Python Forum - How to get filtered result using payload in requests?

I can't get filtered result using payload, I got generic result. In Postman the result is filtered with same payload.

I think I'm doing something wrong, Here is my code:

import requests
import json
import pandas as pd
import time


headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0',
  'Accept': 'application/json, text/plain, */*',
  'Accept-Language': 'en-US,en;q=0.5',
  'Accept-Encoding': 'gzip, deflate, br',
  'Referer': 'https://xxxxxxx',
  'Content-Type': 'application/json;charset=utf-8',
  'Origin': 'https://xxxxx',
  'Connection': 'keep-alive',
  'Cookie': 'ULocation=WW%7Cen; SId=l61eda0332b5f67.36575359_6ef5682a33068fedccdb2eeee5ff26370697b82a; _ga=GA1.2.1306843269.1642962997; _hjSessionUser_1949701=eyJpZCI6IjZiYjUyYjc5LTRkNDAtNTZmNi04OGFhLTlhZDAzYmYzOTNhOSIsImNyZWF0ZWQiOjE2NDI5NjI5OTkxNTMsImV4aXN0aW5nIjp0cnVlfQ==; acceptCookie=1; _gid=GA1.2.115394137.1643138150; _gat=1; _hjSession_1949701=eyJpZCI6ImExNjQ0NWIwLTc3MTgtNDJmNC1iYmViLTg5ZjBmZGY2MzEwYyIsImNyZWF0ZWQiOjE2NDMxMzgxNDk5NTYsImluU2FtcGxlIjpmYWxzZX0=; _hjAbsoluteSessionInProgress=0; _gat_UA-77624103-1=1; ULocation=WW%7Cen',
  'Sec-Fetch-Dest': 'empty',
  'Sec-Fetch-Mode': 'cors',
  'Sec-Fetch-Site': 'same-origin',
  'Cache-Control': 'max-age=0',
  'TE': 'trailers'
}

prods = pd.DataFrame([])

for x in range(1,9):
    url = f"https://xxxxxxx"
    payload = "{\"searchString\":\"\",\"sort\":[\"\",\"\"],\"tmp\":null,\"params\":{\"supplierLocalName\":\"sony\",\"categoryId\":\"1637\"},\"codes\":{\"ean\":[],\"mpn\":[]},\"facetParams\":[{\"name\":\"155712\",\"group\":\"155712\",\"isNum\":false,\"val\":\"wireless\"},{\"name\":\"155712\",\"group\":\"155712\",\"isNum\":false,\"val\":\"wired & wireless\"},{\"name\":\"155712\",\"group\":\"155712\",\"isNum\":false,\"val\":\"true wireless stereo tws\"},{\"name\":\"320952\",\"group\":\"320952\",\"isNum\":false,\"val\":\"headphones\"}],\"p\":{x}}"
    r = requests.post(url, headers=headers, data=payload)
    data = json.loads(r.text)
    prods = prods.append(pd.json_normalize(data['products']), sort=False)
    time.sleep(10)
    print(f'Getting page {x}', 'waiting..')

prods.to_csv('one.csv')

(Feb-06-2022, 12:49 PM)Moto Wrote: [ -> ]payload = "{\"searchString\":\"\",\"sort\":[\"\",\"\"],\"tmp\":null,\"params\":{\"supplierLocalName\":\"sony\",\"categoryId\":\"1637\"},\"codes\":{\"ean\":[],\"mpn\":[]},\"facetParams\":[{\"name\":\"155712\",\"group\":\"155712\",\"isNum\":false,\"val\":\"wireless\"},{\"name\":\"155712\",\"group\":\"155712\",\"isNum\":false,\"val\":\"wired & wireless\"},{\"name\":\"155712\",\"group\":\"155712\",\"isNum\":false,\"val\":\"true wireless stereo tws\"},{\"name\":\"320952\",\"group\":\"320952\",\"isNum\":false,\"val\":\"headphones\"}],\"p\":{x}}"

Payload must be a dictionary. By quoting it you made a string of it. Remove the outer quotes and remove the backslashes.

Thank you for your quick respond.

I remove the outer quotes and the backslashes. but still i get unfiltered data.

But when I use print i get filtered result like:

import requests

url = "https://xxxxxx"

payload = {
    "searchString": "motorola",
    "sort": ["", ""],
    "tmp": None,
    "params": {
        "supplierLocalName": "motorola",
        "keyword": "motorola"
    },
    "codes": {
        "ean": [],
        "mpn": []
    },
    "facetParams": [],
    "p": 0
}
headers = {
    "cookie": "ULocation=WW%7Cen; SId=l61f2c188a5c2f1.67354772_b64ce4c7affb9813b2c09b287df7ec646e32a501; _ga=GA1.2.26180855.1643299209; _hjSessionUser_1949701=eyJpZCI6Ijc3YzQzMjg3LWM5ZDUtNThlNC1hYmZmLWM5MjhkOTAyNTdkYyIsImNyZWF0ZWQiOjE2NDMyOTkyMDk3MjksImV4aXN0aW5nIjp0cnVlfQ==; PHPSESSID=dum577sleve5kc4hbb48p4ij3e; _gid=GA1.2.1356617166.1644147243; _hjAbsoluteSessionInProgress=1; _gat=1; _gat_UA-77624103-1=1; _hjIncludedInPageviewSample=1; _hjSession_1949701=eyJpZCI6IjM4MGRjOWQzLWViMmEtNGMwOS05MzI3LTk4NTBmNTAxYjMxMiIsImNyZWF0ZWQiOjE2NDQyMzMzNjI0NTUsImluU2FtcGxlIjp0cnVlfQ==",
    "authority": "xxxx",
    "pragma": "no-cache",
    "cache-control": "no-cache",
    "accept": "application/json, text/plain, */*",
    "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1",
    "content-type": "application/json;charset=UTF-8",
    "origin": "https://xxxx",
    "sec-fetch-site": "same-origin",
    "sec-fetch-mode": "cors",
    "sec-fetch-dest": "empty",
    "referer": "https://xxxx",
    "accept-language": "en-US,en;q=0.9"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)

any idea?

(Feb-07-2022, 12:08 PM)Moto Wrote: [ -> ]but still i get unfiltered data. But when I use print i get filtered result like:

I don't understand you. How do you know the data is unfiltered as long as you don't print it?

(Feb-13-2022, 05:45 PM)ibreeden Wrote: [ -> ]
(Feb-07-2022, 12:08 PM)Moto Wrote: [ -> ]but still i get unfiltered data. But when I use print i get filtered result like:
I don't understand you. How do you know the data is unfiltered as long as you don't print it?

I successfully export it to CSV file, but the result is not filtered.
When I print it I got filter result.