POST request with form data issue web scraping - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: POST request with form data issue web scraping (/thread-25820.html) |
POST request with form data issue web scraping - hoff1022 - Apr-12-2020 Please help. I'm trying to create a POST request on an .asp site that requires cookies, but the way I handle them seems not to return anything. Read through some questions of similar topic but can't find the _SessionID cookie some are referring to. Please help me formulate this POST request so it works. HEADERS: :authority: safer.fmcsa.dot.gov :method: POST :path: /query.asp :scheme: https accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 accept-encoding: gzip, deflate, br accept-language: en-US,en;q=0.9 cache-control: max-age=0 content-length: 85 content-type: application/x-www-form-urlencoded cookie: ASP.NET_SessionId=ywxszihqlu1yciwe5z5gm4qt; etype=au; ASPSESSIONIDQECTCDRB=KGFOBHBBLKCBKBFIAPEBMIHJ; ASPSESSIONIDQGARCDRA=LKEDBAOBMOMDNGBNBFEMMIPB; ASPSESSIONIDSEBQCBSB=DAMJMNKCNJKHCMDCIJBPKEHD; ASPSESSIONIDCEQRADQC=EIEJCDLBHHCCKHCNJNIMHDKA; ASPSESSIONIDAESTCBQC=KPDPJNHCLOBJENEHPNIFKJLH; LI_carrier=67449; ASPSESSIONIDAGSQADRC=CPIBAKEDDPNFCIPLIGLOKKLA; ASPSESSIONIDAERTDBQD=FMKFHJJAJKNIGCBCCFJFCMNF; AWSALB=Xc7OAuZUmx6vgE5l9NaawsH8oBWjy6eZ3B62kw2rZ5HieoRlMu4SSmVVcaPJPcPjp1fVt9U/T9FaRflgNHwtzmsK4X4e+y+yoGArTfgpb75NWo/ilAek0Qk/sFYI; AWSALBCORS=Xc7OAuZUmx6vgE5l9NaawsH8oBWjy6eZ3B62kw2rZ5HieoRlMu4SSmVVcaPJPcPjp1fVt9U/T9FaRflgNHwtzmsK4X4e+y+yoGArTfgpb75NWo/ilAek0Qk/sFYI origin: https://safer.fmcsa.dot.gov referer: https://safer.fmcsa.dot.gov/CompanySnapshot.aspx sec-fetch-dest: document sec-fetch-mode: navigate sec-fetch-site: same-origin sec-fetch-user: ?1 upgrade-insecure-requests: 1 user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36 FORM DATA searchtype: ANY query_type: queryCarrierSnapshot query_param: USDOT query_string: 2300842 MY CODE SO FAR: def checkDOT(): headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'Accept-Language': 'en-US,en;q=0.9', 'Cache-Control': 'no-cache,no-store,must-revalidate,max-age=0,private', 'upgrade-insecure-requests': '1', 'Connection': 'keep-alive', 'origin': 'https://safer.fmcsa.dot.gov', 'referer': 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx' } s = requests.Session() data = { 'searchtype': 'ANY', 'query_type': 'queryCarrierSnapshot', 'query_param': 'USDOT', 'query_string': '2300842' } params = ( ('pageNumber', '0'), ('itemsPerPage', '15'), ) url = 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx' response = s.get(url, headers=headers, data=data, params=params) if response: print(response.content) else: print("This did not work") RE: POST request with form data issue web scraping - kashcode - Aug-14-2020 On line 27 you send GET not POST |