Python Forum
POST request with form data issue web scraping
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
POST request with form data issue web scraping
#1
Please help. I'm trying to create a POST request on an .asp site that requires cookies, but the way I handle them seems not to return anything. Read through some questions of similar topic but can't find the _SessionID cookie some are referring to. Please help me formulate this POST request so it works.

HEADERS:

:authority: safer.fmcsa.dot.gov
:method: POST
:path: /query.asp
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
content-length: 85
content-type: application/x-www-form-urlencoded
cookie: ASP.NET_SessionId=ywxszihqlu1yciwe5z5gm4qt; etype=au; ASPSESSIONIDQECTCDRB=KGFOBHBBLKCBKBFIAPEBMIHJ; ASPSESSIONIDQGARCDRA=LKEDBAOBMOMDNGBNBFEMMIPB; ASPSESSIONIDSEBQCBSB=DAMJMNKCNJKHCMDCIJBPKEHD; ASPSESSIONIDCEQRADQC=EIEJCDLBHHCCKHCNJNIMHDKA; ASPSESSIONIDAESTCBQC=KPDPJNHCLOBJENEHPNIFKJLH; LI_carrier=67449; ASPSESSIONIDAGSQADRC=CPIBAKEDDPNFCIPLIGLOKKLA; ASPSESSIONIDAERTDBQD=FMKFHJJAJKNIGCBCCFJFCMNF; AWSALB=Xc7OAuZUmx6vgE5l9NaawsH8oBWjy6eZ3B62kw2rZ5HieoRlMu4SSmVVcaPJPcPjp1fVt9U/T9FaRflgNHwtzmsK4X4e+y+yoGArTfgpb75NWo/ilAek0Qk/sFYI; AWSALBCORS=Xc7OAuZUmx6vgE5l9NaawsH8oBWjy6eZ3B62kw2rZ5HieoRlMu4SSmVVcaPJPcPjp1fVt9U/T9FaRflgNHwtzmsK4X4e+y+yoGArTfgpb75NWo/ilAek0Qk/sFYI
origin: https://safer.fmcsa.dot.gov
referer: https://safer.fmcsa.dot.gov/CompanySnapshot.aspx
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36


FORM DATA

searchtype: ANY
query_type: queryCarrierSnapshot
query_param: USDOT
query_string: 2300842



MY CODE SO FAR:

def checkDOT():
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'Accept-Language': 'en-US,en;q=0.9',
        'Cache-Control': 'no-cache,no-store,must-revalidate,max-age=0,private',
        'upgrade-insecure-requests': '1',
        'Connection': 'keep-alive',
        'origin': 'https://safer.fmcsa.dot.gov',
        'referer': 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx'
    }

    s = requests.Session()

    data = {
        'searchtype': 'ANY',
        'query_type': 'queryCarrierSnapshot',
        'query_param': 'USDOT',
        'query_string': '2300842'
    }
    params = (
        ('pageNumber', '0'),
        ('itemsPerPage', '15'),
    )

    url = 'https://safer.fmcsa.dot.gov/CompanySnapshot.aspx'
    response = s.get(url, headers=headers, data=data, params=params)

    if response:
        print(response.content)

    else:
        print("This did not work")
Reply
#2
On line 27 you send GET not POST
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Size scraping issue scrapemasta 0 356 Feb-09-2024, 10:26 AM
Last Post: scrapemasta
  How to scraping data from dinamic site sergio21124444 2 675 Nov-08-2023, 12:43 PM
Last Post: sergio21124444
Question Python request (post/get) Drunknmonkie 1 2,652 Jan-19-2023, 02:02 PM
Last Post: prvncpa
  Post request not saving in DRF Dexty 0 916 Jun-03-2022, 12:35 PM
Last Post: Dexty
  Scraping data from table into existing dataframe vincer58 1 2,003 Jan-09-2022, 05:15 PM
Last Post: vincer58
Brick how to work with 2 or more related tables in a Post request ikurorox 0 1,392 Dec-05-2021, 01:01 AM
Last Post: ikurorox
  Post HTML Form Data to API Endpoints Dexty 0 1,392 Nov-11-2021, 10:51 PM
Last Post: Dexty
  Web scraping data Mike_Eddy 2 2,530 Jul-03-2021, 05:49 PM
Last Post: Mike_Eddy
  Scraping lender data from Ren Ren Dai website using Python. I will pay for that 200$ Hafedh_2021 1 2,752 May-18-2021, 08:41 PM
Last Post: snippsat
  Scraping a page with log in data (security, proxies) iamaghost 0 2,139 Mar-27-2021, 02:56 PM
Last Post: iamaghost

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020