Python Forum
Scraping problems. Pls help with a correct request query.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping problems. Pls help with a correct request query.
#1
Please help to formulate the correct request query. Thank you.
.
Here is the website
http://www.eatonpowersource.com/cross-reference/
.
one needs to enter a search parameter in the form
[Image: pic1.png]
.
This is how the website looks like
.
.
[Image: pic2.png]
.
in Mozilla there are two variants in the field analysis: POST and GET

Method : POST
Link: http://www.eatonpowersource.com/cross-re...e/results/
Parameters
Criteria.SiteSearchTerm
Criteria.CurrentPageNumber=1
Criteria.FilterOptions.SortBy=CompetitorPartNumber
Criteria.FilterOptions.SortOrder=Asc
Criteria.CompetitorPartNumber=0330D0
Criteria.FilterOptions.PageSize=25

and
Method: GET
Link:
http://www.eatonpowersource.com/cross-re...9680055925

Параметры
Criteria.SiteSearchTerm=
Criteria.CurrentPageNumber=1
Criteria.FilterOptions.SortBy=CompetitorPartNumber
Criteria.FilterOptions.SortOrder=Asc
Criteria.CompetitorPartNumber=0330D0
Criteria.FilterOptions.PageSize=25
_=1569680055925

If I send a request query with GET parameters and method, I get the following
[Image: pic3.png]
this is not what is needed

If I send a request query with POST parameters and method, then I get
either 404 - page not found
or error 500
.
# function for web query and recording into a file
def fg_list_bot(_name_element, _output_file):
    s = requests.Session()
    _data = {"Criteria.SiteSearchTerm":"",
             "Criteria.CurrentPageNumber":"1",
             "Criteria.FilterOptions.SortBy":"CompetitorPartNumber",
             "Criteria.FilterOptions.SortOrder":"Asc",
             "Criteria.CompetitorPartNumber":_name_element,
             "Criteria.FilterOptions.PageSize":"25",
             "_":"1569680055925"}

    r = requests.post("_Url", data=_data)
    with open(_output_file, "w") as f: f.write(r.text)
    print(r.status_code)
    input()


import requests
from bs4 import BeautifulSoup
_url = "http://www.eatonpowersource.com/cross-reference/results/"
_name_element = "0330D0"      # text of the search request

_output_file = "Eaton_Vickers.html"
fg_list_bot(_name_element, _output_file)
Reply


Messages In This Thread
Scraping problems. Pls help with a correct request query. - by gtlhbkkj - Sep-29-2019, 02:46 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  POST request with form data issue web scraping hoff1022 1 2,715 Aug-14-2020, 10:25 AM
Last Post: kashcode
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,906 Aug-14-2020, 10:13 AM
Last Post: kashcode
  The correct POST request abhie_lp 5 3,019 Jun-05-2020, 07:27 AM
Last Post: buran
  Scraping problems with Python requests. gtlhbkkj 1 1,895 Jan-22-2020, 11:00 AM
Last Post: gtlhbkkj
  Scraping problems. Pls help with a correct request query. gtlhbkkj 0 1,523 Oct-09-2019, 12:00 PM
Last Post: gtlhbkkj
  web scraping to csv formatting problems bluethundr 4 2,808 Jul-04-2019, 02:00 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020