Python Forum
Scraping problems. Pls help with a correct request query.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping problems. Pls help with a correct request query.
#1
Please do not suggest SELENIUM solutions.

the request in this form does not return results.

def fg_list_bot(_name_element, _output_file):
    print(_name_element)
    s = requests.Session()
    _data = {"searchValue": _name_element,"searchSubmit":"submit[s][/s]"}
    _headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"}  
    r = requests.post(_url, data=_data, headers = _headers)
    with open(_output_file, "w", encoding='utf-8') as f:
        f.write(r.text)
    print(r.status_code)

#
# code beginn
#
import requests
from bs4 import BeautifulSoup
_url = "https://hifi-filter.com/en/catalog/recherche-equivalence.html"
_output_file = "IDLE_HIFI.html"
_name_element = "pi3115"   
fg_list_bot(_name_element, _output_file)
if I add captcha, as below, then it works for some time with different search parameters, when the captcha is fresh.
After some time it does not return results, like the first variant.

def fg_list_bot(_name_element, _output_file):
    print(_name_element)
    s = requests.Session()
_data = {"searchValue": _name_element,"searchSubmit":"submit",
             "g-recaptcha-response":"03AOLTBLSGgHKYeeU_WgH-tOhoUV8UXkBejUCAhxgfuyBKE0QA0PeDOcTlrhTd0zlhTyCVIjjkZrfxWBnBfd6R5_G_XU15ZN8s3nqHljYjvXMHpijXj4TZUIu0t_hBHu65rJb7op28Iz1EplJxP0lbfXJbm3Mif-O6jg-eXb-v_spSH4W2aW4nSvMMrHGy-7iJpOns4O-Ff-P2kit_E7jbrKF6jakyR1f0FlcLGFHAPNaf0w2BhnXvxlFmo6ghDR58jqJmWiRRj0BK8nAMIw0FVI4J1j3hoWDxxNX6bnHXxw-mQb-FEhwM4oHMVCvj-NqzG2gX__H9AXuSU7Ehnl9YwtMi3ssW6V4FuEmVIwpZDPy-nIfSdi7NyuycZj6tgLFyKfefj91oaWCNoNqH48I0MfE6zkfim7KlTfbG0LxGIFpH4MMH1_iNunJ0LJU9s_o8jUA3HP5bL-1jPVAbFC6pnxe07GmyKaSutQ"}    _headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"}  
    r = requests.post(_url, data=_data, headers = _headers)
    with open(_output_file, "w", encoding='utf-8') as f:
        f.write(r.text)
    print(r.status_code)

#
# code beginn
#
import requests
from bs4 import BeautifulSoup
_url = "https://hifi-filter.com/en/catalog/recherche-equivalence.html"
_output_file = "IDLE_HIFI.html"
_name_element = "pi3115"   
fg_list_bot(_name_element, _output_file)
[Image: q02.png]

Thank you !!!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  POST request with form data issue web scraping hoff1022 1 2,703 Aug-14-2020, 10:25 AM
Last Post: kashcode
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,892 Aug-14-2020, 10:13 AM
Last Post: kashcode
  The correct POST request abhie_lp 5 3,001 Jun-05-2020, 07:27 AM
Last Post: buran
  Scraping problems with Python requests. gtlhbkkj 1 1,881 Jan-22-2020, 11:00 AM
Last Post: gtlhbkkj
  Scraping problems. Pls help with a correct request query. gtlhbkkj 6 3,094 Oct-01-2019, 09:22 PM
Last Post: gtlhbkkj
  web scraping to csv formatting problems bluethundr 4 2,790 Jul-04-2019, 02:00 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020