Python Forum
Scraping problems. Pls help with a correct request query.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping problems. Pls help with a correct request query.
#1
Please do not suggest SELENIUM solutions.

the request in this form does not return results.

def fg_list_bot(_name_element, _output_file):
    print(_name_element)
    s = requests.Session()
    _data = {"searchValue": _name_element,"searchSubmit":"submit[s][/s]"}
    _headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"}  
    r = requests.post(_url, data=_data, headers = _headers)
    with open(_output_file, "w", encoding='utf-8') as f:
        f.write(r.text)
    print(r.status_code)

#
# code beginn
#
import requests
from bs4 import BeautifulSoup
_url = "https://hifi-filter.com/en/catalog/recherche-equivalence.html"
_output_file = "IDLE_HIFI.html"
_name_element = "pi3115"   
fg_list_bot(_name_element, _output_file)
if I add captcha, as below, then it works for some time with different search parameters, when the captcha is fresh.
After some time it does not return results, like the first variant.

def fg_list_bot(_name_element, _output_file):
    print(_name_element)
    s = requests.Session()
_data = {"searchValue": _name_element,"searchSubmit":"submit",
             "g-recaptcha-response":"03AOLTBLSGgHKYeeU_WgH-tOhoUV8UXkBejUCAhxgfuyBKE0QA0PeDOcTlrhTd0zlhTyCVIjjkZrfxWBnBfd6R5_G_XU15ZN8s3nqHljYjvXMHpijXj4TZUIu0t_hBHu65rJb7op28Iz1EplJxP0lbfXJbm3Mif-O6jg-eXb-v_spSH4W2aW4nSvMMrHGy-7iJpOns4O-Ff-P2kit_E7jbrKF6jakyR1f0FlcLGFHAPNaf0w2BhnXvxlFmo6ghDR58jqJmWiRRj0BK8nAMIw0FVI4J1j3hoWDxxNX6bnHXxw-mQb-FEhwM4oHMVCvj-NqzG2gX__H9AXuSU7Ehnl9YwtMi3ssW6V4FuEmVIwpZDPy-nIfSdi7NyuycZj6tgLFyKfefj91oaWCNoNqH48I0MfE6zkfim7KlTfbG0LxGIFpH4MMH1_iNunJ0LJU9s_o8jUA3HP5bL-1jPVAbFC6pnxe07GmyKaSutQ"}    _headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"}  
    r = requests.post(_url, data=_data, headers = _headers)
    with open(_output_file, "w", encoding='utf-8') as f:
        f.write(r.text)
    print(r.status_code)

#
# code beginn
#
import requests
from bs4 import BeautifulSoup
_url = "https://hifi-filter.com/en/catalog/recherche-equivalence.html"
_output_file = "IDLE_HIFI.html"
_name_element = "pi3115"   
fg_list_bot(_name_element, _output_file)
[Image: q02.png]

Thank you !!!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  POST request with form data issue web scraping hoff1022 1 2,705 Aug-14-2020, 10:25 AM
Last Post: kashcode
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,893 Aug-14-2020, 10:13 AM
Last Post: kashcode
  The correct POST request abhie_lp 5 3,003 Jun-05-2020, 07:27 AM
Last Post: buran
  Scraping problems with Python requests. gtlhbkkj 1 1,885 Jan-22-2020, 11:00 AM
Last Post: gtlhbkkj
  Scraping problems. Pls help with a correct request query. gtlhbkkj 6 3,095 Oct-01-2019, 09:22 PM
Last Post: gtlhbkkj
  web scraping to csv formatting problems bluethundr 4 2,793 Jul-04-2019, 02:00 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020