Python Forum
Web Scraper with BeautifulSoup4 sometimes no Output
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web Scraper with BeautifulSoup4 sometimes no Output
#3
It work something but not stable,like this it's a little more stable.
It may bye more stable using Selenium in headless mode,
but code under should work ok.
from bs4 import BeautifulSoup
import requests
import time

URL = "https://www.medizinfuchs.de/?params%5Bsearch%5D=10714367&params%5Bsearch_cat%5D=1"
my_headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml; q=0.9,image/webp,image/apng,*/*;q=0.8",
}
page = requests.get(URL, headers=my_headers)
soup = BeautifulSoup(page.text, "lxml")
time.sleep(5)
price_lst = soup.find_all('div', class_="col-xs-24 single")
for price in price_lst:
    print(price.text.strip())
 
Output:
6,81 € 6,84 € 7,14 € 7,23 € 7,36 € 7,39 € 7,53 € 7,60 €
I think is want continuous checker over time should use something like schedule,or set it up schedule at OS level.
Reply


Messages In This Thread
RE: Web Scraper with BeautifulSoup4 sometimes no Output - by snippsat - Nov-26-2021, 05:04 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  How to make scraper send email notification just once themech25 0 1,405 Nov-08-2021, 01:51 PM
Last Post: themech25
  bs4 beautifulsoup4 "module not found" but I confirmed it is installed mjglsn 1 16,444 Feb-01-2020, 12:12 AM
Last Post: metulburr
  A mail scraper for sorting emails into categories anata2047 1 5,510 Sep-03-2019, 11:03 PM
Last Post: micseydel

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020