Web Scraper with BeautifulSoup4 sometimes no Output

[Nitsuj]

Hello,

I need some help with my Python-Script:
I want to make a web scraper to scrape some prices from this website:
https://www.medizinfuchs.de/?params%5Bse...h_cat%5D=1
I wrote following code:

1 2 3 4 5 6 7 8 9 10 11 12 13

from bs4 import BeautifulSoup import requests   URL = "https://www.medizinfuchs.de/?params%5Bsearch%5D=10714367&params%5Bsearch_cat%5D=1"   my_headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36", "Accept":"text/html,application/xhtml+xml,application/xml; q=0.9,image/webp,image/apng,*/*;q=0.8"}     page = requests.get(URL, headers=my_headers)   soup = BeautifulSoup(page.text, "lxml")   for price in soup.select("ul.Apothekenliste div.price"):         print(float(price.text.strip(' \t\n€').replace(',', '.')))

It works sometimes - but too inconsistent.
I really don't know what I should be doing different.

Thanks for your help!

[ghoul]

Well, for me it does not seem to work at all.

[snippsat]

It work something but not stable,like this it's a little more stable.
It may bye more stable using Selenium in headless mode,
but code under should work ok.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

from bs4 import BeautifulSoup import requests import time   URL = "https://www.medizinfuchs.de/?params%5Bsearch%5D=10714367&params%5Bsearch_cat%5D=1" my_headers = {     "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",     "Accept": "text/html,application/xhtml+xml,application/xml; q=0.9,image/webp,image/apng,*/*;q=0.8", } page = requests.get(URL, headers=my_headers) soup = BeautifulSoup(page.text, "lxml") time.sleep(5) price_lst = soup.find_all('div', class_="col-xs-24 single") for price in price_lst:     print(price.text.strip())

Output:
6,81 €
6,84 €
7,14 €
7,23 €
7,36 €
7,39 €
7,53 €
7,60 €

I think is want continuous checker over time should use something like schedule,or set it up schedule at OS level.