Python Forum

Full Version: Scrap a dynamic span
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello Everyone.
I am trying to scrape a website using python. I want to scrape text inside a span, which is inside a div. but the span is thrown automatically there, after every 5 minutes. so I want the script to run forever, and scrap every time the value of span, when it appears.
My code is working fine locally when I have the HTML file in my local machine. but when I try to scrape from the website, it is not working.
Where could be the error?

More details here. Details here

from urllib.request import urlopen
from bs4 import BeautifulSoup
from bs4 import BeautifulSoup as soup  # HTML data structure
from urllib.request import urlopen as uReq  # Web client
import re
import time
import pyperclip


while True:

    page_url = "https://www.example.com/"

    uClient = uReq(page_url)

    page_soup = soup(uClient.read(), "html.parser")

    div = page_soup.find('div',{'class':'content'})

    if div:
        
        span = div.find('span')
        if not span:
            continue
        numbers = span.get_text()
    match = re.search('\d{5,}', numbers)
    card = match.group(0)
    print(card)
    time.sleep(2)