Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scrap a dynamic span
#1
Hello Everyone.
I am trying to scrape a website using python. I want to scrape text inside a span, which is inside a div. but the span is thrown automatically there, after every 5 minutes. so I want the script to run forever, and scrap every time the value of span, when it appears.
My code is working fine locally when I have the HTML file in my local machine. but when I try to scrape from the website, it is not working.
Where could be the error?

More details here. Details here

from urllib.request import urlopen
from bs4 import BeautifulSoup
from bs4 import BeautifulSoup as soup  # HTML data structure
from urllib.request import urlopen as uReq  # Web client
import re
import time
import pyperclip


while True:

    page_url = "https://www.example.com/"

    uClient = uReq(page_url)

    page_soup = soup(uClient.read(), "html.parser")

    div = page_soup.find('div',{'class':'content'})

    if div:
        
        span = div.find('span')
        if not span:
            continue
        numbers = span.get_text()
    match = re.search('\d{5,}', numbers)
    card = match.group(0)
    print(card)
    time.sleep(2)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scrap --Need help Lizardpython 4 953 Oct-01-2023, 11:37 AM
Last Post: Lizardpython
  trying to scrape a span inside a div using beautifulsoup CompleteNewb 2 8,795 Jan-28-2021, 01:20 PM
Last Post: snippsat
  I tried every way to scrap morningstar financials data without success so far sparkt 2 8,167 Oct-20-2020, 05:43 PM
Last Post: sparkt
  select all the span text with same attribute JennyYang 2 2,096 Jul-28-2020, 02:56 PM
Last Post: snippsat
  Web scrap multiple pages anilacem_302 3 3,783 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  Need logic on how to scrap 100K URLs goodmind 2 2,569 Jun-29-2020, 09:53 AM
Last Post: goodmind
  scrap by defining 3 functions zarize 0 1,833 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 1,852 Jan-17-2020, 11:51 AM
Last Post: zarize
  Cannot get selenium to scrap past the first two pages newbie_programmer 0 4,131 Dec-12-2019, 06:19 AM
Last Post: newbie_programmer
  Cannot get contents from ul.li.span.string LLLLLL 8 3,949 Nov-29-2019, 10:30 AM
Last Post: LLLLLL

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020