Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Scrap a dynamic span
#1
Hello Everyone.
I am trying to scrape a website using python. I want to scrape text inside a span, which is inside a div. but the span is thrown automatically there, after every 5 minutes. so I want the script to run forever, and scrap every time the value of span, when it appears.
My code is working fine locally when I have the HTML file in my local machine. but when I try to scrape from the website, it is not working.
Where could be the error?

More details here. Details here



from urllib.request import urlopen
from bs4 import BeautifulSoup
from bs4 import BeautifulSoup as soup  # HTML data structure
from urllib.request import urlopen as uReq  # Web client
import re
import time
import pyperclip


while True:

    page_url = "https://www.example.com/"

    uClient = uReq(page_url)

    page_soup = soup(uClient.read(), "html.parser")

    div = page_soup.find('div',{'class':'content'})

    if div:
        
        span = div.find('span')
        if not span:
            continue
        numbers = span.get_text()
    match = re.search('\d{5,}', numbers)
    card = match.group(0)
    print(card)
    time.sleep(2)



Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  scrap by defining 3 functions zarize 0 260 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 309 Jan-17-2020, 11:51 AM
Last Post: zarize
  Cannot get selenium to scrap past the first two pages newbie_programmer 0 960 Dec-12-2019, 06:19 AM
Last Post: newbie_programmer
  Cannot get contents from ul.li.span.string LLLLLL 8 511 Nov-29-2019, 10:30 AM
Last Post: LLLLLL
  Scrap data from not standarized page? zarize 4 600 Nov-25-2019, 10:25 AM
Last Post: zarize
  page impossible to scrap? :O zarize 2 838 Oct-03-2019, 02:44 PM
Last Post: zarize
  Scrap a value from website harsush 1 380 Aug-29-2019, 01:57 PM
Last Post: snippsat
  Scrap text out of td table from URLS Gochix2020 4 1,268 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  scrap macrotrends mr_byte31 7 2,017 Aug-02-2019, 12:02 AM
Last Post: mr_byte31
  Scrap arbitrage odds -help Gochix2020 3 639 Jul-31-2019, 10:45 AM
Last Post: Gochix2020

Forum Jump:


Users browsing this thread: 1 Guest(s)