Python Forum
Python Web Scraping can not getting all HTML content
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python Web Scraping can not getting all HTML content
#1
I am very new to web scraping in Python, and I am using BeautifulSoup for parsing. After obtaining the HTML data, I will try to access some content under "< div id="root">.</div>", but I don’t have all the HTML sites that will be displayed when I actually click "Check". How can I access, or is the way the website prevents me from accessing the information on the page?

If it doesn’t make sense, I’m going to say "." in that div instead of more subcategories for me to view (I will see when I click "check" on the web page).

This is my beautiful soup code... Undecided Undecided

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

myurl = 'https://www.coolbet.com/en/sports/incoming-bets'

#open connecting and grab content
uClient = uReq(myurl)
page_html = uClient.read()
uClient.close()

#html parsing
page_soup = soup(page_html, "html.parser")

#grabs each product
containers = page_soup.div.findAll("div", {"class":"sc-iuJeZd iJcGXh"})

print(containers)
Larz60+ write Aug-02-2021, 09:14 AM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.
fixed for you this time. Please use bbcode tags on future posts.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Retrieve website content using Python? Vadanane 1 1,196 Jan-16-2023, 09:55 AM
Last Post: Axel_Erfurt
  Python Obstacles | Krav Maga | Wiki Scraped Content [Column Copy] BrandonKastning 4 2,161 Jan-03-2022, 06:59 AM
Last Post: BrandonKastning
  Python Obstacles | Kapap | Wiki Scraped Content [Column Nulling] BrandonKastning 2 1,687 Jan-03-2022, 04:26 AM
Last Post: BrandonKastning
  Scraping the page without distorting content oleglpts 5 2,442 Dec-16-2021, 05:08 PM
Last Post: oleglpts
Sad web scraping HTML - :( Kingoman 22 6,206 Apr-05-2021, 09:50 AM
Last Post: snippsat
  HTML multi select HTML listbox with Flask/Python rfeyer 0 4,529 Mar-14-2021, 12:23 PM
Last Post: rfeyer
  Web Scraping Inquiry (Extracting content from a table in asubdomain) DustinKlent 3 3,663 Aug-17-2020, 10:10 AM
Last Post: snippsat
  Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row BrandonKastning 0 2,329 Mar-22-2020, 06:10 AM
Last Post: BrandonKastning
  requests problem in python script "content type" abdlwafitahiri 4 3,122 Dec-29-2019, 02:29 PM
Last Post: abdlwafitahiri
  Web scraping User Generated Content StephenG93 2 2,915 Oct-10-2018, 12:17 AM
Last Post: StephenG93

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020