Python Forum
Help: Beautiful Soup - Parsing HTML table
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Help: Beautiful Soup - Parsing HTML table
#1
Hi,

I am trying to get a table from this link:
https://www.wunderground.com/history/dai...e/2018-6-1

And the table I am trying to get data about is the "Daily Observations" table.

No matter what I do, I simply can't get any data relevant to that. I've tried...

page_soup = soup(page,'html.parser')
data = page_soup.findAll("table",{"class":"mat-table cdk-table mat-sort ng-star-inserted"})
print(data)
As well as several other findAll() attributes and keep getting an empty list returned.

Help is greatly appreciated. Thanks!
Reply
#2
there must be more code than this.
show it all.
finding the table is just the start, now you need to parse what you have found.
Suggest taking the web scraping tutorials part 1 and 2 on this site in the tutorials forum.
Reply
#3
That site us JavaScript heavily,so use Selenium.
Here a example there are may more here if you search forum.

Also if give browser.page_source to Pandas to pandas.read_html,it will pull out whole table without doing any parsing.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Suggestion request for scrapping html table Vkkindia 3 2,024 Dec-06-2021, 06:09 PM
Last Post: Larz60+
  Beautiful Soup - access a rating value in a class KatMac 1 3,452 Apr-16-2021, 01:27 PM
Last Post: snippsat
  HTML multi select HTML listbox with Flask/Python rfeyer 0 4,617 Mar-14-2021, 12:23 PM
Last Post: rfeyer
  Parsing html page and working with checkbox (on a captcha) straannick 17 11,260 Feb-04-2021, 02:54 PM
Last Post: snippsat
  *Beginner* web scraping/Beautiful Soup help 7ken8 2 2,598 Jan-28-2021, 04:26 PM
Last Post: 7ken8
  Beautiful Soup (suddenly) doesn't get full webpage html j.crater 8 16,779 Jul-11-2020, 04:31 PM
Last Post: j.crater
  Requests-HTML vs Beautiful Soup - How to Choose? robin73 0 3,809 Jun-23-2020, 02:53 PM
Last Post: robin73
  Python3 + BeautifulSoup4 + lxml (HTML -> CSV) - How to loop to next HTML/new CSV Row BrandonKastning 0 2,353 Mar-22-2020, 06:10 AM
Last Post: BrandonKastning
  looking for direction - scrappy, crawler, beautiful soup Sly_Corn 2 2,445 Mar-17-2020, 03:17 PM
Last Post: Sly_Corn
  Beautiful soup truncates results jonesjoz 4 3,861 Mar-09-2020, 06:04 PM
Last Post: jonesjoz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020