Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web scraping "fancy" table
#1
Hi guys,
I am just getting into Python and web scraping, so I am very sorry if this is a basic question. So far, I have been able to scrape information from tables from various website. However, when I want to get access to the prices at this web site nordpoolspot.com/Market-data1/#/nordic/table I run into trouble.

Is it because the page and table are "fancy"? I can't seem to access the data in the usual way -- in particular, I do not seem to have access to the whole page in my "soup" variable. Below is some code which I have tried. 

import requests, bs4

url = 'www.nordpoolspot.com/Market-data1/#/nordic/table' # <-- include http (I can't post links before I am a proven non-spammer)

res = requests.get(url)
res.raise_for_status()
    
soup = bs4.BeautifulSoup(res.content,"lxml")
tmp = soup.select('tr td')

print(len(tmp))
# Output: 0
Reply
#2
(Dec-13-2016, 02:53 PM)acehole60 Wrote: Is it because the page and table are "fancy"? I can't seem to access the data in the usual way -- in particular
It's because JavaScript in this case they use AngularJS for the table.
JavaScript is very common in web-development,but Requests can read what is rendered/executed in DOM.

I talk about tool here Web-scraping part-2
So tool is Selenium and PhantomJs,in tutorial i pass in some values to site.
You don't need need that,just take browser.page_source and pass it to BS.
PhantomJs is for not loading a browser,and is what you want to here(just want source with JavaScript).

I do work i did i quick test.
Here tmp[0:5] output in a Pen.
Reply
#3
Thanks a lot!!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help Scraping links and table from link cartonics 11 1,462 Oct-12-2023, 06:42 AM
Last Post: cartonics
  Scraping data from table into existing dataframe vincer58 1 1,960 Jan-09-2022, 05:15 PM
Last Post: vincer58
  Need help scraping wikipedia table bborusz2 6 3,167 Dec-01-2020, 11:31 PM
Last Post: snippsat
  Web Scraping Inquiry (Extracting content from a table in asubdomain) DustinKlent 3 3,666 Aug-17-2020, 10:10 AM
Last Post: snippsat
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,823 Aug-14-2020, 10:13 AM
Last Post: kashcode
  scraping multiple pages from table bandar 1 2,651 Jun-27-2020, 10:43 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020