Python Forum
Webscraping a site that uses javascript?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Webscraping a site that uses javascript?
#2
(Apr-29-2017, 08:03 PM)bigmit37 Wrote: So It seems that the site uses Javascript., but I have been scouring a webscraping book (old book) and the net, but I can't seem to figure out what I'm supposed to do. 
Do I need a different module? 
Yes if the site uses javascript it changes on the fly, so the data doesnt exist when regular urllib.request. You need to automate a browser by using selenium. IF you look at the HTML that you browser gets and the HTML that python gets, and its different you need to use selenium. You can use PhantomJS to "hide" the browser in the background so to speak. All you really need to do since your just parsing the content is just get the HTML via selenium instead, and then pass that to BeuatifulSoup
Recommended Tutorials:
Reply


Messages In This Thread
RE: Webscraping a site that uses javascript? - by metulburr - Apr-29-2017, 08:30 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscraping news articles by using selenium cate16 7 3,270 Aug-28-2023, 09:58 AM
Last Post: snippsat
  Webscraping with beautifulsoup cormanstan 3 2,096 Aug-24-2023, 11:57 AM
Last Post: snippsat
  Webscraping returning empty table Buuuwq 0 1,441 Dec-09-2022, 10:41 AM
Last Post: Buuuwq
  WebScraping using Selenium library Korgik 0 1,073 Dec-09-2022, 09:51 AM
Last Post: Korgik
  How to get rid of numerical tokens in output (webscraping issue)? jps2020 0 1,979 Oct-26-2020, 05:37 PM
Last Post: jps2020
  Python Webscraping with a Login Website warriordazza 0 2,635 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Help with basic webscraping Captain_Snuggle 2 3,993 Nov-07-2019, 08:07 PM
Last Post: kozaizsvemira
  Can't Resolve Webscraping AttributeError Hass 1 2,344 Jan-15-2019, 09:36 PM
Last Post: nilamo
  How to exclude certain links while webscraping basis on keywords Prince_Bhatia 0 3,271 Oct-31-2018, 07:00 AM
Last Post: Prince_Bhatia
  Webscraping homework Ghigo1995 1 2,681 Sep-23-2018, 07:36 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020