Python Forum
scraping javascript websites with selenium
Thread Rating:
  • 2 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scraping javascript websites with selenium
#2
Let me answer my own question. This seems to be the fastest way for scraping.

from lxml import html
# open some link and wait
data = []
innerHTML = driver.execute_script("return document.body.innerHTML")
htmlElem = html.document_fromstring(innerHTML)
tables = htmlElem.xpath("//div[@id='match-history-content']/div[contains(@id, 'tab-mhistory-')]/table/tbody")
for table in tables:
    this_table = []
    for row in table.xpath(".//tr"):
        this_row = []
        for elm in row.xpath(".//td"):
            this_row.append(elm.text_content())
        this_table.append(this_row)
    data.append(this_table) 
Reply


Messages In This Thread
RE: scraping javascript websites with selenium - by DoctorEvil - Jun-08-2018, 06:40 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscrapping sport betting websites KoinKoin 3 5,495 Nov-08-2023, 03:00 PM
Last Post: LoriBrown
  Scraping div tags with selenium, need help hfakoor2 1 1,081 Mar-12-2023, 08:31 AM
Last Post: hfakoor2
  Web Scraping Sportsbook Websites Khuber79 17 307,991 Mar-17-2021, 12:06 AM
Last Post: Whitesox1
  Web scraping cookie in URL blocks selenium Alex06 2 2,460 Jan-10-2021, 01:43 PM
Last Post: Alex06
  Using Python request without selenium on html form with javascript onclick submit but eraosa 0 3,197 Jan-09-2021, 06:08 PM
Last Post: eraosa
Thumbs Up Issue facing while scraping the data from different websites in single script. Balamani 1 2,139 Oct-20-2020, 09:56 AM
Last Post: Larz60+
  Can urlopen be blocked by websites? peterjv26 2 3,414 Jul-26-2020, 06:45 PM
Last Post: peterjv26
  Python program to write into websites for you pythonDEV333 3 2,537 Jun-08-2020, 12:06 PM
Last Post: pythonDEV333
  question about using javascript on python selenium Kai 1 1,912 Apr-12-2020, 04:28 AM
Last Post: Larz60+
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 7,262 Dec-27-2019, 12:32 PM
Last Post: JuanJuan

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020