Python Forum
scraping javascript websites with selenium
Thread Rating:
  • 2 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scraping javascript websites with selenium
#1
Hello, I hope this is the right subforum for this. I'm trying to scrape a website like this https://www.rezultati.com/utakmica/OUNkA...jek-meca;1

Right now, the best I can do with the code is this - classical search by xpath and then get text.
data=[]
tables = driver.find_elements_by_xpath("//div[@id='match-history-content']/div[contains(@id, 'tab-mhistory-')]/table/tbody")
for table in tables:
	this_table = []
	for row in table.find_elements_by_xpath(".//tr"):
		this_row = []
		for one_element in row.find_elements_by_xpath(".//td"):
			this_row.append(one_element.get_attribute("innerText"))
		this_table.append(this_row)
	data.append(this_table)

# parse...
Is there an another way to scrape this by using requests to directly get some kind of json format? When I open "Networking" tab in Chrome, response for every request is some kind of javascript code. Is the thing I'm asking for even possible to do for this particular website, or any other? How hard would you say this is, maybe it's best for me to just keep on doing it like with the above code...

Thanks for your help!
Reply
#2
Let me answer my own question. This seems to be the fastest way for scraping.

from lxml import html
# open some link and wait
data = []
innerHTML = driver.execute_script("return document.body.innerHTML")
htmlElem = html.document_fromstring(innerHTML)
tables = htmlElem.xpath("//div[@id='match-history-content']/div[contains(@id, 'tab-mhistory-')]/table/tbody")
for table in tables:
    this_table = []
    for row in table.xpath(".//tr"):
        this_row = []
        for elm in row.xpath(".//td"):
            this_row.append(elm.text_content())
        this_table.append(this_row)
    data.append(this_table) 
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscrapping sport betting websites KoinKoin 3 5,337 Nov-08-2023, 03:00 PM
Last Post: LoriBrown
  Scraping div tags with selenium, need help hfakoor2 1 1,032 Mar-12-2023, 08:31 AM
Last Post: hfakoor2
  Web Scraping Sportsbook Websites Khuber79 17 255,937 Mar-17-2021, 12:06 AM
Last Post: Whitesox1
  Web scraping cookie in URL blocks selenium Alex06 2 2,384 Jan-10-2021, 01:43 PM
Last Post: Alex06
  Using Python request without selenium on html form with javascript onclick submit but eraosa 0 3,135 Jan-09-2021, 06:08 PM
Last Post: eraosa
Thumbs Up Issue facing while scraping the data from different websites in single script. Balamani 1 2,075 Oct-20-2020, 09:56 AM
Last Post: Larz60+
  Can urlopen be blocked by websites? peterjv26 2 3,322 Jul-26-2020, 06:45 PM
Last Post: peterjv26
  Python program to write into websites for you pythonDEV333 3 2,449 Jun-08-2020, 12:06 PM
Last Post: pythonDEV333
  question about using javascript on python selenium Kai 1 1,853 Apr-12-2020, 04:28 AM
Last Post: Larz60+
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 7,058 Dec-27-2019, 12:32 PM
Last Post: JuanJuan

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020