Python Forum
Extract data with Selenium and BeautifulSoup
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extract data with Selenium and BeautifulSoup
#1
Hi, I'm trying to extract live betting data from this sports betting site, https://bwin.com this is the code I use
from selenium import webdriver
from bs4 import BeautifulSoup
import time
 

browser = webdriver.Chrome()
url = 'https://sports.bwin.com/sports/events/benfica-tondela-10089945'
browser.get(url)
time.sleep(3)
soup = BeautifulSoup(browser.page_source, 'html.parser')
events = []

for i in soup.select('div[class="name"]'):
    events.append(i.text)
print(events)
if I open the url of the event in this case Benfica-Tondela with this code I extract the data of that event but if I want to extract the data of all the events in live I have to open the events page and click with Selenium to open all event links or is there a faster method? thank you all for your help
Reply
#2
If you have to traverse several pages that contain JavaScript, there are a couple of options.
  1. You can continue to use beautiful soup to parse a page, but selenium when you have to follow a link.
  2. Do it all with selenium
Reply
#3
Thanks for the reply, so doing everything with Selenium or using Beautifulsoap is the same or one method is better than another ?
Reply
#4
Neither fits all websites.
Beautifulsoup where no JavaScript expansion is necessary, Selenium otherwise.
It would be helpful for you if you went through Snippsat's web scraping tutorial,
which only takes a few minutes and will give you a better outlook on which package fits which situation.
web scraping part 1
web scraping part 2
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Logic behind BeautifulSoup data-parsing jimsxxl 7 1,183 Apr-13-2021, 09:06 AM
Last Post: jimsxxl
  Extract data from sports betting sites nestor 4 2,495 Mar-30-2021, 04:37 PM
Last Post: Larz60+
  Extract data from a table Bob_M 3 970 Aug-14-2020, 03:36 PM
Last Post: Bob_M
  Selenium extract id text xzozx 1 853 Jun-15-2020, 06:32 AM
Last Post: Larz60+
  Extract json-ld schema markup data and store in MongoDB Nuwan16 0 1,192 Apr-05-2020, 04:06 PM
Last Post: Nuwan16
  Extract data from a webpage cycloneseb 5 1,361 Apr-04-2020, 10:17 AM
Last Post: alekson
  Clicking on element not triggering event in Selenium Python (Event Key is not in data dkaeloredo 2 1,913 Feb-16-2020, 05:50 AM
Last Post: dkaeloredo
  Cannot Extract data through charts online AgileAVS 0 748 Feb-01-2020, 01:47 PM
Last Post: AgileAVS
  Cannot extract data from the next pages nazmulfinance 4 1,191 Nov-11-2019, 08:15 PM
Last Post: nazmulfinance
  Selenium get data from newly accessed page hoff1022 2 1,255 Oct-09-2019, 06:52 PM
Last Post: hoff1022

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020