Python Forum
Extract data with Selenium and BeautifulSoup - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Extract data with Selenium and BeautifulSoup (/thread-27387.html)



Extract data with Selenium and BeautifulSoup - nestor - Jun-04-2020

Hi, I'm trying to extract live betting data from this sports betting site, https://bwin.com this is the code I use
from selenium import webdriver
from bs4 import BeautifulSoup
import time
 

browser = webdriver.Chrome()
url = 'https://sports.bwin.com/sports/events/benfica-tondela-10089945'
browser.get(url)
time.sleep(3)
soup = BeautifulSoup(browser.page_source, 'html.parser')
events = []

for i in soup.select('div[class="name"]'):
    events.append(i.text)
print(events)
if I open the url of the event in this case Benfica-Tondela with this code I extract the data of that event but if I want to extract the data of all the events in live I have to open the events page and click with Selenium to open all event links or is there a faster method? thank you all for your help


RE: Extract data with Selenium and BeautifulSoup - Larz60+ - Jun-05-2020

If you have to traverse several pages that contain JavaScript, there are a couple of options.
  1. You can continue to use beautiful soup to parse a page, but selenium when you have to follow a link.
  2. Do it all with selenium



RE: Extract data with Selenium and BeautifulSoup - nestor - Jun-05-2020

Thanks for the reply, so doing everything with Selenium or using Beautifulsoap is the same or one method is better than another ?


RE: Extract data with Selenium and BeautifulSoup - Larz60+ - Jun-06-2020

Neither fits all websites.
Beautifulsoup where no JavaScript expansion is necessary, Selenium otherwise.
It would be helpful for you if you went through Snippsat's web scraping tutorial,
which only takes a few minutes and will give you a better outlook on which package fits which situation.
web scraping part 1
web scraping part 2