+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Extract data with Selenium and BeautifulSoup (/thread-27387.html)
Extract data with Selenium and BeautifulSoup - nestor - Jun-04-2020
Hi, I'm trying to extract live betting data from this sports betting site, https://bwin.com this is the code I use
from selenium import webdriver
from bs4 import BeautifulSoup
import time
browser = webdriver.Chrome()
url = 'https://sports.bwin.com/sports/events/benfica-tondela-10089945'
browser.get(url)
time.sleep(3)
soup = BeautifulSoup(browser.page_source, 'html.parser')
events = []
for i in soup.select('div[class="name"]'):
events.append(i.text)
print(events)if I open the url of the event in this case Benfica-Tondela with this code I extract the data of that event but if I want to extract the data of all the events in live I have to open the events page and click with Selenium to open all event links or is there a faster method? thank you all for your help
RE: Extract data with Selenium and BeautifulSoup - Larz60+ - Jun-05-2020
If you have to traverse several pages that contain JavaScript, there are a couple of options.
- You can continue to use beautiful soup to parse a page, but selenium when you have to follow a link.
- Do it all with selenium
RE: Extract data with Selenium and BeautifulSoup - nestor - Jun-05-2020
Thanks for the reply, so doing everything with Selenium or using Beautifulsoap is the same or one method is better than another ?
RE: Extract data with Selenium and BeautifulSoup - Larz60+ - Jun-06-2020
Neither fits all websites.
Beautifulsoup where no JavaScript expansion is necessary, Selenium otherwise.
It would be helpful for you if you went through Snippsat's web scraping tutorial,
which only takes a few minutes and will give you a better outlook on which package fits which situation.
web scraping part 1
web scraping part 2