Extract data with Selenium and BeautifulSoup

Thread Rating:

Thread Modes

Extract data with Selenium and BeautifulSoup

Larz60+
aetate et sapientia
Super Moderators

Posts: 12,023

Threads: 484

Joined: Sep 2016

Reputation: 452

Jun-05-2020, 11:04 AM

If you have to traverse several pages that contain JavaScript, there are a couple of options.

You can continue to use beautiful soup to parse a page, but selenium when you have to follow a link.
Do it all with selenium

Find

Messages In This Thread

RE: Extract data with Selenium and BeautifulSoup - by Larz60+ - Jun-05-2020, 11:04 AM

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Trying to extract style attribute with BeautifulSoup	knight2000	1	6,089	Dec-28-2022, 03:06 AM Last Post: knight2000
	Python Selenium (Dropdown-) data	Robin_at_Cantelli	2	6,989	Dec-29-2021, 03:16 AM Last Post: ondreweil
	Logic behind BeautifulSoup data-parsing	jimsxxl	7	5,854	Apr-13-2021, 09:06 AM Last Post: jimsxxl
	Extract data from sports betting sites	nestor	3	7,563	Mar-30-2021, 04:37 PM Last Post: Larz60+
	Extract data from a table	Bob_M	3	3,457	Aug-14-2020, 03:36 PM Last Post: Bob_M
	Selenium extract id text	xzozx	1	2,734	Jun-15-2020, 06:32 AM Last Post: Larz60+
	Extract json-ld schema markup data and store in MongoDB	Nuwan16	0	3,050	Apr-05-2020, 04:06 PM Last Post: Nuwan16
	Extract data from a webpage	cycloneseb	5	3,907	Apr-04-2020, 10:17 AM Last Post: alekson
	Clicking on element not triggering event in Selenium Python (Event Key is not in data	dkaeloredo	2	5,231	Feb-16-2020, 05:50 AM Last Post: dkaeloredo
	Cannot Extract data through charts online	AgileAVS	0	2,344	Feb-01-2020, 01:47 PM Last Post: AgileAVS

Users browsing this thread: 1 Guest(s)