Python Forum
Logic behind BeautifulSoup data-parsing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Logic behind BeautifulSoup data-parsing
#7
(Apr-12-2021, 03:06 PM)jimsxxl Wrote: So basiclly request_html is the same as Selenium with headless-option (as far as getting html code) ?
Resource wise it will be the same as request_html use pyppeteer(headless) chrome/chromium browser automation.

(Apr-12-2021, 03:06 PM)jimsxxl Wrote: If i would choose to use Selenium this time, would BeautifulSoup be unnessecary then?

I wanted to learn Bs4 in this project, would it be foolish to combine Selenium and BS4 ?
It's fine to send browser.page_source to Bs4 and then do parsing with Bs4.
Example:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
import time

#--| Setup
options = Options()
options.add_argument("--headless")
#options.add_argument("--window-size=1980,1020")
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options)
#--| Parse or automation
url = "https://www.youtube.com/channel/UCwTrHPEglCkDz54iSg9ss9Q/videos"
browser.get(url)
# Send to BS
soup = BeautifulSoup(browser.page_source, 'lxml')
title = soup.select_one('#video-title')
print(title.text)
Output:
WE FISH THE SAME SPOT FOR 12 HOURS - Amazing Results!! | Team Galant
Reply


Messages In This Thread
RE: Logic behind BeautifulSoup data-parsing - by snippsat - Apr-13-2021, 03:33 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  BeautifulSoup not parsing other URLs giddyhead 0 1,220 Feb-23-2022, 05:35 PM
Last Post: giddyhead
  BeautifulSoup: 6k records - but stops after parsing 20 lines apollo 0 1,836 May-10-2021, 05:08 PM
Last Post: apollo
  fetching, parsing data from Wikipedia apollo 2 3,580 May-06-2021, 08:08 PM
Last Post: snippsat
  Extract data with Selenium and BeautifulSoup nestor 3 3,984 Jun-06-2020, 01:34 AM
Last Post: Larz60+
  Fetching and Parsing XML Data FalseFact 3 3,310 Apr-01-2019, 10:21 AM
Last Post: Larz60+
  BeautifulSoup Parsing Error slinkplink 6 9,649 Feb-12-2018, 02:55 PM
Last Post: seco
  Beautifulsoup parsing Larz60+ 7 6,119 Apr-05-2017, 03:07 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020