Combining selenium and beautifulsoup for web scraping

[metulburr]

can I enter into webpages using selenium and then scrape data from there using beautifulsoup?

Yes if you need to get past javascript, you can use selenium to get the full page content and pass it to BS.

from selenium import webdriver
driver = webdriver.Firefox()
driver.get(WEBSITE)
#delay of some kind wait for load time.sleep(3) or selenium wait for an element to be visible
soup = BeautifulSoup(driver.page_source, 'html')

However selenium has methods to get navigate HTML, as you will need it to get past multiple javascript pages/mouse clicks. So it depends really on whether you need BS after already using selenium.

I observed that beautifulsoup never work on those pages needs login, so I need to login first to that website using selenium

You can login to website with requests module and saving cookies, etc. Selenium is not required to login to a website unless it has javascript.