(Sep-23-2020, 12:35 PM)GoldeNx Wrote: Selenium is too slow for me. That's why I try to write this with requestsCan run Selenium in headless(not loading browser) then is faster.
When logged in can send source
browser.page_source
to BS for parse with lxml.This is way easier the trying to do this with Requests/BS on a JavaScript heavy site.
Also when have
.page_source
can do serval operation at same speed.Example setup bid_size and price_sales or more will have good same speed after the setup and catch of
.page_source
is done.from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.keys import Keys from bs4 import BeautifulSoup import time, sys #--| Setup options = Options() options.add_argument("--headless") options.add_argument('--disable-gpu') #options.add_argument('--log-level=3') browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options) #--| Parse or automation browser.get('https://www.morningstar.com/stocks/XOSL/XXL/quote.html') #time.sleep(1) soup = BeautifulSoup(browser.page_source, 'lxml') bid_size = soup.select('div.dp-value.price-down.ng-binding.ng-scope') price_sales = soup.select('li:nth-child(9) > div > div.dp-value.ng-binding') print(price_sales[0].text.strip()) print(bid_size[0].text.strip())
Output:0.49
25.20×1,000