Selenium with headless firefox is slow - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Selenium with headless firefox is slow (/thread-7507.html) |
Selenium with headless firefox is slow - mgtheboss - Jan-13-2018 Hi, I've started experimenting with selenium & headless firefox. The code I am using is similar to what @snippsat shared on a different thread. The relevant portion of the code can be seen below. I've chained the functions but that cannot be an issue as it was slow before chaining. options = Options() options.add_argument("--headless") browser = webdriver.Firefox(firefox_options=options) browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy') print(html.fromstring(browser.page_source).xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')[0].text)I also tried removing dependency on lxml by using find_elements_by_xpath, but that did not make any difference. It takes around 30 seconds to print the output in both cases. Is this normal behavior? Is there anything I can do to make it faster? I've also experimented with headless chrome. That's slow too. Version information: python 3.4.3, selenium 3.8.1, geckodriver 0.19.1-win32, chromedriver 2.35-win32, firefox 57.0.4, chrome 63.0.3239.132 I appreciate the cooperation of forum members. RE: Selenium with headless firefox is slow - metulburr - Jan-13-2018 this code takes me about 3-4 seconds to return the price with phantomjs or chrome. from selenium import webdriver browser = webdriver.PhantomJS('/home/metulburr/phantomjs') browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy') t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]') print(t.text) from selenium import webdriver browser = webdriver.Chrome('/home/metulburr/chromedriver') browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy') t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]') print(t.text)
RE: Selenium with headless firefox is slow - mgtheboss - Jan-13-2018 @metulburr Are you using headless configuration? RE: Selenium with headless firefox is slow - snippsat - Jan-13-2018 (Jan-13-2018, 04:58 PM)mgtheboss Wrote: @metulburr Are you using headless configuration?The first one with PhantomJS is headless. It rather new that both Chrome and FireFox both comes with a headless mode for there driver. I have used mostly PhantomJS for headless mode. In future it may bee that PhantomJS is not needed anymore. With newest version of selenium,gives this message when use PhantomJS. Quote:Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead Here setup that work for me,10-13 seconds run time. FireFox: from selenium import webdriver from selenium.webdriver.firefox.options import Options #-- Setup options = Options() options.add_argument("--headless") browser = webdriver.Firefox(firefox_options=options, executable_path=r"path to geckodriver") #-- Parse browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy') t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]') print(t.text) browser.quit()Chrome: from selenium import webdriver from selenium.webdriver.chrome.options import Options #-- Setup chrome_options = Options() chrome_options.add_argument("--headless") chrome_options.add_argument('--log-level=3') browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'path to chromedriver') #-- Parse browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy') t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]') print(t.text) browser.quit() RE: Selenium with headless firefox is slow - metulburr - Jan-13-2018 Quote:With newest version of selenium,gives this message when use PhantomJS.Wow that would suck. I ve always had problems with headless mode in chrome and firefox. I hope such issues are resolved before getting rid of phantomjs.Quote:Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead Quote:10-13 seconds run time.Your same code for me is also taking a few seconds to run from selenium import webdriver from selenium.webdriver.chrome.options import Options import time start = time.time() #-- Setup chrome_options = Options() chrome_options.add_argument("--headless") chrome_options.add_argument('--log-level=3') browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/home/metulburr/chromedriver') #-- Parse browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy') t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]') print(t.text) browser.quit() end = time.time() print(end-start)
|