Python Forum
Selenium with headless firefox is slow
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Selenium with headless firefox is slow
#1
Hi,

I've started experimenting with selenium & headless firefox. The code I am using is similar to what @snippsat shared on a different thread. The relevant portion of the code can be seen below. I've chained the functions but that cannot be an issue as it was slow before chaining.

options = Options()
options.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=options)
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
print(html.fromstring(browser.page_source).xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')[0].text)
I also tried removing dependency on lxml by using find_elements_by_xpath, but that did not make any difference. It takes around 30 seconds to print the output in both cases.

Is this normal behavior? Is there anything I can do to make it faster?

I've also experimented with headless chrome. That's slow too.

Version information: python 3.4.3, selenium 3.8.1, geckodriver 0.19.1-win32, chromedriver 2.35-win32, firefox 57.0.4, chrome 63.0.3239.132

I appreciate the cooperation of forum members.
Reply
#2
this code takes me about 3-4 seconds to return the price with phantomjs or chrome.

from selenium import webdriver

browser = webdriver.PhantomJS('/home/metulburr/phantomjs')
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
from selenium import webdriver

browser = webdriver.Chrome('/home/metulburr/chromedriver')
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
Output:
Rs. 7596
Recommended Tutorials:
Reply
#3
@metulburr Are you using headless configuration?
Reply
#4
(Jan-13-2018, 04:58 PM)mgtheboss Wrote: @metulburr Are you using headless configuration?
The first one with PhantomJS is headless.
It rather new that both Chrome and FireFox both comes with a headless mode for there driver.
I have used mostly PhantomJS for headless mode.
In future it may bee that PhantomJS is not needed anymore.
With newest version of selenium,gives this message when use PhantomJS.
Quote:Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead

Here setup that work for me,10-13 seconds run time.
FireFox:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options

#-- Setup
options = Options()
options.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=options, executable_path=r"path to geckodriver")
#-- Parse
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
browser.quit()
Chrome:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

#-- Setup
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument('--log-level=3')
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'path to chromedriver')
#-- Parse
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
browser.quit()
Reply
#5
Quote:With newest version of selenium,gives this message when use PhantomJS.
Quote:Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
Wow that would suck. I ve always had problems with headless mode in chrome and firefox. I hope such issues are resolved before getting rid of phantomjs.


Quote:10-13 seconds run time.
Your same code for me is also taking a few seconds to run
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

start = time.time()
#-- Setup
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument('--log-level=3')
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/home/metulburr/chromedriver')
#-- Parse
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
browser.quit()

end = time.time()

print(end-start)
Output:
Rs. 6646 3.7623226642608643
Recommended Tutorials:
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Need help for script access via webdriver to an open web page in Firefox Clixmaster 1 1,214 Apr-20-2023, 05:27 PM
Last Post: farshid
  Connect to existing Firefox session with Selenium euras 0 5,388 Feb-11-2021, 02:54 PM
Last Post: euras
  Headless Chrome: How to login into a webpage? amandacstr 1 4,230 Feb-06-2020, 02:19 AM
Last Post: metulburr
  error when running headless selenium julio2000 2 4,503 Feb-01-2020, 12:41 PM
Last Post: julio2000
  Unable to access javaScript generated data with selenium and headless FireFox. pjn4 0 2,501 Aug-04-2019, 11:10 AM
Last Post: pjn4
  Firefox Selenium (open new tab) oneclick 1 7,690 Dec-29-2018, 06:59 AM
Last Post: hbknjr
  Headless browser Truman 9 9,015 Oct-16-2018, 10:35 PM
Last Post: Truman
  selenium not running firefox Sanlus 5 4,819 Aug-31-2018, 10:37 PM
Last Post: snippsat
  Proxy Variable in Selenium wont work with FireFox Profile Proxy Setting. MIPython 0 8,489 Jul-13-2018, 05:43 PM
Last Post: MIPython
  How to geckodriver anonymous firefox mariolopes 5 7,121 Feb-10-2018, 10:58 PM
Last Post: mariolopes

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020