Hi,
I've started experimenting with selenium & headless firefox. The code I am using is similar to what @
snippsat shared on
a different thread. The relevant portion of the code can be seen below. I've chained the functions but that cannot be an issue as it was slow before chaining.
options = Options()
options.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=options)
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
print(html.fromstring(browser.page_source).xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')[0].text)I also tried removing dependency on lxml by using find_elements_by_xpath, but that did not make any difference. It takes around 30 seconds to print the output in both cases.
Is this normal behavior? Is there anything I can do to make it faster?
I've also experimented with headless chrome. That's slow too.
Version information: python 3.4.3, selenium 3.8.1, geckodriver 0.19.1-win32, chromedriver 2.35-win32, firefox 57.0.4, chrome 63.0.3239.132
I appreciate the cooperation of forum members.
this code takes me about 3-4 seconds to return the price with phantomjs or chrome.
from selenium import webdriver
browser = webdriver.PhantomJS('/home/metulburr/phantomjs')
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)from selenium import webdriver
browser = webdriver.Chrome('/home/metulburr/chromedriver')
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)Output:
Rs. 7596
@
metulburr Are you using headless configuration?
(Jan-13-2018, 04:58 PM)mgtheboss Wrote: [ -> ]@metulburr Are you using headless configuration?
The first one with PhantomJS is headless.
It rather new that both Chrome and FireFox both comes with a headless mode for there driver.
I have used mostly PhantomJS for headless mode.
In future it may bee that PhantomJS is not needed anymore.
With newest version of selenium,gives this message when use PhantomJS.
Quote:Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
Here setup that work for me,10-13 seconds run time.
FireFox:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
#-- Setup
options = Options()
options.add_argument("--headless")
browser = webdriver.Firefox(firefox_options=options, executable_path=r"path to geckodriver")
#-- Parse
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
browser.quit()Chrome:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
#-- Setup
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument('--log-level=3')
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'path to chromedriver')
#-- Parse
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
browser.quit()
Quote:With newest version of selenium,gives this message when use PhantomJS.Quote:Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
Wow that would suck. I ve always had problems with headless mode in chrome and firefox. I hope such issues are resolved before getting rid of phantomjs.
Quote:10-13 seconds run time.
Your same code for me is also taking a few seconds to run
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
start = time.time()
#-- Setup
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument('--log-level=3')
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/home/metulburr/chromedriver')
#-- Parse
browser.get('https://www.myntra.com/watches/fossil/fossil-women-rose-gold-toned-dial-watch-es3352i/759168/buy')
t = browser.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "pdp-price", " " ))]')
print(t.text)
browser.quit()
end = time.time()
print(end-start)Output:
Rs. 6646
3.7623226642608643