![]() |
Headless browser - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Headless browser (/thread-13413.html) |
Headless browser - Truman - Oct-13-2018 Trying to run this simple code ( which should be a part of longer script ): from selenium import webdriver from selenium.webdriver import Firefox from selenium.webdriver.firefox.options import Options opts = Options() opts.set_headless() assert opts.headless # Operating in headless mode browser = webdriver.Firefox(options=opts) browser.get('https://duckduckgo.com')...but from some reason it won't open duckduckgo web site. RE: Headless browser - metulburr - Oct-14-2018 try adding headless=True in the argument of set_headless as show in this tutorialhttps://python-forum.io/Thread-Web-scraping-part-2 RE: Headless browser - Larz60+ - Oct-14-2018 Do you get any errors? I tried it with chromedriver and it worked fine (although code a bit different (see below)): from selenium import webdriver from selenium.webdriver.chrome.options import Options import os class TryDucky: def __init__(self): os.chdir(os.path.dirname(__file__)) self.duck() def duck(self): chrome_options = Options() chrome_options.add_argument("--headless") chrome_options.add_argument('--disable-gpu') chrome_options.add_argument('--log-level=3') browser = webdriver.Chrome(chrome_options=chrome_options) browser.get('https://duckduckgo.com') # Try to find something (otherwise wont see anything since headless) search_comment = browser.find_element_by_xpath('/html/body/div/div[2]/div/div/div[3]/div/div') print(f'\n{search_comment.text}') if __name__ == '__main__': TryDucky()Don't forget if headless, you won't see anything until you actually print it. result: options part of code courtesy snippsat
RE: Headless browser - Truman - Oct-14-2018 (Oct-14-2018, 12:04 AM)metulburr Wrote: try addingadded it and it's struggling to execute the code. After a while I receive this: larz60+, I'm looking at your code and it looks a bit above my league.
RE: Headless browser - metulburr - Oct-14-2018 Check your geckodriver.log file (should be in the same directory as python file) If it says Error: GDK_BACKEND does not match available displays then install pyvirtualdisplay: pip install pyvirtualdisplay seleniumYou might need xvfb too: RE: Headless browser - Truman - Oct-15-2018 metulburr, I installed pyvirtualdisplay but still get the same error. Not sure what did you mean by xvfb, didn't add anything... RE: Headless browser - snippsat - Oct-15-2018 You may need to update FireFox,Selenium and geckodriver. In first post you use older code. opts.set_headless() is Deprecated,but it still work.If you don't get error with first code,can try to get something back. Remember headless mean that you don't see anything as browser dos not start. set_headless(headless=True) to see browser set_headless(headless=False) And comment out #assert opts.headless. from selenium import webdriver from selenium.webdriver import Firefox from selenium.webdriver.firefox.options import Options opts = Options() opts.set_headless() assert opts.headless # Operating in headless mode browser = webdriver.Firefox(options=opts) browser.get('https://duckduckgo.com') logo = browser.find_elements_by_css_selector('#logo_homepage_link') print(logo[0].text) FireFox has messed a lot with setting for there driver,so updated browser may need this.Both this method testet here work fine for me. from selenium import webdriver from selenium.webdriver.firefox.options import Options import time #--| Setup options = Options() options.add_argument("--headless") caps = webdriver.DesiredCapabilities().FIREFOX caps["marionette"] = True browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"geckodriver.exe") #--| Parse browser.get('https://duckduckgo.com') logo = browser.find_elements_by_css_selector('#logo_homepage_link') print(logo[0].text)
RE: Headless browser - Truman - Oct-15-2018 snippsat, your last solution works on my end. Thank you, still don't understand it. I guess that this #--| Setup part can be used in other scripts too. Will save it. by the way, how to stop music I tried with browser.close() quit()but it won't work. also, is there any point to add if I want to listen to music for some time? Would try it myself but first have to figure out how to stop music.
RE: Headless browser - snippsat - Oct-16-2018 (Oct-15-2018, 11:09 PM)Truman Wrote: I guess that this #--| Setup part can be used in other scripts too. Will save it.Yes can use it for all when work parse or automation ,just comment out --headless then it work without.Example search and show images of cars. from selenium import webdriver from selenium.webdriver.firefox.options import Options from selenium.webdriver.common.keys import Keys import time #--| Setup options = Options() #options.add_argument("--headless") caps = webdriver.DesiredCapabilities().FIREFOX caps["marionette"] = True browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"geckodriver.exe") #--| Parse or automation browser.get('https://duckduckgo.com') input_field = browser.find_elements_by_css_selector('#search_form_input_homepage') input_field[0].send_keys('car' + Keys.RETURN) time.sleep(3) images_link = browser.find_elements_by_link_text('Images') images_link[0].click() time.sleep(5) browser.quit() Quote:if I want to listen to music for some time? Would try it myself but first have to figure out how to stop music.No time.sleep() is not for that,you have to push start and stop button on web-site that has music.Also you find button eg bye CSS selector,then call click() on eg stop button.
RE: Headless browser - Truman - Oct-16-2018 Thank you. The next step for this car script is to download these images. Will get back to that later when I finish the script I started. Remember that there is a chapter in Automate the boring stuff with Python about that stuff. regarding push and stop button I tried this but it won't stop the music. browser.find_element_by_css_selector('#playbutton playing')[0].click()'playbutton playing' is what I see when the music starts. |