Learning slenium question - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Learning slenium question (/thread-12676.html) |
Learning slenium question - Larz60+ - Sep-06-2018 This is targeted at snippsat: I have modified a selenium tutorial on YouTube to use headless mode as in your web scraping tutorial, but am getting an error that I don't understand. The code: from selenium import webdriver from selenium.webdriver.firefox.options import Options import time import csv import os MAX_PAGE_NUM = 5 MAX_PAGE_DIG = 3 #--| Setup options = Options() options.set_headless(headless=True) caps = webdriver.DesiredCapabilities().FIREFOX caps["marionette"] = True browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"/home/Larz60p/Drivers/geckodriver-v0.21.0-linux64/geckodriver") with open('result.csv', 'w') as f: f.write("Buyers, Price\n") for i in range(1, MAX_PAGE_NUM + 1): page_num = f'{(MAX_PAGE_DIG - len(str(i))) * "0" + str(i)}' url = f'http://econpy.pythonanywhere.com/ex/{page_num}.html' browser.get(url) time.sleep(2) buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]') prices = caps.find_elements_by_xpath('//span[@class="item-price"]') with open('result.csv', 'a') as f: for n, buyer in enumerate(buyers): f.write(f'{buyer.text}, {prices[n].text}\n') browser.quit()running produces the following error: find _elements_by_path was working in original tutorial code,tutorial was part 2 of https://www.youtube.com/watch?v=zjo9yFHoUl8 RE: Learning slenium question - snippsat - Sep-06-2018 It's not caps.find_elements_by_xpath ,but browser.find_elements_by_xpath .Also when testing out stuff it can okay to have headless=False ,so you see what going on in browser
RE: Learning slenium question - Larz60+ - Sep-06-2018 Snippsat, Thanks. I am in unfamiliar territory, so question my every move I have some websites I want to scrape that use java script extensively, I like the way that selenium works, although it seems slow. |