Sep-06-2018, 09:07 PM
This is targeted at snippsat:
I have modified a selenium tutorial on YouTube to use headless mode as in your web scraping tutorial, but am getting an error that I don't understand.
The code:
tutorial was part 2 of https://www.youtube.com/watch?v=zjo9yFHoUl8
I have modified a selenium tutorial on YouTube to use headless mode as in your web scraping tutorial, but am getting an error that I don't understand.
The code:
from selenium import webdriver from selenium.webdriver.firefox.options import Options import time import csv import os MAX_PAGE_NUM = 5 MAX_PAGE_DIG = 3 #--| Setup options = Options() options.set_headless(headless=True) caps = webdriver.DesiredCapabilities().FIREFOX caps["marionette"] = True browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"/home/Larz60p/Drivers/geckodriver-v0.21.0-linux64/geckodriver") with open('result.csv', 'w') as f: f.write("Buyers, Price\n") for i in range(1, MAX_PAGE_NUM + 1): page_num = f'{(MAX_PAGE_DIG - len(str(i))) * "0" + str(i)}' url = f'http://econpy.pythonanywhere.com/ex/{page_num}.html' browser.get(url) time.sleep(2) buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]') prices = caps.find_elements_by_xpath('//span[@class="item-price"]') with open('result.csv', 'a') as f: for n, buyer in enumerate(buyers): f.write(f'{buyer.text}, {prices[n].text}\n') browser.quit()running produces the following error:
Error:(venv) Larz60p@linux-nnem: SelenmiumTraining:$/run/media/Larz60p/Data-2TB/misc/SelenmiumTraining/venv/bin/python /run/media/Larz60p/Data-2TB/misc/SelenmiumTraining/src/UsingSnippsatMethod.py
Traceback (most recent call last):
File "/.../SelenmiumTraining/src/UsingSnippsatMethod.py", line 25, in <module>
buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]')
AttributeError: 'dict' object has no attribute 'find_elements_by_xpath'
(venv) Larz60p@linux-nnem: SelenmiumTraining:$
find _elements_by_path was working in original tutorial code,tutorial was part 2 of https://www.youtube.com/watch?v=zjo9yFHoUl8