Python Forum

Full Version: Learning slenium question
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
This is targeted at snippsat:

I have modified a selenium tutorial on YouTube to use headless mode as in your web scraping tutorial, but am getting an error that I don't understand.

The code:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time
import csv
import os

MAX_PAGE_NUM = 5
MAX_PAGE_DIG = 3

#--| Setup
options = Options()
options.set_headless(headless=True)
caps = webdriver.DesiredCapabilities().FIREFOX
caps["marionette"] = True
browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"/home/Larz60p/Drivers/geckodriver-v0.21.0-linux64/geckodriver")

with open('result.csv', 'w') as f:
    f.write("Buyers, Price\n")

for i in range(1, MAX_PAGE_NUM + 1):
    page_num = f'{(MAX_PAGE_DIG - len(str(i))) * "0" + str(i)}'
    url = f'http://econpy.pythonanywhere.com/ex/{page_num}.html'
    browser.get(url)
    time.sleep(2)
    buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]')
    prices = caps.find_elements_by_xpath('//span[@class="item-price"]')
    with open('result.csv', 'a') as f:
        for n, buyer in enumerate(buyers):
            f.write(f'{buyer.text}, {prices[n].text}\n')

browser.quit()
running produces the following error:
Error:
(venv) Larz60p@linux-nnem: SelenmiumTraining:$/run/media/Larz60p/Data-2TB/misc/SelenmiumTraining/venv/bin/python /run/media/Larz60p/Data-2TB/misc/SelenmiumTraining/src/UsingSnippsatMethod.py Traceback (most recent call last): File "/.../SelenmiumTraining/src/UsingSnippsatMethod.py", line 25, in <module> buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]') AttributeError: 'dict' object has no attribute 'find_elements_by_xpath' (venv) Larz60p@linux-nnem: SelenmiumTraining:$
find _elements_by_path was working in original tutorial code,
tutorial was part 2 of https://www.youtube.com/watch?v=zjo9yFHoUl8
It's not caps.find_elements_by_xpath,but browser.find_elements_by_xpath.
Also when testing out stuff it can okay to have headless=False,so you see what going on in browser Cool
Snippsat,
Thanks. I am in unfamiliar territory, so question my every move
I have some websites I want to scrape that use java script extensively,
I like the way that selenium works, although it seems slow.