Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Learning slenium question
#1
This is targeted at snippsat:

I have modified a selenium tutorial on YouTube to use headless mode as in your web scraping tutorial, but am getting an error that I don't understand.

The code:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time
import csv
import os

MAX_PAGE_NUM = 5
MAX_PAGE_DIG = 3

#--| Setup
options = Options()
options.set_headless(headless=True)
caps = webdriver.DesiredCapabilities().FIREFOX
caps["marionette"] = True
browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"/home/Larz60p/Drivers/geckodriver-v0.21.0-linux64/geckodriver")

with open('result.csv', 'w') as f:
    f.write("Buyers, Price\n")

for i in range(1, MAX_PAGE_NUM + 1):
    page_num = f'{(MAX_PAGE_DIG - len(str(i))) * "0" + str(i)}'
    url = f'http://econpy.pythonanywhere.com/ex/{page_num}.html'
    browser.get(url)
    time.sleep(2)
    buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]')
    prices = caps.find_elements_by_xpath('//span[@class="item-price"]')
    with open('result.csv', 'a') as f:
        for n, buyer in enumerate(buyers):
            f.write(f'{buyer.text}, {prices[n].text}\n')

browser.quit()
running produces the following error:
Error:
(venv) Larz60p@linux-nnem: SelenmiumTraining:$/run/media/Larz60p/Data-2TB/misc/SelenmiumTraining/venv/bin/python /run/media/Larz60p/Data-2TB/misc/SelenmiumTraining/src/UsingSnippsatMethod.py Traceback (most recent call last): File "/.../SelenmiumTraining/src/UsingSnippsatMethod.py", line 25, in <module> buyers = caps.find_elements_by_xpath('//div[@title="buyer-name"]') AttributeError: 'dict' object has no attribute 'find_elements_by_xpath' (venv) Larz60p@linux-nnem: SelenmiumTraining:$
find _elements_by_path was working in original tutorial code,
tutorial was part 2 of https://www.youtube.com/watch?v=zjo9yFHoUl8
Reply
#2
It's not caps.find_elements_by_xpath,but browser.find_elements_by_xpath.
Also when testing out stuff it can okay to have headless=False,so you see what going on in browser Cool
Reply
#3
Snippsat,
Thanks. I am in unfamiliar territory, so question my every move
I have some websites I want to scrape that use java script extensively,
I like the way that selenium works, although it seems slow.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  slenium instergram auto loggin mister_mister 1 3,530 Jul-02-2018, 04:03 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020