Python Forum

Running code to scrape items from the website and display it

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")

products= []
prices = []
ratings = []

driver.get("https://www.flipkart.com/laptops/~buyback-guarantee-on-laptops-/pr?sid=6bo%2Cb5g&uniq")

content = driver.page_source
soup = BeautifulSoup(content)
for a in soup.findAll('a',href=True, attrs={'class':'_31qSD5'}):
    name=a.find('div', attrs={'class':'_3wU53n'})
    price=a.find('div', attrs={'class':'_1vC4OE _2rQ-NK'})
    rating=a.find('div', attrs={'class':'hGSR34 _2beYZw'})
    products.append(name.text)
    prices.append(price.text)
    ratings.append(rating.text) 

However an error comes :

Error:
runfile('/Users/aarav/Documents/MyProject/Python/Web_scraping.py', wdir='/Users/aarav/Documents/MyProject/Python')
Traceback (most recent call last):

  File "/Users/aarav/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 76, in start
    stdin=PIPE)

  File "/Users/aarav/opt/anaconda3/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)

  File "/Users/aarav/opt/anaconda3/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/chromium-browser/chromedriver': '/usr/lib/chromium-browser/chromedriver'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/aarav/Documents/MyProject/Python/Web_scraping.py", line 7, in <module>
    driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")

  File "/Users/aarav/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
    self.service.start()

  File "/Users/aarav/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 83, in start
    os.path.basename(self.path), self.start_error_message)

WebDriverException: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

What happened?

The error is saying that the path in

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")

should be a full path to an .exe

Please see https://sites.google.com/a/chromium.org/...river/home
https://stackoverflow.com/questions/4055...be-in-path

Well, I found the code in this site and started running it, and it shows this error.
I don't know, but I didn't modify anything while writing the code. How does it work for him and not me?

Have you downloaded the ChromeDriver executable?

Nope, I don't think so. How to download/ check?

The link is in my first post and in the error message you received.

Here a example of a working setup.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import time

#--| Setup
options = Options()
#options.add_argument("--headless")
#options.add_argument("--window-size=1980,1020")
#options.add_argument('--disable-gpu')
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options)
#--| Parse or automation
browser.get("https://www.flipkart.com/laptops/~buyback-guarantee-on-laptops-/pr?sid=6bo%2Cb5g&uniq")
time.sleep(2)
soup = BeautifulSoup(browser.page_source, 'lxml')
# Example of using both to parse
use_bs4 = soup.find('div', class_="col col-7-12")
print(use_bs4.text)
print('*' * 25)
use_sel = browser.find_elements_by_css_selector('div > div._1vC4OE._2rQ-NK')
print(use_sel[0].text)

Output:
Apple MacBook Air Core i5 5th Gen - (8 GB/128 GB SSD/Mac OS Sierra) MQD32HN/A A14664.724,087 Ratings & 2,715 ReviewsIntel Core i5 Processor (5th Gen)8 GB DDR3 RAM64 bit Mac OS Operating System128 GB SSD33.78 cm (13.3 inch) Display1 Year Carry In Warranty
*************************
₹65,990

So this Path is for Windows,do you use Mac?
It's the same way,but use chromedriver_mac64.zip then there is no .exe only chromedriver.

If uncomment like this browser will not start up.

#--| Setup
options = Options()
options.add_argument("--headless")
options.add_argument("--window-size=1980,1020")
options.add_argument('--disable-gpu')

I have used selenium too. Make sure to include the extension .exe, like Chromedriver.exe. Simply writing Chromedriver implies it's a folder.

.exe is for windows, right? What about mac?

(Jun-09-2020, 05:37 PM)pyzyx3qwerty Wrote: [ -> ].exe is for windows, right? What about mac?

(Jun-09-2020, 03:46 PM)snippsat Wrote: [ -> ]So this Path is for Windows,do you use Mac?
It's the same way,but use chromedriver_mac64.zip then there is no .exe only chromedriver.