Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
error in code web scraping
#5
As mention you most check better that name of class is correct,it is only hGSR34 and not hGSR34 _2beYZw.
Also not all product has a rating so need a fix for this.
Most specify which parser that BS shall use or get a Warning message,preferably use lxml.
As you have loop move append lines into the loop,or only get first product in the lists.
soup = BeautifulSoup(browser.page_source, 'lxml')
products = [] #List to store name of the product
prices = [] #List to store price of the product
ratings = [] #List to store rating of the product
browser.get("https://www.flipkart.com/search?q=laptop&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=off&as=off")
content = browser.page_source
soup = BeautifulSoup(content, 'lxml')
for a in soup.findAll('a', href=True, class_="_31qSD5"):
    name = a.find('div', class_="_3wU53n")
    price = a.find('div', class_="_1vC4OE _2rQ-NK")
    rating = a.find('div',class_="hGSR34")
    products.append(name.text)
    prices.append(price.text)
    try:
        ratings.append(rating.text)
    except AttributeError:
        pass
[>>> ratings
['4.2',
 '4.4',
 '4.5',
 '4.3',
 '4.4',
 '4',
 '4.2',
 '3.7',
 '4.6',
 '4.4',
 '4.5',
 '4',
 '4.5',
 '4.4',
 '4.7',
 '4',
 '4.5',
 '4.6',
 '4.2',
 '4.6',
 '4.3',
 '4.3',
 '5']
>>> prices
['₹39,990',
 '₹1,20,990',
 '₹35,990',
 '₹56,990',
 '₹52,990',
 '₹59,990',
 '₹32,990',
 '₹39,990',
 '₹43,990',
 '₹52,990',
 '₹59,990',
 '₹60,990',
 '₹35,990',
 '₹39,990',
 '₹38,990',
 '₹73,990',
 '₹24,990',
 '₹55,990',
 '₹69,990',
 '₹1,01,990',
 '₹59,990',
 '₹54,990',
 '₹61,990',
 '₹75,990']
Reply


Messages In This Thread
error in code web scraping - by alexisbrunaux - Aug-17-2020, 02:28 AM
RE: error in code web scraping - by mlieqo - Aug-17-2020, 06:10 AM
RE: error in code web scraping - by alexisbrunaux - Aug-18-2020, 01:31 AM
RE: error in code web scraping - by ndc85430 - Aug-18-2020, 05:18 AM
RE: error in code web scraping - by snippsat - Aug-18-2020, 10:05 AM
RE: error in code web scraping - by alexisbrunaux - Aug-19-2020, 02:31 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  I am scraping a web page but got an Error Sarmad54 3 2,580 Mar-02-2023, 08:20 PM
Last Post: Sarmad54
  scraping code misses listings kolarmi19 0 1,878 Jan-27-2023, 10:00 AM
Last Post: kolarmi19
  Code Help, web scraping non uniform lists(ul) luke_m 4 4,330 Apr-22-2021, 05:16 PM
Last Post: luke_m
  scraping code nexuz89 0 2,032 Sep-28-2020, 12:16 PM
Last Post: nexuz89
  In need of web scraping code! kolbyng 1 2,413 Sep-21-2020, 06:02 AM
Last Post: buran
  error zomato scraping data syxzetenz 3 4,385 Jun-23-2020, 08:53 PM
Last Post: Gribouillis
  scraping from a website that hides source code PIWI_Protein 1 2,753 Mar-27-2020, 05:08 PM
Last Post: Larz60+
  Web scraping error jithin123 0 3,303 Mar-22-2020, 08:13 PM
Last Post: jithin123
  Web Scraping Error : Not getting expected result adminravi 4 3,393 Oct-08-2019, 09:53 AM
Last Post: snippsat
  Scraping data saving to DB error with Cursor cubangt 3 3,856 May-20-2019, 08:30 PM
Last Post: Yoriz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020