Python Forum
not getting image src in my BeautifulSoup csv file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
not getting image src in my BeautifulSoup csv file
#7
Still now same problem after fix the IndentationError. Getting only one image url for each item where multiple images url getting in my python shell for each item. here is my modified full code:


for page_num in range(1):
    url = "https://www.amazon.com/s?k=wireless+mouse&page={}&crid=22TI4BA3RLK5J&qid=1599517835&sprefix=w%2Caps%2C528&ref=sr_pg_2".format(page_num)
    r = requests.get(url,headers=headers,proxies=proxies,auth=auth).text
    soup = BeautifulSoup(r,'lxml')
    container = soup.find_all('h2',{'class':'a-size-mini a-spacing-none a-color-base s-line-clamp-2'})
    for containers in container:
        product_link = f"https://www.amazon.com{containers.find('a')['href']}"
        #print(f"page_number:{url}\n\nproduct_link:{product_link}")
        #here I am start scraping from details page of each product 
        details_page = requests.get(product_link,headers=headers,proxies=proxies,auth=auth).text
        dpsoup = BeautifulSoup(details_page,'lxml')
        title = dpsoup.find('span', id='productTitle')
        if title is not None:
          title = title.text.strip()
        else:
           title= None
        rating = dpsoup.find('span', id='acrCustomerReviewText')
        if rating is not None:
           rating = rating.text
        else:
           rating = None
        price = dpsoup.find('span', class_='a-size-mini twisterSwatchPrice')
        if price is not None:
           price = price.text
        else:
           price = None
        print(f'\nproduct_link: {product_link}\n\nproduct_title: {title}\n\nproduct_price: {price}\n\nproduct_rating: {rating}\n\n')
        #this is for scrape all gallray image src
        with io.open("amazon.csv", "a",encoding="utf-8") as f:
           for url in dpsoup.select('span.a-button-text > img')[3:10]:
              print(f"image link:{url['src']}")
              writeFile = csv.writer(f)
              writeFile.writerow([url['src'],product_link ,title,rating,price]) 
                

        
Reply


Messages In This Thread
RE: not getting image src in my BeautifulSoup csv file - by farhan275 - Sep-14-2020, 02:35 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  BeautifulSoup Showing none while extracting image url josephandrew 0 1,962 Sep-20-2021, 11:40 AM
Last Post: josephandrew
  Image Scraper (beautifulsoup), stopped working, need to help see why woodmister 9 4,129 Jan-12-2021, 04:10 PM
Last Post: woodmister

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020