Python Forum
web scrape not returning number correctly
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
web scrape not returning number correctly
#1
I am building an application that compares wine prices, but I ran into a problem.
When I try to get the price of the wine (under the variable "price")it returns "--" instead of the number.

It would be very helpfull if someone knows what is the problem.
Don't hesitate to correct any other mistakes you find in the code.

html from vivino.com:
<div class="text-inline-block header-large light wine-price average__number">
   <span class="wine-price-prefix">€</span>
   <span class="wine-price-value">12,40</span>
   <span class="wine-price-suffix"></span>
</div>
my code:
import requests
from bs4 import BeautifulSoup

wine_to_search = ["Kerner 2017"]

user_agent = {
    "user-agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36'
}

for item in wine_to_search:
    url = 'http://vivino.com/search?q=' + item
    page = requests.get(url, headers=user_agent)
    soup = BeautifulSoup(page.content, 'html.parser')

    wine = soup.find(attrs={"bold"}).get_text()
    price = soup.find(attrs={"wine-price-value"}).get_text()

    print(wine)
    print(price)
Reply
#2
(Sep-18-2019, 04:11 PM)willyflapoor Wrote: When I try to get the price of the wine (under the variable "price")it returns "--" instead of the number.
Yeah, im getting the same thing. The number is generated through javascript. IF you turn off your javacsript on the browser and reload the numbers are -- meaning they are generated by javascript. You will need selenium to get the correct HTML with the numbers in it.
Recommended Tutorials:
Reply
#3
Look at Web-scraping part-2 why do i not get all content.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time

#--| Setup
options = Options()
options.add_argument("--window-size=1980,1020")
options.add_argument("--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36")
options.add_argument("--headless")
browser = webdriver.Chrome(executable_path=r'chromedriver.exe', options=options)
#--| Parse or automation
browser.get('https://www.vivino.com/search/wines?q=Kerner+2017')
price_all = browser.find_elements_by_css_selector('span.wine-price-value')
print(price_all[0].text)
Output:
115,50
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,089 Mar-13-2020, 07:59 PM
Last Post: alkaline3

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020