Website Search Assistance Needed

[CaptianNewb]

Hey Snippsat,

Reaching back out to you on this. Any idea why the following:

stock = soup.select_one('#AT-atc-upsell')
if stock.text == 'Add to cart':

isn't returning any data in the above request?

Is it possible that they changed the code on their website https://www.totalwine.com/spirits/tequil...rules=true

I can confirm that it worked on the CCN Code below:

url = 'https://edition.cnn.com/'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
print(soup.find('title').text)

[snippsat]

Check what version on BS you have,pip show beautifulsoup4,if you have really old version select_one may not work.

G:\div_code\egg
λ pip show beautifulsoup4
Name: beautifulsoup4
Version: 4.12.2
Summary: Screen-scraping library
Home-page:
Author:
Author-email: Leonard Richardson <[email protected]>
License:
Location: C:\python311\Lib\site-packages
Requires: soupsieve
Required-by: bs4, yfinance

The newest is 4.12.2, pip install beautifulsoup4 --upgrade
Here is code written without CSS selector,using find().

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36'}
url = 'https://www.totalwine.com/spirits/tequila/blancosilver/3-amigos-blanco-tequila/p/106918750?s=101&igrules=true'
# url = 'https://www.totalwine.com/spirits/tequila/reposado/clase-azul-reposado-tequila/p/218990175?s=101&igrules=true'
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
stock = soup.find('div', class_="addToCartButtonWrapper__1h1EqYZU")
if stock.text == 'Add to cart':
    print('They have it in stock \N{Bottle with Popping Cork}')
    print(f'The price of {stock.button.get("data-product-name")} is ${stock.button.get("data-price")}')
else:
    print("They don't have it")

Output:
They have it in stock 🍾
The price of 3 Amigos Blanco Tequila is $27.99

[CaptianNewb]

MacBook-Pro:Python UserA$ pip show beautifulsoup4
Name: beautifulsoup4
Version: 4.12.2
Summary: Screen-scraping library
Home-page:
Author:
Author-email: Leonard Richardson <[email protected]>
License:
Location: /Users/UserA/Library/Python/3.9/lib/python/site-packages
Requires: soupsieve
Required-by:

The difference I see is that you have "Required-by: bs4, yfinance" whereas my Required-by is null. I'm not sure if that matters.

I'm still getting the same error unfortunately.
<module>
if stock.text == 'Add to cart':
AttributeError: 'NoneType' object has no attribute 'text'


This is blowing my mind. I have no idea why it is continuing to throw that error- if working for you it should work for me as well since you've proved that it works in your environment .... Do you think it would process better to somehow look for The following"

Pick Up
In stock
Claymont
- Aisle 12, Left
, Bay 05

rather than the text on the button? Not the Aisle and Bay# and prob not Claymont (that's the store) , but maybe Pick Up and In stock? This is the text that is next to the Nearby Stores button. Thoughts?

Thanks!

[snippsat]

You most try to troubleshoot to see if get right content back,your network or even contry placment may block something.
Request to get all data first,so you start there.
Also do pip install requests --upgrade to make sure you have newestet version.

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36'}
url = 'https://www.totalwine.com/spirits/tequila/blancosilver/3-amigos-blanco-tequila/p/106918750?s=101&igrules=true'
# url = 'https://www.totalwine.com/spirits/tequila/reposado/clase-azul-reposado-tequila/p/218990175?s=101&igrules=true'
response = requests.get(url, headers=headers)
print(response)
if b'addToCartButtonWrapper__1h1EqYZU' in response.content:
    print('True')

# print(response.content)
# If uncomment line over should see whole content website

Output:
<Response [200]>
True

When it work you get 200 and it will find add to chart button in source.

[CaptianNewb]

Ok, I upgraded requests but that didn't do the trick. When I ran your latest code, I'm getting 403, so it's getting rejected, not sure why since you're able to get through. Do I need to add more detail to the header?

<Response [403]>

I don't have a firewall or anything on the router that would be blocking this. Very strange!

[snippsat]

<Response [403]>

I don't have a firewall or anything on the router that would be blocking this. Very strange!

403 means that server/site understands the request,but refuses to authorize it.
So there nothing wrong with your network or firewall as the site get request,but will not authorize it.
Look first in source code print(response.content) to see if there more info.

It can various reaons for this as you placement as eg contry,as User-Agent works for me that should not be a problem.
But it could be that need a optimizing request headers from your place.
Using proxies or VPN is common way in web-scraping to get around this.
Also using Selenium / Playright can work.

This may not be so easy to fix as you are all new to this.
Can show example of using proxy that you can try.
I now just use free one from this site Proxy list,
those ip's can up/down fast as they are free,so may need try some for it work.

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36'}
proxy = {"http": "34.162.24.17:8585"}
url = 'https://www.totalwine.com/spirits/tequila/blancosilver/3-amigos-blanco-tequila/p/106918750?s=101&igrules=true'
# url = 'https://www.totalwine.com/spirits/tequila/reposado/clase-azul-reposado-tequila/p/218990175?s=101&igrules=true'
response = requests.get(url, headers=headers, proxies=proxy)
print(response)

Output:
<Response [200]>