Python Forum

Full Version: Selenium suddenly fails to find element
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello,

A strange phenomenon occurred when using Selenium: after several successful executions, the command that searches for a particular element fails.
Before Selenium I worked wit BeautifulSoap and also observed such phenomena.
Here is code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import re

url = 'https://www.amazon.com/s?k=9781492092513&ref=nb_sb_noss'
options = Options()
options.add_argument("--headless")
browser = webdriver.Chrome('/usr/bin/chromedriver', options=options)
browser.get(url)
url = browser.find_element(By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')
browser.get(url.get_property('href'))
Here is output:
Output:
Traceback (most recent call last): File "/home/pavel/python_code/amazon_selenium/explore_amazone_find_href_of_book.py", line 11, in <module> url = browser.find_element(By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)') File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element 'value': value})['value'] File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".a-size-mini > a:nth-child(1)"} (Session info: headless chrome=102.0.5005.61)
Any suggestions ?
Thanks.
Amazon has some best protection against scraper,bot...ect.
So it natural that it fails sometime,also if element is not loaded on page then scrape to early will fail.
If work most of the times schedule a new try.
Can use Waits for this or as first test i just throw in time.sleep sometime .
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

.....
tag_url = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')))
print(tag_url.get_property('href'))
I use to have to update my scripts every 3 to 6 months as the website would modify the tags I search for.
(Sep-02-2022, 05:05 PM)snippsat Wrote: [ -> ]Amazon has some best protection against scraper,bot...ect.
So it natural that it fails sometime,also if element is not loaded on page then scrape to early will fail.
If work most of the times schedule a new try.
Can use Waits for this or as first test i just throw in time.sleep sometime .
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

.....
tag_url = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')))
print(tag_url.get_property('href'))

Thanks. Tried your approach. Doesn't work.
It seems that if the number of accesses (or overall access time) to Amazon exceeds a certain limit, access is blocked for a certain period of time.
Surprisingly, using VPN cannot solve the problem either.