Selenium suddenly fails to find element

Selenium suddenly fails to find element - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Selenium suddenly fails to find element (/thread-38095.html)

Selenium suddenly fails to find element - Pavel_47 - Sep-02-2022

Hello,

A strange phenomenon occurred when using Selenium: after several successful executions, the command that searches for a particular element fails.
Before Selenium I worked wit BeautifulSoap and also observed such phenomena.
Here is code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import re

url = 'https://www.amazon.com/s?k=9781492092513&ref=nb_sb_noss'
options = Options()
options.add_argument("--headless")
browser = webdriver.Chrome('/usr/bin/chromedriver', options=options)
browser.get(url)
url = browser.find_element(By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')
browser.get(url.get_property('href'))

Here is output:

Output:Traceback (most recent call last):
  File "/home/pavel/python_code/amazon_selenium/explore_amazone_find_href_of_book.py", line 11, in <module>
    url = browser.find_element(By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')
  File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".a-size-mini > a:nth-child(1)"}
  (Session info: headless chrome=102.0.5005.61)

Any suggestions ?
Thanks.

RE: Selenium suddenly fails to find element - snippsat - Sep-02-2022

Amazon has some best protection against scraper,bot...ect.
So it natural that it fails sometime,also if element is not loaded on page then scrape to early will fail.
If work most of the times schedule a new try.
Can use Waits for this or as first test i just throw in time.sleep sometime .

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

.....
tag_url = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')))
print(tag_url.get_property('href'))

RE: Selenium suddenly fails to find element - metulburr - Sep-03-2022

I use to have to update my scripts every 3 to 6 months as the website would modify the tags I search for.

RE: Selenium suddenly fails to find element - Pavel_47 - Sep-04-2022

(Sep-02-2022, 05:05 PM)snippsat Wrote: Amazon has some best protection against scraper,bot...ect.
So it natural that it fails sometime,also if element is not loaded on page then scrape to early will fail.
If work most of the times schedule a new try.
Can use Waits for this or as first test i just throw in time.sleep sometime .
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

.....
tag_url = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')))
print(tag_url.get_property('href'))

Thanks. Tried your approach. Doesn't work.
It seems that if the number of accesses (or overall access time) to Amazon exceeds a certain limit, access is blocked for a certain period of time.
Surprisingly, using VPN cannot solve the problem either.