Python Forum
Selenium suddenly fails to find element
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Selenium suddenly fails to find element
#1
Hello,

A strange phenomenon occurred when using Selenium: after several successful executions, the command that searches for a particular element fails.
Before Selenium I worked wit BeautifulSoap and also observed such phenomena.
Here is code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import re

url = 'https://www.amazon.com/s?k=9781492092513&ref=nb_sb_noss'
options = Options()
options.add_argument("--headless")
browser = webdriver.Chrome('/usr/bin/chromedriver', options=options)
browser.get(url)
url = browser.find_element(By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')
browser.get(url.get_property('href'))
Here is output:
Output:
Traceback (most recent call last): File "/home/pavel/python_code/amazon_selenium/explore_amazone_find_href_of_book.py", line 11, in <module> url = browser.find_element(By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)') File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element 'value': value})['value'] File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/home/pavel/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".a-size-mini > a:nth-child(1)"} (Session info: headless chrome=102.0.5005.61)
Any suggestions ?
Thanks.
Reply
#2
Amazon has some best protection against scraper,bot...ect.
So it natural that it fails sometime,also if element is not loaded on page then scrape to early will fail.
If work most of the times schedule a new try.
Can use Waits for this or as first test i just throw in time.sleep sometime .
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

.....
tag_url = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')))
print(tag_url.get_property('href'))
metulburr likes this post
Reply
#3
I use to have to update my scripts every 3 to 6 months as the website would modify the tags I search for.
Recommended Tutorials:
Reply
#4
(Sep-02-2022, 05:05 PM)snippsat Wrote: Amazon has some best protection against scraper,bot...ect.
So it natural that it fails sometime,also if element is not loaded on page then scrape to early will fail.
If work most of the times schedule a new try.
Can use Waits for this or as first test i just throw in time.sleep sometime .
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

.....
tag_url = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.a-size-mini > a:nth-child(1)')))
print(tag_url.get_property('href'))

Thanks. Tried your approach. Doesn't work.
It seems that if the number of accesses (or overall access time) to Amazon exceeds a certain limit, access is blocked for a certain period of time.
Surprisingly, using VPN cannot solve the problem either.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to find element in a deeply nested html structure Orientation_group 5 1,198 Oct-09-2023, 10:13 AM
Last Post: Larz60+
  find a hyperlink in Gmail body python 3(imap and selenium) taomihiranga 1 8,117 Dec-30-2020, 05:31 PM
Last Post: Gamer1057
  Beautiful Soup (suddenly) doesn't get full webpage html j.crater 8 16,390 Jul-11-2020, 04:31 PM
Last Post: j.crater
  Find element using xpath engli 3 3,886 May-20-2020, 08:56 AM
Last Post: Larz60+
  selenium click in iframe fails 3Pinter 6 5,024 Apr-29-2020, 12:59 PM
Last Post: Larz60+
  find element...click() dont work windows11 6 3,078 Apr-23-2020, 11:13 PM
Last Post: law
  Find Dynamic Input Element Class Problem wazacko 0 1,576 Apr-03-2020, 11:46 AM
Last Post: wazacko
  bypassing find element click() windows11 1 1,836 Apr-02-2020, 11:55 AM
Last Post: Larz60+
  How find link element zinho 3 3,079 Mar-31-2020, 12:29 PM
Last Post: snippsat
  Clicking on element not triggering event in Selenium Python (Event Key is not in data dkaeloredo 2 4,232 Feb-16-2020, 05:50 AM
Last Post: dkaeloredo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020