+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Scraping a Flexible Element - works at first, and then crashes (/thread-34625.html)
Scraping a Flexible Element - works at first, and then crashes - JonnyB - Aug-14-2021
import selenium
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
executable_path = "C:\Program Files (x86)\chromedriver.exe"
os.environ["webdriver.chrome.driver"] = executable_path
chrome_options = Options()
chrome_options.add_extension('C:\\Program Files (x86)\\adguard.crx')
driver = webdriver.Chrome(executable_path=executable_path, chrome_options=chrome_options)
driver.get("https://pinterest.com")
element = driver.find_element_by_xpath('//div[contains(text(),"Live Posts")]')
element.click()
time.sleep(6)
posts = driver.find_elements_by_class_name('_3MKTqZAzt9ZahG8uCIFoOb')
for post in posts:
title = post.find_element_by_class_name('_1h0SwjHGr__FljITS6J2FA').text
time_ago = post.find_element_by_class_name('_94ZmeYYiS8oR1zgGtS0xI').text
value= post.find_element_by_class_name('_1jlBA3TjmrUfFXXCyWbfpV').text
print(title, value, time_ago)This code works and scrapes the first several lines, but then it crashes with this error:Quote:selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
The element I am trying to scrape is flex, and gets reloaded every dew seconds (at that time when it reloads, the script crashes. the reload takes less than 1 second.
This is image of the flex element: