Scraping with some delay - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Scraping with some delay (/thread-18979.html) |
Scraping with some delay - Truman - Jun-08-2019 from selenium import webdriver from selenium.webdriver.support import ui driver = webdriver.Firefox() driver.get("http://the-internet.herokuapp.com/dynamic_loading/2") button = driver.find_element_by_xpath("//*/div[@id='start']/button") button.click() print("clicked") wait = ui.WebDriver.Wait(driver, 10) wait.until(lambda driver: driver.find_element_by_xpath("//*/div[@id='finish']")) finish_element=driver.find_element_by_xpath("//*/div[@id='finish']/h4") print(finish_element.text) Don't understand why there isn't a webdriver attribute. What should I put instead to do a scraping after the page is in status 'finish'? How to create object 'wait'?
RE: Scraping with some delay - metulburr - Jun-09-2019 its WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC ... WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, ""//*/div[@id='finish']"")))more info about: selenium waits locating elements By locators RE: Scraping with some delay - Truman - Jun-09-2019 from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC driver = webdriver.Firefox() driver.get("http://the-internet.herokuapp.com/dynamic_loading/2") button = driver.find_element_by_xpath("//*/div[@id='start']/button") button.click() print("clicked") WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*/div[@id='finish']"))) finish_element=driver.find_element_by_xpath("//*/div[@id='finish']/h4") print(finish_element.text)Thank you. This works, just without double quotes for xpath. RE: Scraping with some delay - metulburr - Jun-10-2019 Sorry that was a typo |