Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
WebScraping using Selenium library
#1
Hi there, who knows how to properly process WebDriverWait TimeoutException? I am not sure of myself that I do everything right

Let's suppose that we have the following task - we need to parser information from dynamic website where there is show-more button


def get_source_page(uri: str) -> None: # here I created a function for getting source page
    driver.get(uri)
    
    while True: # in this loop I am trying to go through all dynamic pages inside our website main page
        try:
            find_more_element = WebDriverWait(driver=driver, timeout=2).until(
                expected_conditions.element_to_be_clickable((By.CLASS_NAME, "button-show-more"))
            )

            if find_more_element:
                actions = ActionChains(driver)
                actions.move_to_element(find_more_element).click().perform()
                time.sleep(2.5)
        except TimeoutException: # the matter is that WebDriverWait throws an exception when it can't find relevant element with class name[b] "button-show-                    more"[/b]
            with open('index.html', 'w') as file:
                file.write(driver.page_source)

            break

The meaning is that if we went through all pages and there's no more next_page_buttons, the TimeoutException would be thrown and consequently we could write it down into index.html file using with context manager
Reply


Messages In This Thread
WebScraping using Selenium library - by Korgik - Dec-09-2022, 09:51 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscraping news articles by using selenium cate16 7 3,455 Aug-28-2023, 09:58 AM
Last Post: snippsat
  Webscraping with beautifulsoup cormanstan 3 2,241 Aug-24-2023, 11:57 AM
Last Post: snippsat
  Webscraping returning empty table Buuuwq 0 1,485 Dec-09-2022, 10:41 AM
Last Post: Buuuwq
  How to get rid of numerical tokens in output (webscraping issue)? jps2020 0 2,005 Oct-26-2020, 05:37 PM
Last Post: jps2020
  Python Webscraping with a Login Website warriordazza 0 2,690 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Help with basic webscraping Captain_Snuggle 2 4,051 Nov-07-2019, 08:07 PM
Last Post: kozaizsvemira
  Can't Resolve Webscraping AttributeError Hass 1 2,377 Jan-15-2019, 09:36 PM
Last Post: nilamo
  How to exclude certain links while webscraping basis on keywords Prince_Bhatia 0 3,302 Oct-31-2018, 07:00 AM
Last Post: Prince_Bhatia
  Webscraping homework Ghigo1995 1 2,723 Sep-23-2018, 07:36 PM
Last Post: nilamo
  Intro to WebScraping d1rjr03 2 3,518 Aug-15-2018, 12:05 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020