Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
WebScraping using Selenium library
#1
Hi there, who knows how to properly process WebDriverWait TimeoutException? I am not sure of myself that I do everything right

Let's suppose that we have the following task - we need to parser information from dynamic website where there is show-more button


def get_source_page(uri: str) -> None: # here I created a function for getting source page
    driver.get(uri)
    
    while True: # in this loop I am trying to go through all dynamic pages inside our website main page
        try:
            find_more_element = WebDriverWait(driver=driver, timeout=2).until(
                expected_conditions.element_to_be_clickable((By.CLASS_NAME, "button-show-more"))
            )

            if find_more_element:
                actions = ActionChains(driver)
                actions.move_to_element(find_more_element).click().perform()
                time.sleep(2.5)
        except TimeoutException: # the matter is that WebDriverWait throws an exception when it can't find relevant element with class name[b] "button-show-                    more"[/b]
            with open('index.html', 'w') as file:
                file.write(driver.page_source)

            break

The meaning is that if we went through all pages and there's no more next_page_buttons, the TimeoutException would be thrown and consequently we could write it down into index.html file using with context manager
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscraping news articles by using selenium cate16 7 3,164 Aug-28-2023, 09:58 AM
Last Post: snippsat
  Webscraping with beautifulsoup cormanstan 3 1,991 Aug-24-2023, 11:57 AM
Last Post: snippsat
  Webscraping returning empty table Buuuwq 0 1,404 Dec-09-2022, 10:41 AM
Last Post: Buuuwq
  How to get rid of numerical tokens in output (webscraping issue)? jps2020 0 1,956 Oct-26-2020, 05:37 PM
Last Post: jps2020
  Python Webscraping with a Login Website warriordazza 0 2,610 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Help with basic webscraping Captain_Snuggle 2 3,944 Nov-07-2019, 08:07 PM
Last Post: kozaizsvemira
  Can't Resolve Webscraping AttributeError Hass 1 2,318 Jan-15-2019, 09:36 PM
Last Post: nilamo
  How to exclude certain links while webscraping basis on keywords Prince_Bhatia 0 3,247 Oct-31-2018, 07:00 AM
Last Post: Prince_Bhatia
  Webscraping homework Ghigo1995 1 2,654 Sep-23-2018, 07:36 PM
Last Post: nilamo
  Intro to WebScraping d1rjr03 2 3,456 Aug-15-2018, 12:05 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020