How to Caputre Data After Selenium Scroll

[metulburr]

This is a better method to scroll to the bottom. It automatically identified the end of the page instead of arbitrary scrolling (and much faster too).

def scroll_to_bottom(driver):
    driver = self.browser
    SCROLL_PAUSE_TIME = 0.5
    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load page
        time.sleep(SCROLL_PAUSE_TIME)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height

When you scroll down the page should load the html. But it can depend on the website. AFter you scroll to the bottom, you can then obtain driver.page_source