Python Forum

Full Version: Scraping and breaking out of Loop
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello everyone, Need your help with the following script the problem I am running into it pages down in a continuous loading page, until it reaches the bottom, however it does not timeout of after a few seconds. The dynamic loading page can contain 4 to over thousands of elements as I was able to put this script together as it did not have any issues scrolling down in this case to the 300 elements in a pop up window. What can I modify in the script so that it stops at the last element at the bottom of the popup window? Thanks


mlst=[]
actions1 = ActionChains(browser)
while len(WebDriverWait(browser, 0).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']")))) > 0:

    actions1.send_keys(Keys.PAGE_DOWN).perform()
    time.sleep(10)
    
    try:
        x=WebDriverWait(browser, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']")))
        mlst.extend(x)
        print("element found")
        
   
    except TimeoutException as exm:
        
        
        print("Timeout Exception has been thrown. " + str(exm))
        break




for a in mlst:
    print(a.text)
What about replacing the while loop with a for loop?

Then it will stop when it is finished.
(Sep-08-2022, 04:01 AM)XavierPlatinum Wrote: [ -> ]What about replacing the while loop with a for loop?

Then it will stop when it is finished.

Quote:Got it. Thanks for the information. it worked.
(Sep-08-2022, 09:28 PM)giddyhead Wrote: [ -> ]
(Sep-08-2022, 04:01 AM)XavierPlatinum Wrote: [ -> ]What about replacing the while loop with a for loop?

Then it will stop when it is finished.

Quote:Got it. Thanks for the information. it worked.

Quote:I have updated the script as follows. Thanks. Have one more question how can I go about counting the number of elements in a class before another class. For example in a popup windows class _yua_ has 20 names and class _gilua_1' shows up between the 5th and 6th name. How can I go about getting the number 5 due to class _guilua_1 is between the 5th and 6th element. Once again. Thanks

actions1 = ActionChains(browser)
for cltx in WebDriverWait(browser, 0).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']"))): 
    actions1.send_keys(Keys.PAGE_DOWN).perform()
    time.sleep(10)
     
    try:
        x=WebDriverWait(browser, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']")))
        
        print("element found")
         
    
    except TimeoutException as ex:
        #break
        isrunning = 0
        print("Exception has been thrown. " + str(ex))
        break
        
        
        print("No such element")
    else:

        if WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.XPATH, "//h4[@class='_gilua_1']"))):
           
            
            print('Found the text Looking For')
            break
        else:
            print('Else lets see')