Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping and breaking out of Loop
#1
Hello everyone, Need your help with the following script the problem I am running into it pages down in a continuous loading page, until it reaches the bottom, however it does not timeout of after a few seconds. The dynamic loading page can contain 4 to over thousands of elements as I was able to put this script together as it did not have any issues scrolling down in this case to the 300 elements in a pop up window. What can I modify in the script so that it stops at the last element at the bottom of the popup window? Thanks


mlst=[]
actions1 = ActionChains(browser)
while len(WebDriverWait(browser, 0).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']")))) > 0:

    actions1.send_keys(Keys.PAGE_DOWN).perform()
    time.sleep(10)
    
    try:
        x=WebDriverWait(browser, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']")))
        mlst.extend(x)
        print("element found")
        
   
    except TimeoutException as exm:
        
        
        print("Timeout Exception has been thrown. " + str(exm))
        break




for a in mlst:
    print(a.text)
Reply
#2
What about replacing the while loop with a for loop?

Then it will stop when it is finished.
giddyhead likes this post
Reply
#3
(Sep-08-2022, 04:01 AM)XavierPlatinum Wrote: What about replacing the while loop with a for loop?

Then it will stop when it is finished.

Quote:Got it. Thanks for the information. it worked.
Reply
#4
(Sep-08-2022, 09:28 PM)giddyhead Wrote:
(Sep-08-2022, 04:01 AM)XavierPlatinum Wrote: What about replacing the while loop with a for loop?

Then it will stop when it is finished.

Quote:Got it. Thanks for the information. it worked.

Quote:I have updated the script as follows. Thanks. Have one more question how can I go about counting the number of elements in a class before another class. For example in a popup windows class _yua_ has 20 names and class _gilua_1' shows up between the 5th and 6th name. How can I go about getting the number 5 due to class _guilua_1 is between the 5th and 6th element. Once again. Thanks

actions1 = ActionChains(browser)
for cltx in WebDriverWait(browser, 0).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']"))): 
    actions1.send_keys(Keys.PAGE_DOWN).perform()
    time.sleep(10)
     
    try:
        x=WebDriverWait(browser, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='_yua_']")))
        
        print("element found")
         
    
    except TimeoutException as ex:
        #break
        isrunning = 0
        print("Exception has been thrown. " + str(ex))
        break
        
        
        print("No such element")
    else:

        if WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.XPATH, "//h4[@class='_gilua_1']"))):
           
            
            print('Found the text Looking For')
            break
        else:
            print('Else lets see')
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020