Dec-13-2019, 10:25 PM
Below is the code I am writing to try and scrape information from a website. There are multiple options to choose from on a drop down menu and each value represents the results from a different season. I want to scrape the data from every season starting with season 10 (earliest available). When the code is run, it navigates to the site, successfully logins, goes to season 10 and begins going through the data. After going through season 10 it jumps to 20, skipping 11 through 19. Once it gets done with 20, it goes through every season and stops at 41, as I intended. Regardless of the starting season I select, it will complete the same action (skip several season then proceed as intended). Any help or advice on how I could fix this loop would be much appreciated.
from selenium import webdriver from selenium.webdriver.support.ui import Select import time login_screen = driver.get("https://mwomercs.com/profile/leaderboards") username = driver.find_element_by_id("email").send_keys("login_here") password = driver.find_element_by_id("password").send_keys("some_password") submit = driver.find_element_by_xpath("//*[@id='loginForm']/form/button").click() # set up variable for season dropdown selection dropdown = driver.find_element_by_xpath("//*[@id='season']") season = Select(dropdown) # select the season to start on i = 10 season.select_by_value(str(i)) # loop to change seasons at the end of just scraped season while True: source = driver.page_source time.sleep(2) try: driver.find_element_by_partial_link_text("Next").click() except: i += 1 if i <= 41: season = driver.find_element_by_xpath("//*[@id='season']/option[" + str(i) + "]").click() else: break