+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Cannot get selenium to scrap past the first two pages (/thread-23124.html)
Cannot get selenium to scrap past the first two pages - newbie_programmer - Dec-12-2019
I am trying to develop a web scrapper that scrapes leader board information and the code below is what I have so far. Everything works and no error messages are given. However, it will scrap the first page, click the next button, scrap the second page, and then return back to the first page and repeat the process. I cannot figure out how to get it to not repeat the cycle and continue scrapping all the pages until the end. Any help/advice would be much appreciated as I am very new to this.
from selenium import webdriver
import time
from bs4 import BeautifulSoup
driver = webdriver.Chrome(r"C:\Users\hones\Desktop\MWO data\chromedriver.exe")
login_screen = driver.get("https://mwomercs.com/profile/leaderboards")
username = driver.find_element_by_id("email").send_keys("login_here")
password = driver.find_element_by_id("password").send_keys("some_password")
submit = driver.find_element_by_xpath("//*[@id='loginForm']/form/button").click()
while True:
time.sleep(5)
source = driver.page_source
stats_obj = BeautifulSoup(source, 'lxml')
for stat in stats_obj.findAll("tr"):
cells = stat.findAll("td")
print(cells)
driver.find_element_by_xpath("//*[@id='contentBody']/div[2]/div[2]/ul/li/a").click()* For some reason the post, on my screen, does not have the items in the while loop indented but they are indented properly.