Python Forum
Cannot get selenium to scrap past the first two pages
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Cannot get selenium to scrap past the first two pages
#1
I am trying to develop a web scrapper that scrapes leader board information and the code below is what I have so far. Everything works and no error messages are given. However, it will scrap the first page, click the next button, scrap the second page, and then return back to the first page and repeat the process. I cannot figure out how to get it to not repeat the cycle and continue scrapping all the pages until the end. Any help/advice would be much appreciated as I am very new to this.
from selenium import webdriver
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome(r"C:\Users\hones\Desktop\MWO data\chromedriver.exe")

login_screen = driver.get("https://mwomercs.com/profile/leaderboards")
username = driver.find_element_by_id("email").send_keys("login_here")
password = driver.find_element_by_id("password").send_keys("some_password")
submit = driver.find_element_by_xpath("//*[@id='loginForm']/form/button").click()

while True:
    time.sleep(5)
    source = driver.page_source
    stats_obj = BeautifulSoup(source, 'lxml')
    for stat in stats_obj.findAll("tr"):
        cells = stat.findAll("td")
        print(cells)
    driver.find_element_by_xpath("//*[@id='contentBody']/div[2]/div[2]/ul/li/a").click()
* For some reason the post, on my screen, does not have the items in the while loop indented but they are indented properly.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scrap --Need help Lizardpython 4 953 Oct-01-2023, 11:37 AM
Last Post: Lizardpython
  I tried every way to scrap morningstar financials data without success so far sparkt 2 8,167 Oct-20-2020, 05:43 PM
Last Post: sparkt
  Web scrap multiple pages anilacem_302 3 3,781 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  Need logic on how to scrap 100K URLs goodmind 2 2,569 Jun-29-2020, 09:53 AM
Last Post: goodmind
  Scrap a dynamic span hefaz 0 2,658 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 1,833 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 1,852 Jan-17-2020, 11:51 AM
Last Post: zarize
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 7,056 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  Scrap data from not standarized page? zarize 4 3,238 Nov-25-2019, 10:25 AM
Last Post: zarize
  page impossible to scrap? :O zarize 2 3,880 Oct-03-2019, 02:44 PM
Last Post: zarize

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020