Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Cannot get selenium to scrap past the first two pages
#1
I am trying to develop a web scrapper that scrapes leader board information and the code below is what I have so far. Everything works and no error messages are given. However, it will scrap the first page, click the next button, scrap the second page, and then return back to the first page and repeat the process. I cannot figure out how to get it to not repeat the cycle and continue scrapping all the pages until the end. Any help/advice would be much appreciated as I am very new to this.
from selenium import webdriver
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome(r"C:\Users\hones\Desktop\MWO data\chromedriver.exe")

login_screen = driver.get("https://mwomercs.com/profile/leaderboards")
username = driver.find_element_by_id("email").send_keys("login_here")
password = driver.find_element_by_id("password").send_keys("some_password")
submit = driver.find_element_by_xpath("//*[@id='loginForm']/form/button").click()

while True:
    time.sleep(5)
    source = driver.page_source
    stats_obj = BeautifulSoup(source, 'lxml')
    for stat in stats_obj.findAll("tr"):
        cells = stat.findAll("td")
        print(cells)
    driver.find_element_by_xpath("//*[@id='contentBody']/div[2]/div[2]/ul/li/a").click()
* For some reason the post, on my screen, does not have the items in the while loop indented but they are indented properly.
Larz60+ wrote Dec-12-2019, 11:15 AM:
Please post all code, output and errors (in it's entirety) between their respective tags. I did it for you this time, Here are instructions on how to do it yourself next time.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Scrap a dynamic span hefaz 0 592 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 262 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 310 Jan-17-2020, 11:51 AM
Last Post: zarize
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 976 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  Scrap data from not standarized page? zarize 4 612 Nov-25-2019, 10:25 AM
Last Post: zarize
  page impossible to scrap? :O zarize 2 845 Oct-03-2019, 02:44 PM
Last Post: zarize
  Scrap a value from website harsush 1 381 Aug-29-2019, 01:57 PM
Last Post: snippsat
  Scrap text out of td table from URLS Gochix2020 4 1,280 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  scrap macrotrends mr_byte31 7 2,033 Aug-02-2019, 12:02 AM
Last Post: mr_byte31
  Scrap arbitrage odds -help Gochix2020 3 640 Jul-31-2019, 10:45 AM
Last Post: Gochix2020

Forum Jump:


Users browsing this thread: 1 Guest(s)