Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Cannot get selenium to scrap past the first two pages
#1
I am trying to develop a web scrapper that scrapes leader board information and the code below is what I have so far. Everything works and no error messages are given. However, it will scrap the first page, click the next button, scrap the second page, and then return back to the first page and repeat the process. I cannot figure out how to get it to not repeat the cycle and continue scrapping all the pages until the end. Any help/advice would be much appreciated as I am very new to this.
from selenium import webdriver
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome(r"C:\Users\hones\Desktop\MWO data\chromedriver.exe")

login_screen = driver.get("https://mwomercs.com/profile/leaderboards")
username = driver.find_element_by_id("email").send_keys("login_here")
password = driver.find_element_by_id("password").send_keys("some_password")
submit = driver.find_element_by_xpath("//*[@id='loginForm']/form/button").click()

while True:
    time.sleep(5)
    source = driver.page_source
    stats_obj = BeautifulSoup(source, 'lxml')
    for stat in stats_obj.findAll("tr"):
        cells = stat.findAll("td")
        print(cells)
    driver.find_element_by_xpath("//*[@id='contentBody']/div[2]/div[2]/ul/li/a").click()
* For some reason the post, on my screen, does not have the items in the while loop indented but they are indented properly.
Larz60+ wrote Dec-12-2019, 11:15 AM:
Please post all code, output and errors (in it's entirety) between their respective tags. I did it for you this time, Here are instructions on how to do it yourself next time.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Scrap a dynamic span hefaz 0 214 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 190 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 243 Jan-17-2020, 11:51 AM
Last Post: zarize
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 703 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  page impossible to scrap? :O zarize 2 529 Oct-03-2019, 02:44 PM
Last Post: zarize
  Scrap a value from website harsush 1 315 Aug-29-2019, 01:57 PM
Last Post: snippsat
  scrap macrotrends mr_byte31 7 1,406 Aug-02-2019, 12:02 AM
Last Post: mr_byte31
  Scrap arbitrage odds -help Gochix2020 3 535 Jul-31-2019, 10:45 AM
Last Post: Gochix2020
  i am trying to web scrap this .asp nufan0000 1 428 May-30-2019, 02:27 AM
Last Post: Larz60+
  Scrap Yahoo Finance using BS4 mr_byte31 7 2,235 Aug-24-2018, 02:50 PM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)