Python Forum
scrape data 1 go to next page scrape data 2 and so on
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scrape data 1 go to next page scrape data 2 and so on
#7
UPDATE

Wanted to post an update to see if there are any suggestions. I'm nearly there with my code, I was able to find something helpful where the 'Next' button is used instead of the individual 'gotoNextPage()' elements. However, it only appends the last page that it runs through. How can I append each page it clicks through to a master data frame?

driver = webdriver.Chrome()
driver.get('website')
username = driver.find_element_by_id("username")
password = driver.find_element_by_id("password")

username.send_keys("username")
password.send_keys("password"+"\n")

while True:
    driver.implicitly_wait(30)
    table = driver.find_element_by_id('preblockBody')
    information = []
    job_elems = table.find_elements_by_xpath("//*[contains(@class,'pbListingTable')]")
    for value in job_elems:
    #print(value.text)
        information.append(value.text)
        
    try:
        driver.find_element_by_partial_link_text('Next').click()
    except:
        break

driver.quit()
print(information)
Reply


Messages In This Thread
RE: scrape data 1 go to next page scrape data 2 and so on - by alkaline3 - Mar-13-2020, 07:59 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Can't figure out how to scrape grid templeowls 1 133 Jun-13-2024, 09:54 PM
Last Post: Larz60+
  Trying to scrape data from HTML with no identifiers pythonpaul32 2 1,110 Dec-02-2023, 03:42 AM
Last Post: pythonpaul32
  how to scrape page that works dynamicaly? samuelbachorik 0 879 Sep-23-2023, 10:38 AM
Last Post: samuelbachorik
  I am trying to scrape data to broadcast it on Telegram BarryBoos 1 2,473 Jun-10-2023, 02:36 PM
Last Post: snippsat
  Scrape table from multiple pages Nhattanktnn 1 1,025 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  Simple screen scrape is baffling me. monty024 1 1,129 Apr-26-2023, 03:27 PM
Last Post: snippsat
  Help to web scrape from 2 diffrent sources Extra 0 901 Jan-05-2023, 12:39 AM
Last Post: Extra
  How can I web scrape the "alt" attribute from a "img" tag with Python? cisky 1 4,245 Aug-19-2022, 04:59 AM
Last Post: snippsat
  How can I target and scrape a data-stat never5000 5 3,009 Feb-11-2022, 07:59 PM
Last Post: snippsat
Lightbulb Python Obstacles | Kung-Fu | Full File HTML Document Scrape and Store it in MariaDB BrandonKastning 5 3,092 Dec-29-2021, 02:26 AM
Last Post: BrandonKastning

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020