Python Forum
scrape data 1 go to next page scrape data 2 and so on
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scrape data 1 go to next page scrape data 2 and so on
#7
UPDATE

Wanted to post an update to see if there are any suggestions. I'm nearly there with my code, I was able to find something helpful where the 'Next' button is used instead of the individual 'gotoNextPage()' elements. However, it only appends the last page that it runs through. How can I append each page it clicks through to a master data frame?

driver = webdriver.Chrome()
driver.get('website')
username = driver.find_element_by_id("username")
password = driver.find_element_by_id("password")

username.send_keys("username")
password.send_keys("password"+"\n")

while True:
    driver.implicitly_wait(30)
    table = driver.find_element_by_id('preblockBody')
    information = []
    job_elems = table.find_elements_by_xpath("//*[contains(@class,'pbListingTable')]")
    for value in job_elems:
    #print(value.text)
        information.append(value.text)
        
    try:
        driver.find_element_by_partial_link_text('Next').click()
    except:
        break

driver.quit()
print(information)
Reply


Messages In This Thread
RE: scrape data 1 go to next page scrape data 2 and so on - by alkaline3 - Mar-13-2020, 07:59 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Trying to scrape data from HTML with no identifiers pythonpaul32 2 832 Dec-02-2023, 03:42 AM
Last Post: pythonpaul32
  how to scrape page that works dynamicaly? samuelbachorik 0 710 Sep-23-2023, 10:38 AM
Last Post: samuelbachorik
  I am trying to scrape data to broadcast it on Telegram BarryBoos 1 2,077 Jun-10-2023, 02:36 PM
Last Post: snippsat
  Scrape table from multiple pages Nhattanktnn 1 855 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  Simple screen scrape is baffling me. monty024 1 971 Apr-26-2023, 03:27 PM
Last Post: snippsat
  Help to web scrape from 2 diffrent sources Extra 0 824 Jan-05-2023, 12:39 AM
Last Post: Extra
  How can I web scrape the "alt" attribute from a "img" tag with Python? cisky 1 3,884 Aug-19-2022, 04:59 AM
Last Post: snippsat
  How can I target and scrape a data-stat never5000 5 2,785 Feb-11-2022, 07:59 PM
Last Post: snippsat
Lightbulb Python Obstacles | Kung-Fu | Full File HTML Document Scrape and Store it in MariaDB BrandonKastning 5 2,874 Dec-29-2021, 02:26 AM
Last Post: BrandonKastning
  Python Obstacles | American Kenpo | Wiki Scrape URL/Table and Store it in MariaDB BrandonKastning 6 2,831 Dec-29-2021, 12:38 AM
Last Post: BrandonKastning

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020