Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web scrap multiple pages
#2
First you need to find out how many pages there is and then you can jump through individual pages using the _p url query parameter
number_of_pages = 200  # just and example number
url = "https://abom.learningbuilder.com/public/membersearch?model.FirstName=&model.LastName=&model.UniqueId=&model.City=&model.State=&performSearch=true&_p={page_number}&_s=20&_o=LastName&_d="
for page_number in range(number_of_pages):
    soup = make_soup(url.format(page_number=page_number))
    for record in soup.findAll('tr'):
        mydata = ""
        for data in soup.findAll('td'):
            mydata = mydata + "," + data.text
            mydata_saved = mydata_saved + "\n" + mydata[1:]
            print(mydata_saved)
Reply


Messages In This Thread
Web scrap multiple pages - by anilacem_302 - Jun-30-2020, 05:38 PM
RE: Web scrap multiple pages - by mlieqo - Jun-30-2020, 08:14 PM
RE: Web scrap multiple pages - by anilacem_302 - Jul-01-2020, 12:07 PM
RE: Web scrap multiple pages - by mlieqo - Jul-01-2020, 07:50 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scrap --Need help Lizardpython 4 1,063 Oct-01-2023, 11:37 AM
Last Post: Lizardpython
  Scrape table from multiple pages Nhattanktnn 1 885 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  I tried every way to scrap morningstar financials data without success so far sparkt 2 8,304 Oct-20-2020, 05:43 PM
Last Post: sparkt
  Need logic on how to scrap 100K URLs goodmind 2 2,658 Jun-29-2020, 09:53 AM
Last Post: goodmind
  scraping multiple pages from table bandar 1 2,731 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,744 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping Multiple Pages mbadatanut 1 4,252 May-08-2020, 02:30 AM
Last Post: Larz60+
  Scrap a dynamic span hefaz 0 2,723 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 1,878 Feb-18-2020, 03:55 PM
Last Post: zarize
  Looping through multiple pages with changing url Qaruri 2 2,625 Jan-17-2020, 01:55 PM
Last Post: Qaruri

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020