Python Forum
scrape data 1 go to next page scrape data 2 and so on
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scrape data 1 go to next page scrape data 2 and so on
#1
Hi everyone, I'm fairly new to Python and brand new to web scraping. any help from the forum is greatly appreciated Smile

I have some code here that gets me what I want from page 1 of my website.
response = requests.get(url)
response

print(response.text)


soup = BeautifulSoup(r.text, 'html.parser')
results = soup.find(id='preblockBody')
results
print(results.prettify())
job_elems = results.find_all('table', class_='pbListingTable')
for job_elem in job_elems:
    title_elem2 = job_elem.find_all('tr', class_='pbListingTable1')
for pbListingTable1 in job_elem.find_all('tr', {'class':'pbListingTable1'}):
    print(pbListingTable1.text)
    title_elem = job_elem.find_all('tr', class_='pbListingTable0')
for pbListingTable0 in job_elem.find_all('tr', {'class':'pbListingTable0'}):
    print(pbListingTable0.text)
Next, I would like to go to page 2 and do the same thing then page 3 and so on. I'm unsure how to do so because the inspection indicates
Quote:<a href="javascript:gotoNextPage(2)">&nbsp;2&nbsp;</a>

Then i would like to summarize all of the outputs together.

How can I do so? Thank you very much
Reply


Messages In This Thread
scrape data 1 go to next page scrape data 2 and so on - by alkaline3 - Mar-05-2020, 06:09 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Trying to scrape data from HTML with no identifiers pythonpaul32 2 795 Dec-02-2023, 03:42 AM
Last Post: pythonpaul32
  how to scrape page that works dynamicaly? samuelbachorik 0 683 Sep-23-2023, 10:38 AM
Last Post: samuelbachorik
  I am trying to scrape data to broadcast it on Telegram BarryBoos 1 1,995 Jun-10-2023, 02:36 PM
Last Post: snippsat
  Scrape table from multiple pages Nhattanktnn 1 825 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  Simple screen scrape is baffling me. monty024 1 942 Apr-26-2023, 03:27 PM
Last Post: snippsat
  Help to web scrape from 2 diffrent sources Extra 0 814 Jan-05-2023, 12:39 AM
Last Post: Extra
  How can I web scrape the "alt" attribute from a "img" tag with Python? cisky 1 3,763 Aug-19-2022, 04:59 AM
Last Post: snippsat
  How can I target and scrape a data-stat never5000 5 2,744 Feb-11-2022, 07:59 PM
Last Post: snippsat
Lightbulb Python Obstacles | Kung-Fu | Full File HTML Document Scrape and Store it in MariaDB BrandonKastning 5 2,821 Dec-29-2021, 02:26 AM
Last Post: BrandonKastning
  Python Obstacles | American Kenpo | Wiki Scrape URL/Table and Store it in MariaDB BrandonKastning 6 2,787 Dec-29-2021, 12:38 AM
Last Post: BrandonKastning

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020