Python Forum
Looping through multiple pages with changing url
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Looping through multiple pages with changing url
#2
I'm not sure what links you are interested in, but using your first result 'businesses' on line 14 (for example)
You can pull the link to kabbage by adding after line 14
next_url = businesses.h2.a.get('href')
and review link for same using review_url = businesses.div.a.get('href')
I'd also use requests rather that urllib.
see examples in the following two threads:
web scraping part1
web scraping part2
Reply


Messages In This Thread
RE: Looping through multiple pages with changing url - by Larz60+ - Jan-16-2020, 11:11 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Scrape table from multiple pages Nhattanktnn 1 1,025 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,979 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 2,824 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,842 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping Multiple Pages mbadatanut 1 4,324 May-08-2020, 02:30 AM
Last Post: Larz60+
  How to handle tables splitted across multiple web pages ankitjindalbti 2 2,191 Jun-02-2019, 07:33 AM
Last Post: ankitjindalbti
  scraping multiple pages of a website. Blue Dog 14 22,766 Jun-21-2018, 09:03 PM
Last Post: Blue Dog
  Looping and naming multiple files revo 4 3,621 Mar-25-2018, 09:46 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020