Python Forum
Looping through multiple pages with changing url
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Looping through multiple pages with changing url
#2
I'm not sure what links you are interested in, but using your first result 'businesses' on line 14 (for example)
You can pull the link to kabbage by adding after line 14
next_url = businesses.h2.a.get('href')
and review link for same using review_url = businesses.div.a.get('href')
I'd also use requests rather that urllib.
see examples in the following two threads:
web scraping part1
web scraping part2
Reply


Messages In This Thread
RE: Looping through multiple pages with changing url - by Larz60+ - Jan-16-2020, 11:11 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Scrape table from multiple pages Nhattanktnn 1 883 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,876 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 2,728 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,744 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping Multiple Pages mbadatanut 1 4,250 May-08-2020, 02:30 AM
Last Post: Larz60+
  How to handle tables splitted across multiple web pages ankitjindalbti 2 2,123 Jun-02-2019, 07:33 AM
Last Post: ankitjindalbti
  scraping multiple pages of a website. Blue Dog 14 22,503 Jun-21-2018, 09:03 PM
Last Post: Blue Dog
  Looping and naming multiple files revo 4 3,519 Mar-25-2018, 09:46 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020