Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping Multiple Pages
#1
hello - I am having difficult scraping more than one page on a real estate website. any tips or recommendations on my code would be very much appreciated. thanks!

 
import requests
from bs4 import BeautifulSoup
from csv import writer

base_url = 'https://www.rew.ca/properties/areas/kelowna-bc'

for i in range(1,26):
    url = '/page/' + str(i)

    while url:
        response = requests.get(f"{base_url}{url}")
        soup = BeautifulSoup(response.text, "html.parser")
        listings = soup.find_all("article")

        with open("property4.csv", "w") as csv_file:
            csv_writer = writer(csv_file)
            csv_writer.writerow(["title", "type", "price", "location", "bedrooms", "bathrooms", "square feet", "link"])
        for listing in listings:
            location = listing.find(class_="displaypanel-info").get_text().strip()
            price = listing.find(class_="displaypanel-title hidden-xs").get_text().strip()
            link = listing.find("a").get('href').strip()
            title = listing.find("a").get('title').strip()
            type = (listing.find(class_="clearfix hidden-xs").find(class_="displaypanel-info")).get_text()
            bedrooms = (listing.find_all("li")[2]).get_text()
            bathrooms = (listing.find_all("li")[3]).get_text()
            square_feet = (listing.find_all("li")[4]).get_text()
            csv_writer.writerow([title, type, price, location, bedrooms, bathrooms, square_feet, link])
            next_btn = soup.find(class_="paginator-next_page paginator-control")
            url = next_btn.find("a")["href"]
Reply
#2
you need to use the navbar to change pages. This is best done using the .click method of selenium.
this is explained in the scraping tutorial on this forum (part 2)
web scraping part 1
web scraping part 2
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Web scrap multiple pages anilacem_302 3 1,428 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 403 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 1,118 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping not moving to the next pages in a website jithin123 0 427 Mar-23-2020, 06:10 PM
Last Post: jithin123
  Scraping from multiple URLS to print in a single line. jb89 4 728 Jan-29-2020, 06:12 AM
Last Post: perfringo
  Looping through multiple pages with changing url Qaruri 2 635 Jan-17-2020, 01:55 PM
Last Post: Qaruri
  MaxRetryError while scraping a website multiple times kawasso 6 6,512 Aug-29-2019, 05:25 PM
Last Post: kawasso
  How to handle tables splitted across multiple web pages ankitjindalbti 2 722 Jun-02-2019, 07:33 AM
Last Post: ankitjindalbti
  scraping with multiple iframe jansky 1 2,182 Nov-09-2018, 11:12 AM
Last Post: snippsat
  Scraping external URLs from pages Apook 5 1,911 Jul-18-2018, 06:42 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020