Scraping Multiple Pages - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Scraping Multiple Pages (/thread-26621.html) |
Scraping Multiple Pages - mbadatanut - May-07-2020 hello - I am having difficult scraping more than one page on a real estate website. any tips or recommendations on my code would be very much appreciated. thanks! import requests from bs4 import BeautifulSoup from csv import writer base_url = 'https://www.rew.ca/properties/areas/kelowna-bc' for i in range(1,26): url = '/page/' + str(i) while url: response = requests.get(f"{base_url}{url}") soup = BeautifulSoup(response.text, "html.parser") listings = soup.find_all("article") with open("property4.csv", "w") as csv_file: csv_writer = writer(csv_file) csv_writer.writerow(["title", "type", "price", "location", "bedrooms", "bathrooms", "square feet", "link"]) for listing in listings: location = listing.find(class_="displaypanel-info").get_text().strip() price = listing.find(class_="displaypanel-title hidden-xs").get_text().strip() link = listing.find("a").get('href').strip() title = listing.find("a").get('title').strip() type = (listing.find(class_="clearfix hidden-xs").find(class_="displaypanel-info")).get_text() bedrooms = (listing.find_all("li")[2]).get_text() bathrooms = (listing.find_all("li")[3]).get_text() square_feet = (listing.find_all("li")[4]).get_text() csv_writer.writerow([title, type, price, location, bedrooms, bathrooms, square_feet, link]) next_btn = soup.find(class_="paginator-next_page paginator-control") url = next_btn.find("a")["href"] RE: Scraping Multiple Pages - Larz60+ - May-08-2020 you need to use the navbar to change pages. This is best done using the .click method of selenium. this is explained in the scraping tutorial on this forum (part 2) web scraping part 1 web scraping part 2 |