Python Forum
Looping through multiple pages with changing url
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Looping through multiple pages with changing url
#1
Hi, I am new to web scraping and have just managed to write my first working script. However it is only able to extract data from the first page. I have not been able to apply solutions offered online successfully. Will be might glad if someone can assist me write a complete scrip that extracts data from all pages. below is my current working script

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.merchantcircle.com/search?q=self-storage&qn='

#opens connection, grabbing the page
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

#page parser
page_soup = soup(page_html, "html.parser")

businesses = page_soup.findAll("div",{"class":"hInfo vcard"})


filename = "storage.csv"
f = open(filename, "w")

#headers = "brand, product_name, price, shipping\n"
headers = "biz_name, biz_address, biz_phone_num\n"

f.write(headers)

for business in businesses:
		

	#grabs business name
	biz_name = business.h2.a.text.strip()

	#grabs business address
	address = business.find("a",{"class":"directions"})
	biz_address = address.text.strip()	


	#grabs phone number
	phone_num = business.find("a",{"class":"phone digits tel"})
	biz_phone_num = phone_num.text.strip()



	print("biz_name: " + biz_name)
	print("biz_address: " + biz_address)
	print("biz_phone_num: " + biz_phone_num)

	f.write(biz_name + "," + biz_address.replace(",", "|") + "," + biz_phone_num + "\n")

f.close()
Reply


Messages In This Thread
Looping through multiple pages with changing url - by Qaruri - Jan-16-2020, 09:11 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Scrape table from multiple pages Nhattanktnn 1 1,025 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,979 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 2,824 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,842 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping Multiple Pages mbadatanut 1 4,324 May-08-2020, 02:30 AM
Last Post: Larz60+
  How to handle tables splitted across multiple web pages ankitjindalbti 2 2,191 Jun-02-2019, 07:33 AM
Last Post: ankitjindalbti
  scraping multiple pages of a website. Blue Dog 14 22,766 Jun-21-2018, 09:03 PM
Last Post: Blue Dog
  Looping and naming multiple files revo 4 3,621 Mar-25-2018, 09:46 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020