Python Forum
scraping multiple pages of a website.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scraping multiple pages of a website.
#1
Hello All,

I have a website that as 26 pages, that star with 'a' and end with a 'z'.
this is y\the url of the site https://www.usa.gov/federal-agencies/a
I have a scraper that does what I want. I know to all of you python kings
it will be crude. what I need help on is how to scrape all 26 pages.
I have been all over the net looking for how to do it. Just not much out there.
I have found a few way of doing it, but none work. Wall So here I am hoping someone can help. LOL

here is my code
#Python 3.7
from html.parser import HTMLParser
import requests
from bs4 import BeautifulSoup







r = requests.get('https://www.usa.gov/federal-agencies/a')
        
first_page = r.text

soup = BeautifulSoup(first_page, 'html.parser')

page_soup = soup

#page_soup.h1

#page_soup.p
boxes = page_soup.find_all('ul', {'class' : 'one_column_bullet'})
boxes[0].text.strip()


print(boxes)
I tryed all I could think of mostly many for loop.
here that works a bit. it print out the same page 26 times.

#Python 3.7
from html.parser import HTMLParser
import requests
from bs4 import BeautifulSoup
from string import ascii_lowercase


for letter in ascii_lowercase:
        r = requests.get('https://www.usa.gov/federal-agencies/' + letter +' ')
        
        first_page = r.text

        soup = BeautifulSoup(first_page, 'html.parser')

        page_soup = soup.find('h1')


        print(page_soup)
So if some one know how to use my to scrape 26 pages let me know.
Thank you
renny
Reply


Messages In This Thread
scraping multiple pages of a website. - by Blue Dog - Jun-07-2018, 10:07 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Need help opening pages when web scraping templeowls 1 300 Feb-29-2024, 06:45 PM
Last Post: snippsat
  Scrape table from multiple pages Nhattanktnn 1 855 Jun-07-2023, 09:35 AM
Last Post: Larz60+
Information Web-scraping, multiple webpages Pabloty92 1 1,271 Dec-28-2022, 02:09 PM
Last Post: Yoriz
  web scraping for new additions/modifed website? kingoman123 4 2,234 Apr-14-2022, 04:46 PM
Last Post: snippsat
  Scraping lender data from Ren Ren Dai website using Python. I will pay for that 200$ Hafedh_2021 1 2,752 May-18-2021, 08:41 PM
Last Post: snippsat
  Scraping all website text using Python MKMKMKMK 1 2,071 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,820 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 2,676 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,699 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping a Website (HELP) LearnPython2 1 1,739 May-08-2020, 03:20 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020