Python Forum
scraping multiple pages of a website.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scraping multiple pages of a website.
#7
Larz60+ has done wonderful job writing this for you, but I think 'it's too complicated for something that can be done with couple of lines (i.e. OOP, etc is overkill)
first of all - your code. The problem is on line#11.
Here is it with some small changes
import requests
from bs4 import BeautifulSoup
from string import ascii_lowercase
 
base_url  = 'https://www.usa.gov/federal-agencies/'
for letter in ascii_lowercase:
    url = '{}{}'.format(base_url, letter)
    print(url)
    resp = requests.get(url)
    soup = BeautifulSoup(resp.text, 'html.parser')
    for ul in soup.find_all('ul', {'class' : 'one_column_bullet'}):
        print(ul))
Now the nice part
If you inspect the page and what it loads you will notice that it gets all the information as json
so
import requests

url = "https://www.usa.gov/ajax/federal-agencies/autocomplete"
resp = requests.get(url)
print(resp.json())
if you want you can save the json resp as file. Anyway, you get all 548 agencies and the respective url in one get request as a json file.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Messages In This Thread
scraping multiple pages of a website. - by Blue Dog - Jun-07-2018, 10:07 PM
RE: scraping multiple pages of a website. - by buran - Jun-08-2018, 10:33 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Need help opening pages when web scraping templeowls 1 362 Feb-29-2024, 06:45 PM
Last Post: snippsat
  Scrape table from multiple pages Nhattanktnn 1 906 Jun-07-2023, 09:35 AM
Last Post: Larz60+
Information Web-scraping, multiple webpages Pabloty92 1 1,316 Dec-28-2022, 02:09 PM
Last Post: Yoriz
  web scraping for new additions/modifed website? kingoman123 4 2,301 Apr-14-2022, 04:46 PM
Last Post: snippsat
  Scraping lender data from Ren Ren Dai website using Python. I will pay for that 200$ Hafedh_2021 1 2,786 May-18-2021, 08:41 PM
Last Post: snippsat
  Scraping all website text using Python MKMKMKMK 1 2,118 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,899 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 2,745 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,768 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping a Website (HELP) LearnPython2 1 1,790 May-08-2020, 03:20 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020