Larz60+ has done wonderful job writing this for you, but I think 'it's too complicated for something that can be done with couple of lines (i.e. OOP, etc is overkill)
first of all - your code. The problem is on line#11.
Here is it with some small changes
If you inspect the page and what it loads you will notice that it gets all the information as json
so
first of all - your code. The problem is on line#11.
Here is it with some small changes
import requests from bs4 import BeautifulSoup from string import ascii_lowercase base_url = 'https://www.usa.gov/federal-agencies/' for letter in ascii_lowercase: url = '{}{}'.format(base_url, letter) print(url) resp = requests.get(url) soup = BeautifulSoup(resp.text, 'html.parser') for ul in soup.find_all('ul', {'class' : 'one_column_bullet'}): print(ul))Now the nice part
If you inspect the page and what it loads you will notice that it gets all the information as json
so
import requests url = "https://www.usa.gov/ajax/federal-agencies/autocomplete" resp = requests.get(url) print(resp.json())if you want you can save the json resp as file. Anyway, you get all 548 agencies and the respective url in one get request as a json file.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs