Scraping Wikipedia Article (Name in 1 column & URL in 2nd column) ->CSV! Anyone?

[BrandonKastning]

Scraping Wikipedia Article (Name in 1 column & URL in 2nd column) ->CSV! Anyone?

Targeted Columns & All Links

Does anyone know how to accomplish this feat?

I followed this tutorial/blog:
https://www.kindacode.com/article/extrac...ul-soup-4/

This code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

import requests   # BeautifulSoup is imported with the name bas4 import bs4   URL = 'https://en.wikipedia.org/wiki/List_of_counties_in_Washington'   # Fetch all the HTML source from the url response = requests.get(URL)     soup = bs4.BeautifulSoup(response.text, 'html.parser') links = soup.select('a')   # Print out the result for link in links:   print(link.get_text())   if link.get('href') != None:     if 'https://' in link.get('href'):       print(link.get('href'))     else:       print('https://en.wikipedia.org' + link.get('href')) # Convert relative URL to absolute URL     print('----------------------------') # Just a line break

This prints the output with "name on top and URL on bottom", such as:

1 2 3 4 5 6 7 8 9 10 11

Counties ---------------------------- Adams https://en.wikipedia.org/wiki/Adams_County,_Washington ---------------------------- Asotin https://en.wikipedia.org/wiki/Asotin_County,_Washington ---------------------------- Benton https://en.wikipedia.org/wiki/Benton_County,_Washington ----------------------------

I would like to store the name in 1 column and the URL in a 2nd column and send it to a CSV

Thank you everyone for this forum! I will append this thread if I start to find answers, rather than replies as I was doing earlier in error! Appreciate the correction and forum/assistance!

Best Regards,

Brandon Kastning

[snippsat]

I would like to store the name in 1 column and the URL in a 2nd column and send it to a CSV

I would first store it in a data structure like list,dict then create CSV from that.
Get a lot of garbage when get all links from Wiki,eg link to different languages link.
Should specify tag in html better,so get only table that you want

Output:
Magyar
https://hu.wikipedia.org/wiki/Washington_megy%C3%A9inek_list%C3%A1ja
----------------------------
Nederlands
https://nl.wikipedia.org/wiki/Lijst_van_county%27s_in_Washington
----------------------------
日本語
https://ja.wikipedia.org/wiki/%E3%83%AF%E3%82%B7%E3%83%B3%E3%83%88%E3%83%B3%E5%B7%9E%E3%81%AE%E9%83%A1%E4%B8%80%E8%A6%A7

Example.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

import requests import bs4   URL = "https://en.wikipedia.org/wiki/List_of_counties_in_Washington" response = requests.get(URL) lst1 = [] lst2 = [] soup = bs4.BeautifulSoup(response.text, "html.parser") links = soup.select("a") for link in links:     print(link.get_text())     lst1.append(link.get_text())     if link.get("href") != None:         if "https://" in link.get("href"):             print(link.get("href"))             lst2.append(link.get("href"))         else:              print(f"https://en.wikipedia.org link.get('href')")              lst2.append(f"https://en.wikipedia.org{link.get('href')}")

So now can zip()and also dict() this together.

Output:
>>> record = dict(zip(lst1, lst2))
>>> record
{'': 'https://foundation.wikimedia.org/wiki/Privacy_policy',
 '"Area Transferred"': "https://en.wikipedia.org link.get('href')",
 '"Article XI, Section 3: New Counties"': 'https://en.wikipedia.org '
                                          "link.get('href')",
 '"Chapter 77 (S.B. 297), Changing Name of Chehalis County"': 'https://en.wikipedia.org '
                                                              "link.get('href')",
 '"Chehalis – Thumbnail History"': "https://en.wikipedia.org link.get('href')",

I guess the info you want is in there,so if do call like this.

1 2 3 4

>>> record['Alabama'] 'https://en.wikipedia.org/wiki/List_of_boroughs_and_census_areas_in_Alaska' >>> record['Massachusetts'] 'https://en.wikipedia.org/wiki/List_of_counties_in_Michigan'

Also if want the language link of in Hebrew📜

1 2	>>> record['עברית'] 'https://frr.wikipedia.org/wiki/Washington_Counties'

[Larz60+]

FYI: County data for all US states is available from: https://www.census.gov/geographies/refer...files.html Use link under Counties.
The link will get you a zip file which extracts to a text file containing State, GEOID, CountyName and links to additional data (files for which can be downloaded from the same page)

[BrandonKastning]

Larz60,

Thank you for the link to the U.S. Census Data! :)

FYI: County data for all US states is available from: https://www.census.gov/geographies/refer...files.html Use link under Counties.
The link will get you a zip file which extracts to a text file containing State, GEOID, CountyName and links to additional data (files for which can be downloaded from the same page)

[Larz60+]

Thank you for the link to the U.S. Census Data! :)

Your welcome.

You may also be interested in this url: https://www2.census.gov/
here, though it takes a while to find and know what each link is for, contains the entire download tree for census public files.
Lots of useful stuff here.