(Apr-11-2020, 12:36 AM)BadWhite Wrote: but have you tried to run the code?Yes.
import requests from bs4 import BeautifulSoup #from Data import row # Collect and parse first page headers = {'User-agent': 'Mozilla/5.0'} page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm', headers=headers) soup = BeautifulSoup(page.content, 'html.parser') # Pull all text from the BodyText div artist_name_list = soup.find(class_='BodyText') # Pull text from all instances of <a> tag within BodyText div artist_name_list_items = artist_name_list.find_all('a') # Create for loop to print out all artists' names for artist_name in artist_name_list_items: print(artist_name.text)
Output:Zabaglia, Niccola
Zaccone, Fabian
Zadkine, Ossip
Zaech, Bernhard
Zagar, Jacob
Zagroba, Idalia
Zaidenberg, A.
Zaidenberg, Arthur
Zaisinger, Matthäus
Zajac, Jack
Zak, Eugène
Zakharov, Gurii Fillipovich
Zakowortny, Igor
Zalce, Alfredo
Zalopany, Michele
Zammiello, Craig
Zammitt, Norman
Zampieri, Domenico
Zampieri, called Domenichino, Domenico
Zanartú, Enrique Antunez
Zanchi, Antonio
Zanetti, Anton Maria
Zanetti Borzino, Leopoldina
Zanetti I, Antonio Maria, conte
Zanguidi, Jacopo
Zanini, Giuseppe
Zanini-Viola, Giuseppe
Zanotti, Giampietro
Zao Wou-Ki
Zas-Zie
Zie-Zor
nextpage
BadWhite Wrote:why you have added "headers" variable?That was what i explain first,the site return 455
The request was rejected
without user agent.import requests from bs4 import BeautifulSoup #from Data import row # Collect and parse first page page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm') print(page.status_code)
Output:445
So when get this no more scraping is possible,using a user agent we identify as browser in this case Firefox.The get 200 OK and can continue to scrape.
The problem most be something on your side here a run in a other environment colab.
As you see it work fine there to.