1st you should modify:
r = requests.get(url, headers=headers) soup = BeautifulSoup(r.content, 'html.parser')to
response = requests.get(url, headers=headers) if response.status_code == 200: soup = BeautifulSoup(response.content, 'html.parser') links = soup.find_all('a') for link in links: print(link.get('href')) print(link.text) else: print('Unable to open URL: {}'.format(url))