+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: 403 Forbidden Error (/thread-27760.html)
403 Forbidden Error - Evil_Patrick - Jun-20-2020
How to avoid this error while crawling any website?
from bs4 import BeautifulSoup
import requests
source = requests.get("https://www.hltv.org/")
print(source.status_code)Output:403RE: 403 Forbidden Error - snippsat - Jun-20-2020
(Jun-20-2020, 06:19 AM)Evil_Patrick Wrote: How to avoid this error while crawling any website?Using a header user_agent is one way.
from bs4 import BeautifulSoup
import requests
user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'}
source = requests.get("https://www.hltv.org/", headers=user_agent)
print(source.status_code)Output:200