Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
why I can't scrape a website?
#2
try adding in a bogus header
import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
}

url = 'https://map.naver.com'
req = requests.get(url, headers=headers)
bs = BeautifulSoup(req.text, 'html.parser')
print(bs)
However based on the results being javascript, you may need selenium instead of requests to obtain the correct HTML.
Recommended Tutorials:
Reply


Messages In This Thread
why I can't scrape a website? - by kmkim319 - Sep-18-2019, 03:46 PM
RE: why I can't scrape a website? - by metulburr - Sep-18-2019, 06:03 PM
RE: why I can't scrape a website? - by kmkim319 - Sep-20-2019, 06:07 PM
RE: why I can't scrape a website? - by metulburr - Sep-20-2019, 06:38 PM
RE: why I can't scrape a website? - by snippsat - Sep-20-2019, 07:41 PM
RE: why I can't scrape a website? - by kmkim319 - Sep-25-2019, 03:49 AM
RE: why I can't scrape a website? - by metulburr - Sep-25-2019, 11:31 AM
RE: why I can't scrape a website? - by kmkim319 - Sep-27-2019, 03:14 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Unable to Scrape Website muhamdasim 2 2,628 Dec-27-2021, 07:49 PM
Last Post: JohnRaven
  how to scrape a website from a keyword list greenpine 2 2,394 Dec-04-2020, 03:50 PM
Last Post: greenpine
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,215 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  Read url from CSV and Scrape website Prince_Bhatia 3 10,279 Jan-08-2020, 09:08 AM
Last Post: binaryanimal
  How do i scrape website whose page changes using javsacript _dopostback function and Prince_Bhatia 1 7,248 Aug-06-2018, 09:45 AM
Last Post: wavic
  Scrape A tags from a website Prince_Bhatia 1 4,233 Oct-15-2017, 12:56 AM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020