Python Forum
Can urlopen be blocked by websites?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Can urlopen be blocked by websites?
#2
Use Requests and not urllib,also a User agent that site use.
import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36'
}

url = 'https://www.nseindia.com/option-chain'
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'lxml')
print(soup.find('title').text)
Output:
NSE - Option Chain
I guess you can parse anything on this site,turn off JavaScripts in browser then reload.
What you see now is what you get with Requests/BS.
This is common problem that Selenium solve.
There are many Threads about this here if search,one with a other stock site.
Reply


Messages In This Thread
Can urlopen be blocked by websites? - by peterjv26 - Jul-26-2020, 02:13 PM
RE: Can urlopen be blocked by websites? - by snippsat - Jul-26-2020, 05:25 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscrapping sport betting websites KoinKoin 3 5,552 Nov-08-2023, 03:00 PM
Last Post: LoriBrown
  Web Scraping Sportsbook Websites Khuber79 17 320,037 Mar-17-2021, 12:06 AM
Last Post: Whitesox1
Thumbs Up Issue facing while scraping the data from different websites in single script. Balamani 1 2,151 Oct-20-2020, 09:56 AM
Last Post: Larz60+
  Python program to write into websites for you pythonDEV333 3 2,573 Jun-08-2020, 12:06 PM
Last Post: pythonDEV333
  prevent getting blocked maneesh7787 3 2,347 Dec-11-2019, 08:41 AM
Last Post: buran
  Scraping Websites to post on Telegram kobryan 1 2,692 Oct-19-2019, 07:03 AM
Last Post: metulburr
  Scraping Websites to post on Telegram kobryan 0 3,452 Oct-09-2019, 04:11 PM
Last Post: kobryan
  SSLCertVerificationError using urllib (urlopen) FalseFact 1 5,930 Mar-31-2019, 08:34 AM
Last Post: snippsat
  Error: module 'urllib' has no attribute 'urlopen' mitmit293 2 15,124 Jan-29-2019, 02:32 PM
Last Post: snippsat
  Scrapping .aspx websites boxingowl88 3 8,321 Oct-10-2018, 05:35 PM
Last Post: stranac

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020