Python Forum

Very much a python beginner and have stumbled into this problem whilst following the Udacity python course. I realize that the course uses python2, however, I'm trying to replicate using python3.7.2.

The lesson that's causing me a problem is when using the urllib module. I have consulted the python3.7.2 documentation on using the module and have used the example given:

import urllib.request
with urllib.request.urlopen('http://www.python.org/') as f:
    print(f.read(300))

However, I get the output:

Output:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

I've tried looking around for an answer however have had no luck. If anyone could shed some light it would be much appreciated!

Many thanks.

most people use requests library now instead.

try this and see if you get the same error.

import requests
r = requests.get(URL, verify=False)
print(r.text)

Just an addition to metulburr's comments:

or if binary like zip, pdf, doc, xls etc.:

data = r.content

(Feb-21-2019, 09:54 PM)metulburr Wrote: [ -> ]most people use requests library now instead.

try this and see if you get the same error.

import requests
r = requests.get(URL, verify=False)
print(r.text)

This seems to be achieving what I wanted now, thank you! One question though, what does the 'verify=False' part do?

Any idea why the urllib wasn't working for me though?

(Feb-22-2019, 07:36 AM)tomfry Wrote: [ -> ]Any idea why the urllib wasn't working for me though?

Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib

(Feb-22-2019, 07:53 AM)snippsat Wrote: [ -> ]

(Feb-22-2019, 07:36 AM)tomfry Wrote: [ -> ]Any idea why the urllib wasn't working for me though?

Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib

Thanks for the heads up

(Feb-22-2019, 07:41 AM)tomfry Wrote: [ -> ][quote='metulburr' pid='72353' dateline='1550786053']
most people use requests library now instead.

try this and see if you get the same error.

import requests
r = requests.get(URL, verify=False)
print(r.text)

Thanks for your post from Feb 2019; it's a year later and I have the same problem but get a fail on

import requests
ModuleNotFoundError: No module named 'requests'

import urllib.request
has no error, but

[python]x = urllib.request.urlopen('https://www.google.com/')
fails like it did for tomfry, the original poster.

Quote:import requests
ModuleNotFoundError: No module named 'requests'

pip install requests