Beginner: urllib error - tomfry - Feb-21-2019
Very much a python beginner and have stumbled into this problem whilst following the Udacity python course. I realize that the course uses python2, however, I'm trying to replicate using python3.7.2.
The lesson that's causing me a problem is when using the urllib module. I have consulted the python3.7.2 documentation on using the module and have used the example given:
import urllib.request
with urllib.request.urlopen('http://www.python.org/') as f:
print(f.read(300)) However, I get the output:
Output: Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 563, in error
result = self._call_chain(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 755, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>
I've tried looking around for an answer however have had no luck. If anyone could shed some light it would be much appreciated!
Many thanks.
RE: Beginner: urllib error - metulburr - Feb-21-2019
most people use requests library now instead.
try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)
RE: Beginner: urllib error - Larz60+ - Feb-22-2019
Just an addition to metulburr's comments:
or if binary like zip, pdf, doc, xls etc.:
data = r.content
RE: Beginner: urllib error - tomfry - Feb-22-2019
(Feb-21-2019, 09:54 PM)metulburr Wrote: most people use requests library now instead.
try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)
This seems to be achieving what I wanted now, thank you! One question though, what does the 'verify=False' part do?
Any idea why the urllib wasn't working for me though?
RE: Beginner: urllib error - snippsat - Feb-22-2019
(Feb-22-2019, 07:36 AM)tomfry Wrote: Any idea why the urllib wasn't working for me though? Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib
RE: Beginner: urllib error - tomfry - Feb-22-2019
(Feb-22-2019, 07:53 AM)snippsat Wrote: (Feb-22-2019, 07:36 AM)tomfry Wrote: Any idea why the urllib wasn't working for me though? Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib
Thanks for the heads up
RE: Beginner: urllib error - deac33 - May-03-2020
(Feb-22-2019, 07:41 AM)tomfry Wrote: [quote='metulburr' pid='72353' dateline='1550786053']
most people use requests library now instead.
try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)
Thanks for your post from Feb 2019; it's a year later and I have the same problem but get a fail on
import requests
ModuleNotFoundError: No module named 'requests'
import urllib.request
has no error, but
[python]x = urllib.request.urlopen('https://www.google.com/')
fails like it did for tomfry, the original poster.
RE: Beginner: urllib error - Larz60+ - May-03-2020
Quote:import requests
ModuleNotFoundError: No module named 'requests'
pip install requests
|