Python Forum
Beginner: urllib error - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Beginner: urllib error (/thread-16297.html)



Beginner: urllib error - tomfry - Feb-21-2019

Very much a python beginner and have stumbled into this problem whilst following the Udacity python course. I realize that the course uses python2, however, I'm trying to replicate using python3.7.2.

The lesson that's causing me a problem is when using the urllib module. I have consulted the python3.7.2 documentation on using the module and have used the example given:

import urllib.request
with urllib.request.urlopen('http://www.python.org/') as f:
    print(f.read(300))
However, I get the output:

Output:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request self._send_request(method, url, body, headers, encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output self.send(msg) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send self.connect() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect server_hostname=server_hostname) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket session=session File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 563, in error result = self._call_chain(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 755, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open response = self._open(req, data) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open '_open', req) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open context=self._context, check_hostname=self._check_hostname) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>
I've tried looking around for an answer however have had no luck. If anyone could shed some light it would be much appreciated!

Many thanks.


RE: Beginner: urllib error - metulburr - Feb-21-2019

most people use requests library now instead.

try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)



RE: Beginner: urllib error - Larz60+ - Feb-22-2019

Just an addition to metulburr's comments:

or if binary like zip, pdf, doc, xls etc.:
data = r.content



RE: Beginner: urllib error - tomfry - Feb-22-2019

(Feb-21-2019, 09:54 PM)metulburr Wrote: most people use requests library now instead.

try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)

This seems to be achieving what I wanted now, thank you! One question though, what does the 'verify=False' part do?

Any idea why the urllib wasn't working for me though?


RE: Beginner: urllib error - snippsat - Feb-22-2019

(Feb-22-2019, 07:36 AM)tomfry Wrote: Any idea why the urllib wasn't working for me though?
Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib Wink


RE: Beginner: urllib error - tomfry - Feb-22-2019

(Feb-22-2019, 07:53 AM)snippsat Wrote:
(Feb-22-2019, 07:36 AM)tomfry Wrote: Any idea why the urllib wasn't working for me though?
Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib Wink

Thanks for the heads up Smile


RE: Beginner: urllib error - deac33 - May-03-2020

(Feb-22-2019, 07:41 AM)tomfry Wrote: [quote='metulburr' pid='72353' dateline='1550786053']
most people use requests library now instead.

try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)


Thanks for your post from Feb 2019; it's a year later and I have the same problem but get a fail on

import requests
ModuleNotFoundError: No module named 'requests'

import urllib.request
has no error, but

[python]x = urllib.request.urlopen('https://www.google.com/')
fails like it did for tomfry, the original poster.


RE: Beginner: urllib error - Larz60+ - May-03-2020

Quote:import requests
ModuleNotFoundError: No module named 'requests'
pip install requests