Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Beginner: urllib error
#1
Very much a python beginner and have stumbled into this problem whilst following the Udacity python course. I realize that the course uses python2, however, I'm trying to replicate using python3.7.2.

The lesson that's causing me a problem is when using the urllib module. I have consulted the python3.7.2 documentation on using the module and have used the example given:

import urllib.request
with urllib.request.urlopen('http://www.python.org/') as f:
    print(f.read(300))
However, I get the output:

Output:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request self._send_request(method, url, body, headers, encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output self.send(msg) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send self.connect() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect server_hostname=server_hostname) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket session=session File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 563, in error result = self._call_chain(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 755, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open response = self._open(req, data) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open '_open', req) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open context=self._context, check_hostname=self._check_hostname) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>
I've tried looking around for an answer however have had no luck. If anyone could shed some light it would be much appreciated!

Many thanks.
Reply
#2
most people use requests library now instead.

try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)
Reply
#3
Just an addition to metulburr's comments:

or if binary like zip, pdf, doc, xls etc.:
data = r.content
Reply
#4
(Feb-21-2019, 09:54 PM)metulburr Wrote: most people use requests library now instead.

try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)

This seems to be achieving what I wanted now, thank you! One question though, what does the 'verify=False' part do?

Any idea why the urllib wasn't working for me though?
Reply
#5
(Feb-22-2019, 07:36 AM)tomfry Wrote: Any idea why the urllib wasn't working for me though?
Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib Wink
Reply
#6
(Feb-22-2019, 07:53 AM)snippsat Wrote:
(Feb-22-2019, 07:36 AM)tomfry Wrote: Any idea why the urllib wasn't working for me though?
Unfortunately, it's a sad story still without a happy ending,so always use Requests is better in all cases.
Can also look at Web-Scraping part-1,you will not see urllib Wink

Thanks for the heads up Smile
Reply
#7
(Feb-22-2019, 07:41 AM)tomfry Wrote: [quote='metulburr' pid='72353' dateline='1550786053']
most people use requests library now instead.

try this and see if you get the same error.
import requests
r = requests.get(URL, verify=False)
print(r.text)


Thanks for your post from Feb 2019; it's a year later and I have the same problem but get a fail on

import requests
ModuleNotFoundError: No module named 'requests'

import urllib.request
has no error, but

[python]x = urllib.request.urlopen('https://www.google.com/')
fails like it did for tomfry, the original poster.
Reply
#8
Quote:import requests
ModuleNotFoundError: No module named 'requests'
pip install requests
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  urllib.error.HTTPError: HTTP Error 404: Not Found ckkkkk 4 1,726 Mar-03-2020, 11:30 AM
Last Post: snippsat
  python beginner HTTP Error 500 leofcastro 0 427 Jan-24-2020, 04:37 PM
Last Post: leofcastro
  SSLCertVerificationError using urllib (urlopen) FalseFact 1 2,532 Mar-31-2019, 08:34 AM
Last Post: snippsat
  Error: module 'urllib' has no attribute 'urlopen' mitmit293 2 8,640 Jan-29-2019, 02:32 PM
Last Post: snippsat
  urllib request urlopen? nutgut 4 2,723 Apr-14-2018, 01:12 PM
Last Post: nutgut
  urllib urlopen getting error 400 on 1 specific page glidecode 4 1,576 Mar-01-2018, 11:01 PM
Last Post: glidecode

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020