Python Forum

Hi!
I’m new with Python and I have been following some tutorials and based on one of them I have the following piece of code:

from scrapy import Selector
from urllib.request import urlopen

html = urlopen("https://www.pythonparatodos.com.br/formulario.html")
sel = Selector(text = html.read())
lista = sel.xpath('//input[@type="text"]')
print(lista)
for selector in lista:
    print(selector)

Thanks in advance!

The URL is easily accessed through a browser but when I run the code I got the following errors:

Error:
C:\Users\username\Envs\webcrawler_part_ii\Scripts\python.exe C:/Users/username/PycharmProjects/webcrawler_part_ii/aula44_scrapy_lxml_exemplos.py
Traceback (most recent call last):
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1257, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1303, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1252, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1012, in _send_output
    self.send(msg)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 952, in send
    self.connect()
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1426, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\username\PycharmProjects\webcrawler_part_ii\aula44_scrapy_lxml_exemplos.py", line 5, in <module>
    html = urlopen("https://www.pythonparatodos.com.br/formulario.html")
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Users\username\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)>

Process finished with exit code 1

I’ve found some possible solutions on google to this with certifi or with the two lines of code below but none of them worked. Can someone help me on this?

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

Use Requests and not urllib.
Site also need a User-agent or will not give access.

from scrapy import Selector
import requests

user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36'}
html = requests.get("https://www.pythonparatodos.com.br/formulario.html", headers=user_agent)
sel = Selector(text = html.content)
lista = sel.xpath('//input[@type="text"]')
#print(lista)
for selector in lista:
    print(selector)

Output:
<Selector xpath='//input[@type="text"]' data='<input type="text" name="nome" size="...'>
<Selector xpath='//input[@type="text"]' data='<input type="text" name="email" size=...'>
<Selector xpath='//input[@type="text"]' data='<input type="text" name="celular" siz...'>

Hi All,
I try to post a question but can not find the icon to do it.
So I drop here and hopefully someone shows me where it is.
Thanks!

(May-09-2022, 10:31 PM)duonght Wrote: [ -> ]I try to post a question but can not find the icon to do it.

Up in right corner there is button Post Thread.
Click on it and write your question-

(May-09-2022, 02:58 PM)snippsat Wrote: [ -> ]Use Requests and not urllib.
Site also need a User-agent or will not give access.

from scrapy import Selector
import requests

user_agent = {'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36'}
html = requests.get("https://www.pythonparatodos.com.br/formulario.html", headers=user_agent)
sel = Selector(text = html.content)
lista = sel.xpath('//input[@type="text"]')
#print(lista)
for selector in lista:
    print(selector)

Output:
<Selector xpath='//input[@type="text"]' data='<input type="text" name="nome" size="...'>
<Selector xpath='//input[@type="text"]' data='<input type="text" name="email" size=...'>
<Selector xpath='//input[@type="text"]' data='<input type="text" name="celular" siz...'>

Thank you! It works.
Out of curiosity, is there any specific reason for the failure with urllib?