urllib.request.ProxyHandler works with bad proxy - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: urllib.request.ProxyHandler works with bad proxy (/thread-33418.html) |
urllib.request.ProxyHandler works with bad proxy - spacedog - Apr-24-2021 I'm using urllib.request to read pages for scraping. Things were connecting and reading OK but I wanted to add some exception handling in-case one of my rotating proxies was bad. So I made a bad proxy to test: '1.1.1.1:3' and expected things to fail... but it succeeded instead. Hmmm. Ok, then I used 'xxxx' as he proxy and it still pulled the page - so clearly I'm doing things wrong. I tried to use Fiddler so see if it was using my IP and port instead but I could not see that info in Fiddler. Here's my code: proxy_handler = urllib.request.ProxyHandler({'http': 'xxxx'}) # 'xxxx' was '161.35.4.201:80' opener = urllib.request.build_opener(proxy_handler) opener.addheaders = [('User-agent', 'Mozilla/5.0')] urllib.request.install_opener(opener) page = urllib.request.urlopen(self.url) data = page.read()What's the point of urllib.request.ProxyHandler if it doesn't use the proxy I pass into it? Any help would be great. Thanks. |