Python Forum
urllib.request.ProxyHandler works with bad proxy
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
urllib.request.ProxyHandler works with bad proxy
#1
I'm using urllib.request to read pages for scraping. Things were connecting and reading OK but I wanted to add some exception handling in-case one of my rotating proxies was bad. So I made a bad proxy to test:
'1.1.1.1:3'

and expected things to fail... but it succeeded instead. Hmmm. Ok, then I used 'xxxx' as he proxy and it still pulled the page - so clearly I'm doing things wrong. I tried to use Fiddler so see if it was using my IP and port instead but I could not see that info in Fiddler. Here's my code:
proxy_handler = urllib.request.ProxyHandler({'http': 'xxxx'})  # 'xxxx' was '161.35.4.201:80'
opener = urllib.request.build_opener(proxy_handler)
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)  
page = urllib.request.urlopen(self.url)
data = page.read()
What's the point of urllib.request.ProxyHandler if it doesn't use the proxy I pass into it?

Any help would be great. Thanks.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Prevent urllib.request from using my local proxy spacedog 0 454 Apr-24-2021, 08:55 PM
Last Post: spacedog
  Need help with XPath using requests,time,urllib.request and BeautifulSoup spacedog 3 536 Apr-24-2021, 02:48 AM
Last Post: bowlofred
  Help with urllib.request Brian177 2 842 Apr-21-2021, 01:58 PM
Last Post: Brian177
  urllib.request ericmt123 2 741 Dec-21-2020, 06:53 PM
Last Post: Larz60+
  Cannot open url link using urllib.request Askic 5 1,072 Oct-25-2020, 04:56 PM
Last Post: Askic
  ImportError: cannot import name 'Request' from 'request' abhishek81py 1 1,670 Jun-18-2020, 08:07 AM
Last Post: buran
  get file by proxy and header using urllib.request.urlretrieve randyjack 0 817 Mar-12-2020, 09:22 AM
Last Post: randyjack
  why does my python request cannot work with proxy and cannot get captured by fiddler szhangpy 0 971 Jun-27-2019, 10:48 PM
Last Post: szhangpy
  urllib request error 404 Coco 2 2,912 May-11-2019, 02:47 PM
Last Post: Larz60+
  Download entire web pages and save them as html file with urllib.request fyec 2 10,661 Jul-13-2018, 10:12 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020