Python Forum
urllib.request.ProxyHandler works with bad proxy
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
urllib.request.ProxyHandler works with bad proxy
#1
I'm using urllib.request to read pages for scraping. Things were connecting and reading OK but I wanted to add some exception handling in-case one of my rotating proxies was bad. So I made a bad proxy to test:
'1.1.1.1:3'

and expected things to fail... but it succeeded instead. Hmmm. Ok, then I used 'xxxx' as he proxy and it still pulled the page - so clearly I'm doing things wrong. I tried to use Fiddler so see if it was using my IP and port instead but I could not see that info in Fiddler. Here's my code:
proxy_handler = urllib.request.ProxyHandler({'http': 'xxxx'})  # 'xxxx' was '161.35.4.201:80'
opener = urllib.request.build_opener(proxy_handler)
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)  
page = urllib.request.urlopen(self.url)
data = page.read()
What's the point of urllib.request.ProxyHandler if it doesn't use the proxy I pass into it?

Any help would be great. Thanks.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  urllib can't find "parse" rjdegraff42 6 2,188 Jul-24-2023, 05:28 PM
Last Post: deanhystad
  how can I correct the Bad Request error on my curl request tomtom 8 5,082 Oct-03-2021, 06:32 AM
Last Post: tomtom
  Prevent urllib.request from using my local proxy spacedog 0 2,884 Apr-24-2021, 08:55 PM
Last Post: spacedog
  Need help with XPath using requests,time,urllib.request and BeautifulSoup spacedog 3 2,860 Apr-24-2021, 02:48 AM
Last Post: bowlofred
  Help with urllib.request Brian177 2 2,879 Apr-21-2021, 01:58 PM
Last Post: Brian177
  urllib.request ericmt123 2 2,449 Dec-21-2020, 06:53 PM
Last Post: Larz60+
  Cannot open url link using urllib.request Askic 5 6,684 Oct-25-2020, 04:56 PM
Last Post: Askic
  urllib is not a package traceback cc26 3 5,415 Aug-28-2020, 09:34 AM
Last Post: snippsat
  ImportError: cannot import name 'Request' from 'request' abhishek81py 1 3,934 Jun-18-2020, 08:07 AM
Last Post: buran
  get file by proxy and header using urllib.request.urlretrieve randyjack 0 2,265 Mar-12-2020, 09:22 AM
Last Post: randyjack

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020