How to resolve a 401 error, when the credentials work fine in a browser

[PyCurious]

Hi,

I'm using the requests package to check sites for connectivity. 2 of the sites require credentials, and 2 don't.


One of the sites that require credentials, works fine, it returns a 200. The other one returns a 401 Unauthorized.

Both sites present no problem, when logging in via the browser.

request = requests.get(url,timeout=30, verify = False, auth = HTTPDigestAuth( id, pw ))

I've also tried Basic Authentication, but still get the 401 Unauthorized.

Any suggestions about what I can try next?

By the way, is there a third party tool - open source or commercial - that can be used for testing connectivity when developing modules like these? Like a Ping enhancement, command line or gui.

[Larz60+]

from requests.auth import HTTPBasicAuth
import sys

request = requests.get(url,timeout=30, verify = False, auth = HTTPDigestAuth( id, pw ))

if response.status_code == 401:    
    response = requests.get(url, auth=HTTPBasicAuth('user', 'pass')

if response == 200:
    ...
else:
    print("still got an error")
    sys.exit(-1)

[PyCurious]

Hi, thanks for the tips. Your version helped me to notice some structural & syntactic errors in the code. For example,

request = requests.get(url,timeout=30, verify = False, auth = HTTPDigestAuth( id, pw ))

should be

response = requests.get(url,timeout=30, verify = False, auth = HTTPDigestAuth( id, pw ))

Am I correct in stating that your

response = requests.get(url, auth=HTTPBasicAuth('user', 'pass')

should be

response = requests.get(url, auth=HTTPBasicAuth(id, pw)

?

Fixing this removed some false positives.

At any rate, I'm stilling getting a 401 from both the Authentications.

If there's anything else I can try, please let know, thanks.

---

Hi,

It's because the call requires a wait time of at least 15 seconds or so. Which parameter tells the requests to wait for a 200 response?

[Larz60+]

I actually never use it.
Usually the only sites that I scrape are open enough that's it's not needed.
I have run across enough instances where others use it that it was in the back of my mind, therefore the post.
requests documentation here: https://requests.readthedocs.io/en/maste...entication
shows that it should be used as I posted.
I only capture the return (in my variable response) because I usually use status_code, but also other parts of the returned structure.

[PyCurious]

Hi Larz,

I know what's going on now, and (in principle, at least) how to adjust the second authentication attempt appropriately. The header from the first request tells us what type of Authentication to use, in the retry.

So really all I need is to understand the nature of that header data - it looks like a simple dictionary - and proceed according to the type of Authentication returned.

If you have any other tips or links for this, supply them if you can. But I'm on the right track now, and will probably figure this out shortly.

Thanks much,

I actually never use it.
Usually the only sites that I scrape are open enough that's it's not needed.
I have run across enough instances where others use it that it was in the back of my mind, therefore the post.
requests documentation here: https://requests.readthedocs.io/en/maste...entication
shows that it should be used as I posted.
I only capture the return (in my variable response) because I usually use status_code, but also other parts of the returned structure.