Python Forum

I am trying to use curl to read read large response.
I ran that from command line and redirected to text file and it is 30Mb file

using python I can put one record at a time but not everything that I want.
using following code which is simple.

    try:
        r = requests.get(url, auth=('username', 'password'), headers=headers, verify=False)
        r.raise_for_status()
    except requests.exceptions.RequestException as error:  
        print("Error:", error)
         exit()

    output = r.json()

But what is the output of your code? Does it show the error message or the json?

(Apr-05-2022, 08:11 AM)ibreeden Wrote: [ -> ]But what is the output of your code? Does it show the error message or the json?

if I pull one record it shows in json format.

if I do try to pull all the record it fails with error.
I can do same via curl command on Linux and redirect that to text file and I have 30mb file with all json data in it.

Did you look at Streaming requests? Would it work in your case?
Or a package like json_stream?

(Apr-05-2022, 02:24 PM)buran Wrote: [ -> ]Did you look at Streaming requests? Would it work in your case?
Or a package like json_stream?

Great, json_stream worked. now how do I encode it

with requests.get(url, auth=('username', 'password'),headers=headers, verify=False, stream=True) as response:
            data = json_stream.requests.load(response)
         print(type(data))

<class 'json_stream.base.TransientStreamingJSONObject'>

The point of streaming is that the server sends chunks of data at a time, as it's ready. So, you'll need to have read the whole thing before processing the data. The docs should tell you how to do that with this library.

I try two method describe in that document but didn't work that is why ask. searching around didn't find any example either.

here is what I try

# Option 1: supply json_stream.encoding.default as the default argument
print(json.dumps(data, default=default))

# Option 2: supply json_stream.encoding.JSONStreamEncoder as the cls argument
# This allows you to created your own subclass to further customise encoding
print(json.dumps(data, cls=JSONStreamEncoder))

both prints empty dictionary.

It is working with just request.get without json_stream. it was timing issue.

I have added timeout=200 and I got the whole result back.

Thank you everyone who help.