+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: request.get to read large response (/thread-36836.html)
request.get to read large response - pythonlearner1 - Apr-05-2022
I am trying to use curl to read read large response.
I ran that from command line and redirected to text file and it is 30Mb file
using python I can put one record at a time but not everything that I want.
using following code which is simple.
try:
r = requests.get(url, auth=('username', 'password'), headers=headers, verify=False)
r.raise_for_status()
except requests.exceptions.RequestException as error:
print("Error:", error)
exit()
output = r.json()
RE: request.get to read large response - ibreeden - Apr-05-2022
But what is the output of your code? Does it show the error message or the json?
RE: request.get to read large response - pythonlearner1 - Apr-05-2022
(Apr-05-2022, 08:11 AM)ibreeden Wrote: But what is the output of your code? Does it show the error message or the json?
if I pull one record it shows in json format.
if I do try to pull all the record it fails with error.
I can do same via curl command on Linux and redirect that to text file and I have 30mb file with all json data in it.
RE: request.get to read large response - buran - Apr-05-2022
Did you look at Streaming requests? Would it work in your case?
Or a package like json_stream?
RE: request.get to read large response - pythonlearner1 - Apr-05-2022
(Apr-05-2022, 02:24 PM)buran Wrote: Did you look at Streaming requests? Would it work in your case?
Or a package like json_stream?
Great, json_stream worked. now how do I encode it
with requests.get(url, auth=('username', 'password'),headers=headers, verify=False, stream=True) as response:
data = json_stream.requests.load(response)
print(type(data))<class 'json_stream.base.TransientStreamingJSONObject'>
RE: request.get to read large response - ndc85430 - Apr-05-2022
The point of streaming is that the server sends chunks of data at a time, as it's ready. So, you'll need to have read the whole thing before processing the data. The docs should tell you how to do that with this library.
RE: request.get to read large response - pythonlearner1 - Apr-05-2022
I try two method describe in that document but didn't work that is why ask. searching around didn't find any example either.
here is what I try
# Option 1: supply json_stream.encoding.default as the default argument
print(json.dumps(data, default=default))
# Option 2: supply json_stream.encoding.JSONStreamEncoder as the cls argument
# This allows you to created your own subclass to further customise encoding
print(json.dumps(data, cls=JSONStreamEncoder))
both prints empty dictionary.
RE: request.get to read large response - pythonlearner1 - Apr-05-2022
It is working with just request.get without json_stream. it was timing issue.
I have added timeout=200 and I got the whole result back.
Thank you everyone who help.