Python Forum
How to return the next page from json recursively?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to return the next page from json recursively?
#1
I have an api that returns me a json of 5 out of 5 data, for example. In the example I have an output with 9 data. And in the json it gives me a nextBatchToken token which when added to the parameter returns me the next page of the json. I want to save the final.json with the 9 data at once.

I have the following output:
"data": {
"totalCount": 9,
"detections": [
    {
        "source": "detection",
        "uuid": "6122200f",
        "detectionTime": "2022-03-27T01:32:41Z",
        "ingestionTime": "2022-03-27T01:45:34Z",
        "filters": [...]
},
"nextBatchToken": "TOKEN"
}
}
My code:
import requests, json
import datetime, time
archive = open("final.json", "w")
with archive as finaljson:
    date_time_start = datetime.datetime(2022, 3, 26, 21, 00, 00)
    date_time_end = datetime.datetime(2022, 3, 27, 9, 00, 00)
    start = int((time.mktime(date_time_start.timetuple())))
    end = int((time.mktime(date_time_end.timetuple())))
    # response = json.dumps(r.text)
    url_base = 'URL'
    url_path = '/PATH'
    token = 'MY_YOKEN'
    nextBatchToken = ''
    query_params = {
        'nextBatchToken': nextBatchToken,
        'end': end,
        'size': '5',
        'start': start,
        'riskLevels':'critical'
        }
    headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json;charset=utf-8'}
    response = requests.get(url_base + url_path, params=query_params, headers=headers)
    # archive = response.text
    decodedResponse = json.loads(response.text)
    # decodedResponse = json.dumps(response.json())
    r = decodedResponse['data']
    count = r['totalCount']
    json.dump(response.json(), archive)
    for i in r:
        nextBatchToken = r['nextBatchToken']
        if 'nextBatchToken' in r:
            print('nextBatchToken') #I add the variable here, oculting
            nextBatchToken = r['nextBatchToken']
        else:
            print('cabo')
archive.close()
I tried to make an increment shape but I can't send the json response to the variable and print the next page on this for but no success...

In the output of the file, instead of leaving the token at least once (I changed it for example), it is leaving 4 times and does not add to the parameter.

The json only exits with 5 and the nextBachToken included.
Output:
C:\Users\Sandson\Desktop λ python vai.py MY_TOKEN MY_TOKEN MY_TOKEN MY_TOKEN
The idea is to take the skiptoken, access the next page of the json and save it in csv.
In larger json, I have to do it manually and it's tedious.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help Needed | Read Outlook email Recursively & download attachment Vinci141 1 4,097 Jan-07-2022, 07:38 PM
Last Post: cubangt
Star Recursively convert nested dicts to dict subclass Alfalfa 1 2,906 Jan-22-2021, 05:43 AM
Last Post: buran
  [split] script: remove all "carriage return" from my json variable pete 2 2,814 May-05-2020, 03:22 PM
Last Post: deanhystad
  Python script that recursively zips folders WITHOUT nesting the folder inside the zip umkc1 1 2,858 Feb-11-2020, 09:12 PM
Last Post: michael1789
  How to get full path of specified hidden files matching pattern recursively SriRajesh 4 3,951 Jan-18-2020, 07:12 PM
Last Post: SriRajesh
  Find a given file recursively inside a directory alinaveed786 1 1,950 Jul-01-2019, 01:53 PM
Last Post: ichabod801
  Reading json behind a login page ebolisa 3 2,443 May-26-2019, 05:15 PM
Last Post: heiner55
  Return in correct json format UtiliseIT 3 2,944 May-13-2019, 11:24 AM
Last Post: snippsat
  script: remove all "carriage return" from my json variable mfran2002 4 11,228 Feb-20-2019, 05:07 AM
Last Post: mfran2002
  Call functions recursively in tree structure using python dubru 1 2,295 Feb-15-2019, 06:43 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020