Paginate API in python

[shantanu97]

I am having trouble writing pagination code in Python

How does the API request work?
First call by the client will be the regular endpoint of the API. The response from the call contains pagination metadata for the subsequent calls. The API limits the request to 100 records, and if the endpoint has more, then I would have to paginate, and used "Next Page Link" from the response to make subsequent calls. The "NextPageLink" would be null on the last page, where the client would end the call.

You can check the basic structure for the API response here in the API Doc

I just want to paginate API and get all data inside the csv file.

When I run the Python code it doesn't bring any data, it just brings headers. Also, I am not sure my API looping is working correctly or not.

Any help what I am doing wrong in the code.

import requests
import csv
import pandas as pd

results = []  # empty list
csvheader = []
headers = {}
params = {}

url = "https://api-peopletray.azurewebsites.net/api/reporting/2321B899-0907-4D2F-9BDD-66D924E699DD/checklists"

try:
    while True:
        response = requests.get(url, data=params, headers=headers)
        # print(response.status_code)  # for debug
        # print(response.text)         # for debug

        json_data = response.json()
        results += json_data['data']

        if "nextPageLink" not in json_data['nextPageLink']:
            break

        url = json_data['nextPageLink']
        
except Exception as ex:
    print("Exception:", ex)        

with open('APIData.csv','w',encoding='UTF8', newline='')as f:    
    writer = csv.writer(f)
    writer.writerow(csvheader)
    writer.writerows(results)   

APIData.csv file attached.

[deanhystad]

You are using a writer to write dictionaries. You should be using a DictWriter. The writer is iterating through your dictionaries to write the rows, and when you iterate through a dictionary you get the keys, not the values.

Using the link provided in you post I wrote this:

import json
import csv
import pandas as pd

response = """{
    "totalRecords": 992,
    "maxPageSize": 100,
    "currentPageSize": 1,
    "nextPageLink": null,
    "nextCursor": 12935,
    "data": [
        {
            "id": 12729,
            "checklistName": "Signature",
            "description": "Signature",
            "user": "Wagner Robin",
            "userId": 3196,
            "checklistDate": "03-01-2020",
            "accountCode": "DM24"
        }
   ]
}"""

response = json.loads(response)
data = [response["data"]]

with open('data.csv','w',encoding='UTF8', newline='')as f:    
    writer = csv.DictWriter(f, data[0][0].keys())
    writer.writeheader()
    for page in data:
        writer.writerows(page)

df = pd.read_csv("data.csv")
print(df)

Which produces this output:

Output:
      id checklistName description          user  userId checklistDate accountCode
0  12729     Signature   Signature  Wagner Robin    3196    03-01-2020        DM24