[split] API in Python and Postman

[buran]

yes, look at csv module. You can use csv.DictWriter.

[fioranosnake]

Hi, Im having trouble trying to extract the elements from the JSON structure.


Are you able to help with an example if possible?

Thanks again

[buran]

import csv

# response_data = response.json()
response_data = {
"Results": {
"10000": {
"Code1": "9108423",
"Code2": "0",
"Code3": "not available",
"Code4": "Confirmed",
"Code5": "N/A",
"Code6": "N/A"
},
"10008": {
"Code1": "410661",
"Code2": "0",
"Code3": "not available",
"Code4": "Confirmed",
"Code5": "N/A",
"Code6": "N/A"
},
"10009": {
"Code1": "1066228",
"Code2": "0",
"Code3": "available",
"Code4": "Confirmed",
"Code5": "N/A",
"Code6": "7"
}
},
"Override": [],
"Summary": {
"Total": 864,
"Inferrence": 12,
"GlobalUnits": 7206,
"TotalDivereged": 3808,
"ErrorCount": 2,
"ProcessingTimeSeconds": 14.591021
},
"APIMessage": {
"StatusCode": 200,
"Message": "SUCCESS"
}
}

with open('data.csv', 'w') as f:
    fieldnames = list(response_data["Results"].values())[0].keys() # you can split this in multiple steps
    wrtr = csv.DictWriter(f, fieldnames=fieldnames)
    wrtr.writeheader()
    wrtr.writerows(response_data["Results"].values())

all you need is to extract value for key "results" (it's a dict) and pass its values to csv.DictWriter.writerows()

[fioranosnake]

Brilliant .. any thoughts on why its placing a blank line between each row in the csv?

[buran]

oh, sorry - because you are on windows and I tested it on linux. to avoid this it should be opened with newline='' - it's in the docs.
change with open('data.csv', 'w') as f:
to

with open('data.csv', 'w', newline='') as f:

this way it will work on all OS

[fioranosnake]

great stuff once again!

[fioranosnake]

Hi again,

The app is working great.. but.. due to the number of items being posted its beginning to crash the service.

The code currently looks like

import requests
import json
import os
import csv
import pandas as pd
import logging


with open('List.txt', 'r') as f:
        d = {k: v.strip() for (k, v) in enumerate(f, start=1)}
        LIST = json.dumps(d)

# As a temporary stage write the file out as a JSON formatted text file  
 with open('out.txt', 'w') as f_out:  
        f_out.write(LIST)


with open("out.txt", "r") as f: # values_from_file.txt
        LIST2 = json.load(f)


#specify url
url = 'https://myapi.php'

#Specify APIKey 
APIKey = 'mykey' 

#Set Payload to be passsed to API
payload = {"Parameters":{"Key": APIKey
                            },
           "DATA":LIST2
            }
    
#Fill in headers
headers = {'content-type': 'application/json'}

response = requests.post(url, data=json.dumps(payload), headers=headers)
response_data = response.json()

with open('finalresults.csv', 'w', newline='') as f:
        fieldnames = list(response_data["TestResults"].values())[0].keys()
        wrtr = csv.DictWriter(f, fieldnames=fieldnames)
        wrtr.writeheader()
        wrtr.writerows(response_data["TestResults"].values())

The file (list.txt) being passed has over 100K items. Is it possible to split this up into separate files containing around 5K rows and loop through these passing them to the API in turn to create a single output at the end?

Thanks for any advice,

Fiorano