+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Initializing, reading and updating a large JSON file (/thread-36615.html)
Initializing, reading and updating a large JSON file - medatib531 - Mar-10-2022
Hello,
I want to work with a large JSON file in the following form:
{"1": {"data": "abc"},"2": {"data": "xyz"},...}I want be continually fetching data from my source and adding the data in the above indexed format. Each data block is around 60kbytes, and the index is expected to grow to a million or more, so I will be handling several Gigabytes of data that might not potentially fit into memory.So basically what I want to do when my code runs is to open the json data file (e.g. "out.json"), find the greatest index (e.g. 120000) and continue appending from there.
I am not familiar though with handling JSON files in python, let alone large ones. I tried to do something with the code below. However I don't want to store the whole thing into memory, and I want to incrementally write into the file.
Any help appreciated.
import json
import random
with open("out.json", "r+") as file:
jsondata = json.load(file)
#Find largest index??
for i in range(largest_index+1,1000000):
#continue in next index
jsondata[str(i)]={"data":random.random()}
file.seek(0)
json.dump(jsondata, file)
file.truncate()