Python Forum

Dear all,

(Note: I don't have technical background )
 
I have a big json file (Twiitter tweets), 2.6 GB. I need to work with that data and my win10 system (6 GB RAM) can not handle the pressure. When I start looping through the file my laptop freezes. 

I though I'd transfer data from json to sqlite so I can easily quarry the data and get insights. SQLite3 library seems like it works just by loading everything to RAM and then creating the .sql file so I don't think it will work coz i still need go line by line to fill in the tables. 

How would one solve this problem? do you have any suggestions? if there are example code or/and tutorials on this, could you pls share? 

Regards,
Maxi

Can you post the code you tried that has a problem? The minimum amount that reproduces the problem? It's possible there's a really simple solution, that will let you get past needing to add more technology (such as SQlite).

The more that I think about it...
If memory is the issue, how are going to convert the JSON file to anything else? (SQlite included.)

There is ijson: https://pypi.python.org/pypi/ijson/ which is an iterative json reader.
I have never tried it, but it may solve your problem