JSON to sql(lite) - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: JSON to sql(lite) (/thread-5027.html) |
JSON to sql(lite) - BeerLover - Sep-14-2017 Dear all, (Note: I don't have technical background ) I have a big json file (Twiitter tweets), 2.6 GB. I need to work with that data and my win10 system (6 GB RAM) can not handle the pressure. When I start looping through the file my laptop freezes. I though I'd transfer data from json to sqlite so I can easily quarry the data and get insights. SQLite3 library seems like it works just by loading everything to RAM and then creating the .sql file so I don't think it will work coz i still need go line by line to fill in the tables. How would one solve this problem? do you have any suggestions? if there are example code or/and tutorials on this, could you pls share? Regards, Maxi RE: JSON to sql(lite) - micseydel - Sep-15-2017 Can you post the code you tried that has a problem? The minimum amount that reproduces the problem? It's possible there's a really simple solution, that will let you get past needing to add more technology (such as SQlite). RE: JSON to sql(lite) - micseydel - Sep-15-2017 The more that I think about it... If memory is the issue, how are going to convert the JSON file to anything else? (SQlite included.) RE: JSON to sql(lite) - Larz60+ - Sep-15-2017 There is ijson: https://pypi.python.org/pypi/ijson/ which is an iterative json reader. I have never tried it, but it may solve your problem |