Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
JSON to sql(lite)
#1
Dear all,

(Note: I don't have technical background Sad )
 
I have a big json file (Twiitter tweets), 2.6 GB. I need to work with that data and my win10 system (6 GB RAM) can not handle the pressure. When I start looping through the file my laptop freezes. 

I though I'd transfer data from json to sqlite so I can easily quarry the data and get insights. SQLite3 library seems like it works just by loading everything to RAM and then creating the .sql file so I don't think it will work coz i still need go line by line to fill in the tables. 

How would one solve this problem? do you have any suggestions? if there are example code or/and tutorials on this, could you pls share? 

Regards,
Maxi
Reply
#2
Can you post the code you tried that has a problem? The minimum amount that reproduces the problem? It's possible there's a really simple solution, that will let you get past needing to add more technology (such as SQlite).
Reply
#3
The more that I think about it...
If memory is the issue, how are going to convert the JSON file to anything else? (SQlite included.)
Reply
#4
There is ijson: https://pypi.python.org/pypi/ijson/ which is an iterative json reader.
I have never tried it, but it may solve your problem
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Importing csv data into sql lite cant figure it out. jimmyvegas29 5 3,697 Jul-16-2018, 02:02 AM
Last Post: jimmyvegas29

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020