Mar-20-2018, 11:05 AM
The best way is to load the data in a line at a time and save into a reliable database (Personally, I'd use PostgreSQL).
This may only work if it's not time critical.
In that case, you need to incrementally read the file line by line, and build
a hashed index (or SQL index) of:
hash code, key and file location (which you can get with filepointer.tell())
then to access a specific record, first look it up (using hash code) in the index, and seek the record from the file pointer.
This may only work if it's not time critical.
In that case, you need to incrementally read the file line by line, and build
a hashed index (or SQL index) of:
hash code, key and file location (which you can get with filepointer.tell())
then to access a specific record, first look it up (using hash code) in the index, and seek the record from the file pointer.