Sep-11-2019, 12:03 PM
(This post was last modified: Sep-11-2019, 12:25 PM by AlekseyPython.)
Python 3.7.3
In my task, it is necessary to store in memory a very large quantity of records (~ 10,000,000 elements), each of which is on the order of 5-10 fields of different types (bool, string, integer, fixed-point number, date and time). It is necessary to be economical not only with memory, but also with processor time (otherwise I won’t wait for the processing to complete).
Initially, the data is stored in csv- files, which must be read by filling out the collections (for further processing).
Now I thing to use a NumPy- structured arrays (in my old C- program, I successfully used vector of structures).
What choice could you offer and why?
In my task, it is necessary to store in memory a very large quantity of records (~ 10,000,000 elements), each of which is on the order of 5-10 fields of different types (bool, string, integer, fixed-point number, date and time). It is necessary to be economical not only with memory, but also with processor time (otherwise I won’t wait for the processing to complete).
Initially, the data is stored in csv- files, which must be read by filling out the collections (for further processing).
Now I thing to use a NumPy- structured arrays (in my old C- program, I successfully used vector of structures).
What choice could you offer and why?