Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Please, advise collections for my task
Python 3.7.3

In my task, it is necessary to store in memory a very large quantity of records (~ 10,000,000 elements), each of which is on the order of 5-10 fields of different types (bool, string, integer, fixed-point number, date and time). It is necessary to be economical not only with memory, but also with processor time (otherwise I won’t wait for the processing to complete).

Initially, the data is stored in csv- files, which must be read by filling out the collections (for further processing).

Now I thing to use a NumPy- structured arrays (in my old C- program, I successfully used vector of structures).

What choice could you offer and why?
Why don't you try it out? Just couple of lines of code and you have initial idea what it will take. I have experience with smaller files (~200K lines) and the time consumed was so tiny that I had to measure it separately (it was ~0.2 sec). I read the file line by line and built nested dictionary along the way from data on rows for fast lookup later.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Absolutely new to python - basic advise needed mariolucas75 2 140 Jun-12-2020, 08:36 PM
Last Post: Yoriz
  AttributeError: module 'collections' has no attribute 'namedtuple' epgs1975 2 286 May-04-2020, 08:10 PM
Last Post: epgs1975
  Problem with importing and using collections module pythomdummy 3 438 Nov-14-2019, 08:48 AM
Last Post: Gribouillis
  What's the use case of collections.counter? magic 2 723 Sep-21-2018, 02:04 AM
Last Post: Mekire
  Drills/Exercises for collections, itertools, and common built-in data structures? Cranberry 2 1,219 Apr-26-2018, 05:43 PM
Last Post: Cranberry
  General advise for building a video manager ffrree 1 1,267 Oct-25-2017, 01:24 AM
Last Post: Larz60+

Forum Jump:

Users browsing this thread: 1 Guest(s)