Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Please, advise collections for my task
#1
Python 3.7.3

In my task, it is necessary to store in memory a very large quantity of records (~ 10,000,000 elements), each of which is on the order of 5-10 fields of different types (bool, string, integer, fixed-point number, date and time). It is necessary to be economical not only with memory, but also with processor time (otherwise I won’t wait for the processing to complete).

Initially, the data is stored in csv- files, which must be read by filling out the collections (for further processing).

Now I thing to use a NumPy- structured arrays (in my old C- program, I successfully used vector of structures).

What choice could you offer and why?
Quote
#2
Why don't you try it out? Just couple of lines of code and you have initial idea what it will take. I have experience with smaller files (~200K lines) and the time consumed was so tiny that I had to measure it separately (it was ~0.2 sec). I read the file line by line and built nested dictionary along the way from data on rows for fast lookup later.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Absolutely new to python - basic advise needed mariolucas75 2 140 Jun-12-2020, 08:36 PM
Last Post: Yoriz
  AttributeError: module 'collections' has no attribute 'namedtuple' epgs1975 2 286 May-04-2020, 08:10 PM
Last Post: epgs1975
  Problem with importing and using collections module pythomdummy 3 438 Nov-14-2019, 08:48 AM
Last Post: Gribouillis
  What's the use case of collections.counter? magic 2 723 Sep-21-2018, 02:04 AM
Last Post: Mekire
  Drills/Exercises for collections, itertools, and common built-in data structures? Cranberry 2 1,219 Apr-26-2018, 05:43 PM
Last Post: Cranberry
  General advise for building a video manager ffrree 1 1,267 Oct-25-2017, 01:24 AM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)