Python Forum
Please, advise collections for my task
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Please, advise collections for my task
#1
Python 3.7.3

In my task, it is necessary to store in memory a very large quantity of records (~ 10,000,000 elements), each of which is on the order of 5-10 fields of different types (bool, string, integer, fixed-point number, date and time). It is necessary to be economical not only with memory, but also with processor time (otherwise I won’t wait for the processing to complete).

Initially, the data is stored in csv- files, which must be read by filling out the collections (for further processing).

Now I thing to use a NumPy- structured arrays (in my old C- program, I successfully used vector of structures).

What choice could you offer and why?
Reply
#2
Why don't you try it out? Just couple of lines of code and you have initial idea what it will take. I have experience with smaller files (~200K lines) and the time consumed was so tiny that I had to measure it separately (it was ~0.2 sec). I read the file line by line and built nested dictionary along the way from data on rows for fast lookup later.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  count certain task in task manager[solved] kucingkembar 2 1,083 Aug-29-2022, 05:57 PM
Last Post: kucingkembar
  cs.collections error DaveG 7 1,687 Apr-06-2022, 04:18 PM
Last Post: Larz60+
  Trying to understand how isinstance(values, collections.Iterable) work. quazirfan 7 4,095 Aug-10-2021, 08:10 AM
Last Post: snippsat
  Schedule a task and render/ use the result of the task in any given time klllmmm 2 2,033 May-04-2021, 10:17 AM
Last Post: klllmmm
  How to create a task/import a task(task scheduler) using python Tyrel 7 3,627 Feb-11-2021, 11:45 AM
Last Post: Tyrel
  I need advise with developing a brute forcing script fatjuicypython 11 4,925 Aug-21-2020, 09:20 PM
Last Post: Marbelous
  AttributeError: 'collections.OrderedDict' object has no attribute 'value_counts Kristenl2784 4 7,266 Jul-17-2020, 01:50 AM
Last Post: palladium
  collections.OrderedDict Bert123456 2 1,748 Jul-09-2020, 08:51 PM
Last Post: Bert123456
  Absolutely new to python - basic advise needed mariolucas75 2 2,022 Jun-12-2020, 08:36 PM
Last Post: Yoriz
  AttributeError: module 'collections' has no attribute 'namedtuple' epgs1975 2 10,238 May-04-2020, 08:10 PM
Last Post: epgs1975

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020