Python Forum
Thread Rating:
  • 1 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Is Python Suitable?
I don't know if Python is a suitable language for what I want to do. In summary:

Read a file of about 6GB into an array (list I presume). The array/list has 6,000,000 entries each of about 1,000 bytes containing a mix of text, integer and real numbers.

Process the data. Loading it all into RAM is very desirable because effectively random subsets of the data must be processed up to about 1,000 times. Anyway, the subsets are very large.

In my youth, about 50 years ago, I used Fortran, Algol and Pascal. All of these have no important problems, but I was curious to learn something of Python to see if a claim made in a tutorial, that it can be used for anything, is correct. 

My experience says try it.
(In 64 bit python)
Looks like the limit is on number of list entries, not the size of each so I think you'll be ok
Depends what you do with each entry... Do you need to have everything in RAM at the same time, or are you processing things sequentially? In some case you can do things in two passes, a first one to get just the global info from each line (leading to potentially much less memory usage), and a second to process things sequentially.
Unless noted otherwise, code in my posts should be understood as "coding suggestions", and its use may require more neurones than the two necessary for Ctrl-C/Ctrl-V.
Your one-stop place for all your GIMP needs:
I would also say, yes, it's surely worth a try.
And: you should consider using numpy from the beginning. There are some tutorials online, but I can't evaluate the quality. Probably those who work with numpy already can say more about that.

How much experience do you have with python already? Are you 'familiar' with importing modules and so on?

Possibly Related Threads…
Thread Author Replies Views Last Post
  Suitable Data Modelling Packages JoeDainton123 3 327 Jun-27-2021, 01:43 PM
Last Post: Caprone

Forum Jump:

User Panel Messages

Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020