Iterating Large Files

[Gribouillis]

I realize that I will need quick access to the (huge) data I save.

The first thing to clarify is how quick must the access be and how huge is the data file, especially the third file created by the others. Ram is typically small. On my computer, I have 16GB of Ram. Clearly, I wont be able to keep tens of GB in Ram at a time.

Se the main question is what do you want to do with the third file? You can still create histograms if the huge file is written on disk.

You could perhaps have a look at the PyTables module, based on HDF5, which was specially designed to handle large sets of data.