Iterating Large Files

[Gribouillis]

I see Memory leaks due to append method.

Well, the most obvious recommendation would be to write a file sequentially instead of filling a numpy array until there is a memory leak. I'm not a numpy expert but the strategy would be

with open('export.txt', 'wb') as ofh:
    diff = [] # this is a python list, not a numpy array
    for <iteration within the input>:
        diff.extend(...) # use list's extend() or append() methods which are fast
        if len(diff) > 10000: # save diff list and reset it when it becomes too long
            np.savetxt(ofh, diff)
            diff = []

It is probably not lightspeed but it can do the work for reasonably sized files. As an example the following python loop with 1 billion numbers take less than 2 minutes on my computer

>>> def f():
...     L = []
...     start = time.time()
...     for i in range(10**9):
...         L.append(i)
...         if len(L) >= 10000:
...             L = []
...     print(time.time() - start, 'seconds')

beware of numpy.append() which rewrites the whole array each time. Benchmark your procedures.

As for the file, I use sometimes the trick to write a file on a ramdisk, which is fast and doesn't need real disk access. By using this trick, you could perhaps write directly segments of numpy arrays to the file, such as in

np.savetxt(ofh, x[i] - y[j:k])