Working with large volume of data (RAM is not enough)

[Larz60+]

Reading a file line by line (if text) will only use enough memory for the actual record (line):
Note: none of this code has been tested.
Instead of:

with open('Myfile.txt') as fp:
    buffer = fp.readlines()
for line in buffer:
   ... do stuff

which will read in the entire file,

use to read record by record:

with open('Myfile.txt') as fp:
    for line in fp:
       ... do stuff

only one record at a time.

This however doesn't help if it's a binary file. In this instance, you can read in chunks:
In which case open file as 'rb' and read chunk by chunk (keep in mind last chunk can be any size up to chunksize, including 0):

fp.read(chunksize)