pickle or txt

[Gribouillis]

Pickle can store several objects by successive calls to dump. These objects can be retrieved one by one, which solves your memory issue.

from pathlib import Path
import pickle

this_dir = Path(__file__).parent

idata = ['some spam', '4 slices of ham', '12 eggs']

filename = this_dir / 'idata.pkl'

# pickle a sequence of objects one by one
with filename.open('wb') as ofh:
    pkl = pickle.Pickler(ofh)
    for x in idata:
        pkl.dump(x)

# unpickle a sequence of objects one by one
with filename.open('rb') as ifh:
    pkl = pickle.Unpickler(ifh)
    try:
        while True:
            x = pkl.load()
            print(x)
    except EOFError:
        pass

Output:
λ python paillasse/pf/pick.py
some spam
4 slices of ham
12 eggs