pickle problem

[DPaul]

Hi,
I can write and read pickle files, no problem.
But if I lookup how to merge a pickle file to an existing pickle file, I get somewhat confused,
I find:

pickle.dump((pickle_new), open(old_pickle, 'ab'))  # does not seem to work

I have control over the alternatives:
a) I have a text file or list that i can append to the old_pickle, line by line.
b) I have said text also as a pickle file that i could append to the old_pickle.
What is the best way?
Obviously old and new files follow the same format.
thx,
Paul

[buran]

what kind of object is pickle_new, exactly?
And what exactly you expect to happen? At best, I think, you will end up with pickled pickle.Pickler object

[DPaul]

Hi Buran,
Pickle_old is a *.pkl file created with

pkl = pickle.Pickler(pickle_old)
for line in sourcefile:
        pkl.dump(line)

Pickle_new is done exactly the same, but afterwards i have to append it to pickle.old.

My confusion is that some say I have to read the old file first and then
combine that with the new file, and then save both together, but that seems cumbersome.
thx,
Paul

[buran]

Sorry, your explanation is even more confusing to me. I'm not using pickle (because of the well-known security issues, but not only), but what you show is not at all how I understand the pickle working. Maybe I am not correct in my understanding.

[DPaul]

OK, I did not invent this, I got the code from one
of your most learned colleagues on this forum.
I still don't understand however, how i can append one
pickle file to another in a straightforward manner.
If that is not possible, I'd like to know.
Paul

[deanhystad]

Pickling is just a way to serialize a data structure. If you are using it for data storage it makes sense you would data = pickle.load(file), modify data, pickle.dump(data, file).

EXAMPLE 1

import pickle

animals = ["tiger", "lion", "giraffe"]
pickled = pickle.dumps(animals)
print(pickled)

things = pickle.loads(pickled)
animals = things + ["cow", "sheep", "pig"]
pickled = pickle.dumps(animals)
print(pickled)
things = pickle.loads(pickled)
print(things)

Output:
b'\x80\x04\x95\x1e\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x05tiger\x94\x8c\x04lion\x94\x8c\x07giraffe\x94e.'
b'\x80\x04\x952\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x05tiger\x94\x8c\x04lion\x94\x8c\x07giraffe\x94\x8c\x03cow\x94\x8c\x05sheep\x94\x8c\x03pig\x94e.'
['tiger', 'lion', 'giraffe', 'cow', 'sheep', 'pig']

It does not make sense appending to a pickled file. You are concatenating pickles.

EXAMPLE 2

import pickle

animals = ["tiger", "lion", "giraffe"]
pickled = pickle.dumps(animals)
print(pickled)
things = pickle.loads(pickled)
print(things)

pickled = pickled + pickle.dumps(["cow", "sheep", "pig"])
print(pickled)
things = pickle.loads(pickled)
print(things)

Output:
b'\x80\x04\x95\x1e\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x05tiger\x94\x8c\x04lion\x94\x8c\x07giraffe\x94e.'
['tiger', 'lion', 'giraffe']
b'\x80\x04\x95\x1e\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x05tiger\x94\x8c\x04lion\x94\x8c\x07giraffe\x94e.\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x03cow\x94\x8c\x05sheep\x94\x8c\x03pig\x94e.'
['tiger', 'lion', 'giraffe']

Appending to "pickled" makes it longer, but it is not properly formatted to make a longer list.

Or are you trying to save multiple data structures to a pickle file?

EXAMPLE 3

import pickle

jungle = ["tiger", "lion", "giraffe"]
farm = ["cow", "sheep", "pig"]

with open("test.bin", "wb") as file:
    pickle.dump(jungle, file)

with open("test.bin", "ab") as file:  # Appending to an existing pickle file
    pickle.dump(farm, file)

with open("test.bin", "rb") as file:
    while True:
        try:
            print(pickle.load(file))  # Need to call pickle.load() for each pickle.dump() used to make the file
        except EOFError:
            break

Output:
['tiger', 'lion', 'giraffe']
['cow', 'sheep', 'pig']

I don't see this as being particularly useful, but there may be a special case where this makes sense. Usually if you want to pickle multiple data structures you put them in a list and pickle the list.

EXAMPLE 4

import pickle

jungle = ["tiger", "lion", "giraffe"]
farm = ["cow", "sheep", "pig"]

with open("test.bin", "wb") as file:
    pickle.dump([jungle, farm], file)


with open("test.bin", "rb") as file:
    a, b = pickle.load(file)
print(a, b)

Output:
['tiger', 'lion', 'giraffe'] ['cow', 'sheep', 'pig']

Is Gribouillis the learned colleague? I found the link here:

https://python-forum.io/thread-38018.htm...ght=pickle

I don't know if this is one of the "special cases" where it makes sense to append to a "pickled" file. As shown in EXAMPLE 3, you'll need to call pickle.load() multiple times if you called pickle.dump() multiple times, or used pickle.dump() to append to an existing file.

[DPaul]

@deanh : thanks for a much appreciated answer.
I found the pickle snippets I read on various other websites
a trifle confusing, because they did not say what I wanted to hear.

If i combine your comments with what Gribouillis wrote,
I understand I was barking up the wrong tree.
I do have a plan B, I'll do the appending somewhere else before
creating the pickle file. Piece of cake.
Paul

[Gribouillis]

What is the issue? It is just a feature of the pickle module that a pickle file can store several serialized Python objects one after the other. Do you have a problem appending new objects to an old pickle file? Can you post a reproducible example where this does not work?

[DPaul]

What I understand now from appending to an existing pickle file,
is that you end up with 2... objects in the same pickle file.
Hence my confusion, i originally thought i could do something like list.append.
Consider it solved.
Paul

[Gribouillis]

you end up with 2... objects in the same pickle file.

Here is an example of this feature with sockets: A client connects to a server and sends three Python objects by successive calls to pickle.dump() in the same file-like object. The server receives the Python objects and prints them.

Server code:

# pickserver.py
import pickle
import socket
import sys

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

# Bind the socket to the port
server_address = ('localhost', 10000)
print('starting up on {} port {}'.format(*server_address))
sock.bind(server_address)

# Listen for incoming connections
sock.listen(1)

while True:
    # Wait for a connection
    print('waiting for a connection')
    connection, client_address = sock.accept()
    try:
        print('connection from', client_address)
        connection.shutdown(socket.SHUT_WR)
        file = connection.makefile(mode='rb')
        unpic = pickle.Unpickler(file)
        # Receive Python objects and print them
        while True:
            try:
                obj = unpic.load()
            except EOFError:
                break
            else:
                print(f'received: {repr(obj)}')
    finally:
        # Clean up the connection
        connection.close()

Client code:

# pickclient.py
import pickle
import socket
import sys

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Connect the socket to the port where the server is listening
server_address = ('localhost', 10000)
print('connecting to {} port {}'.format(*server_address))
sock.connect(server_address)
sock.shutdown(socket.SHUT_RD)
file = sock.makefile(mode='wb')
pick = pickle.Pickler(file)

try:
    # Send Python objects
    data = [
        10,
        [2, 'spam', 'eggs'],
        (5, 7, 11),
    ]
    for obj in data:
        print(f'sending: {repr(obj)}')
        pick.dump(obj)

finally:
    print('closing socket')
    sock.close()

Output on the server side:

Output:
λ python paillasse/pf/pickserver.py
starting up on localhost port 10000
waiting for a connection
connection from ('127.0.0.1', 34014)
received: 10
received: [2, 'spam', 'eggs']
received: (5, 7, 11)
waiting for a connection

Output on the client side

Output:
λ python paillasse/pf/pickclient.py
connecting to localhost port 10000
sending: 10
sending: [2, 'spam', 'eggs']
sending: (5, 7, 11)
closing socket