About command 'joblib.load'

[middlestudent]

Hi, deanhystad. Many thanks for the kind reply.

The pkl file (about 32 mb) contains joint angle information of all time frames. I need to access all of them in my simulation work. Therefore, I would like to read all contents of the pkl file into a .txt format file. It is ideal that the data format in the .txt file is string for it can be used directly.

I have tried the recommended code:

import pickle
import json

data = list(range(1000))

with open("data.pkl", "wb") as file:
    pickle.dump(data, file)

with open("data.json", "w") as file:
    json.dump(data, file)

Yet, running error occurred as follows:

Traceback (most recent call last):
File "<stdin>", line 2, in <module>
_pickle.UnpicklingError: invalid load key, '\x04'.

May I know whether there is a way to read all contents of a pkl file (about 32mb) into a txt file? Many thanks. Yuan

Why are you using a pickle file? It is an odd choice for any kind of data archiving.

Normally a picke file will be much smaller than a file that saves the same information as text. For example, the program below creates a 3KB pickle file and a 5KB json file.

import pickle
import json

data = list(range(1000))

with open("data.pkl", "wb") as file:
    pickle.dump(data, file)

with open("data.json", "w") as file:
    json.dump(data, file)

The ellipsis indicate that not all the data is printed. This is a kindness performed by many python packages when asked to convert large aarrays/lists to a string.

Output:
[ 0.04885907, -0.9572657 , -0.03051978],
...,   <- Indicates not all data was printed
[-0.09611149, -0.9011042 , -0.04491276],

What do you want done? What is the purpose of the txt file? if you want a text based archive file that you can move from machine to machine, I suggest using json format.

import pickle
import json

data = list(range(1000))

with open("data.pkl", "rb") as file:
    data = pickle.load(file)

with open("data.json", "w") as file:
    json.dump(data, file)

If you want a machine independent binary format, I would look at using numpy savez

https://numpy.org/doc/stable/reference/g...savez.html

If you just want a text file so you can look at it. Hey, you said this thing is 32MB