About command 'joblib.load'

[middlestudent]

Dear All,

I applied joblib.load command to access a pkl file (about 32 mb). The commands are as follows:

import joblib
output = joblib.load('/home/yuan/motion_data/output.pkl') 
file=open('/home/yuan/motion_data/data.txt', 'w')
output_str=str(output)
file.write(output_str)
file.close()

Finally, the content of data.txt is as follows:
{1: {'pred_cam': array([[1.0553476 , 0.01870966, 0.16412038],
[1.069203 , 0.02064601, 0.1643776 ],
[1.0785335 , 0.02310706, 0.16229492],
...,
[1.0573285 , 0.02703779, 0.13177261],
[1.0552672 , 0.03153537, 0.11069966],
[1.0567206 , 0.01587683, 0.11882061]], dtype=float32), 'orig_cam': array([[ 0.35814508, 0.63670237, -1.4263681 , 0.76267481],
[ 0.3628471 , 0.64506152, -1.40570545, 0.75517556],
[ 0.36604017, 0.65073807, -1.39128891, 0.74778145],
...,
[ 0.260801 , 0.46364623, -1.1766514 , 0.75617363],
[ 0.26573757, 0.47242235, -1.2046633 , 0.74890999],
[ 0.26649146, 0.47376259, -1.23360063, 0.75865443]]), 'verts': array([[[ 0.04407381, -0.97446126, -0.03605718],
[ 0.04699752, -0.96045536, -0.04436788],
[ 0.04885907, -0.9572657 , -0.03051978],
...,
[-0.09611149, -0.9011042 , -0.04491276],
[-0.09540198, -0.9027417 , -0.04659075],
[-0.09525803, -0.9002895 , -0.05021533]],

[[ 0.03895233, -0.9746224 , -0.03506504],
[ 0.04191986, -0.96062875, -0.04343657],
[ 0.04388162, -0.95741135, -0.02961013],
...,
[-0.10088779, -0.90039927, -0.04299552],
[-0.10020136, -0.9020486 , -0.04467536],
[-0.1000717 , -0.8996091 , -0.04830974]],

[[ 0.03465913, -0.97380745, -0.03328603],
[ 0.03771285, -0.9597902 , -0.041634 ],
[ 0.03962472, -0.9565819 , -0.02780933],
...,
[-0.10485663, -0.8991722 , -0.04168364],
[-0.10416569, -0.9008289 , -0.04335511],
[-0.10401696, -0.8983972 , -0.04699068]],

...,

[[-0.07685035, -0.9303616 , -0.09653595],
[-0.08428618, -0.9180497 , -0.08947386],
[-0.07680301, -0.91232145, -0.0999243 ],
...,
[ 0.01785447, -0.8800436 , 0.01579271],
[ 0.01639317, -0.88178813, 0.01616666],
[ 0.01396976, -0.8801436 , 0.01920865]],

[[-0.07783334, -0.9165481 , -0.10046807],
[-0.08480501, -0.9039606 , -0.09363362],
[-0.07679046, -0.8985981 , -0.10374232],
...,
[ 0.0154784 , -0.870908 , 0.01387965],
[ 0.01397827, -0.8726083 , 0.01418496],
[ 0.01157701, -0.8709144 , 0.01714143]],

[[-0.06782427, -0.9243063 , -0.09490766],
[-0.07560667, -0.9120439 , -0.08857992],
[-0.06730784, -0.90634763, -0.09830321],
...,
[ 0.01604176, -0.8735553 , 0.02478523],
[ 0.01463456, -0.87535316, 0.02498878],
[ 0.01198672, -0.8737435 , 0.02775544]]], dtype=float32), 'pose': array([[ 2.7535458 , -0.17776728, 1.3685989 , ..., -0.05098459,
-0.09550607, 0.32302174],
[ 2.7576423 , -0.1838874 , 1.3607229 , ..., -0.05438529,
-0.0949294 , 0.31558415],
[ 2.7435265 , -0.19096005, 1.3850489 , ..., -0.05228679,
-0.09980644, 0.31189477],
...,
[-2.2775056 , -0.20539762, 2.1377597 , ..., 0.02529589,
-0.09647566, 0.38429365],
[-2.2966738 , -0.17553501, 2.121143 , ..., 0.02220837,
-0.11700007, 0.3954935 ],
[-2.2970102 , -0.13022876, 2.1242604 , ..., 0.02945088,
-0.11544169, 0.39877096]], dtype=float32), 'betas': array([[-0.41963667, 0.9198057 , 0.7524594 , ..., 0.4803348 ,
0.57579195, -0.08520291],
[-0.4237536 , 0.8905467 , 0.76004124, ..., 0.49365988,
0.59498155, -0.08117235],
[-0.41443723, 0.8947872 , 0.8012768 , ..., 0.4965011 ,
0.58433056, -0.08212025],
...,
[-0.3477986 , 1.6840414 , 0.7496066 , ..., 0.40488756,
0.422943 , -0.06801946],
[-0.36453918, 1.8236454 , 0.6945118 , ..., 0.42299372,
0.4311611 , -0.03063928],
[-0.3718453 , 1.798913 , 0.71441627, ..., 0.4222765 ,
0.4136417 , -0.03111994]], dtype=float32), 'joints3d': array([[[ 0.04802994, -0.8844861 , -0.08683674],
[-0.05735164, -0.75794244, 0.02535598],
[-0.14529027, -0.6305741 , -0.11340863],
...,
[-0.00445663, -0.9176111 , -0.08591709],
[ 0.01135602, -0.9115541 , 0.05360784],
[-0.09287867, -0.89153147, -0.04215819]],

[[ 0.04305333, -0.884678 , -0.08603188],
[-0.06059415, -0.75722826, 0.02656614],
[-0.14957744, -0.63000906, -0.11135776],
...,
[-0.00958662, -0.9174992 , -0.0847047 ],
[ 0.00724642, -0.9112784 , 0.05474848],
[-0.09756899, -0.89084816, -0.04025927]],

[[ 0.03917626, -0.8837117 , -0.08411616],
[-0.06457388, -0.75608784, 0.02769488],
[-0.15164557, -0.63040334, -0.11157695],
...,
[-0.01349489, -0.9164252 , -0.08309188],
[ 0.00275986, -0.9103344 , 0.0564117 ],
[-0.10150856, -0.8896405 , -0.03890951]],

...,

[[-0.11441365, -0.8512248 , -0.04346032],
[ 0.03886513, -0.72610986, -0.03916029],
[ 0.05068082, -0.6475865 , 0.13365033],
...,
[-0.07579784, -0.8909767 , -0.01703947],
[ 0.00456081, -0.8584138 , -0.12312402],
[ 0.01712222, -0.8701596 , 0.0137368 ]],

[[-0.11293334, -0.8358046 , -0.04848906],
[ 0.04326482, -0.7170633 , -0.04007216],
[ 0.06035992, -0.6425956 , 0.13077132],
...,
[-0.07728785, -0.8774177 , -0.0211476 ],
[ 0.00702812, -0.84840786, -0.12386235],
[ 0.01525279, -0.8610977 , 0.01176213]],

[[-0.10980754, -0.8449346 , -0.04609802],
[ 0.04113907, -0.7193877 , -0.0299248 ],
[ 0.05009299, -0.6414673 , 0.1406691 ],
...,
[-0.07406253, -0.8847761 , -0.01615553],
[ 0.01514768, -0.85267365, -0.113653 ],
[ 0.01550194, -0.863771 , 0.02253311]]], dtype=float32), 'joints2d': None, 'bboxes': array([[463.15440369, 745.79455566, 651.57544807, 651.57544807],
[463.15440369, 745.79455566, 651.57544807, 651.57544807],
[462.98329416, 745.73912278, 651.62287433, 651.62287433],
...,
[658.63358587, 696.33063598, 473.58786246, 473.58786246],
[644.6357459 , 702.81260299, 483.49472952, 483.49472952],
[640.34393602, 703.68983861, 484.19950397, 484.19950397]]), 'frame_ids': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103,
104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,
130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,
156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,
169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220,
221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,
247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259,
260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272,
273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285,
286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298,
299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311,
312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324,
325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337,
338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350,
351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363,
364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376,
377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389])}}

My questions are :
1. the file data.txt is only 9 kb, much less than the orignal pkl file. Is it normal?
2. In the data.txt file, there are many .... It seems that the data in thr original pkl file was not read completely. If yes, is there any solution to read the pkl file completely?

May I get your suggestion? Many thanks.

[deanhystad]

Why are you using a pickle file? It is an odd choice for any kind of data archiving.

Normally a picke file will be much smaller than a file that saves the same information as text. For example, the program below creates a 3KB pickle file and a 5KB json file.

import pickle
import json

data = list(range(1000))

with open("data.pkl", "wb") as file:
    pickle.dump(data, file)

with open("data.json", "w") as file:
    json.dump(data, file)

The ellipsis indicate that not all the data is printed. This is a kindness performed by many python packages when asked to convert large aarrays/lists to a string.

Output:
[ 0.04885907, -0.9572657 , -0.03051978],
...,   <- Indicates not all data was printed
[-0.09611149, -0.9011042 , -0.04491276],

What do you want done? What is the purpose of the txt file? if you want a text based archive file that you can move from machine to machine, I suggest using json format.

import pickle
import json

data = list(range(1000))

with open("data.pkl", "rb") as file:
    data = pickle.load(file)

with open("data.json", "w") as file:
    json.dump(data, file)

If you want a machine independent binary format, I would look at using numpy savez

https://numpy.org/doc/stable/reference/g...savez.html

If you just want a text file so you can look at it. Hey, you said this thing is 32MB

[middlestudent]

Hi, deanhystad. Many thanks for the kind reply.

The pkl file (about 32 mb) contains joint angle information of all time frames. I need to access all of them in my simulation work. Therefore, I would like to read all contents of the pkl file into a .txt format file. It is ideal that the data format in the .txt file is string for it can be used directly.

I have tried the recommended code:

import pickle
import json

data = list(range(1000))

with open("data.pkl", "wb") as file:
    pickle.dump(data, file)

with open("data.json", "w") as file:
    json.dump(data, file)

Yet, running error occurred as follows:

Traceback (most recent call last):
File "<stdin>", line 2, in <module>
_pickle.UnpicklingError: invalid load key, '\x04'.

May I know whether there is a way to read all contents of a pkl file (about 32mb) into a txt file? Many thanks. Yuan

Why are you using a pickle file? It is an odd choice for any kind of data archiving.

Normally a picke file will be much smaller than a file that saves the same information as text. For example, the program below creates a 3KB pickle file and a 5KB json file.

import pickle
import json

data = list(range(1000))

with open("data.pkl", "wb") as file:
    pickle.dump(data, file)

with open("data.json", "w") as file:
    json.dump(data, file)

The ellipsis indicate that not all the data is printed. This is a kindness performed by many python packages when asked to convert large aarrays/lists to a string.

Output:
[ 0.04885907, -0.9572657 , -0.03051978],
...,   <- Indicates not all data was printed
[-0.09611149, -0.9011042 , -0.04491276],

What do you want done? What is the purpose of the txt file? if you want a text based archive file that you can move from machine to machine, I suggest using json format.

import pickle
import json

data = list(range(1000))

with open("data.pkl", "rb") as file:
    data = pickle.load(file)

with open("data.json", "w") as file:
    json.dump(data, file)

If you want a machine independent binary format, I would look at using numpy savez

https://numpy.org/doc/stable/reference/g...savez.html

If you just want a text file so you can look at it. Hey, you said this thing is 32MB

[deanhystad]

The "recommended" code of mine in your post doesn't do any unpickling. It is incapable of generating an unpickling error. Not sure what you are talking about.

To print out long numpy arrays without ". . ." you can set the numpy print options

https://numpy.org/doc/stable/reference/g...tions.html

import joblib
import sys
import numpy as np

np.set_printoptions(threshold=sys.maxsize)
output = joblib.load('/home/yuan/motion_data/output.pkl') 
with open('/home/yuan/motion_data/data.txt', 'w') as file:
    print(output, file=file)

[middlestudent]

Hi, deanhystad. Using your method, I can obtain complete data (about 165 mb) now. Thank you very much. Yuan

The "recommended" code of mine in your post doesn't do any unpickling. It is incapable of generating an unpickling error. Not sure what you are talking about.

To print out long numpy arrays without ". . ." you can set the numpy print options

https://numpy.org/doc/stable/reference/g...tions.html

import joblib
import sys
import numpy as np

np.set_printoptions(threshold=sys.maxsize)
output = joblib.load('/home/yuan/motion_data/output.pkl') 
with open('/home/yuan/motion_data/data.txt', 'w') as file:
    print(output, file=file)