Python Forum
Question about deserializing some numbers (bug??)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Question about deserializing some numbers (bug??)
#1
Smile 
Hi! In order to deserialize bytes object, we use pickle.loads():

import pickle
import numpy as np
pickle.loads(np.float64(0.34103))

and the expected result is like below (because np.float64(0.34103) is not bytes objects, appropriate errors are expected)
---------------------------------------------------------------------------
UnpicklingError Traceback (most recent call last)
<ipython-input-19-5c07606a60f1> in <module>
----> 1 pickle.loads(np.float64(0.34103))

UnpicklingError: invalid load key, '\xc1'.


Here we have some questions that some numbers (it is rare) like 0.34104 prints the following result without errors.
pickle.loads(np.float64(0.34104))
=> True

This occurs only when the converted bytes start with b'\x88 (for example 0.04263, 0.08526, 0.11651 ...)
np.float64(0.34104).tobytes()
=> b'\x88.\xa8o\x99\xd3\xd5?'
Can anyone answer whether this issue is Python bugs?

Any answer will be highly appreciated. Smile
Reply
#2
It seems that you are attempting an invalid operation. pickle.loads() takes for its first argument a bytes sequence that was created by pickle.dumps(). Here you are passing a numpy.float64 instance. One cannot just pass anything to pickle.loads() and expect it to work.
Reply
#3
(Oct-27-2020, 07:43 AM)Gribouillis Wrote: It seems that you are attempting an invalid operation. pickle.loads() takes for its first argument a bytes sequence that was created by pickle.dumps(). Here you are passing a numpy.float64 instance. One cannot just pass anything to pickle.loads() and expect it to work.

Regardless of invalid operation, can anyone explain why "pickle.loads(np.float64(0.34104))" prints "True"?
Reply
#4
It turns out that pickle.loads() starts by converting its argument to BytesIO (unless the argument is a str, see in pickle.py), and it turns out that a numpy float can be converted to a BytesIO
>>> import numpy as np
>>> thing = np.float64(0.34104)
>>> import io
>>> file = io.BytesIO(thing)
>>> file.getvalue()
b'\x88.\xa8o\x99\xd3\xd5?'
It means that pickle.loads() thinks that it receives a sequence of bytes. This leaves two questions open
  1. Shouldn't pickle.loads() check more thoroughly its arguments?
  2. Why are numpy floats convertible to BytesIO?
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020