Python Forum
Corrupted numpy arrays when save to file.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Corrupted numpy arrays when save to file.
#1
I take a numpy array, and write it to a gzip file like this:
f = gzip.GzipFile("frames.npy.gz", "a")
np.save(file=f, arr=arr)
f.close()
However, when I try and extract it (on mac) with Archive Utility I get
Error:
Error 32 - Broken pipe
a similar error appears using Unarchiver. Using unzip in terminal results in the error:
Error:
End-of-central-directory signature not found.
I am writing to the gzip file in different processes which could definitely be causing the problem, yet I do that elsewhere in the script, with absolutely no problem extracting the file.

Force extracting the file, this is the data contained within:
ìNUMPY�v�{'descr': '|u1', 'fortran_order': False, 'shape': (24000, 18000, 4), }
This is located at the top - this looks fine to me.
Most of the file is then filled with:
ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇ
About 167 million lines down, there is some more data like this:
HXeˇP`mˇHXeˇ5ERˇ7GT
An then it continues with:
ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇ
for the rest of the file. This is clearly what is causing the problem.
The annoying this is, the arrays themselves are perfectly fine, so I can't figure out what is causing the corruption (if it is even python).
Reply
#2
Have you already tried f = gzip.open("frames.npy.gz", "a") instead of f = gzip.GzipFile("frames.npy.gz", "a")?
Reply
#3
(Dec-14-2019, 10:58 AM)ibreeden Wrote: Have you already tried f = gzip.open("frames.npy.gz", "a") instead of f = gzip.GzipFile("frames.npy.gz", "a")?
Yeah I have - unfortunately it doesn't make a difference.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [Numpy] How to store different data type in one numpy array? water 7 540 Mar-26-2024, 02:18 PM
Last Post: snippsat
  Pandas dataframes and numpy arrays bytecrunch 1 1,325 Oct-11-2022, 08:08 PM
Last Post: Larz60+
  Numpy returns "TypeError: unsupported operand type(s) for *: 'numpy.ufunc' and 'int'" kalle 2 2,602 Jul-19-2022, 06:31 AM
Last Post: paul18fr
  comparing floating point arrays to arrays of integers in Numpy amjass12 0 1,625 Jul-26-2021, 11:58 AM
Last Post: amjass12
  [solved] Save a matplotlib figure into hdf5 file paul18fr 1 2,513 Jun-08-2021, 05:58 PM
Last Post: paul18fr
  Python numpy fft from data file magnet1 1 2,753 Feb-06-2020, 07:30 AM
Last Post: magnet1
  Numpy arrays and compatability with Fortran arrays merrittr 0 1,872 Sep-03-2019, 03:54 AM
Last Post: merrittr
  save my sensor data from the bme680 into a json or csv file Plastefuchs84 1 3,142 Aug-23-2019, 03:04 AM
Last Post: Plastefuchs84
  Converting set of tuples to set of 2D numpy arrays ThemePark 0 2,607 Aug-04-2019, 05:07 PM
Last Post: ThemePark
  Is there a way to save a CSV file as a python object amjass12 4 2,743 Jul-16-2019, 12:00 PM
Last Post: amjass12

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020