Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Corrupted numpy arrays when save to file.
#1
I take a numpy array, and write it to a gzip file like this:
f = gzip.GzipFile("frames.npy.gz", "a")
np.save(file=f, arr=arr)
f.close()
However, when I try and extract it (on mac) with Archive Utility I get
Error:
Error 32 - Broken pipe
a similar error appears using Unarchiver. Using unzip in terminal results in the error:
Error:
End-of-central-directory signature not found.
I am writing to the gzip file in different processes which could definitely be causing the problem, yet I do that elsewhere in the script, with absolutely no problem extracting the file.

Force extracting the file, this is the data contained within:
ìNUMPY�v�{'descr': '|u1', 'fortran_order': False, 'shape': (24000, 18000, 4), }
This is located at the top - this looks fine to me.
Most of the file is then filled with:
ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇ
About 167 million lines down, there is some more data like this:
HXeˇP`mˇHXeˇ5ERˇ7GT
An then it continues with:
ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇ
for the rest of the file. This is clearly what is causing the problem.
The annoying this is, the arrays themselves are perfectly fine, so I can't figure out what is causing the corruption (if it is even python).
Quote
#2
Have you already tried f = gzip.open("frames.npy.gz", "a") instead of f = gzip.GzipFile("frames.npy.gz", "a")?
Quote
#3
(Dec-14-2019, 10:58 AM)ibreeden Wrote: Have you already tried f = gzip.open("frames.npy.gz", "a") instead of f = gzip.GzipFile("frames.npy.gz", "a")?
Yeah I have - unfortunately it doesn't make a difference.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Python numpy fft from data file magnet1 1 285 Feb-06-2020, 07:30 AM
Last Post: magnet1
  Numpy arrays and compatability with Fortran arrays merrittr 0 384 Sep-03-2019, 03:54 AM
Last Post: merrittr
  save my sensor data from the bme680 into a json or csv file Plastefuchs84 1 612 Aug-23-2019, 03:04 AM
Last Post: Plastefuchs84
  Converting set of tuples to set of 2D numpy arrays ThemePark 0 742 Aug-04-2019, 05:07 PM
Last Post: ThemePark
  numpy.copy / numpy.delete paul18fr 3 684 Jul-26-2019, 01:51 PM
Last Post: paul18fr
  Is there a way to save a CSV file as a python object amjass12 4 551 Jul-16-2019, 12:00 PM
Last Post: amjass12
  "erlarge" a numpy-matrix to numpy-array PhysChem 2 736 Apr-09-2019, 04:54 PM
Last Post: PhysChem
  numpy subtraction of two arrays devenuro 1 7,638 Sep-24-2018, 12:55 AM
Last Post: Mekire
  save 2d array to file and load back ian 3 8,306 May-18-2018, 05:00 AM
Last Post: scidam

Forum Jump:


Users browsing this thread: 1 Guest(s)