Python Forum

Full Version: Create a 2-channel numpy file from two cvs file containing a 9x9 matrix
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hello everyone, I work in a CNN project for my thesis and I have problem with my Python code. I hope that somebody can help me. Thank you.

I would like to take two .cvs files with the same name (each is a 9x9 matrix) from a folder and create a unique file of the size 9x9x2 and save it in another folder.

This is my code, right now the problem is that the final file has been saved as a 9x18 instead of 9x9x2.

import os, glob
from collections import defaultdict

dirs = ['/content/drive/MyDrive/Colab Notebooks/Nuova cartella/mu1/real', '/content/drive/MyDrive/Colab Notebooks/Nuova cartella/mu1/imag']
file_pattrn = r'*.csv'
unique_files = defaultdict(list)

for d in dirs:
  for i in glob.iglob(os.path.join(d, file_pattrn)):
     unique_files[os.path.basename(i)].append(i)

destination = '/content/drive/MyDrive/Colab Notebooks/New_Dataset'
for unique_filename, copies in unique_files.items():
  with open(os.path.join(destination, unique_filename), 'w') as f:
      for copy in copies:
          with open(copy, 'r') as cp:
            for line in cp:
                f.write(line)
Why not something like (untested)
import numpy as np
for unique_filename, copies in unique_files.items():
    a = np.array([np.loadtxt(copy) for copy in copies])
    np.savetxt(os.path.join(destination, unique_filename), a)
Well actually, the above code doesn't work because numpy.savetxt() cannot handle 3D arrays. You can save the 3D array under binary format by using numpy.save() with the allow_pickle=True option. Here is an example with 3x3 arrays
import numpy as np
import io

file1 = io.StringIO("""\
1.000000000000000000e+00 2.000000000000000000e+00 3.000000000000000000e+00
5.000000000000000000e+00 6.000000000000000000e+00 7.000000000000000000e+00
9.000000000000000000e+00 1.000000000000000000e+01 1.100000000000000000e+01
""")

file2 = io.StringIO("""\
-1.000000000000000000e+00 -2.000000000000000000e+00 -3.000000000000000000e+00
-5.000000000000000000e+00 -6.000000000000000000e+00 -7.000000000000000000e+00
-9.000000000000000000e+00 -1.000000000000000000e+01 -1.100000000000000000e+01
""")

out = io.BytesIO()
a = np.array([np.loadtxt(f) for f in (file1, file2)])
a = np.moveaxis(a, 0, -1)
np.save(out, a, allow_pickle=True)
out.seek(0)
print(np.load(out, allow_pickle=True))
print(a.shape)
Output:
[[[ 1. -1.] [ 2. -2.] [ 3. -3.]] [[ 5. -5.] [ 6. -6.] [ 7. -7.]] [[ 9. -9.] [ 10. -10.] [ 11. -11.]]] (3, 3, 2)