Python Forum
Automating to save generated data
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Automating to save generated data
#1
Hey everyone:

I want to save a particular number of values in maps I create. For example, when creating (4064x1) values, I want to save first (1000x1) in map1, next (1000x1) in map2 and so on. The last map will have remaining (64x1) elements.

Now the issue is I want to automate this as the number 4064 varies based on data I analyze. Here is simplsitic version of something I tried and is working (L is 1000 and index is reset to zero as I move from one map to the other).

Any suggestions on how to make this creative?

L = 1000
count = 0
fp1 = np.memmap('map1.dat', dtype='float64', mode='w+', shape=(L,1))
fp2 = np.memmap('map2.dat', dtype='float64', mode='w+', shape=(L,1))
fp3 = np.memmap('map3.dat', dtype='float64', mode='w+', shape=(L,1))
...

if count < L:
    fp1[index,0] = delta

if count == L:
    index = 0

if L <= count < 2*L:
    fp2[index,0] = delta

if count == 2*L:
    index = 0

if 2*L <= count < 3*L:
    fp3[index,0]=delta

...

count += 1 
Reply
#2
I didn't test it, but something like the following should help you:

L = 1000
data = np.random.rand(4064, 1)

for j in range(data.shape[0] // L):
    mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L,1))
    mapper = data[j*L:j*L+L]

if data.shape[0] % L != 0:
    mapper = np.memmap(f'map{j+1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L,1))
    mapper = data[j*L + L:]
Reply
#3
Thanks, I tried that. Although it runs fine, I wonder why the information is not stores in the map.dat files created.

That is, when I read via print(f[:10]), I get all zeros. Do you know what's happening?


L = 1000
data = np.random.rand(4064, 1)
print(data.shape[0])

for j in range(data.shape[0] // L):
    mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L, 1))
    mapper = data[j * L:j * L + L]
    print(mapper[:10])

if data.shape[0] % L != 0:
    mapper = np.memmap(f'map{j + 1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L, 1))
    mapper = data[j * L + L:]
    print(mapper.shape)

# Reading the maps
L = 10**3
M = 1
f = np.memmap('map1.dat', dtype='float64', mode='r', shape=(L,M))
print(f[:10])


(Aug-12-2020, 12:51 AM)scidam Wrote: I didn't test it, but something like the following should help you:

L = 1000
data = np.random.rand(4064, 1)

for j in range(data.shape[0] // L):
    mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L,1))
    mapper = data[j*L:j*L+L]

if data.shape[0] % L != 0:
    mapper = np.memmap(f'map{j+1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L,1))
    mapper = data[j*L + L:]
Reply
#4
Using mapper[:] instead of mapper on line-7 and line-12 (in the shared code) solved the problem.

Thanks again!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Automating to generate multiple arrays Robotguy 1 1,780 Nov-05-2020, 08:14 AM
Last Post: Gribouillis
  Automating PyTables Dataset Creation and Append Robotguy 1 1,768 Oct-18-2020, 08:35 PM
Last Post: jefsummers
  save my sensor data from the bme680 into a json or csv file Plastefuchs84 1 3,103 Aug-23-2019, 03:04 AM
Last Post: Plastefuchs84
  save parameters an data from image to csv pifko22 0 1,916 Apr-02-2019, 10:22 PM
Last Post: pifko22
  save video frames into pandas data-frame tofi 0 2,648 Oct-18-2018, 07:02 PM
Last Post: tofi
  Automating the code using os.walk kiton 20 14,160 Apr-13-2017, 06:15 PM
Last Post: kiton

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020