Python Forum

Hey everyone:

I want to save a particular number of values in maps I create. For example, when creating (4064x1) values, I want to save first (1000x1) in map1, next (1000x1) in map2 and so on. The last map will have remaining (64x1) elements.

Now the issue is I want to automate this as the number 4064 varies based on data I analyze. Here is simplsitic version of something I tried and is working (L is 1000 and index is reset to zero as I move from one map to the other).

Any suggestions on how to make this creative?

L = 1000
count = 0
fp1 = np.memmap('map1.dat', dtype='float64', mode='w+', shape=(L,1))
fp2 = np.memmap('map2.dat', dtype='float64', mode='w+', shape=(L,1))
fp3 = np.memmap('map3.dat', dtype='float64', mode='w+', shape=(L,1))
...

if count < L:
    fp1[index,0] = delta

if count == L:
    index = 0

if L <= count < 2*L:
    fp2[index,0] = delta

if count == 2*L:
    index = 0

if 2*L <= count < 3*L:
    fp3[index,0]=delta

...

count += 1 

I didn't test it, but something like the following should help you:

L = 1000
data = np.random.rand(4064, 1)

for j in range(data.shape[0] // L):
    mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L,1))
    mapper = data[j*L:j*L+L]

if data.shape[0] % L != 0:
    mapper = np.memmap(f'map{j+1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L,1))
    mapper = data[j*L + L:]

Thanks, I tried that. Although it runs fine, I wonder why the information is not stores in the map.dat files created.

That is, when I read via print(f[:10]), I get all zeros. Do you know what's happening?

L = 1000
data = np.random.rand(4064, 1)
print(data.shape[0])

for j in range(data.shape[0] // L):
    mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L, 1))
    mapper = data[j * L:j * L + L]
    print(mapper[:10])

if data.shape[0] % L != 0:
    mapper = np.memmap(f'map{j + 1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L, 1))
    mapper = data[j * L + L:]
    print(mapper.shape)

# Reading the maps
L = 10**3
M = 1
f = np.memmap('map1.dat', dtype='float64', mode='r', shape=(L,M))
print(f[:10])

(Aug-12-2020, 12:51 AM)scidam Wrote: [ -> ]I didn't test it, but something like the following should help you:

L = 1000
data = np.random.rand(4064, 1)

for j in range(data.shape[0] // L):
    mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L,1))
    mapper = data[j*L:j*L+L]

if data.shape[0] % L != 0:
    mapper = np.memmap(f'map{j+1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L,1))
    mapper = data[j*L + L:]

Using mapper[:] instead of mapper on line-7 and line-12 (in the shared code) solved the problem.

Thanks again!