Automating to save generated data - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Data Science (https://python-forum.io/forum-44.html) +--- Thread: Automating to save generated data (/thread-28964.html) |
Automating to save generated data - Robotguy - Aug-11-2020 Hey everyone: I want to save a particular number of values in maps I create. For example, when creating (4064x1) values, I want to save first (1000x1) in map1, next (1000x1) in map2 and so on. The last map will have remaining (64x1) elements. Now the issue is I want to automate this as the number 4064 varies based on data I analyze. Here is simplsitic version of something I tried and is working (L is 1000 and index is reset to zero as I move from one map to the other). Any suggestions on how to make this creative? L = 1000 count = 0 fp1 = np.memmap('map1.dat', dtype='float64', mode='w+', shape=(L,1)) fp2 = np.memmap('map2.dat', dtype='float64', mode='w+', shape=(L,1)) fp3 = np.memmap('map3.dat', dtype='float64', mode='w+', shape=(L,1)) ... if count < L: fp1[index,0] = delta if count == L: index = 0 if L <= count < 2*L: fp2[index,0] = delta if count == 2*L: index = 0 if 2*L <= count < 3*L: fp3[index,0]=delta ... count += 1 RE: Automating to save generated data - scidam - Aug-12-2020 I didn't test it, but something like the following should help you: L = 1000 data = np.random.rand(4064, 1) for j in range(data.shape[0] // L): mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L,1)) mapper = data[j*L:j*L+L] if data.shape[0] % L != 0: mapper = np.memmap(f'map{j+1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L,1)) mapper = data[j*L + L:] RE: Automating to save generated data - Robotguy - Aug-12-2020 Thanks, I tried that. Although it runs fine, I wonder why the information is not stores in the map.dat files created. That is, when I read via print(f[:10]), I get all zeros. Do you know what's happening? L = 1000 data = np.random.rand(4064, 1) print(data.shape[0]) for j in range(data.shape[0] // L): mapper = np.memmap(f'map{j}.dat', dtype='float64', mode='w+', shape=(L, 1)) mapper = data[j * L:j * L + L] print(mapper[:10]) if data.shape[0] % L != 0: mapper = np.memmap(f'map{j + 1}.dat', dtype='float64', mode='w+', shape=(data.shape[0] % L, 1)) mapper = data[j * L + L:] print(mapper.shape) # Reading the maps L = 10**3 M = 1 f = np.memmap('map1.dat', dtype='float64', mode='r', shape=(L,M)) print(f[:10]) (Aug-12-2020, 12:51 AM)scidam Wrote: I didn't test it, but something like the following should help you: RE: Automating to save generated data - Robotguy - Aug-12-2020 Using mapper[:] instead of mapper on line-7 and line-12 (in the shared code) solved the problem. Thanks again! |