Jun-14-2018, 01:50 PM
(This post was last modified: Jun-14-2018, 02:09 PM by RomanRettich. Edited 2 times in total.)

Hey guys,

I am trying to get into 'multiprocessing' since a few days. I have a few hundred files which contain a few thousand numbers which I want to save in a matrix. I found a code in the internet that seems applicable for my problem, but I did not get into it by now (I am relatively new to python). Here is my changed version of the code (explanation further down)

I would be very happy about an answer!

Thanks in advance and best regards,

Max

sorry, here's the whole traceback

Please, post the entire traceback that you get. We need to see the whole thing. Do not just give us the last line.

Take a time to read What to include in a post

Also, PEP8 recommends using 4 spaces per indentation level - fix the indentation in writedata function

I am trying to get into 'multiprocessing' since a few days. I have a few hundred files which contain a few thousand numbers which I want to save in a matrix. I found a code in the internet that seems applicable for my problem, but I did not get into it by now (I am relatively new to python). Here is my changed version of the code (explanation further down)

import numpy as np import multiprocessing import math def writedata(rs,out_q): data_sim=np.zeros((4281,80)) for kkk in rs: rs_str=str(kkk) fnrs = '../OUTPUT_FILES' + '/Datei' + rs_str data_sim_rec=np.genfromtxt(fnrs) data_sim_rec=np.array([data_sim_rec[:,1]]).T data_sim[:,kkk]=data_sim_rec[:,0] out_q.put(data_sim) rs=np.arange(1,80,1) nprocs=10 out_q=multiprocessing.Queue() chunksize=int(math.ceil(len(rs)/float(nprocs))) procs=[] for i in range(nprocs): p=multiprocessing.Process(target=writedata,args=(rs[chunksize*i:chunksize*(i+1)],out_q)) procs.append(p) p.start() resultdict={} for i in range(nprocs): resultdict.update((out_q.get())) for p in procs: p.join() data_sim=resultdictI define a function called writedata which goes into a folder with exemplary 80 files, reads in the data in data_sim_rec and saves it in a Matrix data_sim in the appropriate row. This matrix shall be saved into a queue. I distribute the jobs to the processors by a list with numbers for the loop. Then, the data shall be saved in resultdict. The error message is:

dictionary update sequence element #0 has length 80, 2 is required.I know that I probably mix data types and that the 'update'-line is wrong as well, but I already tried many things which did not work.

I would be very happy about an answer!

Thanks in advance and best regards,

Max

sorry, here's the whole traceback

Traceback (most recent call last): File "FORUM.py", line 31, in <module> resultdict.update((out_q.get())) ValueError: dictionary update sequence element #0 has length 80; 2 is required

**buran**wrote Jun-14-2018, 01:57 PM:Please, post the entire traceback that you get. We need to see the whole thing. Do not just give us the last line.

Take a time to read What to include in a post

Also, PEP8 recommends using 4 spaces per indentation level - fix the indentation in writedata function