parallel for loop with multiprocessing

parallel for loop with multiprocessing - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: parallel for loop with multiprocessing (/thread-19563.html)

parallel for loop with multiprocessing - dervast - Jul-04-2019

Hi all,
I have a for loop that takes as input rows of a dataframe and returns back a new dataframe row that needs to be appended.

for i in range(0,1000):
    new_df=new_df.append(myfunction(small_pd.loc[i],listOfUePatterns) )

this type of for loop should be easily parallelized.

I tried the below

import multiprocessing

num_cores = multiprocessing.cpu_count()

results = Parallel(n_jobs=num_cores)(myfunction(small_pd.loc,listOfUePatterns)(i) for i in range(0,1000))

but it does not work. My guess is that the output of Parallel cant handle a dataframe row.

Can you please help me make it work?
Regards
Alex