Python Forum

Full Version: parallel for loop with multiprocessing
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi all,
I have a for loop that takes as input rows of a dataframe and returns back a new dataframe row that needs to be appended.

for i in range(0,1000):
    new_df=new_df.append(myfunction(small_pd.loc[i],listOfUePatterns) )
this type of for loop should be easily parallelized.


I tried the below
import multiprocessing

num_cores = multiprocessing.cpu_count()

results = Parallel(n_jobs=num_cores)(myfunction(small_pd.loc,listOfUePatterns)(i) for i in range(0,1000))
but it does not work. My guess is that the output of Parallel cant handle a dataframe row.

Can you please help me make it work?
Regards
Alex