Python Forum
parallel for loop with multiprocessing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
parallel for loop with multiprocessing
#1
Hi all,
I have a for loop that takes as input rows of a dataframe and returns back a new dataframe row that needs to be appended.

for i in range(0,1000):
    new_df=new_df.append(myfunction(small_pd.loc[i],listOfUePatterns) )
this type of for loop should be easily parallelized.


I tried the below
import multiprocessing

num_cores = multiprocessing.cpu_count()

results = Parallel(n_jobs=num_cores)(myfunction(small_pd.loc,listOfUePatterns)(i) for i in range(0,1000))
but it does not work. My guess is that the output of Parallel cant handle a dataframe row.

Can you please help me make it work?
Regards
Alex
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Pandas, How to trigger parallel loop Mekala 4 556 Oct-29-2020, 12:58 PM
Last Post: Mekala
  python multiprocessing import Pool, cpu_count: causes forever loop | help to remove Hassibayub 0 448 Jun-18-2020, 05:27 PM
Last Post: Hassibayub
  Parallel iteration with for loop Josh_Python890 1 897 Jul-19-2019, 11:50 PM
Last Post: metulburr
  Multiprocessing my Loop/Iteration (Try...Except) Jompie96 7 1,397 Jun-19-2019, 12:59 PM
Last Post: noisefloor

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020