Python Forum
parallel for loop with multiprocessing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
parallel for loop with multiprocessing
#1
Hi all,
I have a for loop that takes as input rows of a dataframe and returns back a new dataframe row that needs to be appended.

for i in range(0,1000):
    new_df=new_df.append(myfunction(small_pd.loc[i],listOfUePatterns) )
this type of for loop should be easily parallelized.


I tried the below
import multiprocessing

num_cores = multiprocessing.cpu_count()

results = Parallel(n_jobs=num_cores)(myfunction(small_pd.loc,listOfUePatterns)(i) for i in range(0,1000))
but it does not work. My guess is that the output of Parallel cant handle a dataframe row.

Can you please help me make it work?
Regards
Alex
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Pool multiprocessing - know current status in loop? korenron 0 254 Jul-28-2021, 08:49 AM
Last Post: korenron
  Pandas, How to trigger parallel loop Mekala 4 930 Oct-29-2020, 12:58 PM
Last Post: Mekala
  python multiprocessing import Pool, cpu_count: causes forever loop | help to remove Hassibayub 0 738 Jun-18-2020, 05:27 PM
Last Post: Hassibayub
  Need help on Parallel Programming Kishore01 0 446 Mar-23-2020, 03:14 PM
Last Post: Kishore01
  Parallel iteration with for loop Josh_Python890 1 1,116 Jul-19-2019, 11:50 PM
Last Post: metulburr
  Multiprocessing my Loop/Iteration (Try...Except) Jompie96 7 2,129 Jun-19-2019, 12:59 PM
Last Post: noisefloor
  Parallel Python Manjunath 2 2,572 Oct-11-2016, 11:56 AM
Last Post: Manjunath

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020