Python Forum
parallel for loop with multiprocessing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
parallel for loop with multiprocessing
#1
Hi all,
I have a for loop that takes as input rows of a dataframe and returns back a new dataframe row that needs to be appended.

for i in range(0,1000):
    new_df=new_df.append(myfunction(small_pd.loc[i],listOfUePatterns) )
this type of for loop should be easily parallelized.


I tried the below
import multiprocessing

num_cores = multiprocessing.cpu_count()

results = Parallel(n_jobs=num_cores)(myfunction(small_pd.loc,listOfUePatterns)(i) for i in range(0,1000))
but it does not work. My guess is that the output of Parallel cant handle a dataframe row.

Can you please help me make it work?
Regards
Alex
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  PyRun_SimpleFile calling multiprocessing Python Class cause endless init loop Xeno 2 988 Sep-19-2022, 02:32 AM
Last Post: Xeno
  parallel loop python caro 4 1,735 Jun-16-2022, 08:46 PM
Last Post: woooee
  Pool multiprocessing - know current status in loop? korenron 0 1,604 Jul-28-2021, 08:49 AM
Last Post: korenron
  Pandas, How to trigger parallel loop Mekala 4 2,647 Oct-29-2020, 12:58 PM
Last Post: Mekala
  python multiprocessing import Pool, cpu_count: causes forever loop | help to remove Hassibayub 0 1,823 Jun-18-2020, 05:27 PM
Last Post: Hassibayub
  Parallel iteration with for loop Josh_Python890 1 2,128 Jul-19-2019, 11:50 PM
Last Post: metulburr
  Multiprocessing my Loop/Iteration (Try...Except) Jompie96 7 4,509 Jun-19-2019, 12:59 PM
Last Post: noisefloor

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020