Python Forum
How to collate Multiprocessing-Process results?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to collate Multiprocessing-Process results?
Let us consider the following code where I calculate the factorial of 4 really large numbers, saving each output to a separate .txt file (out_mp_{idx}.txt). I use multiprocessing (4 processes) to reduce the computation time. Though this works fine, I want to output all the four results in one file. Of course I can open each of the four files I create (from the code below) and append to a new file, but that's not my choice (below is just a simplistic version of my code, I have too many files to handle, which defeats the purpose of time-saving via multiprocessing). Is there a better way to automate such that the results from the processes are all dumped/appended to some file?

I also tried process.immap route, but that's not as computationally efficient as the below code.

from multiprocessing import Process
import os
import time

tic = time.time()

def factorial(n, idx):  # function to calculate the factorial

    num = 1
    while n >= 1:
        num *= n
        n = n - 1

    with open(f'out_mp_{idx}.txt', 'w') as f0:  # saving output to a separate file

def My_prog():

    jobs = []
    N = [10000, 20000, 40000, 50000]  # numbers for which factorial is desired
    n_procs = 4
    # executing multiple processes
    for i in range(n_procs):  
        p = Process(target=factorial, args=(N[i], i))

    for j in jobs:

    for j in jobs:

    print(f'Exec. Time:{time.time()-tic} [s]')

if __name__=='__main__':
The write-up on multiprocessing is quite good here

It's worth a read if you haven't done so already.

Forum Jump:

User Panel Messages

Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020