Python Forum
Multiprocessing Module Running An Infinite Child Process Even After Completion
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Multiprocessing Module Running An Infinite Child Process Even After Completion
#1
I have a sample code which uses joblibs and multiprocessing modules. The code works fine when run from the command line, but when I package it as an executable using Pyinstaller, the multiple processes spawn as new instances infinitely (creating new child process id in backend).

After scoping lot of sources I was suggested to use multiprocessing.freeze_support() at the beginning of my if __name__ == "__main__": block.

This resolved the infinite invocation of my main script, but after completion of the code, there was a left out child process still running in background invoked with a new process id always with below process name :

Output:
user 2741 1 28 14:25 pts/5 00:00:00 /home/user/ParallelProcessing -E -s -B -S -c from multiprocessing.semaphore_tracker import main;main(5)
After few seconds (Process ID changes)

Output:
user 2745 1 28 14:25 pts/5 00:00:00 /home/user/ParallelProcessing -E -s -B -S -c from multiprocessing.semaphore_tracker import main;main(5)
I would just like to know what additional part am I missing in my code to avoid such issues.

Code :
from joblib import Parallel, delayed                            
import multiprocessing                            
import time                            
inputs = range(10)                             
def processInput(i):                            
    print("Started : ",i)                            
    time.sleep(i)                            
    print("Completed : ",i)                            
    return i * i


def main():                            
    num_cores = multiprocessing.cpu_count()                            
    backend = 'threading'                            
    print(num_cores)                            
    results = Parallel(n_jobs=num_cores, backend=backend)(delayed(processInput)(i) for i in inputs)                            
    print(results)


if __name__ == "__main__":                            
    multiprocessing.freeze_support()                            
    main()
Have even tried handling via :
backend=threading argument during Parallel() call.
But to no use.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Using one child class method in another child class garynewport 5 1,490 Jan-11-2023, 06:07 PM
Last Post: garynewport
  Failed to execute child process (No such file or directory) uriel 1 1,616 Sep-15-2022, 03:48 PM
Last Post: Gribouillis
  Python multiprocessing Pool apply async wait for process to complete sunny9495 6 6,223 Apr-02-2022, 06:31 AM
Last Post: sunny9495
  Get stdout of a running process yok0 0 2,976 Aug-20-2020, 10:12 AM
Last Post: yok0
  Can a module be executed even if the computer running it does not install it? CFYB 5 3,344 Feb-08-2020, 01:56 PM
Last Post: snippsat
  Running function from parent module which has a loop in it. ta2909i 1 2,654 Nov-18-2019, 07:04 PM
Last Post: Gribouillis
  Python ModuleNotFoundError: No module named 'sender_model' when running an Alembic mi fcardinaliuk 1 3,887 Nov-13-2019, 07:57 PM
Last Post: micseydel
  How I can use multiprocessing with upickled module variable? AlekseyPython 3 4,012 Oct-31-2019, 06:09 AM
Last Post: AlekseyPython
  How to sharing object between multiple process from main process using Pipe Subrata 1 3,620 Sep-03-2019, 09:49 PM
Last Post: woooee
  XML Parsing Child karthi_python 1 1,855 May-16-2019, 01:37 PM
Last Post: karthi_python

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020