+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: waiting for many processes in parallel (/thread-20779.html)
waiting for many processes in parallel - Skaperen - Aug-30-2019
i have started about 20 processes using
subprocess.Popen(command,stdout=PIPE) and have gotten EOF on all their pipes. now i want the parent script to wait for all of them to exit. for various reasons some could take several minutes to exit. i'd like to allow up to 5 minutes for the whole bunch. is there a way to wait for them all as a group with just a single timeout? normally i would just for loop through the list of objects i got from Popen and call each.join(). but putting the 300 second timeout in there could end up being a long time if several are slow. is there a better way to wait for them all while giving up after 5 minutes?
RE: waiting for many processes in parallel - wavic - Aug-31-2019
http://www.blog.pythonlibrary.org/2016/05/17/python-101-how-to-timeout-a-subprocess/
?
RE: waiting for many processes in parallel - Skaperen - Sep-02-2019
i'm not using .communicate(). each process has a pipe its stdout goes to.
there are many processes. i need to exit quickly/immediately if ALL of the processes exit in a timely manner (the usual case).
the issue is how to wait on them ALL in a manner that when they ALL exit it wakes up. the timer is good enough for the timeout event if one or more stays running.