Python Forum
waiting for many processes in parallel
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
waiting for many processes in parallel
#1
i have started about 20 processes using subprocess.Popen(command,stdout=PIPE) and have gotten EOF on all their pipes. now i want the parent script to wait for all of them to exit. for various reasons some could take several minutes to exit. i'd like to allow up to 5 minutes for the whole bunch. is there a way to wait for them all as a group with just a single timeout? normally i would just for loop through the list of objects i got from Popen and call each.join(). but putting the 300 second timeout in there could end up being a long time if several are slow. is there a better way to wait for them all while giving up after 5 minutes?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
http://www.blog.pythonlibrary.org/2016/0...ubprocess/

?
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#3
i'm not using .communicate(). each process has a pipe its stdout goes to.

there are many processes. i need to exit quickly/immediately if ALL of the processes exit in a timely manner (the usual case).

the issue is how to wait on them ALL in a manner that when they ALL exit it wakes up. the timer is good enough for the timeout event if one or more stays running.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  pip stops waiting for python walker 6 970 Nov-28-2023, 06:55 PM
Last Post: walker
  Waiting for input from serial port, then move on KenHorse 2 834 Oct-17-2023, 01:14 AM
Last Post: KenHorse
  processes shall be parallel flash77 4 1,066 Sep-20-2022, 11:46 AM
Last Post: DeaD_EyE
  Sharing imported modules with Sub Processes? Stubblemonster 2 1,463 May-02-2022, 06:42 AM
Last Post: Stubblemonster
  Waiting for heavy functions question philipbergwerf 14 3,279 Apr-29-2022, 07:31 PM
Last Post: philipbergwerf
  How to create waiting process? samuelbachorik 4 1,931 Sep-02-2021, 05:41 PM
Last Post: bowlofred
  Killing processes via python Lavina 2 2,560 Aug-04-2021, 06:20 AM
Last Post: warnerarc
  How to share a numpy array between 2 processes on Windows? qstdy 0 2,137 Jan-29-2021, 04:24 AM
Last Post: qstdy
  Waiting and listening test 2 2,100 Nov-13-2020, 04:43 PM
Last Post: michael1789
  sharing variables between two processes Kiyoshi767 1 1,848 Nov-07-2020, 04:00 AM
Last Post: ndc85430

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020