i would like to run multiple commands in a parallel pool. i do not need to wait for the commands to be started or to end. i need to be able to set the pool size at least in advance to limit the concurrency. i need to be able to submit commands after they start running. when a command finishes, it needs to start the next one with no action needed by my code. after submitting the last command i need to wait for everything to be done. commands are in the form of a list of strings like in subprocess.Popen(). surely, something like this has been created, already.
it looks like that module is intended to call functions. i need to execute a command. i probably can do that by having each process running a function do subprocess.call(). but, that means 2 processes per command. if i use the threaded version, can a variety of threads each do subprocess.call() to run 1 process per command, N of them concurrently?
what i ultimately need to have is commands launched in processes when the main parent code calls the method to submit a command (list/tuple of strings). if the number of running processes is less than the maximum, then start a new process. otherwise append the command to the FIFO Queue. one of the effects of this is that fewer commands are submitted than the max_processes, then fewer processes will exist. the processes will not exist until they are needed. when a worker process exits, it is only because a command exited. there must be no need for special output or signals from the child because there would be no code to handle it in the command.
a command is a list of strings like subprocess.Popen() takes.
(Jul-30-2019, 12:15 AM)Skaperen Wrote: [ -> ]a command is a list of strings like subprocess.Popen() takes.
A string is a string,
subprocess.Popen is a callable, which can be submitted to a concurrent future.
subprocess.Popen() is one way to execute a command. another is subprocess.call(). in both cases you pass in the only argument, the list of strings, such as:
subprocess.call(['sudo','mount','-r','/dev/cdrom0','/mnt/cdrom'])