running multiple commands in a parallel pool

[Skaperen]

i would like to run multiple commands in a parallel pool. i do not need to wait for the commands to be started or to end. i need to be able to set the pool size at least in advance to limit the concurrency. i need to be able to submit commands after they start running. when a command finishes, it needs to start the next one with no action needed by my code. after submitting the last command i need to wait for everything to be done. commands are in the form of a list of strings like in subprocess.Popen(). surely, something like this has been created, already.

[Yoriz]

What about concurrent.futures

[Skaperen]

it looks like that module is intended to call functions. i need to execute a command. i probably can do that by having each process running a function do subprocess.call(). but, that means 2 processes per command. if i use the threaded version, can a variety of threads each do subprocess.call() to run 1 process per command, N of them concurrently?

[Yoriz]

What is a command ?

[Skaperen]

what i ultimately need to have is commands launched in processes when the main parent code calls the method to submit a command (list/tuple of strings). if the number of running processes is less than the maximum, then start a new process. otherwise append the command to the FIFO Queue. one of the effects of this is that fewer commands are submitted than the max_processes, then fewer processes will exist. the processes will not exist until they are needed. when a worker process exits, it is only because a command exited. there must be no need for special output or signals from the child because there would be no code to handle it in the command.

---

a command is a list of strings like subprocess.Popen() takes.

[Yoriz]

a command is a list of strings like subprocess.Popen() takes.

A string is a string, subprocess.Popen is a callable, which can be submitted to a concurrent future.

[Skaperen]

subprocess.Popen() is one way to execute a command. another is subprocess.call(). in both cases you pass in the only argument, the list of strings, such as:

    subprocess.call(['sudo','mount','-r','/dev/cdrom0','/mnt/cdrom'])