Python Forum
running multiple commands in a parallel pool
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
running multiple commands in a parallel pool
#1
i would like to run multiple commands in a parallel pool. i do not need to wait for the commands to be started or to end. i need to be able to set the pool size at least in advance to limit the concurrency. i need to be able to submit commands after they start running. when a command finishes, it needs to start the next one with no action needed by my code. after submitting the last command i need to wait for everything to be done. commands are in the form of a list of strings like in subprocess.Popen(). surely, something like this has been created, already.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
What about concurrent.futures
Reply
#3
it looks like that module is intended to call functions. i need to execute a command. i probably can do that by having each process running a function do subprocess.call(). but, that means 2 processes per command. if i use the threaded version, can a variety of threads each do subprocess.call() to run 1 process per command, N of them concurrently?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
What is a command ?
Reply
#5
what i ultimately need to have is commands launched in processes when the main parent code calls the method to submit a command (list/tuple of strings). if the number of running processes is less than the maximum, then start a new process. otherwise append the command to the FIFO Queue. one of the effects of this is that fewer commands are submitted than the max_processes, then fewer processes will exist. the processes will not exist until they are needed. when a worker process exits, it is only because a command exited. there must be no need for special output or signals from the child because there would be no code to handle it in the command.

a command is a list of strings like subprocess.Popen() takes.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#6
(Jul-30-2019, 12:15 AM)Skaperen Wrote: a command is a list of strings like subprocess.Popen() takes.

A string is a string, subprocess.Popen is a callable, which can be submitted to a concurrent future.
Reply
#7
subprocess.Popen() is one way to execute a command. another is subprocess.call(). in both cases you pass in the only argument, the list of strings, such as:
    subprocess.call(['sudo','mount','-r','/dev/cdrom0','/mnt/cdrom'])
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Suggestions for coding of multiple commands astroashton 3 995 Nov-30-2022, 11:18 PM
Last Post: astroashton
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 737 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  super multiple parallel classes catlessness 2 1,294 Jun-07-2022, 02:35 PM
Last Post: deanhystad
  execute commands inside a running application mr_byte31 3 1,294 Apr-12-2022, 08:50 AM
Last Post: Larz60+
  Controlling what get outputted to stdout when running external commands Daring_T 4 2,111 Jan-30-2022, 05:40 PM
Last Post: bowlofred
  Pool multiprocessing - know current status in loop? korenron 0 1,604 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 2,101 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  Running script on multiple files Afrodizzyjack 1 2,468 May-14-2021, 10:49 PM
Last Post: Yoriz
  Multiproccessing ValueError: Pool not running when running parallel functions Gilush 1 3,268 Jun-06-2020, 12:49 AM
Last Post: Gilush
  Use dynamic variable from parallel running python script Sicksym 0 1,823 May-15-2020, 02:52 PM
Last Post: Sicksym

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020