threads and subprocess.call

[Skaperen]

can i have 2 threads each call subprocess.call() and have 2 commands running in parallel without being blocked by the big global lock?

[Gribouillis]

Why would you be blocked? It simply means that each of the threads is waiting for an external process to complete. Threads that wait are a very common thing.

[Skaperen]

i don't understand enough of the thread limitations in Python to know if there would be some kind of blockage or not. i do have a script that could benefit from two "step synchronized" threads.

"step synchronized" is like this. think of 2 threads each doing 3 tasks in a specific order, 1, 2, and 3. the 2 threads have slight variations to do (such carrying out the 3 steps on 2 different files). but there is enough interrelation (such as exchanging file between the threads) that the threads need to synchronize their steps so that each does not proceed to step 2 until both have completed step 1. likewise, each must wait for both to finish step 2 before starting step 3.

the real case i have today is genuinely just 2 threads, but 12 critical steps (they absolutely cannot be done out of order or data or security is likely to be lost). two of the steps are each thread to run the "cp" command to copy some data. the sing thread test is very fast for small data. but i can imagine it being quite slow for very large files.

i encounter this need often but have never seen methods to do this kind of thing in a straight forward way. it can be done by setting up a semaphore for each step and waiting on that. one possible way would be a discrete function for each step.

[snippsat]

Using call or run it will not run subprocesses in parallel.
Popen release so it run in parallel.
Here a test.
Running this it wait 20-sec then do ping one bye one.

from subprocess import Popen, call, run

# ping -c 4 google.com ## Linux
commands = [
    'sleep 20',
    'ping -n 4 google.com',
    'ping -n 4 cnn.com'
]
# Try run in parallel
processes = [call(cmd, shell=True) for cmd in commands]

'''
for p in processes:
    p.wait()'''

Now you see that ping and sleep start in parallel all at once.

from subprocess import Popen, call, run

# ping -c 4 google.com ## Linux
commands = [
    'sleep 20',
    'ping -n 4 google.com',
    'ping -n 4 cnn.com'
]
# Try run in parallel
processes = [Popen(cmd, shell=True) for cmd in commands]

'''
for p in processes:
    p.wait()'''

[Skaperen]

that's certainly "a way" to run the commands in parallel. if my objective were only that, it is like i would do it very similar to this (though not likely in a comprehension). i need to do it in threads so my question is as posted. if i need to avoid threads, it could be much harder to do. i might have to use multiprocessing. i have not put much thought into this, yet, and won't until i know which directions to focus on.