pipeline between 2 programs

[Skaperen]

in a function, you need to run 2 programs, their names given in the two arguments of the function call, with a pipeline between them, the output of the program/command given in argument 1 going to the input of the program/command given in argument 2. argument 3 is a list of argument strings for the program/command given in argument 1. argument 4 is a list of argument strings for the program/command given in argument 2. the output to the pipeline could be many millions of lines of data, so reading this into the python program is not practical.

what methods will you use to achieve this?

[nilamo]

First, I'd open a pipe [os.pipe()], then use subprocess.Popen's stdin/stdout parameters as the read/write handles of that pipe. That way, (hopefully) the first program pipes it's output into the second program without ever touching python at all.

[Skaperen]

sounds like how i did it in an unnamed language spelled with one letter. so, has anyone figured out how to do this with python calls?

[nilamo]

I don't actually know of any programs on windows that use stdin, so I can't really test this that well, but it at least runs...

import os
import subprocess
import sys

chain = sys.argv[1:3]
args = sys.argv[3:]

pipe_read, pipe_write = os.pipe()

first = subprocess.Popen([chain[0], args[0]], stdout=pipe_write)
second = subprocess.Popen([chain[1], args[1]], stdin=pipe_read)

first.wait()
second.wait()

[DeaD_EyE]

Just a bit simpler:

def pipe(command1, command2):
    pout = subprocess.Popen(command1, stdout=subprocess.PIPE)
    pin = subprocess.Popen(command2, stdin=pout.stdout)
    pout.wait()
    pin.wait()

If you want to pipe many processes:

def many_pipe(*commands):
    first, *rest = [shlex.split(cmd) for cmd in commands]
    procs = [subprocess.Popen(first, stdout=subprocess.PIPE)]
    for cmd in rest:
        proc = subprocess.Popen(cmd, stdin=procs[-1].stdout, stdout=subprocess.PIPE)
        procs.append(proc)
    [p.wait() for p in procs]
    return procs[-1].stdout.read().decode(sys.getdefaultencoding())

I use shlex.split to split the command. If you depend on user input, you should not use shlex.split.

If you relay on Python2, you don't have the nice argument unpacking.
You can 'clone' this functionallity:

commands = ['ls -l', 'grep foo', 'sort']
first, rest = commands.pop(0), commands

[nilamo]

I didn't know you could use the pipe parameters like that. I bet it works better than using a pipe like I was, in addition to looking better.

[Skaperen]

well, at least it works. and is not delayed by trying to buffer everything in python. and i like simpler. and it seems obvious how to chain many commands into a long pipeline without shlex.split. i'd just use a list of commands or for a quickie function, commands as a variable list of arguments.

edit:

i do worry that there may be some holdups at the chain of .wait() method calls but that should be a minimal issue in most cases.

[nilamo]

i do worry that there may be some holdups at the chain of .wait() method calls but that should be a minimal issue in most cases.

What happens to a running sub-command if the parent Python process finishes? Does it continue running to completion, or does the kernel kill it since it's an orphaned process?

[Skaperen]

PID 1 takes over as the parent and it continues to run.

[Skaperen]

now i need to have one python script read from the outputs of N other processes in parallel and fully intermixed so i know which of those processes it came from.