pipeline between 2 programs - Printable Version

pipeline between 2 programs - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: pipeline between 2 programs (/thread-10607.html)

Pages: 1 2

pipeline between 2 programs - Skaperen - May-28-2018

in a function, you need to run 2 programs, their names given in the two arguments of the function call, with a pipeline between them, the output of the program/command given in argument 1 going to the input of the program/command given in argument 2. argument 3 is a list of argument strings for the program/command given in argument 1. argument 4 is a list of argument strings for the program/command given in argument 2. the output to the pipeline could be many millions of lines of data, so reading this into the python program is not practical.

what methods will you use to achieve this?

RE: pipeline between 2 programs - nilamo - Jun-25-2018

First, I'd open a pipe [os.pipe()], then use subprocess.Popen's stdin/stdout parameters as the read/write handles of that pipe. That way, (hopefully) the first program pipes it's output into the second program without ever touching python at all.

RE: pipeline between 2 programs - Skaperen - Jun-26-2018

sounds like how i did it in an unnamed language spelled with one letter. so, has anyone figured out how to do this with python calls?

RE: pipeline between 2 programs - nilamo - Jun-26-2018

I don't actually know of any programs on windows that use stdin, so I can't really test this that well, but it at least runs...

import os
import subprocess
import sys

chain = sys.argv[1:3]
args = sys.argv[3:]

pipe_read, pipe_write = os.pipe()

first = subprocess.Popen([chain[0], args[0]], stdout=pipe_write)
second = subprocess.Popen([chain[1], args[1]], stdin=pipe_read)

first.wait()
second.wait()

RE: pipeline between 2 programs - DeaD_EyE - Jun-26-2018

Just a bit simpler:

def pipe(command1, command2):
    pout = subprocess.Popen(command1, stdout=subprocess.PIPE)
    pin = subprocess.Popen(command2, stdin=pout.stdout)
    pout.wait()
    pin.wait()

If you want to pipe many processes:

def many_pipe(*commands):
    first, *rest = [shlex.split(cmd) for cmd in commands]
    procs = [subprocess.Popen(first, stdout=subprocess.PIPE)]
    for cmd in rest:
        proc = subprocess.Popen(cmd, stdin=procs[-1].stdout, stdout=subprocess.PIPE)
        procs.append(proc)
    [p.wait() for p in procs]
    return procs[-1].stdout.read().decode(sys.getdefaultencoding())

I use shlex.split to split the command. If you depend on user input, you should not use shlex.split.

If you relay on Python2, you don't have the nice argument unpacking.
You can 'clone' this functionallity:

commands = ['ls -l', 'grep foo', 'sort']
first, rest = commands.pop(0), commands

RE: pipeline between 2 programs - nilamo - Jun-26-2018

I didn't know you could use the pipe parameters like that. I bet it works better than using a pipe like I was, in addition to looking better.

RE: pipeline between 2 programs - Skaperen - Jun-26-2018

well, at least it works. and is not delayed by trying to buffer everything in python. and i like simpler. and it seems obvious how to chain many commands into a long pipeline without shlex.split. i'd just use a list of commands or for a quickie function, commands as a variable list of arguments.

edit:

i do worry that there may be some holdups at the chain of .wait() method calls but that should be a minimal issue in most cases.

RE: pipeline between 2 programs - nilamo - Jun-26-2018

(Jun-26-2018, 08:16 PM)Skaperen Wrote: i do worry that there may be some holdups at the chain of .wait() method calls but that should be a minimal issue in most cases.

What happens to a running sub-command if the parent Python process finishes? Does it continue running to completion, or does the kernel kill it since it's an orphaned process?

RE: pipeline between 2 programs - Skaperen - Jun-26-2018

PID 1 takes over as the parent and it continues to run.

RE: pipeline between 2 programs - Skaperen - Jun-30-2018

now i need to have one python script read from the outputs of N other processes in parallel and fully intermixed so i know which of those processes it came from.