running a pipeline of commands

[Skaperen]

i want to run a pipeline of commands in my python function. what is the best way to have those commands passed to the function? what is the best way for the caller to describe how the output should be handled or returned? should the function wait for all the commands to finish or return to the caller first?

[Gribouillis]

The question is a little abstract, could you provide some examples of what you want to do?

[Skaperen]

you know how subprocess.call() can run a single command. my function will run a command pipeline like a shell does with |. it will be given a list of commands, so a list of lists of strings and the like. the way to give it a pipeline of commands is simple. but different ways to deal with the output is up in the air. i'm trying to figure what might be wanted. for example providing a buffer of bytes, or a pipe to read, or a list pf lines, or a generator.

[Axel_Erfurt]

subprocess multiple commands

import subprocess

def subprocess_multi(cmd):
    
    process = subprocess.Popen(cmd,stdout=subprocess.PIPE, shell=True)
    my_stdout = process.communicate()[0].strip().decode("utf-8")
    print(my_stdout)
    
subprocess_multi("ls /home; echo 'Hello World'; ls /")

[Skaperen]

i'm not looking for how to do it. i already have it working. i'm looking at features to add such as ways to return the output to the caller. read my original post, again.

[DeaD_EyE]

i want to run a pipeline of commands in my python function.

Commands in sense of Python-Function or Commands in sense of running a process outside of Python with subprocess?

what is the best way to have those commands passed to the function? what is the best way for the caller to describe how the output should be handled or returned? should the function wait for all the commands to finish or return to the caller first?

If its process related (running commands somewhere), then you can use the POSIX standard.
A process has stdin, stderr and stdout which can be connected with a pipe in your bash
In addition, the process uses signalling to shut down or reload configuration.

My question is, do you want to replicate subprocess in a way to easily pipe commands
or do you want to Pipe Python-Functions like in your shell?

I would try it with the Pipe operator. You can make a class and define what happens if you use the | operator.
Then you have to specify what happens, if using the | operator and which data is exchanged between the two Functions/Processes.

[Skaperen]

Commands in sense of Python-Function or Commands in sense of running a process outside of Python with subprocess?

i didn't know there was a "Python-Function" sense. i don't think of functions as commands. does this redirect print() to input()?

what i want to do is an expansion of subprocess.call so that shell command pipelines where stdout of a process is piped to stdin of the next process.

i have a working function that does this. each command is a list of strings (bytes,bytearray,str). a pipeline is a list of commands. tuple can be used in place of list. in the latest version, what i am trying to do is allow the caller to specify alternate ways to handle the output of the pipeline, stdout of the last process. the default is to return a readable file that is the read end of a unidirectional pipe, just like POSIX shells do. i want to do this with mostly an out= keyword parameter in the function call. i have named this function "runpipeline".

[Skaperen]

here is my latest thoughts for the out= keyword argument parameter for the runpipeline() function. i do not have the code for this complete. yet.

command pipeline function runpipeline(cmd,out=)

def runpipeline(pipeline,out=None):

the default handling of output is to return a readable file to the caller and not wait
for the command pipeline to finish.

the out= keyword argument selects an alternate way to return the output.
using out= enables waiting for the command pipeline to complete so the
entire output is captured and returned or stored as specified.

out=bytes           one buffer with newlines as bytes
out=bytearray       one buffer with newlines as bytearray
out=str             one buffer with newlines as str (decoded from bytes)

out=list,bytes      list of lines as bytes
out=list,bytearray  list of lines as bytearray
out=list,str        list of lines as str (decoded from bytes)
		   
out=tuple,bytes     tuple of lines as bytes
out=tuple,bytearray tuple of lines as bytearray
out=tuple,str       tuple of lines as str (decoded from bytes)

out=<int>       outputs to the given file descriptor opened for write or append.

out=<filename>  output is written to the named file (given in str, bytes, or bytearray).
                if name of file ends with ".gz", ".bz2", or ".xz" then a compress command
                will be added to the pipeline (caller should not include compression)
                a mode= option can be used in this case.  it is ignored in other cases.
                it allows use of "a", or "w", or "x".

out=<openfile>  output is written to the file already opened for write or append.
                writing takes place as the command pipeline runs even though return
                waits for the command pipeline to finish.  file must be binary mode.

out=2           outputs to the current stderr.
out=1           outputs to the current stdout.
out=True        outputs to the current stdout.
out=False       discards output (equivalent to outputting to "/dev/null" on POSIX
out=None        behaves as if out= was not provided, no wait, returns readable pipe

a command is a list or tuple of arguments (str, bytes, or bytearray, or a mixture).
a pipeline is a list or tuple of commands (list or tuple of str, bytes, or bytearray).
if only one level of list or tuple is given then it is made into a pipeline of one
command that has arguments.  if zero levels of list or tuple are given then it is
made into a pipeline of one command with no arguments.

[Gribouillis]

Controlling complexity is the essence of computer programming.

[Skaperen]

there is complexity in what this does in the end. what it presents to the caller is intended to be the controlled complexity. so, what the caller can do is something like:

result=runpipeline(pipeline=(cmd1,cmd2),out=(tuple,str))

once i have the API design out of beta, i can then look at how i can restructure the implementation into less complex layers. the user of runpipeline() should see no more than this interface, not need to see more (if it is designed well), and not see changes as i redesign the implementation.

i expect to later look at an in= keyword argument for what is to be transferred to stdin of the first process. my first implementation just used "/dev/null". that actually broke some programs. what is needed is a device that waits when read and provides no data. a lame pipe can do this.

now i am looking at some ambiguity in read(), read1(), readinto(), and readinto1(). it is unclear how to differentiate between EOF and a slow process. which gives a zero or empty result.