subprocess.Popen for lively data reading?

[Skaperen]

is subprocess.Popen the best choice for lively data reading? what i mean by lively data reading is that i can read the data immediately after the kernel makes it available from the child process that was start in that call writes it out to the kernel, instead of waiting for the child process to exit/end and get all of the output all at one. think of reading the output of the ping command.

[gontajones]

I like to use this way to monitor output of a child process:

import subprocess
import os

my_cmd = ['ping', 'www.google.com']

p = subprocess.Popen(
    my_cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    bufsize=1,
    preexec_fn=os.setsid
)

with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print('OUTPUT: ' + str(line))

[Skaperen]

what if you have a child process that periodically outputs to stderr amongst output to stout. can you monitor that, prefix each line with "e" or "o" depending on whether it was written to stderr or stdout, and write that somewhere just after it gets written by the child?

or, just deal with N child processes running concurrently?

[gontajones]

You can do this using threads:

import subprocess
import os
from threading import Thread


def std_output(pipe):
    with pipe:
        for line in iter(pipe.readline, b''):
            print('o ' + str(line))


def err_output(pipe):
    with pipe:
        for line in iter(pipe.readline, b''):
            print('e ' + str(line))


my_cmd = ['ping', 'www.google.com']

p = subprocess.Popen(
    my_cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    bufsize=1,
    preexec_fn=os.setsid
)

th_out = Thread(target=std_output, args=(p.stdout,))
th_err = Thread(target=err_output, args=(p.stderr,))
th_out.start()
th_err.start()

[Skaperen]

i want to put together a way to do it without threading. i want to be able to mix analyses of the data without the complexities of threading. i'm just trying figure out early whether it is better to start all these child processes with subprocess or with os. coming from C programming, using os will be familiar (fork, etc). but if subprocess can easily do it, that should be the way to go. i just want to know in studying subprocess will take lots of time for this.