Python Forum
subprocess.Popen for lively data reading?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
subprocess.Popen for lively data reading?
#1
is subprocess.Popen the best choice for lively data reading? what i mean by lively data reading is that i can read the data immediately after the kernel makes it available from the child process that was start in that call writes it out to the kernel, instead of waiting for the child process to exit/end and get all of the output all at one. think of reading the output of the ping command.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
I like to use this way to monitor output of a child process:

import subprocess
import os

my_cmd = ['ping', 'www.google.com']

p = subprocess.Popen(
    my_cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    bufsize=1,
    preexec_fn=os.setsid
)

with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print('OUTPUT: ' + str(line))
Reply
#3
what if you have a child process that periodically outputs to stderr amongst output to stout. can you monitor that, prefix each line with "e" or "o" depending on whether it was written to stderr or stdout, and write that somewhere just after it gets written by the child?

or, just deal with N child processes running concurrently?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
You can do this using threads:

import subprocess
import os
from threading import Thread


def std_output(pipe):
    with pipe:
        for line in iter(pipe.readline, b''):
            print('o ' + str(line))


def err_output(pipe):
    with pipe:
        for line in iter(pipe.readline, b''):
            print('e ' + str(line))


my_cmd = ['ping', 'www.google.com']

p = subprocess.Popen(
    my_cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    bufsize=1,
    preexec_fn=os.setsid
)

th_out = Thread(target=std_output, args=(p.stdout,))
th_err = Thread(target=err_output, args=(p.stderr,))
th_out.start()
th_err.start()
Reply
#5
i want to put together a way to do it without threading. i want to be able to mix analyses of the data without the complexities of threading. i'm just trying figure out early whether it is better to start all these child processes with subprocess or with os. coming from C programming, using os will be familiar (fork, etc). but if subprocess can easily do it, that should be the way to go. i just want to know in studying subprocess will take lots of time for this.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Information subprocess.Popen() suddenly giving me grief? davecotter 3 532 Dec-13-2023, 10:49 PM
Last Post: davecotter
  Reading All The RAW Data Inside a PDF NBAComputerMan 4 1,276 Nov-30-2022, 10:54 PM
Last Post: Larz60+
  Reading Data from JSON tpolim008 2 1,031 Sep-27-2022, 06:34 PM
Last Post: Larz60+
  Use subprocess.Popen and time.sleep chucky831 2 1,890 Aug-11-2022, 07:53 PM
Last Post: carecavoador
  Help reading data from serial RS485 korenron 8 13,596 Nov-14-2021, 06:49 AM
Last Post: korenron
  Help with WebSocket reading data from anoter function korenron 0 1,300 Sep-19-2021, 11:08 AM
Last Post: korenron
  Fastest Way of Writing/Reading Data JamesA 1 2,139 Jul-27-2021, 03:52 PM
Last Post: Larz60+
  Reading data to python: turn into list or dataframe hhchenfx 2 5,279 Jun-01-2021, 10:28 AM
Last Post: Larz60+
  Reading data from mysql. stsxbel 2 2,163 May-23-2021, 06:56 PM
Last Post: stsxbel
  Subprocess.Popen() not working when reading file path from csv file herwin 13 14,621 May-07-2021, 03:26 PM
Last Post: herwin

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020