Python Forum
builing a long variable-length command-pipeline
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
builing a long variable-length command-pipeline
#1
there are many tools that allow executing system commands and some can provide input and/or capture output.  but suppose you have a series of commands with the number of commands known only when the commands become known that need to be run as a pipeline (output of a command goes to input of the next).  and you may need to provide input to the first and/or capture output from the last.  how do you do this?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
sys.stdout.write() # pipe out
sys.stdin.read()  # pipe in
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#3
(Dec-12-2017, 05:24 AM)wavic Wrote: sys.stdout.write() # pipe out
sys.stdin.read()  # pipe in

that builds a pipeline?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
You read from standard out and write to standard in, right?

#!/usr/bin/env python3

import sys

if len(sys.argv) > 1:
    pipe_in = sys.stdin.read().strip()
    sys.stdout.write('{} {} \n'.format(pipe_in, ' '.join(sys.argv[1:])))
else:
    pipe_in = sys.stdin.read().strip()
    sys.stdout.write('{} \n'.format(pipe_in))
You have to be careful with the quotes. If you put double quotes for the strings you get EOL error because of the bash quoting. I've needed some time to get it during this example. I have to test it before to publish Confused  

victor@jerry:/tmp$ echo hello | ./piped.py world
hello world

victor@jerry:/tmp$ echo hello | ./piped.py
hello

victor@jerry:/tmp$ echo hello | ./piped.py | ./piped.py
hello

victor@jerry:/tmp$ echo hello | ./piped.py beautiful | ./piped.py world
hello beautiful world

victor@jerry:/tmp$ echo hello | ./piped.py beautiful | ./piped.py beautiful world
hello beautiful beautiful world
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#5
shells like bash can take a particular syntax and understand a series of commands to be put into a pipeline, when such a command string with that syntax is given to it.  it makes a number of system calls to construct that pipeline, including the 2 you showed, many times.  what i am looking for, is code in Python that carries out a similar setup, execution, and completion.  i am not looking for a few example syscalls.  i am looking for how Python coders would carry out those steps.  part of the reason is to see what Python library functions might get used, how they would be used, and how they fit into the steps.  i am less interested in how a command pipeline string would parsed.  you could just assumed the commands to be pipelined are in a list ... a list of commands ... a list of lists of strings (a list of lists of parsed command tokens).  this could be defined as a function run_pipeline(list_of_lists_of tokens):.   fyi, like so many things, i've done this, before, in C.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#6
subprocess has PIPE build in.
shell=False is default,and do not use shell=True so can pass in the whole shell pipe command.
Because of security reasons.

So can look at a command like this grep -o "hello" foo.txt | wc -l
It will search for hello in foo.txt,split at |.
stdout=subprocess.PIPE which tells subprocess to send that output to the respective file handler.
import subprocess

# grep -o "hello" foo.txt | wc -l
grep_proc = subprocess.Popen(['grep', '-o', 'hello', 'foo.txt'],stdout=subprocess.PIPE)
wc_proc = subprocess.Popen(['wc', '-l'],stdin=grep_proc.stdout, stdout=subprocess.PIPE)
result, arg_1 = wc_proc.communicate()
print(result.decode())
Reply
#7
can you show how to connect TWO commands with that built-in PIPE in subprocess?  can you then expand that to N commands from a list of commands?

btw, the .communicate() method is bad in some cases because it waits for the command to complete before you get any data.  using this in a pipeline would preclude parallel execution of the commands.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to write a part of powershell command as a variable? ilknurg 2 1,084 Jul-26-2022, 11:31 AM
Last Post: ilknurg
  Os command output in variable shows wrong value paulo79 2 1,467 Apr-09-2022, 03:48 PM
Last Post: ndc85430
  How to use a variable in linux command in python code? ilknurg 2 1,547 Mar-14-2022, 07:21 AM
Last Post: ndc85430
  Command output to Variable ironclaw 1 1,743 Aug-26-2021, 06:55 PM
Last Post: bowlofred
  Correct syntax for a variable inside a quotes: MP4Box command JonnyDriller 2 2,710 Feb-02-2020, 01:22 AM
Last Post: JonnyDriller
  Long command with characters not working in Python on Oracle Linux 7 iaas_infra 10 6,131 Jul-19-2019, 04:53 PM
Last Post: ichabod801
  Insert a variable in a Python > Cellular AT command ElectronicsNut 1 2,043 Jul-07-2019, 02:26 PM
Last Post: joe_momma
  pipeline between 2 programs Skaperen 11 5,760 Jul-02-2018, 10:48 PM
Last Post: Skaperen
  search and store a value from linux command output in to variable python prazy29 1 3,226 Apr-06-2018, 01:56 PM
Last Post: wavic
  [subprocess]>Run a cmd command and get output into a variable CSA75 4 22,710 Mar-13-2017, 09:33 PM
Last Post: Ofnuts

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020