Python Forum
how to keep a Popen instance existant in a function return?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
how to keep a Popen instance existant in a function return?
#1
i have a function that is intended to start a subprocess.Popen instance and return the file object reference of the read end of the stdout pipe of that process. normally i can read this file OK. but this fails when created in a function. i have determined that the cause is the Popen instance object is dereferenced when the function returns resulting in the file object being return is now closed. the caller tries to read it and gets the exception "ValueError: I/O operation on closed file."

how can i prevent the Popen instance object from being dereferenced?
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
Can you post some code that shows this?
Reply
#3
as you can see, this code takes a list of commands and runs them in a stdout to stdin pipeline. it returns the read side of the pipe from the last process in the pipeline.
from os import write
from subprocess import PIPE,Popen
from sys import argv,stderr
from time import sleep

def cmdsout(cmds,**opts):
    """Run a command (list/tuple of list/tuple of str/bytes/bytearray)
returning its last output as a big str/bytes/bytearray buffer of newline separated lines."""

# for POSIX/BSD/Unix/Linux systems
# tested on Ubuntu Linux
# each command can be either a list or tuple
# each command arg must be the same type within a command, one of str or bytes or bytearray
# but can be different between commands
# the command pipeline can be either a list or tuple

    if not isinstance(cmds,(list,tuple)):
        raise TypeError('argument 1 (cmds) must be a list or tuple (of list/tuple of str/bytes/bytearray).')
    if not all(isinstance(cmd,(list,tuple))for cmd in cmds):
        raise TypeError('argument 1 (cmds) must be a list or tuple of list or tuple (of str/bytes/bytearray).')
    if not all(all(isinstance(arg,(str,bytes,bytearray))for arg in cmd)for cmd in cmds):
        raise TypeError('argument 1 (cmds) must be a list or tuple of list or tuple of str or bytes or bytearray.')
    for num in range(len(cmds)):
        if not all(isinstance(arg,type(cmds[num][0]))for arg in cmds[num]):
            print('cmd =',repr(cmd),flush=1)
            raise TypeError(f'argument 1 (cmds) cmd #{num} has not all args the same type of str/bytes/bytearray.')
    procs = []
    for cmd in cmds[:-1]:
        procs.append(Popen(cmd,stdout=PIPE,universal_newlines=isinstance(cmd[0],str),**opts))
        opts['stdin']=procs[-1].stdout # connect next stdin to previous stdout
    with Popen(cmds[-1],stdout=PIPE,universal_newlines=isinstance(cmds[-1][0],str),**opts) as lastproc:
        return (lastproc.stdout)

version = '0.0.0'

def main(cmdpath,args):
    slow = 1/32

# test the cmdsout() function with a little command pipeline that lists
# files with ls, filters the ls output for python files, and sorts the
# output by date/time.

    # each command is a list or tuple of str or bytes or bytearray
    cmd1=('ls','-l','--full-time')+tuple(args)
    cmd2=[b'egrep',b'\.py$']
    cmd3=[b'sort',b'-k6']

    # each command pipeline is a list or tuple of commands
    cmds=(cmd1,cmd2,cmd3)

    # the command pipeline is passed to cmdsout in the only argument
    # the return value is the stdout file
    out=cmdsout(cmds)
    print('',file=stderr)

    for line in out:
        line=line.rstrip()
        sleep(slow)
        write(1,line)
        write(1,b'\n')
    print('',file=stderr)
    print(f'got a {type(out).__name__} of {type(line).__name__}',file=stderr)

    return 0

def help():
    print('See a time sorted list of Python files.')
    return

def version():
    print(version)
    return

if __name__ == '__main__':
    cmdpath = argv.pop(0)
    if argv and argv[0][:2]=='--':
        if '--help' in argv:
            help()
        if '--version' in argv:
            version()
        exit(1)
    try:
        result=main(cmdpath,argv)
    except BrokenPipeError:
        exit(141)
    except KeyboardInterrupt:
        print(flush=True)
        exit(98)
    if isinstance(result,(bytes,bytearray)):
        result=''.join(chr(x)for x in result)
    if isinstance(result,str):
        print(result,file=stderr,flush=True)
        result=1
    if result is None or result is True:
        result=0
    elif result is False:
        result=1
    exit(int(float(result)))
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
What is the purpose of the with on line 31?

Yeah, I suspect that is the problem. The with tells python to tear down and release the resources when the object goes out of scope. When you return, the Popen __exit__() is called and the io object is closed.
Reply
#5
i will recode to try without with, tomorrow. bedtime for me. too much youtube.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#6
Popen has the default keyword argument: close_fds=True
Maybe you can still use the with-block, but then without closing the file descriptors for stdin, stdout and stderr.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#7
undoing the with block worked. it looks simpler, so i will just leave it that way. there are 2 other versions of that function, each with their own tester. the 1st returns a big single buffer of output. the 2nd returns a list of bytes, one per line. now i have a working 3rd.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#8
the 3 functions i created (without and with simple test code) are located here:
http://ipal.net/python-forum/cmdsbuf.py
http://ipal.net/python-forum/cmdsbuf_test.py
http://ipal.net/python-forum/cmdslist.py
http://ipal.net/python-forum/cmdslist_test.py
http://ipal.net/python-forum/cmdsout.py
http://ipal.net/python-forum/cmdsout_test.py
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  nested function return MHGhonaim 2 562 Oct-02-2023, 09:21 AM
Last Post: deanhystad
  return next item each time a function is executed User3000 19 2,171 Aug-06-2023, 02:29 PM
Last Post: deanhystad
  function return boolean based on GPIO pin reading caslor 2 1,131 Feb-04-2023, 12:30 PM
Last Post: caslor
  return vs. print in nested function example Mark17 4 1,674 Jan-04-2022, 06:02 PM
Last Post: jefsummers
  How to invoke a function with return statement in list comprehension? maiya 4 2,749 Jul-17-2021, 04:30 PM
Last Post: maiya
  Function - Return multiple values tester_V 10 4,319 Jun-02-2021, 05:34 AM
Last Post: tester_V
  Get return value from a threaded function Reverend_Jim 3 16,804 Mar-12-2021, 03:44 AM
Last Post: Reverend_Jim
  Return not exiting function?? rudihammad 3 5,190 Dec-01-2020, 07:11 PM
Last Post: bowlofred
  Why does my function return None? vg100h 3 2,155 Oct-29-2020, 06:17 AM
Last Post: vg100h
  Function will not return variable that I think is defined Oldman45 6 3,435 Aug-18-2020, 08:50 PM
Last Post: deanhystad

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020