how to keep a Popen instance existant in a function return?

[Skaperen]

i have a function that is intended to start a subprocess.Popen instance and return the file object reference of the read end of the stdout pipe of that process. normally i can read this file OK. but this fails when created in a function. i have determined that the cause is the Popen instance object is dereferenced when the function returns resulting in the file object being return is now closed. the caller tries to read it and gets the exception "ValueError: I/O operation on closed file."

how can i prevent the Popen instance object from being dereferenced?

[bowlofred]

Can you post some code that shows this?

[Skaperen]

as you can see, this code takes a list of commands and runs them in a stdout to stdin pipeline. it returns the read side of the pipe from the last process in the pipeline.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98

from os import write from subprocess import PIPE,Popen from sys import argv,stderr from time import sleep   def cmdsout(cmds,**opts):     """Run a command (list/tuple of list/tuple of str/bytes/bytearray) returning its last output as a big str/bytes/bytearray buffer of newline separated lines."""   # for POSIX/BSD/Unix/Linux systems # tested on Ubuntu Linux # each command can be either a list or tuple # each command arg must be the same type within a command, one of str or bytes or bytearray # but can be different between commands # the command pipeline can be either a list or tuple       if not isinstance(cmds,(list,tuple)):         raise TypeError('argument 1 (cmds) must be a list or tuple (of list/tuple of str/bytes/bytearray).')     if not all(isinstance(cmd,(list,tuple))for cmd in cmds):         raise TypeError('argument 1 (cmds) must be a list or tuple of list or tuple (of str/bytes/bytearray).')     if not all(all(isinstance(arg,(str,bytes,bytearray))for arg in cmd)for cmd in cmds):         raise TypeError('argument 1 (cmds) must be a list or tuple of list or tuple of str or bytes or bytearray.')     for num in range(len(cmds)):         if not all(isinstance(arg,type(cmds[num][0]))for arg in cmds[num]):             print('cmd =',repr(cmd),flush=1)             raise TypeError(f'argument 1 (cmds) cmd #{num} has not all args the same type of str/bytes/bytearray.')     procs = []     for cmd in cmds[:-1]:         procs.append(Popen(cmd,stdout=PIPE,universal_newlines=isinstance(cmd[0],str),**opts))         opts['stdin']=procs[-1].stdout # connect next stdin to previous stdout     with Popen(cmds[-1],stdout=PIPE,universal_newlines=isinstance(cmds[-1][0],str),**opts) as lastproc:         return (lastproc.stdout)   version = '0.0.0'   def main(cmdpath,args):     slow = 1/32   # test the cmdsout() function with a little command pipeline that lists # files with ls, filters the ls output for python files, and sorts the # output by date/time.       # each command is a list or tuple of str or bytes or bytearray     cmd1=('ls','-l','--full-time')+tuple(args)     cmd2=[b'egrep',b'\.py$']     cmd3=[b'sort',b'-k6']       # each command pipeline is a list or tuple of commands     cmds=(cmd1,cmd2,cmd3)       # the command pipeline is passed to cmdsout in the only argument     # the return value is the stdout file     out=cmdsout(cmds)     print('',file=stderr)       for line in out:         line=line.rstrip()         sleep(slow)         write(1,line)         write(1,b'\n')     print('',file=stderr)     print(f'got a {type(out).__name__} of {type(line).__name__}',file=stderr)       return 0   def help():     print('See a time sorted list of Python files.')     return   def version():     print(version)     return   if __name__ == '__main__':     cmdpath = argv.pop(0)     if argv and argv[0][:2]=='--':         if '--help' in argv:             help()         if '--version' in argv:             version()         exit(1)     try:         result=main(cmdpath,argv)     except BrokenPipeError:         exit(141)     except KeyboardInterrupt:         print(flush=True)         exit(98)     if isinstance(result,(bytes,bytearray)):         result=''.join(chr(x)for x in result)     if isinstance(result,str):         print(result,file=stderr,flush=True)         result=1     if result is None or result is True:         result=0     elif result is False:         result=1     exit(int(float(result)))

[bowlofred]

What is the purpose of the with on line 31?

---

Yeah, I suspect that is the problem. The with tells python to tear down and release the resources when the object goes out of scope. When you return, the Popen __exit__() is called and the io object is closed.

[Skaperen]

i will recode to try without with, tomorrow. bedtime for me. too much youtube.

[DeaD_EyE]

Popen has the default keyword argument: close_fds=True
Maybe you can still use the with-block, but then without closing the file descriptors for stdin, stdout and stderr.

[Skaperen]

undoing the with block worked. it looks simpler, so i will just leave it that way. there are 2 other versions of that function, each with their own tester. the 1st returns a big single buffer of output. the 2nd returns a list of bytes, one per line. now i have a working 3rd.

[Skaperen]

the 3 functions i created (without and with simple test code) are located here:
http://ipal.net/python-forum/cmdsbuf.py
http://ipal.net/python-forum/cmdsbuf_test.py
http://ipal.net/python-forum/cmdslist.py
http://ipal.net/python-forum/cmdslist_test.py
http://ipal.net/python-forum/cmdsout.py
http://ipal.net/python-forum/cmdsout_test.py