Python Forum

what i would like to add to subprocess.Popen():

something i often did when coding in C that is not convenient in Python is changing the working directory or other setting in the forked child process before calling exec*() to start a new executable. perhaps that execution needed to be done in that context. or, perhaps, that context was needed to find that executable file. the code, in POSIX, would call fork() and then in the child process it would call chdir() with the appropriate directory and finally call exec*(). i could do this in Python. but i would prefer to do this using Python tools like subprocess.Popen(). but there is no means to run some specific code between fork() and exec*() in the child process and no means to supply a directory path name for it to do a chdir() with.

i would like to see subprocess.Popen() improved to have these capabilities. an option chdir= would pass a string (str or bytes) of the path to a directory to change to. the call would exit the child and raise the appropriate exception if the chdir() fails. another keyword argument would have a name like precall= and would refer to a function to be called in the child process before exec*() is called.

i do not know if these calls would be appropriate for platforms other than POSIX.

Use the cwd= keyword argument of subprocess.Popen() and other functions, for example

>>> import subprocess as sp
>>> import os
>>> sp.run(['ls'], cwd=os.path.expanduser('~'))

is cwd= the function to be called in the context of the forked child process?

(Aug-22-2023, 02:15 AM)Skaperen Wrote: [ -> ]is cwd= the function to be called in the context of the forked child process?

I'm not sure about what this means, but I think the answer is yes because the cwd parameter does not change the working directory of the parent process. Here is what the documentation says

Output:
If cwd is not None, the function changes the working directory to cwd before executing the child. cwd can be a string, bytes or path-like object. On POSIX, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path.

Changed in version 3.6: cwd parameter accepts a path-like object on POSIX.

Changed in version 3.7: cwd parameter accepts a path-like object on Windows.

Changed in version 3.8: cwd parameter accepts a bytes object on Windows.

OK. i think that is what i wanted as chdir=. apparently my thinking gave it some other semantics as i scanned the documentation originally. this time it looks right. the future may still have a need for other settings. maybe they can be done in the parent or maybe not. best would be a function, maybe with args. then you can play dangerously, like root changing effective user (BTDT in C). for a current project, cwd= is good enough. it is a function to run a pipeline of commands, and i chose to use the same argument cwd= to be consistent with Popen() and maybe others.

(Aug-22-2023, 05:29 PM)Skaperen Wrote: [ -> ]the future may still have a need for other settings. maybe they can be done in the parent or maybe not. best would be a function, maybe with args.

I think you could do a lot by calling a Python function in a child process with the 'multiprocessing' module before calling the subprocess.Popen(). Here is an example

import os
import multiprocessing as mp
import subprocess as sp

def target():
    print(f'In child: {os.getpid()}')
    print('Child doing some stuff...')
    os.chdir('/')
    sp.run(['ls'])

p = mp.Process(target=target)
p.start()
print(f'In parent: {os.getpid()}')
p.join()

Output:
λ python paillasse/pf/subproccom.py
In parent: 4035
In child: 4036
Child doing some stuff...
bin   cdrom  etc   lib    lib64   lost+found  mnt  proc  run   snap  sys  usr
boot  dev    home  lib32  libx32  media       opt  root  sbin  srv   tmp  var

You can also use the env= option of subprocess.Popen() to fine-tune environment variables.

Popen() already has forking taking place. i really want to avoid doing it twice.

(Aug-24-2023, 10:48 PM)Skaperen Wrote: [ -> ]i really want to avoid doing it twice.

Unless you fork hundreds of processes, how is that a problem? The computer is working for you, nobody will be harmed.

on some systems, process limits exist for various reasons. i don't want my script to run ever so briefly in 3 processes when it can do so in 2. if the user is limited to 10 processes, it can make difference if they are already running 8. many graphical systems and display managers eat those up rather readily. i've run into this issue before on some highly shared systems with thousands of users.