redirect STDIO in the Python code

[Skaperen]

i need to have the ability to do standard I/O redirection within a CLI script in Python. I've done this a few times in C. to do this in Python i could just follow the way in C using os.open(), os.dup2(), and os.close(). but that might not let Python know what is going on, if it needs to know. is there a good Pythonic way to do this?

[Gribouillis]

i need to have the ability to do standard I/O redirection within a CLI script in Python.

What does that mean? Can you give details about what you want to do exactly and why you want to do it?

[Skaperen]

i want my Python script to change STDOUT and/or STDERR as used by all references to or copies of sys.stdout and sys.stderr to be some other form of output, such as an opened pipe to another process, to affect other code i include such as imported modules. in C i would do this by substituting file descriptors 1 and 2 with others by using syscall dup2(). tests doing this in Python with a few methods from the os module have worked. i would suspect this is rather unpythonic.

the reason i want to do this is that i want to have my Python code that invokes the module set up special handling of the output such as special file names and/or special processing to avoid duplication of data. for example, i want to compress data as it is written rather than write uncompressed data and compress it later (because the script may run for a long time, like many days or weeks). there may also be multiple invocations with different forms of output.

doing this in a bash script is messy and i want to avoid that.

[Gribouillis]

Why not just replacing sys.stdout and sys.stderr by custom file objects that send the data to the other forms of output?

For example the IDLE IDE interposes in front of these streams to send the output to a tkinter text window instead of the standard files.

[Skaperen]

Why not just replacing sys.stdout and sys.stderr by custom file objects that send the data to the other forms of output?

For example the IDLE IDE interposes in front of these streams to send the output to a tkinter text window instead of the standard files.

when any Python library forks a new process, does it make whatever is in sys.stdout, which may be using some other fd, such as fd 6, instead of fd 1, become fd 1, so that the new process gets it as fd 1 and likewise sys.stderr becomes fd 2?

[Gribouillis]

when any Python library forks a new process, does it make whatever is in sys.stdout, which may be using some other fd, such as fd 6, instead of fd 1, become fd 1

There is a lot of confusion here because you are mixing low-level concepts that belong to the C implementation of Python (such as file descriptors) with high-level concepts such as sys.stdin and sys.stdout which are abstract Python file objects that don't even need a file descriptor. The original question is too broad. If you want to do OS level operations in CPython, then use the available functions in modules such as os and subprocess. Most of what you do in C can be done in CPython, you are allowed to use functions such as os.fork or os.dup2, etc.

[Skaperen]

my big concern is that sys.stderr and/or sys.stdout, either a reference or (parts of) the object itself, could be cached by whatever code that accesses sys.stderr and/or sys.stdout has cached. as such, i would rather not try to update sys.stderr and/or sys.stdout in any way, and, instead, update the system reference (file descriptor in POSIX), instead.