Python Forum

the script being developed will basically be like a shell script, running many command with some logic between many of them. at the start of the script i want to redirect STDOUT and/or STDERR to output to a specified file, to record all the output from all the commands. in bash, i can do that with this line in the script:

exec &>/tmp/bigscript.log

using my past C experience i came up with this Python3 code:

import os
fd = os.open('/tmp/bigscript.log',os.WRONLY|os.O_CREAT|os.O_TRUNC,mode=0o600)
if fd!=0:os.dup2(fd,0)
if fd!=1:os.dup2(fd,1)
os.close(fd)

is this code good enough or is there a more expected (and pythonic) way to do this?

(Dec-05-2023, 07:10 AM)Skaperen Wrote: [ -> ]is this code good enough or is there a more expected (and pythonic) way to do this?

I think it is ok if you want to do this very specific thing, redirect every output of subprocesses to the file.

I would probably choose another way. First you can redirect the streams when the script is invoked, for example let the script be

# myscript.py
import subprocess as sp
import sys

# print our args
print(sys.argv)
sys.stdout.flush()

# run some command which creates output
sp.run(["date"])

It can be invoked at the console with a redirection

Output:
λ python paillasse/pf/myscript.py --spam eggs ham >/tmp/bigscript.log 2>&1 
λ 

Let us check the content of /tmp/bigscript.log

Output:
λ cat /tmp/bigscript.log
['paillasse/pf/myscript.py', '--spam', 'eggs', 'ham']
mar. 05 déc. 2023 11:09:13 CET
λ 

If we don't want to redirect it at the console, we can interpose a driver script which does the redirection

# driver.py
from pathlib import Path
import subprocess as sp
import sys

myscript = Path(__file__).parent / "myscript.py"

with Path("/tmp/bigscript.log").open("w") as ofile:
    sp.run([sys.executable, myscript, *sys.argv[1:]], stdout=ofile, stderr=sp.STDOUT)

Let us invoke the driver

Output:
λ python paillasse/pf/driver.py --spam eggs ham
λ 

We now check the content of the output file

Output:
λ cat /tmp/bigscript.log
['/home/eric/Projets/Scratch/2023-01/paillasse/pf/myscript.py', '--spam', 'eggs', 'ham']
mar. 05 déc. 2023 11:11:43 CET
λ 

Pythonic way to redirect stdout and stderr:

import sys
from contextlib import redirect_stdout, redirect_stderr


def foo():
    print("STDOUT: Hello World")
    print("STDERR: Hello World", file=sys.stderr)


def main():
    foo()


if __name__ == "__main__":
    print("No redirect")
    main()

    print("Redirecting stdout and stderr to a file")
    with open("logfile.log", "a") as logfile:
        with redirect_stdout(logfile):
            with redirect_stderr(logfile):
                main()

I would use tee. Then you get the output on stdout and stderr, and it's written to a file.
But in this case, stderr is not written to the file.

python3 your_program.py | tee logfile.txt

If you want to redirect also stderr, you could redirect it:

python3 your_program.py 2>&1 | tee logfile.txt

(Dec-05-2023, 01:06 PM)DeaD_EyE Wrote: [ -> ]Pythonic way to redirect stdout and stderr:

But this won't work with subprocesses because it is too high level.

It could work, if you set stderr and stdout to subprocess.PIPE.

(Dec-07-2023, 04:56 AM)DeaD_EyE Wrote: [ -> ]It could work, if you set stderr and stdout to subprocess.PIPE.

I tried it but it doesn't work. Here is my script

# myscript.py
from contextlib import redirect_stdout, redirect_stderr
from pathlib import Path
import subprocess as sp
import sys


def main():
    # print our args
    print(sys.argv)
    sys.stdout.flush()

    # run some command which creates output
    sp.run(["date"], stdout=sp.PIPE, stderr=sp.PIPE) # OUTPUT IS LOST


with Path("/tmp/bigscript.log").open("w") as ofile:
    with redirect_stdout(ofile), redirect_stderr(ofile):
        main()

Output:
λ python paillasse/pf/myscript.py
λ cat /tmp/bigscript.log
['paillasse/pf/myscript.py']
λ 

I don't know where the output of sp.run(["date"], stdout=sp.PIPE, stderr=sp.PIPE) was sent.

On the other hand IT WORKS if I call instead explicitly

    sp.run(["date"], stdout=sys.stdout, stderr=sys.stderr)

This could be the pythonic way to go, but I don't think I could do this and redirect to an abstract file object without fileno, in order to make a tee for example.

i want to have this more integrated so that the redirection is done by the script itself and minimize the number of files. of course, having all the steps needed to do the redirection done in a function or class method so it can be done in one line would be.

what i want to accomplish is how best to do in Python what i have done in Bash and/or C. in C it is trivial because STDOUT and STDERR are just file descriptors which can be changed.

classic redirection is done at command level where one adds on shell syntax when typing in a command or doing the same in a shell script. what i want is to integrate this so that the redirection is done in the script itself so that i don't have to type it in each time or add another script file.

(Dec-07-2023, 06:32 PM)Skaperen Wrote: [ -> ]i want to have this more integrated so that the redirection is done by the script itself and minimize the number of files

The more pythonic way could be to use @DeaD_EyE 's suggestion plus subprocess.check_output() or Popen.communicate(). Here is a script

# myscript.py
from contextlib import redirect_stdout, redirect_stderr
from pathlib import Path
import subprocess as sp
import sys


def main():
    # print our args
    print(sys.argv)

    # run some command which creates output
    data = sp.check_output(['date'], stderr=sp.STDOUT, encoding='utf8')
    print(data, end='')

with Path("/tmp/bigscript.log").open("w") as ofile:
    with redirect_stdout(ofile), redirect_stderr(ofile):
        main()

Output:
λ cat /tmp/bigscript.log
['paillasse/pf/myscript.py']
ven. 08 déc. 2023 09:38:16 CET
λ