Subprocess use and modify inputs

[PierreLCV]

Dear all,
I started python code in 2024 (a good resolution is not it ?) to use for a job project and i am currently blocked with the subprocess use.
Basically, i have to run python script (written in CPython with CPython package: numpy, scipy...) in an environnement which does not support CPython. No problem, i have a tricks : subprocess
Indeed, as you know this function allows to run a code with a specified python version for instance. There is just after my code :

import os
import subprocess
os.chdir(r"My_code_location")
Python312 = r"C:\Users...\AppData\Local\Programs\Python\Python312\python.exe"
subprocess.call([Python312,'pyNukiyama.py'])

With this code i can run my pyNukiyama.py script in my environnement which does not support this last code with python 3.12 installed on the computer. So great, it works !
However, now i need to interact with this pyNukiyama.py script to change the inputs used in this last script (basically, to perform loop with values sent in pyNukiyama.py and retrieve output values).
Thus my question how to do that ?
Maybe, modify the script pyNukiyama.py before run the subprocess with it ? Or, not use a script saved but directly the code in subprocess (pyNukiyama contains just 10 lines...) ? Any other idea ?

Thank you for your help !

Best regards,
Pierre

[DeaD_EyE]

pyNukiyama.py

#!/usr/bin/env python3

def greet(name=None):
    print(f"Hello {name or 'world'}")

if __name__ == "__main__":
    # this block runs only, if the program was executed with the python interpreter
    # if this code were imported, this block is not executed
    greet("Test")

app.py

#!/usr/bin/env python3

from pyNukiyama import greet
# importing the function greet from pyNukiyama.py


if __name__ == "__main__":
    greet()

If you have a long-running CPU-Intensive python-program, it could make sense to run it as a process beside your application. But the interactivity of subprocess requires additional steps.

A Process can get:

• command line arguments
  
• signals like sigkill, sigterm, sigint, ...
  
• via standard input, standard output and standard error
  
• return value to communicate if the process was successful. 0 means OK
  

Another option is multiprocessing, where Python does the low level stuff.
Then you have to work with queues, because multiprocessing.Process does not return the value of a function. Instead it's a Python-Interpreter-Process managed by Python itself.

If the function is not cpu intensive and waits for IO, then threading is an alternative, but here is also a queue required, if you need to process the return value from the other process/function.

asyncio is even better, but first you should learn python basics.

[PierreLCV]

Dear DeaD_Eye,
Thank you for your answer, i am not sure to understand what you mean or if you understand what i wrote... :)
Basically, i would like to have code in which in can specify three values (V,P,T) for (Velocity, Pressure, Temperature), send these three values to pyNukiyama (inputs necessary for this module) executed with subprocess and retrieve the output of pyNukiyama with last inputs. Thus, i will be able to use these pyNukiyama results in my main code.
I hope this is more clear now :)

Thank you for your help,
best regards,
Pierre

[roryelila68]

For simple interactions and infrequent changes, modifying the script may be sufficient. For more complex interactions or frequent changes, stdin/stdout or temporary files are better choices. incredibox

[DeaD_EyE]

Basically, i would like to have code in which in can specify three values (V,P,T) for (Velocity, Pressure, Temperature), send these three values to pyNukiyama (inputs necessary for this module) executed with subprocess and retrieve the output of pyNukiyama with last inputs. Thus, i will be able to use these pyNukiyama results in my main code.

Is the function CPU-intensive? If not, then importing it as a module is better than executing it via subprocess. That would have been the first example.

Almost all beginners make the mistake of executing modules via subprocess. In most cases, this is unnecessary and makes the code unnecessarily complicated.

Even those beginners don't know Unix and process handling good enough. There are many pitfalls, which could be prevented, if you use the normal import. Examples with subprocess are often toy-examples.

In real-world applications it's much more complicated to do it right.


Example with subprocess:

pyNukiyama.py

#!/usr/bin/env python3
import sys


def compute(a, b, c):
    """
    Print the result of the calculation to stdout
    """
    print(a * 2, b * 3, c * 4)


if __name__ == "__main__":
    try:
        # at index 0 is the filename of the program
        # index 1, 2, 3 should be the arguments
        compute(sys.argv[1:4])
    except IndexError:
        print("Program requires 3 Arguments")
        sys.exit(1)

app.py

#!/usr/bin/env python3
import sys
import subprocess


def call_external_function(a, b, c):
    a, b, c = map(str, (a, b, c))
    # now a, b, c are str
    # calling to Python
    proc = subprocess.run(
        [sys.executable, "pyNukiyama.py", a, b, c], capture_output=True, encoding="utf8"
    )
    if proc.returncode != 0:
        raise RuntimeError("Problem with pyNukiyama")

    # proc.stdout is a str. To get floats back, you must convert them.
    values = tuple(map(float, proc.stdout.splitlines()[0].split()))

    print(f"Input: {a=}, {b=}, {c=}")
    print(f"Output: {values=}")


if __name__ == "__main__":
    call_external_function(1, 2, 3)

Then run app.py
app.py starts a subprocess with the same interpreter and sends the values a, b and c as arguments to the process. Then the process does its calculation and prints the result to stdout. The calling process splits the stdout and converts it from str to float. The encoding is given, so you get str. Otherwise, stdout, stderr are bytes.

Then you can decide to make it much more complicated. For example, reading from stdin in a loop.


main2.py

#!/usr/bin/env python3
import sys
from subprocess import PIPE, Popen
from random import randint


def main():
    # -u is for unbuffered stdin, otherwise it won't work
    proc = Popen([sys.executable, "-u", "pyNukiyama.py"], encoding="utf8", stdin=PIPE, stdout=PIPE)

    for _ in range(10):
        a, b, c = [randint(0, 100) for _ in range(3)]

        proc.stdin.write(f"{a} {b} {c}\n")
        proc.stdin.flush() # you need to flush the input

        line = proc.stdout.readline()
        a, b, c = map(float, line.split())
        print(f"{a=}, {b=}, {c=}")


if __name__ == "__main__":
    main()

pyNukiyama.py

#!/usr/bin/env python3
import sys

def compute(line):
    try:
        a, b, c = map(float, line.split())
    except (IndexError, ValueError):
        print("Error")
    else:
        print(a * 2, b * 3, c * 4)

def main():
    for line in sys.stdin:
        compute(line)


if __name__ == "__main__":
    main()

[deanhystad]

What is the environment you are using that supports subprocess.call but not numpy?