Using subprocess to execute complex command with many arguments

Using subprocess to execute complex command with many arguments - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Using subprocess to execute complex command with many arguments (/thread-39872.html)

Using subprocess to execute complex command with many arguments - medatib531 - Apr-26-2023

I have a shell script that is as follows:

LOAD_RRD=sysmon.rrd
DISK_RRD=disk.rrd
DISKTEMP_RRD=disktemp.rrd

rrdtool graph load1h.png \
		-Y -u 1.1 -l 0 -L 5 -v "Load" -w 700 -h 200 -t "Load stats - `/bin/date`" \
        --start end-"1h" --x-grid MINUTE:1:MINUTE:5:MINUTE:10:0:%R \
		-c ARROW\#000000 \
		DEF:load1=$LOAD_RRD:load1:AVERAGE \
		DEF:load5=$LOAD_RRD:load5:AVERAGE \
		DEF:load15=$LOAD_RRD:load15:AVERAGE \
		LINE1:load1\#ff0000:"Load average 1 min" \
		LINE1:load5\#ff6600:"Load average 5 min" \
		LINE2:load15\#ffaa00:"Load average 15 min" \
		COMMENT:"	\j" \
		COMMENT:"\j" \
		COMMENT:"	" \
		GPRINT:load15:MIN:"Load 15 min minimum\: %lf" \
		GPRINT:load15:MAX:"Load 15 min maximum\: %lf" \
		GPRINT:load15:AVERAGE:"Load 15 min average\: %lf" \
		COMMENT:"	\j" \
		COMMENT:"	" \
		COMMENT:"	\j" >/dev/null;

I want to convert it to python using the subprocess module and run similar commands but with manipulating the internal variables and parameters.

I was thinking of copying the whole command in a single python string cmd_str then run

subprocess.run(cmd_str, shell=True)

But I'm not on the best way to escape all those internal quotes and other special symbols.
Any suggestions?

RE: Using subprocess to execute complex command with many arguments - Gribouillis - Apr-26-2023

(Apr-26-2023, 01:47 PM)medatib531 Wrote: Any suggestions?

Yes you just pass a list of the command's arguments as strings

subprocess.run([ "rrdtool",  "graph", "load1h.png",
      "-Y",  "-u", "1.1", "-l", "0", "-L", "5", "-v", "Load", ...])

Also I see that there is a Python API to rrdtool. Why not use that?

RE: Using subprocess to execute complex command with many arguments - medatib531 - Apr-26-2023

But what is the correct way to escape the quotes that are originally in the shell command?
E.g. for "Load" should it be something like:

subprocess.run([ "rrdtool",  "graph", "load1h.png", "-Y",  "-u", "1.1", "-l", "0", "-L", "5", "-v", '"Load"', ...])

?
Also do I have to somehow escape the backslashes, dollar and percent symbols as well, or are they fine within double quotes in the list of strings?

RE: Using subprocess to execute complex command with many arguments - DeaD_EyE - Apr-26-2023

https://docs.python.org/3/library/shlex.html#shlex.split

import shlex


my_command = """
rrdtool graph load1h.png \
        -Y -u 1.1 -l 0 -L 5 -v "Load" -w 700 -h 200 -t "Load stats - `/bin/date`" \
        --start end-"1h" --x-grid MINUTE:1:MINUTE:5:MINUTE:10:0:%R \
        -c ARROW\#000000 \
        DEF:load1=$LOAD_RRD:load1:AVERAGE \
        DEF:load5=$LOAD_RRD:load5:AVERAGE \
        DEF:load15=$LOAD_RRD:load15:AVERAGE \
        LINE1:load1\#ff0000:"Load average 1 min" \
        LINE1:load5\#ff6600:"Load average 5 min" \
        LINE2:load15\#ffaa00:"Load average 15 min" \
        COMMENT:"   \j" \
        COMMENT:"\j" \
        COMMENT:"   " \
        GPRINT:load15:MIN:"Load 15 min minimum\: %lf" \
        GPRINT:load15:MAX:"Load 15 min maximum\: %lf" \
        GPRINT:load15:AVERAGE:"Load 15 min average\: %lf" \
        COMMENT:"   \j" \
        COMMENT:"   " \
        COMMENT:"   \j" >/dev/null;
""".strip().replace("\\", "")


my_cmd = shlex.split(my_command)
print(my_cmd)

The last field ">/dev/null" is wrong, but I guess you still want to suppress stdout:

subprocess.run(my_cmd, stdout=subprocess.DEVNULL)

Just get rid of the >/dev/null

RE: Using subprocess to execute complex command with many arguments - Gribouillis - Apr-26-2023

(Apr-26-2023, 02:27 PM)medatib531 Wrote: But what is the correct way to escape the quotes that are originally in the shell command?

I ran a command to show the arguments received by your program and I get this list, that you can pass to the subprocess.run command (the first argument is missing, it should be the program, 'rrdtool')

[ 'graph', 'load1h.png', '-Y', '-u', '1.1', '-l', '0', '-L', '5', '-v', 'Load',
 '-w', '700', '-h', '200', '-t', 'Load stats - mer. 26 avril 2023 17:11:35 CEST', 
'--start', 'end-1h', '--x-grid', 'MINUTE:1:MINUTE:5:MINUTE:10:0:%R', '-c', 'ARROW#000000', 
'DEF:load1=sysmon.rrd:load1:AVERAGE', 'DEF:load5=sysmon.rrd:load5:AVERAGE', 
'DEF:load15=sysmon.rrd:load15:AVERAGE', 'LINE1:load1#ff0000:Load average 1 min', 
'LINE1:load5#ff6600:Load average 5 min', 'LINE2:load15#ffaa00:Load average 15 min',
 'COMMENT:   \\j', 'COMMENT:\\j', 'COMMENT:   ', 'GPRINT:load15:MIN:Load 15 min minimum\\: %lf', 
'GPRINT:load15:MAX:Load 15 min maximum\\: %lf',
 'GPRINT:load15:AVERAGE:Load 15 min average\\: %lf', 'COMMENT:   \\j', 'COMMENT:   ', 'COMMENT:   \\j']

@DeaD_EyE 's method using the shlex module is not perfect because it doesn't take into account the string interpolation of the shell, for example the variables $LOAD_RRD, etc

By the way, here is my executable "echoargs" script to show arguments

#!/usr/bin/env python3
# echoargs script
import sys
 
if __name__ == '__main__':
    print(sys.argv)

If you run your shell script by removing > /dev/null and you replace rrdtool with echoargs , it will print the arguments to the console in the same way that they are received by the invoked command after the shell actually processed them.

Note that the shell ran /bin/date and replaced it in the result by today's date (in french because it is my locals). So you may need to work a little bit on this, but Python's datetime module should be invoked instead of the /bin/date command to get a more pythonic script.

RE: Using subprocess to execute complex command with many arguments - medatib531 - Apr-27-2023

Worked, thank you all!