Confused about python execution

[jpezz]

I copy my programs to a directory that is a ram disk (for speed). Note that execution speed is critical. Startup time is not.
cp ~key/Programs.Source/py3*.py ~key/Programs # Copy to RAM disk

Then I execute:
python3 -m compileall ~key/Programs/py3*

This creates the subdirectory __pycache__ and the .pyc files

Now, presently, I execute my programs by having in the bash script:
python3 Programs/[xxx.py]
Is this the fastest way for them to execute or should I be executing them as
python3 Programs/__pycache__/[xxx.pyc]
or simply
Programs/__pycache__[/xxx.pyc]


It was my understanding that if the subdirectory __pycache_ existed, it would automatically use the program name there but I am not too sure any more.

My objective is to have them execute as quickly as possible.

Also, at the time I originally built the programs, I did not use
#/usr/bin/python3
because I built them with python2 knowing I would eventually go over to python3. I never got around to adding in that line.
Is there any significant difference in time to use that line and do this:
Programs/[xxx.py]
instead of this
#/usr/bin/python3 Programs/[xxx.py]
or is it just good practice so as to identify that this is a Python3 script vs any other type and that the shell knows to call /usr/bin/python3 to run it? Even 40 years ago, I knew to use the #!/bin/sh header but didn't do it for my python programs.
(Note, there was no bash - only sh and csh back in those days)

[Gribouillis]

My objective is to have them execute as quickly as possible.

Usually, when a program is slow, the bottleneck is not the time it takes to load the program in memory. The actions that you are suggesting such as storing the program in RAM or using compiled Python files only alter the time that it takes to load the program. It will not accelerate the algorithms or the execution of the Python code.

The only case I can think of where this could change things is if the program does almost nothing and the loading time becomes significant with respect to the real execution time.

Compiled Python files .pyc are automatically used when a module such as «spam» is imported and the spam.pyc file is more recent than the spam.py file. Normally you don't need to worry about this.

Here is an example on my computer with a program «pyspam.py» that simply prints «Hi» and does nothing else. As you can see the running time is about the same for the .py file, the .pyc file and the .pyc file copied to ramdisk.

Output:
λ time python3 paillasse/pf/pyspam.py
hi

real    0m0,048s
user    0m0,032s
sys     0m0,016s
λ time python3 paillasse/pf/__pycache__/pyspam.cpython-310.pyc
hi

real    0m0,050s
user    0m0,046s
sys     0m0,004s
λ cp paillasse/pf/__pycache__/pyspam.cpython-310.pyc /media/ramdisk
λ time python3 /media/ramdisk/pyspam.cpython-310.pyc
hi

real    0m0,049s
user    0m0,040s
sys     0m0,008s

[jpezz]

Thanks for your reply. I see a notable difference in my program speeds when using ram disks for program execution and for storing output information (which is copied to the real disk periodically) but that's not the real subject of my question.

Using your example, it appears that executing
python3 paillasse/pf/pyspam.py

is notably slower than
python3 paillasse/pf/__pycache__/pyspam.cpython-310.pyc

but it is not clear that in the first example that you created the pyc file before the first example. To clarify my question, if the pyc file exists AND is newer AND I specify the py file, does python use the pyc such that the two commands take the same time and it uses the pyc automatically meaning I don't have to explicitly use it in the command line or should I be using the pyc file by name as shown in the second example? Your example clearly shows the extra benefit of the pyc file but it is not clear as to whether it existed prior to running the py file or whether you created it between those two examples.

I am also curious as to whether using the use of the

#/usr/bin/python3

so as to directly call the program speeds anything up vs having to specify

python3 program.py

.


My guess is that pre-compiling and creating the pyc file removes all the comment lines (I have LOTS of them mostly to help me remember what and why I was doing when I wrote each line of the code), merging the includes into the code and doing the interpretation ahead of time to increase the execution speed and reduce the loading time.

My guess is also that the only purposes of the shebang is to tell the executing program (i.e. python, python2, python3) and the person running it that code being executed is indeed really intended for that compiler.

[menator01]

The shebang more or less is for nix os. Useless for windows os

[Gribouillis]

I see a notable difference in my program speeds when using ram disks for program execution and for storing output information

If your program does Input/Output, it will be faster if you read and write from RAM, but storing the program itself in RAM is not necessary, there is very little benefit. The program is always loaded to RAM when it is executed.

Your example clearly shows the extra benefit of the pyc file

It does not. If the program runs during more than a few seconds, the difference is neglectible. Try with real programs and show the performance measurements.

#/usr/bin/python3

The shebang line starts with #! , not # . It does not speed up anything.

if the pyc file exists AND is newer AND I specify the py file, does python use the pyc

No, as far as I know. Python uses automatically the .pyc file when modules are imported for the first time. I don't think it does this for the main program.

My guess is also that the only purposes of the shebang is to tell the executing program

This guess is wrong. In Linux (and Unixes in general), if the file has a shebang and it is executable, the OS uses the shebang to know which program to call (here python3).

Again, apart from Input/Output in RAM, which can be really useful, all these «optimizations» seem futile to me.