looking fo documentation for module files

[Gribouillis]

Clearly these prefixes are wrong but is is becoming more and more difficult to find where it may come from. What do you have in sitecustomize.py and usercustomize.py ? Do you by any chance redefine site.PREFIXES in these modules ?

[Skaperen]

i've never changed any file with either of those names. since you did not say where/how they are found by the engine, i ran a full system-wide search in root. i only found "/etc/python3.8/sitecustomize.py" for the first one:

Output:
lt1a/forums/1 /home/forums 9> ls --full-time /etc/python3.8/sitecustomize.py
-rw-r--r-- 1 root root 155 2020-03-13 06:14:16.000000000 -0400 /etc/python3.8/sitecustomize.py
lt1a/forums/1 /home/forums 10> box /etc/python3.8/sitecustomize.py
+----</etc/python3.8/sitecustomize.py>----------------+
| # install the apport exception handler if available |
| try:                                                |
|     import apport_python_hook                       |
| except ImportError:                                 |
|     pass                                            |
| else:                                               |
|     apport_python_hook.install()                    |
+-----------------------------------------------------+
lt1a/forums/1 /home/forums 11> 

for "usercustomize.py" i found 3 files in a 7 year old backup from 2015.

Output:
rw-r--r-- 1 phil phil   5 2015-04-19 06:43:05.564927659 -0400 /home/black/u2/forums/.local/lib/python2.7/dist-packages/usercustomize.py
-rw-r--r-- 1 phil phil 151 2015-04-19 06:43:47.964926793 -0400 /home/black/u2/forums/.local/lib/python2.7/dist-packages/usercustomize.pyc
-rw-r--r-- 1 phil phil   5 2015-04-19 06:42:51.972927936 -0400 /home/black/u2/forums/.local/lib/python3.4/dist-packages/usercustomize.py

the contents of the 2 smaller .py files is "pass\n". so both of these appear to be benign. i have never coded "site.PREFIXES" anywhere i know of.

[Gribouillis]

You can use the -S option to disable the import of the site module. Does the following command give the same prefixes?

Output:
$ python3.8 -S -c "import sys; print(sys.prefix, sys.exec_prefix, sys.base_exec_prefix)"
/usr /usr /usr

i only found "/etc/python3.8/sitecustomize.py"

This looks strange too. There should be a copy in the python library. What is the output of

$ python3.8 -S -c "import sitecustomize; print(sitecustomize)"
<module 'sitecustomize' from '/usr/lib/python3.8/sitecustomize.py'>

[Skaperen]

sorry, these are in reverse order because the highlight-reply mechanism on the website does it that way:

This looks strange too. There should be a copy in the python library. What is the output of

Output:
lt1a/forums/3 /home/forums 4> python3.8 -S -c "import sitecustomize; print(sitecustomize)"
<module 'sitecustomize' from '/usr/lib/python3.8/sitecustomize.py'>
lt1a/forums/3 /home/forums 5> ls -dl --full-time /usr/lib/python3.8/sitecustomize.py
lrwxrwxrwx 1 root root 31 2022-06-22 16:18:18.000000000 -0400 /usr/lib/python3.8/sitecustomize.py -> /etc/python3.8/sitecustomize.py
lt1a/forums/3 /home/forums 6> box /etc/python3.8/sitecustomize.py
+----</etc/python3.8/sitecustomize.py>----------------+
| # install the apport exception handler if available |
| try:                                                |
|     import apport_python_hook                       |
| except ImportError:                                 |
|     pass                                            |
| else:                                               |
|     apport_python_hook.install()                    |
+-----------------------------------------------------+
lt1a/forums/3 /home/forums 7>

You can use the -S option to disable the import of the site module. Does the following command give the same prefixes?

Output:
lt1a/forums/3 /home/forums 7> python3.8 -S -c "import sys; print(sys.prefix, sys.exec_prefix, sys.base_exec_prefix)"
/usr /usr /usr
lt1a/forums/3 /home/forums 8>

[Gribouillis]

Output:
lt1a/forums/3 /home/forums 7> python3.8 -S -c "import sys; print(sys.prefix, sys.exec_prefix, sys.base_exec_prefix)"
/usr /usr /usr
lt1a/forums/3 /home/forums 8>

This means that before the import of the 'site' module, the prefixes are CORRECT! They get corrupted somewhere when you run without the -S option. Try this perhaps

Output:
python3.8 -S -c "import site; print(site.PREFIXES)"

and compare with

Output:
python3.8 -S -c "import site; site.main(); print(site.PREFIXES)"

[Skaperen]

Output:
lt1a/forums/3 /home/forums 7> python3.8 -S -c "import sys; print(sys.prefix, sys.exec_prefix, sys.base_exec_prefix)"
/usr /usr /usr
lt1a/forums/3 /home/forums 8>

This means that before the import of the 'site' module, the prefixes are CORRECT! They get corrupted somewhere when you run without the -S option. Try this perhaps

Output:
python3.8 -S -c "import site; print(site.PREFIXES)"

and compare with

Output:
python3.8 -S -c "import site; site.main(); print(site.PREFIXES)"

Output:
lt1a/forums/1 /home/forums 47> python3.8 -S -c "import site; print(site.PREFIXES)"
['/usr', '/usr']
lt1a/forums/1 /home/forums 48> python3.8 -S -c "import site; site.main(); print(site.PREFIXES)"
['/usr', '/usr']
lt1a/forums/1 /home/forums 49>

looks the same and correct to me.

[Skaperen]

even without -S those two commands get the same correct prefix. so the corruption must be somewhere else?

Output:
lt1a/forums/1 /home/forums 50> python3.8 -c "import site; print(site.PREFIXES)"
['/usr', '/usr']
lt1a/forums/1 /home/forums 51> python3.8 -c "import site; site.main(); print(site.PREFIXES)"
['/usr', '/usr']
lt1a/forums/1 /home/forums 52>

[Skaperen]

the problem seems to be transient and has appeared to have gone away. no idea when it did that so no idea how far back the correct answers are misleading us.

Output:
python3.8 -c "import site; print(site.getsitepackages())"
['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']

i didn't change anything related to python that i am aware of. i did run a full backup minus the /home directory (need to clear more space on the USB drive which will eventually become one of 2 or 3 backup drives).

[Skaperen]

it might be based on how i'm using symlinks for the "py" and "python" commands. i was trying other stuff this evening and noticed the problem would come and go. so i studied that more closely and noticed the problem always happens when i use "py" or "python" as the engine name and never when i used "python3.8". i found a source of "../../.." and it looks like this is the source. but something is still off as if is replicating the path to python3.8 it should work going that way since it really runs python3.8 either way. i think the merger of /bin and /usr/bin by Ubuntu is a factor in this.

Output:
 lt1a/forums/3 /home/forums 12> py -S -c "import site;print(site.getsitepackages())"
['/usr/host/bin/../../../local/lib/python3.8/dist-packages', '/usr/host/bin/../../../lib/python3/dist-packages', '/usr/host/bin/../../../lib/python3.8/dist-packages']
lt1a/forums/3 /home/forums 13> python -S -c "import site;print(site.getsitepackages())"
['/usr/host/bin/../../../local/lib/python3.8/dist-packages', '/usr/host/bin/../../../lib/python3/dist-packages', '/usr/host/bin/../../../lib/python3.8/dist-packages']
lt1a/forums/3 /home/forums 14> python3 -S -c "import site;print(site.getsitepackages())"
['/usr/host/bin/../../../local/lib/python3.8/dist-packages', '/usr/host/bin/../../../lib/python3/dist-packages', '/usr/host/bin/../../../lib/python3.8/dist-packages']
lt1a/forums/3 /home/forums 15> python3.8 -S -c "import site;print(site.getsitepackages())"
['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']
lt1a/forums/3 /home/forums 16> wh py
lrwxrwxrwx 1 root 22 Aug  8 01:02 /usr/host/bin/py -> ../../../bin/python3.8
lrwxrwxrwx 1 root 22 Aug  8 01:02 /usr/local/bin/py -> ../../../bin/python3.8
lt1a/forums/3 /home/forums 17> wh python
lrwxrwxrwx 1 root 22 Aug  8 01:02 /usr/host/bin/python -> ../../../bin/python3.8
lrwxrwxrwx 1 root 22 Aug  8 01:02 /usr/local/bin/python -> ../../../bin/python3.8
lrwxrwxrwx 1 root 7 Apr 15  2020 /usr/bin/python -> python3
lrwxrwxrwx 1 root 7 Apr 15  2020 /bin/python -> python3
lt1a/forums/3 /home/forums 18> wh python3
lrwxrwxrwx 1 root 22 Aug  8 01:02 /usr/host/bin/python3 -> ../../../bin/python3.8
lrwxrwxrwx 1 root 22 Aug  8 01:02 /usr/local/bin/python3 -> ../../../bin/python3.8
lrwxrwxrwx 1 root 9 May 25 00:58 /usr/bin/python3 -> python3.8
lrwxrwxrwx 1 root 9 May 25 00:58 /bin/python3 -> python3.8
lt1a/forums/3 /home/forums 19> wh python3.8
-rwxr-xr-x 1 root 5502744 Jun 22 16:18 /usr/bin/python3.8
-rwxr-xr-x 1 root 5502744 Jun 22 16:18 /bin/python3.8
lt1a/forums/3 /home/forums 20> box /usr/host/bin/wh
+----</usr/host/bin/wh>--------------------------------+
| #!/bin/bash                                          |
| which -a "$@"|while read n;do /bin/ls -dGl "$n";done |
+------------------------------------------------------+
lt1a/forums/3 /home/forums 21>

edit:

the fact that my symlinks go to ../../../{bin,lib} instead of ../../../usr/{bin,lib} may be the problem. more investigating. i wonder if a venv would have gotten around this.

[Gribouillis]

the fact that my symlinks go to ../../../{bin,lib} instead of ../../../usr/{bin,lib} may be the problem.

I'm glad you are so close to solve the issue. You could perhaps start by replacing the symlinks targets with absolute paths.