Python Forum

I am running Debian Buster/Sid and have Pythons 2.7.16 and 3.7.2

System Python is by default /usr/lib/Pythonxx/dist-packages
PyPy installs by default into /usr/local/lib/Pythonxx/dist-packages.

After a April Debian update of system python, I began to install modules from PyPy with the occasional note that (presumably) system modules could not be removed during install, and I assumed all was well. sys.path shows that distro paths trump PyPy (site paths). Debian in its infinite idiocy decided to deprecate the 'site' prefix.

In fact, all was working well, and everything was peaches and cream

But THEN, the other day I grabbed some *new* python entries from the Debian repository, which appeared to mess with the system. Everything still appears to work, but

buster:/# python3 -v

# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone/app
# destroy plone.app
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone/app
# destroy plone.app
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone/app
# destroy plone.app
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone/formwidget
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# destroy plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/plone
# possible namespace for /usr/local/lib/python3.7/dist-packages/repoze
# destroy repoze
# possible namespace for /usr/local/lib/python3.7/dist-packages/repoze
# destroy repoze
# possible namespace for /usr/local/lib/python3.7/dist-packages/repoze
# destroy repoze
... (lots more)
# possible namespace for /usr/local/lib/python3.7/dist-packages/ruamel
# possible namespace for /usr/local/lib/python3.7/dist-packages/scikits
# possible namespace for /usr/local/lib/python3.7/dist-packages/scikits
# destroy scikits
# possible namespace for /usr/local/lib/python3.7/dist-packages/snowflake
# destroy sphinxcontrib
# destroy sphinxcontrib
# destroy sphinxcontrib
# possible namespace for /usr/local/lib/python3.7/dist-packages/z3c
# destroy z3c
# possible namespace for /usr/local/lib/python3.7/dist-packages/z3c/recipe
# possible namespace for /usr/local/lib/python3.7/dist-packages/z3c
# destroy z3c
# possible namespace for /usr/local/lib/python3.7/dist-packages/z3c
# destroy z3c
# possible namespace for /usr/local/lib/python3.7/dist-packages/zc
# possible namespace for /usr/local/lib/python3.7/dist-packages/zc
# destroy zc
# possible namespace for /usr/local/lib/python3.7/dist-packages/zc
# destroy zc
...
# possible namespace for /usr/local/lib/python3.7/dist-packages/zc
# destroy zc
# possible namespace for /usr/local/lib/python3.7/dist-packages/zc/recipe
...
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope/app
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope/app
# destroy zope.app
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope/app
... 
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zope
# destroy zope
# possible namespace for /usr/local/lib/python3.7/dist-packages/zopyx
# possible namespace for /usr/local/lib/python3.7/dist-packages/zopyx
# destroy zopyx
# possible namespace for /usr/local/lib/python3.7/dist-packages/zopyx
# destroy zopyx
# possible namespace for /usr/local/lib/python3.7/dist-packages/zopyx/txng3
# possible namespace for /usr/local/lib/python3.7/dist-packages/zopyx/txng3
# destroy zopyx.txng3
# possible namespace for /usr/local/lib/python3.7/dist-packages/zopyx/txng3/ext
# possible namespace for /usr/lib/python3/dist-packages/pyhoca
# possible namespace for /usr/lib/python3/dist-packages/stetl
# possible namespace for /usr/lib/python3/dist-packages/xstatic
# possible namespace for /usr/lib/python3/dist-packages/xstatic
# destroy xstatic
# possible namespace for /usr/lib/python3/dist-packages/xstatic/pkg
# possible namespace for /usr/lib/python3/dist-packages/xstatic
# destroy xstatic
# possible namespace for /usr/lib/python3/dist-packages/xstatic
# destroy xstatic
# possible namespace for /usr/lib/python3/dist-packages/xstatic/pkg
# destroy xstatic.pkg
# possible namespace for /usr/lib/python3/dist-packages/xstatic
# destroy xstatic
# possible namespace for /usr/lib/python3/dist-packages/xstatic
# destroy xstatic
# possible namespace for /usr/lib/python3/dist-packages/xstatic/pkg
# destroy xstatic.pkg
.....
# cleanup[2] removing zc.recipe
# destroy zc.recipe
# cleanup[2] removing zest
# destroy zest
# cleanup[2] removing zestreleaser
# destroy zestreleaser
# cleanup[2] removing zope
# destroy zope
# cleanup[2] removing zope.app
# destroy zope.app
# cleanup[2] removing zopyx
# destroy zopyx
# cleanup[2] removing zopyx.txng3
# destroy zopyx.txng3
# cleanup[2] removing zopyx.txng3.ext
# destroy zopyx.txng3.ext
# cleanup[2] removing paste
# destroy paste
# cleanup[2] removing pyhoca
# destroy pyhoca
# cleanup[2] removing stetl
...
# cleanup[3] wiping posix
# cleanup[3] wiping codecs
# cleanup[3] wiping _codecs
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping importlib._bootstrap
# cleanup[3] wiping sys
# cleanup[3] wiping builtins

Any ideas on how to fix this mess?
Or even, should I bother?
'import zope'
Works fine

This leads to a second question, assuming all is actually OK, or all I need to do is yank some duplicates out of the PyPy installed dirs that conflict with system (which itself should be fairly up to date).
I have the option of Debian updating the 3.72 system python to 3.82...
The thing is : will the thousands of already imported modules from PyPy be ignored in their /local/lib/python37/dist-packages dir?
Can I hack them into site.py?

(I have used this basic method with perlpath with good results!)

you should not use -v, must be -V

python3 -V

or

python3 --version

Thank you. Comes up now simply as Python 3.7.2

However, what in gawds name were all those errors in the -v flag about?

python --help

Quote:-v : verbose (trace import statements); also PYTHONVERBOSE=x
can be supplied multiple times to increase verbosity

Quote:After a April Debian update of system python, I began to install modules from PyPy with the occasional note that (presumably) system modules could not be removed during install, and I assumed all was well. sys.path shows that distro paths trump PyPy (site paths). Debian in its infinite idiocy decided to deprecate the 'site' prefix.

Let's call it PyPi,PyPy is somthing else
A little confusion here "distro paths trump PyPy (site paths)",no there is no trump PyPi.

It work like this on Debian.
dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu.
Debian Python Wiki:

Quote:dist-packages instead of site-packages.
Third party Python software installed from Debian packages goes into dist-packages, not site-packages.
This is to reduce conflict between the system Python, and any from-source Python build you might install manually.

Means that if you manually install Python from source,it uses the site-packages directory.
site-packages or dist-packages is the place Python install 3-party modules/package.

With a default Debian distro no mess from you it will install to dist-packages.
The placement really don't matter as eg pip install requests or remove pip uninstall requests will find correct folder always.
As note so has Windows always used site-packages,as folder for 3-party installed modules/package.

The above path makes absolute sense. Site paths SHOULD be clearly distinct from dist(ro) paths. Unfortunately, Debian is not making much sense these days:

>>> print(sys.path )
['', '/usr/lib/python37.zip',
 '/usr/lib/python3.7', 
'/usr/lib/python3.7/lib-dynload', 
'/root/.local/lib/python3.7/site-packages', 
'/usr/local/lib/python3.7/dist-packages', 
'/usr/lib/python3/dist-packages', 
'/usr/lib/python3/dist-packages/linkgrammar',
 '/usr/lib/python3.7/dist-packages']

The only site path in under /root/.local and it is *empty*

Pypi (thank you for clearing this up!) is installing to /usr/local, which is below the distro dumping ground, so appears safe.

Have no clue about the python37.zip entry. It is not there.

In short, it *appears* that:
System Python packages (Debian) -> /usr/lib
User Python Packages(PyPi) -> /usr/local/lib

Ideally, assuming this behaves like Perl, there should be no conflicts: Debian packages should always take precedence over Pypi packages, if there is duplication of names.

But... are paths searched also for versions?
Assume: foo.3.98 is in Debian, and foo.4.47 is in Pypi dirs.
Package bar has a requirment of foo>4.0
Will it stop at at the first parsed Debian dir, and assume the package does not meet requirements, or go on and look in the Pypi dirs for the updated version?

This is not a production system, so I really dont care about versioning with projects (yet!). My main objective is to get as complete a development system as possible (without needing to rely on cloud) so that everything is in place and working when I do start doing some serious projects. Like perhaps remaking my old ecommerce OsCommerce site in Django.

One of the problems with a Debian distro, that that the maintainers expect everyone to stay within their ecosystem, but that does have a tendency to be restrictive. So my basic concern is whether I have to be concerned about any possible conflicts of system python with user addons. In Perl I did not have problems between system and CPAN addons.

I would like to avoid venv as much as possible. It reminds me too much of wine 'bottles' .

What's the problem?
You talk about "Pypi dirs" PyPi has no dirs,all is done by pip which install to Python version shown in pip -V.
Here a demo i use pyenv Simple Python Version Managemet then it will always be python and pip to current system wide installation.
You may need to use python3 and pip3.

tom@tom:~$ python -V
Python 3.7.3

# Placement of python
tom@tom:~$ which python
/home/tom/.pyenv/shims/python

# This show python version and where pip install to
tom@tom:~$ pip -V
pip 19.1.1 from /home/tom/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip (python 3.7)

# Install
tom@tom:~$ pip install logzero
Collecting logzero ......  
Installing collected packages: logzero
Successfully installed logzero-1.5.0

# Show placement of install and version
tom@tom:~$ pip show logzero
Name: logzero
Version: 1.5.0
Summary: Robust and effective logging for Python 2 and 3
Home-page: https://github.com/metachris/logzero
Author: Chris Hager
Author-email: [email protected]
License: MIT license
Location: /home/tom/.pyenv/versions/3.7.3/lib/python3.7/site-packages
Requires: 
Required-by: 

So with test that you also can do,we see that pip install to python3.7/site-packages
pip will do this same procedure for all packages on PyPi.
Uninstall is:

tom@tom:~$ pip uninstall logzero
Uninstalling logzero-1.5.0:
  Would remove:
    /home/tom/.pyenv/versions/3.7.3/lib/python3.7/site-packages/logzero-1.5.0.dist-info/*
    /home/tom/.pyenv/versions/3.7.3/lib/python3.7/site-packages/logzero/*
Proceed (y/n)? y
  Successfully uninstalled logzero-1.5.0

This it how is should work when use Python 3.7.3,it should be similar for you just that you use python3 and pip3.

Quote:would like to avoid venv as much as possible. It reminds me too much of wine 'bottles' .

venv is now build into Python and very easy to use,of course do not need to use it, but can be useful for some project.
Demo:

# Make
tom@tom:~$ python -m venv my_env
tom@tom:~$ cd my_env

# Activate
tom@tom:~/my_env$ source bin/activate

# Test pip,see not that it point it my_env folder
(my_env) tom@tom:~/my_env$ pip -V
pip 19.0.3 from /home/tom/my_env/lib/python3.7/site-packages/pip (python 3.7)

# Install,will now install to my_env folder which is like a new python install
(my_env) tom@tom:~/my_env$ pip install logzero
Collecting logzero .....  
Installing collected packages: logzero
Successfully installed logzero-1.5.0

Its not actually a problem, as just trying to wrap my head around the packaging management.

Have to check Debian with pip -V. Right now in Win7 where pip -v points to anaconda's site-packages, which is apparently where it should be.