Paths

[millpond]

Lets start at the beginning.

Awhile back I started taking an interest in Python, and began to build a Python37 system in C:\Python37.

Then I had heard of Anaconda(3.7), and installed it also. I had thought at the time Anaconda was simply an extension of C:\python37.

The result was pip was first installing into C/Python37, and then into the Anaconda.

To complicate matters, they were both sharing the same --users directory under ROAMING, \Python37.

However, there were no apparent problems caused by this. I simply started installing and updating everything with --users.
This is not a particular problem as I have not started any projects which might be versison specific.

But.. Problems came. The first time it was the result of a hibernation program which BSODd and messed up a bunch of files in the shared --users dir.
But I was able to track down the errors, and reinstall the dozen or so affected modules. Out of around 5000.
Everything resumed to working again.

I renamed pip and Python.exe so as to clearly differentiate the two in my path statement. They were coexisting fine.

Then a second problem occurred. I had left the machine on during a hot day, and turned the fan off when going out for the day. On arrival the machine was quite frozen
and required a cold reboot. Problems. Soft errors on about a dozen problems and a low level tester showed 100% OK. But *something* was messing with the Python37 install as I
found when switching directories around. I copied over a mirror install (albeint with less modules) from another machine, and all seems to be working.
BUT...

Somewhere along the line of testing I moved the --users site dir (Roaming\Python37\site-packages ) out of the way to see if the error was there. It
persisted, and then I moved it back.

HOWEVER...

It had created a *NEW* --user dir in ROAMING\Python\Python37\site-packages, and set the path to it on BOTH Anaconda and Default python installs.
It also changed the directory structure on the new users directory. Scripts, libs, that sort of thing.
I no longer had access to thousands of installed modules!

The error I was encountering before I switched things around was a null bytes error, but absolutely nothing was pointing to an aberrant file. Pip would ownload the install file, write it out to /TEMP but never get further. I reinstalled pip, setuptools, pkginfo, and anything similar I could find.

The *problem* is that I would like to get my old --user directory back, but

import sys
sys.path.append("C:\\Users\\Administrator\\AppData\\Roaming\\Python37\\site-packages")

Only seems to work in a python shell, and not at all in 'Python -c' command line mode.

I would like to be able to manipulate my python path in a permanent manner.

I am a bit leery about using an environment variable as the space for them is about used up, with all the packages and tools I have on this machine.
I have symlinked a bunch of paths just to fit everything!

So: to simplify -

I need a means of changing PYTHONPATH thats 'sticky'.

I also need guidance for useful logging. Adding a logging file to pip only told me to check the logs at the point of failure I was experiencing, with no reference to the troublesome module causing the NULL BYTE error.

[Gribouillis]

I would like to be able to manipulate my python path in a permanent manner.

Create a file

Output:
ROAMING\Python\Python37\site-packages\usercustomize.py

In this file, write these lines

import site
site.addsitedir("C:\\Users\\Administrator\\AppData\\Roaming\\Python37\\site-packages")

In this file, you can also manipulate sys.path if you want. Don't use the environment variable PYTHONPATH when you have several pythons.

You said you had a NULL byte error while installing a module with pip. What is the name of the module?

[snippsat]

It had created a *NEW* --user dir in ROAMING\Python\Python37\site-packages, and set the path to it on BOTH Anaconda and Default python installs.

No and no no.
Do not mix python(python.org) with Anaconda,they are stand alone and should have nothing to with each other.
Anaconda need to have (base)(newer version always) environment active,then will python and pip always use Anaconda.
Anaconda has a prompt that activate Anaconda environment,i use cmder so i do activate environment manually.

(base) C:\Users\Tom>

See (base),now will use Anaconda.

(base) C:\Users\Tom>python -V
Python 3.7.3

(base) C:\Users\Tom>python
Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Ana
conda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

(base) C:\Users\Tom>pip -V
pip 20.0.2 from G:\Anaconda3\lib\site-packages\pip (python 3.7)

If i just use cmd.
Use my main python 3.8 version set in Environment Variables Path.
The only change in Path done.

C:\Users\Tom>python
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\Users\Tom>pip -V
pip 20.1.1 from c:\python38\lib\site-packages\pip (python 3.8)

Also all other python version can be access with py.
py.exe get installed to Windows folder when install Python,so it's always in Path.

G:\div_code
λ py -3.7 -V
Python 3.7.3

G:\div_code
λ py -3.6 -V
Python 3.6.4

G:\div_code
λ py -2.7 -V
Python 2.7.9                                                                                             
                                                                                                                
G:\div_code                                                                                                            
λ # Using pip to install to 2.7                                                                            
λ py -2.7 -m pip install logzero                                                                               
Requirement already satisfied: logzero in c:\python27\lib\site-packages

[millpond]

I would like to be able to manipulate my python path in a permanent manner.

Create a file

Output:
ROAMING\Python\Python37\site-packages\usercustomize.py

In this file, write these lines

import site
site.addsitedir("C:\\Users\\Administrator\\AppData\\Roaming\\Python37\\site-packages")

In this file, you can also manipulate sys.path if you want. Don't use the environment variable PYTHONPATH when you have several pythons.

You said you had a NULL byte error while installing a module with pip. What is the name of the module?

YES!!! It works!!!!
Forgive the enthusiam. I have spent quite a while searching google and my etexts for this solution.
And many, many thanks.

The only issue I had at first was broken symplinks, which were easily fixed.
I have it now updating/installing about a dozen files and clearly seeing a "requirement already up-to-date" in *both* dirs in the the /user/Roaming dir.

Now... both python 3.7 and Anaconda 3.7 installs accessing the 2 --user dirs as can be seen from calling sys.path on each.

Two more issues have popped up while writing this....

The first was that .pth files in the added dir suddenly needed line returns after the first statement semicolons.

The second is unresolved however, and occurs immediately after the module is downloaded, either as primary or a dependency. and is of the form:

    ERROR: Command errored out with exit status 1:
     command: 'c:\python37\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Temp\\pip-install-pmkv08uw\\bokeh\\setup.py'"'"'; __file__='"'"'C:\\Temp\\pip-install-pmkv08uw\\bokeh\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Temp\pip-pip-egg-info-63npni7_'
         cwd: C:\Temp\pip-install-pmkv08uw\bokeh\
    Complete output (28 lines):
    Traceback (most recent call last):
      File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2866, in get_entry_map
        ep_map = self._ep_map
      File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2824, in __getattr__
        raise AttributeError(attr)
    AttributeError: _ep_map
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Temp\pip-install-pmkv08uw\bokeh\setup.py", line 140, in <module>
        cmdclass=get_cmdclass()
      File "c:\python37\lib\site-packages\setuptools\__init__.py", line 164, in setup
        _install_setup_requires(attrs)
      File "c:\python37\lib\site-packages\setuptools\__init__.py", line 154, in _install_setup_requires
        dist = MinimalDistribution(attrs)
      File "c:\python37\lib\site-packages\setuptools\__init__.py", line 146, in __init__
        distutils.core.Distribution.__init__(self, filtered)
      File "c:\python37\lib\site-packages\setuptools\dist.py", line 427, in __init__
        for ep in pkg_resources.iter_entry_points('distutils.setup_keywords'):
      File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 655, in <genexpr>
        for entry in dist.get_entry_map(group).values()
      File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2869, in get_entry_map
        self._get_metadata('entry_points.txt'), self
      File "c:\python37\lib\site-packages\pkg_resources\__init__.py", line 2549, in parse_map
        raise ValueError("Entry points must be listed in groups")
    ValueError: Entry points must be listed in groups
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This errors occurs in some but not all module installs/upgrades. Typical examples are
bokeh
nmrglue
lmfit
ai
absl_py
ipdb

The install script is a .bat file:

for /f %%i in (requirements.txt) do c:\python37\python -m pip install %%i -c constraints.txt --no-cache --user --no-color --upgrade --upgrade-strategy eager  --index-url=https://pypi.python.org/simple   

I have also tried:

for /f %%i in (requirements.txt) do c:\python37\python -m pip install %%i -c constraints.txt  --user --no-color  --no-deps  --index-url=https://pypi.python.org/simple   

With installed files showing up as not needing install, and uninstalled (or moved off pythonpath) modules giving the above error.

[Gribouillis]

You could perhaps try to reinstall or upgrade pkg_resources.

Also listen to Snippsat's advice about Anaconda, which I don't use. Normally one does not mix two installs of python.

[millpond]

@snippsat
The problem I have with Anaconda is that conda refuses to initialize and spits out my (large) environment as a cryptic error message.

The two 3.7 versions have been coexisting for quite a while until a recent system crash putzed up some files which were readily reinstalled. I do currently have an occasional _ep_map error (discussed above) which is common to both installs, even when the original user dir was moved off-path.

I am in the process of learning Python, and as I intend in the near future to vacation in a region which is totally off-grid - I need all my possible tools installed beforehand.

I have no current Python projects other than simple scripts, so version conflicts are not an issue - yet.

I am looking forward to moving to Python 3.8 and would really, really like to use my 3.7 modules: If possible. If not - I might wait till Anaconda goes to 3.8 and hopefully conda will work with that.

I do this with Perl. I have 5.10 site directories living with a 5.20 install - just at the low end of the perlpath. Though having CPAN mirrored also helps.

Oddly, while I have a py 1.9 module, I do not have a py.exe Reinstalling did not help.
This would seem to be my best option at the moment if I want to use 3.8

The link to cmder is much appreciated. I really need a better option to the cmd window!

---

You could perhaps try to reinstall or upgrade pkg_resources.

Also listen to Snippsat's advice about Anaconda, which I don't use. Normally one does not mix two installs of python.

As I mention (before I read your reply) - I do intend to migrate to Anaconda 3.8 as soon as it is released, as conda is already problematic here. Py looks like a possible option is I can figure out how to get the executable installed. I just want to avoid env.

pkg_resources isnt upgradable as it isnt a real module. --force-reinstall pip and/or pkginfo might help (in the morning!).

It seems like there might be a type(ing) issue going on.
Is there any way to eliminate that problem?
I'll be happy to hack out any annoying part of the system.

[Gribouillis]

I think there are useful information to read in this page Installing Packages. For example you could try the line

Output:
python -m pip install --upgrade pip setuptools wheel

(or perhaps it's 'py' on your system...)

[snippsat]

I am looking forward to moving to Python 3.8 and would really, really like to use my 3.7 modules: If possible.

Follow this install setup for Python 3.8.5.
If want to access any other version(python.org versions) as eg 3.7 use py py -3.7 some_code.py
Make it simple have this Python 3.8 as main version in Path and nothing else pointing to any Python version.

Restart pc.
Test python and pip in cmd as in link.

---

The problem I have with Anaconda is that conda refuses to initialize and spits out my (large) environment as a cryptic error message.

You most activate base,eg bye using Anaconda Prompt.
Here how i do manually as i use cmder.

# cd to Scripts folder
G:\
λ cd Anaconda3\Scripts\

# Activate base
G:\Anaconda3\Scripts
λ activate base

# See now that there is (base)
(base) G:\Anaconda3\Scripts
λ cd ..

(base) G:\Anaconda3
λ cd ..

# Now from any folder will pip and conda always use Anaconda when (base) is active
(base) G:\
λ pip -V
pip 20.0.2 from G:\Anaconda3\lib\site-packages\pip (python 3.7)

(base) G:\
λ conda -V
conda 4.8.3

Go out of base environment.

(base) G:\
λ deactivate
deactivate

# Now will use Python 3.8,as this is main version set in Path 
G:\
λ python -V
Python 3.8.3

G:\
λ pip -V
pip 20.1.1 from c:\python38\lib\site-packages\pip (python 3.8)

[millpond]

Here conda cannot seem to find the cli module, although it is clearly here. Will not activate on either Win7 machine.
One reason to maybe wait for the next version.

Is it safe to assume that Python3.8 will not putz with other installs or site-packages?
It seems to have a different install method, and I certainly do not want it under Windows, or will tolerate long paths (without symlinking to abbreviated versions).

If it will not delete anything, hopefully py will work fine on it.And be able to use 3.7, 3.5, and 2.7 installs.

BTW - Cmder works fine here. Powershell derived.... Finally figured out what the lambda character is all about!

Addendum:
Tried conda init, and activate in Cmder, and it finally seems to work:

C:\Programs\Anaconda3\Scripts
(base) λ pip -V
pip 20.1.1 from c:\programs\anaconda3\lib\site-packages\pip (python 3.7)

C:\Programs\Anaconda3\Scripts
(base) λ conda -V
conda 4.6.14
(base) λ python -V
Python 3.7.3

I now get the (base) prompt in Cmder. Conda init, and Conda activate no longer give errors.
It looks like it is working!

---

[snippsat]

Is it safe to assume that Python3.8 will not putz with other installs or site-packages?

Yes.

It seems to have a different install method, and I certainly do not want it under Windows, or will tolerate long paths (without symlinking to abbreviated versions).

It's the same as previous version.

If it will not delete anything, hopefully py will work fine on it.And be able to use 3.7, 3.5, and 2.7 installs.

It will not delete anything Windows only see Python trough Environment Variables Path.
So have one main version eg Python 3.8 in Environment Variables Path.
Then use py for other version or activate base for Anaconda.

I now get the (base) prompt in Cmder. Conda init, and Conda activate no longer give errors.
It looks like it is working!

Looks okay

Here a run with loguru look that pip -V always tell you which version you install to.

# Python 3.8
G:\
λ pip -V
pip 20.1.1 from c:\python38\lib\site-packages\pip (python 3.8)

# Install 3.8
G:\
λ pip install loguru
Collecting loguru .....     
Successfully installed colorama-0.4.3 loguru-0.5.1 win32-setctime-1.0.1

# Python 3.7
G:\
λ py -3.7 -m pip -V
pip 20.0.2 from C:\Python37\lib\site-packages\pip (python 3.7)

# Install Python 3.7
G:\
λ py -3.7 -m pip install loguru
Requirement already satisfied: loguru in c:\python37\lib\site-packages (0.4.1)

# To Anaconda activate first
G:\Anaconda3\Scripts
λ activate.bat

# To python 3.7 Anaconda
(base) G:\Anaconda3\Scripts
λ pip -V
pip 20.0.2 from G:\Anaconda3\lib\site-packages\pip (python 3.7)

(base) G:\Anaconda3\Scripts
λ pip install loguru
Collecting loguru ..... 
Successfully installed loguru-0.5.1 win32-setctime-1.0.1