networkx package is not visible in singularity container image

[erdemath]

I have singularity-ce version 3.10.2 on my ubuntu 20.04. I created my own Python library in a self built singularity container. I installed networkx package
through pip in the library with no error. However, I do not see networkx package when I do 'pip list'. Also, I cannot run my python code through the container
because the code says;

Error:
import networkx as nx
ModuleNotFoundError: No module named 'networkx'

[Larz60+]

see: https://pypi.org/project/networkx/ for installation instructions
There are two ways to install:

• package alone pip install networkx
  
• with all dependencies: pip install networkx[all]
  

make sure pip and python are for same version:
current pip: pip -V
current python: python -V

[erdemath]

Yes, this how I installed. But, I do not see it inside the container if I shell into the container as a user. I only see it if I shell into the container as the root. The main goal is to run python in HPC environment wherein I do not have sudo privileges.

[ibreeden]

Perhaps try pip3 install networkx.

[erdemath]

This is how I performed the installation, and it did not work either. Installation in container for some python packages is not as in the actual linux environment. I was suggested to craft some container recipe into a def file for clear installation.

[Larz60+]

I took a quick look at singularity, and couldn't find much about python.
I see a hint that it might be using anaconda, in this case, the package install command is 'conda'

From your 'container' (which I think is a virtual environment), can you access a command line (as in terminal, or 'cmd' in windows)?
if so, please execute commands:

• Python -V
  
• pip -V
  
• Show results in a new post.
  

also, privately run pip list from command line and make sure you see your container, etc.

[erdemath]

From your 'container' (which I think is a virtual environment), can you access a command line (as in terminal, or 'cmd' in windows)?

Yes! Also, yes container is a virtual environment.

All were just fixed today by creating custom Ubuntu container. We embedded python library with all the necessary packages into this container.
For those who are interested, here is the hierarchy

1 - Generate definition file by the command touch blahblah.def
2 - Describe the recipe for the desired container, by the followings
2.1 - apt-get update
2.2 - apt-get install python-pip
2.3 - apt-get install networkx==version
2.4 - ......
3 - sudo singularity build blahblah.sif blahblah.def

also, privately run pip list from command line and make sure you see your container, etc.

All the packages are in the container.

[Larz60+]

please execute commands:

• Python -V
  
• pip -V
  
• Show results in a new post.
  
• Make sure singularity Container environment is activated if you find wrong pip is being used.
  

also, privately run pip list from command line and make sure you see your container, etc.
I think the pip that you have in singularity container is not the one being run.
Above simple test will show if this is true.

[erdemath]

[*] First, here is the complete recipe for my .def file
BootStrap: library
From: ubuntu:20.04

%post
export DEBIAN_FRONTEND=noninteractive
apt-get -y update
apt-get -y install build-essential
apt-get -y install zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev git wget libbz2-dev
apt-get -y install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt-get -y install python3.10
apt-get -y install python3-pip
apt-get -y install pkg-config
apt-get -y install libhdf5-dev
apt-get -y install python-h5py-doc
apt-get -y install gfortran
apt-get -y install python3-scipy
apt-get -y install texlive-latex-extra --no-install-recommends
apt-get -y install python3-sklearn python3-sklearn-lib
pip3 install --no-binary=h5py h5py==2.10.0
pip3 install networkx
pip3 install neo
pip3 install pylops
pip3 install elephant[extras]
pip3 install -U scikit-learn
pip3 install pandas
pip3 install PyWavelets

%environment
export LC_ALL=C

[*] Python3 -V
Python 3.8.10

[*] pip -V
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

[*] Lastly, when I type pip list, I see all the packages that I installed by pip3.
In conclusion, installation is successful.

[Larz60+]

You see all of the packages you installed to python 3.8.10

[*] Python3 -V
Python 3.8.10

you install python3.10. without specifying subversion (like 3.10.7)

apt-get -y install python3.10

This does not specify the specific subversion, so not sure what you get

I don't install python As above, I use pyenv, which allows selection of a particular version and subversion
so I can have both python 3.10.5, and 3.10.7 installed (and many more) but specifically select 3.10.5 as the version to use for one specific virtual environment and 3.10.7 for another, and the proper version of pip will follow. (If interested, please see: https://python-forum.io/thread-15441.html)

you then install pip3

apt-get -y install python3-pip

As you can see by:

[*] Python3 -V
Python 3.8.10

and

pip -V
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

the system pip and python are for python 3.8.10, not 3.10. and system pip for 3.8.10,
not for version 3.10.?

you can install pip to a specific version of python using:
python3.10.5 -m pip install package-name which will install package-name to version 3.10.5

It's complicated if not using pyenv. Easy if so.

Hope this is understandable.