to get a list of pip packages that were installed

[Skaperen]

i have most files from a previous Ubuntu Linux system that had a few pips installed, saved on a spare disk. i'd like to extract a list of which pip packages that were install. anyone know how, like where pip keeps its list?

[Larz60+]

Since the pip you used on the other system is not available, there won't be any history, unless you can bring up the old system.

If you used a virtual system, there will be a list of installed packages available in your virtual environment path tree
under venv --> lib64 --> python.3.10 --> site-packages (assume venv name is venv, replace with one assigned, also lib64 is lib if not 64bit, and proper python version)

This is also available if no virtual environment was used, under the installed python version path, but will contain all loaded packages. Probably a lot more than what you're looking for.

[Skaperen]

there are a bunch of missing files. i tried to run it in chroot but python can't run due to system libs missing. how does pip know what is installed? is that stored on pypi?

edit:

right now everything runs as root. it's a fresh install of Ubuntu 20.04 that i am customizing. is venv safe that way?

[Skaperen]

it looks like /usr/local/bin/python3.6/dist-packages is where to look (on the old system). but it looks like i have 94 packages installed on my new system, already. it looks like the old system had 117. i thought it had only about 20. could be dependencies. i have only installed one pip package, myself (netifaces 2 days ago).

i wish package managers would keep a dependency list with what other packages every package depends on (OpenBSD does this much) plus a list that each administrator (or user if this is allowed by local policy) "depends on" (the top dependency layer). having both the Ubuntu repository (that i have a complete list of wile paths each package installs) and PyPi repository (that i want a complete list of) together could make that tricky.

now, i am curious how those other 93 package got installed, some of them i have no idea what they do.

[Larz60+]

many packages install others that are needed dependencies, so they tend to add up. Especially if virtual environments are not used. I always create a separate virtual environment for each project and include a requirements.text file which I update every time I add a new package ( pip freeze > requirements.txt ). This tends to keep the repository slim, and is a great help if a new python is installed (I use pyenv (see snippsats video: ) to bring the environment up to date.
After new python install, use pip install -r requirements.txt.

[snippsat]

now, i am curious how those other 93 package got installed, some of them i have no idea what they do.

As mention so can one pip install have many sub dependencies that get installed.
93 is not a lot at all,i did help someone with install of a machine learning project this one project did install 100 package's.
pip list gives packages without version as freeze do.
Not recommend to move over all for OS install,a least pip list will install new packages and not many old as freeze(make more sense if done in virtual environment)

pip list > requirements.txt
# New OS or virtual enviroment
pip install -r requirements.txt

If want have full control over dependencies look a Poetry
It's a great tool for Python packaging and dependency management.
Will give tree of what eg Request has as sub dependencies.
Quick demo.

G:\div_code
λ poetry --version
Poetry version 1.1.13

G:\div_code
λ poetry new request_test
Created package request_test in request_test

G:\div_code
λ cd request_test\

# Install Requests
G:\div_code\request_test
λ poetry add requests
Using version ^2.27.1 for requests
.....

# Show dependencies that installed and needed bye Requests 
G:\div_code\request_test
λ poetry show requests
name         : requests
version      : 2.27.1
description  : Python HTTP for Humans.

dependencies
 - certifi >=2017.4.17
 - charset-normalizer >=2.0.0,<2.1.0
 - idna >=2.5,<4
 - urllib3 >=1.21.1,<1.27

# Show in tree view,pytest is a default install
G:\div_code\request_test
λ poetry show --tree
pytest 5.4.3 pytest: simple powerful testing with Python
├── atomicwrites >=1.0
├── attrs >=17.4.0
├── colorama *
├── more-itertools >=4.0.0
├── packaging *
│   └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
├── pluggy >=0.12,<1.0
├── py >=1.5.0
└── wcwidth *
requests 2.27.1 Python HTTP for Humans.
├── certifi >=2017.4.17
├── charset-normalizer >=2.0.0,<2.1.0
├── idna >=2.5,<4
└── urllib3 >=1.21.1,<1.27

[Skaperen]

it would be nice to have a dependencies list, one line per package, the package name at the start of the line, and all dependency packages follow on the line, delimited by some character or space, perhaps tab or comma.

[Larz60+]

You can get the dependencies from PyPi in the json file they produce.
Here's a simple program I wrote last night that does that for one module.
It was quick, so i wrote it for just one module. You can change it to accept package name from stdin,
or better yet from a file. I'll bet there's an index file somewhere on PyPi that you could use, or scrape one by listing all pages.
At any rate, here's the code, it is what it is:

import requests
import json
from pathlib import Path
import sys
 
 
def get_dependencies(package_name):
    pkginfo = {}
 
    url = f"https://pypi.org/pypi/{package_name}/json"
    response = requests.get(url)
    if response.status_code == 200:
        pkginfo = json.loads(response.content)
    else:
        print(f"Unable to fetch url: {url}")
        sys.exit(-1)
    print(f"\nOther packages: {pkginfo['info']['requires_dist']}")
    print(f"python versions: {pkginfo['info']['requires_python']}")
 
 
def main():
    get_dependencies('pyflowchart')
 
if __name__ == '__main__':
    main()

returns:

Output:
Other packages: ['astunparse', 'chardet']
python versions: >=3.6

To get the full list, follow the tree as each dependencies may also have some dependencies.

[snippsat]

i wish package managers would keep a dependency list with what other packages every package depends on

poetry dos this,i give a demo before in Thread.
Can do one more.

G:\div_code
λ poetry --version
Poetry version 1.1.13

# Make environment 
G:\div_code
λ poetry new pan_env
Created package pan_env in pan_env
G:\div_code
λ cd pan_env\

# Install pandas 
G:\div_code\pan_env
λ poetry add pandas
Creating virtualenv pan-env-G1a7vIpL-py3.10 in C:\Users\Tom\AppData\Local\pypoetry\Cache\virtualenvs
Using version ^1.4.3 for pandas

Updating dependencies
Resolving dependencies... (33.9s)

Writing lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.9)
  • Installing six (1.16.0)
  • Installing atomicwrites (1.4.0)
  • Installing attrs (21.4.0)
  • Installing colorama (0.4.5)
  • Installing more-itertools (8.13.0)
  • Installing numpy (1.23.0)
  • Installing packaging (21.3)
  • Installing pluggy (0.13.1)
  • Installing py (1.11.0)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2022.1)
  • Installing wcwidth (0.2.5)
  • Installing pandas (1.4.3)
  • Installing pytest (5.4.3)

Now if using --tree will create a dependency tree.

G:\div_code\pan_env
λ poetry show --tree
pandas 1.4.3 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.21.0
├── python-dateutil >=2.8.1
│   └── six >=1.5
└── pytz >=2020.1
pytest 5.4.3 pytest: simple powerful testing with Python
├── atomicwrites >=1.0
├── attrs >=17.4.0
├── colorama *
├── more-itertools >=4.0.0
├── packaging *
│   └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
├── pluggy >=0.12,<1.0
├── py >=1.5.0
└── wcwidth *  

So we see that Pandas depend on numpy, python-dateutil(which need six), and pytz.

[Skaperen]

can it make a flat list of dependencies?