how to get a full list of pypi packages

[Skaperen]

how can i get a full list of pypi packages?

[snippsat]

There is a API for this PyPIJSON.
Eg:

try:
    import xmlrpclib
except ImportError:
    import xmlrpc.client as xmlrpclib

client = xmlrpclib.ServerProxy('https://pypi.python.org/pypi')
# get a list of package names
packages = client.list_packages()
print(packages[:5])

Output:
['pycovjson', 'django-devserver', 'sl_pip', 'rio', 'django-supermigrate']

This do list all with description Index of Packages

[nilamo]

pip search *

Careful, you'll get A LOT of results.

[Skaperen]

pip search *

Careful, you'll get A LOT of results.

i get this:

Output:
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/search.py", line 43, in run
    pypi_hits = self.search(query, options)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/search.py", line 60, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/lib/python2.7/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 767, in request
    return self.parse_response(response.raw)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1493, in parse_response
    return u.close()
  File "/usr/lib/python2.7/xmlrpclib.py", line 800, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<class 'requests.exceptions.ConnectionError'>:('Connection aborted.', error(104, 'Connection reset by peer'))">

so it looks like the server dropped the connection.  maybe the send buffer overflowed.

[wavic]

pip search "*"
I suggest you to pipe it to less :D

[Skaperen]

looks like i need to code in a retry.  sometimes it works and sometimes it fails no matter where i pipe it to, but piping it to vbuf seems to work well.  i got a 5942 line list now.

[wavic]

looks like i need to code in a retry. sometimes it works and sometimes it fails no matter where i pipe it to, but piping it to vbuf seems to work well. i got a 5942 line list now.

I piped it only to wc -l to count the output lines. Why do you need this :)

[Skaperen]

Why do you need this :)

i want to build a local search that works in a different way.  it will display packages before and after the ones that match.  i might also do a "close spelling" match (e.g. "fobar" == "foobar", etc).

[wavic]

I would look at the source code which provide the search functionality of pip and just modify it a bit.

[Larz60+]

snippsat's method works.
This will get you a list of packages.
if you take each entry in that list you can get additional information

1. get a list of releases with releases = self.client.package_releases(item)
2. get package URL with release_urls(package_name, version) - Actually gets a lot more including MD5, python version, etc.
3. From the release list, get detailed information on that release with:
    a. release_data(package_name, version)
4. You can also get change log info and more

see: https://wiki.python.org/moin/PyPIXmlRpc?...PyPiXmlRpc
Sounds like something (with a GUI wrapper) that should go on the Code Idea list

Here's Snippsat's code modified to get additional information

import json
import HasInternet
try:
    import xmlrpclib
except ImportError:
    import xmlrpc.client as xmlrpclib
from collections import defaultdict
import time


class GetPyPi:
    def __init__(self):
        self.use_pkg_file = False
        self.packages = defaultdict(lambda: defaultdict(list))
        self.pkg_filename = 'packages.json'
        self.json_name = None
        self.web_available = False
        if HasInternet.has_connection():
            self.web_available = True
        if self.web_available:
            self.client = xmlrpclib.ServerProxy('https://pypi.python.org/pypi')

    def try_package_release(self):
        if self.web_available:
            allpackages = None
            print('updating package list')
            if self.use_pkg_file:
                with open(self.pkg_filename, 'r') as f:
                    j = f.read()
                    allpackages = json.loads(j)
            else:
                allpackages = self.client.list_packages()
                allpackages.sort()
                with open(self.pkg_filename, 'w') as f:
                    j = json.dumps(allpackages)
                    f.write(j)
            for n, item in enumerate(allpackages):
                print('Fetching release data for: {}'.format(item))
                releases = self.client.package_releases(item)
                for release in releases:
                    time.sleep(.2)
                    release_data = self.client.release_data(item, release)
                    print('release_data: {}'.format(release_data))
        else:
            print('Please check internet connection')


if __name__ == '__main__':
    gpp = GetPyPi()
    gpp.try_package_release()

You also need the HasInternet.py module:

import subprocess


def has_connection():
    website = "google.com"
    ping = subprocess.Popen(["ping", website], stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE)
    out, error = ping.communicate()
    if 'Sent = 4, Received = 4' in out.decode('utf-8'):
        return True
    return False

if __name__ == '__main__':
    print(has_connection())