how to get a full list of pypi packages

[Skaperen]

how can i get a full list of pypi packages?

[snippsat]

There is a API for this PyPIJSON.
Eg:

1 2 3 4 5 6 7 8 9

try:     import xmlrpclib except ImportError:     import xmlrpc.client as xmlrpclib   client = xmlrpclib.ServerProxy('https://pypi.python.org/pypi') # get a list of package names packages = client.list_packages() print(packages[:5])

Output:
['pycovjson', 'django-devserver', 'sl_pip', 'rio', 'django-supermigrate']

This do list all with description Index of Packages

[nilamo]

1	pip search *

Careful, you'll get A LOT of results.

[Skaperen]

1	pip search *

Careful, you'll get A LOT of results.

i get this:

Output:
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/search.py", line 43, in run
    pypi_hits = self.search(query, options)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/search.py", line 60, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/lib/python2.7/xmlrpclib.py", line 1243, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1602, in __request
    verbose=self.__verbose
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 767, in request
    return self.parse_response(response.raw)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1493, in parse_response
    return u.close()
  File "/usr/lib/python2.7/xmlrpclib.py", line 800, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<class 'requests.exceptions.ConnectionError'>:('Connection aborted.', error(104, 'Connection reset by peer'))">

so it looks like the server dropped the connection.  maybe the send buffer overflowed.

[wavic]

pip search "*"
I suggest you to pipe it to less :D

[Skaperen]

looks like i need to code in a retry.  sometimes it works and sometimes it fails no matter where i pipe it to, but piping it to vbuf seems to work well.  i got a 5942 line list now.

[wavic]

looks like i need to code in a retry. sometimes it works and sometimes it fails no matter where i pipe it to, but piping it to vbuf seems to work well. i got a 5942 line list now.

I piped it only to wc -l to count the output lines. Why do you need this :)

[Skaperen]

Why do you need this :)

i want to build a local search that works in a different way.  it will display packages before and after the ones that match.  i might also do a "close spelling" match (e.g. "fobar" == "foobar", etc).

[wavic]

I would look at the source code which provide the search functionality of pip and just modify it a bit.

[Larz60+]

snippsat's method works.
This will get you a list of packages.
if you take each entry in that list you can get additional information

1. get a list of releases with releases = self.client.package_releases(item)
2. get package URL with release_urls(package_name, version) - Actually gets a lot more including MD5, python version, etc.
3. From the release list, get detailed information on that release with:
    a. release_data(package_name, version)
4. You can also get change log info and more

see: https://wiki.python.org/moin/PyPIXmlRpc?...PyPiXmlRpc
Sounds like something (with a GUI wrapper) that should go on the Code Idea list

Here's Snippsat's code modified to get additional information

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

import json import HasInternet try:     import xmlrpclib except ImportError:     import xmlrpc.client as xmlrpclib from collections import defaultdict import time     class GetPyPi:     def __init__(self):         self.use_pkg_file = False         self.packages = defaultdict(lambda: defaultdict(list))         self.pkg_filename = 'packages.json'         self.json_name = None         self.web_available = False         if HasInternet.has_connection():             self.web_available = True         if self.web_available:             self.client = xmlrpclib.ServerProxy('https://pypi.python.org/pypi')       def try_package_release(self):         if self.web_available:             allpackages = None             print('updating package list')             if self.use_pkg_file:                 with open(self.pkg_filename, 'r') as f:                     j = f.read()                     allpackages = json.loads(j)             else:                 allpackages = self.client.list_packages()                 allpackages.sort()                 with open(self.pkg_filename, 'w') as f:                     j = json.dumps(allpackages)                     f.write(j)             for n, item in enumerate(allpackages):                 print('Fetching release data for: {}'.format(item))                 releases = self.client.package_releases(item)                 for release in releases:                     time.sleep(.2)                     release_data = self.client.release_data(item, release)                     print('release_data: {}'.format(release_data))         else:             print('Please check internet connection')     if __name__ == '__main__':     gpp = GetPyPi()     gpp.try_package_release()

You also need the HasInternet.py module:

1 2 3 4 5 6 7 8 9 10 11 12 13 14

import subprocess     def has_connection():     website = "google.com"     ping = subprocess.Popen(["ping", website], stdout=subprocess.PIPE,                             stderr=subprocess.PIPE)     out, error = ping.communicate()     if 'Sent = 4, Received = 4' in out.decode('utf-8'):         return True     return False   if __name__ == '__main__':     print(has_connection())