downloading source for every version

downloading source for every version - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Forum & Off Topic (https://python-forum.io/forum-23.html)
+--- Forum: Bar (https://python-forum.io/forum-27.html)
+--- Thread: downloading source for every version (/thread-18908.html)

Pages: 1 2

downloading source for every version - Skaperen - Jun-05-2019

downloading source for every version of Python from www.python.org is such a bitch. they have a mix of .tar.bz2 and .tar.xz and i didn't see .tar.gz anywhere which i was sure they'd have. i think i need to do a brute-force download of the whole site.

RE: downloading source for every version - buran - Jun-05-2019

https://www.python.org/downloads/source/
*.tgz and *.tar.xz files...

RE: downloading source for every version - Skaperen - Jun-06-2019

i see no tar.xz files for many versions, such as 2.6 through 2.6.7 (is tar.bz2). then 2.6.8 is in tar.xz they need to do some compression conversion.

RE: downloading source for every version - buran - Jun-06-2019

OK, I didn't check every single one...

RE: downloading source for every version - Skaperen - Jun-06-2019

i was trying to download every single one. i ended up throwing together a script that tried everything and every way i could think of. after about half an hour i think i now have ever version. now i see the pattern that could have made it go faster, had i known it before.

RE: downloading source for every version - DeaD_EyE - Jun-06-2019

Output:In [15]: sorted(tgz, key=by_version)                                                                                                                                                         
Out[15]: 
['https://www.python.org/ftp/python/2.0.1/Python-2.0.1.tgz',
 'https://www.python.org/ftp/python/2.1.3/Python-2.1.3.tgz',
 'https://www.python.org/ftp/python/2.2/Python-2.2.tgz',
 'https://www.python.org/ftp/python/2.2.1/Python-2.2.1.tgz',
 'https://www.python.org/ftp/python/2.2.2/Python-2.2.2.tgz',
 'https://www.python.org/ftp/python/2.2.3/Python-2.2.3.tgz',
 'https://www.python.org/ftp/python/2.3/Python-2.3.tgz',
 'https://www.python.org/ftp/python/2.3.1/Python-2.3.1.tgz',
 'https://www.python.org/ftp/python/2.3.2/Python-2.3.2.tgz',
 'https://www.python.org/ftp/python/2.3.3/Python-2.3.3.tgz',
 'https://www.python.org/ftp/python/2.3.4/Python-2.3.4.tgz',
 'https://www.python.org/ftp/python/2.3.5/Python-2.3.5.tgz',
 'https://www.python.org/ftp/python/2.3.6/Python-2.3.6.tgz',
 'https://www.python.org/ftp/python/2.3.7/Python-2.3.7.tgz',
 'https://www.python.org/ftp/python/2.4/Python-2.4.tgz',
 'https://www.python.org/ftp/python/2.4.1/Python-2.4.1.tgz',
 'https://www.python.org/ftp/python/2.4.2/Python-2.4.2.tgz',
 'https://www.python.org/ftp/python/2.4.3/Python-2.4.3.tgz',
 'https://www.python.org/ftp/python/2.4.4/Python-2.4.4.tgz',
 'https://www.python.org/ftp/python/2.4.5/Python-2.4.5.tgz',
 'https://www.python.org/ftp/python/2.4.6/Python-2.4.6.tgz',
 'https://www.python.org/ftp/python/2.5/Python-2.5.tgz',
 'https://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz',
 'https://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz',
 'https://www.python.org/ftp/python/2.5.3/Python-2.5.3.tgz',
 'https://www.python.org/ftp/python/2.5.4/Python-2.5.4.tgz',
 'https://www.python.org/ftp/python/2.5.5/Python-2.5.5.tgz',
 'https://www.python.org/ftp/python/2.5.6/Python-2.5.6.tgz',
 'https://www.python.org/ftp/python/2.6/Python-2.6.tgz',
 'https://www.python.org/ftp/python/2.6.1/Python-2.6.1.tgz',
 'https://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz',
 'https://www.python.org/ftp/python/2.6.3/Python-2.6.3.tgz',
 'https://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz',
 'https://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz',
 'https://www.python.org/ftp/python/2.6.6/Python-2.6.6.tgz',
 'https://www.python.org/ftp/python/2.6.7/Python-2.6.7.tgz',
 'https://www.python.org/ftp/python/2.6.8/Python-2.6.8.tgz',
 'https://www.python.org/ftp/python/2.6.9/Python-2.6.9.tgz',
 'https://www.python.org/ftp/python/2.7/Python-2.7.tgz',
 'https://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz',
 'https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz',
 'https://www.python.org/ftp/python/2.7.10/Python-2.7.10rc1.tgz',
 'https://www.python.org/ftp/python/2.7.11/Python-2.7.11rc1.tgz',
 'https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz',
 'https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz',
 'https://www.python.org/ftp/python/2.7.12/Python-2.7.12rc1.tgz',
 'https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz',
 'https://www.python.org/ftp/python/2.7.13/Python-2.7.13rc1.tgz',
 'https://www.python.org/ftp/python/2.7.14/Python-2.7.14rc1.tgz',
 'https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz',
 'https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz',
 'https://www.python.org/ftp/python/2.7.15/Python-2.7.15rc1.tgz',
 'https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz',
 'https://www.python.org/ftp/python/2.7.16/Python-2.7.16rc1.tgz',
 'https://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz',
 'https://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz',
 'https://www.python.org/ftp/python/2.7.4/Python-2.7.4.tgz',
 'https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz',
 'https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz',
 'https://www.python.org/ftp/python/2.7.7/Python-2.7.7rc1.tgz',
 'https://www.python.org/ftp/python/2.7.7/Python-2.7.7.tgz',
 'https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz',
 'https://www.python.org/ftp/python/2.7.9/Python-2.7.9rc1.tgz',
 'https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz',
 'https://www.python.org/ftp/python/3.0/Python-3.0.tgz',
 'https://www.python.org/ftp/python/3.0.1/Python-3.0.1.tgz',
 'https://www.python.org/ftp/python/3.1/Python-3.1.tgz',
 'https://www.python.org/ftp/python/3.1.1/Python-3.1.1.tgz',
 'https://www.python.org/ftp/python/3.1.2/Python-3.1.2.tgz',
 'https://www.python.org/ftp/python/3.1.3/Python-3.1.3.tgz',
 'https://www.python.org/ftp/python/3.1.4/Python-3.1.4.tgz',
 'https://www.python.org/ftp/python/3.1.5/Python-3.1.5.tgz',
 'https://www.python.org/ftp/python/3.2/Python-3.2.tgz',
 'https://www.python.org/ftp/python/3.2.1/Python-3.2.1.tgz',
 'https://www.python.org/ftp/python/3.2.2/Python-3.2.2.tgz',
 'https://www.python.org/ftp/python/3.2.3/Python-3.2.3.tgz',
 'https://www.python.org/ftp/python/3.2.4/Python-3.2.4.tgz',
 'https://www.python.org/ftp/python/3.2.5/Python-3.2.5.tgz',
 'https://www.python.org/ftp/python/3.2.6/Python-3.2.6.tgz',
 'https://www.python.org/ftp/python/3.2.6/Python-3.2.6rc1.tgz',
 'https://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz',
 'https://www.python.org/ftp/python/3.3.1/Python-3.3.1.tgz',
 'https://www.python.org/ftp/python/3.3.2/Python-3.3.2.tgz',
 'https://www.python.org/ftp/python/3.3.3/Python-3.3.3.tgz',
 'http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz',
 'https://www.python.org/ftp/python/3.3.5/Python-3.3.5rc1.tgz',
 'http://www.python.org/ftp/python/3.3.5/Python-3.3.5rc2.tgz',
 'http://www.python.org/ftp/python/3.3.5/Python-3.3.5.tgz',
 'http://www.python.org/ftp/python/3.3.5/Python-3.3.5rc1.tgz',
 'https://www.python.org/ftp/python/3.3.6/Python-3.3.6.tgz',
 'https://www.python.org/ftp/python/3.3.6/Python-3.3.6rc1.tgz',
 'https://www.python.org/ftp/python/3.3.7/Python-3.3.7rc1.tgz',
 'https://www.python.org/ftp/python/3.3.7/Python-3.3.7.tgz',
 'http://www.python.org/ftp/python/3.4.0/Python-3.4.0rc3.tgz',
 'https://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz',
 'https://www.python.org/ftp/python/3.4.1/Python-3.4.1rc1.tgz',
 'https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz',
 'https://www.python.org/ftp/python/3.4.10/Python-3.4.10.tgz',
 'https://www.python.org/ftp/python/3.4.10/Python-3.4.10rc1.tgz',
 'https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tgz',
 'https://www.python.org/ftp/python/3.4.2/Python-3.4.2rc1.tgz',
 'https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz',
 'https://www.python.org/ftp/python/3.4.3/Python-3.4.3rc1.tgz',
 'https://www.python.org/ftp/python/3.4.4/Python-3.4.4rc1.tgz',
 'https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tgz',
 'https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz',
 'https://www.python.org/ftp/python/3.4.5/Python-3.4.5rc1.tgz',
 'https://www.python.org/ftp/python/3.4.6/Python-3.4.6rc1.tgz',
 'https://www.python.org/ftp/python/3.4.6/Python-3.4.6.tgz',
 'https://www.python.org/ftp/python/3.4.7/Python-3.4.7rc1.tgz',
 'https://www.python.org/ftp/python/3.4.7/Python-3.4.7.tgz',
 'https://www.python.org/ftp/python/3.4.8/Python-3.4.8.tgz',
 'https://www.python.org/ftp/python/3.4.8/Python-3.4.8rc1.tgz',
 'https://www.python.org/ftp/python/3.4.9/Python-3.4.9.tgz',
 'https://www.python.org/ftp/python/3.4.9/Python-3.4.9rc1.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc3.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b2.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc1.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b4.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a1.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a2.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b3.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a4.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b1.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc4.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a3.tgz',
 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc2.tgz',
 'https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz',
 'https://www.python.org/ftp/python/3.5.1/Python-3.5.1rc1.tgz',
 'https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz',
 'https://www.python.org/ftp/python/3.5.2/Python-3.5.2rc1.tgz',
 'https://www.python.org/ftp/python/3.5.3/Python-3.5.3rc1.tgz',
 'https://www.python.org/ftp/python/3.5.3/Python-3.5.3.tgz',
 'https://www.python.org/ftp/python/3.5.4/Python-3.5.4rc1.tgz',
 'https://www.python.org/ftp/python/3.5.4/Python-3.5.4.tgz',
 'https://www.python.org/ftp/python/3.5.5/Python-3.5.5rc1.tgz',
 'https://www.python.org/ftp/python/3.5.5/Python-3.5.5.tgz',
 'https://www.python.org/ftp/python/3.5.6/Python-3.5.6rc1.tgz',
 'https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz',
 'https://www.python.org/ftp/python/3.5.7/Python-3.5.7.tgz',
 'https://www.python.org/ftp/python/3.5.7/Python-3.5.7rc1.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a1.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0rc2.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a4.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a3.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b3.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0rc1.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b1.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a2.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b4.tgz',
 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b2.tgz',
 'https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz',
 'https://www.python.org/ftp/python/3.6.1/Python-3.6.1rc1.tgz',
 'https://www.python.org/ftp/python/3.6.2/Python-3.6.2rc2.tgz',
 'https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz',
 'https://www.python.org/ftp/python/3.6.2/Python-3.6.2rc1.tgz',
 'https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz',
 'https://www.python.org/ftp/python/3.6.3/Python-3.6.3rc1.tgz',
 'https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz',
 'https://www.python.org/ftp/python/3.6.4/Python-3.6.4rc1.tgz',
 'https://www.python.org/ftp/python/3.6.5/Python-3.6.5rc1.tgz',
 'https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz',
 'https://www.python.org/ftp/python/3.6.6/Python-3.6.6rc1.tgz',
 'https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz',
 'https://www.python.org/ftp/python/3.6.7/Python-3.6.7rc2.tgz',
 'https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz',
 'https://www.python.org/ftp/python/3.6.7/Python-3.6.7rc1.tgz',
 'https://www.python.org/ftp/python/3.6.8/Python-3.6.8rc1.tgz',
 'https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a1.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b2.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b5.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a4.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a3.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a2.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0rc1.tgz',
 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b1.tgz',
 'https://www.python.org/ftp/python/3.7.1/Python-3.7.1rc1.tgz',
 'https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz',
 'https://www.python.org/ftp/python/3.7.1/Python-3.7.1rc2.tgz',
 'https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz',
 'https://www.python.org/ftp/python/3.7.2/Python-3.7.2rc1.tgz',
 'https://www.python.org/ftp/python/3.7.3/Python-3.7.3rc1.tgz',
 'https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz',
 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a1.tgz',
 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a3.tgz',
 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a2.tgz',
 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0b1.tgz',
 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a4.tgz']

RE: downloading source for every version - Skaperen - Jun-06-2019

that code doesn't seem to be working for me. but then, i'm running on 3.5.2.

Output:Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'tgz' is not defined

RE: downloading source for every version - DeaD_EyE - Jun-06-2019

I made this one minute before lunch. Just 20 lines code.
Here as an example, but it has dependencies: requests and requests_html

"""
This script list or download Python source files
It was requested here: https://python-forum.io/Thread-downloading-source-for-every-version

Dependencies: requests, requests_html
Task: Remove external dependencies
"""


import re
import sys
try:
    import requests
    import requests_html
except ImportError:
    print('Please install requests and requests_html.', file=sys.stdout)
    sys.exit(255)


def sorter(url):
    """
    This sorter is used to sort the
    python by version string.
    
    Early version have only two digits.
    """
    v1 = re.search(r'(\d+)\.(\d+)\.(\d+)', url)
    v2 = re.search(r'(\d+)\.(\d+)', url)
    if v1:
        return tuple(map(int, v1.groups()))
    elif v2:
        return tuple(map(int, [*v2.groups(), 0]))
    else:
        return (0,0,0)


def get_python_sources(filetype='tgz'):
    """
    Grab all links from python.org/downloads/source/
    Return the source file archives sorted as a list.
    """
    url = 'https://www.python.org/downloads/source/'
    with requests_html.HTMLSession() as session:
        req = session.get(url)
    links = [link for link in req.html.links if link.endswith(filetype)]
    return sorted(links, key=sorter)


def print_python_sources():
    """
    Print all python source links to stdout.
    """
    for link in get_python_sources():
        print(link)


def download_python_sources():
    """
    Download all python source files to current location.
    """
    for link in get_python_sources():
        file = link.split('/')[-1]
        # find a better way to get the file name from url
        # maybe urllib.parse?
        # yarl is an alternative with more abstraction
        print('Downloading', file)
        req = requests.get(link, stream=True)
        with open(file, 'wb') as fd:
            for chunk in req.iter_content(chunk_size=64*1024**1):
                # what is the ideal chunk_size?
                # how to detect the best chunk_size?
                fd.write(chunk)


def help():
    print('python3', sys.argv[0], '<list|download>')
    sys.exit(1)


if __name__ == '__main__':
    if len(sys.argv) != 2:
        help()
    cmd = sys.argv[1].lower()
    if cmd not in ('download', 'list'):
        help()        
    try:
        if cmd == 'list':
            print_python_sources()
        elif cmd == 'download':
            download_python_sources()
    except Exception as e:
        # catching all exceptions here, the're bubbleing up
        # extend this to handle different cases
        print('Error:', e)

RE: downloading source for every version - Skaperen - Jun-07-2019

nice, but i already got every version. in fact i got 2 of every version. they had tar.bz2 and at one point switched to tar.xz while leaving the older ones as tar.bz2. they also have everything in tgz name pattern, but i originally was trying tar.gz because that is the common standard. i ran a script that tried everything based on a few patterns i did see. it also tried at least 8 different extensions for each version.

RE: downloading source for every version - Gribouillis - Jun-07-2019

DeaD_EyE Wrote:print('Please install requests and requests_html.', file=sys.stdout)

@DeaD_EyE Have you considered replacing this print with

import subprocess
import sys
subprocess.call([sys.executable, '-m', 'pip', 'install', '--user', 'requests', 'requests_html'])
import requests
import requests_html