Python Forum
downloading source for every version
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
downloading source for every version
#1
downloading source for every version of Python from www.python.org is such a bitch. they have a mix of .tar.bz2 and .tar.xz and i didn't see .tar.gz anywhere which i was sure they'd have. i think i need to do a brute-force download of the whole site.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
https://www.python.org/downloads/source/
*.tgz and *.tar.xz files...
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
i see no tar.xz files for many versions, such as 2.6 through 2.6.7 (is tar.bz2). then 2.6.8 is in tar.xz they need to do some compression conversion.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
OK, I didn't check every single one...
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#5
i was trying to download every single one. i ended up throwing together a script that tried everything and every way i could think of. after about half an hour i think i now have ever version. now i see the pattern that could have made it go faster, had i known it before.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#6
Output:
In [15]: sorted(tgz, key=by_version) Out[15]: ['https://www.python.org/ftp/python/2.0.1/Python-2.0.1.tgz', 'https://www.python.org/ftp/python/2.1.3/Python-2.1.3.tgz', 'https://www.python.org/ftp/python/2.2/Python-2.2.tgz', 'https://www.python.org/ftp/python/2.2.1/Python-2.2.1.tgz', 'https://www.python.org/ftp/python/2.2.2/Python-2.2.2.tgz', 'https://www.python.org/ftp/python/2.2.3/Python-2.2.3.tgz', 'https://www.python.org/ftp/python/2.3/Python-2.3.tgz', 'https://www.python.org/ftp/python/2.3.1/Python-2.3.1.tgz', 'https://www.python.org/ftp/python/2.3.2/Python-2.3.2.tgz', 'https://www.python.org/ftp/python/2.3.3/Python-2.3.3.tgz', 'https://www.python.org/ftp/python/2.3.4/Python-2.3.4.tgz', 'https://www.python.org/ftp/python/2.3.5/Python-2.3.5.tgz', 'https://www.python.org/ftp/python/2.3.6/Python-2.3.6.tgz', 'https://www.python.org/ftp/python/2.3.7/Python-2.3.7.tgz', 'https://www.python.org/ftp/python/2.4/Python-2.4.tgz', 'https://www.python.org/ftp/python/2.4.1/Python-2.4.1.tgz', 'https://www.python.org/ftp/python/2.4.2/Python-2.4.2.tgz', 'https://www.python.org/ftp/python/2.4.3/Python-2.4.3.tgz', 'https://www.python.org/ftp/python/2.4.4/Python-2.4.4.tgz', 'https://www.python.org/ftp/python/2.4.5/Python-2.4.5.tgz', 'https://www.python.org/ftp/python/2.4.6/Python-2.4.6.tgz', 'https://www.python.org/ftp/python/2.5/Python-2.5.tgz', 'https://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz', 'https://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz', 'https://www.python.org/ftp/python/2.5.3/Python-2.5.3.tgz', 'https://www.python.org/ftp/python/2.5.4/Python-2.5.4.tgz', 'https://www.python.org/ftp/python/2.5.5/Python-2.5.5.tgz', 'https://www.python.org/ftp/python/2.5.6/Python-2.5.6.tgz', 'https://www.python.org/ftp/python/2.6/Python-2.6.tgz', 'https://www.python.org/ftp/python/2.6.1/Python-2.6.1.tgz', 'https://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz', 'https://www.python.org/ftp/python/2.6.3/Python-2.6.3.tgz', 'https://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz', 'https://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz', 'https://www.python.org/ftp/python/2.6.6/Python-2.6.6.tgz', 'https://www.python.org/ftp/python/2.6.7/Python-2.6.7.tgz', 'https://www.python.org/ftp/python/2.6.8/Python-2.6.8.tgz', 'https://www.python.org/ftp/python/2.6.9/Python-2.6.9.tgz', 'https://www.python.org/ftp/python/2.7/Python-2.7.tgz', 'https://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz', 'https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz', 'https://www.python.org/ftp/python/2.7.10/Python-2.7.10rc1.tgz', 'https://www.python.org/ftp/python/2.7.11/Python-2.7.11rc1.tgz', 'https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz', 'https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz', 'https://www.python.org/ftp/python/2.7.12/Python-2.7.12rc1.tgz', 'https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz', 'https://www.python.org/ftp/python/2.7.13/Python-2.7.13rc1.tgz', 'https://www.python.org/ftp/python/2.7.14/Python-2.7.14rc1.tgz', 'https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz', 'https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz', 'https://www.python.org/ftp/python/2.7.15/Python-2.7.15rc1.tgz', 'https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz', 'https://www.python.org/ftp/python/2.7.16/Python-2.7.16rc1.tgz', 'https://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz', 'https://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz', 'https://www.python.org/ftp/python/2.7.4/Python-2.7.4.tgz', 'https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz', 'https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz', 'https://www.python.org/ftp/python/2.7.7/Python-2.7.7rc1.tgz', 'https://www.python.org/ftp/python/2.7.7/Python-2.7.7.tgz', 'https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz', 'https://www.python.org/ftp/python/2.7.9/Python-2.7.9rc1.tgz', 'https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz', 'https://www.python.org/ftp/python/3.0/Python-3.0.tgz', 'https://www.python.org/ftp/python/3.0.1/Python-3.0.1.tgz', 'https://www.python.org/ftp/python/3.1/Python-3.1.tgz', 'https://www.python.org/ftp/python/3.1.1/Python-3.1.1.tgz', 'https://www.python.org/ftp/python/3.1.2/Python-3.1.2.tgz', 'https://www.python.org/ftp/python/3.1.3/Python-3.1.3.tgz', 'https://www.python.org/ftp/python/3.1.4/Python-3.1.4.tgz', 'https://www.python.org/ftp/python/3.1.5/Python-3.1.5.tgz', 'https://www.python.org/ftp/python/3.2/Python-3.2.tgz', 'https://www.python.org/ftp/python/3.2.1/Python-3.2.1.tgz', 'https://www.python.org/ftp/python/3.2.2/Python-3.2.2.tgz', 'https://www.python.org/ftp/python/3.2.3/Python-3.2.3.tgz', 'https://www.python.org/ftp/python/3.2.4/Python-3.2.4.tgz', 'https://www.python.org/ftp/python/3.2.5/Python-3.2.5.tgz', 'https://www.python.org/ftp/python/3.2.6/Python-3.2.6.tgz', 'https://www.python.org/ftp/python/3.2.6/Python-3.2.6rc1.tgz', 'https://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz', 'https://www.python.org/ftp/python/3.3.1/Python-3.3.1.tgz', 'https://www.python.org/ftp/python/3.3.2/Python-3.3.2.tgz', 'https://www.python.org/ftp/python/3.3.3/Python-3.3.3.tgz', 'http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz', 'https://www.python.org/ftp/python/3.3.5/Python-3.3.5rc1.tgz', 'http://www.python.org/ftp/python/3.3.5/Python-3.3.5rc2.tgz', 'http://www.python.org/ftp/python/3.3.5/Python-3.3.5.tgz', 'http://www.python.org/ftp/python/3.3.5/Python-3.3.5rc1.tgz', 'https://www.python.org/ftp/python/3.3.6/Python-3.3.6.tgz', 'https://www.python.org/ftp/python/3.3.6/Python-3.3.6rc1.tgz', 'https://www.python.org/ftp/python/3.3.7/Python-3.3.7rc1.tgz', 'https://www.python.org/ftp/python/3.3.7/Python-3.3.7.tgz', 'http://www.python.org/ftp/python/3.4.0/Python-3.4.0rc3.tgz', 'https://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz', 'https://www.python.org/ftp/python/3.4.1/Python-3.4.1rc1.tgz', 'https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz', 'https://www.python.org/ftp/python/3.4.10/Python-3.4.10.tgz', 'https://www.python.org/ftp/python/3.4.10/Python-3.4.10rc1.tgz', 'https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tgz', 'https://www.python.org/ftp/python/3.4.2/Python-3.4.2rc1.tgz', 'https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz', 'https://www.python.org/ftp/python/3.4.3/Python-3.4.3rc1.tgz', 'https://www.python.org/ftp/python/3.4.4/Python-3.4.4rc1.tgz', 'https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tgz', 'https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz', 'https://www.python.org/ftp/python/3.4.5/Python-3.4.5rc1.tgz', 'https://www.python.org/ftp/python/3.4.6/Python-3.4.6rc1.tgz', 'https://www.python.org/ftp/python/3.4.6/Python-3.4.6.tgz', 'https://www.python.org/ftp/python/3.4.7/Python-3.4.7rc1.tgz', 'https://www.python.org/ftp/python/3.4.7/Python-3.4.7.tgz', 'https://www.python.org/ftp/python/3.4.8/Python-3.4.8.tgz', 'https://www.python.org/ftp/python/3.4.8/Python-3.4.8rc1.tgz', 'https://www.python.org/ftp/python/3.4.9/Python-3.4.9.tgz', 'https://www.python.org/ftp/python/3.4.9/Python-3.4.9rc1.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc3.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b2.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc1.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b4.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a1.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a2.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b3.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a4.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b1.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc4.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a3.tgz', 'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc2.tgz', 'https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz', 'https://www.python.org/ftp/python/3.5.1/Python-3.5.1rc1.tgz', 'https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz', 'https://www.python.org/ftp/python/3.5.2/Python-3.5.2rc1.tgz', 'https://www.python.org/ftp/python/3.5.3/Python-3.5.3rc1.tgz', 'https://www.python.org/ftp/python/3.5.3/Python-3.5.3.tgz', 'https://www.python.org/ftp/python/3.5.4/Python-3.5.4rc1.tgz', 'https://www.python.org/ftp/python/3.5.4/Python-3.5.4.tgz', 'https://www.python.org/ftp/python/3.5.5/Python-3.5.5rc1.tgz', 'https://www.python.org/ftp/python/3.5.5/Python-3.5.5.tgz', 'https://www.python.org/ftp/python/3.5.6/Python-3.5.6rc1.tgz', 'https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz', 'https://www.python.org/ftp/python/3.5.7/Python-3.5.7.tgz', 'https://www.python.org/ftp/python/3.5.7/Python-3.5.7rc1.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a1.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0rc2.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a4.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a3.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b3.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0rc1.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b1.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a2.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b4.tgz', 'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b2.tgz', 'https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz', 'https://www.python.org/ftp/python/3.6.1/Python-3.6.1rc1.tgz', 'https://www.python.org/ftp/python/3.6.2/Python-3.6.2rc2.tgz', 'https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz', 'https://www.python.org/ftp/python/3.6.2/Python-3.6.2rc1.tgz', 'https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz', 'https://www.python.org/ftp/python/3.6.3/Python-3.6.3rc1.tgz', 'https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz', 'https://www.python.org/ftp/python/3.6.4/Python-3.6.4rc1.tgz', 'https://www.python.org/ftp/python/3.6.5/Python-3.6.5rc1.tgz', 'https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz', 'https://www.python.org/ftp/python/3.6.6/Python-3.6.6rc1.tgz', 'https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz', 'https://www.python.org/ftp/python/3.6.7/Python-3.6.7rc2.tgz', 'https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz', 'https://www.python.org/ftp/python/3.6.7/Python-3.6.7rc1.tgz', 'https://www.python.org/ftp/python/3.6.8/Python-3.6.8rc1.tgz', 'https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a1.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b2.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b5.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a4.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a3.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a2.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0rc1.tgz', 'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b1.tgz', 'https://www.python.org/ftp/python/3.7.1/Python-3.7.1rc1.tgz', 'https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz', 'https://www.python.org/ftp/python/3.7.1/Python-3.7.1rc2.tgz', 'https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz', 'https://www.python.org/ftp/python/3.7.2/Python-3.7.2rc1.tgz', 'https://www.python.org/ftp/python/3.7.3/Python-3.7.3rc1.tgz', 'https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz', 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a1.tgz', 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a3.tgz', 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a2.tgz', 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0b1.tgz', 'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a4.tgz']
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#7
that code doesn't seem to be working for me. but then, i'm running on 3.5.2.

Output:
Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'tgz' is not defined
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#8
I made this one minute before lunch. Just 20 lines code.
Here as an example, but it has dependencies: requests and requests_html

"""
This script list or download Python source files
It was requested here: https://python-forum.io/Thread-downloading-source-for-every-version

Dependencies: requests, requests_html
Task: Remove external dependencies
"""


import re
import sys
try:
    import requests
    import requests_html
except ImportError:
    print('Please install requests and requests_html.', file=sys.stdout)
    sys.exit(255)


def sorter(url):
    """
    This sorter is used to sort the
    python by version string.
    
    Early version have only two digits.
    """
    v1 = re.search(r'(\d+)\.(\d+)\.(\d+)', url)
    v2 = re.search(r'(\d+)\.(\d+)', url)
    if v1:
        return tuple(map(int, v1.groups()))
    elif v2:
        return tuple(map(int, [*v2.groups(), 0]))
    else:
        return (0,0,0)


def get_python_sources(filetype='tgz'):
    """
    Grab all links from python.org/downloads/source/
    Return the source file archives sorted as a list.
    """
    url = 'https://www.python.org/downloads/source/'
    with requests_html.HTMLSession() as session:
        req = session.get(url)
    links = [link for link in req.html.links if link.endswith(filetype)]
    return sorted(links, key=sorter)


def print_python_sources():
    """
    Print all python source links to stdout.
    """
    for link in get_python_sources():
        print(link)


def download_python_sources():
    """
    Download all python source files to current location.
    """
    for link in get_python_sources():
        file = link.split('/')[-1]
        # find a better way to get the file name from url
        # maybe urllib.parse?
        # yarl is an alternative with more abstraction
        print('Downloading', file)
        req = requests.get(link, stream=True)
        with open(file, 'wb') as fd:
            for chunk in req.iter_content(chunk_size=64*1024**1):
                # what is the ideal chunk_size?
                # how to detect the best chunk_size?
                fd.write(chunk)


def help():
    print('python3', sys.argv[0], '<list|download>')
    sys.exit(1)


if __name__ == '__main__':
    if len(sys.argv) != 2:
        help()
    cmd = sys.argv[1].lower()
    if cmd not in ('download', 'list'):
        help()        
    try:
        if cmd == 'list':
            print_python_sources()
        elif cmd == 'download':
            download_python_sources()
    except Exception as e:
        # catching all exceptions here, the're bubbleing up
        # extend this to handle different cases
        print('Error:', e)
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#9
nice, but i already got every version. in fact i got 2 of every version. they had tar.bz2 and at one point switched to tar.xz while leaving the older ones as tar.bz2. they also have everything in tgz name pattern, but i originally was trying tar.gz because that is the common standard. i ran a script that tried everything based on a few patterns i did see. it also tried at least 8 different extensions for each version.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#10
DeaD_EyE Wrote:print('Please install requests and requests_html.', file=sys.stdout)

@DeaD_EyE Have you considered replacing this print with
import subprocess
import sys
subprocess.call([sys.executable, '-m', 'pip', 'install', '--user', 'requests', 'requests_html'])
import requests
import requests_html
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  best way to transfer databases from old version of PostgreSQL to new version Larz60+ 3 3,506 Jun-20-2018, 07:13 AM
Last Post: Skaperen

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020