Posts: 4,580
Threads: 1,472
Joined: Sep 2016
downloading source for every version of Python from www.python.org is such a bitch. they have a mix of .tar.bz2 and .tar.xz and i didn't see .tar.gz anywhere which i was sure they'd have. i think i need to do a brute-force download of the whole site.
Tradition is peer pressure from dead people
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Posts: 8,093
Threads: 154
Joined: Sep 2016
Posts: 4,580
Threads: 1,472
Joined: Sep 2016
Jun-06-2019, 01:05 AM
(This post was last modified: Jun-06-2019, 01:05 AM by Skaperen.)
i see no tar.xz files for many versions, such as 2.6 through 2.6.7 (is tar.bz2). then 2.6.8 is in tar.xz they need to do some compression conversion.
Tradition is peer pressure from dead people
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Posts: 8,093
Threads: 154
Joined: Sep 2016
OK, I didn't check every single one...
Posts: 4,580
Threads: 1,472
Joined: Sep 2016
i was trying to download every single one. i ended up throwing together a script that tried everything and every way i could think of. after about half an hour i think i now have ever version. now i see the pattern that could have made it go faster, had i known it before.
Tradition is peer pressure from dead people
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Posts: 2,025
Threads: 9
Joined: May 2017
Output: In [15]: sorted(tgz, key=by_version)
Out[15]:
['https://www.python.org/ftp/python/2.0.1/Python-2.0.1.tgz',
'https://www.python.org/ftp/python/2.1.3/Python-2.1.3.tgz',
'https://www.python.org/ftp/python/2.2/Python-2.2.tgz',
'https://www.python.org/ftp/python/2.2.1/Python-2.2.1.tgz',
'https://www.python.org/ftp/python/2.2.2/Python-2.2.2.tgz',
'https://www.python.org/ftp/python/2.2.3/Python-2.2.3.tgz',
'https://www.python.org/ftp/python/2.3/Python-2.3.tgz',
'https://www.python.org/ftp/python/2.3.1/Python-2.3.1.tgz',
'https://www.python.org/ftp/python/2.3.2/Python-2.3.2.tgz',
'https://www.python.org/ftp/python/2.3.3/Python-2.3.3.tgz',
'https://www.python.org/ftp/python/2.3.4/Python-2.3.4.tgz',
'https://www.python.org/ftp/python/2.3.5/Python-2.3.5.tgz',
'https://www.python.org/ftp/python/2.3.6/Python-2.3.6.tgz',
'https://www.python.org/ftp/python/2.3.7/Python-2.3.7.tgz',
'https://www.python.org/ftp/python/2.4/Python-2.4.tgz',
'https://www.python.org/ftp/python/2.4.1/Python-2.4.1.tgz',
'https://www.python.org/ftp/python/2.4.2/Python-2.4.2.tgz',
'https://www.python.org/ftp/python/2.4.3/Python-2.4.3.tgz',
'https://www.python.org/ftp/python/2.4.4/Python-2.4.4.tgz',
'https://www.python.org/ftp/python/2.4.5/Python-2.4.5.tgz',
'https://www.python.org/ftp/python/2.4.6/Python-2.4.6.tgz',
'https://www.python.org/ftp/python/2.5/Python-2.5.tgz',
'https://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz',
'https://www.python.org/ftp/python/2.5.2/Python-2.5.2.tgz',
'https://www.python.org/ftp/python/2.5.3/Python-2.5.3.tgz',
'https://www.python.org/ftp/python/2.5.4/Python-2.5.4.tgz',
'https://www.python.org/ftp/python/2.5.5/Python-2.5.5.tgz',
'https://www.python.org/ftp/python/2.5.6/Python-2.5.6.tgz',
'https://www.python.org/ftp/python/2.6/Python-2.6.tgz',
'https://www.python.org/ftp/python/2.6.1/Python-2.6.1.tgz',
'https://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz',
'https://www.python.org/ftp/python/2.6.3/Python-2.6.3.tgz',
'https://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz',
'https://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz',
'https://www.python.org/ftp/python/2.6.6/Python-2.6.6.tgz',
'https://www.python.org/ftp/python/2.6.7/Python-2.6.7.tgz',
'https://www.python.org/ftp/python/2.6.8/Python-2.6.8.tgz',
'https://www.python.org/ftp/python/2.6.9/Python-2.6.9.tgz',
'https://www.python.org/ftp/python/2.7/Python-2.7.tgz',
'https://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz',
'https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz',
'https://www.python.org/ftp/python/2.7.10/Python-2.7.10rc1.tgz',
'https://www.python.org/ftp/python/2.7.11/Python-2.7.11rc1.tgz',
'https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz',
'https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz',
'https://www.python.org/ftp/python/2.7.12/Python-2.7.12rc1.tgz',
'https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz',
'https://www.python.org/ftp/python/2.7.13/Python-2.7.13rc1.tgz',
'https://www.python.org/ftp/python/2.7.14/Python-2.7.14rc1.tgz',
'https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz',
'https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz',
'https://www.python.org/ftp/python/2.7.15/Python-2.7.15rc1.tgz',
'https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz',
'https://www.python.org/ftp/python/2.7.16/Python-2.7.16rc1.tgz',
'https://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz',
'https://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz',
'https://www.python.org/ftp/python/2.7.4/Python-2.7.4.tgz',
'https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz',
'https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz',
'https://www.python.org/ftp/python/2.7.7/Python-2.7.7rc1.tgz',
'https://www.python.org/ftp/python/2.7.7/Python-2.7.7.tgz',
'https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz',
'https://www.python.org/ftp/python/2.7.9/Python-2.7.9rc1.tgz',
'https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz',
'https://www.python.org/ftp/python/3.0/Python-3.0.tgz',
'https://www.python.org/ftp/python/3.0.1/Python-3.0.1.tgz',
'https://www.python.org/ftp/python/3.1/Python-3.1.tgz',
'https://www.python.org/ftp/python/3.1.1/Python-3.1.1.tgz',
'https://www.python.org/ftp/python/3.1.2/Python-3.1.2.tgz',
'https://www.python.org/ftp/python/3.1.3/Python-3.1.3.tgz',
'https://www.python.org/ftp/python/3.1.4/Python-3.1.4.tgz',
'https://www.python.org/ftp/python/3.1.5/Python-3.1.5.tgz',
'https://www.python.org/ftp/python/3.2/Python-3.2.tgz',
'https://www.python.org/ftp/python/3.2.1/Python-3.2.1.tgz',
'https://www.python.org/ftp/python/3.2.2/Python-3.2.2.tgz',
'https://www.python.org/ftp/python/3.2.3/Python-3.2.3.tgz',
'https://www.python.org/ftp/python/3.2.4/Python-3.2.4.tgz',
'https://www.python.org/ftp/python/3.2.5/Python-3.2.5.tgz',
'https://www.python.org/ftp/python/3.2.6/Python-3.2.6.tgz',
'https://www.python.org/ftp/python/3.2.6/Python-3.2.6rc1.tgz',
'https://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz',
'https://www.python.org/ftp/python/3.3.1/Python-3.3.1.tgz',
'https://www.python.org/ftp/python/3.3.2/Python-3.3.2.tgz',
'https://www.python.org/ftp/python/3.3.3/Python-3.3.3.tgz',
'http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz',
'https://www.python.org/ftp/python/3.3.5/Python-3.3.5rc1.tgz',
'http://www.python.org/ftp/python/3.3.5/Python-3.3.5rc2.tgz',
'http://www.python.org/ftp/python/3.3.5/Python-3.3.5.tgz',
'http://www.python.org/ftp/python/3.3.5/Python-3.3.5rc1.tgz',
'https://www.python.org/ftp/python/3.3.6/Python-3.3.6.tgz',
'https://www.python.org/ftp/python/3.3.6/Python-3.3.6rc1.tgz',
'https://www.python.org/ftp/python/3.3.7/Python-3.3.7rc1.tgz',
'https://www.python.org/ftp/python/3.3.7/Python-3.3.7.tgz',
'http://www.python.org/ftp/python/3.4.0/Python-3.4.0rc3.tgz',
'https://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz',
'https://www.python.org/ftp/python/3.4.1/Python-3.4.1rc1.tgz',
'https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz',
'https://www.python.org/ftp/python/3.4.10/Python-3.4.10.tgz',
'https://www.python.org/ftp/python/3.4.10/Python-3.4.10rc1.tgz',
'https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tgz',
'https://www.python.org/ftp/python/3.4.2/Python-3.4.2rc1.tgz',
'https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz',
'https://www.python.org/ftp/python/3.4.3/Python-3.4.3rc1.tgz',
'https://www.python.org/ftp/python/3.4.4/Python-3.4.4rc1.tgz',
'https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tgz',
'https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz',
'https://www.python.org/ftp/python/3.4.5/Python-3.4.5rc1.tgz',
'https://www.python.org/ftp/python/3.4.6/Python-3.4.6rc1.tgz',
'https://www.python.org/ftp/python/3.4.6/Python-3.4.6.tgz',
'https://www.python.org/ftp/python/3.4.7/Python-3.4.7rc1.tgz',
'https://www.python.org/ftp/python/3.4.7/Python-3.4.7.tgz',
'https://www.python.org/ftp/python/3.4.8/Python-3.4.8.tgz',
'https://www.python.org/ftp/python/3.4.8/Python-3.4.8rc1.tgz',
'https://www.python.org/ftp/python/3.4.9/Python-3.4.9.tgz',
'https://www.python.org/ftp/python/3.4.9/Python-3.4.9rc1.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc3.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b2.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc1.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b4.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a1.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a2.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b3.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a4.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0b1.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc4.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0a3.tgz',
'https://www.python.org/ftp/python/3.5.0/Python-3.5.0rc2.tgz',
'https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz',
'https://www.python.org/ftp/python/3.5.1/Python-3.5.1rc1.tgz',
'https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz',
'https://www.python.org/ftp/python/3.5.2/Python-3.5.2rc1.tgz',
'https://www.python.org/ftp/python/3.5.3/Python-3.5.3rc1.tgz',
'https://www.python.org/ftp/python/3.5.3/Python-3.5.3.tgz',
'https://www.python.org/ftp/python/3.5.4/Python-3.5.4rc1.tgz',
'https://www.python.org/ftp/python/3.5.4/Python-3.5.4.tgz',
'https://www.python.org/ftp/python/3.5.5/Python-3.5.5rc1.tgz',
'https://www.python.org/ftp/python/3.5.5/Python-3.5.5.tgz',
'https://www.python.org/ftp/python/3.5.6/Python-3.5.6rc1.tgz',
'https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz',
'https://www.python.org/ftp/python/3.5.7/Python-3.5.7.tgz',
'https://www.python.org/ftp/python/3.5.7/Python-3.5.7rc1.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a1.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0rc2.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a4.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a3.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b3.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0rc1.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b1.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0a2.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b4.tgz',
'https://www.python.org/ftp/python/3.6.0/Python-3.6.0b2.tgz',
'https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz',
'https://www.python.org/ftp/python/3.6.1/Python-3.6.1rc1.tgz',
'https://www.python.org/ftp/python/3.6.2/Python-3.6.2rc2.tgz',
'https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz',
'https://www.python.org/ftp/python/3.6.2/Python-3.6.2rc1.tgz',
'https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz',
'https://www.python.org/ftp/python/3.6.3/Python-3.6.3rc1.tgz',
'https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz',
'https://www.python.org/ftp/python/3.6.4/Python-3.6.4rc1.tgz',
'https://www.python.org/ftp/python/3.6.5/Python-3.6.5rc1.tgz',
'https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz',
'https://www.python.org/ftp/python/3.6.6/Python-3.6.6rc1.tgz',
'https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz',
'https://www.python.org/ftp/python/3.6.7/Python-3.6.7rc2.tgz',
'https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz',
'https://www.python.org/ftp/python/3.6.7/Python-3.6.7rc1.tgz',
'https://www.python.org/ftp/python/3.6.8/Python-3.6.8rc1.tgz',
'https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a1.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b2.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b5.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a4.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a3.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0a2.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0rc1.tgz',
'https://www.python.org/ftp/python/3.7.0/Python-3.7.0b1.tgz',
'https://www.python.org/ftp/python/3.7.1/Python-3.7.1rc1.tgz',
'https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz',
'https://www.python.org/ftp/python/3.7.1/Python-3.7.1rc2.tgz',
'https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz',
'https://www.python.org/ftp/python/3.7.2/Python-3.7.2rc1.tgz',
'https://www.python.org/ftp/python/3.7.3/Python-3.7.3rc1.tgz',
'https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz',
'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a1.tgz',
'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a3.tgz',
'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a2.tgz',
'https://www.python.org/ftp/python/3.8.0/Python-3.8.0b1.tgz',
'https://www.python.org/ftp/python/3.8.0/Python-3.8.0a4.tgz']
Posts: 4,580
Threads: 1,472
Joined: Sep 2016
that code doesn't seem to be working for me. but then, i'm running on 3.5.2.
Output: Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'tgz' is not defined
Tradition is peer pressure from dead people
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Posts: 2,025
Threads: 9
Joined: May 2017
I made this one minute before lunch. Just 20 lines code.
Here as an example, but it has dependencies: requests and requests_html
"""
This script list or download Python source files
It was requested here: https://python-forum.io/Thread-downloading-source-for-every-version
Dependencies: requests, requests_html
Task: Remove external dependencies
"""
import re
import sys
try:
import requests
import requests_html
except ImportError:
print('Please install requests and requests_html.', file=sys.stdout)
sys.exit(255)
def sorter(url):
"""
This sorter is used to sort the
python by version string.
Early version have only two digits.
"""
v1 = re.search(r'(\d+)\.(\d+)\.(\d+)', url)
v2 = re.search(r'(\d+)\.(\d+)', url)
if v1:
return tuple(map(int, v1.groups()))
elif v2:
return tuple(map(int, [*v2.groups(), 0]))
else:
return (0,0,0)
def get_python_sources(filetype='tgz'):
"""
Grab all links from python.org/downloads/source/
Return the source file archives sorted as a list.
"""
url = 'https://www.python.org/downloads/source/'
with requests_html.HTMLSession() as session:
req = session.get(url)
links = [link for link in req.html.links if link.endswith(filetype)]
return sorted(links, key=sorter)
def print_python_sources():
"""
Print all python source links to stdout.
"""
for link in get_python_sources():
print(link)
def download_python_sources():
"""
Download all python source files to current location.
"""
for link in get_python_sources():
file = link.split('/')[-1]
# find a better way to get the file name from url
# maybe urllib.parse?
# yarl is an alternative with more abstraction
print('Downloading', file)
req = requests.get(link, stream=True)
with open(file, 'wb') as fd:
for chunk in req.iter_content(chunk_size=64*1024**1):
# what is the ideal chunk_size?
# how to detect the best chunk_size?
fd.write(chunk)
def help():
print('python3', sys.argv[0], '<list|download>')
sys.exit(1)
if __name__ == '__main__':
if len(sys.argv) != 2:
help()
cmd = sys.argv[1].lower()
if cmd not in ('download', 'list'):
help()
try:
if cmd == 'list':
print_python_sources()
elif cmd == 'download':
download_python_sources()
except Exception as e:
# catching all exceptions here, the're bubbleing up
# extend this to handle different cases
print('Error:', e)
Posts: 4,580
Threads: 1,472
Joined: Sep 2016
nice, but i already got every version. in fact i got 2 of every version. they had tar.bz2 and at one point switched to tar.xz while leaving the older ones as tar.bz2. they also have everything in tgz name pattern, but i originally was trying tar.gz because that is the common standard. i ran a script that tried everything based on a few patterns i did see. it also tried at least 8 different extensions for each version.
Tradition is peer pressure from dead people
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Posts: 4,543
Threads: 69
Joined: Jan 2018
DeaD_EyE Wrote:print('Please install requests and requests_html.', file=sys.stdout)
@ DeaD_EyE Have you considered replacing this print with
import subprocess
import sys
subprocess.call([sys.executable, '-m', 'pip', 'install', '--user', 'requests', 'requests_html'])
import requests
import requests_html
|