Automating the code using os.walk

[Ofnuts]

I would just try

Output:
wget https://archive.org/download/archiveteam-twitter-stream-2012-01/archiveteam-twitter-2012-01.tar

on your remote machine. wget is usually quite reliable.

Thank you for suggestion. I am downloading the file this way now and will test it. The speed, however, is relatively slow--1-2MB/s on a 10 Gigabit channel.

Your Python code isn't that fast either. There are better protocols than HTTP to transfer large files. FTP is already somewhat faster but it will not take advantage of all the bandwidth.

As a minimum compute the MD5 on both sides of the transfer and compare them.