Python Forum
Transferring large amounts of files
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Transferring large amounts of files
#1
Hopefully someone can help me find some good resources for this. I want to write something in Python that can transfer large amounts of files (TB's of files) in the shortest amount of time possible. Looking around on Google I found something called GridFTP. I really like the way it describes how the files are transferred and I would like to write something in Python that transfers files the same way. Just setting up a GridFTP server isn't really going to solve the problem, because this is just a piece of a much larger pie. I've done file transfers with Python before, but this seems like it's much more complicated. I'm not a huge fan of duplicating work so I've been looking around for a module that does this same thing, but haven't found anything yet. Anyone have any idea's, or know of something off hand?
Reply
#2
Why not just use rsync? Or robocopy if you're on windows.

If it's just a piece of the puzzle, then work on the pieces that someone else hasn't already solved for you :p
Reply
#3
The other pieces of the puzzle are clear and have already been worked out. This one however is the main piece and the most difficult.

I'm not asking anyone to give me code that will solve the problem (unless it's a publicly available module that others are using). I'm simply using a resource to hopefully speed up my research on what my options are. I was under the impression that this was a main selling point to using the Internet. :P

I have looked at using rsync type solutions, but after looking over this document, I think the way GridFTP transfers will better serve this project. I have some ideas on how to do this, but the more resources I have to confirm my thoughts the better. Thanks in advance!
Reply
#4
You're task is far from uncommon. I suggest looking through here: https://pypi.python.org/pypi?%3Aaction=s...mit=search
Reply
#5
Thank you for the link. I'm new enough at developing with Python that I sometimes forget about the things like PyPi.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Transferring argument to a thread issues msloat 1 1,599 Feb-27-2022, 06:12 PM
Last Post: msloat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020