Python Forum
How do I copy files faster with python?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How do I copy files faster with python?
#1
Hey guys I know that this issue has already been adressed in a previous threads (https://stackoverflow.com/questions/2207...files-fast ), but I didn't manage to apply the solution to my own code (sorry, I'm rather new to Python) so I thought I´d ask again. Here´s my problem:

I created a simple app that compares the content of two folders and copies the new files of one folder into the other (I used the filecmp-statement for the comparison, so far so good). As for the copy-part, I used the shutil.copy-module.

As it has already been discussed in previous threads, the shutil.copy-module is way slower than the native command for copying files in windows (e.g. it took my code about 94 seconds to copy 800MB, while it took the ordinary copy and paste method in windows about 30 seconds for the same files).

I would be really grateful if somebody could have a look at my code and tell me how to implement a faster copying method (e.g. subprocess.call or native command) in my code. Thanks in advance for your help!

import shutil   
import filecmp
import os

comparison = filecmp.dircmp(r"C:\Users\j2the\Documents\Test2", r"C:\Users\j2the\Documents\Test1")

def analyze():
    for e in comparison.right_only:
        print(e)
        os.chdir(r"C:\Users\j2the\Documents\Test1")
        shutil.copy(e, r"C:\Users\j2the\Documents\Test2")

analyze()
Reply
#2
you can use subprocess call with xcopy on windows:
from subprocess import call
call(['xcopy', 'filepath source', 'filepath dest', '/K/O/X'])
the /K keeps attributes, /O maintains ownership, /X maintains file audit info
Reply
#3
Do you need a copy or reference to the file (hardlink) would do? You can create latter for example with os.link()
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#4
Try shutil.copyfileobj directly and set the buffer-size.


with open(filea, 'rb') as fda:
    with open(fileb, 'wb' as fdb:
        shutil.copyfileobj(fda, fdb, length=64*1024*1)
Try it with bigger values for length.
The standard is currently 16 KiB big chunks. On modern FS it's a little bit less.
But you can try to find the bast value for your os/host.

EDIT: This does not include copymode, this is an extra task you have to do,
if you use copyfileobj directly.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#5
(Jul-14-2019, 11:32 PM)Larz60+ Wrote: you can use subprocess call with xcopy on windows:
from subprocess import call
call(['xcopy', 'filepath source', 'filepath dest', '/K/O/X'])
the /K keeps attributes, /O maintains ownership, /X maintains file audit info

@Larz60+ thanks a lot for your help. I already thought about using the subprocess call module. The problem is that I don't get the statement to work with a for loop. For example if I try this:

for e in comparison.right_only:
        status = subprocess.call("copy e C:\Users\j2the\Documents\Test2", shell=True)
I get an message saying that the file cannot be found.
Do you happen to know how I can implement the subprocess module within my for loop? Thanks in advance for your help!
Reply
#6
(Jul-14-2019, 07:52 PM)steckinreinhart619 Wrote: I created a simple app that compares the content of two folders and copies the new files of one folder into the other

Is it for repeated usage (keeping files in sync)? What should happen in scenario when file in first directory is changed without renaming after it's already copied to second directory?
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#7
(Jul-17-2019, 10:04 AM)perfringo Wrote:
(Jul-14-2019, 07:52 PM)steckinreinhart619 Wrote: I created a simple app that compares the content of two folders and copies the new files of one folder into the other

Is it for repeated usage (keeping files in sync)? What should happen in scenario when file in first directory is changed without renaming after it's already copied to second directory?

Yes the code is basically meant for keeping files in sync, which means that when a file name in the source directory is changed, the older filer in the destination directory will be replaced by the new file.
Reply
#8
I personally prefer avoid copying files whenever possible (except backup Smile).

If it's not some sort of backup solution then just an idea: one can use os.walk() to hardlink destination to same file in source. If file in source is changed the hardlink should hold (keeps same inode number on *nix and the file index on Windows). If file renaming occur then based on os.stat() attribute st_ino one should be able to update the filename. os.stat() should be cheaper than actual copying of files.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Copy Paste excel files based on the first letters of the file name Viento 2 347 Feb-07-2024, 12:24 PM
Last Post: Viento
  Create new folders and copy files cocobolli 3 1,334 Mar-22-2023, 10:23 AM
Last Post: Gribouillis
  Copy only hidden files and folders with rsync Cannondale 2 953 Mar-04-2023, 02:48 PM
Last Post: Cannondale
  Compare filename with folder name and copy matching files into a particular folder shantanu97 2 4,390 Dec-18-2021, 09:32 PM
Last Post: Larz60+
  Copy documents to Teams using python SallySmith 0 2,339 Mar-23-2021, 04:27 AM
Last Post: SallySmith
  Copy files from subfolders into same name of subfolders at other directory rathoreanil 1 2,305 Oct-12-2020, 01:30 AM
Last Post: Larz60+
  need to define date range of files to copy over OTH 4 3,060 Aug-07-2020, 12:29 PM
Last Post: buran
  Search for files and copy them to different location jayser247 4 2,419 Apr-25-2020, 12:50 AM
Last Post: jayser247
  To Copy text values to Excel using Python ksasi2k3 16 15,551 Dec-26-2019, 11:59 AM
Last Post: ashutoshdeodhar
  How do I copy files without case sensitive? mcesmcsc 8 4,851 Dec-18-2019, 02:19 PM
Last Post: mcesmcsc

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020