How do I copy files faster with python?

[steckinreinhart619]

Hey guys I know that this issue has already been adressed in a previous threads (https://stackoverflow.com/questions/2207...files-fast ), but I didn't manage to apply the solution to my own code (sorry, I'm rather new to Python) so I thought I´d ask again. Here´s my problem:

I created a simple app that compares the content of two folders and copies the new files of one folder into the other (I used the filecmp-statement for the comparison, so far so good). As for the copy-part, I used the shutil.copy-module.

As it has already been discussed in previous threads, the shutil.copy-module is way slower than the native command for copying files in windows (e.g. it took my code about 94 seconds to copy 800MB, while it took the ordinary copy and paste method in windows about 30 seconds for the same files).

I would be really grateful if somebody could have a look at my code and tell me how to implement a faster copying method (e.g. subprocess.call or native command) in my code. Thanks in advance for your help!

1 2 3 4 5 6 7 8 9 10 11 12 13

import shutil   import filecmp import os   comparison = filecmp.dircmp(r"C:\Users\j2the\Documents\Test2", r"C:\Users\j2the\Documents\Test1")   def analyze():     for e in comparison.right_only:         print(e)         os.chdir(r"C:\Users\j2the\Documents\Test1")         shutil.copy(e, r"C:\Users\j2the\Documents\Test2")   analyze()

[Larz60+]

you can use subprocess call with xcopy on windows:

1 2	from subprocess import call call(['xcopy', 'filepath source', 'filepath dest', '/K/O/X'])

the /K keeps attributes, /O maintains ownership, /X maintains file audit info

[perfringo]

Do you need a copy or reference to the file (hardlink) would do? You can create latter for example with os.link()

[DeaD_EyE]

Try shutil.copyfileobj directly and set the buffer-size.

1 2 3

with open(filea, 'rb') as fda:     with open(fileb, 'wb' as fdb:         shutil.copyfileobj(fda, fdb, length=64*1024*1)

Try it with bigger values for length.
The standard is currently 16 KiB big chunks. On modern FS it's a little bit less.
But you can try to find the bast value for your os/host.

EDIT: This does not include copymode, this is an extra task you have to do,
if you use copyfileobj directly.

[steckinreinhart619]

you can use subprocess call with xcopy on windows:

1 2	from subprocess import call call(['xcopy', 'filepath source', 'filepath dest', '/K/O/X'])

the /K keeps attributes, /O maintains ownership, /X maintains file audit info

@Larz60+ thanks a lot for your help. I already thought about using the subprocess call module. The problem is that I don't get the statement to work with a for loop. For example if I try this:

1 2	for e in comparison.right_only:         status = subprocess.call("copy e C:\Users\j2the\Documents\Test2", shell=True)

I get an message saying that the file cannot be found.
Do you happen to know how I can implement the subprocess module within my for loop? Thanks in advance for your help!

[perfringo]

I created a simple app that compares the content of two folders and copies the new files of one folder into the other

Is it for repeated usage (keeping files in sync)? What should happen in scenario when file in first directory is changed without renaming after it's already copied to second directory?

[steckinreinhart619]

I created a simple app that compares the content of two folders and copies the new files of one folder into the other

Is it for repeated usage (keeping files in sync)? What should happen in scenario when file in first directory is changed without renaming after it's already copied to second directory?

Yes the code is basically meant for keeping files in sync, which means that when a file name in the source directory is changed, the older filer in the destination directory will be replaced by the new file.

[perfringo]

I personally prefer avoid copying files whenever possible (except backup ).

If it's not some sort of backup solution then just an idea: one can use os.walk() to hardlink destination to same file in source. If file in source is changed the hardlink should hold (keeps same inode number on *nix and the file index on Windows). If file renaming occur then based on os.stat() attribute st_ino one should be able to update the filename. os.stat() should be cheaper than actual copying of files.