best way to copy a big binary file

[Skaperen]

i also want to be able to output a progress indicator, such as the number of bytes copied so far. that rules out using anything that carries out the whole copy before completing. i'm thinking of implementing my copier as a generator that yields an int of the number of byte copied so far.

for number_of_bytes in copy_gen(fin,fout):
    print(f'\r{number_of_bytes} ',end='',flush=True)

[Larz60+]

Oh, you want to slow it down.

[snippsat]

i also want to be able to output a progress indicator

Use tqdm A Fast, Extensible Progress Bar for Python and CLI.
If i write a test,also rewrite previous code to use for loop bye using functools.partial.

from functools import partial
from tqdm import tqdm
import os

def file_copy(file_name, path_out, buffer_size=1024000):
    with tqdm(total=os.path.getsize(file_name), unit='B', unit_scale=True) as pbar:
        with open(file_name, 'rb') as fsrc,open(f'{path_out}/{file_name}', 'wb') as fdst:
                for byte in tqdm(iter(partial(fsrc.read, buffer_size), b'')):
                    fdst.write(byte)
                    pbar.update(len(byte))

if __name__ == '__main__':
    path_out = 'C:/code/out'
    file_name = '2.1GB.rar'
    file_copy(file_name, path_out)

Okay speed(6-sec for 2.1GB) with this buffer size as i have set to default.

[Skaperen]

how does tqdm know how big the file is?

[snippsat]

how does tqdm know how big the file is?

See that i use os.path.getsize

λ ptpython
>>> import os

>>> os.path.getsize('2.1GB.rar')
2178168457

Return size in bytes,then use unit='B', unit_scale=True,this tell tqdm to count bytes in block size.

If eg use 16 kb buffer_size,will see that the progress bar get a more stuttering way,because of smaller block size.

file_copy(file_name, path_out, buffer_size=16384)

[Skaperen]

i'm still asking how best to copy a binary file leaving open the possibility of adding in some kind of progress indication which may need to be something custom, or something else that could be done between steps of the copy. so that means doing the copy a little at a time. it also needs to be for open file objects although doing special opens may be possible. being portable is important so it works on other platforms that CPython has been properly ported to.