Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 best way to copy a big binary file
#11
i also want to be able to output a progress indicator, such as the number of bytes copied so far. that rules out using anything that carries out the whole copy before completing. i'm thinking of implementing my copier as a generator that yields an int of the number of byte copied so far.
for number_of_bytes in copy_gen(fin,fout):
    print(f'\r{number_of_bytes} ',end='',flush=True)
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Quote
#12
Oh, you want to slow it down.
Quote
#13
Skaperen Wrote:i also want to be able to output a progress indicator
Use tqdm A Fast, Extensible Progress Bar for Python and CLI.
If i write a test,also rewrite previous code to use for loop bye using functools.partial.
from functools import partial
from tqdm import tqdm
import os

def file_copy(file_name, path_out, buffer_size=1024000):
    with tqdm(total=os.path.getsize(file_name), unit='B', unit_scale=True) as pbar:
        with open(file_name, 'rb') as fsrc,open(f'{path_out}/{file_name}', 'wb') as fdst:
                for byte in tqdm(iter(partial(fsrc.read, buffer_size), b'')):
                    fdst.write(byte)
                    pbar.update(len(byte))

if __name__ == '__main__':
    path_out = 'C:/code/out'
    file_name = '2.1GB.rar'
    file_copy(file_name, path_out)
Okay speed(6-sec for 2.1GB) with this buffer size as i have set to default.
İmage
Quote
#14
how does tqdm know how big the file is?
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Quote
#15
Skaperen Wrote:how does tqdm know how big the file is?
See that i use os.path.getsize
λ ptpython
>>> import os

>>> os.path.getsize('2.1GB.rar')
2178168457
Return size in bytes,then use unit='B', unit_scale=True,this tell tqdm to count bytes in block size.

If eg use 16 kb buffer_size,will see that the progress bar get a more stuttering way,because of smaller block size.
file_copy(file_name, path_out, buffer_size=16384)
Quote
#16
i'm still asking how best to copy a binary file leaving open the possibility of adding in some kind of progress indication which may need to be something custom, or something else that could be done between steps of the copy. so that means doing the copy a little at a time. it also needs to be for open file objects although doing special opens may be possible. being portable is important so it works on other platforms that CPython has been properly ported to.
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  how would you copy a file? Skaperen 17 522 Oct-04-2019, 07:15 AM
Last Post: Skaperen

Forum Jump:


Users browsing this thread: 1 Guest(s)