Python Forum
Progress Finished Question - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Progress Finished Question (/thread-9983.html)

Pages: 1 2 3 4


RE: Progress Finished Question - malonn - May-09-2018

Every bit helps me understand. Thanks for the link; I will be sure to read through it. Hopefully I can get something working.


RE: Progress Finished Question - malonn - May-11-2018

Well, I figured I'd follow up on this and post my results after trial and error and Google-fu. The 'shutil' module just doesn't support returning progress complete. Even threading couldn't make it happen. The interpreter will not process another line until the copy or move function has completed. I've learned that multithreading doesn't truly perform tasks in parallel, it rapidly switches back and forth between tasks. This doesn't work for shutil.copy()/shutil.move() because the rapid switching pauses until the copy or move function has completed, just like the standard, singlethreaded way. I've read a topic or two of multi-processing, but wth...

I found that I could write my own copy function from scratch (which I may end up doing eventually); shutil utilizes it's own 'copyfileobject' function that I could re-write including some code for progress feedback, but wth. If any more experienced Pythonites want to try out some code and find my conclusions incorrect, please don't hesitate to share--it's a learning experience for me.


RE: Progress Finished Question - wavic - May-12-2018

Yes, shutil.copy doesn't tell you its progress. You can check the size of the file using os.stat

file_zise = os.stat('original_big_file.big').st_size

# you doing this periodicaly every second while in another thread or process the copying is on the go
new_size = os.stat('new_big_file.big').st_size
print(new_size / file_size * 100) # print the percentage of copying file to original



RE: Progress Finished Question - malonn - May-12-2018

Yeah, I tried that when I was experimenting. Like I said in my previous post, threading doesn't use true, parallel, concurrent threads, rather it switches back and forth rapidly. Even with quick switches, shutil.copy() pauses the interpreter until the copy process is complete--it will not switch to other threads while copying. That's what my testing has led me to believe, at least. If someone wants to show me otherwise, please do.

Thanks for your input, @wavic. I may look into multiprocessing because it offers true parallel tasking. Another option would be writing my own copy function.


RE: Progress Finished Question - wavic - May-12-2018

As I said I never used threading. Maybe I have to look at it these days. What you want to do is not a complicated task. I may try to do the same to see what is going one with these threads.


RE: Progress Finished Question - wavic - May-12-2018

It will be better if you show us the modified code.


RE: Progress Finished Question - malonn - May-21-2018

Well, I've come back to this because, well, why not. I've decided to go the custom copy function route and have come up with this (very rough code--just proof of concept for myself):
import shutil, os, sys

with open('397.64-desktop.exe', 'rb') as f:
    with open('C:\\Users\Mark\Downloads\Ordenador\driver.exe', 'wb') as g:
        size = os.stat('397.64-desktop.exe').st_size
        while True:
            moving = f.read(1024576)
            g.write(moving)
            size2 = os.stat('C:\\Users\Mark\Downloads\Ordenador\driver.exe').st_size
            print('\r', size2 / size * 100)
It works, but needs refinement (BIG time). I've noticed, however, that it is nowhere near as fast as shutil, or native OS copying. How can I improve the speed of transfers. I know of course you're limited to the IO of the drive but in my testing (NVME SSD) it's waaay slower than the drive can handle.

@wavic - I'd love to see code examples. Helps me learn, for one. But in my testing (code is deleted at this point), built-in Python functions halt at the copy/move line until the transfer is complete. Proving me wrong is always welcome.


RE: Progress Finished Question - wavic - May-21-2018

Since this is SSD you can increase the buffer size. For example 10M.
Also because of that buffer, you could know exactly how many bytes are copied. You can do some math instead calling os.stat in every iteration.

Or you can use tqdm.
Something like this:

import os
import tqdm

src = './source/big_file.big'
dest = './path/big_file.big'

f_size = os.stats('./source/big_file.big').st_size
buff = 10485760 # 1024**2 * 10 = 10M

num_chunks = f_size // buff + 1

with open(src, 'rb') as src_f, open(dest, 'wb') as dest_f:
    try:
        for _ in tqdm(range(num_chunks)):
            chunk = src_f.read(buff)
            dest_f.write(chunk)
    except IOError as e:
        print(e)
    finally:
        print(f'Done! Copied {f_size} bytes.')
It will give you nice progress bar.

I didn't tested the script.


RE: Progress Finished Question - malonn - May-21-2018

Great, wavic. I will look into this code further. I wonder what is "tqdm" (I'll be Googling it too)? Does it require to install a 3rd party library? Also, I have seen but never read about "try", "except" and "finally"... maybe a little explanation of the code? I'm quite new to this.


RE: Progress Finished Question - wavic - May-21-2018

In my post tqdm is a link. Just go and see it.
Try/except construction allows you to catch the errors or a specific error. A simple example will explain it better. My English is not so good as I want.

try:
    num = float(input('Input a number: '))
    num2 = float(input('A second please: '))
    result = num / num2
    print(f'{num} / {num2} = {result}')
except ValueError:
    print('Incorect input. Abort!')
except ZeroDivisionError:
    print('Zerro division not allowed. Abort!')
finally:
    print('We did some nice math. Buy!')
So you try to run some block of code and if there is an error you catch it and can take some actions.
This can replace the if/elif/else in most of the cases.