Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Progress Finished Question
#11
Every bit helps me understand. Thanks for the link; I will be sure to read through it. Hopefully I can get something working.
Reply
#12
Well, I figured I'd follow up on this and post my results after trial and error and Google-fu. The 'shutil' module just doesn't support returning progress complete. Even threading couldn't make it happen. The interpreter will not process another line until the copy or move function has completed. I've learned that multithreading doesn't truly perform tasks in parallel, it rapidly switches back and forth between tasks. This doesn't work for shutil.copy()/shutil.move() because the rapid switching pauses until the copy or move function has completed, just like the standard, singlethreaded way. I've read a topic or two of multi-processing, but wth...

I found that I could write my own copy function from scratch (which I may end up doing eventually); shutil utilizes it's own 'copyfileobject' function that I could re-write including some code for progress feedback, but wth. If any more experienced Pythonites want to try out some code and find my conclusions incorrect, please don't hesitate to share--it's a learning experience for me.
Reply
#13
Yes, shutil.copy doesn't tell you its progress. You can check the size of the file using os.stat

file_zise = os.stat('original_big_file.big').st_size

# you doing this periodicaly every second while in another thread or process the copying is on the go
new_size = os.stat('new_big_file.big').st_size
print(new_size / file_size * 100) # print the percentage of copying file to original
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#14
Yeah, I tried that when I was experimenting. Like I said in my previous post, threading doesn't use true, parallel, concurrent threads, rather it switches back and forth rapidly. Even with quick switches, shutil.copy() pauses the interpreter until the copy process is complete--it will not switch to other threads while copying. That's what my testing has led me to believe, at least. If someone wants to show me otherwise, please do.

Thanks for your input, @wavic. I may look into multiprocessing because it offers true parallel tasking. Another option would be writing my own copy function.
Reply
#15
As I said I never used threading. Maybe I have to look at it these days. What you want to do is not a complicated task. I may try to do the same to see what is going one with these threads.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#16
It will be better if you show us the modified code.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#17
Well, I've come back to this because, well, why not. I've decided to go the custom copy function route and have come up with this (very rough code--just proof of concept for myself):
import shutil, os, sys

with open('397.64-desktop.exe', 'rb') as f:
    with open('C:\\Users\Mark\Downloads\Ordenador\driver.exe', 'wb') as g:
        size = os.stat('397.64-desktop.exe').st_size
        while True:
            moving = f.read(1024576)
            g.write(moving)
            size2 = os.stat('C:\\Users\Mark\Downloads\Ordenador\driver.exe').st_size
            print('\r', size2 / size * 100)
It works, but needs refinement (BIG time). I've noticed, however, that it is nowhere near as fast as shutil, or native OS copying. How can I improve the speed of transfers. I know of course you're limited to the IO of the drive but in my testing (NVME SSD) it's waaay slower than the drive can handle.

@wavic - I'd love to see code examples. Helps me learn, for one. But in my testing (code is deleted at this point), built-in Python functions halt at the copy/move line until the transfer is complete. Proving me wrong is always welcome.
Reply
#18
Since this is SSD you can increase the buffer size. For example 10M.
Also because of that buffer, you could know exactly how many bytes are copied. You can do some math instead calling os.stat in every iteration.

Or you can use tqdm.
Something like this:

import os
import tqdm

src = './source/big_file.big'
dest = './path/big_file.big'

f_size = os.stats('./source/big_file.big').st_size
buff = 10485760 # 1024**2 * 10 = 10M

num_chunks = f_size // buff + 1

with open(src, 'rb') as src_f, open(dest, 'wb') as dest_f:
    try:
        for _ in tqdm(range(num_chunks)):
            chunk = src_f.read(buff)
            dest_f.write(chunk)
    except IOError as e:
        print(e)
    finally:
        print(f'Done! Copied {f_size} bytes.')
It will give you nice progress bar.

I didn't tested the script.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#19
Great, wavic. I will look into this code further. I wonder what is "tqdm" (I'll be Googling it too)? Does it require to install a 3rd party library? Also, I have seen but never read about "try", "except" and "finally"... maybe a little explanation of the code? I'm quite new to this.
Reply
#20
In my post tqdm is a link. Just go and see it.
Try/except construction allows you to catch the errors or a specific error. A simple example will explain it better. My English is not so good as I want.

try:
    num = float(input('Input a number: '))
    num2 = float(input('A second please: '))
    result = num / num2
    print(f'{num} / {num2} = {result}')
except ValueError:
    print('Incorect input. Abort!')
except ZeroDivisionError:
    print('Zerro division not allowed. Abort!')
finally:
    print('We did some nice math. Buy!')
So you try to run some block of code and if there is an error you catch it and can take some actions.
This can replace the if/elif/else in most of the cases.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Process finished with exit code 137 (interrupted by signal 9: SIGKILL) erdemath 2 9,384 Apr-18-2022, 08:40 PM
Last Post: erdemath
  Progress bar bnadir55 1 1,781 Apr-11-2022, 01:52 PM
Last Post: deanhystad
  Using .hdf5 files only once they are finished writing pyhill00 7 2,722 Nov-25-2021, 06:01 PM
Last Post: pyhill00
  How to check if a file has finished being written leocsmith 2 7,689 Apr-14-2021, 04:21 PM
Last Post: perfringo
  Progress Indicator for Xmodem 0.4.6 KenHorse 1 1,931 Jan-30-2021, 07:12 PM
Last Post: bowlofred
  process finished with exit code -1073741819 (0xC0000005) GMCobraz 8 5,301 Sep-01-2020, 08:19 AM
Last Post: GMCobraz
  How to stop Xmodem after bin file transfer was finished shaya2103 0 2,474 Nov-27-2019, 04:33 PM
Last Post: shaya2103
  How can I add a progress bar for my software? aquerci 8 3,692 Nov-16-2019, 04:20 PM
Last Post: aquerci
  wget progress bar anasrocks 1 4,690 Jun-06-2019, 03:12 PM
Last Post: heiner55
  Process finished with exit code -107374819 (0xC0000375) mrazko 2 8,395 Apr-05-2019, 12:46 PM
Last Post: mrazko

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020