Oct-12-2017, 11:54 PM
os.walk is using os.scandir. At least in Python 3.6.
However, I have tried to make it faster using concurrent.futures.ProcessPoolExecutor to run it on all cores but don't know how much. I was thinking that the md5 hashing is going to be a CPU heavy operation but it turns out that it's not. On my old laptop. It depends on the disk performance.
I have changed from md5 to sha1 because of the possibility of equal hash sums from different files. It's minimal but still... I may change it again to blake2b. If it turns out that the CPU is doing well I will try to use asyncio instead. I have tried already but without success. This library is making my head to explode. Also, will try to take pieces of the file for hashing, not the whole file. According to their website, blake2b can do 1GB per second.
I did a few more changes. If I run the script to scan my /home dir it exits with errors if I am using the web browser at the same time for example. Which is normal because of the .chache directory. I am ignoring those errors but seems it's better to make it skip the whole dir.
However, I have tried to make it faster using concurrent.futures.ProcessPoolExecutor to run it on all cores but don't know how much. I was thinking that the md5 hashing is going to be a CPU heavy operation but it turns out that it's not. On my old laptop. It depends on the disk performance.
I have changed from md5 to sha1 because of the possibility of equal hash sums from different files. It's minimal but still... I may change it again to blake2b. If it turns out that the CPU is doing well I will try to use asyncio instead. I have tried already but without success. This library is making my head to explode. Also, will try to take pieces of the file for hashing, not the whole file. According to their website, blake2b can do 1GB per second.
I did a few more changes. If I run the script to scan my /home dir it exits with errors if I am using the web browser at the same time for example. Which is normal because of the .chache directory. I am ignoring those errors but seems it's better to make it skip the whole dir.