Oct-13-2017, 07:22 AM
Tried blake2b but got better results with xxhash.
I found out that Blocksize is of much significance. Overall got better results(with larger files) when used 2**17(131072 bytes) instead of 1mb.
Using concurrent.futures worsened the execution time and multiprocessing.Pool seemed to have no effect, idk may it has to do something with how hashes are calculated.
My current updated script took 610 seconds(10 minutes) to find 14170 duplicates in 53,491 Files and 7,988 Folders of total 165 GB.
Now I'll try to change block/buffer size according to the file size but i don't know the exact effect buffer size have on the hashing speed in accordance to filesize.
I found out that Blocksize is of much significance. Overall got better results(with larger files) when used 2**17(131072 bytes) instead of 1mb.
Using concurrent.futures worsened the execution time and multiprocessing.Pool seemed to have no effect, idk may it has to do something with how hashes are calculated.
My current updated script took 610 seconds(10 minutes) to find 14170 duplicates in 53,491 Files and 7,988 Folders of total 165 GB.
Now I'll try to change block/buffer size according to the file size but i don't know the exact effect buffer size have on the hashing speed in accordance to filesize.