fast hash function - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: fast hash function (/thread-1201.html) |
fast hash function - verstapp - Dec-13-2016 hi guys, i'm looking for a faster-than-the-default-adler32/md5/etc non-crypto hash function. i have googled and found things like pyfasthash, xxhash, etc. but they only appear to be available in [C] source form. not being a C developer, and not wanting to acquire the baggage that that entails, i would much prefer something that i could pip install. any ideas? RE: fast hash function - sixteenornumber - Dec-13-2016 Use %timeit to compare them. here is an example.pip install pyhashxx some additional info on various algs: https://cyan4973.github.io/xxHash/ sample script # in ipython from pyhashxx import hashxx import hashlib def hashxx_test(x): result = hashxx(x) def hashlib_md5_test(x): m = hashlib.md5() m.update(x) test_str_one = "Hello World!" test_str_two = "ABCD1234"*2**17 #1MB test_str_three = "ABCD1234"*2**23 # 64MB # 12 Bytes %timeit hashlib_md5_test(test_str_one) # 1670 ns %timeit hashxx_test(test_str_one) # 359 ns (4.7x speedup) 1MB %timeit hashlib_md5_test(test_str_two) # 3840 us %timeit hashxx_test(test_str_two) # 417 us (9.2x speedup) # 64MB %timeit hashlib_md5_test(test_str_two) # 225 ms %timeit hashxx_test(test_str_two) # 29.6 ms (7.6x speedup) RE: fast hash function - Larz60+ - Dec-13-2016 Many (most, all) hash functions involve a lot of bit shifting, and masking that is why most are written in C Any book on compiler design or Algorithms will include many examples, but usually in 'C' or smalltalk RE: fast hash function - verstapp - Dec-13-2016 thanks, guys. |