Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
fast hash function
#1
hi guys,
i'm looking for a faster-than-the-default-adler32/md5/etc non-crypto hash function.
i have googled and found things like pyfasthash, xxhash, etc. but they only appear to be available in [C] source form. not being a C developer, and not wanting to acquire the baggage that that entails, i would much prefer something that i could pip install.
any ideas?
Reply
#2
Use %timeit to compare them. here is an example.

pip install pyhashxx
some additional info on various algs: https://cyan4973.github.io/xxHash/
sample script
# in ipython
from pyhashxx import hashxx
import hashlib

def hashxx_test(x):
    result = hashxx(x)

def hashlib_md5_test(x):
    m = hashlib.md5()
    m.update(x)

test_str_one = "Hello World!"
test_str_two = "ABCD1234"*2**17 #1MB
test_str_three = "ABCD1234"*2**23 # 64MB

# 12 Bytes
%timeit hashlib_md5_test(test_str_one)
# 1670 ns
%timeit hashxx_test(test_str_one)
# 359 ns (4.7x speedup)

1MB
%timeit hashlib_md5_test(test_str_two)
# 3840 us
%timeit hashxx_test(test_str_two)
# 417 us (9.2x speedup)

# 64MB
%timeit hashlib_md5_test(test_str_two)
# 225 ms
%timeit hashxx_test(test_str_two)
# 29.6 ms (7.6x speedup)
Reply
#3
Many (most, all) hash functions involve a lot of bit shifting, and masking that is why most are written in C
Any book on compiler design or Algorithms will include many examples, but usually in 'C' or smalltalk
Reply
#4
thanks, guys.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [SOLVED] How to crack hash with hashlib Milan 0 1,326 Mar-09-2023, 08:25 PM
Last Post: Milan
  Hash command works differently for me in CMD and Spyder ZweiDCG 3 2,302 Sep-10-2019, 01:10 PM
Last Post: DeaD_EyE
  length constraint on phrase hash to password javaben 0 1,880 Aug-21-2019, 05:34 PM
Last Post: javaben
  Create file archive that contains crypto hash ED209 1 2,006 May-29-2019, 03:05 AM
Last Post: heiner55
  Fastest dict/map method when 'key' is already a hash? tasket 6 3,909 Apr-20-2019, 06:40 PM
Last Post: tasket
  hash v2 and v3 help Normalitie 7 4,269 Mar-22-2018, 01:57 PM
Last Post: DeaD_EyE
  how to generate sha256 hash for each line of my txt file | using python version 3.6.4 rajtekken5 2 9,030 Feb-11-2018, 01:41 PM
Last Post: rajtekken5
  virtualenv activate.ps1 hash error po20 2 3,765 Jan-13-2018, 09:21 AM
Last Post: po20
  Hash function in Python rachelrosemond 3 3,991 Sep-29-2017, 02:57 PM
Last Post: ichabod801
  Python Hash list check here2learn 4 5,355 Feb-27-2017, 08:35 PM
Last Post: here2learn

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020