Quote:I don't think that matters, tbh. Hashing millions of *anything* is going to take a long time.
First, fast is relative,
Hashing individual keys can be extremely fast.
Case in point - My background is telecommunications (At one of Largest telecommunications companies
in the world at the time)
We processed no less than 80 million calls per day, each starting out as a bit stream,
Then hashing, formatting into records, identifying customer, breaking into time of day and date segments,
(billing was done on duration, number of parties, and time of day in one minute segments), NPA, NXX and LATA,
and much more.
That's 4,000,000 calls per minute (all multi record)
Typical time for a complete run was 20 to 30 minutes (a complete days processing). This was done
in 'C' on HP 9000's.
We would not use relational databases at this point, but rather a home built structure that I guess
would be extremely similar to a Python dictionary. I can get into details if requested.
Of course we didn't use SHA, the hash was a modified version from what found in 'Dragon book'
by Aho, Sethi and Ullman. (not the real title, but known everywhere by this name)
Hash tables were fixed length (usually a prime number of entries) with lateral expansion (linked lists). This
proved to have amazingly fast access times.
To make this more interesting, although large blocks were used, data was stored on disk.
This all done back in 1992 - 1994