Counting Duplicates in large Data Set

[jmair]

My students and I were discussing the difference between possible and probability when it comes to lottery numbers. Is the sequence 1 2 3 4 5 6 just as likely to happen as 76 31 7 54 29 18 ?

My question is, what would be a good format to record six random numbers and what method to count the duplicates?

[perfringo]

Simple simulation would do: run n-times random.sample on the desired range and convert result to tuple and feed to collections Counter. Then inspect results. If order doesn't matter, then before converting to tuple use sorted. I tried with n=1000000 and there was no 1, 2, 3, 4, 5, 6 in results (I used range(1, 49)).

[deanhystad]

Use a set instead of a tuple. Should be faster. But I don't think this as a possible solution.

"Simple solution" is not going to work because the numbers are staggering. I am not surprised that 1 million combinations did not produce a single 1, 2, 3, 4, 5, 6. One million is a pretty small sample size when there are almost 14 million combinations. For the kind of test you propose I would suggest 1 billion combinations. and how are you going to store the 320 MB Counter dictionary?

[paul18fr]

Hi

When speaking about duplicates for numbers, I'm alway thinking to "np.unique" => here bellow an example. Note at the same time Numpy is fast even for a huge array size.

Paul

import numpy as np

MyList=[0, 1, 10, 5, 2, 1, -1, 8, 2, 1, 5, 1, 1, 1, -1]
MyList=np.asarray(MyList)

UniqueList = np.unique(MyList, return_index=True, return_counts=True)

n = np.shape(UniqueList[0])[0]
for i in range(n):
    print(f"for {UniqueList[0][i]}  => {UniqueList[2][i]} occurence(s)")

Provinding:

Output:
for -1  => 2 occurence(s)
for 0  => 1 occurence(s)
for 1  => 6 occurence(s)
for 2  => 2 occurence(s)
for 5  => 2 occurence(s)
for 8  => 1 occurence(s)
for 10  => 1 occurence(s)