Dice detection using DBSCAN

[Termiik]

Hi there,

I'm trying to make dice detection system using a Raspberry with a camera. I want to use the live video feed for this.

Now I'm able to detect the dice, but there is a slight problem.
Eventually there will be multiple dice rolled by a machine. This means it is possible that there will be dice right next to eachother.

The problem is that when I put 2 dice against eachother, the number on the dice will be calculated as if it was one dice. Now I've kinda fixed it by saying there cannot be more then 6 points in a cluster.

But now if I have a 2 on a dice and it's against a 5 for example it still counts as 6. Now i've tried tweaking the eps value, but the distance between the 2 pips on a 2 dice roll are greater than one of the pips from the 2 next to another dice roll. I've added some pictures for you to see. Down here is the code

import cv2
import numpy as np
from picamera2 import Picamera2
from sklearn import cluster

# Setup van de camera
picam2 = Picamera2()
picam2.preview_configuration.main.size = (1920, 1080)
picam2.video_configuration.controls.FrameRate = 25
picam2.preview_configuration.main.format = "RGB888"
picam2.start()

params = cv2.SimpleBlobDetector_Params()

params.filterByInertia = True
params.minInertiaRatio = 0.6

detector = cv2.SimpleBlobDetector_create(params)

def get_blobs(frame):
    frame_blurred = cv2.medianBlur(frame, 7)
    frame_gray = cv2.cvtColor(frame_blurred, cv2.COLOR_BGR2GRAY)
    blobs = detector.detect(frame_gray)
    
    return blobs

def get_dice_from_blobs(blobs):
    
    X = []
    for b in blobs:
        pos = b.pt
        
        if pos!= None:
            X.append(pos)
            
    X = np.asarray(X)
    
    if len(X) > 0:
        
        clustering = cluster.DBSCAN(eps=60, min_samples=1).fit(X)
        
        num_dice = max(clustering.labels_) + 1
        
        dice = []
        
        for i in range(num_dice):
            X_dice = X[clustering.labels_ == i]
            
            centroid_dice = np.mean(X_dice, axis=0)
            
            dice.append([len(X_dice), *centroid_dice])
            
        return dice
    
    else:
        
        return[]

def overlay_info(frame, dice, blobs):
    
    for b in blobs:
        pos = b.pt
        r = b.size / 2
        
        cv2.circle(frame, (int(pos[0]), int(pos[1])),
                   int(r), (255, 0, 0), 2)
        
        for d in dice:
            
            textsize = cv2.getTextSize(
                str(d[0]), cv2.FONT_HERSHEY_PLAIN, 3, 2)[0]
            
            cv2.putText(frame, str(d[0]),
                        (int(d[1] - textsize[0] / 2),
                         int(d[2] + textsize[1] / 2)),
                        cv2.FONT_HERSHEY_PLAIN, 3, (0, 255, 0), 2)

while True:
    # Neem een afbeelding van de camera
    frame = picam2.capture_array()
    
    blobs = get_blobs(frame)
    dice = get_dice_from_blobs(blobs)
    out_frame = overlay_info (frame, dice, blobs)
    
    cv2.imshow('frame', frame)
    
    res = cv2.waitKey(1)
    
    if res & 0xFF == ord('q'):
        break
    
    
picam2.stop()
cv2.destroyAllWindows()

[Pedroski55]

I have 3 questions:

1: Will your dice always be that creamy-white colour?
2.: Will the spots always be blue?
3. Will the camera always be at the same height above the dice?

If so, should be easy to walk through the pixels, looking for white and blue.

I'll try with one die first, see how that goes.

[Termiik]

I have 3 questions:

1: Will your dice always be that creamy-white colour?
2.: Will the spots always be blue?
3. Will the camera always be at the same height above the dice?

If so, should be easy to walk through the pixels, looking for white and blue.

I'll try with one die first, see how that goes.

1: yes it will always be the same colour.
2: The spots are actually black. these spots are the ones that are detected and drawn onto the image.
3: yes the camera will always be at the same height above the dice.

one dice works and 2 aswell, but as soon as they touch eachother, I can't seem to seperate the 2

[Pedroski55]

Aha, that makes life easier! The dice in the photo should all be the same size!

cv2 uses top left as 0,0 and y goes down from there. You can identify a pixel like this: pixel = dice_copy[y,x]

What I have done so far is:

1. Rotate the image if it is higher than wide.
2. Coming from the left, for each x, go down y, until you find the first white pixel, which is the leftmost corner of the first die. (if the die is level, the next corner will be either below this point, or right of this point.)
3. Then, find either the next lower corner, or the next upper corner, from the left.
4. Now you know the first corner and you know the dx and dy from the first corner to the other corner. dx + dy = the bounding box of this dice. (Don't forget the offset from x = 0)
5. Draw a 6 pixel thick green line on the right side of this die.
6. Now search for the spots in the die. The green line is where you stop as you go across x.
7. When you find a spot, mine are blue, get its leftmost and rightmost pixel. Increase your count by 1.
8. Draw a box around this spot, I use a red line.
9. Make everything within the red box transparent, by setting the alpha channel to 0. Then a blue pixel looks like [255 0 0 0]
10. Now you keep going across and down, looking for the next blue pixel, repeat, until, for this die the function find_blue_spots returns None. You have all the spots on this die.
11. Make all white pixels on this die transparent, as above with the blue spots.
12. Now look for white spots again. Repeat.

The only problem I have is when the die is tilted right by a very small angle, say < 3 degrees, I get a wrong count, because the green line is wrong. Still working on that! This comes from the fact that the edge of the die, which looks like a straight line, is actually a series of steps, except when it is horizontal / vertical.

Had a good idea today, but no time to try it, yet.

Also, I am working with pngs exported from Inkscape, not those grainy images you have, but, once this works I will try it on your image. Maybe post a couple more?

When I have tried out my new idea, I will put the code here, if you are interested.

I know cv2 has many functions to find things, but I have no idea how they actually work, internally. An image is just a rectangle full of pixels, it is possible just to use that fact!

[Pedroski55]

It's been a long time! I got involved in a house renovation project here in Barbate, don't have time for anything lately!

I tried to do this with my own functions, but when I tried them on real photos, noise in the photos made that impossible!

Not sure how to post images here, but you can try this with your pictures. You just need to tweak the numbers to suit your images.

# bilateral filter preserves edges
# cv.bilateralFilter( src, d, sigmaColor, sigmaSpace[, dst[, borderType]] ) -> dst

I just put 2 dice together and took a photo with my mobile, then I cropped it so I just have the 2 dice and a small area around them.

import cv2 as cv

# a font for writing the number on the dice image
font = cv.FONT_HERSHEY_SIMPLEX
destination = 'cv2/images/steps/'
name = input('Enter the name of the picture you want to analyse ... ')
# write on this image when we have the contours
original_image = cv.imread(destination + name)
original_image.shape # (963, 1592, 3)
savename = 'final_output.png'

# get the image as greyscale
image = cv.imread(destination + name, cv.IMREAD_GRAYSCALE)

# have a look at the grey image
cv.imwrite(destination + 'grey.png', image)

# make the image black and white
# bilateral filter preserves edges
# cv.bilateralFilter(	src, d, sigmaColor, sigmaSpace[, dst[, borderType]]	) ->	dst
# https://docs.open# cv.org/3.4/d4/d86/group__imgproc__filter.html#ga9d7064d478c95d60003cf839430737ed
# the docs may be old tweak parameters to suit
#blur = cv.bilateralFilter(image,9,75,75)
blur = cv.bilateralFilter(image,5,75,75)
cv.imwrite(destination + 'blur.png', blur)
ret,thresh = cv.threshold(blur,0,255,cv.THRESH_BINARY+cv.THRESH_OTSU)
cv.imwrite(destination + 'thresh.png', thresh) # some noise may be left
# now get clean contours
contours, hierarchy = cv.findContours(thresh, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)
len(contours) # may not be correct may contain noise contours
    
# draw all the contours on the original image:
cv.drawContours(original_image, contours, -1, (0,255,0), 5)
cv.imwrite(destination + 'contours.png', original_image)

# have a look at the areas within the contours
for i in range(len(contours)):
    # Calculate contour area
    area = cv.contourArea(contours[i])
    print(f'The area of this contour is: {area}')

# by first looking at areas output you can see each die is about 370000+ pixels
# a contour with that area must be a die
areas = [cv.contourArea(contours[i]) for i in range(len(contours))]

# a dictionary to take the numbers
# the key is the area of each die
d = {num:[] for num in areas if num > 360000} # tweak the number to suit your pictures
# d = {376399.5: [], 382368.5: []}

# assign the spots to the dice
# the actual number on each die will be the length of d[key]
for num in areas:    
    if num > 360000:
        key = num
        continue
    # small imperfections cause small contours, filter them out
    # adjust the numbers to suit
    elif num < 55500 and num > 14500:
        d[key].append(num)
        
total = 0
for i in range(len(contours)):
    # Calculate contour area
    area = cv.contourArea(contours[i])
    print(f'The area of this contour is: {area}')
    if area > 310000:
        M = cv.moments(contours[i])
        centroid_x = int(M['m10'] / M['m00'])
        centroid_y = int(M['m01'] / M['m00'])
        dice_centre = (centroid_x, centroid_y)
        number = len(d[area])
        total = total + number
        print('\n***********************************************')
        print(f'The total value so far of the throw is {total}')
        print('\n***********************************************')
        cv.putText(original_image, str(number), dice_centre, font, 8,(0,255,0, 255),5,cv.LINE_AA)
        
print(f'Saving png as {destination + savename} ... ')
cv.imwrite(destination + savename, original_image)

Vrolijk kerstfeest!