Making a phone app that can help you win at the card game SET — real-time computer vision and a small neural network. A fun weekend project.
In high school my good friend Laura introduced me to the card game SET. What I love about it is how the deceptively simple rules create a brain teaser in the form of a visual perception working memory search puzzle.
This write-up has three parts:
The game is built around four core features for each card — shape, color, number and shading. Click below to start an animation showing the three types of each feature. You can then click any feature again to review its types.
The full deck can be visualized as spanning each of the four features and their three types:
You can click each of the features below to toggle them ON or OFF to build up the full deck!
So the core of the game is finding SETs, which makes understanding what makes a SET the single rule you need in order to play. A SET is a group of three cards, and those three cards have to cover the feature space in a special way: each of the four features spanned by the group has to be either all the SAME or all DIFFERENT across the three cards. This is one of those things where seeing it helps far more than reading it. Below is an example of a SET.
Underneath the cards you can see how it satisfies the requirement, for each of the four features in turn. Try clicking on a feature row to flip it between SAME and DIFFERENT and see more examples of good SETs.
One detail you might notice: because there are no identical cards in the full deck, you must always have at least one feature that is DIFFERENT. So a SET could be DIFFERENT for one feature and the SAME in all the others — but it could equally be DIFFERENT for every single one.
So now you have learned the most important part of the game! But I promise it is one of those things that gets tricky and causes slip-ups — which is a good part of what makes it fun. As a warm up, below is a mini practice game where you can test your understanding of what a SET really is. I will give you two cards, and you choose the third card that completes them into an official SET.
Worth knowing before you start: any two cards have exactly one card that completes them. Not sometimes, not usually — always exactly one. So only one of the three choices can ever be right, and if you work through the four features one at a time you can always find it. Go ahead and get some practice until you feel comfortable.
Ok, now that you have practiced, let's get to the fun of the real game. A dealer shuffles the cards and lays twelve of them face up on the table in a rectangle, so they can be seen by every player.
The game has no turns. It just starts once everyone agrees to start, and you all begin searching at once. The first person to call out “SET!” gets control of the board and gets to quickly pick up the SET they found. The other players then check it together to make sure it really is a SET.
If a player calls SET and does not have one, or the three cards they picked turn out to be wrong, they lose a point and the cards are returned to the table. If the three cards do make up a SET, the player keeps them in a stack in front of them, which counts as one point scored, and the dealer puts out three fresh cards to replace the three that were just taken.
Try it out below!
Now, sometimes twelve cards will contain no SET at all. If after a while every player agrees there is none to be found, the dealer deals three more cards for a total of fifteen. Importantly, when a SET is then found among those fifteen, the three cards taken are not replaced — the board drops back down to twelve and play carries on from there.
The game continues until the deck runs out, and at the very end there may be a few cards left on the table that do not make a SET. The player with the most points — the biggest stack of SETs in front of them — wins.
Here is a clip of it working in an early development mode. Later I added more polish, with a voice that tells me when it has found a SET and can read it out if I want.
App and code download: coming soon.
Fair warning: this part was written automatically by AI from my project notes. It covers the details of the process, but it's honestly a bit junky to read.
SET's booklet gives the odds of a dead layout: about 1 in 33 for twelve cards, about 1 in 2500 for fifteen. Once you can generate and solve millions of layouts, those are checkable in a minute — and they check out. Twelve random cards contain no set 1 time in 31; fifteen, 1 in 3008. Both well inside an "about".
Except that is not the situation you are ever in. Twelve random cards is not what sits on the table. A real board is what is left after every set anyone could find has already been taken away — and set-rich boards are precisely the ones that get dismantled first. The set-poor ones survive and pile up.
Simulate actual games instead of random draws and the number moves a long way:
The obvious first move is to threshold on brightness: cards are white, tables are not. This works until the light is uneven, and then it fails in the worst possible way — it does not fail loudly, it just quietly loses an entire shadowed corner of the table.
The better signal is saturation. A SET card is almost colorless; hardwood, carpet and skin are not. Threshold on lack of color instead, judge brightness relative to a heavily blurred copy of the image rather than an absolute number, and the shadowed cards come back.
The most useful habit of the whole project came from here: count the blobs after every stage. Not look at the overlay — count. Overlays are reassuring and routinely lie. Counting found three separate root causes that looking had missed, including a morphological "cleanup" step that was quietly welding nineteen adjacent cards into a single blob.
Finding a card is easy. Deciding it is two green striped ovals is not. Hand-written rules — count blobs, measure hue, measure interior brightness — got 39.4% of cards fully right. A tiny k-nearest-neighbour classifier on the identical features got 54.7%. Shrinking the crop 12% inwards, so the table underneath stopped bleeding into the white balance, got it to 71.6%.
71.6% per card sounds respectable. Here is what it means for a table of twelve:
That multiplication is the villain of this project, and it comes back at the end.
A neural network would fix the reading, but a neural network wants tens of thousands of labelled examples and I had a deck of cards and a Saturday.
So: don't collect the data. Measure the deck and rebuild it. Macro photographs of a handful of cards, then measure everything — the exact ink colors from solid symbols only, the paper color, symbol width as a fraction of card width (0.245; my eyeball guess had been 0.170), the outline thickness, the stripe spacing, the stripe duty cycle solved backwards from how much darker a striped symbol is than an open one.
Then render cards from those measurements — and jitter every single constant on every card. Size, spacing, outline weight, stripe pitch, color temperature, exposure, blur, sensor noise, JPEG artefacts, and a slice of table bleeding in at one edge because the crop is never perfect.
The training data is never written to disk. It is generated on the fly, one gradient step per card, and thrown away: an infinite stream of cards that do not exist.
The thing that made it work was calibration, not architecture. The network is unremarkable. What mattered was measuring the real deck carefully enough that the fake cards landed in the same neighbourhood as the real ones.
I measured the stripe direction from real cards: the variance across rows was 45.1 against 4.1 down columns, so the stripes ran vertically. I built the renderer that way. My collaborator picked up a card, looked at it, and said the stripes were horizontal.
He was right. My measurement had been reading the symbol outlines, not the stripes inside them. It was a confident, objective, entirely wrong number — and it took one human glance to overturn.
Four three-way heads on one shared trunk, rather than one 81-way classifier. Every card then teaches all four heads at once — "two green striped ovals" is an example of two, of green, of striped and of oval simultaneously — and when it fails you can see which attribute failed.
| parameters | 329,460 |
|---|---|
| model file | 1.34 MB |
| input | 72 × 112 rectified crop |
| detection | 20 ms per frame |
| end to end | ~29 fps on a 2019 Snapdragon 855 |
One discovery en route that had nothing to do with machine learning. We had asked the camera for a 16:9 stream, because that is what video looks like. But the sensor is natively 4:3, so asking for 16:9 makes it crop — and held upright, it crops the width. We had been paying a third of the field of view for a shape nobody wanted. Switching to 4:3 gave the picture back and left black bands top and bottom, which is now where all the buttons live.
Said in passing, after playing with the app for ten minutes.
Remember that the detector finds cards by looking for things without color. Solid symbols are the most saturated ink on the card. So a solid card punches three large holes in its own mask — and the more symbols, the worse it gets:
| solid symbols | how much of the card the detector sees |
|---|---|
| one | 87.5% |
| two | 77.3% |
| three | 67.6% |
Holes alone are survivable. What kills it is the noise-removal step that runs afterwards. Three symbols span 87% of the card's width, leaving a rim of about six pixels down each side — and the cleanup erodes four pixels from the outside edge and four more from the inside of each hole. The rim is cut through, the card stops being one connected shape, and it vanishes.
Filling the holes before the cleanup instead of after fixes it completely:
The part worth keeping is why nobody found it sooner. There was a benchmark. The benchmark said everything was fine, and it said so twice, for two independent reasons: its photographs came from a camera setup we had since replaced, and it happened to contain no examples of the hard case. A week earlier I had looked at this exact code, seen the benchmark disagree with a handful of real captures, and concluded the captures were unrepresentative.
They were not. The benchmark was.
Detection is essentially solved: 99.3% of cards found, 99.3% of what it finds is a card. Reading is at 93.9% per card, up from 71.6%.
And a table needs every card right.
Getting to 99.5% per card would put tables at 94%, and there are two known routes: pool the reading across frames instead of deciding fresh every time, or fine-tune on the real labelled cards and give up the "never saw a real card" property. Neither is built. The first was designed in detail and deliberately abandoned — a card at a bad angle produces the same wrong answer thirty frames in a row, so pooling would deliver a confidently wrong result and then latch it, which is worse than flickering.
It plays SET. It beats me at spotting sets. It is wrong sometimes, and it does not know when. That seems like the right place to stop.
np.cross.cv2.erode treats everything outside the image as foreground, so open symbols rendered as disconnected arcs.From Brian Sciences: more projects and research.