Training one model to recognize three different card games

Pokemon cards have a clean, rounded frame with the HP in the top right corner. Magic cards use a dark border with a text box that takes up the bottom third. Yu-Gi-Oh cards cram attribute icons, levels, and dense stat blocks into a layout that looks nothing like the other two. When I set out to build TCG Scanner, I figured the visual differences would make recognition easy. Wrong.

Three models, one oversized app

My first approach was obvious: train a separate Core ML model for each game. Pokemon model, Magic model, Yu-Gi-Oh model. Each handling its own card set.

It worked. Accuracy was decent. But the app bundle hit 180MB. Three models, each carrying its own learned representations of what a trading card looks like -- the shared features (rectangular card, artwork in the center, text at the bottom) were being learned three times over. Users on older iPhones were abandoning the download. You can't iterate your way out of that.

One model, multiple tasks

The fix was architectural. Instead of three independent models, I trained a single multi-task model. First layer classifies which game the card belongs to. Then the network branches into game-specific heads that handle identification within that game.

The shared backbone learns general card features once: edges, text regions, color distributions, holographic patterns. The game-specific branches learn the stuff that actually differs. Bundle dropped to roughly 60MB -- a third of the original, with comparable accuracy.

Building the training set

I started with my own collection. A few thousand cards across all three games, photographed on my desk under different lighting. Not nearly enough, so I supplemented with images scraped from public card databases -- sites that host high-resolution scans of every card ever printed.

The real world gap was still a problem though. Database scans are perfectly lit, perfectly flat, perfectly centered. Actual user photos are taken at angles, under fluorescent lights, with glare from sleeves. I leaned hard on data augmentation: random rotations up to 15 degrees, brightness and contrast shifts, and synthetic glare overlays that mimic the reflection patterns you get from holographic cards under overhead lighting. That augmentation pipeline did more for real world accuracy than any model architecture change.

The Japanese card problem

This one blindsided me. Collectors frequently mix Japanese-language cards into their English collections, especially for Pokemon and Yu-Gi-Oh where Japanese sets release months earlier. My model would see a Japanese Pokemon card and confidently classify it as Yu-Gi-Oh. The Japanese text density and layout cues were throwing off the game classifier.

Fixed it by adding a language detection branch early in the pipeline, before the game classification step. Model first determines whether the card text is English, Japanese, or Korean, then routes to language-aware classification. Minimal overhead to inference time but eliminated an entire category of misclassification.

Where it landed

After the language branch and a few rounds of hard-negative mining, the model hit 94% correct identification on the first scan. Point your camera at a card and you usually get the right result immediately. For the remaining cases, adjusting the angle or lighting and scanning again pushes accuracy to 99% within three attempts.

Keeping up with new sets

All three games release new cards quarterly, sometimes more. Can't retrain manually every time. So I built a pipeline: new card images get pulled from public databases as sets release, run through the augmentation pipeline, and fed into a fine-tuning job that updates only the game-specific heads. Shared backbone stays frozen unless something fundamental changes. Whole retraining cycle takes about a day on a Mac with an M-series chip, and I can validate against a held-out test set before pushing the update.

The pipeline isn't glamorous. It's a handful of Python scripts, a cron job, and a TestFlight build that I sanity check myself before going live. But it means the app stays current without me dropping everything when a new set drops.

-- Daniel