How I built real-time card scanning with Vision

CardValue Pro lets you point your phone's camera at a trading card and get an estimated value in about a second. Getting there took longer than I'd like to admit.

Starting with OCR

First version was simple. Grab a camera frame, run VNRecognizeTextRequest, pull out the card name and set information, look it up in a pricing database. For a clean card sitting flat on a table under decent light, this worked fine.

Then I tested it on actual cards in actual conditions. Foil cards threw reflections right across the text. Holographic surfaces confused the text recognizer into returning garbage. Curved cards in someone's hand warped the text enough that recognition fell apart. And vintage cards with worn printing or unusual fonts were basically invisible to OCR.

I spent a while tuning recognition parameters -- language correction, minimum text height, custom word lists. Helped on the margins. But OCR alone couldn't reliably identify a card when the text wasn't cooperating.

Adding image classification

What does work, even with glare and curves, is recognizing the card visually. The artwork, the border style, the layout. Humans do this all the time. You don't read the tiny set number to know you're looking at a 1986 Fleer Jordan.

I added VNClassifyImageRequest as a second signal and trained a Core ML model on a dataset of card images. The classifier handles the cases OCR can't: it identifies the card from visual features regardless of surface reflections or text legibility. OCR still runs in parallel and contributes when it gets a confident read. The final identification combines both signals, weighted by confidence scores. If OCR returns high confidence text, that gets priority. If not, the classifier carries the load.

The camera pipeline

Real-time scanning runs on AVCaptureSession feeding frames to the Vision pipeline. Processing every frame was overkill and tanked performance, so I throttle to every third frame. Simple counter increments on each captureOutput callback and skips frames where the count isn't divisible by three. Keeps the analysis current without saturating the processor.

Results feed back to SwiftUI through an ObservableObject that publishes identified card data. Camera preview stays live and smooth while identification results appear in an overlay.

The freeze bug

Early testers on iPhone 12 and 13 reported the camera preview freezing for a second or two at a time. On my iPhone 15 Pro, everything was fluid. Classic "works on my machine."

Problem was that I was running Vision inference on the main thread. On newer hardware with a faster Neural Engine, inference completed quickly enough that the UI stayed responsive. On older phones, each inference call blocked the main thread long enough to stall the preview layer. Camera kept capturing frames, but the display wasn't updating.

Fixing it

Moved all Vision request execution to a dedicated background DispatchQueue. Capture delegate receives frames on the session queue, hands them to the inference queue, and results dispatch back to main only for the final UI update. I also added frame dropping: if the inference queue is still processing when a new frame arrives, the new frame gets discarded. No point queuing up stale frames.

Don't do expensive work on the main thread. I knew this. I just didn't notice the violation because my development device was fast enough to hide it.

What it got us

Recognition latency dropped from around 800ms to roughly 120ms. Preview never freezes, even on an iPhone 12. The app feels like pointing a camera at a card and immediately seeing what it's worth, which is what it should have felt like from the start.

The frame dropping approach means we occasionally skip a frame that would have given a good read, but since the camera is continuous, the next processed frame picks it up. In practice, identification happens within the first second of pointing at a card, fast enough that users perceive it as instant.

I'm still iterating on the Core ML model -- the training set needs more vintage cards and non-English sets. But the pipeline itself has been stable. Parallel OCR plus classification, throttled frames, background inference, and frame dropping handles the messy reality of scanning physical objects with a phone camera.

-- Daniel