On-device AI is bigger than you think

Everyone's talking about cloud models getting smarter. GPT-4, Claude, Gemini, whatever comes next. But the more interesting story for mobile developers is happening on the other end: smaller models running directly on the phone.

Apple has been building toward this for years

Core ML shipped in 2017. The Neural Engine has been in every A-series and M-series chip since. Apple Intelligence was the moment they said it out loud: AI should run locally first, and only reach out to the cloud when it has to.

Google's going a different direction. Gemini Nano is on Android devices, but Google's instinct is to route things through their infrastructure. They want your queries flowing through their servers. Apple wants computation to stay on your device. I'm biased, but if you care about privacy and latency, Apple's bet is more interesting.

Why this matters if you build mobile apps

Local inference takes milliseconds. A server round trip takes hundreds of milliseconds on a good day. For anything interactive -- suggestions while typing, camera features, adaptive UI -- that gap is the difference between responsive and laggy.

Privacy gets simpler too. Data stays on the device. You don't need a privacy policy paragraph explaining why a kid's math homework is hitting a server. For education apps, that's a big deal.

Then there's offline. Your app works on a plane, in a subway tunnel, at a school with bad wifi. And there are no inference API bills. The user's phone is your compute. When you're one person running a studio, not paying per request is a real relief.

What I'm thinking about for my own apps

In Math Hero, spaced repetition already runs locally. A fully on-device tutor that watches how a student solves a problem, catches misconceptions, and adjusts in real time -- that's what I want to build next. The pieces are getting close.

In CardValue Pro, I use the camera to scan trading cards. Right now the image pipeline is fairly straightforward. But on-device vision models are getting good enough that you could do card grading and condition assessment locally. No API call, no waiting, no cost per scan.

The hybrid bet

I think the best mobile AI apps will be hybrid. On-device for the fast, frequent, private stuff: text suggestions, image classification, personalized recommendations, real time feedback. Cloud models for the heavy lifting: multi-step reasoning, large context synthesis, anything that needs a frontier model.

The hard part is drawing that line well. What runs locally and what gets sent out. Developers who get that right will build noticeably better apps. Developers who send everything to the cloud will have apps that feel slow and cost too much to run.

WWDC is what I'm watching

A few specific things: Can we run larger models on-device without blowing up memory? Will Apple open up their foundation models the way they opened Vision and NaturalLanguage? And will Xcode get tooling for profiling on-device model performance, not just code generation?

The developer tooling story is what will decide adoption. A powerful Neural Engine doesn't help much if deploying a model to it requires a research background.

The hardware is there. The frameworks are getting there. I've started prototyping locally for Math Hero, and even the rough version is faster than I expected.

-- Daniel