What I learned building an AI tutor for high school math

When I added an AI tutor to Math Hero, I assumed the hard part would be infrastructure. API calls, streaming, token costs. That stuff was routine. The hard part was getting it to teach instead of just answering.

The first version made things worse

Any LLM's default behavior is to be helpful. Ask "what's the derivative of 3x^2?" and it says "6x." Correct, and totally useless for a student who needs to understand the power rule.

My first prototype was a chat window bolted onto the lesson screen. Students could ask anything, the AI would answer everything. Usage went up. Completion rates went down. Kids were just asking the chatbot instead of thinking.

The system prompt is where it got fixed

I rewrote the system prompt to be much more restrictive. Simplified version:

You are a Socratic math tutor for high school students.
NEVER solve the problem for the student.
Instead, ask a guiding question that helps them identify their next step.
If the student is stuck, give a hint about which concept applies,
but do not perform the calculation.
If the student asks you to "just tell me the answer," explain that
working through it themselves is how the learning happens.

You have to tell the model what not to do, over and over. LLMs want to be maximally helpful, and in education that means minimum learning. I also inject the current lesson context -- what topic, what problem, what they've gotten wrong -- so the tutor responds to what's actually happening, not in the abstract.

Guardrails I didn't think I'd need

The tutor is restricted to math topics the app covers. Students will absolutely try to get it to write essays or talk about random stuff. I validate input and output; if the response drifts off topic, it gets caught before rendering.

I also enforce age-appropriate language in the system prompt and run a lightweight content filter on outputs. The audience is 14-18. Parents trust this app with their kids, and one jailbreak screenshot on Twitter would be a real problem.

For accuracy, I don't rely on the model for computation. LLMs aren't calculators. Anything involving actual math gets validated against the known correct answers in the lesson data. The AI handles pedagogy. It doesn't do arithmetic.

Where the tutor lives in the UI

I burned a few weeks building a separate chat tab before realizing it was wrong. Students would finish chatting, hit back, and lose their context. What worked was making the tutor an inline collapsible panel inside the problem view. It can see what problem you're on, what you've tried, what hints you've already gotten. It needs to feel like part of the lesson, not a separate thing you switch to.

Mistakes, roughly in order of how long they took me to figure out

I gave the AI too much personality. The first version was quirky and encouraging. Students found it annoying. They wanted clear, short guidance. I stripped the persona back to almost nothing.

I didn't rate limit conversations. Some kids would go 30 messages deep trying to trick the AI into giving answers. I added a soft cap -- after a few exchanges on one problem, the tutor suggests reviewing the lesson material. Outcomes actually got better after this.

I underestimated how much latency matters in context. A 3-second response in a standalone chat feels fine. A 3-second delay when you're stuck mid-problem feels like forever. Streaming responses and shorter prompts brought the perceived wait down enough.

What the data showed

Three things correlated with better problem solving rates:

  1. Hints tied to the specific problem, not generic topic explanations
  2. One guiding question at a time instead of multi-step breakdowns
  3. The tutor appearing proactively after two wrong attempts, instead of waiting for students to ask for help

The students who improved most weren't the heavy chatters. They were the ones who got one good nudge and then solved it themselves.

-- Daniel