Latency matters more than model selection when building AI tutoring systems

Everyone has an opinion on which LLM to use for AI tutoring. GPT-4o vs Claude vs fine-tuned open-source. It's the wrong question to start with. What actually keeps students engaged is response latency. Specifically, speech start latency - how fast the voice response kicks in after a student finishes talking. Past roughly 1.5 seconds, they check out. They assume the system froze. No amount of model quality recovers that tutoring rhythm once it's broken. The pipeline goes: ASR → context lookup → LLM call → TTS → avatar sync → output. Every step costs time, and they compound. Most teams obsess over the LLM step. In practice, ASR and the TTS-to-avatar-sync handoff are where things go sideways. Whisper handles ASR accuracy well, but its latency behavior needs real attention in streaming contexts. Add a 3D avatar with lip sync and you've got a rendering layer on top of everything else - let that slip past a certain threshold and the uncanny valley effect kills the experience faster than a slow response would. For production systems, the targets I'd aim for: speech start latency under 1 second, full response under 2 seconds, whiteboard or shared context sync under 500ms. Reachable, but not without deliberate work at each layer. Streaming TTS is probably the highest-leverage thing you can do. Don't wait for the full LLM response before starting audio. Play audio while tokens are still coming in - in practice, that alone can shave a full second off how fast the response feels. Get the architecture right for latency first. Then worry about what the model says.