Your voice agent probably isn't slow because of the LLM.
Summary
A developer debunks the common belief that LLM latency is the primary cause of slow voice agents, explaining that delays often stem from earlier stages like audio capture, VAD, and STT. They recommend logging specific latency metrics and testing various STT/TTS providers and orchestration frameworks to diagnose issues.
Similar Articles
6 months running a production voice agent for service businesses. The latency math is way harder than the demos suggest.
After 6 months running a voice AI agent for service businesses, the author reveals that real-world latency is bimodal (median ~800ms, p95 ~2.4s) and this p95 determines user perception. Issues like VAD misfires, function call degradation with long prompts, and TTS quality matter more than LLM choice, with multilingual support adding significant costs.
Our voice agent's p99 was 280ms. Competitor's was 450ms. Users said ours felt slower. We measured why.
A voice agent team found that despite lower end-to-end latency (280ms vs competitor's 450ms), users perceived it as slower due to poor barge-in interrupt rate (380ms vs 60ms). They identified three fixes—memory pinning, VAD threshold tuning, and smaller TTS chunks—that improved barge-in rate from 41% to 89% at 100ms, making users feel it's faster.
@DanKornas: Real-time voice agents need more than an LLM call—they need transport, speech components, turn handling, and a path to …
TEN is a framework for building real-time multimodal conversational AI agents, offering configurable STT, LLM, and TTS components, a visual designer, and deployment options including self-hosting and split deployment.
I swapped the TTS in my voice agent and it cut the lag people actually feel more than anything else
The author shares their experience swapping the TTS in their voice agent to a custom model (Banter 1) designed for bilingual Arabic-English conversations, which significantly reduced perceived lag.
Voice agent latency: how to tell if it’s the model or the network
A guide on distinguishing whether latency in voice agents stems from the model or the network, helping developers identify bottlenecks.