A voice agent team found that despite lower end-to-end latency (280ms vs competitor's 450ms), users perceived it as slower due to poor barge-in interrupt rate (380ms vs 60ms). They identified three fixes—memory pinning, VAD threshold tuning, and smaller TTS chunks—that improved barge-in rate from 41% to 89% at 100ms, making users feel it's faster.
Shipped our voice agent into production last quarter. The dashboard said we were faster than every named competitor. p99 end-to-end latency at 280ms. The biggest competitor was 450ms. We were genuinely faster. User research panel said our agent felt slower. 8 percentage points on a 5-point Likert. Statistically significant. Two weeks of investigation later, we figured out the panel was measuring barge-in, not latency. Barge-in is the time between the user starting to interrupt and the agent shutting up. The end-to-end clock measures the agent's response time. The barge-in clock measures what the user actually waits through when they want to take control of the conversation. Different numbers. Our end-to-end was 280ms. Our barge-in was 380ms. Competitor's was 60ms. How we measured it: 1. Synthetic corpus of 500 recorded interruption attempts from prior support calls. Feed each one to a copy of the agent, measure time from first syllable to agent stopping. 2. OTel spans on the production pipeline. One span when VAD fires, one when TTS interrupts. Subtract. Both methods. Synthetic for A/B testing. Production for the actual distribution. Our barge-in interrupt rate at 100ms threshold was 41%. At 250ms it was 89% but 250ms is too slow to feel responsive. The fix was three things: 1. Pin the audio buffer pages in memory. libc::mlock on the buffer. Audio pages were occasionally paging to swap when the model weights were active, costing 150ms on detection. After pin, VAD caught speech within 25ms. 2. VAD threshold tuning. Default was 0.6. Tested 0.4 to 0.65. 0.5 was best. 4% earlier detection with only 1.2% increase in false positives. 3. TTS interrupt path. Our TTS streamed in 200ms chunks. When VAD fired, the audio queue still played 400ms of buffered chunks. We dropped chunk size to 30ms and flushed the queue immediately on VAD fire. More network overhead. Worth it. Four weeks of work. Barge-in interrupt rate at 100ms threshold moved from 41% to 89%. p99 latency actually went up slightly (280ms to 305ms) because of smaller TTS chunks. The dashboard got worse. Users say the agent feels faster than the 450ms competitor now. The mental shift that stuck: voice agent latency is the dashboard number. Barge-in interrupt rate is the user number. Once you measure both, the dashboard becomes a debugging tool and the user metric becomes the product KPI. Curious how other teams measure what users actually experience separately from what their dashboards report.
After 6 months running a voice AI agent for service businesses, the author reveals that real-world latency is bimodal (median ~800ms, p95 ~2.4s) and this p95 determines user perception. Issues like VAD misfires, function call degradation with long prompts, and TTS quality matter more than LLM choice, with multilingual support adding significant costs.
A developer debunks the common belief that LLM latency is the primary cause of slow voice agents, explaining that delays often stem from earlier stages like audio capture, VAD, and STT. They recommend logging specific latency metrics and testing various STT/TTS providers and orchestration frameworks to diagnose issues.
The author shares their experience swapping the TTS in their voice agent to a custom model (Banter 1) designed for bilingual Arabic-English conversations, which significantly reduced perceived lag.
An open-weights 8B parameter voice model achieves only 110ms latency, faster than average human conversation latency of 200-250ms. It can be run locally and is freely available via a GitHub repository.
The author argues that for live voice agents, STT latency and real-time behavior are more critical than raw transcription accuracy, and proposes a different evaluation scorecard.