Voice booking agent on ElevenLabs + AWS: tool turns take 5 s. How would you fix it?

Reddit r/AI_Agents Tools
voice-agents elevenlabs aws-lambda latency llm-judge tool-calling booking

Summary

A dev shares results from building a voice booking agent on ElevenLabs with AWS Lambda tools, achieving accurate pricing/availability but struggling with 5s P95 latency on tool turns and keeping booking steps ordered. The post asks the community how to reduce tool-turn latency, enforce step ordering, and make the post-call LLM judge reliable.

**TL;DR:** Hosted LLM on ElevenLabs, deterministic tools on AWS, an LLM judge after every call. Prices and availability are now right. The problems are latency on tool turns and keeping the booking steps in order. Diagram attached. **Setup** - ElevenLabs: STT, turn-taking, TTS (`eleven_v4_turbo`), hosted LLM GLM 5.2 with reasoning off, one prompt per language (EN/ES) - AWS Lambda tools: `quote` (same pricing function the checkout charges), `availability` (live), `prepare_booking` (issues a one-time code; checkout won't accept a booking without it) - Per-call state in DynamoDB, so the agent doesn't re-ask - After each call, Claude Haiku 4.5 on Bedrock judges the transcript against 4 checks **Results** (128 calls, 108 simulated) Price 101/103 · availability 76/76 · permission + checkout 54/70 · no re-asking 120/126 **Problems** - First audio on tool turns: P95 5.2 s (target 2.5 s). Tools take 105–454 ms; the rest is the model - The booking order lives in the prompt, so the agent sometimes skips steps - The judge sometimes misses failures **Questions** Booking order: ElevenLabs Workflow nodes, a server-side state machine, or both? How do you get tool turns under 2.5 s on ElevenLabs? How do you make a post-call LLM judge trustworthy? More detail in the comments.
Original Article

Similar Articles

6 months running a production voice agent for service businesses. The latency math is way harder than the demos suggest.

Reddit r/ArtificialInteligence

After 6 months running a voice AI agent for service businesses, the author reveals that real-world latency is bimodal (median ~800ms, p95 ~2.4s) and this p95 determines user perception. Issues like VAD misfires, function call degradation with long prompts, and TTS quality matter more than LLM choice, with multilingual support adding significant costs.

Building an AI voice agent from scratch: the parts that actually took our time

Reddit r/artificial

The author shares a postmortem on building a production phone-based AI voice agent, revealing that most engineering time was consumed by telephony infrastructure, turn detection, observability, and failure handling rather than core LLM behavior. They suggest using managed platforms like Vapi, Retell, or Dasha from the start to focus engineering effort on business logic.

Your voice agent probably isn't slow because of the LLM.

Reddit r/AI_Agents

A developer debunks the common belief that LLM latency is the primary cause of slow voice agents, explaining that delays often stem from earlier stages like audio capture, VAD, and STT. They recommend logging specific latency metrics and testing various STT/TTS providers and orchestration frameworks to diagnose issues.