Which LLM are you running for real-time voice agents? What we’re hearing about dedicated deployments and serverless

This post brings together feedback from colleagues and customers, along with what we’re currently seeing in real-time voice agent deployments. These are qualitative observations across different providers, configurations, prompts and workloads. Most of the feedback concerns models running with thinking disabled or kept to a minimum. The same priorities keep coming up: reliable instruction following, correct tool calls, and fast, predictable responses throughout a conversation. Here’s what we’re hearing about models on dedicated endpoints: Gemma 4 31B. The most common production default, with a good balance of latency and capability. Some providers advertise under 200 ms TTFT, a few under 100 ms. BF16 and FP8 get better tool-calling feedback; we've heard of tool-calling regressions on some FP4 configurations, so the exact deployment matters. It also fine-tunes well for conversational behavior. Qwen3 235B Instruct. Valued for strict instruction following. Teams with tightly scripted workflows like how literally it follows prompts; some find it less likely to reinterpret instructions than GPT or Claude. One team in production moved from GPT-4.1 to Qwen3 235B Instruct (thinking off) on dedicated instances, and LLM latency went from 0.8–1.5 s to about 0.3–0.5 s. GLM 5.2 (thinking off). The strongest quality feedback on dedicated deployments; it came out ahead in the A/B tests Jared shared, but dedicated hosting costs more. On serverless, reported TTFT around 450 ms reflects particularly favorable conditions. Cache misses or congestion can push it into seconds, making predictable latency much harder to achieve for live voice. Nemotron 3 Ultra. Used with a small thinking budget (e.g. 256 tokens) to balance capability and responsiveness. Few providers support a token budget rather than a thinking level. gpt-oss-120b on Groq/Cerebras. Fast, but needs careful prompting and explicit tool instructions. Some teams report throttling and worse P99 at scale on shared capacity. Mercury 2. A diffusion LLM with very low reported TTFT and tool calling with adjustable reasoning. Kimi K2.7/K3. Less favored for real-time voice, because of generation speed (some report low double-digit tok/s on K2.7) and the cost of dedicated deployments Closed models GPT-4.1. Still the tool-calling reference for several teams; some tried newer models and came back to it. GPT-5.6 Luna. Positive feedback for speed and capability. Gemini Flash. 2.5 Flash had the most consistent p90/p99 in one team's repeated measurements. 3.5 Flash is smarter but slower; one team saw about 1.1 s average voice-to-voice. A few deployment lessons keep coming up alongside the model recommendations: Measure what the caller hears. TTFT, the first usable chunk for TTS, and the first audible response are different milestones. Specify which one you’re measuring. Check longer conversations. An endpoint can perform well with a short context and behave differently later in the call. Evaluate latency across the session. Separate cache hits and misses. Prefix caching can help, but replica routing, eviction or changes near the beginning of the prompt can affect reuse. Check both conditions with realistic conversation lengths. Test tools on the actual serving configuration. Wrong tool selection, incorrect arguments, failed dependent calls and poor recovery from tool errors need separate checks. Include your production prompts, quantization and tool schemas. Include region and concurrency in the comparison. A model recommendation becomes much more useful when it includes where it runs, the hardware configuration and how many calls overlap. There’s also a cost question: what happens when nobody is calling? Some teams describe almost no overnight traffic, with only occasional urgent calls outside business hours. Keeping dedicated GPUs running through those periods adds idle cost. Releasing capacity can reduce spending, but teams also need to understand whether they can reliably get that capacity back before traffic returns. For voice, we’d evaluate P95/P99 latency at realistic concurrency, throughout longer conversations, and separately for cache hits and misses. We’d also test tool calling on the actual serving configuration. Voice agents need predictable response times throughout a session. That’s a key consideration when choosing and sizing a dedicated endpoint. If you’re running voice agents in production, which model and provider are you using, and how does latency hold up under load? Hardware, peak concurrency and P95 figures would be especially useful, if you can share them.