Building voice AI agents that take turns like humans — the gotchas nobody warns you about

Spent months building real-time voice AI agents — 1:1 personas and a multi-agent setup where several agents run a social deduction game. Lessons that cost me real time and money: Turn-taking is the whole game. Stop the instant a human speaks, wait for real silence, reply in short turns. Monologues kill it. "getUserMedia succeeded" ≠ audio flowing. OS mute keeps the track silent, VAD never fires, agent sits stuck on "listening." Measure RMS, don't trust the permission. Muting the mic track does NOT stop billing on a server-side Realtime API. VAD runs on the model server. You have to turn off turn detection in a session update to actually pause it. Never feed the agent's own TTS back into STT. Echo and self-listening loops are instant death. Filter taps, breathing, mobile feedback too. Role should change with the room. Active in 1:1, mostly quiet in a group — step in only on silence or when invited. For multi-agent orchestration, don't let models free-run. An external orchestrator that owns whose turn it is beats agents deciding among themselves. Still messy for me: barge-in and false-interrupt filtering on mobile. How do you handle it?