A team shares how they built a production voice-agent stack using self-hosted Pipecat, Mai Transcribe 2, a Sarvam conversational LLM, and a Malayalam fine-tuned VoxCPM2 TTS, achieving 20 concurrent low-latency voice streams on a single L4 GPU with an automated WhatsApp post-call handoff.
Most voice agent setups work great in English, but trying to get natural-sounding, low-latency conversational AI in regional Indian languages like Malayalam is a massive headache. Standard off-the-shelf models either sound robotic, mess up the pronunciation, or cost way too much per minute to make sense for local businesses. Over the past few months, my team and I built a production voice stack from the ground up to solve this, and I wanted to share our architecture for anyone else working on non-English or custom voice pipelines. Here is the exact stack we are running right now: Orchestration: Self-hosted Pipecat worker setup. It handles interruptions and state management cleanly without the overhead of paid wrapper platforms. STT: Mai Transcribe 2. It has been surprisingly accurate at picking up regional accents and fast speech without lagging the pipeline. LLM: A Sarvam conversational model tuned for natural dialogue flow, complete with custom objection handling and clean fallback routing to human agents when a lead needs to escalate. TTS & Hardware: We fine-tuned VoxCPM2 specifically for Malayalam so the cadence and tone actually sound like a local speaker. We optimized this to run 20 concurrent voice generations on a single L4 GPU. Post-Call Workflow: Because we are an official Meta Tech Partner, we connected the agent directly to the WhatsApp API. The second a caller expresses interest on the phone, the agent triggers an automated WhatsApp message with their custom quotation before they even hang up. The hardest part was definitely getting the TTS fine-tuning right and squeezing 20 concurrent streams out of a single L4 without latency spikes, plus getting the voice pipeline and WhatsApp handoff to feel instantaneous. Because local pricing here in India is super cheap, our margins on domestic clients are pretty thin even with our low compute costs. However, this exact self-hosted architecture works for English and other languages too. I'm currently looking to connect with a few agency owners or sales-focused folks outside of India—especially in the US, UK, or the Gulf where there is either a huge Malayalam-speaking diaspora or a demand for high-margin custom voice deployments. I love handling the backend engineering, fine-tuning, and fulfillment, so if anyone is great at client acquisition and wants to team up on a 50-50 profit split for international clients, I'm always down to chat and share some voice demos. Happy to answer any technical questions about the Pipecat setup, the L4 optimization, or how we handled the VoxCPM2 fine-tuning in the comments!
A tutorial demonstrating a low-latency voice agent setup with Pipecat PhoneLLM Alpha 1 on Modal, using Deepgram transcription and Cartesia voice, with code and video guides.
A developer shares a postmortem of building a Hindi-English voice agent for fintech, highlighting challenges with number readback, code-mixed TTS, latency under load, and compliance. Key fix was choosing TTS with first-class support for Indian code-mixing and testing at real concurrency.
NVIDIA announces Magpie Multilingual TTS, an open-weights text-to-speech model supporting 12 languages with low-latency deployment via NVIDIA NIM for building production voice agents.
OpenAI details its rearchitected WebRTC stack designed to deliver low-latency voice AI at scale for over 900 million users. The post explains how new split-relay and transceiver architectures optimize media routing and connection setup for real-time interactions like ChatGPT voice.
A practitioner roundup of which LLMs teams are running for real-time voice agents, covering dedicated vs. serverless trade-offs (Gemma 4 31B, Qwen3 235B, GLM 5.2, Nemotron 3 Ultra, GPT-4.1/5.6, Gemini Flash), along with deployment lessons on measuring latency, caching, tool testing and idle GPU cost.