@iluciddreaming: The four-layer architecture of voice assistants: VAD → Speech Recognition → LLM → Speech Synthesis. These four layers often come from different service providers. Wanting to replace one of them often means rewriting the entire pipeline. Hugging Face open-sourced speech-to-speech, modularizing these four layers so you can swap them out like components…
Summary
Hugging Face has open-sourced a modular speech-to-speech tool, which separates the four layers (VAD, speech recognition, LLM, and speech synthesis) and supports replacement, allowing users to flexibly swap components or run locally without rewriting the entire pipeline.
View Cached Full Text
Cached at: 07/13/26, 03:57 PM
Voice Assistant’s Four-Layer Architecture: VAD → Speech Recognition → LLM → Speech Synthesis. These four layers often come from different providers. Replacing one usually means rewriting the entire pipeline.
Hugging Face open-sourced speech-to-speech, modularizing these four layers so you can swap out individual components like parts.
• VAD, STT, LLM, and TTS are each replaceable — no lock-in to a single vendor • Swap OpenAI Realtime API for a local version without rewriting the entire logic • All models can run locally, so voice data never leaves your machine
Similar Articles
@FeitengLi: Actually, these problems can be well solved: 1. Ditch whisper, switch to an ASR model. Qwen3-ASR is great with few hallucinations, and there are other ASR options. Whisper has many hallucinations and requires 30s segments. Qwen3-ASR gets more accurate with longer audio, supporting up to 20…
Recommends using Qwen3-ASR instead of Whisper to reduce hallucinations, using LattifAI tools for precise audio-text alignment and subtitle generation, and introducing their own OmniVAD-Kit project for voice activity detection.
huggingface/speech-to-speech
Hugging Face's speech-to-speech is an open-source, modular voice-agent pipeline (VAD->STT->LLM->TTS) with an OpenAI Realtime-compatible WebSocket API, supporting swappable components and local or hosted models.
@FakeMaidenMaker: Explosive! This open-source project converts text to human-like voice for free, can clone anyone's voice, and adjust timbre with text! GitHub has garnered 30K stars, from Mianbao Intelligent OpenBMB, VoxCPM previously topped both GitHub and HuggingFace charts. Do...
VoxCPM2 is an open-source speech synthesis model from OpenBMB, using a tokenizer-free diffusion autoregressive architecture, supporting 30 languages, voice design, and controllable voice cloning. It can clone a voice with just one sentence, or create a brand new voice using text, outputting 48kHz high-quality audio, and is commercially usable.
@Chenzeze777: Found an open-source voice synthesis model that I just had to share. 2 billion parameters, trained on 2 million hours of data, supports 30 languages plus 9 Chinese dialects—just input text and it synthesizes speech, including Sichuanese, Cantonese, and Northeastern dialects. The craziest part? Use natural language to describe a voice—like "young female, gentle and sweet"—and it creates a brand-new voice from scratch without needing any reference audio.
Introducing an open-source voice synthesis model with 2 billion parameters and 2 million hours of training. It supports 30 languages and 9 Chinese dialects, allows voice description via natural language, can clone voices from a 3-second recording, delivers 48kHz studio-quality audio, and is free for commercial use under the Apache-2.0 license.
@cevenif: Bro, it's time to say goodbye to those paid voice tools! The open-source and free Voicebox has arrived, completely crushing paid giants like ElevenLabs and WisprFlow. Features: Voice cloning - instantly become anyone, Global voice input - accessible anytime...
An open-source, free local voice AI studio that supports voice cloning, voice generation, and global dictation. No API key required, runs entirely locally, and serves as a free alternative to ElevenLabs and WisprFlow.