FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
Summary
Introduces FunAudioLLM, a model family that combines SenseVoice for multilingual ASR, emotion detection, and audio event detection with CosyVoice for natural speech generation, enabling enhanced voice interactions with LLMs through open-source models and code.
View Cached Full Text
Cached at: 07/15/26, 04:21 PM
Paper page - FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
Source: https://huggingface.co/papers/2407.04051
Abstract
FunAudioLLM enhances voice interactions by integrating SenseVoice for multilingual speech recognition, emotion detection, and audio event detection with CosyVoice for natural speech generation across languages, timbres, and styles.
This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition,emotion recognition, andaudio event detection; and CosyVoice, which facilitatesnatural speech generationwith control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels inmulti-lingual voice generation, zero-shot in-context learning,cross-lingual voice cloning, andinstruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such asspeech-to-speech translation,emotional voice chat,interactive podcasts, andexpressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM.
View arXiv pageView PDFGitHub22.2kautoAdd to collection
Get this paper in your agent:
hf papers read 2407\.04051
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### omote-ai/sensevoice-asr UpdatedMar 6
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2407.04051 in a dataset README.md to link it from this page.
Spaces citing this paper1
Collections including this paper15
Similar Articles
Audio perception layer for LLM agents, with a memory that grows through use
An experimental open-source framework that enables LLM agents to perceive and recognize non-speech audio events using local models (CLAP, Whisper, Silero VAD) and a growing concept memory. The system uses event-gated recognition, fingerprinting, and symbol-based reasoning, with no formal benchmarks yet.
@paulabartabajo_: Advice for AI engineers If you're building voice agents, stop wiring up 3 separate models, for audio-to-text, text-to-a…
Announces liquid-audio, an open-source repository for Liquid AI's end-to-end speech-to-speech LFM models (LFM2-Audio-1.5B and LFM2.5-Audio-1.5B) with interleaved and sequential generation modes and fine-tuning support.
Liberating LLM Capabilities in Full-Duplex Speech Models
Proposes Listen-Write-Speak (LWS), a text-first tri-channel paradigm that allows a single autoregressive LLM to continuously listen, write visible text, and speak in real-time, enabling full-duplex speech interaction without architectural modifications.
A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
A comprehensive survey reviewing the trustworthiness challenges of Large Audio Language Models (LALMs), including vulnerabilities like cross-modal jailbreaking and acoustic backdoors, and proposing a defense-in-depth roadmap.
Open-LLM-VTuber/Open-LLM-VTuber
Open-LLM-VTuber is an open-source voice-interactive AI companion with a Live2D avatar, supporting real-time conversations and visual perception, fully operable offline.