Paper page - VoxMind: An End-to-End Agentic Spoken Dialogue System

Source: https://huggingface.co/papers/2604.15710

Abstract

VoxMind enhances spoken dialogue models with agentic capabilities through a “Think-before-Speak” mechanism and dynamic tool management to improve task completion rates while maintaining conversational quality.

Recentend-to-end spoken dialogue modelsenable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporatingagentic capabilitiesis therefore essential: by enablingtool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equipend-to-end spoken dialogue modelswith comprehensive agentic abilities. Leveraging our curated 470-hourAgentChat dataset, we incorporate a “Think-before-Speak” mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose aMulti-Agent Dynamic Tool Managementarchitecture. By asynchronously delegatingretrieval tasksto an auxiliary agent aligned with the main model’s reasoning trajectory, this system effectively decouplesinference latencyfrom toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, thetask completion rateincreases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving generalconversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.

View arXiv pageView PDFGitHub21Add to collection

Get this paper in your agent:

hf papers read 2604\.15710

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### leungtianle/VoxMind 8B• Updatedabout 7 hours ago • 17 • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.15710 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.15710 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.