MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

# MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

Source: https://arxiv.org/html/2511.10262

He Zhang¹, Wenqian Cui²¹, Haoning Xu², Xiao-Hui Li³, Lei Zhu³, Haoli Bai³, Shaohua Ma¹², Irwin King²

¹Tsinghua University, ²The Chinese University of Hong Kong, ³Huawei Technologies

## Abstract

Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Additionally, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark.

Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench

## 1 Introduction

Speech Language Models (SLMs) are foundation models designed for seamless, end-to-end interaction with users through speech. These models excel in comprehending various aspects of user speech, including meaning, prosody, voice characteristics, and in generating appropriate responses. Most SLMs are built to support half-duplex communication (HD-SLMs), wherein the model first "listens" to a complete user query and then "speaks" the corresponding response. A more advanced subset of SLMs, known as Full-Duplex Speech Language Models (FD-SLMs), specializes in enabling full-duplex communication. This capability allows the model to concurrently listen and speak during real-time interactions, fostering complex conversational features, such as interruptions and backchannels, thereby creating a more dynamic and engaging user experience.

When evaluating FD-SLMs, existing benchmarks primarily assess the model's ability to interact in a single round of interaction. For instance, Full-Duplex-Bench generates scenarios where the user engages in continuous speech for one turn, pauses mid-turn, or interrupts the assistant. Full-Duplex-Bench v1.5 builds on this by introducing speech inputs with overlapping interruptions, backchannels, and similar dynamics. However, real-world conversations often unfold in a multi-round format, and evaluating FD-SLMs in such settings is crucial to ensure the model consistently delivers timely and high-quality speech even after multiple rounds of communication.

Multi-round communication evaluation for FD-SLMs is challenging, mainly due to the following features in full duplex dialogues:

1) **Blurred Turn Boundary**. Unlike HD-SLMs, where conversations are naturally organized in a turn-by-turn manner, full duplex communication occurs spontaneously, without adhering to a strict turn-taking structure. Consequently, there are no clear indicators marking the start or end of each speaking turn for either the user or the assistant. This lack of defined boundaries makes it difficult to assess the response quality for a specific communication round.

2) **Context Inconsistency**. In natural full-duplex communication datasets, the user's speech in each round depends on the ground-truth assistant's responses from previous rounds. However, during evaluation, the model's responses to earlier rounds may diverge significantly from the ground truth, creating a context mismatch. This inconsistency leads the model to operate in scenarios that would never occur in real-world conversations, ultimately diminishing the reliability of the evaluation.

Another notable limitation of existing benchmarks is their predominant focus on evaluating FD-SLMs' ability to manage various conversational features, while largely overlooking other critical capabilities of the model, such as instruction following and ensuring safe outputs. These capabilities need to be evaluated accordingly since they pose unique challenges to FD-SLMs. For example, if the user keeps interrupting the model for multiple rounds, can the model still maintain expected behaviors (e.g., properly follow instructions)?

| Feature | MTR-DuplexBench | Other Benchmarks |
|---------|-----------------|-----------------|
| Multi-round Dialogue | ✓ | |
| Turn-by-turn Evaluation | ✓ | |
| Conversational Features | ✓ | ✓ |
| Dialogue Quality | ✓ | |
| Instruction Following | ✓ | |
| Safety | ✓ | |

To address the challenges and limitations mentioned above, we introduce a novel benchmark, MTR-DuplexBench, designed to comprehensively evaluate FD-SLMs in a multi-round communication setting. First, our benchmark employs an innovative methodology to segment continuous full-duplex dialogues into discrete turns, enabling effective evaluation of FD-SLMs for each turn separately. This approach directly tackles key challenges of the blurred turn boundary and context inconsistency. Second, our benchmark provides a more comprehensive evaluation framework compared to existing benchmarks, encompassing aspects such as dialogue quality, conversational features, instruction following, and safety. This holistic approach ensures thorough and reliable assessments of FD-SLMs.

To summarize our contributions:

1. We propose a novel benchmark, MTR-DuplexBench, for comprehensive evaluation of FD-SLMs in multi-turn interactions, covering conversational features, dialogue quality, instruction following, and safety.

2. We introduce a turn segmentation methodology for segmenting continuous full-duplex dialogues into discrete turns, enabling turn-by-turn evaluations of FD-SLMs.

3. We demonstrate through experiments that existing FD-SLMs struggle to maintain consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our proposed benchmark.

## 2 Related Works

### 2.1 Speech Language Models

Speech Language Models (SLMs) refer to speech-based autoregressive foundation models that enable end-to-end spoken dialogue interactions. As mentioned before, SLMs can be categorized into HD-SLMs and FD-SLMs.

**Half Duplex Speech Language Models (HD-SLMs)** refer to SLMs that enable "listening" and "speaking" in a turn-by-turn fashion. In this setup, the user provides a complete query to the model, which then generates the corresponding spoken response. Therefore, HD-SLMs are designed to handle the concatenated sequence comprising the user query and the assistant's response. HD-SLMs are typically continually trained upon existing Text Large Language Model (TLM) checkpoints through a two-stage process:

1) **Pre-training**. In this initial stage, the model uses extensive speech data for autoregressive speech generation. This can involve unsupervised learning with pure speech data or text-speech alignment pairs to transfer knowledge from text to speech.

2) **Instruction Tuning**. After pre-training, the model is trained on speech instruction tuning datasets, enabling it to respond effectively to spoken queries. During inference, some HD-SLMs respond in speech directly, while others first generate text responses as guidance and then produce the corresponding speech.

**Full Duplex Speech Language Models (FD-SLMs)** refer to SLMs that enable "listening" and "speaking" simultaneously to support advanced conversational features such as interruption and backchannel. FD-SLMs are typically built upon HD-SLMs, and they can be divided into two categories:

1) **Cascaded FD-SLMs** divide the full duplex dialogues into different states. The SLM then decides to listen or speak based on the current dialogue state. Cascaded FD-SLMs are typically trained on synthetic full duplex data to enable simple conversation features like interruption.

2) **End-to-end FD-SLMs**, on the other hand, directly learn from real-world full duplex data to enable more natural, human-like conversational dynamics.

### 2.2 Full Duplex Benchmarks

Full duplex benchmarks focus on evaluating FD-SLMs on various features related to full duplex communications. Most full duplex benchmarks focus on evaluating one round of spoken interactions. Specifically, Full-Duplex-Bench devises one round of spoken prompts to evaluate FD-SLM's ability for pause handling, backchanneling, smooth turn taking, and user interruptions. Full-Duplex-Bench v1.5 extends the evaluation scenarios to overlapping speech, but is still limited to one round of interaction.

Other full-duplex benchmarks incorporate multi-round spoken prompts but prioritize overall model response quality over individual round performance. They also focus on single scenarios, limiting comprehensive assessments of FD-SLM's multi-round interaction capabilities. For example, FD-Bench uses up to five rounds of user prompts, mainly addressing user interruptions with synthetic speech. In contrast, our benchmark provides diverse evaluations for multi-round full-duplex communication using both natural and synthetic dialogue data.

Similarly, Talking Turns includes multi-round data but solely predicts turn-taking timings without round-by-round dialogue evaluation. Moreover, their data collection relies on human interactions with various models, making it resource-intensive to scale and adapt to new models. Our benchmark, however, implements an automated evaluation pipeline with round-by-round metrics, removing the need for human intervention.

Additionally, concurrent work evaluates multi-round full-duplex interactions across four scenarios (Daily, Correction, Entity Tracking, and Safety), focusing on task-level outcomes instead of turn-level behaviors. It utilizes a separate Speech LLM as an automated examiner, which does not guarantee consistent interaction data for the same task, limiting evaluation stability. In contrast, our benchmark uses the same evaluation data for each round, allowing for stable assessments and fair model comparisons.

## 3 MTR-DuplexBench

This section illustrates the technical details of our proposed MTR-DuplexBench benchmark. We start by introducing the full duplex turn segmentation methodology, which segments continuous full duplex dialogues into discrete turns to enable turn-by-turn evaluation. Subsequently, we outline the overall structure of the benchmark, encompassing all evaluation dimensions.

### 3.1 Full Duplex Turn Segmentation Methodology

The full-duplex turn segmentation methodology is designed to address the challenges of blurred turn boundaries and context inconsistency in real-world full duplex dialogues and enable turn-by-turn evaluations. It contains an algorithm to identify the start and end points of each user turn boundary within a full-duplex dialogue. Then, the appropriate time period for the assistant's response is determined accordingly. We provide an overview of the algorithm below.

The turn segmentation algorithm addresses the blurred turn boundary challenge and can be divided into four steps:

1) **Information Extraction**. Initially, given speech audio from both channels A_c where c ∈ {user, assistant}, we employ the Whisper-timestamped package in conjunction with Silero Voice Activity Detection (VAD) and the medium-sized Whisper model. These tools extract the transcriptions and the start/end timestamps of each VAD segment, denoted as S^c ← Extract(A_c) = {(T_j^start, T_j^end, text_j^c)}_j=1^n. The VAD segments are fragmented and typically co-occur due to the overlapping nature of full-duplex communication.