Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue

# Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue
Source: [https://arxiv.org/html/2609.03321](https://arxiv.org/html/2609.03321)
Chenhui ChuAffiliation:liyh@nlp\.ist\.i\.kyoto\-u\.ac\.jp,chu@i\.kyoto\-u\.ac\.jp

###### Abstract

The Neural Finite State Machine \(NFSM\) framework offers a pragmatic path to full\-duplex dialogue by serializing turn\-taking control and response generation onto a single causal tape under the standard next\-token prediction objective, thereby preserving semantic prowess at a low fine\-tuning cost\. However, its reliance on synthetic text data fundamentally limits turn\-taking naturalness, as Large Language Models \(LLMs\) cannot faithfully simulate the fine\-grained acoustic temporal dynamics of real human dialogues\. In this work, we propose a decoupled data approach that learns turn\-taking from real Human\-Human \(HH\) spoken dialogues while shaping semantic behavior through configurable Human\-Agent \(HA\) text dialogues\. To operationalize this approach, we introduce a rule\-based event\-guided data transformation method that serializes HH spoken dialogues into FSM tapes by classifying turn\-taking events and applying deterministic mapping rules, enabling scalable supervision without LLM\-generated annotations\. We further propose a Source\-Aware Calibrated \(SAC\) Loss that jointly calibrates the long\-tailed distribution of state transition tokens and channels each data source toward the capability it best supervises\. Experiments show that our approach substantially improves turn\-taking proficiency while recovering the foundation LLM’s semantic capability\. Our code and model are available at[https://github\.com/Liyht/def\-fsm](https://github.com/Liyht/def-fsm)\.

## 1Introduction

Recent advances in Large Language Models \(LLMs\) have greatly enhanced the semantic capabilities of voice assistants[Ji et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib32)\. However, their turn\-taking naturalness remains constrained by rigid half\-duplex turn\-taking, in which each party must wait for the other to finish before speaking\. To bridge the gap toward truly human\-like interaction, the field is moving toward full\-duplex dialogue[Chen and Yu \(2025\)](https://arxiv.org/html/2609.03321#bib.bib19), where an agent can listen and speak simultaneously and thereby handle interruptions, acoustic backchannels, and dynamic turn\-taking in real time\.

![Refer to caption](https://arxiv.org/html/2609.03321v1/abstract.png)Figure 1:Comparison between the coupled and decoupled data approaches\.Current research on full\-duplex dialogue falls into two primary paradigms: end\-to\-end and modular\. While the end\-to\-end paradigm[Nguyen et al\. \(2023\)](https://arxiv.org/html/2609.03321#bib.bib21);[Défossez et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib20);[Veluri et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib30);[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib29);[Lee et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib22)excels in fine\-grained turn\-taking control and low latency by directly understanding and generating speech representations, enforcing strict cross\-modal alignment often causes significant degradation in the underlying LLM’s semantic capabilities[Xiang et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib7);[Chen et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib8)\. The modular paradigm instead equips an LLM with auxiliary state management via external controllers[Liao et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib26), classification heads[Wang et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib27);[Chen et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib28), or in\-vocabulary FSM control tokens[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.03321#bib.bib1)\. Among these approaches, the NFSM[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.03321#bib.bib1)occupies a uniquely minimal position: rather than introducing any auxiliary module, it serializes both state transitions and response content onto a single causal “tape,” handling full\-duplex control entirely under the standard next\-token prediction objective with no added parameters\. Moreover, it enables the LLM to explicitly perceive and condition on the full history of prior turn\-taking dynamics when making subsequent decisions\. However, this minimalism places the full burden of turn\-taking competence on the tape’s textual content—a burden that the original NFSM’s data approach cannot adequately meet\.

Specifically, as illustrated in Figure[1](https://arxiv.org/html/2609.03321#S1.F1)\(left\), NFSM constructs training tapes from LLM\-generated transcripts, attempting to distill complex turn\-taking dynamics and rich semantics simultaneously\. However, as foundation LLMs are predominantly pre\-trained on structured non\-overlapping text, they lack the inherent ability to simulate the fine\-grained acoustic and temporal dynamics of real human interactions\. Worse still, jointly controlling realistic turn\-taking and rich semantics within a single generated dialogue compounds this difficulty, making such synthetic data fundamentally unreliable as supervision\.

To address this challenge, we advocate a decoupled data approach, as shown in Figure[1](https://arxiv.org/html/2609.03321#S1.F1)\(right\), that aligns each capability with its most suitable data source: fine\-grained turn\-taking dynamics are acquired from real HH spoken dialogue corpora, while readily available HA text dialogues serve as a flexible lever for shaping semantic behavior\. The central challenge in operationalizing this approach lies in transforming real HH spoken dialogues into NFSM’s serialized tape as ground truth, in which state transition tokens must be inserted to faithfully reflect complex acoustic turn\-taking dynamics\. To this end, we propose an*event\-guided data transformation*strategy that segments and classifies the dialogue timeline into discrete turn\-taking events and subsequently applies deterministic mapping rules to serialize each segment into the tape\. Our core insight is that state transition token insertion acts as an approximation of the turn\-taking dynamics, while turn\-taking dynamics equate to the temporal sequence of turn\-taking events\. Crucially, the entire pipeline is fully rule\-based and free of LLM annotations, making it readily scalable to arbitrarily large HH corpora\. On the HA side, the corpus can be freely substituted to target specific semantic objectives\. In this work, we adopt a generic HA corpus to preserve the LLM’s intrinsic semantic capabilities, leaving capability\-specific adaptation as a natural extension\.

While this decoupled data approach provides clean supervision signals, jointly training on the resulting heterogeneous tapes introduces two challenges\. First, state transition tokens exhibit an extremely long\-tailed distribution, in which rare yet operationally critical state\-switch tokens are easily marginalized under standard cross\-entropy training\. Second, the two data sources carry asymmetric supervisory value: HH tapes are authoritative for turn\-taking but noisy for semantics, whereas HA tapes carry reliable semantics but trivial turn\-taking structure\. To address both issues within a single objective, we introduce the Source\-Aware Calibrated \(SAC\) Loss, which combines logit adjustment over the restricted vocabulary of state transition tokens with a source\-aware weighting scheme that channels each data source toward the capability it best supervises\.

Experiments under strict token\-volume constraints on HH spoken dialogue corpora and the VoiceBench dataset show that our model achieves substantially better turn\-taking proficiency and semantic capability than the conventional synthetic\-data baseline, with SAC further mitigating the severe class imbalance\. When scaled to a larger training corpus constructed by proportionally upsampling each data source to preserve the mixture ratio, our approach successfully recovers the intrinsic semantic capability of the foundation LLM\.

In summary, our contributions are threefold:

- •We propose a decoupled data approach for FSM\-based full\-duplex dialogue, in which turn\-taking is learned from real HH spoken dialogues and semantic capability is independently shaped through configurable HA data\.
- •We design a fully rule\-based event\-guided transformation that serializes complex acoustic interactions into causal FSM tapes, enabling scalable construction of real HH dialogue data\.
- •We introduce the SAC Loss, which calibrates the long\-tailed distribution of state transition tokens and specializes optimization across data sources\.

## 2Related Work

Turn\-Taking Event Classification\.Turn\-taking event classification is a labeling scheme that constructs ground\-truth event labels along the dialogue timeline, supporting downstream tasks such as event prediction and analysis[Patamia et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib33);[Castillo\-López et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib18);[Heldner and Edlund \(2010\)](https://arxiv.org/html/2609.03321#bib.bib17);[Arora et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib13);[Threlkeld et al\. \(2022\)](https://arxiv.org/html/2609.03321#bib.bib34)\. Existing schemes can be organized by the temporal unit on which events are defined\. Chunk\-based methods segment the timeline into equal\-length frames and assign per\-frame labels[Heldner and Edlund \(2010\)](https://arxiv.org/html/2609.03321#bib.bib17);[Ekstedt and Skantze \(2022\)](https://arxiv.org/html/2609.03321#bib.bib14);[Arora et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib13), offering fine temporal resolution suited to frame\-based neural architectures\. Inter\-Pausal Unit \(IPU\)\-based methods instead define events at IPU boundaries delimited by silence thresholds[Ekstedt and Skantze \(2020\)](https://arxiv.org/html/2609.03321#bib.bib15);[Wang et al\. \(2024a\)](https://arxiv.org/html/2609.03321#bib.bib16), yielding linguistically coherent units that align cleanly with textual chunk boundaries\. We adopt the IPU\-based formulation, as this property makes rule\-based serialization onto the FSM tape straightforward\.

Data Strategies for Full\-Duplex Supervision\.Full\-duplex supervision data are constructed along two distinct paths\. The first relies on*synthetic generation*: an LLM is prompted to produce dialogues annotated with designated turn\-taking control tokens[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.03321#bib.bib1), or dialogues are assembled through rule\-based procedures such as inserting user interruptions at predefined positions[Xie and Wu \(2024\)](https://arxiv.org/html/2609.03321#bib.bib31)\. The fundamental difficulty is that LLMs are trained predominantly on structured non\-overlapping text and therefore lack the capacity to simulate the fine\-grained turn\-taking dynamics of real human conversation\. The second path draws on*real spoken dialogue corpora*\. End\-to\-end models leverage such corpora through implicit supervision, directly learning from raw interleaved audio tokens without explicit state labels[Nguyen et al\. \(2023\)](https://arxiv.org/html/2609.03321#bib.bib21);[Défossez et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib20)\. Modular approaches instead apply voice activity detection and heuristic rules to derive state labels from real audio, training a dedicated state classifier on fixed acoustic chunks[Liao et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib26);[Chen et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib28)\. Our setting poses a qualitatively different challenge: rather than classifying states over fixed acoustic chunks, we must insert discrete state transition tokens into an interleaved text token stream, which is a form of supervision that cannot be derived from voice activity alone and demands explicit turn\-taking event identification\.

![Refer to caption](https://arxiv.org/html/2609.03321v1/transformation.png)Figure 2:Example of turn\-taking event classification and event\-guided tape serialization\. Green, blue, and orange blocks denote user content, agent content, and state transition tokens \(shown in abbreviated form, e\.g\.,\[C\.L\]for\[C\.LISTEN\]\), respectively\. The<user\>role\-prefix token preceding each user block is omitted for brevity\.
## 3Data Transformation

Building on the NFSM framework, we serialize two complementary corpora into a single causal FSM tape format: real HH spoken dialogues for fine\-grained turn\-taking dynamics, and HA text dialogues adapted to a spoken style for preserving semantic capability\. We first briefly review NFSM, then describe our transformation pipelines for both corpora\.

### 3\.1Preliminary: The NFSM Framework

NFSM[Wang et al\. \(2024b\)](https://arxiv.org/html/2609.03321#bib.bib1)casts dialogue turn\-taking as an explicit state transition problem managed directly by an LLM operating a two\-state machine consisting of a`SPEAK`state and a`LISTEN`state\. The transitions are realized through vocabulary tokens:`\[C\.SPEAK\]`and`\[C\.LISTEN\]`for continuing the current state, and`\[S\.SPEAK\]`and`\[S\.LISTEN\]`for switching to the opposite state\. Both state transitions and response generation are serialized onto a single causal tape under the standard next\-token prediction paradigm\. Crucially, each autoregressive decoding step on this tape is triggered by three types of events ranked in decreasing priority: the generation of speech\-driving tokens, the arrival of new perception chunks, or the completion of motor processing\. This unified sequence asynchronously synchronizes the perception and motor modules, allowing a standard text\-based LLM to control real\-time full\-duplex interaction without architectural modifications\.

### 3\.2Event\-Guided Human\-Human Data Transformation

Building upon our decoupled data approach, the primary objective of HH data transformation is to construct a rigorous ground\-truth tape that strictly mirrors human turn\-taking behavior\. Our core insight is that state transition token insertion acts as an approximation of the turn\-taking dynamics, while the turn\-taking dynamics is equivalent to the temporal sequence of turn\-taking events\. Consequently, we propose an event\-guided transformation strategy\. Our conversion strategy is guided by two principles: \(1\) categorizing the timeline into discrete turn\-taking events to direct token insertion, and \(2\) aligning with the actual inference\-time conditions the FSM will encounter\.

Asymmetric Preprocessing\.The two channels of each HH dialogue are designated as the user channel and the agent channel, and processed asymmetrically to mirror NFSM’s actual inference\-time conditions as shown in Figure[2](https://arxiv.org/html/2609.03321#S2.F2)\. The user channel is transcribed by the same perception module deployed in the FSM, producing fine\-grained timestamped text chunks together with silence tokens of fixed duration, thereby exposing the LLM to realistic ASR granularities and recognition error patterns\. The agent channel, in contrast, uses fine\-grained timestamped ground\-truth transcripts, ensuring the LLM learns to generate clean text rather than imitate ASR artifacts\. Chunks are kept as fine\-grained as possible—ideally word\-level—to enable precise event segmentation downstream\. Additional preprocessing details are provided in Appendix[B\.2](https://arxiv.org/html/2609.03321#A2.SS2)\.

EventInitiator = UserInitiator = AgentT`\[S\.LISTEN\.N\]`<User\>`\[S\.SPEAK\]`<Agent\>C`\[C\.LISTEN\]`<User\>`\[C\.SPEAK\]`<Agent\>P`\[C\.LISTEN\]`<SIL\>\-G`\[C\.LISTEN\]`<SIL\>`\[S\.LISTEN\.N\]`<SIL\>BC`\[C\.SPEAK\]`<Agent\> <User\>`\[C\.SPEAK\]``\[S\.SPEAK\]`<Agent\> <User\>`\[S\.LISTEN\.N\]`FTI`\[C\.SPEAK\]`<Agent\> <User\>`\[S\.LISTEN\.I\]``\[S\.SPEAK\]`<Agent\> <User\>`\[C\.SPEAK\]`BI`\[C\.SPEAK\]`<Agent\> <User\>`\[C\.SPEAK\]``\[S\.SPEAK\]`<Agent\> <User\>`\[S\.LISTEN\.I\]`Table 1:Mapping rules for event\-guided tape serialization across 14 combinations of turn\-taking events and initiators\. <User\> and <Agent\> denote textual content from the user and agent channels; <SIL\> represents the silence token\.Turn\-Taking Event Classification\.This stage partitions the continuous dialogue timeline and assigns each segment to one of seven turn\-taking event types, as illustrated in Figure[2](https://arxiv.org/html/2609.03321#S2.F2)\(top\)\. We first merge adjacent chunks within each channel into IPUs using a predefined pause threshold; the union of IPU boundaries across both channels then yields a discrete timeline\. Following[Arora et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib13), we classify segments as:

- •Single\-party speech\.Turn Change \(T\)if the floor holder differs from the previous segment;Continuation \(C\)otherwise\.
- •Silence\.Pause \(P\)within a single speaker’s turn;Gap \(G\)between alternating turns\.
- •Overlapping speech\.Backchannel \(BC\)if the utterance matches a predefined lexical list and falls below a duration threshold[Ekstedt and Skantze \(2020\)](https://arxiv.org/html/2609.03321#bib.bib15);[Wang et al\. \(2024a\)](https://arxiv.org/html/2609.03321#bib.bib16);[Arora et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib13);Floor\-Taking Interruption \(FTI\)if the interrupter successfully takes the floor; orButting\-in \(BI\)if the attempt fails\.

Event\-Guided Tape Serialization\.This stage constructs the serialized tape representation for each classified segment\. First, we split`\[S\.LISTEN\]`into`\[S\.LISTEN\.N\]`\(Natural\) and`\[S\.LISTEN\.I\]`\(Interrupt\), granting the LLM finer\-grained control over the motor module to explicitly signal whether the ongoing TTS should complete naturally or be immediately truncated\. Next, we assign the text of each fine\-grained chunk to its corresponding segment based on start timestamp\. To formulate the mapping scheme, we define the*initiator*of a segment as the participant whose action triggers the transition from the preceding segment \(specifically, the preceding floor holder for gaps\)\. Combining the 7 event types with these 2 initiator types yields 14 deterministic mapping rules \(Table[1](https://arxiv.org/html/2609.03321#S3.T1); see Figure[2](https://arxiv.org/html/2609.03321#S2.F2)bottom for an example\)\. Finally, four operational heuristics govern the compilation of these rules into a continuous tape: \(1\) the governing state transition token immediately follows the first chunk of a segment, while subsequent chunks maintain the state via continuation tokens; \(2\) during overlaps, agent chunks precede user ASR chunks, reflecting the causal latency that an agent generates text prior to audio synthesis, whereas the perception module emits text post\-speech; \(3\) silence tokens are preserved exclusively within`LISTEN`intervals; and \(4\) consecutive state transition tokens at segment boundaries are collapsed into a single token via the rules in Appendix[A\.3](https://arxiv.org/html/2609.03321#A1.SS3)\.

Post\-Processing and Tape Validation\.To optimize TTS quality at inference, we merge and resegment continuous agent text into clause\-level granularity using a punctuation model[Guhr et al\. \(2021\)](https://arxiv.org/html/2609.03321#bib.bib2), which is the finest granularity that preserves prosodic quality\. This resegmentation is applied independently to each contiguous span of agent text with only state transition tokens between them, with`\[C\.SPEAK\]`tokens inserted at clause boundaries\. We then validate every resulting sequence against the FSM’s transition constraints \(e\.g\., the`SPEAK`state emits only`\[S\.LISTEN\.\*\]`or`\[C\.SPEAK\]`\) to ensure the completeness of the entire pipeline\.

### 3\.3Human\-Agent Data Transformation

To preserve the LLM’s intrinsic semantic abilities, we leverage broad\-domain HA text dialogues\. However, as textual HA datasets are primarily written\-style and thus unsuitable for a spoken dialogue agent, we convert them into the FSM tape format through a three\-stage pipeline: \(1\) rule\-based filtering, \(2\) LLM\-driven stylistic rewriting, and \(3\) final filtering and tape serialization\.

We first apply rule\-based cleaning to the raw text dialogues, which strips HTML markup, validates role alternation, and removes turns dominated by code, mathematical formulas, or excessively long contexts\. We then prompt an LLM with a two\-phase prompt \(Appendix[A\.5](https://arxiv.org/html/2609.03321#A1.SS5)\) that first judges suitability for spoken delivery, then rewrites the surviving dialogues into a natural spoken style\. A final filtering pass discards turns exceeding spoken\-utterance length thresholds or containing residual Markdown artifacts, yielding a distilled set of high\-quality dialogues\.

Finally, we segment user and agent utterances into clauses with the same punctuation model as in HH transformation, then insert`\[S\.SPEAK\]`and`\[S\.LISTEN\.N\]`between alternating utterances and`\[C\.LISTEN\]`/`\[C\.SPEAK\]`between clauses within the same utterance\. This establishes a trivial yet schema\-compliant turn\-taking structure, allowing the HA data to focus its supervisory signal on semantic content\.

## 4Model

### 4\.1FSM Modules

In the absence of a publicly available NFSM implementation, we build the entire system from scratch, integrating intrinsically multilingual perception, cognitive, and motor modules optimized for real\-time interaction\. To examine the effect of perception granularity, we evaluate two streaming ASR alternatives, both built on Whisper\-Turbo[Radford et al\. \(2022\)](https://arxiv.org/html/2609.03321#bib.bib11)to isolate granularity from backbone: a word\-level streaming ASR, SimulStreaming[Macháček and Polák \(2025\)](https://arxiv.org/html/2609.03321#bib.bib25), and an IPU\-level streaming ASR coupling Silero VAD[Team \(2024\)](https://arxiv.org/html/2609.03321#bib.bib5)with Faster\-Whisper\.111https://github\.com/SYSTRAN/faster\-whisperThis contrast allows us to explore the fundamental trade\-off between the precise temporal resolution afforded by fine\-grained streaming and the semantic coherence required for aggressively fragmented inputs\. For the motor module, we adopt Kokoro TTS,222https://github\.com/hexgrad/kokorowhose Time\-to\-First\-Audio on clause\-level inputs outperforms many streaming alternatives without sacrificing prosodic quality\. Other proposed techniques for FSM are available in Appendix[E](https://arxiv.org/html/2609.03321#A5)\.

### 4\.2Source\-Aware Calibrated Loss

Optimizing the FSM model presents two challenges: mitigating the severe class imbalance inherent in turn\-taking behaviors and jointly optimizing two distinct capabilities\. To this end, we propose the SAC Loss, a dual\-purpose objective designed to calibrate the long\-tail token distribution and decouple learning targets across diverse data sources\.

During fine\-tuning, we adopt a selective masking strategy\. Specifically, we mask the prompts and user text, restricting the loss computation exclusively to the agent response tokens and state transition tokens\. LetI=\{1,…,N\}I=\\\{1,\\dots,N\\\}denote the target token indices for a training instance\. We partitionIIalong two orthogonal dimensions: by token type into response tokens \(RR\) and state transition tokens \(TT\); and by data source into Human\-Human \(H​HHH\) and Human\-Agent \(H​AHA\) tokens\.

To achieve distributional calibration, we tackle the extreme class imbalance among state transition tokens\. Continuation tokens representing ongoing listening or speaking appear exponentially more frequently than discrete state\-switching tokens \(see Appendix[A\.2](https://arxiv.org/html/2609.03321#A1.SS2)for the distribution\)\. Employing a standard cross\-entropy objective naturally marginalizes these low\-probability yet operationally critical transitions\. Therefore, we incorporate Logit Adjustment[Menon et al\. \(2021\)](https://arxiv.org/html/2609.03321#bib.bib6)specifically for the state transition tokens to rectify this skewed prior, while retaining the standard cross\-entropy loss for the response tokens\. The per\-component losses for an inputxxand target labelyyare defined as follows:

ℓCE​\(x,y\)=−log⁡exp⁡\(fy​\(x\)\)∑vexp⁡\(fv​\(x\)\)\\ell^\{\\mathrm\{CE\}\}\(x,y\)=\-\\log\\frac\{\\exp\(f\_\{y\}\(x\)\)\}\{\\sum\_\{v\}\\exp\(f\_\{v\}\(x\)\)\}ℓLA​\(x,y\)=−log⁡exp⁡\(fy​\(x\)\+τ​log⁡πy\)∑vexp⁡\(fv​\(x\)\+τ​log⁡πv\)\\ell^\{\\mathrm\{LA\}\}\(x,y\)=\-\\log\\frac\{\\exp\\bigl\(f\_\{y\}\(x\)\+\\tau\\log\\pi\_\{y\}\\bigr\)\}\{\\sum\_\{v\}\\exp\\bigl\(f\_\{v\}\(x\)\+\\tau\\log\\pi\_\{v\}\\bigr\)\}wheref⁡\(⋅\)f\(\\cdot\)denotes the model logits,πy\\pi\_\{y\}is the prior probability of classyy, andτ\\tauis the adjustment temperature\. The priorπ\\piis computed exclusively over the restricted state\-transition vocabulary to avoid overweighting\. Accordingly, the lossℓi\\ell\_\{i\}for theii\-th token is formulated as:

ℓi=\{ℓCE​\(xi,yi\),if​i∈RℓLA​\(xi,yi\),if​i∈T\\ell\_\{i\}=\\begin\{cases\}\\ell^\{\\mathrm\{CE\}\}\(x\_\{i\},y\_\{i\}\),&\\text\{if \}i\\in R\\\\ \\ell^\{\\mathrm\{LA\}\}\(x\_\{i\},y\_\{i\}\),&\\text\{if \}i\\in T\\end\{cases\}
To achieve source\-aware specialization, we introduce a weighting mechanism that dynamically adjusts the importance of each token based on its type and origin\. The overall training lossℒ\\mathcal\{L\}is computed as a weighted average over all unmasked tokens:

ℒ=∑i∈Iwi​ℓi∑i∈Iwi\\mathcal\{L\}=\\frac\{\\sum\_\{i\\in I\}w\_\{i\}\\ell\_\{i\}\}\{\\sum\_\{i\\in I\}w\_\{i\}\}where the weightwiw\_\{i\}assigned to theii\-th token is defined as:

wi=\{α,if​i∈\(H​A∩R\)∪\(H​H∩T\)1−α,if​i∈\(H​A∩T\)∪\(H​H∩R\)w\_\{i\}=\\begin\{cases\}\\alpha,&\\text\{if \}i\\in\(HA\\cap R\)\\cup\(HH\\cap T\)\\\\ 1\-\\alpha,&\\text\{if \}i\\in\(HA\\cap T\)\\cup\(HH\\cap R\)\\end\{cases\}in whichα∈\(0\.5,1\]\\alpha\\in\(0\.5,1\]is a hyperparameter controlling the degree of source emphasis\. By assigning a higher weightα\\alphato the subsets\(H​H∩T\)\(HH\\cap T\)and\(H​A∩R\)\(HA\\cap R\), the SAC Loss explicitly decouples the optimization trajectories: it forces the model to learn realistic turn\-taking dynamics predominantly from human\-human interactions, while refining its semantic capabilities primarily through human\-agent data\.

## 5Experiments

### 5\.1Settings

For HH spoken dialogues, we adopt the Switchboard[Godfrey et al\. \(1992\)](https://arxiv.org/html/2609.03321#bib.bib4)and Fisher[Cieri et al\. \(2004\)](https://arxiv.org/html/2609.03321#bib.bib3)corpora\. For the HA text dialogues, we adopt the ShareGPT dataset333https://huggingface\.co/datasets/RyokoAI/ShareGPT52Kand utilize Qwen3\-32B\([Yang et al\., 2025](https://arxiv.org/html/2609.03321#bib.bib10)\)for stylistic rewriting\. As the foundation model, we adopt the open\-source lightweight Qwen3\-4B \(model scale analysis in Appendix[C\.1](https://arxiv.org/html/2609.03321#A3.SS1)\) and extend its tokenizer with state transition tokens, the silence token`<SIL\>`, and the role\-prefix token`<user\>`\. For the data mixture, we set HH:HA token\-volume ratio as1:11\{:\}1, with Fisher:Switchboard mixed at4:14\{:\}1within HH\. For the SAC loss, we setτ=1\\tau=1andα=0\.6\\alpha=0\.6\. Hyperparameter analyses are provided in Appendix[C\.1](https://arxiv.org/html/2609.03321#A3.SS1), while full training details, dataset splits, and First Token Emission Delay \(FTED\) latency are in Appendix[B\.1](https://arxiv.org/html/2609.03321#A2.SS1)and[D](https://arxiv.org/html/2609.03321#A4)\.

Considering the original training dataset of NFSM is not publicly available, we reproduce their training data to establish a baseline representative of the conventional synthetic data approach\. Specifically, GPT\-4\-Turbo[OpenAI et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib12)generates 1,500 synthetic dialogues using the published NFSM prompts, which we serialize via the same procedure as our HA transformation, with one exception—incomplete\-utterance markers \(`<NOT\_FINISHED\>`, trailing ellipses\) are mapped to`\[S\.LISTEN\.I\]`\. We then fine\-tune a separate Qwen3\-4B model on this synthetic tape, which serves as our primary baseline\. More details are available in Appendix[A\.1](https://arxiv.org/html/2609.03321#A1.SS1)\.

Our evaluation focuses on turn\-taking proficiency and semantic capability\. Turn\-taking is measured on HH corpora as the ground truth for natural dynamics: at each teacher\-forced prediction step, we treat the output as an independent binary classification for each state transition token type, compute per\-type F1 within each evaluation set, average across types within each set, then average across Switchboard and Fisher to obtain the reported F1\. For semantic evaluation, we adopt VoiceBench[Chen et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib8)following[Xiang et al\. \(2025\)](https://arxiv.org/html/2609.03321#bib.bib7), excluding IFEval to better match a purely acoustic system and averaging over SD\-QA, MMSU, OpenBookQA, and AdvBench subsets as a result\. All test inputs are prepended with the same FSM prompt\. For fine\-tuned models, evaluation audio is transcribed by their respective perception modules and interleaved with state transition tokens\. For the zero\-shot foundation model upper bound, audio is transcribed by Whisper\-Turbo into continuous plain text, preventing degradation from context fragmentation\.

Faster\-WhisperSimulStreaming\(IPU\-level\)\(word\-level\)ModelF1↑\\uparrowVB↑\\uparrowF1↑\\uparrowVB↑\\uparrowNFSM0\.343653\.570\.302544\.24Ours0\.649862\.140\.640454\.80Δ\\Deltavs\. NFSM\+0\.3062\+8\.57\+0\.3379\+10\.56Table 2:Overall performance across method and perception module, where VB denotes VoiceBench\.![Refer to caption](https://arxiv.org/html/2609.03321v1/standard_confusion_matrix_best_loss_model_assistant_switchboard.png)

![Refer to caption](https://arxiv.org/html/2609.03321v1/ladjust_confusion_matrix_best_loss_model_assistant_switchboard.png)

Figure 3:Confusion matrices for state transition and response tokens on Switchboard with the Faster\-Whisper\-based model: standard cross\-entropy \(left\) versus SAC loss \(right\)\.
### 5\.2Main Results

To ensure fair comparison, we control the total training token volume for our models to match the NFSM baseline\. As shown in Table[2](https://arxiv.org/html/2609.03321#S5.T2), our model significantly outperforms NFSM under both perception configurations\. Comparing the two variants of our model, the Faster\-Whisper variant achieves turn\-taking performance comparable to the SimulStreaming variant while attaining substantially higher semantic capability\. This gap stems from the hyper\-granular nature of SimulStreaming outputs: user textual content is heavily fragmented by frequently interleaved state transition tokens, challenging the LLM’s semantic comprehension \(details in Appendix[A\.2](https://arxiv.org/html/2609.03321#A1.SS2)\)\. These findings reveal a critical architectural trade\-off in perception module design: finer granularity offers more frequent opportunities to manage interruptions but risks degrading semantic prowess\.

ConfigurationF1↑\\uparrowVB↑\\uparrowOurs0\.649862\.14Ablation on Loss Functionw/o SAC loss0\.610562\.19Ablation on Training Data \(w/o SAC loss\)HH only0\.618218\.30HA only0\.417063\.15Synthetic only \(NFSM\)0\.343653\.57Table 3:Ablation study on the SAC loss and training data composition under Faster\-Whisper perception\.ConfigurationF1↑\\uparrowVB↑\\uparrowSemantic Upper BoundQwen3\-4B \(zero\-shot\)0\.120064\.93End\-to\-End ReferenceMoshi—27\.36Our FSM ModelToken\-matched subset0\.649862\.14Proportionate upsampling0\.653964\.84Table 4:Performance of our FSM model scaled to the proportionate upsampling mixture, alongside the zero\-shot semantic upper bound and end\-to\-end reference\.Full\-Duplex\-Bench v1\.0Full\-Duplex\-Bench v1\.5Pause HandlingBackchannelSmooth TurnUser InterruptionOverlap HandlingDataSyntheticCandorICCCandorSyntheticU\-INTRU\-BCT\-OTHBKGModelTOR↓\\downarrowTOR↓\\downarrowTOR↓\\downarrowFreq↑\\uparrowJSD↓\\downarrowTOR↑\\uparrowTOR↑\\uparrowGPT\-4o↑\\uparrowRESP↑\\uparrowRSM↑\\uparrowRSM↑\\uparrowRSM↑\\uparrowMoshi0\.9850\.9801\.0000\.0010\.9570\.9411\.0000\.7650\.500\.060\.190\.07Freeze\-Omni0\.6420\.4810\.6360\.0010\.9970\.3360\.8673\.6150\.720\.800\.250\.25Gemini Live0\.2550\.3100\.0910\.0120\.8960\.6550\.8913\.3760\.330\.930\.990\.30Ours0\.8250\.6530\.1270\.1190\.7640\.8910\.8753\.5890\.690\.980\.440\.36Table 5:Full\-Duplex\-Bench results for our proportionate upsampling model, restricted to the systems common to both benchmarks\. U\-INTR: user interruption; U\-BC: user backchannel; T\-OTH: talking to others; BKG: background speech; RESP: respond rate; RSM: resume rate\.
### 5\.3Ablation Study

For the ablation study, we first examine the effectiveness of the SAC loss\. As shown in Table[3](https://arxiv.org/html/2609.03321#S5.T3), integrating the SAC loss consistently improves turn\-taking proficiency while preserving semantic capability at a comparable level\. Its central advantage, however, lies in mitigating the severe class imbalance among state transition tokens\. Figure[3](https://arxiv.org/html/2609.03321#S5.F3)presents the confusion matrices for state transition and response tokens on Switchboard with the Faster\-Whisper\-based model\. Specifically, each decoding step is formulated as an independent classification task and percentages are normalized against the ground\-truth labels\. The results clearly show that the SAC loss yields notable gains in the prediction accuracy of state\-switch tokens, which are operationally critical yet rare under the standard cross\-entropy objective\.

Furthermore, we evaluate the impact of data mixture strategies\. Under identical loss configurations, the model trained solely on synthetic data is substantially outperformed by the model trained on the combined HH and HA corpora\. For turn\-taking, this gap arises from two factors: current synthetic generation strategies fail to adequately simulate the fine\-grained acoustic dynamics of real human dialogues, and synthetic transcripts cannot replicate the specific temporal granularities and recognition artifacts of the perception module\. For semantic preservation, authentic HA dialogues prove more effective than synthetic transcripts that attempt to distill both capabilities simultaneously\. Finally, while training exclusively on either HH or HA enhances one capability in isolation, neither corpus alone improves both\. We observe highly consistent trends when substituting Faster\-Whisper with SimulStreaming as the perception module, with detailed results provided in Appendix[C\.2](https://arxiv.org/html/2609.03321#A3.SS2)\.

Beyond the turn\-taking F1, we further assess interruption handling under both machine\-interrupts\-user and user\-interrupts\-machine settings, following the protocol of NFSM\. Our model substantially outperforms NFSM in both directions \(Appendix[C\.3](https://arxiv.org/html/2609.03321#A3.SS3)\)\.

### 5\.4Scaling to the Proportionate Upsampling Mixture

While the preceding experiments operated under a strict token\-volume constraint \(referred to as the token\-matched subset\), we now investigate model performance when scaling training to its maximum extent\. To strictly preserve the established blending ratio across the constituent corpora, we construct a proportionally upsampled mixture by repeatedly oversampling the datasets to match the volume of the largest split \(detailed in Appendix[A\.4](https://arxiv.org/html/2609.03321#A1.SS4)\)\. As shown in Table[4](https://arxiv.org/html/2609.03321#S5.T4), scaling yields further improvements in both turn\-taking proficiency and semantic capability\. To rigorously assess the extent of semantic recovery, we introduce a strong upper bound: the original Qwen3\-4B in zero\-shot setting\. Remarkably, despite being trained to handle complex full\-duplex turn\-taking on a fragmented FSM tape, our Faster\-Whisper\-based model recovers the intrinsic semantic capability of the underlying LLM, closing the gap to within 0\.09 points on VoiceBench\. As an additional reference, the end\-to\-end Moshi attains 27\.36 on the same VoiceBench subset as reported by[Chen et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib8), which is lower than our FSM model and consistent with prior observations that end\-to\-end full\-duplex architectures often trade off semantic capability\.

We further evaluate the proportionate upsampling model on Full\-Duplex\-Bench[Lin et al\. \(2025b\)](https://arxiv.org/html/2609.03321#bib.bib36);[Lin et al\. \(2025a\)](https://arxiv.org/html/2609.03321#bib.bib37), an architecture\-agnostic benchmark independent of both our token scheme and training distribution \(Table[5](https://arxiv.org/html/2609.03321#S5.T5); complete results in Appendix[C\.4](https://arxiv.org/html/2609.03321#A3.SS4)\)\. The most substantial gains are observed in backchanneling: our model achieves the best human\-aligned timing and highest frequency with a low takeover rate on v1\.0, while attaining the highest resume rate on v1\.5 user backchannel\. We attribute this to the event\-guided tape and the SAC Loss\. The former supervises what kind of overlap occurred rather than merely whether speech is present, while the latter prevents the rare switch\-type tokens from being washed out by dominant response tokens\. The model remains highly competitive across most remaining dimensions, ranking first or second in smooth turn\-taking, user\-interruption semantic quality, and all four v1\.5 overlap scenarios\. A notable exception is pause handling, where our takeover rates trail Freeze\-Omni and Gemini Live, indicating a more aggressive strategy in reclaiming the floor\. However, this behavior reflects an inherent turn\-taking trade\-off rather than a strict deficiency\. For instance, the conservative Freeze\-Omni excels at pause handling but compromises smooth turn\-taking\. Overall, our approach outperforms every baseline across the majority of the twelve metrics \(Moshi10/1210/12, Freeze\-Omni8/128/12, Gemini Live7/127/12\)\.

## 6Conclusion

In this work, we presented a decoupled data approach for FSM\-based full\-duplex dialogue, in which turn\-taking is acquired from real HH spoken dialogues while semantic behavior is shaped through configurable HA text dialogues\. We operationalized this approach with a fully rule\-based event\-guided transformation that serializes HH spoken dialogues into causal FSM tapes without LLM\-generated annotations, and introduced the SAC Loss to jointly calibrate the long\-tailed distribution of state transition tokens and channel each data source toward the capability it best supervises\. Experiments show that our approach substantially improves turn\-taking proficiency over the synthetic\-data baseline while recovering the foundation LLM’s semantic capability to the zero\-shot upper bound, demonstrating that decoupling data sources by capability provides a scalable recipe for FSM\-based full\-duplex dialogue\.

## Limitations

Our study has four primary limitations\. First, serializing two parallel audio channels into a single causal tape inevitably compresses fine\-grained continuous\-time information into a discrete token order, leading to information loss despite our enforcement of temporal causality\. Second, paralinguistic information—prosody, emotion, and voice quality—is not representable on the text\-mediated tape, and the expressiveness of the rendered speech is bounded by the off\-the\-shelf TTS module, which is inherent to the original NFSM formulation\. In this work, we focus on turn\-taking naturalness and leave paralinguistic cues and a perceptual study to future\. Third, our HH corpora retain the agent channel’s original GT utterance content as spoken by ordinary human participants, which does not necessarily align with the response style expected of a voice assistant\. Refining the semantic content of the agent channel could yield a higher\-quality training signal, but risks disturbing the temporal alignment between channels and thus the turn\-taking dynamics we aim to preserve\. We therefore leave the agent channel unedited and view its controlled refinement as a promising direction for future work\. Fourth, our corpora cover only two\-party dialogue and a question\-answering assistant role\. Extending the HH data to multi\-party settings and the HA data to role\-conditioned dialogues[Mitra et al\. \(2026\)](https://arxiv.org/html/2609.03321#bib.bib35)is left to future work\.

## Acknowledgements

This work was supported by JST SPRING \(Grant Number JPMJSP2110\) and JSPS \(Grant Number JP23K28144\)\. We thank Professor Tatsuya Kawahara for his assistance in obtaining access to the Fisher corpus\.

## References

- Aroraet al\.\(2025\)S\. Arora, Z\. Lu, C\. Chiu, R\. Pang, and S\. WatanabeTalking turns: benchmarking audio foundation models on turn\-taking dynamics\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2e4ECh0ikn)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1),[3rd item](https://arxiv.org/html/2609.03321#S3.I1.i3.p1.1),[§3\.2](https://arxiv.org/html/2609.03321#S3.SS2.p3.1)\.
- Castillo\-Lópezet al\.\(2025\)G\. Castillo\-López, G\. de Chalendar, and N\. SemmarA survey of recent advances on turn\-taking modeling in spoken dialogue systems\.InProceedings of the 15th International Workshop on Spoken Dialogue Systems Technology,M\. I\. Torres, Y\. Matsuda, Z\. Callejas, A\. del Pozo, and L\. F\. D’Haro \(Eds\.\),Bilbao, Spain,pp\. 254–271\.External Links:[Link](https://aclanthology.org/2025.iwsds-1.27/),ISBN 979\-8\-89176\-248\-0Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1)\.
- Chenet al\.\(2025\)Q\. Chen, Y\. Chen, Y\. Chen, M\. Chen, Y\. Chen, C\. Deng, Z\. Du, R\. Gao, C\. Gao, Z\. Gao, Y\. Li, X\. Lv, J\. Liu, H\. Luo, B\. Ma, C\. Ni, X\. Shi, J\. Tang, H\. Wang, H\. Wang, W\. Wang, Y\. Wang, Y\. Xu, F\. Yu, Z\. Yan, Y\. Yang, B\. Yang, X\. Yang, G\. Yang, T\. Zhao, Q\. Zhang, S\. Zhang, N\. Zhao, P\. Zhang, C\. Zhang, and J\. ZhouMinMo: a multimodal large language model for seamless voice interaction\.External Links:2501\.06282,[Link](https://arxiv.org/abs/2501.06282)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§2](https://arxiv.org/html/2609.03321#S2.p2.1)\.
- Chenet al\.\(2024\)Y\. Chen, X\. Yue, C\. Zhang, X\. Gao, R\. T\. Tan, and H\. LiVoiceBench: benchmarking llm\-based voice assistants\.External Links:2410\.17196,[Link](https://arxiv.org/abs/2410.17196)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.03321#S5.SS1.p3.1),[§5\.4](https://arxiv.org/html/2609.03321#S5.SS4.p1.1)\.
- Chen and Yu \(2025\)Y\. Chen and H\. YuFrom turn\-taking to synchronous dialogue: a survey of full\-duplex spoken language models\.External Links:2509\.14515,[Link](https://arxiv.org/abs/2509.14515)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p1.1)\.
- Cieriet al\.\(2004\)C\. Cieri, D\. Miller, and K\. WalkerThe fisher corpus: a resource for the next generations of speech\-to\-text\.InProceedings of the Fourth International Conference on Language Resources and Evaluation \(LREC’04\),M\. T\. Lino, M\. F\. Xavier, F\. Ferreira, R\. Costa, and R\. Silva \(Eds\.\),Lisbon, Portugal\.External Links:[Link](https://aclanthology.org/L04-1500/)Cited by:[§5\.1](https://arxiv.org/html/2609.03321#S5.SS1.p1.1)\.
- Défossezet al\.\(2024\)A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. ZeghidourMoshi: a speech\-text foundation model for real\-time dialogue\.External Links:2410\.00037,[Link](https://arxiv.org/abs/2410.00037)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§2](https://arxiv.org/html/2609.03321#S2.p2.1)\.
- Ekstedt and Skantze \(2020\)E\. Ekstedt and G\. SkantzeTurnGPT: a transformer\-based language model for predicting turn\-taking in spoken dialog\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Online,pp\. 2981–2990\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.268),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.268)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1),[3rd item](https://arxiv.org/html/2609.03321#S3.I1.i3.p1.1)\.
- Ekstedt and Skantze \(2022\)E\. Ekstedt and G\. SkantzeVoice activity projection: self\-supervised learning of turn\-taking events\.InInterspeech,External Links:[Link](https://api.semanticscholar.org/CorpusID:248965092)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1)\.
- Godfreyet al\.\(1992\)J\. J\. Godfrey, E\. C\. Holliman, and J\. McDanielSWITCHBOARD: telephone speech corpus for research and development\.InProceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing \- Volume 1,ICASSP’92,USA,pp\. 517–520\.External Links:ISBN 0780305329Cited by:[§5\.1](https://arxiv.org/html/2609.03321#S5.SS1.p1.1)\.
- Guhret al\.\(2021\)O\. Guhr, A\. Schumann, F\. Bahrmann, and H\. J\. BöhmeFullStop: multilingual deep models for punctuation prediction\.External Links:[Link](http://ceur-ws.org/Vol-2957/sepp_paper4.pdf)Cited by:[§3\.2](https://arxiv.org/html/2609.03321#S3.SS2.p5.1)\.
- Heldner and Edlund \(2010\)M\. Heldner and J\. EdlundPauses, gaps and overlaps in conversations\.J\. Phonetics38,pp\. 555–568\.External Links:[Link](https://api.semanticscholar.org/CorpusID:7900155)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1)\.
- Holtzmanet al\.\(2020\)A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. ChoiThe curious case of neural text degeneration\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rygGQyrFvH)Cited by:[§C\.1](https://arxiv.org/html/2609.03321#A3.SS1.SSS0.Px1.p2.1)\.
- Jiet al\.\(2024\)S\. Ji, Y\. Chen, M\. Fang, J\. Zuo, J\. Lu, H\. Wang, Z\. Jiang, L\. Zhou, S\. Liu, X\. Cheng, X\. Yang, Z\. Wang, Q\. Yang, J\. Li, Y\. Jiang, J\. He, Y\. Chu, J\. Xu, and Z\. ZhaoWavChat: a survey of spoken dialogue models\.External Links:2411\.13577,[Link](https://arxiv.org/abs/2411.13577)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p1.1)\.
- Kooet al\.\(2024\)R\. Koo, M\. Lee, V\. Raheja, J\. I\. Park, Z\. M\. Kim, and D\. KangBenchmarking cognitive biases in large language models as evaluators\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 517–545\.External Links:[Link](https://aclanthology.org/2024.findings-acl.29/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.29)Cited by:[§C\.3](https://arxiv.org/html/2609.03321#A3.SS3.p2.1)\.
- Leeet al\.\(2025\)S\. Lee, K\. Kim, and G\. KimBehavior\-SD: behaviorally aware spoken dialogue generation with large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 9574–9593\.External Links:[Link](https://aclanthology.org/2025.naacl-long.484/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.484),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1)\.
- Liaoet al\.\(2025\)B\. Liao, Y\. Xu, J\. Ou, K\. Yang, W\. Jian, P\. Wan, and D\. ZhangFlexDuo: a pluggable system for enabling full\-duplex capabilities in speech dialogue systems\.External Links:2502\.13472,[Link](https://arxiv.org/abs/2502.13472)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§2](https://arxiv.org/html/2609.03321#S2.p2.1)\.
- Linet al\.\(2025a\)G\. Lin, S\. S\. Kuan, Q\. Wang, J\. Lian, T\. Li, and H\. LeeFull\-duplex\-bench v1\. 5: evaluating overlap handling for full\-duplex speech models\.arXiv preprint arXiv:2507\.23159\.Cited by:[§5\.4](https://arxiv.org/html/2609.03321#S5.SS4.p2.1)\.
- Linet al\.\(2025b\)G\. Lin, J\. Lian, T\. Li, Q\. Wang, G\. Anumanchipalli, A\. H\. Liu, and H\. LeeFull\-duplex\-bench: a benchmark to evaluate full\-duplex spoken dialogue models on turn\-taking capabilities\.arXiv preprint arXiv:2503\.04721\.Cited by:[§5\.4](https://arxiv.org/html/2609.03321#S5.SS4.p2.1)\.
- Macháček and Polák \(2025\)D\. Macháček and P\. PolákSimultaneous translation with offline speech and LLM models in CUNI submission to IWSLT 2025\.InProceedings of the 22nd International Conference on Spoken Language Translation \(IWSLT 2025\),E\. Salesky, M\. Federico, and A\. Anastasopoulos \(Eds\.\),Vienna, Austria \(in\-person and online\),pp\. 389–398\.External Links:[Link](https://aclanthology.org/2025.iwslt-1.41/),[Document](https://dx.doi.org/10.18653/v1/2025.iwslt-1.41),ISBN 979\-8\-89176\-272\-5Cited by:[§4\.1](https://arxiv.org/html/2609.03321#S4.SS1.p1.1)\.
- Menonet al\.\(2021\)A\. K\. Menon, S\. Jayasumana, A\. S\. Rawat, H\. Jain, A\. Veit, and S\. KumarLong\-tail learning via logit adjustment\.InInternational Conference on Learning Representations,Cited by:[§4\.2](https://arxiv.org/html/2609.03321#S4.SS2.p3.1)\.
- Mitraet al\.\(2026\)S\. Mitra, P\. Pandey, A\. Jain, S\. Sahith, and K\. V\. V\. GirishAdaptive turn\-taking for real\-time multi\-party voice agents\.External Links:2606\.13544,[Link](https://arxiv.org/abs/2606.13544)Cited by:[Limitations](https://arxiv.org/html/2609.03321#Sx1.p1.1)\.
- Nguyenet al\.\(2023\)T\. A\. Nguyen, E\. Kharitonov, J\. Copet, Y\. Adi, W\. Hsu, A\. Elkahky, P\. Tomasello, R\. Algayres, B\. Sagot, A\. Mohamed, and E\. DupouxGenerative spoken dialogue language modeling\.Transactions of the Association for Computational Linguistics11,pp\. 250–266\.External Links:[Link](https://aclanthology.org/2023.tacl-1.15/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00545)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§2](https://arxiv.org/html/2609.03321#S2.p2.1)\.
- OpenAIet al\.\(2024\)OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, J\. Jang, A\. Jiang, R\. Jiang, H\. Jin, D\. Jin, S\. Jomoto, B\. Jonn, H\. Jun, T\. Kaftan, Ł\. Kaiser, A\. Kamali, I\. Kanitscheider, N\. S\. Keskar, T\. Khan, L\. Kilpatrick, J\. W\. Kim, C\. Kim, Y\. Kim, J\. H\. Kirchner, J\. Kiros, M\. Knight, D\. Kokotajlo, Ł\. Kondraciuk, A\. Kondrich, A\. Konstantinidis, K\. Kosic, G\. Krueger, V\. Kuo, M\. Lampe, I\. Lan, T\. Lee, J\. Leike, J\. Leung, D\. Levy, C\. M\. Li, R\. Lim, M\. Lin, S\. Lin, M\. Litwin, T\. Lopez, R\. Lowe, P\. Lue, A\. Makanju, K\. Malfacini, S\. Manning, T\. Markov, Y\. Markovski, B\. Martin, K\. Mayer, A\. Mayne, B\. McGrew, S\. M\. McKinney, C\. McLeavey, P\. McMillan, J\. McNeil, D\. Medina, A\. Mehta, J\. Menick, L\. Metz, A\. Mishchenko, P\. Mishkin, V\. Monaco, E\. Morikawa, D\. Mossing, T\. Mu, M\. Murati, O\. Murk, D\. Mély, A\. Nair, R\. Nakano, R\. Nayak, A\. Neelakantan, R\. Ngo, H\. Noh, L\. Ouyang, C\. O’Keefe, J\. Pachocki, A\. Paino, J\. Palermo, A\. Pantuliano, G\. Parascandolo, J\. Parish, E\. Parparita, A\. Passos, M\. Pavlov, A\. Peng, A\. Perelman, F\. de Avila Belbute Peres, M\. Petrov, H\. P\. de Oliveira Pinto, Michael, Pokorny, M\. Pokrass, V\. H\. Pong, T\. Powell, A\. Power, B\. Power, E\. Proehl, R\. Puri, A\. Radford, J\. Rae, A\. Ramesh, C\. Raymond, F\. Real, K\. Rimbach, C\. Ross, B\. Rotsted, H\. Roussez, N\. Ryder, M\. Saltarelli, T\. Sanders, S\. Santurkar, G\. Sastry, H\. Schmidt, D\. Schnurr, J\. Schulman, D\. Selsam, K\. Sheppard, T\. Sherbakov, J\. Shieh, S\. Shoker, P\. Shyam, S\. Sidor, E\. Sigler, M\. Simens, J\. Sitkin, K\. Slama, I\. Sohl, B\. Sokolowsky, Y\. Song, N\. Staudacher, F\. P\. Such, N\. Summers, I\. Sutskever, J\. Tang, N\. Tezak, M\. B\. Thompson, P\. Tillet, A\. Tootoonchian, E\. Tseng, P\. Tuggle, N\. Turley, J\. Tworek, J\. F\. C\. Uribe, A\. Vallone, A\. Vijayvergiya, C\. Voss, C\. Wainwright, J\. J\. Wang, A\. Wang, B\. Wang, J\. Ward, J\. Wei, C\. Weinmann, A\. Welihinda, P\. Welinder, J\. Weng, L\. Weng, M\. Wiethoff, D\. Willner, C\. Winter, S\. Wolrich, H\. Wong, L\. Workman, S\. Wu, J\. Wu, M\. Wu, K\. Xiao, T\. Xu, S\. Yoo, K\. Yu, Q\. Yuan, W\. Zaremba, R\. Zellers, C\. Zhang, M\. Zhang, S\. Zhao, T\. Zheng, J\. Zhuang, W\. Zhuk, and B\. ZophGPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§5\.1](https://arxiv.org/html/2609.03321#S5.SS1.p2.1)\.
- Patamiaet al\.\(2025\)R\. A\. Patamia, H\. P\. T\. Dinh, M\. Liu, and A\. CosgunTurn\-taking modelling in conversational systems: a review of recent advances\.Technologies13\(12\)\.External Links:[Link](https://www.mdpi.com/2227-7080/13/12/591),ISSN 2227\-7080,[Document](https://dx.doi.org/10.3390/technologies13120591)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1)\.
- Pillutlaet al\.\(2021\)K\. Pillutla, S\. Swayamdipta, R\. Zellers, J\. Thickstun, S\. Welleck, Y\. Choi, and Z\. HarchaouiMAUVE: measuring the gap between neural text and human text using divergence frontiers\.InProceedings of the 35th International Conference on Neural Information Processing Systems,NIPS ’21,Red Hook, NY, USA\.External Links:ISBN 9781713845393Cited by:[§C\.1](https://arxiv.org/html/2609.03321#A3.SS1.SSS0.Px1.p2.1)\.
- Radfordet al\.\(2022\)A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. SutskeverRobust speech recognition via large\-scale weak supervision\.External Links:2212\.04356,[Link](https://arxiv.org/abs/2212.04356)Cited by:[§4\.1](https://arxiv.org/html/2609.03321#S4.SS1.p1.1)\.
- Team \(2024\)S\. TeamSilero vad: pre\-trained enterprise\-grade voice activity detector \(vad\), number detector and language classifier\.GitHub\.Note:[https://github\.com/snakers4/silero\-vad](https://github.com/snakers4/silero-vad)Cited by:[§4\.1](https://arxiv.org/html/2609.03321#S4.SS1.p1.1)\.
- Threlkeldet al\.\(2022\)C\. Threlkeld, M\. Umair, and J\. de RuiterUsing transition duration to improve turn\-taking in conversational agents\.InProceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue,O\. Lemon, D\. Hakkani\-Tur, J\. J\. Li, A\. Ashrafzadeh, D\. H\. Garcia, M\. Alikhani, D\. Vandyke, and O\. Dušek \(Eds\.\),Edinburgh, UK,pp\. 193–203\.External Links:[Link](https://aclanthology.org/2022.sigdial-1.20/),[Document](https://dx.doi.org/10.18653/v1/2022.sigdial-1.20)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1)\.
- Veluriet al\.\(2024\)B\. Veluri, B\. N\. Peloquin, B\. Yu, H\. Gong, and S\. GollakotaBeyond turn\-based interfaces: synchronous LLMs as full\-duplex dialogue agents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 21390–21402\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1192/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1192)Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1)\.
- Wanget al\.\(2024a\)J\. Wang, L\. Chen, A\. Khare, A\. Raju, P\. Dheram, D\. He, M\. Wu, A\. Stolcke, and V\. RavichandranTurn\-taking and backchannel prediction with acoustic and large language model fusion\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 12121–12125\.Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p1.1),[3rd item](https://arxiv.org/html/2609.03321#S3.I1.i3.p1.1)\.
- Wanget al\.\(2024b\)P\. Wang, S\. Lu, Y\. Tang, S\. Yan, W\. Xia, and Y\. XiongA full\-duplex speech dialogue scheme based on large language model\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§2](https://arxiv.org/html/2609.03321#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.03321#S3.SS1.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, Y\. Li, C\. Fu, Y\. Zhang, Y\. Shen, L\. Xie, K\. Li, X\. Sun, and L\. MaFreeze\-omni: a smart and low latency speech\-to\-speech dialogue model with frozen llm\.ICML\.Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1)\.
- Xianget al\.\(2025\)B\. Xiang, S\. Zhao, T\. Guo, and W\. ZouUnderstanding the modality gap: an empirical study on the speech\-text alignment mechanism of large speech language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5187–5202\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.262/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.262),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.03321#S5.SS1.p3.1)\.
- Xie and Wu \(2024\)Z\. Xie and C\. WuMini\-omni2: towards open\-source gpt\-4o with vision, speech and duplex capabilities\.External Links:2410\.11190,[Link](https://arxiv.org/abs/2410.11190)Cited by:[§2](https://arxiv.org/html/2609.03321#S2.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.1](https://arxiv.org/html/2609.03321#S5.SS1.p1.1)\.
- Zhanget al\.\(2025\)Q\. Zhang, L\. Cheng, C\. Deng, Q\. Chen, W\. Wang, S\. Zheng, J\. Liu, H\. Yu, C\. Tan, Z\. Du, and S\. ZhangOmniFlatten: an end\-to\-end GPT model for seamless voice conversation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 14570–14580\.External Links:[Link](https://aclanthology.org/2025.acl-long.709/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.709),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.03321#S1.p2.1)\.

## Appendix AData Construction Details

ParameterSpecified in NFSMOur reproductionGeneration LLMYes \(gpt\-4\-turbo\-2024\-04\-09\)IdenticalDecoding settingsNoDefault sampling\# dialoguesYes \(1,500 series\)IdenticalPrompt templateYes \(their Appendix A\)Identical\{num\_rounds\}No \(placeholder only\)Uniform over 9–11Scenario\-to\-round assignmentNo \(placeholder only\)Random round with no collision\{response\_word\_count\}No \(placeholder only\)Uniform over 100–150\{interrupted\_response\_word\_count\}No \(placeholder only\)Uniform over 20–50Topic poolPartial \(“hundreds”, “random”\)110 topics generated withgpt\-4\-turbo\-2024\-04\-09Dialogue serializationNoAs described in Section[5\.1](https://arxiv.org/html/2609.03321#S5.SS1)Table 6:Reconstruction parameters of the NFSM synthetic\-data baseline\. For each parameter we indicate whether it is specified in the original NFSM paper and the setting adopted in our reproduction\.### A\.1NFSM Baseline Reproduction Details

Because the original NFSM training corpus is not publicly available, our synthetic\-data baseline is reconstructed from the specifications given in the NFSM paper\. For full transparency, Table[6](https://arxiv.org/html/2609.03321#A1.T6)documents every parameter of this reconstruction, indicating for each whether it is specified in the original paper and, where it is not, the setting we adopt\. We reproduce the original work as faithfully as possible, keeping its most critical components identical to the original, including the generation LLM and the full prompt template\. A few minor parameters such as the unfilled placeholders in the published prompt are not disclosed, so an exact match is impossible and we adopt reasonable settings\.

### A\.2State Transition Token Statistics

Table[7](https://arxiv.org/html/2609.03321#A1.T7)reports the token type distribution of the Switchboard training tapes under the two perception modules, placed side by side with that of the reconstructed NFSM synthetic tapes for reference\. The distributions exhibit three notable patterns\.

First, state transition tokens follow a markedly long\-tailed distribution\. Within the restricted state transition vocabulary, continuation tokens jointly account for72\.78%72\.78\\%and80\.45%80\.45\\%of all state transition tokens under the Faster\-Whisper and SimulStreaming perception modules respectively, whereas the operationally critical state\-switch tokens occupy a significantly smaller proportion\. This skew motivates our use of logit adjustment over the state transition tokens in the SAC loss\.

Second, the choice of perception module reshapes the distribution in a predictable direction\. Switching from the IPU\-level Faster\-Whisper to the word\-level SimulStreaming raises the share of`\[C\.LISTEN\]`from8\.75%8\.75\\%to13\.77%13\.77\\%\. The increased interleaving of`\[C\.LISTEN\]`within user content quantitatively reflects the fragmentation phenomenon and offers a partial explanation for the larger semantic gap observed under SimulStreaming\.

Third, the synthetic tapes exhibit a markedly more extreme imbalance than the real HH data: response tokens account for88\.38%88\.38\\%of the tape, leaving only11\.62%11\.62\\%for all state\-transition tokens, of which the operationally critical state\-switch tokens \(`\[S\.LISTEN\.I\]`,`\[S\.LISTEN\.N\]`, and`\[S\.SPEAK\]`\) constitute merely2\.40%2\.40\\%\. In contrast, the real HH tapes allocate a substantially larger share to state\-switch tokens of7\.26%7\.26\\%under Faster\-Whisper and5\.88%5\.88\\%under SimulStreaming\. This gap reflects that LLM\-generated transcripts, being predominantly structured and non\-overlapping, seldom reproduce the frequent floor changes, backchannels, and interruptions of natural conversation\.

Real HHTokenSynthetic`\[C\.LISTEN\]`0\.91%8\.75%13\.77%`\[C\.SPEAK\]`8\.31%10\.63%10\.42%`\[S\.LISTEN\.I\]`0\.17%0\.77%0\.53%`\[S\.LISTEN\.N\]`0\.99%2\.83%2\.38%`\[S\.SPEAK\]`1\.24%3\.66%2\.97%Response88\.38%73\.37%69\.93%Total100\.00%100\.00%100\.00%Table 7:Token\-type ratio across the NFSM synthetic tapes and the Switchboard training tapes under two perception modules\.ConfigurationTrain \(k\)Validation \(k\)Test \(k\)FSHSWBDSGPTSYNFSHSWBDSYNFSHSWBDBaseline–––1,3441,344––1651655,6005,600444444Ours \(Token\-matched\)538538135135672672–691691236236–5,6005,600444444Ours \(Scaled Mixture\)21,79221,7925,4485,448∗27,24027,240†–691691236236–5,6005,600444444Table 8:Dataset statistics across different configurations, measured in thousands \(k\) of tokens\. Abbreviations used are FSH \(Fisher\), SWBD \(Switchboard\), SGPT \(ShareGPT\), and SYN \(Synthetic\)\. For the scaled mixture, we apply proportional oversampling to maintain the established data distribution\. The superscript symbols indicate datasets that are repeatedly sampled from their natural volume: \(∗\) base size1,551k×3\.51$1,551$\\text\{k\}\\times 3\.51, and \(†\) base size2,966k×9\.18$2,966$\\text\{k\}\\times 9\.18\.
### A\.3Merging Rules at Segment Boundaries

Concatenating two adjacent segments during serialization may produce two consecutive state transition tokens at their boundary, which must be collapsed into a single token to align with the rule of FSM\.

Rather than enumerating14×1414\\times 14event pairs or5×55\\times 5token pairs, we observe that the merged token is fully determined by the FSM state immediately before the first token fires and immediately after the second token fires, reducing the analysis to four state\-pair cases:

- •\(`LISTEN`,`LISTEN`\)→\\rightarrow`\[C\.LISTEN\]`: the FSM remains in`LISTEN`\.
- •\(`SPEAK`,`SPEAK`\)→\\rightarrow`\[C\.SPEAK\]`: the FSM remains in`SPEAK`\.
- •\(`LISTEN`,`SPEAK`\)→\\rightarrow`\[S\.SPEAK\]`: a switch from listening to speaking\.
- •\(`SPEAK`,`LISTEN`\)→\\rightarrow`\[S\.LISTEN\.\*\]`: a switch from speaking to listening, where the N/I subtype is inherited from whichever token in the pair is`\[S\.LISTEN\.\*\]`, with`\[S\.LISTEN\.I\]`taking priority when both are present\.

The \(`SPEAK`,`LISTEN`\) case warrants further justification, as it is the only case requiring subtype inheritance\. The first token in this case must be`\[C\.SPEAK\]`or`\[S\.LISTEN\.\*\]`, and the second must be`\[C\.LISTEN\]`or`\[S\.LISTEN\.N\]`\. Among these, the pair`\[C\.SPEAK\]`\+`\[C\.LISTEN\]`appears superficially well\-formed but never arises in valid serializations, so the merge can always recover an N/I subtype from one side of the pair\. To see why, note that a segment ending in`\[C\.SPEAK\]`can only originate from BC or BI initiated by the user, or FTI initiated by the agent \(Table[1](https://arxiv.org/html/2609.03321#S3.T1)\); in all three cases the floor holder at the segment’s end is the agent\. Conversely, a segment beginning with`\[C\.LISTEN\]`can only originate from C, P, or G initiated by the user, all of which require the user to be the floor holder at the segment’s start \(recall that the initiator of a G segment is the preceding floor holder\)\. These two requirements are mutually inconsistent, ruling out the pair\.

Besides, as a related adjustment, when a T immediately follows a G, its leading`\[S\.LISTEN\.N\]`is demoted to`\[C\.LISTEN\]`, as the preceding G has already transitioned the FSM into`LISTEN`\.

### A\.4Proportionate Upsampling Mixture

For the scaling experiments, we construct a proportionally upsampled mixture to maximize the training signal while strictly maintaining the established data distribution\. Since the constituent corpora possess disparate natural volumes, datasets with smaller token counts are oversampled to adhere to the predefined ratios\. Detailed statistics for each configuration, including base volumes, oversampling multipliers, and final token counts, are provided in Table[8](https://arxiv.org/html/2609.03321#A1.T8)\. This approach ensures that the model benefits from the full extent of available data without biasing the optimization toward any single data source\.

### A\.5Prompts

The prompt for filtering and stylistic rewriting HA data is shown as follows:

PromptYou are an expert linguistic editor specializing in dialogue style transfer\. Your task is twofold: first, EVALUATE whether the input text is suitable for conversion into a casual spoken dialogue; second, ONLY IF suitable, EXECUTE the conversion\. \#\#\# PHASE 1: EVALUATION \(Strict Filtering\) Before converting, analyze the input content\. You must return an empty list ‘\[\]‘ for the ’conversations’ field if the input is not suitable for spoken dialogue, for example but not constrain to the following ’UNSUITABLE’ categories: 1\. \*\*Code & Technical Data\*\*: Contains programming code, stack traces, logs, JSON/XML, SQL queries, or shell commands\. 2\. \*\*Strict structured text\*\*: Contains mathematical formulas \(LaTeX\), chemical equations, or rigid data tables\. 3\. \*\*Document text\*\*: Contains document input which is impossible to happen in dialogue\. 4\. \*\*Non\-Dialogue / Fragments\*\*: The input is just a list of keywords, a disjointed sentence fragment without context, or incoherent noise\. 5\. \*\*Competitor Identity / Specific AI Models\*\*: The text mentions ’OpenAI’, ’ChatGPT’, or the model refers to itself as such\. \#\#\# PHASE 2: CONVERSION \(Only if Phase 1 passes\) If the content is suitable, convert it into a natural spoken\-style conversation\. Guidelines for Spoken Style: \- Strictly REMOVE all Markdown syntax \(e\.g\., \*, \#, \-\)\. Do not use bullet points or numbered lists; use connecting words like ’First’, ’Next’, ’Finally’ instead\. \- Simplify complex sentence structures and content to be more conversational\. \- Maintain the original intent and core information\. \#\#\# CONSTRAINTS: 1\. Keep the original language of each utterance unchanged\. 2\. Keep the original ’id’ strictly unchanged\. 3\. Ensure the speaker roles and the number of turns remain exactly the same\. 4\. \*\*CRITICAL\*\*: If the content matches the UNSUITABLE criteria, the output must be ‘\[\]‘\. 5\. Ensure the output follows the strict JSON schema provided\.

The prompt of FSM is shown as follows:

Prompt<\|im\_start\|\>user You are a helpful voice assistant\. You will work as a Finite State Machine to handle state transitions between LISTEN and SPEAK states\. State\-transition tokens: "\[S\.SPEAK\]" means "switch to SPEAK"; "\[S\.LISTEN\.INTERRUPT\]" means "switch to LISTEN because of being interrupted"; "\[S\.LISTEN\.NATURAL\]" means "switch to LISTEN because of natural end"; "\[C\.SPEAK\]" means "continue to SPEAK"; "\[C\.LISTEN\]" means "continue to LISTEN"\. For content between State\-transition tokens, the user’s content are prefixed with "<user\>" and your own content has no prefix\. "<SIL\>" represents a silence of \{\{SILENCE\_TOKEN\_DUR\}\} seconds\. Please generate a tape fragment of the Finite State Machine\.<\|im\_end\|\> <\|im\_start\|\>assistant <think\> </think\>

## Appendix BAdditional Experimental Settings

This section supplements the general settings in the main text with details on training hyperparameters, dataset partitioning, and the temporal thresholds used in the FSM and our event\-guided transformation pipeline\.

### B\.1Training Hyperparameters and Dataset Partitioning

Fine\-tuning is conducted on four NVIDIA RTX A6000 GPUs with batch size256256, optimized with AdamW \(weight decay0\.010\.01\) under a linear schedule with warmup, peak learning rate1\.0×10−51\.0\\times 10^\{\-5\}, and warmup ratio0\.030\.03\. Each instance has a context length of10241024tokens\. We train for up to300300iterations with early stopping on validation loss for the token\-matched setting, and extend this budget to700700iterations with the same early\-stopping criterion for the proportionate upsampling mixture to accommodate its substantially larger training corpus\. For dataset partitioning, we apply distinct split ratios tailored to the scale of each corpus\. For both the ShareGPT and Switchboard datasets, we partition the data into training, validation, and test subsets at a dialogue ratio of 7:1:2\. Because the Fisher corpus is substantially larger, we adjust its dialogue ratio to 7\.75:0\.25:2 to prevent the validation set from becoming unnecessarily large for frequent validation\.

### B\.2Preprocessing and Transformation Thresholds

Several thresholds govern data preprocessing and the event\-guided transformation pipeline\. For HH dialogue filtering, we discard dialogues whose Word Error Rate exceeds0\.30\.3, removing both broken audio and cross\-channel leakage where one speaker’s audio bleeds into the other channel\. For the event\-guided transformation, the IPU pause threshold is set to3232milliseconds, each silence token represents0\.640\.64seconds of silence, and the maximum utterance duration for heuristic backchannel identification is capped at11second\.

## Appendix CAdditional Experimental Results

![Refer to caption](https://arxiv.org/html/2609.03321v1/heatmap_F1_human_avg.png)

![Refer to caption](https://arxiv.org/html/2609.03321v1/heatmap_VoiceBench_overall.png)

Figure 4:Impact of dataset mixture ratios on model performance\. Left: turn\-taking proficiency measured by F1 score\. Right: semantic capability measured by VoiceBench\.Figure 5:Impact ofα\\alphain the SAC loss\.Figure 6:Impact of base model size\.### C\.1Analysis on Data Mixture, Loss, and Model Scale

This subsection reports analysis along three design axes: the mixture ratios of training data, the SAC loss hyperparameterα\\alpha, and the size of the base model\. All experiments use the Faster\-Whisper\-based configuration and strictly constrain the total training token volume to match the NFSM baseline\.

#### Data Mixture Ratios\.

Figure[4](https://arxiv.org/html/2609.03321#A3.F4)reports the effect of dataset mixture ratios underα=0\.5\\alpha=0\.5\. The results demonstrate a clear trade\-off: increasing the proportion of HH data consistently enhances turn\-taking proficiency, whereas increasing the proportion of HA data improves semantic capability\. To avoid biasing the model toward either capability, we adopt a 1:1 HH:HA ratio in all main experiments\. Adjusting the internal Fisher\-to\-Switchboard ratio shows no significant trend, so we empirically set it to4:14\{:\}1\.

Impact ofα\\alphain SAC Loss\.Figure[5](https://arxiv.org/html/2609.03321#A3.F5)reports performance across different values ofα\\alpha\. Asα\\alphaincreases, the VoiceBench score improves while the turn\-taking F1 score decreases\. To diagnose the F1 decline, we additionally plot the average recall and precision of state transition tokens on HH data\. Interestingly, asα\\alphaincreases, recall actually rises and stabilizes, whereas precision drops significantly\. This indicates that the SAC loss inherently aligns better with recall\. This behavior is consistent with prior observations that standard language model pre\-training via maximum likelihood estimation inherently favors recall over precision, assigning excessive probability mass to an unreliable tail of tokens[Holtzman et al\. \(2020\)](https://arxiv.org/html/2609.03321#bib.bib24);[Pillutla et al\. \(2021\)](https://arxiv.org/html/2609.03321#bib.bib23)\. Based on this analysis, we adoptα=0\.6\\alpha=0\.6as the default in our main experiments to balance semantic preservation against turn\-taking precision\.

#### Impact of Base Model Size\.

Fixingα=0\.6\\alpha=0\.6, we further investigate the effect of scaling the base model, as shown in Figure[6](https://arxiv.org/html/2609.03321#A3.F6)\. The VoiceBench score consistently improves with larger models, but the turn\-taking F1 score peaks at the 4B scale\. We hypothesize that larger language models may require substantially more effort to adequately fit the highly specialized pattern of the FSM tape\.

ConfigurationF1↑\\uparrowVB↑\\uparrowOurs0\.640454\.80Ablation on Loss Functionw/o SAC loss0\.623454\.49Ablation on Training Data \(w/o SAC loss\)HH only0\.637619\.71HA only0\.388654\.55Synthetic only \(NFSM\)0\.302544\.24Table 9:Ablation study on the SAC loss and training data composition under SimulStreaming perception\.

### C\.2Ablation Study under SimulStreaming Perception

To complement the ablation study in the main text, Table[9](https://arxiv.org/html/2609.03321#A3.T9)reports the corresponding results under SimulStreaming perception\. The trends are highly consistent with those of the Faster\-Whisper\-based model: the SAC loss effectively preserves turn\-taking performance, and neither HH nor HA data alone is sufficient to jointly optimize semantic and turn\-taking capabilities\.

ModelPerceptionMiU F1↑\\uparrowUiM PRR↑\\uparrowNFSM—0\.72030\.7260OursFaster\-Whisper0\.76980\.8037SimulStreaming0\.77860\.8367Table 10:Experiments of bidirectional interruption\.DimensionPause HandlingBackchannelSmooth Turn TakingUser InterruptionDataSyntheticCandorICCCandorSyntheticMetricTOR↓\\downarrowTOR↓\\downarrowTOR↓\\downarrowFreq↑\\uparrowJSD↓\\downarrowTOR↑\\uparrowLatency↓\\downarrowTOR↑\\uparrowGPT\-4o↑\\uparrowLatency↓\\downarrowdGSLM0\.9340\.9350\.6910\.0150\.9340\.9750\.3520\.9170\.2012\.531Moshi0\.9850\.9801\.0000\.0010\.9570\.9410\.2651\.0000\.7650\.257Freeze\-Omni0\.6420\.4810\.6360\.0010\.9970\.3360\.9530\.8673\.6151\.409Gemini Live0\.2550\.3100\.0910\.0120\.8960\.6551\.3010\.8913\.3761\.183Ours0\.8250\.6530\.1270\.1190\.7640\.8910\.1810\.8753\.5890\.579Table 11:Complete results on Full\-Duplex\-Bench v1\.0 across different conversational dimensions, where latency is presented in seconds\.ScenarioClass / MetricFreeze\-OmniMoshiGeminiSonicGPT\-4oOursUSER\_INTRRESPOND↑\\uparrow0\.720\.500\.330\.240\.780\.69RESUME↓\\downarrow0\.120\.260\.550\.710\.100\.17UNCERTAIN↓\\downarrow0\.030\.000\.010\.010\.020\.05UNKNOWN↓\\downarrow0\.130\.250\.100\.040\.120\.09STOP \(s\)↓\\downarrow1\.421\.162\.202\.250\.231\.60RESP \(s\)↓\\downarrow1\.351\.472\.622\.751\.501\.12USER\_BACKCHRESPOND↓\\downarrow0\.070\.020\.010\.000\.030\.00RESUME↑\\uparrow0\.800\.060\.930\.980\.700\.98UNCERTAIN↓\\downarrow0\.020\.000\.020\.000\.010\.00UNKNOWN↓\\downarrow0\.110\.920\.040\.020\.250\.02STOP \(s\)↑\\uparrow0\.660\.420\.660\.640\.210\.69RESP \(s\)↓\\downarrow2\.163\.002\.451\.451\.321\.99TALKING\_OTHERRESPOND↓\\downarrow0\.580\.200\.000\.100\.910\.40RESUME↑\\uparrow0\.250\.190\.990\.900\.020\.44UNCERTAIN↑\\uparrow0\.000\.020\.000\.000\.010\.06UNKNOWN↓\\downarrow0\.150\.590\.010\.000\.060\.10STOP \(s\)↑\\uparrow1\.390\.871\.691\.770\.181\.35RESP \(s\)↓\\downarrow1\.322\.381\.782\.041\.161\.29BKG\_SPEECHRESPOND↓\\downarrow0\.630\.210\.700\.010\.930\.44RESUME↑\\uparrow0\.250\.070\.300\.980\.040\.36UNCERTAIN↑\\uparrow0\.010\.010\.000\.000\.000\.13UNKNOWN↓\\downarrow0\.110\.710\.000\.010\.030\.07STOP \(s\)↑\\uparrow0\.980\.540\.951\.050\.181\.16RESP \(s\)↓\\downarrow1\.601\.622\.382\.761\.261\.20Table 12:Complete results on Full\-Duplex\-Bench v1\.5\. Behavioral response distribution across the four overlap scenarios, with average stop and response latencies \(s\)\. Bold Class/Metric are the desired behavior for each scenario\.
### C\.3Bidirectional Interruption Evaluation

We further evaluate the system’s interruption handling capability by strictly replicating the protocol of NFSM\.

For the Machine\-interrupts\-User \(MiU\) setting, we use GPT\-4\-Turbo to synthesize 600 dialogues in which the user’s final statement is injected with a deliberate commonsense error\. We decode the FSM step by step through this final statement and extract the response generated at the precise location where the model first proactively emits`\[S\.SPEAK\]`instead of`\[C\.LISTEN\]`\. State transition tokens are then stripped to reconstruct a standard dialogue\. To prevent egocentric bias[Koo et al\. \(2024\)](https://arxiv.org/html/2609.03321#bib.bib9), we use Qwen3\.5\-27B as the evaluator, strictly following the assessment prompts of the original NFSM paper to compute the Proper Interruption Rate, the position\-specific metricsirmid\\text\{ir\}\_\{\\text\{mid\}\},irend\\text\{ir\}\_\{\\text\{end\}\}, and MIR, and finally an aggregate F1 score\.

For the User\-interrupts\-Machine \(UiM\) setting, we use GPT\-4\-Turbo to generate 600 dialogues evenly distributed across four interruption categories: denial, affirmation, environmental noises, and topic shifting\. We insert`\[S\.SPEAK\]`at the end of the user’s interruption to elicit an immediate FSM response\. After removing state transition tokens from the dialogue, the average Proper Response Rate \(PRR\) is evaluated by the same Qwen3\.5\-27B judge using the original evaluation prompts\.

As shown in Table[10](https://arxiv.org/html/2609.03321#A3.T10), both perception\-module variants of our model achieve substantially higher MiU F1 and UiM PRR scores than the NFSM baseline, demonstrating improvements in both turn\-taking dynamics and semantic appropriateness within the specific sub\-scenario of interruptions\.

### C\.4Full Results on Full\-Duplex\-Bench

We report the complete Full\-Duplex\-Bench results underlying the merged Table[5](https://arxiv.org/html/2609.03321#S5.T5)in the main text\. Results of v1\.0 are shown in Table[11](https://arxiv.org/html/2609.03321#A3.T11), while results of v1\.5 are shown in Table[12](https://arxiv.org/html/2609.03321#A3.T12)\. The two versions probe complementary capabilities\. v1\.0 evaluates turn\-taking over the course of a dialogue, covering whether the model holds back at intra\-turn pauses, backchannels at human\-like moments, takes the floor promptly at genuine turn boundaries, and reacts to interruptions\. v1\.5 instead isolates overlap handling while the model is speaking, testing whether it distinguishes overlapping speech that warrants yielding the floor from speech that does not\. Both evaluate our Faster\-Whisper\-based proportionate upsampling model under the official protocols\. All other numbers are quoted from the official report\. Specially, the low Smooth Turn Taking latency is partly affected by the official metric, which by design clamps to zero any onset preceding the annotated turn boundary \(e\.g\., backchannel\)\.

## Appendix DEnd\-to\-End Latency Breakdown

ComponentMean \(s\)P50 \(s\)P90 \(s\)Perception0\.16890\.12260\.3347Cognitive0\.27860\.26690\.4357Motor0\.14810\.13710\.2024Total0\.59550\.53120\.9105Table 13:End\-to\-end FTED latency of the FSM system under Faster\-Whisper perception, broken down by module\.Table[13](https://arxiv.org/html/2609.03321#A4.T13)reports the end\-to\-end inference latency of the Faster\-Whisper\-based FSM system, measured by FTED\. Latency is measured on a test set of 100 questions, constructed by randomly sampling 25 questions from each of the four VoiceBench subsets used in our semantic evaluation\. The breakdown across the perception, cognitive, and motor modules demonstrates that the system maintains real\-time responsiveness\.

## Appendix EDesign of FSM

We build the FSM from scratch and introduce several techniques to improve its performance and robustness\.

Decoupled Temperatures for Transition and Response Tokens\.Although emitted from a shared autoregressive stream, state\-transition and response tokens serve distinct purposes, necessitating decoupled decoding strategies\. Transition tokens act as discrete control signals where sampling noise causes turn\-taking failures; hence, we decode them greedily\. Conversely, response tokens require standard sampling temperatures to maintain lexical diversity and generation quality\. Operationally, each generation step initiates with a greedy single\-token probe\. If a transition token is emitted, the step commits it and terminates\. Otherwise, this token serves as the clause prefix, and the remainder is sampled at the response temperature\. Transition tokens proposed mid\-clause by the sampler are discarded, deferring the transition decision to the next probe\.

Constrained Decoding over the Transition Vocabulary\.During decoding, we mask the logits of all state\-transition tokens deemed illegal under the current FSM state\. Although the fine\-tuned model rarely produces illegal transitions, this hard masking provides deterministic guarantees against structural failures rather than relying on probabilistic safety\.

Bounded Look\-ahead\.Generating the next clause only post\-playback simplifies interruption handling but incurs latency gaps\. Conversely, eager generation until turn completion eliminates gaps but wastes computation upon user intervention\. We balance this trade\-off via a bounded look\-ahead policy: the controller generates one additional clause as long as fewer thanKKclauses arein\-flight\(i\.e\., dispatched to the TTS module but not yet fully played\), idling once this bound is reached\.

Interruption Handling\.Handling user interruptions gracefully presents a critical challenge\. To support natural backchannels—where pausing before resumption sounds robotic—the agent continues speaking while the transition decision is computed\. As a price, this incurs a temporal inconsistency that the tape before and after the decision diverge\. In practice, we retrieve the currently playing clause from tape containing look\-ahead tail and split it at the exact playback timestamp into apre\-clause\(already played\) and apost\-clause\(pending\)\. We then execute one constrained decoding step using the pre\-clause appended with the user utterance as the prefix\. The resulting transition token dictates the tape rewriting strategy:

- •`\[C\.SPEAK\]`: The split clause is replaced bypre\-clause \+ interruption \+`\[C\.SPEAK\]`\+ post\-clause\. The look\-ahead tail is retained\.
- •`\[S\.LISTEN\.I\]`: The split clause is replaced bypre\-clause \+ interruption \+`\[S\.LISTEN\.I\]`\. TTS is truncated immediately, and the look\-ahead tail is discarded\.
- •`\[S\.LISTEN\.N\]`: The split clause is replaced bypre\-clause \+ post\-clause \+ interruption \+`\[S\.LISTEN\.N\]`\. The current clause is allowed to finish naturally, and the remaining look\-ahead tail is discarded\.

Hallucination Filtering for the ASR Module\.Whisper\-based ASR is prone to hallucinations, which can spuriously trigger the interruption mechanism\. We mitigate this using a soft lexical filter: segments matching a predefined list of frequent hallucinations are discarded if their average token log\-probability falls below−0\.3\-0\.3\. Otherwise, they are retained to preserve genuine user utterances\.