Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

# Modality Separation and Semantic Coherence for Full-Duplex SLMs
Source: [https://arxiv.org/html/2607.06540](https://arxiv.org/html/2607.06540)
## Hierarchical Acoustic\-Semantic Modeling: Modality Separation and Semantic Coherence for Full\-Duplex SLMs

Zhenyu Liu1,2,Yunxin Li1,2,Xuanyu Zhang1,2,Qixun Teng1,Shenyuan Jiang1,Haolan Chen, Minjun Zhao,Fanbo Meng,Yu Xu,Yancheng He,Baotian Hu1,2, ✉,Haizhou Li2,3,Min Zhang1,2 1School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China 2Center for Language, Intelligence and Machines, Shenzhen Loop Area Institute, Shenzhen, China 3School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China

[![[Uncaptioned image]](https://arxiv.org/html/2607.06540v1/figures/huggingface-logo.png)https://huggingface\.co/HIT\-TMG/Lychee\-FD](https://huggingface.co/HIT-TMG/Lychee-FD-1.5) [![[Uncaptioned image]](https://arxiv.org/html/2607.06540v1/figures/github-mark.png)https://github\.com/HITsz\-TMG/Lychee\-FD](https://github.com/HITsz-TMG/Lychee-FD) [![[Uncaptioned image]](https://arxiv.org/html/2607.06540v1/figures/website-logo.png)https://hitsz\-tmg\.github\.io/Lychee\-FD](https://hitsz-tmg.github.io/Lychee-FD)

Correspondence:[liuzhenyuhit@gmail\.com](https://arxiv.org/html/2607.06540v1/mailto:[email protected]),[25s151197@stu\.hit\.edu\.cn](https://arxiv.org/html/2607.06540v1/mailto:[email protected]),[liyx@hit\.edu\.cn](https://arxiv.org/html/2607.06540v1/mailto:[email protected]) [hubaotian@hit\.edu\.cn](https://arxiv.org/html/2607.06540v1/mailto:[email protected]),[zhangmin2021@hit\.edu\.cn](https://arxiv.org/html/2607.06540v1/mailto:[email protected]),[haizhouli@cuhk\.edu\.cn](https://arxiv.org/html/2607.06540v1/mailto:[email protected])

###### Abstract

Developing seamless, high\-performance, native intelligent full\-duplex Spoken Language Models \(SLMs\) remains a critical challenge and long\-standing goal for the speech and NLP community\. Despite notable progress, recent endeavors are fundamentally constrained by severemodality interference, which causes substantial knowledge degradation and compromises semantic integrity — ultimately making full\-duplex SLMs feel unnatural and unintelligent\. In this paper, through an exhaustive fine\-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises frominherent gradient conflictsbetween acoustic and semantic modeling when the two modalities are forced to share a deep parameter space\. Guided by this key insight, we introduceLychee\-FD, a native end\-to\-end full\-duplex framework designed to mitigate modality interference\. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross\-modality coherence via a dedicated semantic alignment channel\. Extensive experiments on multiple full\-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence \(\+7\.4%on Spoken QA\) and full\-duplex interaction fluidity \(\+28\.5%on FullDuplexBench 1\.5\) without compromising inference efficiency\. To the best of our knowledge, this work is the first to achieve two key advances: 1\) uncovering and elucidating the root cause of modality interference in full\-duplex SLMs, and 2\) designing an elegant hierarchical model together with a practical solution for seamless, high\-performance, native intelligent full\-duplex SLMs\.

## 1Introduction

The rapid evolution of Large Language Models \(LLMs\) has fundamentally reshaped our daily lives, establishing them as ubiquitous assistants capable of complex reasoning and instruction following\. Within this landscape, Spoken Language Models \(SLMs\) represent a significant paradigm shift from text\-based to voice\-based interaction\. Despite recent advancements in Omni\-modal models capable of seamless voice interaction\(openai2024gpt4ocard;DBLP:conf/acl/ZhanDYZZLZYZL0F24;li2025uni;DBLP:journals/corr/abs\-2001\-08361;10\.5555/3600270\.3602446;DBLP:journals/pami/XuZC23;DBLP:journals/corr/abs\-2501\-16327\), a critical disparity remains between artificial agents and authentic human conversation\. Currently, most voice interactions are constrained to a rigid half\-duplex mode, where the system strictly alternates between listening and speaking in a sequential manner\. In contrast, authentic human conversation is inherently full\-duplex, requiring the ability to continuously process incoming audio streams while concurrently generating responses\(73837945\-26a5\-3e38\-a7b9\-daf138681621;DBLP:journals/corr/abs\-2501\-01957;DBLP:journals/corr/abs\-2411\-13577\)\. The forced turn\-taking of half\-duplex systems disrupts the fluidity of interaction, creating an artificial barrier between user and agent\.

![Refer to caption](https://arxiv.org/html/2607.06540v1/x1.png)Figure 1:Visualization of the efficiency and intelligence trade\-off\. Existing paradigms face a dilemma when extending a half\-duplex SLM \(black star\) to a full\-duplex one\. Native End\-to\-End models \(blue diamond\) sacrifice accuracy for efficiency, while Thinker\-Talker models \(red diamond\) preserve knowledge but incur prohibitive inference costs\. In contrast, our proposed Hierarchical framework \(green diamond\) combines low latency with high accuracy, significantly outperforming all full\-duplex SLM baselines\.To bridge this gap, developing native Full\-Duplex SLMs \(FDSLMs\) has emerged as a critical challenge and a long\-standing goal for the speech and NLP communities\(73837945\-26a5\-3e38\-a7b9\-daf138681621;DBLP:journals/tmlr/AroraCCPWADLLW25;DBLP:journals/tacl/NguyenKCAHETASM23;DBLP:conf/kdd/LinWHSSL22\)\. This full\-duplex paradigm demands complex pragmatic reasoning, enabling the agent to handle spontaneous interruptions, inject natural backchannels, and dynamically manage turn\-taking without explicit system\-level triggers\(DBLP:conf/interspeech/HuHCCGZCLBG25;DBLP:conf/chi/LiuSXHYZ025\)\. Consequently, achieving seamless full\-duplex communication is widely regarded as the next critical milestone in human\-machine interaction, promising to unlock a truly natural, fluid, and immersive conversation experience\(DBLP:journals/corr/abs\-2502\-09940;DBLP:journals/corr/abs\-2501\-01957;DBLP:journals/corr/abs\-2503\-04721\)\.

Despite progress in FDSLM development\(DBLP:journals/corr/abs\-2408\-05211;DBLP:conf/iclr/ZengDLZJD025\), current methods still suffer from severeModality Interference\. As illustrated in Figure[1](https://arxiv.org/html/2607.06540#S1.F1), adapting a half\-duplex base model \(black star\) into a native End\-to\-End architecture \(blue diamond\) precipitates significant knowledge degradation\. This degradation is further corroborated by state\-of\-the\-art models such as Moshi\(DBLP:journals/corr/abs\-2410\-00037\), which reports a significant drop in speech intelligence \(\-12\.7% on LlamaQ\) after full\-duplex alignment\. Historically, this type of interference has been a notorious and persistent optimization bottleneck across deep learning\(DBLP:journals/corr/Ruder17a;Caruana1997;NEURIPS2020\_3fe78a8a;NEURIPS2018\_432aca3a\)\. While some approaches\(DBLP:conf/icml/WangLFZS000M25;chen2025funaudiochattechnicalreport\)attempt to circumvent this interference by adopting Thinker\-Talker architectures \(red diamond\), they require intricate multi\-stage training and introduce significant latency\. These limitations indicate that existing paradigms either fail in knowledge retention or inference efficiency\. Consequently, the central research question of this work is:How can we resolve this modality interference to simultaneously achieve high inference efficiency and robust knowledge retention in full\-duplex SLMs?

To uncover the root cause of modality interference, we conduct an exhaustive fine\-grainedmodel optimization dynamics analysis\(shown in Figure[2](https://arxiv.org/html/2607.06540#S2.F2)\)\. By quantifying the geometric relationships between the gradient vectors of semantic and acoustic objectives, we demonstrate theinherent gradient conflictsthat fundamentally cause this interference\. First, by calculating the layer\-wise gradient cosine similarity, we reveal severeoptimization divergencein gradient directions\. While the gradient directions of text and speech objectives are synergistic in shallow layers, they become increasingly orthogonal and negative in deeper layers\. This empirically demonstrates that forcing acoustic and semantic modeling to update within a shared parameter space inevitably fractures the optimization trajectory\. Second, by evaluating the gradient magnitude ratio, we identify severesemantic dilution\. The temporal alignment of sparse text tokens \(e\.g\., 3Hz\) with dense audio frames \(e\.g\., 25Hz\) via padding tokens consistently suppresses the magnitude of semantic gradients across all layers\. Consequently, the optimization landscape becomes overwhelmingly dominated by acoustic modeling, effectively suppressing semantic modeling\.

Inspired by these insights, we introduceLychee\-FD, a native end\-to\-end full\-duplex framework designed to mitigate modality interference with two architectural innovations\. First, we propose ahierarchical parameter separationstrategy for optimization divergence in deep layers\. Specifically, we separate the deep layers into independent acoustic and semantic heads\. By executing these heads in parallel, we maintain the original model depth, thereby preserving inference efficiency\. Second, to counter semantic dilution, we introduce asemantic alignment channelto generate coherent internal monologues\. By utilizing continuous textual supervision as a semantic anchor, we preserve the robustness of semantic modeling during training\. Extensive experiments demonstrate that Lychee\-FD achieves the state\-of\-the\-art performance, specifically delivering an average 7\.4% improvement on Spoken QA tasks and a 28\.5% gain on FullDuplexBench 1\.5\. Ultimately, our approach effectively mitigates modality interference, simultaneously achieving high inference efficiency and robust knowledge retention\.

Our contributions are summarized as follows:

- •For the first time, we uncover the root cause of modality interference in full\-duplex SLMs\. Supported by an exhaustive fine\-grained analysis of model optimization dynamics, we reveal that these obstacles stem frominherent gradient conflictsbetween acoustic and semantic modeling when forced into a shared deep parameter space\.
- •We presentLychee\-FD, the first fully native intelligent full\-duplex framework\. We propose a hierarchical parameter separation strategy that decouples acoustic and semantic modeling, bridged by an elegantly designed semantic alignment channel to preserve coherent internal monologues and robust semantics\.
- •We advance the state\-of\-the\-art across multiple full\-duplex benchmarks\. To forge ahead the full\-duplex research in the communities, we open\-source our framework, training pipeline, and model weights at[https://github\.com/HITsz\-TMG/Lychee\-FD](https://github.com/HITsz-TMG/Lychee-FD)\.

## 2Related Work

### 2\.1Spoken Language Model

SLMs have evolved from cascading ASR\-LLM\-TTS pipelines\(DBLP:conf/emnlp/ZhangLZZWZQ23;DBLP:journals/corr/abs\-2407\-04051;DBLP:journals/taslp/BorsosMVKPSRTGTZ23;10842513;DBLP:journals/taslp/ZeghidourLOST22\)to unified architectures\. Current methods are mainly grouped into two paradigms:

#### Thinker\-Talker Architectures\.

This paradigm decouples acoustic generation from the LLM backbone to mitigate modality interference\(DBLP:journals/corr/abs\-2503\-20215;DBLP:conf/icml/WangLFZS000M25;DBLP:conf/iclr/FangGZMZ025\)\. For instance,DBLP:journals/corr/abs\-2503\-20215adopts a dual\-track autoregressive architecture\. Despite their stability, these systems often require intricate, multi\-stage training curricula and suffer from inference bottlenecks caused by the separate generation modules\.

#### Native End\-to\-End Architectures\.

Conversely, these architectures embed speech and text into a shared semantic space, enabling direct speech\-to\-speech reasoning\(li2025perception;DBLP:journals/corr/abs\-2408\-16725;DBLP:conf/emnlp/MitsuiMWHS24;DBLP:conf/iclr/ZengDLZJD025\)\. For examples,DBLP:journals/corr/abs\-2507\-16632introduces latent audio encoding to capture paralinguistic cues\.

Crucially, while existing SLMs remain fundamentally bottlenecked by rigid, half\-duplex turn\-taking mechanisms, our work advances the field by establishing a native Full\-Duplex framework\. By enabling simultaneous listening and speaking, we aim to unlock a truly natural, fluid, and immersive paradigm for future human\-machine interaction\.

### 2\.2Full Duplex Speech Interaction

To achieve turn\-free and fluid conversation, recent research has explored full\-duplex paradigms that transcend rigid turn\-taking\. These efforts can be broadly categorized into three paradigms:

#### Full\-Duplex Dialogue System\.

Early efforts primarily leverage Voice Activity Detection \(VAD\) as a dialogue manager to control the speaking process of half\-duplex SLMs\(DBLP:journals/corr/abs\-2509\-06502;DBLP:journals/corr/abs\-2502\-13472;DBLP:journals/corr/abs\-2509\-23938;DBLP:journals/corr/abs\-2502\-14145\)\. Although making some progress, these cascaded systems suffer from high latency and error propagation, prompting a shift towards unified modeling\.

#### Time\-Division Multiplexing \(TDM\)\.

To address these limitations, recent works use the SLM itself as the dialogue manager\. TDM approaches flatten listening and speaking tokens into a single temporal sequence\. However, this leads to increasing computational complexity and limits long\-context interaction\(DBLP:conf/acl/ZhangCDCWZLYTDZ25;DBLP:conf/emnlp/VeluriPYGG24;DBLP:conf/emnlp/ZhangCH0XXZ0024;DBLP:journals/corr/abs\-2505\-17060\)\.

#### Channel\-Division Multiplexing \(CDM\)\.

Instead of flattening, CDM approaches explicitly model concurrent input and output streams, offering the most integrated form of interaction\(coreteam2025mimoaudioaudiolanguagemodels;DBLP:journals/tacl/NguyenKCAHETASM23;DBLP:journals/corr/abs\-2410\-11190\)\. For example,DBLP:journals/corr/abs\-2410\-00037andchen2025funaudiochattechnicalreportintegrate time\-aligned text and audio streams to provide explicit semantic guidance for speech generation\. However, these fully shared architectures entangle distinct modalities in a single parameter space, leaving them highly susceptible to severe knowledge degradation\.

Unlike previous works that struggle with modality interference, Lychee\-FD provides the first fundamental solution to the core problem within FDSLMs\. By resolving the inherent deep\-layer gradient conflicts between acoustic and semantic modeling, our native end\-to\-end framework successfully achieves ultra\-low latency with robust knowledge retention, substantially pushing forward the frontier of research for the speech and NLP communities\.

![Refer to caption](https://arxiv.org/html/2607.06540v1/x2.png)

\(a\)Gradient Cosine Similarity
![Refer to caption](https://arxiv.org/html/2607.06540v1/x3.png)

\(b\)Gradient Magnitude Ratio

Figure 2:Optimization Dynamics Visualization\.\(a\) Gradient Cosine Similarity:The transition to negative values in deep layers reveals conflicting optimization directions between semantic and acoustic modeling, motivating our hierarchical parameter separation\.\(b\) Gradient Magnitude Ratio:The consistently lower ratio for “Aligned” \(Red\) compared to “Dense” \(Blue\) indicates that sparse alignment dilutes semantic supervision, motivating our semantic alignment channel\.

## 3Hierarchical Acoustic\-Semantic Modeling

### 3\.1Model Optimization Dynamics Analysis of Modality Interference

To empirically investigate the fundamental root cause of the modality interference and understand the learning process within FDSLMs, we conducted an in\-depth model optimization dynamics analysis, shown in Figure[2](https://arxiv.org/html/2607.06540#S2.F2)\. We utilized a native CDM architecture initialized with weights from StepAudio\-2\-mini\(DBLP:journals/corr/abs\-2507\-16632\)\. We performed forward passes on 1K samples from the training set to accumulate gradients for the cross\-entropy losses of both text token generation \(ℒtext\\mathcal\{L\}\_\{\\text\{text\}\}\) and speech token generation \(ℒspeech\\mathcal\{L\}\_\{\\text\{speech\}\}\), without updating the model parameters\. Specifically, the layer\-wise gradient vectors are formally defined as follows:

𝐠text\(l\)=∇θ\(l\)ℒtext,\\mathbf\{g\}\_\{\\text\{text\}\}^\{\(l\)\}=\\nabla\_\{\\theta^\{\(l\)\}\}\\mathcal\{L\}\_\{\\text\{text\}\},\(1\)𝐠speech\(l\)=∇θ\(l\)ℒspeech,\\mathbf\{g\}\_\{\\text\{speech\}\}^\{\(l\)\}=\\nabla\_\{\\theta^\{\(l\)\}\}\\mathcal\{L\}\_\{\\text\{speech\}\},\(2\)where∇θ\(l\)\\nabla\_\{\\theta^\{\(l\)\}\}denotes the gradient operator with respect to the layer parametersθ\(l\)\\theta^\{\(l\)\}, and the resulting tensors are flattened into vectors\. By analyzing the geometric relationships of these gradient vectors, we quantitatively uncover the fundamental interaction dynamics between the two modalities during the entire learning process\.

#### Optimization Divergence\.

We investigated the compatibility of the two optimization objectives by calculating the cosine similarity𝒮\(l\)\\mathcal\{S\}^\{\(l\)\}between text and speech gradient vectors:

𝒮\(l\)=cos⁡\(𝐠text\(l\),𝐠speech\(l\)\)\.\\mathcal\{S\}^\{\(l\)\}=\\cos\(\\mathbf\{g\}\_\{\\text\{text\}\}^\{\(l\)\},\\mathbf\{g\}\_\{\\text\{speech\}\}^\{\(l\)\}\)\.\(3\)As shown in Figure[2\(a\)](https://arxiv.org/html/2607.06540#S2.F2.sf1), the similarity reveals a distinct layer\-wise pattern\. In the shallow layers \(0\-9\), the cosine similarity is positive, indicating that the two modalities share synergistic optimization directions, focusing on common low\-level features processing\. However, as the depth increases, the similarity drops sharply, turning negative and fluctuating in the deeper layers\. This trend empirically confirms our hypothesis regarding the dual nature of speech: while shallow layers can share representations, the deep layers face a fundamental conflict between acoustic and semantic modeling\. Forcing a unified set of parameters to resolve these opposing gradient directions inevitably leads to sub\-optimal performance, validating the root cause of the observed modality interference\. Moreover, we confirm these findings on Moshi architecture in Section[5](https://arxiv.org/html/2607.06540#S5)and provide a global gradient influence analysis in in Appendix[9](https://arxiv.org/html/2607.06540#S9)to demonstrate the causal destructive impact of this modality conflict\.

![Refer to caption](https://arxiv.org/html/2607.06540v1/x4.png)Figure 3:Two mainstream architecture paradigms of SLMs and our proposed Lychee\-FD\. Our design features a hierarchical parameter separation strategy to resolve deep\-layer modality conflicts and a semantic alignment channel to enforce robust knowledge retention without sacrificing inference efficiency\.
#### Semantic Dilution\.

Prevalent Full\-Duplex SLMs typically address the frequency mismatch between text \(approximately 3Hz\) and audio \(typically 25Hz\) by interleaving padding tokens to enforce temporal alignment\(DBLP:journals/corr/abs\-2410\-00037;DBLP:journals/corr/abs\-2507\-16632;chen2025funaudiochattechnicalreport\)\. To evaluate the impact of this alignment on optimization, we compared the ratioℛ\(l\)\\mathcal\{R\}^\{\(l\)\}of gradient magnitudes between the two modalities:

ℛ\(l\)=‖𝐠text\(l\)‖/‖𝐠speech\(l\)‖\.\\mathcal\{R\}^\{\(l\)\}=\|\|\\mathbf\{g\}\_\{\\text\{text\}\}^\{\(l\)\}\|\|/\|\|\\mathbf\{g\}\_\{\\text\{speech\}\}^\{\(l\)\}\|\|\.\(4\)As illustrated in Figure[2\(b\)](https://arxiv.org/html/2607.06540#S2.F2.sf2), we observe a substantial disparity between the continuous text supervision \(Dense\) and the sparse text with padding \(Aligned\)\. Specifically, the gradient magnitude ratio in the Aligned setting is consistently suppressed across all layers, suggesting that the introduction of padding tokens effectively dilutes the density of semantic supervision\. Consequently, the optimization dynamics become dominated by acoustic reconstruction, which drives the observed degradation in knowledge retention\.

### 3\.2Model Design: Modality Separation and Semantic Coherence

Guided by these insights, we introduceLychee\-FD, a native end\-to\-end framework designed to resolve modality interference\. Rather than circumventing these optimization bottlenecks via high\-latency Thinker\-Talker models \(as illustrated in Figure[3](https://arxiv.org/html/2607.06540#S3.F3)\), our architecture fundamentally tackles them via two targeted innovations: aHierarchical Parameter Separationstrategy to disentangle conflicting optimization directions, and aSemantic Alignment Channelto counteract semantic dilution and enforce robust knowledge retention\.

#### Shared Backbone Foundation\.

We choose StepAudio\-2\-mini\(DBLP:journals/corr/abs\-2507\-16632\)as our half\-duplex backbone due to its public availability\. We employ the Whisper\-v3\-large encoder for input audio processing\. For acoustic output, we adopt the CosyVoice2 tokenizer to convert audio into discrete speech tokens at a 25Hz frame rate\. Crucially, to ensure precise temporal alignment for full\-duplex interaction, we utilize a 25Hz frame rate, distinct from the setting adopted by Step\-Audio\-2\.

#### Hierarchical Parameter Separation\.

Guided by the observation that acoustic and semantic gradients become orthogonal in deeper layers, we design a hierarchical Transformer architecture\. We retain a unified Transformer backbone for the shallow layers to leverage shared low\-level feature processing\. Formally, given the input embedding sequence𝐄∈ℝN×d\\mathbf\{E\}\\in\\mathbb\{R\}^\{N\\times d\}, the shared representation𝐇shared\\mathbf\{H\}\_\{\\text\{shared\}\}is mathematically formulated as follows:

𝐇shared=ℱshared​\(𝐄;θshared\),\\mathbf\{H\}\_\{\\text\{shared\}\}=\\mathcal\{F\}\_\{\\text\{shared\}\}\(\\mathbf\{E\};\\theta\_\{\\text\{shared\}\}\),\(5\)whereℱshared\\mathcal\{F\}\_\{\\text\{shared\}\}denotes the stack of shared Transformer layers parameterized byθshared\\theta\_\{\\text\{shared\}\}\.

In the deeper layers, we physically disentangle the parameters into three specialized heads: theSemantic Headfor text generation, theAcoustic Headfor speech synthesis, and theControl Headfor interaction management \(e\.g\., stop/start signals\)\. These heads operate in parallel as follows:

𝐎m=ℱheadm​\(𝐇shared;θm\),m∈\{T,A,C\}\\mathbf\{O\}^\{m\}=\\mathcal\{F\}\_\{\\text\{head\}\}^\{m\}\(\\mathbf\{H\}\_\{\\text\{shared\}\};\\theta\_\{m\}\),\\quad m\\in\\\{T,A,C\\\}\(6\)wheremmrepresents the modality \(Text, Acoustic, Control\), and𝐎m\\mathbf\{O\}^\{m\}denotes the output logits for each head\. The total cross\-entropy lossℒ\\mathcal\{L\}is computed as the summation of the next\-token prediction losses for each specific head as follows:

ℒ=−∑m∈\{T,A,C\}∑tlog⁡P​\(ytm\|y<t,𝐄;θ\),\\mathcal\{L\}=\-\\sum\_\{m\\in\\\{T,A,C\\\}\}\\sum\_\{t\}\\log P\(y\_\{t\}^\{m\}\|y\_\{<t\},\\mathbf\{E\};\\theta\),\(7\)whereytmy\_\{t\}^\{m\}denotes the ground\-truth token for modalitymmat steptt\. This hierarchical split effectively isolates the conflicting optimization objectives, allowing the model to articulate high\-fidelity acoustic responses without corrupting its underlying semantic modeling\.

#### Semantic Alignment Channel\.

To counter the semantic dilution caused by sparse alignment in speech\-native tasks, we incorporate a semantic alignment channel to generate coherent internal monologues\. During training, these monologues serve as a semantic anchor, maintaining high\-magnitude gradient flow for the language modeling objective\. Specifically, we organize the parallel generation streams as follows:

YT\\displaystyle Y^\{\\text\{T\}\}=\[t1,t2,⋯,tn,<EOT\>,<pad\>,⋯,<pad\>\],\\displaystyle=\[t\_\{1\},t\_\{2\},\\cdots,t\_\{n\},\\texttt\{<EOT\>\},\\texttt\{<pad\>\},\\cdots,\\texttt\{<pad\>\}\],YA\\displaystyle Y^\{\\text\{A\}\}=\[a1,a2,⋯,an,an\+1,an\+2,⋯,<EOS\>\],\\displaystyle=\[a\_\{1\},a\_\{2\},\\cdots,a\_\{n\},a\_\{n\+1\},a\_\{n\+2\},\\cdots,\\texttt\{<EOS\>\}\],YC\\displaystyle Y^\{\\text\{C\}\}=\[<Start\>,c1,⋯,⋯,⋯,<Stop\>\],\\displaystyle=\[\\texttt\{<Start\>\},c\_\{1\},\\cdots,\\cdots,\\cdots,\\texttt\{<Stop\>\}\],whereYCY^\{\\text\{C\}\}employs special tokens \(e\.g\.,<Start\>and<Stop\>\) to manage the onset and offset of the model’s speech\. By explicitly modeling the text channel alongside the acoustic channel, we ensure high\-magnitude gradient flow for the language modeling objective, thereby preserving robust knowledge retention\.

ModelTypeLlamaQWebQTriviaQAAvg\.Takeover RateS→TS\{\\rightarrow\}TS→SS\{\\rightarrow\}SS→TS\{\\rightarrow\}TS→SS\{\\rightarrow\}SS→TS\{\\rightarrow\}TS→SS\{\\rightarrow\}SS→TS\{\\rightarrow\}TS→SS\{\\rightarrow\}SFreeze\-OmniSystem\-level71\.350\.738\.325\.824\.323\.944\.633\.499\.6VITA 1\.5System\-level75\.751\.041\.829\.235\.026\.050\.835\.4100dGSLMNative–1\.3–0\.2–0\.4–0\.6100FLM\-audioNative41\.336\.715\.614\.510\.510\.422\.420\.599\.5MoshiNative62\.354\.725\.319\.619\.117\.435\.530\.593\.8SALMONN\-omni∗Native67\.061\.733\.728\.132\.924\.244\.538\.099\.9Fun\-Audio\-ChatNative72\.364\.326\.224\.429\.627\.742\.738\.899\.9StepAudio\-2\-miniHalf\-duplex74\.762\.039\.930\.839\.529\.851\.340\.9–Lychee\-FD \(Ours\)Native73\.765\.338\.333\.942\.539\.451\.546\.2100w/oS​e​mSem\-C​h​a​n​n​e​lChannel69\.361\.034\.131\.534\.230\.145\.940\.899\.6w/oP​a​r​a​mParam\-S​e​pSep67\.036\.034\.622\.536\.624\.246\.127\.698\.5Table 1:Performance comparison on spoken question answering benchmarks\. We report accuracy \(Acc\) in both speech\-to\-text \(S→TS\{\\rightarrow\}T\) and speech\-to\-speech \(S→SS\{\\rightarrow\}S\) settings\. We also report average takeover rate \(TOR\) across three benchmarks\.Bolddenotes best results andunderlineddenotes second best\.∗denotes our implementation\.

## 4Experiments

### 4\.1Baselines

To ensure a comprehensive evaluation, we compare our proposed method against a diverse set of representative and competitive full\-duplex SLMs\. These baselines cover the primary architectural paradigms currently explored in the field:

#### System\-Level Full\-Duplex Models

include Freeze\-Omni\(DBLP:conf/icml/WangLFZS000M25\)and VITA\-1\.5\(DBLP:journals/corr/abs\-2501\-01957\), which integrate an external VAD module to manage the dialogue state of a standard half\-duplex SLM\.

#### Native Full\-Duplex Models

include dGSLM\(DBLP:journals/tacl/NguyenKCAHETASM23\), FLM\-Audio\(DBLP:journals/corr/abs\-2509\-02521\), Moshi\(DBLP:journals/corr/abs\-2410\-00037\)and Fun\-Audio\-Chat\(chen2025funaudiochattechnicalreport\)\. Specifically, Fun\-Audio\-Chat adopts a Thinker\-Talker architecture, while the others utilize a CDM architecture\.

### 4\.2Training Data Construction

Given the scarcity of open\-source full\-duplex datasets, we developed an automated pipeline to synthesize high\-quality training data covering three key interaction behaviors: Interruptions, User Backchannels, and AI Backchannels\. We employed multiple agents to simulate realistic User\-Assistant dialogues, injecting rule\-based constraints to trigger diverse interruption types \(e\.g\., topic switching, follow\-up queries\) and natural backchannels\. To ensure acoustic robustness, we synthesized speech from the transcripts via CosyVoice 2DBLP:journals/corr/abs\-2412\-10117, coupled with 80K predefined voice prompts for zero\-shot cloning\. After rigorous filtering to remove samples with logical inconsistencies or low audio quality, we curated a final dataset of approximately 140K full\-duplex dialogue instances, providing a diverse and reliable foundation for our experiments\. We provide more details of our data pipeline in Appendix[11](https://arxiv.org/html/2607.06540#S11)\.

### 4\.3Implementation Details

We optimize our model using AdamW\(DBLP:conf/iclr/LoshchilovH19\)with a cosine learning rate scheduler\. All experiments are conducted on 8 NVIDIA H20 GPUs, with a global batch size of 32 and a learning rate of 3e\-6\. We set the warmup ratio to 0\.1 and train for 1 epoch, which takes approximately 16 hours\. For inference, we employ greedy sampling for both text and speech token generation\. We evaluate our model with three random seeds and report their average performance\. Regarding the hierarchical architecture configuration, unless otherwise specified, we utilize a shared backbone of 24 Transformer layers\. The specialized heads are configured with 4 layers for the text channel, 4 layers for the speech channel, and 2 layers for the control channel\. This yields approximately 10B total parameters, a choice supported by the marginal gain analysis in Appendix[7](https://arxiv.org/html/2607.06540#S7)\.

### 4\.4Spoken Question Answering

#### Metrics\.

To evaluate the speech intelligence capabilities of our model, we follow previous work\(DBLP:journals/corr/abs\-2410\-00037\)and utilize three standard spoken question answering benchmarks: LlamaQ, WebQ, and TriviaQA\. We report the accuracy \(Acc\) under both speech\-to\-text \(S→TS\\rightarrow T\) and speech\-to\-speech \(S→SS\\rightarrow S\) settings\. For speech\-to\-speech setting, we leverage Whisper\-large\-v3\(DBLP:conf/icml/RadfordKXBMS23\)to obtain the transcription of generated speech\. Additionally, we report the takeover rate \(TOR\) across three benchmarks to quantify the frequency of model responses, serving as an indicator of the model’s turn\-taking behavior\.

#### Result\.

As presented in Table[1](https://arxiv.org/html/2607.06540#S3.T1), Lychee\-FD demonstrates superior spoken question answering capabilities\. It achieves the highest average accuracy across both speech\-to\-text \(S→TS\\rightarrow T\) and speech\-to\-speech \(S→SS\\rightarrow S\) settings\. Compared to the previous SOTA native full\-duplex model, Fun\-Audio\-Chat, Lychee\-FD delivers a substantial improvement of 7\.4% inS→SS\\rightarrow Saccuracy and 8\.8% inS→TS\\rightarrow Taccuracy\. Even when compared to system\-level pipelines like VITA\-1\.5, our end\-to\-end approach demonstrates superior reasoning capabilities \(10\.8% inS→SS\\rightarrow Sand 0\.7% inS→TS\\rightarrow T\), validating the effectiveness of our framework\. Furthermore, Lychee\-FD maintains a perfect TOR of 100%, confirming that this exceptional knowledge retention is achieved without compromising seamless, real\-time interaction stability\.

ModelFDBenchFullDuplexBench 1\.0FullDuplexBench 1\.5SRR↑\\uparrowSIR↑\\uparrowEIR↓\\downarrowSRIR↑\\uparrowFSED\.↓\\downarrowIRD↓\\downarrowI\-TOR↑\\uparrowB\-Freq↑\\uparrowB\-TOR↓\\downarrowT\-TOR↑\\uparrowP\-TOR↓\\downarrowStop↓\\downarrowIRR↑\\uparrowBRR↑\\uparrowStop↓\\downarrowLat\.↓\\downarrowdGSLM––––––91\.71\.569\.197\.593\.52523––––Freeze\-Omni12\.957\.225\.729\.5667541377\.50\.163\.633\.646\.3138027\.063\.06602066VITA 1\.521\.046\.116\.378\.33036992599\.52\.581\.858\.888\.815236\.038\.012222140FLM\-audio7\.569\.41\.00\.9989340891\.00\.361\.896\.656\.5457910\.043\.02439983Moshi41\.478\.822\.173\.91895142187\.55\.136\.476\.454\.188561\.026\.010713034Lychee\-FD \(Ours\)86\.399\.70\.495\.8637121094\.514\.623\.498\.310\.084078\.069\.0570826

Table 2:Performance comparison of full\-duplex interaction capabilities and efficiency\. Despite interaction behavior metrics, we also report efficiency metrics of each benchmark \(FSED, IRD, Lat\. Stop\.\), measured in milliseconds \(ms\)\.↑\\uparrowindicates higher is better\.Bolddenotes best results andunderlineddenotes second best\.
#### Ablation\.

To systematically investigate the individual contributions of our two proposed innovations, we conducted ablation studies on two variants:w/o Sem\-Channel, which replaces the semantic alignment channel with sparse time\-aligned text, andw/o Param\-Sep, which removes the hierarchical parameter separation to employ a fully shared architecture\. As shown in Table[1](https://arxiv.org/html/2607.06540#S3.T1), when applying the time\-aligned text, we observe a significant performance decline in bothS→TS\\rightarrow T\(5\.6%\) andS→SS\\rightarrow S\(5\.4%\) settings\. This parallel degradation confirms our hypothesis regarding semantic dilution: the sparse supervision provided by time\-aligned text fails to sustain robust linguistic modeling, which in turn causes knowledge degradation\. In contrast, thew/o Param\-Sepsetting reveals severe modality interference\. While its text generation capability remains relatively stable, its speech accuracy suffers a catastrophic drop to 27\.6%\. This disparity validates our optimization dynamics analysis \(as illustrated in Figure[2](https://arxiv.org/html/2607.06540#S2.F2)\): without physically disentangling the shared parameters, the optimization landscape becomes dominated by semantic modeling, effectively suppressing the learning of acoustic features in deep layers\.

#### Discussion\.

We highlight two critical phenomena that distinguish Lychee\-FD from existing paradigms\. First, our model not only recovers the performance of its half\-duplex backbone \(StepAudio\-2\-mini\) but surpasses it in bothS→TS\\rightarrow T\(\+0\.2%\) andS→SS\\rightarrow S\(\+5\.3%\) settings\. This performance gain, achieved without increasing model depth, strongly validates that our framework realizes robust knowledge retention while maintaining the inference efficiency of the native end\-to\-end model\. This underscores the pivotal role of explicit semantic modeling in driving model intelligence, offering valuable insights for future human\-machine interaction systems\. Second, Lychee\-FD achieves the smallest modality gap among all native CDM models \(e\.g\., FLM\-Audio and Moshi\) and remains competitive with Thinker\-Talker architectures, without requiring intricate multi\-stage training\. This demonstrates that by physically decoupling semantic and acoustic modeling, Lychee\-FD effectively resolves modality interference\. Consequently, our approach allows the model to articulate nature acoustic responses without corrupting its underlying semantic logic, providing a streamlined solution to the efficiency\-intelligence trade\-off\.

### 4\.5Full\-duplex Chatting

#### Metrics\.

For the Full\-duplex Chatting, we select three mainstream benchmarks:FDBench\(DBLP:conf/interspeech/PengCNMN0C25\),FullDuplexBench 1\.0, andFullDuplexBench 1\.5\(DBLP:journals/corr/abs\-2503\-04721\)\. We follow the recommended settings of these benchmarks for evaluating both baselines and our method\. Specifically, FDBench assesses turn\-taking and interruption behaviors using Success\-Replies Rate \(SRR\), Success\-Interrupts Rate \(SIR\), Early\-Interrupts Rate \(EIR\), and Success\-Replies\-to\-Interrupts Rate \(SRIR\), alongside timing metrics such as First\-Speech\-Emit\-Delay \(FSED\) and Interrupt\-Response\-Delay \(IRD\)\. FullDuplexBench 1\.0 consists of four subsets: interruption \(I\), assistant backchannel \(B\), turn\-taking \(T\), and user pause \(P\)\. In addition to the takeover rate \(TOR\) and Interruption stop delay \(Stop\.\), we also report the frequency of generated backchannels during user speech \(B\-Freq\)\. Finally, FullDuplexBench 1\.5 utilizes GPT\-4o\-1124 to classify model responses to user interruptions and backchannels into four categories \(Response, Resume, Uncertain, or Unknown\), reporting the Interruption\-Response Rate \(IRR\), Backchannel\-Resume Rate \(BRR\), as well as the interruption stop delay and response latency\.

#### Result\.

As presented in Table[2](https://arxiv.org/html/2607.06540#S4.T2), Lychee\-FD achieves state\-of\-the\-art performance across 10 of the 11 evaluated interaction metrics, demonstrating superior capability in managing complex conversational dynamics, including interruption, backchanneling, dynamic turn\-taking, and pause handling\. While VITA\-1\.5 exhibits a marginally higher I\-TOR on FullDuplexBench 1\.0, its consistently high B\-TOR and P\-TOR reveal a tendency towards aggressive, indiscriminate speech rather than intelligent turn\-taking\. In contrast, Lychee\-FD maintains a balanced interaction profile, effectively distinguishing between user pauses and interruptions\. Notably, on the challenging FullDuplexBench 1\.5, our model delivers a substantial 28\.5% average improvement over system\-level baselines \(e\.g\., Freeze\-Omni\) that rely on external VAD\. Crucially, this empirical breakthrough directly validates our core scientific insight: by resolving the inherent gradient conflicts that plague fully shared architectures, Lychee\-FD preserves robust semantic awareness during simultaneous listening and speaking\. Consequently, this result not only proves that our model realizes truly natural, fluid, and immersive interaction, but also confirms that LLMs can intrinsically function as highly effective dialog managers\. By eliminating handcrafted signal processing modules, our approach provides an efficient, practical solution to the core problem of full\-duplex communication, substantially pushing forward the frontier of native end\-to\-end SLMs\.

#### Latency\.

We evaluate inference efficiency through two categories: first response latency \(FSED, Lat\.\) and interruption response latency \(IRD, Stop\.\)\. Regarding the former, Lychee\-FD achieves the lowest latency across all benchmarks\. Since our hierarchical parameter separation strategy introduces no additional model depth, we can leverage standard pipeline parallelism for speedup, demonstrating the efficiency of our architectures\. Further details about this real\-time parallel inference are provided in Appendix[10](https://arxiv.org/html/2607.06540#S10)\. For Interruption Response Latency, our model demonstrates an even greater advantage, achieving the lowest stop latency when meeting an interruption \(e\.g\., 570ms Stop\. on FullDuplexBench 1\.5\)\. It is worth noting that interruption latency is a function of both model processing speed and interaction accuracy—since a model must first correctly identify an interruption before it can stop generating\. The superior performance confirms that our architecture incurs no computational overhead, and that its robust semantic awareness enables rapid, accurate reactions to user interventions\.

### 4\.6Speech Generation

![[Uncaptioned image]](https://arxiv.org/html/2607.06540v1/x5.png)

Figure 4:Comparison of speech synthesis quality via UTMOS\. Lychee\-FD \(w/o Param\-Sep\) denotes the variant without hierarchical parameter separation strategy\.
![[Uncaptioned image]](https://arxiv.org/html/2607.06540v1/x6.png)

Figure 5:Gradient Cosine Similarity on the Moshi architecture\. The transition from task synergy in shallow layers to task conflict in deep layers confirms that modality interference is a universal phenomenon in fully shared end\-to\-end SLMs\.

#### Metrics\.

To investigate the quality of the generated speech, we evaluate the content consistency and speech naturalness of the model on LlamaQ dataset\. Specifically, we report the Word Error Rate \(WER\) between the generated text and the transcribed speech to measure content consistency\. Additionally, we employ UTMOS\(DBLP:conf/interspeech/SaekiXNKTS22\), a trained speech quality assessment model, to score the naturalness of the generated audio\.

#### Result\.

High\-fidelity speech synthesis serves as the cornerstone of voice interaction, significantly elevating the immersive quality of full\-duplex conversations\. As illustrated in Figure[4](https://arxiv.org/html/2607.06540#S4.F4), Lychee\-FD achieves the highest UTMOS score of 4\.50, surpassing Freeze\-Omni, Moshi and even its half\-duplex backbone\. This result empirically validates that our proposed hierarchical parameter separation strategy effectively prevents acoustic modeling from semantic interference, thereby preserving fine\-grained prosodic details that are often lost in shared architectures\. When the parameter separation is removed, we observe a significant decline in speech quality\. This finding corroborates our hypothesis regarding modality interference from an acoustic perspective, demonstrating that forcing the acoustic head to share deep layers with text processing objectives inevitably degrades audio fidelity\. Ultimately, by physically disentangling the modeling pathways, Lychee\-FD not only improves generation accuracy but also synthesizes speech with richer acoustic details, delivering a more natural and enjoyable conversational experience\.

## 5Optimization Dynamics Analysis Extension

To verify the universality of our findings, we extend the gradient cosine similarity analysis to Moshi\(DBLP:journals/corr/abs\-2410\-00037\), investigating whether deep\-layer gradient conflicts are architecture\-specific artifacts or a fundamental bottleneck across different FDSLMs\. Following our established methodology, we find that Moshi’s optimization dynamics exhibit a strikingly similar trend \(Figure[5](https://arxiv.org/html/2607.06540#S4.F5)\)\. Shallow layers \(0\-19\) maintain positive similarity \(𝒮\(l\)\>0\\mathcal\{S\}^\{\(l\)\}\>0\), indicating task synergy\. Conversely, as depth increases, the similarity sharply declines to distinctly negative values in deep layers \(23\-31\)\. This highlights the optimization divergence between text and speech tasks\. This cross\-architecture validation empirically confirms that inherent gradient conflicts between semantic and acoustic modeling are not isolated anomalies, but a universal bottleneck in fully shared paradigms\. Consequently, it firmly substantiates the necessity and generalizability of our proposed Hierarchical Parameter Separation\.

## 6Conclusion

In this paper, we address a critical challenge and a long\-standing goal for the speech and NLP communities: developing seamless, high\-performance native full\-duplex spoken language models\. To the best of our knowledge, we are the first to uncover the fundamental root cause of modality interference\. Through an in\-depth analysis of model optimization dynamics, we reveal that the inability to simultaneously listen and speak stems from inherent gradient conflicts within a shared deep parameter space, which specifically manifest as optimization divergence and semantic dilution\. Inspired by these observations, we introduce Lychee\-FD\. Our framework elegantly resolves these bottlenecks by decoupling conflicting modalities via a hierarchical parameter separation strategy, coupled with a semantic alignment channel to enforce robust knowledge retention\. Extensive experiments demonstrate that Lychee\-FD significantly advances the state\-of\-the\-art performance, successfully reconciling ultra\-low latency interaction with deep speech intelligence\. Ultimately, our work substantially pushes forward the frontier of full\-duplex SLMs research, paving the way for the next generation of natural, fluid, and immersive human\-machine interaction\.

## Limitations

While Lychee\-FD enables the development of a high\-performance native full\-duplex framework with highly responsive acoustic processing capabilities, deploying such systems in unconstrained, real\-world open\-mic scenarios remains a promising yet challenging next research frontier\. At present, our model implements highly sensitive interruption detection and can effectively suspend speech output upon any user barge\-in\. However, discriminating between intentional user instructions and incidental background side\-talk \(as illustrated in Appendix[8\.2](https://arxiv.org/html/2607.06540#S8.SS2)\) remains a non\-trivial challenge to address\. This limitation is tentatively attributed primarily to the current data synthesis paradigm, which is largely centered on direct two\-party interactions and lacks detailed intent annotations for complex multi\-speaker scenarios\. This issue points to a critical gap in data coverage within the field, rather than stemming from an architectural bottleneck of the hierarchical framework we propose\. Moving forward, we hope our preliminary work can provide a modest impetus for subsequent research in the community\. By constructing diverse open\-mic datasets and integrating prosodic features for intent disambiguation, future studies may build upon this initial framework to develop truly context\-aware and adaptive selective interruption mechanisms for next\-generation SLMs\.

## References

## 7Layer Ablation

![Refer to caption](https://arxiv.org/html/2607.06540v1/x7.png)Figure 6:Layer ablation study on model performance and computation cost\. Accuracy on LlamaQ saturates after 4 separated layers, while computational overhead grows linearly\. This indicates that 4 layers represent the optimal trade\-off between performance gains and inference efficiency\.To explore the impact of the number of separated layers, we conduct a layer ablation study on both model performance on LlamaQ and static inference cost for speech, as illustrated in Figure[6](https://arxiv.org/html/2607.06540#S7.F6)\. We observe that accuracy follows a steep upward trajectory in the initial stages, surging from 36\.0 to 65\.4 as the depth increases to 4 layers\. This indicates that a relatively shallow separation is sufficient to resolve the primary modality conflicts\. However, beyond this point, the performance gain saturates, yielding diminishing returns, while the computational overhead continues to grow linearly\. Consequently, we identify 4 layers as the optimal configuration\. This choice represents the most favorable trade\-off, securing the vast majority of the performance gain while minimizing the additional parameter budget, thereby ensuring high inference efficiency\.

## 8Taxonomy of Full\-Duplex Scenarios

To intuitively demonstrate the interaction quality of Lychee\-FD, we move beyond isolated examples and analyze user interventions through two basic dimensions: the presence of voice activity and the user’s actual intent to intervene\. Since the default state \(no voice activity and no intent\) simply requires the model to continue listening or speaking, we focus on the remaining three representative scenarios: intentional interruption \(voice with strong intent to intervene\), passive backchannel \(voice with intent to encourage continuation\), and side\-talk \(voice with no intent towards the agent\)\.

![Refer to caption](https://arxiv.org/html/2607.06540v1/x8.png)Figure 7:A case study demonstrating Lychee\-FD’s capability in handling complex turn\-taking dynamics\. Lychee\-FD successfully generates backchannels, halts immediately upon interruption, and provides a contextually accurate response to the user’s specific query without misinterpreting passive feedback\.![Refer to caption](https://arxiv.org/html/2607.06540v1/x9.png)Figure 8:An error analysis illustrating a common limitation in handling side\-talk during full\-duplex interaction\. While the model correctly detects voice activity and halts its speech, it fails to recognize that the user’s utterance is directed at a third party, highlighting a key area for future community research\.### 8\.1Case Study: Interruption and Backchannel

In Figure[7](https://arxiv.org/html/2607.06540#S8.F7), we present a real\-world conversation sample that illustrates the first two scenarios\. As the model begins explaining a recipe, it first employs a natural backchannel \(“Uh\-huh”\) to acknowledge the user’s start\. Crucially, the model exhibits two key capabilities when facing different user interventions:\(1\) Intentional Interruption:When the user interrupts with a specific clarification question \(“what exactly is guanciale?”\), Lychee\-FD halts its speech output almost instantaneously, avoiding the awkward talking\-over phenomenon common in half\-duplex systems\.\(2\) Passive Backchannel:Later, as the model explains the definition of guanciale, the user interjects with a short backchannel \(“I see”\)\. Here, Lychee\-FD demonstrates precise intent understanding\. Instead of misinterpreting this acoustic signal as a barge\-in command—a common failure in fully\-shared models due to semantic dilution—the model correctly identifies it as a passive signal of agreement\. Benefiting from our proposed Semantic Alignment Channel, which preserves robust language modeling during simultaneous listening, Lychee\-FD seamlessly resumes its explanation without unnecessary pauses or topic fragmentation\. This interaction confirms that our Lychee\-FD effectively maintains the model’s language capabilities even during rapid turn\-switching, enabling a fluid, seamless, and truly natural conversational experience\.

### 8\.2Error Analysis: Side\-Talk

Despite the strong performance in standard interactions, we observe certain limitations in handling complex acoustic environments, particularly regarding side\-talk\. In a native full\-duplex setting, the model continuously processes incoming audio\. As shown in Figure[8](https://arxiv.org/html/2607.06540#S8.F8), when the user briefly speaks to a third party in the background \(e\.g\., asking a roommate about food\), the model incorrectly interprets this background conversation as a direct interruption\. Consequently, it halts its current explanation and attempts to respond to the irrelevant query\.

This error indicates that while Lychee\-FD is highly responsive to voice activity, its ability to distinguish between user\-to\-agent commands and user\-to\-human side\-talk remains constrained\. We argue that this is not an architectural limitation of our framework, but rather a common challenge for current native SLMs, largely due to the lack of multi\-speaker intent annotations in existing data synthesis paradigms\. By highlighting this scenario, we hope to provide a blueprint for the community\. Future work will focus on integrating intent detection or utilizing prosodic cues to improve the model’s robustness in open\-mic, multi\-speaker environments\.

## 9Global Gradient Influence Analysis

![Refer to caption](https://arxiv.org/html/2607.06540v1/figures/influence/gradient_heatmap_sci_style.png)Figure 9:Global gradient influence scores among Text and Speech tasks\.Left: The fully shared baseline suffers from severe destructive interference between semantic and acoustic modeling \(negative scores\)\. Right: Our Hierarchical Parameter Separation not only eliminates this conflict but also fosters constructive synergy \(positive scores\) between modalities\.To further validate the effectiveness of our proposed Hierarchical Parameter Separation, we investigate modality interference from a causal perspective\. Inspired byDBLP:journals/corr/abs\-2601\-02967, we quantify how updating parameters for one task causally affects the performance of another\. Specifically, we calculate the normalized global Influence ScoreIm→iI\_\{m\\rightarrow i\}of taskmmon taskii\(m,i∈\{T,A\}m,i\\in\\\{T,A\\\}\):

Im→i=ℒi​\(θ\)−ℒi​\(θ−η​𝐠m\)ℒi​\(θ\)−ℒi​\(θ−η​𝐠i\),I\_\{m\\rightarrow i\}=\\frac\{\\mathcal\{L\}\_\{i\}\(\\theta\)\-\\mathcal\{L\}\_\{i\}\(\\theta\-\\eta\\mathbf\{g\}\_\{m\}\)\}\{\\mathcal\{L\}\_\{i\}\(\\theta\)\-\\mathcal\{L\}\_\{i\}\(\\theta\-\\eta\\mathbf\{g\}\_\{i\}\)\},\(9\)whereθ\\thetadenotes the global model parameters, and𝐠m=∇θℒm\\mathbf\{g\}\_\{m\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{m\}is the gradient derived solely from the objective of taskmm\. The denominator represents the loss reduction when taskiiis updated using its own gradient, serving as a normalization factor\. A negative score \(Im→i<0I\_\{m\\rightarrow i\}<0\) indicates destructive interference, meaning that optimizing taskmmdegrades the performance of taskii\. Conversely, a positive score \(Im→i\>0I\_\{m\\rightarrow i\}\>0\) implies constructive synergy\.

Figure[9](https://arxiv.org/html/2607.06540#S9.F9)visualizes the influence scores between the text and speech tasks\. In the fully shared baseline \(left\), the update of the text task exerts a negative influence on the speech task \(\-0\.129\)\. This empirically confirms our hypothesis: forcing conflicting modalities to update within a shared parameter space causes destructive interference, where acoustic learning actively corrupts semantic modeling\. In contrast, under our Hierarchical Parameter Separation \(right\), this destructive interference is entirely resolved\. Notably, the influence score of text on speech shifts to a positive value \(0\.205\)\. Furthermore, the constructive synergy from speech to text is significantly amplified \(from 0\.312 to 0\.773\)\. This demonstrates that our architecture effectively disentangles conflicting optimization dynamics, allowing the model to acquire robust speech capabilities while mutually reinforcing its language intelligence\.

## 10Real\-Time Inference Algorithm

Deploying Lychee\-FD for real\-time full\-duplex interaction presents a unique system\-level challenge\. Standard LLM inference engines, such as vLLM, are fundamentally designed around a linear layer topology \(L1→L2→…→LDL\_\{1\}\\rightarrow L\_\{2\}\\rightarrow\\dots\\rightarrow L\_\{D\}\) and a single autoregressive output stream\. However, our proposed hierarchical architecture requires the simultaneous generation of text, audio, and control signals\. Forcing this multi\-head architecture into a single GPU or a linear pipeline would result in the sequential execution of the specialized heads, introducing severe latency bottlenecks that disrupt the fluidity of real\-time conversation\.

To bridge the gap between theoretical modality decoupling and practical deployment, we introduceDirected Acyclic Graph Pipeline Parallelism \(DAG\-PP\)by customizing the vLLM engine\. Instead of linear tensor passing, we implement a 1\-to\-N tensor broadcast mechanism at the branching point of the shared backbone\. As illustrated in Algorithm[1](https://arxiv.org/html/2607.06540#algorithm1), the intermediate hidden states \(𝐇shared\\mathbf\{H\}\_\{\\mathrm\{shared\}\}\) are broadcasted via NCCL to multiple GPUs\. This allows the Semantic, Acoustic, and Control heads to execute strictly in parallel across different devices\. Crucially, this parallel execution ensures that the effective model depth on the critical path remains unchanged compared to the half\-duplex backbone,significantly reducing the inference latency bottleneck of multi\-head architectures\.A distributed synchronization barrier is then employed to collect the multi\-stream logits for synchronous sampling before the next autoregressive step\.

This algorithm\-system co\-design demonstrates that our hierarchical parameter separation not only resolves gradient conflicts during training but also unlocks strict physical parallelism during inference\. By hiding the computational overhead of multi\-modal generation behind parallel execution, Lychee\-FD achieves the state\-of\-the\-art interaction intelligence while strictly adhering to the ultra\-low latency constraints of full\-duplex spoken dialogues\.

Input:Text tokens

𝐗T\\mathbf\{X\}^\{T\}, Acoustic tokens

𝐗A\\mathbf\{X\}^\{A\}, Control tokens

𝐗C\\mathbf\{X\}^\{C\}, User speech

𝐔\\mathbf\{U\}
Output:Text logits

𝐎T\\mathbf\{O\}^\{T\}, Acoustic logits

𝐎A\\mathbf\{O\}^\{A\}, Control logits

𝐎C\\mathbf\{O\}^\{C\}
/\*Initial Embedding \(GPU 0\) \*/

𝐄m←Embeddingm​\(𝐗m\),∀m∈\{T,A,C\}\\mathbf\{E\}^\{m\}\\leftarrow\\mathrm\{Embedding\}\_\{m\}\(\\mathbf\{X\}^\{m\}\),\\quad\\forall m\\in\\\{T,A,C\\\};

𝐄U←AudioEncoder​\(𝐔\)\\mathbf\{E\}^\{U\}\\leftarrow\\mathrm\{AudioEncoder\}\(\\mathbf\{U\}\);

𝐄←𝐄T\+𝐄A\+𝐄C\+𝐄U\\mathbf\{E\}\\leftarrow\\mathbf\{E\}^\{T\}\+\\mathbf\{E\}^\{A\}\+\\mathbf\{E\}^\{C\}\+\\mathbf\{E\}^\{U\};

/\*Shared Backbone Execution \(GPU 0\) \*/

𝐇\(0\)←𝐄\\mathbf\{H\}^\{\(0\)\}\\leftarrow\\mathbf\{E\};

for*l←1l\\leftarrow 1toLsharedL\_\{\\mathrm\{shared\}\}*do

𝐇\(l\)←ℱlshared​\(𝐇\(l−1\)\)\\mathbf\{H\}^\{\(l\)\}\\leftarrow\\mathcal\{F\}^\{\\mathrm\{shared\}\}\_\{l\}\(\\mathbf\{H\}^\{\(l\-1\)\}\);

/\*DAG Branching \*/

NCCL\_Broadcast\(

𝐇shared→GPUT,GPUA,GPUC\\mathbf\{H\}\_\{\\mathrm\{shared\}\}\\rightarrow\\mathrm\{GPU\}\_\{T\},\\mathrm\{GPU\}\_\{A\},\\mathrm\{GPU\}\_\{C\}\);

/\*Specialized Heads Execution \*/

parallel for

m∈\{T,A,C\}m\\in\\\{T,A,C\\\}on

GPUm\\mathrm\{GPU\}\_\{m\}
𝐇m←𝐇shared\\mathbf\{H\}^\{m\}\\leftarrow\\mathbf\{H\}\_\{\\mathrm\{shared\}\};

for*k←1k\\leftarrow 1toLmL\_\{m\}*do

𝐇m←ℱkm​\(𝐇m\)\\mathbf\{H\}^\{m\}\\leftarrow\\mathcal\{F\}^\{m\}\_\{k\}\(\\mathbf\{H\}^\{m\}\);

/\*Distributed Synchronization \*/

Barrier\_Synchronize\(\);

return

\(𝐎T,𝐎A,𝐎C\)\(\\mathbf\{O\}^\{T\},\\mathbf\{O\}^\{A\},\\mathbf\{O\}^\{C\}\);

Algorithm 1Directed Acyclic Graph Pipeline Parallelism \(DAG\-PP\)
## 11Data Synthesis Pipeline Details

To address the scarcity of full\-duplex interaction data, we developed an automated data synthesis pipeline\. This pipeline orchestrates interactions between aUser Agentand anAssistant Agent, review by aReviewer Agent\. The process explicitly models complex conversational behaviors including interruptions and backchannels\.

### 11\.1Agent Architecture

- •User Agentis initialized with a specificPersona\(randomly sampled from a pool of diverse profiles\) and aSpeaking Style\(sampled from 19 distinct styles such as Concise and logical, Impatient, Humorous and witty\)\. The agent is instructed to act authentically rather than helpfully\.
- •Assistant Agentgenerates responses based on the conversation history to simulate a realistic AI assistant\.
- •Reviewer Agentevaluates dialogue turns based on persona adherence, event execution quality, and logical flow\.

### 11\.2Interaction Behavior Modeling

#### Interruption Generation\.

We implemented a two\-stage mechanism to generate naturalistic interruptions\. Random interruptions are scheduled between turns 2 and 4\.

1. 1\.Planning Phase:The User Agent analyzes the Assistant’s current response context to determine a validInterruption Motivation\(Correction, Deeper Inquiry, Topic Shift, Strong Emotional Reaction, or Impatience\)\. It then inserts a placeholder tag<interruption/\>at the precise logical point within the Assistant’s text\.
2. 2\.Execution Phase:Conditioned on the chosen motivation and the context prior to the interruption point, the User Agent generates the specific interruption utterance\.

#### Backchannel Injection\.

Backchannels are injected probabilistically \(p=0\.5p=0\.5\) during post\-processing\.

- •User Backchannels:The User Agent reviews the Assistant’s response to insert feedback signals \(uh\-huh, gotcha\) wrapped in<user\_backchannel\>tags\.
- •AI Backchannels:Similarly, the system generates backchannels for the User’s speech to simulate active listening by the Assistant\.

### 11\.3Quality Control

We employ a rigorous filtering process\. AReviewer Agentscores the final dialogue on a scale of 1\-5 across three dimensions: Persona Consistency, Quality of Interruption Event, and Naturalness of Backchannels\. Dialogues with low logical consistency or failed event executions are discarded\.

### 11\.4Prompt Templates

We provide the core system prompts used in our pipeline below\.

Prompt 1: User Role\-Play InstructionSystem Instruction:You are a person in a real\-time voice conversation\. In the conversation history, your lines are marked with "speaker": "You"\. You are talking to the person marked "speaker": "Other"\.You are NOT an AI assistant\.Your task is to speak naturally based on your persona\. React authentically, don’t try to be helpful\.Persona Details:\-Persona:\{persona\}\-Communication Style:\{style\}Conversation Rules:\- Speak, don’t write: Use filler words \(e\.g\., "um", "uh", "like"\), hesitations, and natural phrasing\. \- Stay in character: Let your persona guide your responses\.Task:Now, it’s your turn\. Generate your next response as "You"\.

Prompt 2: Interruption Planning \(Motivation & Placement\)Context:The assistant is currently saying: “\{context\}”Task: Plan and Place the InterruptionYour goal is to find the perfect moment to interrupt, driven by your persona\.Action 1: Plan the Interruption’s Motivation\.First, think aboutwhyyour persona would interrupt here\. Choose a motivation that fits your character:•Correction: The assistant’s response contains a point that may not be entirely accurate, and you want to clarify or refine it\.•Deeper Inquiry: You need to ask for clarification on a key point before they move on\.•Topic Shift: What they said reminds you of something else, and you want to change the subject\.•Strong Emotional Reaction: You are surprised, excited, or disagree strongly and can’t hold it in\.•Impatience: You want to cut to the chase or stop a lengthy explanation\.Action 2: Place the Interruption Marker\.Based on your chosen motivation, find the most natural point in the assistant’s speech to jump in\. Insert ONLY the empty tag pair<interruption\></interruption\>at that precise spot\.

Prompt 3: Interruption Utterance GenerationContext:You just decided to interrupt the assistant while they were saying: “\{context\}” Your motivation for interrupting is:\{motivation\}Task: Deliver the InterruptionNow, say the words you would use to interrupt\. Your utterance must sound spontaneous and directly reflect your motivation\.\[Detailed examples for motivations provided to the model: Correction, Deeper Inquiry, Topic Shift, Strong Emotional Reaction, Impatience\]Output:Generate ONLY the interrupting phrase itself\.

Prompt 4: User Backchannel GenerationTask:As the assistant is speaking, you want to show you’re listening\. The assistant’s last utterance was: “\{context\}”\.•Action 1:Think of a short, spoken backchannel phrase \(e\.g\., "uh\-huh", "gotcha", "right", "mhm"\)\.•Action 2:Find the most natural point in the assistant’s speech to insert this backchannel, wrapped in<user\_backchannel\>tags\.

Prompt 5: Final Dialogue Quality ReviewRole:You are a meticulous evaluator of simulated spoken dialogues\.Scoring Criteria:1\.Persona Consistency & Depth \(1\-5\):Does the user’s speech effectively embody the assigned persona \(\{persona\}\) and style \(\{style\}\)?2\.Quality of Interruption Event \(1\-5\):Is the interruption timed perfectly and motivated by the persona? Does it feel natural or forced?3\.Naturalness of Backchannels \(1\-5\):Are backchannels subtle and placed to improve flow, or are they distracting/robotic?Output:Provide scores and a detailed justification explaining your reasoning\.