Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

# Instruction-Following Speech Language Models without Instruction Tuning
Source: [https://arxiv.org/html/2607.02214](https://arxiv.org/html/2607.02214)
## Unlocking Speech–Text Compositional Powers: Instruction\-Following Speech Language Models without Instruction Tuning

###### Abstract

Instruction tuning for speech language models \(SLMs\) is substantially more challenging than for text\-based large language models \(LLMs\), as it requires learning a new modality and a wide range of speech\-specific instructions in addition to those supported by text LLMs\. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large\-scale speech pre\-training and instruction\-tuning datasets\. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences\. In this paper, we proposeSpeechCombine, an instruction\-following speech language model trainedwithout any instruction tuning, using only a single round of speech pre\-training on 30k hours of data\. Starting from a text LLM base model, we perform continuous pre\-training on speech utterances to obtain a speech\-adapted model, and then directly combine its weights with the weight difference between the instruction\-tuned and base versions of the text LLM\. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain\. These findings suggest a new direction for SLM training that avoids reliance on massive speech data\.

Machine Learning, ICML

## 1Introduction

Recent advances in large language models \(LLMs\) have catalyzed significant progress in speech language models \(SLMs\)\. One key capability of SLMs, as with text LLMs, is the ability to follow*instructions*\. Unlike text LLMs, however, SLMs must handle a much broader spectrum of instructions\. In particular, SLM instructions can be categorized into three types\.

∙\\bulletText\-oriented instructions\.These include instructions that standard text LLMs can already follow, such as question answering \(QA\), reasoning, and article generation\. The key difference is that, for SLMs, the input instructions may be conveyed in either text or speech, and the model’s responses can likewise be produced in either modality\.

∙\\bulletSpeech understanding instructions\.These include instructions that query information from input speech utterances, particularly paralinguistic attributes such as emotion or emphasis\. For these instructions, the input must contain speech, while the output may be in either speech or text\.

∙\\bulletSpeech generation instructions\.These include instructions that require generating speech utterances subject to specific constraints, such as emotion or emphasized words\. For this category, the input requirements may be expressed in either speech or text, but the output must be speech\.

In summary, instruction following in SLMs introduces two additional levels of complexity beyond those faced by text LLMs\. First, an SLM must follow the same instructions as a text LLM, but in a new modality\. Second, it must handle entirely new, speech\-specific instructions that text LLMs cannot address\. As a result, instruction tuning for SLMs is substantially more challenging than for text LLMs\.

To perform instruction tuning for SLMs, most existing methods largely replicate the training pipeline used for text LLMs\(Dinget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib9)\)\. Usually, an SLM is first pre\-trained on a large speech corpus using a next\-token prediction objective, and then fine\-tuned on datasets containing various speech instructions using supervised fine\-tuning \(SFT\) and/or reinforcement learning \(RL\)\(Ghoshet al\.,[2026](https://arxiv.org/html/2607.02214#bib.bib57)\)\. In other paradigms, only speech instruction\-tuning is performed\(Zhanget al\.,[2023](https://arxiv.org/html/2607.02214#bib.bib62)\)\. The training datasets typically consist of discrete speech tokens derived from real speech and/or synthetic speech generated from text datasets originally used for text LLM training\.

However, this paradigm is heavily bottlenecked by thedata inflationproblem\. For example, the sentence“How are you”comprises no more than five tokens in text form, whereas the corresponding speech utterance typically lasts around one second and expands to roughly 60–200 speech tokens\(Wanget al\.,[2025b](https://arxiv.org/html/2607.02214#bib.bib21)\), representing a∼\\sim20×\\timesincrease relative to text\.111Some speech tokenizers\(Zenget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib10)\)have a code rate of 25Hz, but each speech code typically consists of 8 tokens\.Such inflation makes it significantly more difficult to scale the effective dataset size for SLM training\. As a result, compared to text LLM training, SLM training must solve a substantially more challenging problem using far less effective data, which can lead to compromises in both knowledge acquisition and instruction\-following capabilities\.

More importantly, although SLM training is often initialized from a text LLM, subsequent training on massive speech tokens can lead to severecatastrophic forgetting\. As a result, a substantial portion of SLM training is devoted to re\-learning knowledge and skills in the original text LLM, often in a much harder\-to\-learn speech representation\. If the knowledge and capabilities of the text LLM could be better preserved, SLM training would not need to be conducted at such an extensive scale\. Some efforts\(Wanget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib28)\)attempt to address this issue by freezing the text LLM and training only adapters for speech input and output\. However, the resulting SLMs can only support text\-oriented instructions, but not the other speech\-related instructions\.

In summary, existing SLM training paradigms involve an inherent trade\-off between preserving the knowledge and capabilities of text LLMs and acquiring new, speech\-specific skills\. This raises a natural question:Is it possible to achieve both objectives and thereby build more powerful SLMs?

In this paper, we proposeSpeechCombine, an SLM with strong instruction\-following capabilities across all three instruction categories, whose training procedure is surprisingly simple: no iterative rounds of training, but onlya single round of pre\-trainingwith a standard next\-token prediction objective on just 30k hours of speech data\. The core idea behindSpeechCombineis to directly transfer the instruction\-following capabilities of text LLMs to a new modality and new skills, inspired by recent advances in model merging\(Huanget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib5)\)\.

Figure[1](https://arxiv.org/html/2607.02214#S1.F1)illustrates the basic idea\. Given a text LLM base model and its instruction\-tuned counterpart, we first compute their parameter difference, denoted asΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}\(blue arrow\), which can be interpreted as a direction encoding instruction\-following capability\. Starting again from the base model, we then perform continuous pre\-training on speech data, yielding a second weight differenceΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}\(black arrow\) that captures knowledge of the speech modality\. Finally,SpeechCombineis constructed by transplantingΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}onto the speech\-adapted model \(red arrow\), based on the hypothesis that instruction\-following capabilities can generalize to the speedh domain\. In essence, our approach combines the speech adaptation directionΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}with the instruction\-following directionΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}in parameter space, hence the nameSpeechCombine\.

![Refer to caption](https://arxiv.org/html/2607.02214v1/x1.png)Figure 1:Illustration ofSpeechCombinein weight space\.Our experiments show that this combination unleashes surprising compositional effects, enabling the model to follow instructions across all three categories\. The ability to follow speech understanding and generation instructions is particularly striking because neitherΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}norΔ​𝜽i​n​s​t​r\\Delta\\bm\{\\theta\}\_\{instr\}has been exposed to these instructions during training\. Furthermore, we find that this combination strategy also transfers other capabilities of the text LLM to the speech domain, such as*long\-thinking capabilities*\. These findings open up a new possibility for SLM training, escaping from the undue ordeal of inefficient data piling in today’s world\.

## 2Related Work

Speech Language Models\.SLMs can be classified into two primary categories: native multimodal and modular\-aligned models\. Native multimodal models\(Défossezet al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib37); Xie and Wu,[2024](https://arxiv.org/html/2607.02214#bib.bib38); Chenet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib1); Dinget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib9); Zenget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib10); Genget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib11); Wuet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib12); Xuet al\.,[2025a](https://arxiv.org/html/2607.02214#bib.bib61); Ghoshet al\.,[2026](https://arxiv.org/html/2607.02214#bib.bib57)\)employ a large language model \(LLM\) as the core reasoning backbone and extend it to the speech domain via audio encoders and speech decoders\. In contrast, modular\-aligned models\(Shaoet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib26); Luet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib27); Wanget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib28); Fathullahet al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib29); Kanget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib30); Wanget al\.,[2025a](https://arxiv.org/html/2607.02214#bib.bib60)\)freeze the backbone LLM while training modality adapters to grant the LLM audio capabilities\. These methods either require large\-scale training or are confined to text\-related instructions\. Existing approaches typically rely on large\-scale audio instruction turning data, which is often expensive at scale\. We aim to bridge this gap\.

Expressive Speech Synthesis\.Recent expressive TTS is moving from label\-based control toward open\-vocabulary, natural\-language instructions for steering paralinguistic attributes\. HiStyle\(Zhanget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib16)\)improves text\-to\-style alignment with a hierarchical style predictor, while ParaStyleTTS\(Louet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib17)\)enhances robust paralinguistic control without reference audio\. OV\-InstructTTS\(Renet al\.,[2026](https://arxiv.org/html/2607.02214#bib.bib18)\)and FlexiVoice\(Chenet al\.,[2026](https://arxiv.org/html/2607.02214#bib.bib19)\)further strengthen instruction\-following for expressive/zero\-shot settings, the latter leveraging preference optimization to disentangle content, timbre, and style\. Beyond single utterances, DeepDubbing\(Daiet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib20)\)uses contextual InstructTTS for consistent audiobook narration\.

Speech Tokenizers\.Speech tokenizers can be categorized intodiscreteorcontinuoustokenization\. Recent discrete tokenizers emphasize low bitrates and enhanced feature factorization: TaDiCodec\(Wanget al\.,[2025b](https://arxiv.org/html/2607.02214#bib.bib21)\)employs a text\-aware diffusion approach for ultra\-low\-rate modeling, while LongCat\-Audio\-Codec\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib22)\)offers a streaming\-friendly tokenizer\-detokenizer solution tailored for practical SLMs\. To improve generation quality, DSA\-Tokenizer\([Zhaoet al\.,](https://arxiv.org/html/2607.02214#bib.bib23)\)disentangles semantic and acoustic features through hierarchical fusion\. Alternatively, ProsodyLM\(Qianet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib2)\)represents speech as transcripts augmented with explicit word\-level prosody tokens\. Several recent works\(Yanget al\.,[2025b](https://arxiv.org/html/2607.02214#bib.bib31); Heet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib32); Liet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib33)\)explorecontinuoustokenization\. and aim to bypass the inherent information loss of quantization\.

Model Merging\.Model merging aims to integrate multiple capabilities within a single parameter space by combining model weights\. Early works such as Task Arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2607.02214#bib.bib6)\)and Model Soups\(Wortsmanet al\.,[2022](https://arxiv.org/html/2607.02214#bib.bib7)\)demonstrate that simple linear combinations of fine\-tuned models can yield robust multi\-task performance\. Subsequent studies, including Chat Vector\(Huanget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib5)\), BILLY\(Paiet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib4)\), and Preference Vector\(Lianget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib3)\), further show that task\- or alignment\-specific behaviors can be represented as weight\-difference vectors and transferred across models without additional training\. Model merging has also been explored as a strategy to mitigate catastrophic forgetting in multimodal adaptation\(Chenet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib1); Sokaret al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib14); Yu and Ananiadou,[2025](https://arxiv.org/html/2607.02214#bib.bib15)\)\. Fun\-Audio\-Chat\(Chenet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib1)\)adopts the model merging idea in speech SLM, but it does not utilize the text LLM base model, which is shown to yield a more robust foundation for model merging\(Anonymous,[2025](https://arxiv.org/html/2607.02214#bib.bib35)\)\.

## 3Method

In this section, we describe howSpeechCombineis trained\. Our goal is to retain the knowledge and instruction\-following capabilities of a text LLM, while teaching the model to follow speech\-related instructions\.

### 3\.1TheSpeechCombineFramework

![Refer to caption](https://arxiv.org/html/2607.02214v1/x2.png)Figure 2:TheSpeechCombineframework and inference pipeline\.Figure[2](https://arxiv.org/html/2607.02214#S3.F2)illustrates the overall framework ofSpeechCombine\. Speech inputs are first converted into discrete speech tokens by the encoder module and then fed into the LLM\. The LLM’s generated outputs, also in the form of speech tokens, are subsequently converted back into speech by the decoder\. In this paper, we focus exclusively on training the LLM module\. The encoder and decoder are instantiated using a pre\-trained, open\-source system \(Section[3\.5](https://arxiv.org/html/2607.02214#S3.SS5)\) and are kept frozen throughout training\.

Section[1](https://arxiv.org/html/2607.02214#S1)and Figure[1](https://arxiv.org/html/2607.02214#S1.F1)have already provided a brief overview of our weight combination framework for the LLM module\. Here, we present a more formal description\. Let𝜽b​a​s​e\\bm\{\\theta\}\_\{base\}and𝜽i​n​s​t\\bm\{\\theta\}\_\{inst\}denote the parameters of a text LLM base model and instruct model, respectively\. Starting from the*base model*, we perform continuous pre\-training on a speech corpus, yielding parameters𝜽s​p​e​e​c​h\\bm\{\\theta\}\_\{speech\}\. We then define two directions in parameter space:

Δ​𝜽i​n​s​t=𝜽i​n​s​t−𝜽b​a​s​e,Δ​𝜽s​p​e​e​c​h=𝜽s​p​e​e​c​h−𝜽b​a​s​e\.\\Delta\\bm\{\\theta\}\_\{inst\}=\\bm\{\\theta\}\_\{inst\}\-\\bm\{\\theta\}\_\{base\},~~\\Delta\\bm\{\\theta\}\_\{speech\}=\\bm\{\\theta\}\_\{speech\}\-\\bm\{\\theta\}\_\{base\}\.\(1\)Δ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}andΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}contain two complementary capabilities:Δ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}encodes instruction\-following behavior but contains no knowledge of speech, whileΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}captures knowledge of the speech modality but lacks the ability to follow natural language instructions\. OurSpeechCombinemodel, parameterized by𝜽S​C\\bm\{\\theta\}\_\{SC\}, is obtained by linearly combining these two directions,*i\.e\.*,

𝜽S​C=𝜽b​a​s​e\+λ​Δ​𝜽s​p​e​e​c​h\+Δ​𝜽i​n​s​t,\\bm\{\\theta\}\_\{SC\}=\\bm\{\\theta\}\_\{base\}\+\\lambda\\Delta\\bm\{\\theta\}\_\{speech\}\+\\Delta\\bm\{\\theta\}\_\{inst\},\(2\)whereλ∈\(0,1\]\\lambda\\in\(0,1\]is a soft weight\.

One way to interpret the above combination is as follows\. The instruction\-following capabilities encoded inΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}are tailored to the knowledge present in the base model\. Equation \([2](https://arxiv.org/html/2607.02214#S3.E2)\) can thus be viewed as transferringΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}onto the speech\-adapted model,𝜽b​a​s​e\+λ​Δ​𝜽s​p​e​e​c​h\\bm\{\\theta\}\_\{base\}\+\\lambda\\Delta\\bm\{\\theta\}\_\{speech\}, which incorporates new knowledge of speech\. The success of this transfer depends on how close the speech\-adapted model remains to the original base model\. A smaller value ofλ\\lambdakeeps the speech\-adapted model closer to the base model, making it easier forΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}to generalize\. However, choosingλ\\lambdatoo small would limit the acquisition of speech knowledge\. Therefore, this trade\-off must be carefully balanced\.

One important caveat is that the continuous pre\-training on speech must start from the base model𝜽b​a​s​e\\bm\{\\theta\}\_\{base\}rather than the instruction\-tuned model𝜽i​n​s​t\\bm\{\\theta\}\_\{inst\}, which is a major distinction from existing SLM training recipes that also involve model merging\(Chenet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib1)\)\. Training from base is crucial in retaining the compositional structure of knowledge and skill inside the SLM to allow for a successful combination\.

Since the training ofSpeechCombineonly involves one round of pre\-training, the algorithm design boils down to designing a good recipe for the continuous pre\-training, which we detail in Sections[3\.2](https://arxiv.org/html/2607.02214#S3.SS2)to[3\.5](https://arxiv.org/html/2607.02214#S3.SS5)\.

### 3\.2Knowledge of Speech

Before introducing the details of speech pre\-training \(which aims to learn speech knowledge\), it is important to explain what speech knowledge is\. Speech carries multiple levels of information, including ❶content, which can typically be transcribed into text; ❷prosody, which describes properties such as pitch, intonation, rhythm, and loudness; and ❸timbre, which characterizes the speaker’s voice\. Since the majority of the speech\-related instructions are about the prosodic aspect of speech, such as expressive generation, word emphasis detection,*etc\.*, we limit ourselves to the first two levels of information in this paper\. Future generalization to timbre is easily achievable in our framework\.

The speech modality differs from text in two fundamental ways\. First, speech token sequences differ substantially from text token sequences in their structure and vocabulary\. Second, speech carries much richer information than text\. As a result, during pre\-training, an SLM must achieve two goals: ❶become familiar with speech token sequences, and ❷understand the information in speech\(content and prosody in this paper\)\. Section[3\.3](https://arxiv.org/html/2607.02214#S3.SS3)describes how we design the pre\-training scheme to achieve these two goals\.

### 3\.3Pre\-training Data Structure

We adopt the standard next\-token prediction objective for pre\-training\. As a result, the key design choice in our pre\-training recipe lies in constructing training data that can achieve the two goals described above under the next\-token prediction regime\. To this end, we design the following pre\-training data structure:

\[S1 cap\] \[S1 text\] \[S1 speech\] \[S1 cap\] \[S2 cap\] \[S2 text\] \[S2 speech\] \[S2 cap\]⋯\\cdots

‘S1’ stands for sentence 1; ‘speech’ stands for speech token sequence; ‘cap’ stands for speech caption\.

The speech captions describe the information conveyed in speech, including both content and prosody, in natural language, so the\[cap\]section is essential for achieving the second goal outlined in Section[3\.2](https://arxiv.org/html/2607.02214#S3.SS2)\. Each\[speech\]section contains the speech token sequence for a single sentence and familiarizes the SLM with the structure of speech token sequences, thereby addressing the first goal\. Sections[3\.4](https://arxiv.org/html/2607.02214#S3.SS4)and[3\.5](https://arxiv.org/html/2607.02214#S3.SS5)detail our designs of the speech caption and speech token sequence, respectively\.

During both pre\-training and inference, each\[speech\]section is always preceded by a\[text\]section, which contains the*exact transcription*of the corresponding speech tokens\. Although this design introduces some redundancy, we show that it is necessary to facilitate the transfer of text\-based instruction\-following capabilities to speech \(Section[3\.6](https://arxiv.org/html/2607.02214#S3.SS6)\)\. Section[3\.5](https://arxiv.org/html/2607.02214#S3.SS5)describes how we reduce this redundancy through speech tokenization\.

The\[cap\]section may appear before or after\[speech\]\. We adopt a randomization strategy: for each sentence,\[cap\]appears before\[speech\]with probability0\.30\.3, appears after\[speech\]with probability0\.30\.3, and is omitted with probability0\.40\.4\. This ensures that the pre\-training data contain sufficient instances of the following three transition patterns: ❶\[cap\]\[text\]\[speech\], which teaches knowledge for speech generation; ❷\[text\]\[speech\] \[cap\], which teaches knowledge for speech understanding; and ❸\[text\]\[speech\]\[text\]\[speech\], which teaches knowledge for multi\-sentence speech generation\. Other pre\-training details can be found in Appendix[A\.1](https://arxiv.org/html/2607.02214#A1.SS1)\.

### 3\.4Speech Caption

To generate a caption for a speech utterance, we first extract the following attributes from the utterance: average pitch level \(high, medium, or low\), average speaking rate \(high, medium, or low\), emphasized words, perceived emotion, emotional arousal \(i\.e\., the intensity of the emotion\), and the text transcription\. We then randomly select a subset of these attributes to include in the caption: specifically, we sample an integerkkuniformly from11to the total number of available attributes for the sentence and includekkattributes in the caption\. Finally, the selected attributes are provided to GPT\-OSS 120B\(Agarwalet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib54)\), which is prompted to generate a natural\-language description that covers all supplied attributes\. Below is an example generated caption:

<cap\>The spoken excerpt is a read passage rendered at a medium tempo, featuring a high pitch, spotlighting the word “nose”, and expressing anger\.</cap\>

Additional details are provided in Appendix[A\.2](https://arxiv.org/html/2607.02214#A1.SS2)\.

### 3\.5Speech Tokenization

To reduce redundancy between the\[text\]–\[speech\]pairs, we aim to adopt a tokenization scheme whose encoded information has minimal overlap with the text transcription\.

To this end, we adopt the prosody tokenization scheme proposed inQianet al\.\([2025](https://arxiv.org/html/2607.02214#bib.bib2)\), which encodes only prosodic information in speech and therefore produces substantially shorter token sequences than schemes that encode the full waveform\. Specifically, for each word in a speech utterance, the prosody tokenization scheme generates five numerical values describing the word’s median pitch, pitch range, pitch slope, duration, and energy\. These values are quantized into 512 discrete levels, yielding a sequence of discrete*prosody tokens*\. We introduce delimiter tokens<speech\>and</speech\>to enclose each speech token sequence\. Additional details are provided in Appendix[A\.3](https://arxiv.org/html/2607.02214#A1.SS3)\.

### 3\.6Inference

Once the model has been pre\-trained and the instruction direction has been integrated according to Equation \([2](https://arxiv.org/html/2607.02214#S3.E2)\), the resultingSpeechCombineis ready to answer real user queries\. This naturally raises our next research questions: What input–output format should be used during inference? CanSpeechCombine, which is never trained on the inference tasks themselves, reliably follow this format?

The following shows our inference template, based on a model derived fromQWEN3\-8B\(Yanget al\.,[2025a](https://arxiv.org/html/2607.02214#bib.bib59)\)\.

<\|im\_start\|\> User What is the emotion of the following utterance?\[text\]\[speech\]<\|im\_end\|\>

<\|im\_start\|\> Assistant \[text\]\[speech\]\[text\]\[speech\]\.\.\.<\|im\_end\|\>

As can be observed, our answer template is the same as that of the corresponding LLM instruct model, except that it includes\[speech\]sections to enable speech\-mode interaction\. Since each\[speech\]section is always accompanied by its corresponding\[text\]section \(as discussed in Section[3\.3](https://arxiv.org/html/2607.02214#S3.SS3)\),SpeechCombinerelies on an external ASR system to transcribe an input speech query into the\[text\]section before prosody tokens are extracted, as illustrated in Figure[2](https://arxiv.org/html/2607.02214#S3.F2)\. We leave internalizing the ASR module within the model as a direction for future work\.

The answer template above also clarifies whySpeechCombine, despite not being trained on this inference format, is still able to produce correct responses\. This is because the instruction\-following direction of the text LLM \(Δ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}\) already recognizes most of the template structure\. The only additional element, the\[speech\]section, is readily handled by the speech pre\-training direction \(Δ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}\)\. The\[text\]sections serve as anchors for transferring the text LLM’s capabilities, which is why the\[text\]and\[speech\]sections must always appear in pairs\.

To further improve the stability ofSpeechCombinefollowing this format, we introduceformat forcing\. Specifically, when generating a\[text\]section, we mask out the speech\-token vocabulary to prevent unintended speech outputs\. When speech mode is enabled, we boost the generation probability of the delimiter<speech\>to ensure that a speech section is produced at the end of each text section\. Additional inference details are provided in Appendix[A\.4](https://arxiv.org/html/2607.02214#A1.SS4)\.

### 3\.7Enabling Long Thinking

Section[3\.6](https://arxiv.org/html/2607.02214#S3.SS6)shows that, as long as the answer template closely matches that of the text LLM, the instruction\-following abilities of the instruct model \(𝜽i​n​s​t\\bm\{\\theta\}\_\{inst\}\) can be readily transferred toSpeechCombine\. We thus wonder: Can other abilities in𝜽i​n​s​t\\bm\{\\theta\}\_\{inst\}be transferred as well?

We thus seek to elicit another prominent capability of many text LLMs –long thinking, by modifying the answer template in the same manner used to invoke long thinking in text LLMs\. For example, when the text LLM isQWEN3\-8B, we simply insert a<think\>\[text\]</think\>section immediately afterAssistant\. This is implemented by forcing the first token afterAssistantto be<think\>\. Additional design choices that stabilize the onset of long\-thinking generation are discussed in Appendix[A\.5](https://arxiv.org/html/2607.02214#A1.SS5)\.

Unlike other SLMs,SpeechCombineacquires long thinking almost‘for free’– without any modification to the training procedure or weight combination, but simply through a change in the answer template\. Accordingly, we enable the long\-thinking mode by default in this paper\.

## 4Experiments

Table 1:Results on text\-oriented tasks\. The metric is accuracy \(%\)↑\\uparrowfor all the tasks\. Among all the methods inGroup B, except forGPT\-4o\-Audiowith significantly larger model size, the best results are noted in bold, second best results underlined\.In this section, we present experiments to evaluate the effectiveness ofSpeechCombine\.222Github:[https://github\.com/CongruiDu/SpeechCombine](https://github.com/CongruiDu/SpeechCombine) Demo Webpage:[https://auspicious3000\.github\.io/SpeechCombine\-Demo](https://auspicious3000.github.io/SpeechCombine-Demo)Unlike most existing work in this area, our approach is not directly trained on any of the evaluation tasks\. We divide the evaluation tasks into two categories based on the required levels of skill combination:

Shallow combination\.This category includes text\-oriented tasks that the text LLM can already perform, but posed and answered in the*speech*modality\. These tasks primarily require combining the problem\-solving capabilities inΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}with the speech input–output format learned inΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}, which is relatively straightforward\.

Deep combination\.This category includes speech\-related tasks, such as speech generation and speech understanding, that are completely unseen during training\. These tasks require transferring the instruction\-following capabilities inΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}to the newly acquired speech knowledge inΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}, which is much more challenging\.

We will first present results on shallow combination in Section[4\.2](https://arxiv.org/html/2607.02214#S4.SS2), then deep combination in Sections[4\.3](https://arxiv.org/html/2607.02214#S4.SS3)and[4\.4](https://arxiv.org/html/2607.02214#S4.SS4)\.

### 4\.1Experimental Configurations

We choose theQWEN3\-8B\-baseand\-instructas our𝜽b​a​s​e\\bm\{\\theta\}\_\{base\}and𝜽i​n​s​t\\bm\{\\theta\}\_\{inst\}, respectively\. For the continuous pre\-training on speech, we apply LoRA\-based tuning\(Huet al\.,[2022](https://arxiv.org/html/2607.02214#bib.bib40)\)with rank 64 andα\\alpha16\. The pre\-training datasets includeLibrilight\(Kahnet al\.,[2020](https://arxiv.org/html/2607.02214#bib.bib41)\),BEAT\(Liuet al\.,[2022](https://arxiv.org/html/2607.02214#bib.bib42)\),CREMA\-D\(Caoet al\.,[2014](https://arxiv.org/html/2607.02214#bib.bib43)\),ESD\(Zhouet al\.,[2022](https://arxiv.org/html/2607.02214#bib.bib44)\),JL Corpus\(Jameset al\.,[2018](https://arxiv.org/html/2607.02214#bib.bib45)\),EmoV\-DB\(Adigweet al\.,[2018](https://arxiv.org/html/2607.02214#bib.bib46)\),Expresso\(Nguyenet al\.,[2023](https://arxiv.org/html/2607.02214#bib.bib47)\),MEAD\(Wanget al\.,[2020](https://arxiv.org/html/2607.02214#bib.bib48)\)TESS\(Pichora\-Fuller and Dupuis,[2020](https://arxiv.org/html/2607.02214#bib.bib49)\)\. We followQianet al\.\([2025](https://arxiv.org/html/2607.02214#bib.bib2)\)to extract the prosody tokens and usewhisper\-large\-v3\(Radfordet al\.,[2023](https://arxiv.org/html/2607.02214#bib.bib51)\)to extract text transcriptions, and the total amount of data, after dropping extration failures, is only around 30k hours\.λ\\lambdais set to 0\.85\.

We include two groups of baselines\.Group Aconsists of alternative training methods using comparable training data and the same base models as forSpeechCombine\. This group is more controlled to better study the effectiveness ofSpeechCombinecompared to the common SLM training paradigms\. It includes \-

∙\\bulletASR \+ Text LLM:Input speech is converted to text usingwhisper\-large\-v3and then fed to theQWEN3\-8B\-instructmodel;

∙\\bulletContinuous Pre\-Train:Continuous pre\-training on theQWEN3\-8B\-instructmodel \(instead ofbase\), using the same 30k hours of speech data;

∙\\bulletContinuous Pre\-Train \+ SFT:Continuous pre\-training on theQWEN3\-8B\-basemodel, using the same 30k hours of speech data, and then further fine\-tuning on around 10,000 hours of speech instruction\-tuning data fromSIFT\-50M\(Pandeyet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib55)\),InstructS2S\-200K\(Fanget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib56)\), andVoiceAssistant\-400K\(Xie and Wu,[2024](https://arxiv.org/html/2607.02214#bib.bib38)\), using SFT;

Group Bconsists of state\-of\-the\-art SLMs, including:GPT\-4o\-Audio\(OpenAI,[2024](https://arxiv.org/html/2607.02214#bib.bib13)\),GLM\-4\-Voice\(9B\)\(Zenget al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib10)\),Audio Flamingo 3\(Ghoshet al\.,[2026](https://arxiv.org/html/2607.02214#bib.bib57)\),Step\-Audio2\-mini\(7B\)\(Wuet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib12)\),Step\-Audio2\-mini\-think\(7B\)\(Wuet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib12)\),OSUM\-EChat\(3B\)\(Genget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib11)\),QWEN\-2\.5\-Omni\(Xuet al\.,[2025b](https://arxiv.org/html/2607.02214#bib.bib58)\),Kimi\-Audio\(7B\)\(Dinget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib9)\), andFun\-Audio\-Chat\(8B\)\(Chenet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib1)\)\. All baselines undergo multiple rounds of training on substantially larger datasets thanSpeechCombine\. For example,Fun\-Audio\-Chatwas trained on millions of hours of speech data\(Chenet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib1)\), over 100×\\timesthe size of our training data\. All the models are of similar sizes, except forGPT\-4o\-Audio, whose results are thus excluded from bolding\.

### 4\.2Text\-Oriented Tasks

We consider two text\-oriented tasks: QA and logical reasoning\. QA is included because it represents a common application for SLM, while logical reasoning is used to challenge the models with more complex questions\. For QA, we evaluate onOpenbookQA,SDQA, andMMSUinVoiceBench\(Chenet al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib24)\)\. For logical reasoning, we evaluate onGSM8kEval,MLCpro\-en,TruthfulEval, andMLCinURO\-Bench\(Yanet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib25)\)\. Additional details are provided in Appendix[B\.1](https://arxiv.org/html/2607.02214#A2.SS1)\.

Table 2:Results on speech understanding and generation tasks\. Among all the methods inGroup B, except forGPT\-4o\-Audiowith significantly larger model size, the best results are noted in bold, second best results underlined\.Table[1](https://arxiv.org/html/2607.02214#S4.T1)summarizes the results\. Among the Group A methods,ASR \+ TextLLMrepresents the topline performance, because it does not suffer from any catastrophic forgetting\. Nevertheless,SpeechCombineperforms on par or even better compared with this topline\. With additional training, the other two methods in Group A yield much compromised performance, underperformingSpeechCombine\. However, the catastrophic forgetting is not as severe as we anticipated, which we suspect is due to the small dataset size for the additional training\. As shown in Sections[4\.3](https://arxiv.org/html/2607.02214#S4.SS3)and[4\.4](https://arxiv.org/html/2607.02214#S4.SS4), although Group A methods decently retain text LLM’s knowledge, it is at the cost of severe failures in speech\-related tasks, reflecting difficult trade\-offs in these conventional training paradigms\.

Among the Group B methods with comparable model sizes,SpeechCombineachieves either the top or second\-best results across all datasets\. This performance is particularly notable given that the top\-performing SLM baselines are trained on extensive math and reasoning datasets to strengthen these capabilities\(Dinget al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib9)\), whereasSpeechCombineis not\. These results demonstrate that our weight combination strategy successfully preserves the original capabilities of the text LLM, while seamlessly integrating speech\-mode interaction without interfering with correct content generation\.

### 4\.3Speech Understanding Tasks

![Refer to caption](https://arxiv.org/html/2607.02214v1/x3.png)Figure 3:Excerpts of thinking processes ofSpeechCombine\. Non\-text sections are removed for readability\.We consider two speech understanding tasks: emotion understanding and emphasis detection\. Emotion understanding requires identifying the speaker’s emotion based on prosodic cues in an utterance\. We evaluate this task on theUnderEMO\-enbenchmark fromURO\-Bench\. Emphasis detection aims to identify words in an utterance that are emphasized by the speaker\. We adapt theEmphAssessdataset\(de Seysselet al\.,[2024](https://arxiv.org/html/2607.02214#bib.bib50)\), which was originally a benchmark for mimicking emphasis patterns from a reference utterance\. We treat the reference utterance as the input speech and use the provided emphasis annotations as ground truth\. Performance is evaluated using precision, recall, and F1 score computed over the predicted emphasized words\. Additional details are provided in Appendix[B\.2](https://arxiv.org/html/2607.02214#A2.SS2)\.

Table[2](https://arxiv.org/html/2607.02214#S4.T2)\(Panel a\) reports the results\. As shown,SpeechCombinedoes not achieve top\-tier performance on emotion understanding\. This is likely due to the insufficient emotion information in the speech captions: only about 100 hours of the pre\-training corpus include emotion captions\. In contrast, more than 15k hours contain emphasis\-related information\. As a result,SpeechCombineachieves state\-of\-the\-art performance on the emphasis detection task, with substantial margins over all baselines\. Group A methods, with no or insufficient adaptation to speech, fail in speech understanding tasks, especially for emphasis detection\. Yet their performance does not drop to zero because they can still partially infer the answers based on text only\.

One important caveat is that speech understanding capabilities are inherently more difficult to elicit than shallow combination capabilities\. In practice, we find thatSpeechCombinetends to answer such questions primarily based on textual content, rather than prosodic cues, unless additional measures are taken to explicitly encourage prosody\-aware reasoning\. For example, in the emphasis detection task, we prepend“Based on the prosody”to the long\-thinking section to force the model to attend to prosodic information\. Section[4\.5](https://arxiv.org/html/2607.02214#S4.SS5)provides a more detailed analysis of this behavior\. This design choice introduces a mild advantage over baseline methods\. Nevertheless, the strong performance ofSpeechCombineremains encouraging: It confirms that the weight combination strategy is capable of cultivating novel speech understanding capabilities, even though additional mechanisms are currently required to activate them\.

### 4\.4Speech Generation Tasks

We consider two speech generation tasks: expressive speech generation and emphasis generation\. Expressive speech generation requires producing speech utterances that satisfy given emotion constraints\. We evaluate this task using theGenEmotion\-enbenchmark fromURO\-Bench\. Performance is measured using a composite score that combines word error rate with the correctness of the expressed emotion, as determined by an external emotion classifier\. Emphasis generation involves generating speech utterances that satisfy given emphasis requirements\. For this task, we again adapt theEmphAssessdataset by converting emphasis labels into emphasis constraints, and we use the provided emphasis detector to evaluate whether the generated speech emphasizes the required words\. Performance is reported using precision, recall, and F1 score\. Additional details are provided in Appendix[B\.3](https://arxiv.org/html/2607.02214#A2.SS3)\.

Table[2](https://arxiv.org/html/2607.02214#S4.T2)\(Panel b\) reports the results\. Surprisingly, speech generation capabilities prove easier to elicit than speech understanding capabilities, and do not require any task\-specific enforcement mechanisms\. The resulting generation quality is strong:SpeechCombineachieves the second\-best performance on emotion generation and the best performance on emphasis generation, with clear margins over the baselines\. Group A methods, in particular, achieve the worst performance due to the insufficient acquisition of speech skills\. We encourage readers to refer to our demo webpage to judge the perceptual quality of the generation\.

### 4\.5Visualizing the Thinking Process

![Refer to caption](https://arxiv.org/html/2607.02214v1/Figures/openbookqa_ablation_lambda.png)\(a\)QA
![Refer to caption](https://arxiv.org/html/2607.02214v1/Figures/emphaunder_ablation_lambda.png)\(b\)Emphasis Detection
![Refer to caption](https://arxiv.org/html/2607.02214v1/Figures/genemotion_ablation_lambda.png)\(c\)Expressive Speech Generation

Figure 4:Performance across differentλ\\lambda\.To gain further insight into howSpeechCombineleverages its combined capabilities, we select three representative tasks –OpenbookQA, emphasis detection onEmphAssess, andGenEmo– one from each task category\. For each task, we visualize the model’s reasoning process and answer for a representative example\.

Figure[3](https://arxiv.org/html/2607.02214#S4.F3)\(a\) illustrates the thinking process for a QA example, which closely resembles the typical reasoning behavior of text LLMs\. This observation further confirms that the original capabilities of the text LLM are well preserved\.

Figure[3](https://arxiv.org/html/2607.02214#S4.F3)\(b\) and \(c\) show the thinking process for the same emphasis detection question with \(c\) and without \(b\) prepending“Based on the prosody”to the thinking \(Section[4\.3](https://arxiv.org/html/2607.02214#S4.SS3)\)\. Interestingly, without the forcing,SpeechCombineattempts to infer what should be emphasized based on the text, which leads to an incorrect answer\. Only when it is forced to focus on prosody does it start to reason what is actually being emphasized by the speaker, which leads to the correct answer\. This contrast suggests that while weight combination enables the emergence of speech understanding capabilities, it is not sufficient on its own to reliably signal when and how these capabilities should be invoked\.

Figure[3](https://arxiv.org/html/2607.02214#S4.F3)\(d\) shows the thinking process for an expressive speech generation task\. Remarkably, the model is able to reason about the most appropriate prosodic styles to accentuate the requested emotion of excitement, exhibiting a previously unseen form of long\-thinking behavior\. Specifically, the model considers increasing pitch or speaking rate, and ultimately decides to increase energy\. This observation provides strong evidence that long\-thinking behavior has been extended to solve unseen, speech\-centric tasks\.

### 4\.6Ablation Studies

Table 3:Ablation study results\.Combination weight\.As discussed in Section[3\.1](https://arxiv.org/html/2607.02214#S3.SS1), the parameterλ\\lambdain Equation \([2](https://arxiv.org/html/2607.02214#S3.E2)\) controls the trade\-off between acquiring speech\-specific knowledge and preserving the transferability of text\-based instruction\-following capabilities\. We examine how different choices ofλ\\lambdaaffect performance, using the same three tasks considered in Section[4\.5](https://arxiv.org/html/2607.02214#S4.SS5)\.

Figure[4](https://arxiv.org/html/2607.02214#S4.F4)\(solid curves\) plots task performance as a function ofλ\\lambda, revealing clear contrasting trends\. Asλ\\lambdaincreases, performance on QA, a text\-oriented task, degrades due to greater perturbations to the original text LLM weights, while performance on the two speech\-related tasks improves as speech knowledge is more strongly integrated\. Despite this trade\-off, there exists a broad sweet spot forλ\\lambdathat yields competitive, well\-balanced performance across all tasks\.

Effect of long thinking\.We also plot the corresponding performance curves forSpeechCombinein non\-thinking mode in Figure[4](https://arxiv.org/html/2607.02214#S4.F4)\(dashed curves\)\. As shown, performance without long thinking is consistently lower, confirming that long thinking provides meaningful benefits even on unseen tasks\. Nevertheless, the non\-thinking model still achieves competitive performance, further validating that the strong results primarily stem from the effective skill combination enabled by our weight combination scheme\.

Pre\-training data structure\.Our pre\-training data contains three sections,\[speech\],\[cap\],\[text\]\. The first is essential because it is the only section with speech information\. To study whether the other two sections are necessary, we remove them from pre\-training and evaluate the resulting models on the same three tasks\. As shown in Table[3](https://arxiv.org/html/2607.02214#S4.T3), when the\[cap\]is removed, the speech generation and understanding performance degrades significantly, because\[cap\]introduces the knowledge of speech generation and understanding\. When\[text\]is removed, the speech generation and understanding performance is also significantly impacted, because\[text\]serves as an anchor for fusing speech knowledge\.

Weight directions\.To study the effect ofΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}, we removeΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}, but instead use task\-specific in\-context examples to provide instruction\-following guidance\. The results in Table[3](https://arxiv.org/html/2607.02214#S4.T3)show that in\-context learning can achieve better performance on the text\-oriented task than the no\-thinkingSpeechCombine, but worse for speech understanding and generation, which suggests it cannot completely replaceΔ​𝜽i​n​s​t\\Delta\\bm\{\\theta\}\_\{inst\}\. On the other hand, ifΔ​𝜽s​p​e​e​c​h\\Delta\\bm\{\\theta\}\_\{speech\}is removed, the model becomesASR \+ Text LLM\(Tables[1](https://arxiv.org/html/2607.02214#S4.T1)and[2](https://arxiv.org/html/2607.02214#S4.T2)\), which excels in text\-oriented tasks but fails in speech\-related tasks\.

More ablation studies are shown in Appendix[C](https://arxiv.org/html/2607.02214#A3)\.

## 5Conclusion and Limitations

In this paper, we explore weight combination as an alternative paradigm for training instruction\-following SLMs that avoids multi\-round training on massive data\. The resulting empirical findings exceed our expectations:SpeechCombineachieves state\-of\-the\-art performance across multiple tasks, with less than 1% of training data for existing SLMs\.

Despite these promising results,SpeechCombinein its current form has several limitations\. First, its output formatting is unstable, requiring format forcing for correction\. Second, the prosody tokenization scheme used in this work does not encode timbre, voice quality, or accent information, which limits its applicability to a broader range of speech tasks\. Finally, reliance on an external ASR system introduces additional latency and transcription errors during inference\. Nevertheless, this paper unveils great potential of our proposed paradigm\. Beyond enabling the development of competitive SLMs today, this paradigm provides a scalable path for SLMs to rapidly inherit emerging capabilities from text LLMs as they continue to evolve in the future\.

## Impact Statement

This paper presents work whose goal is to advance the field of machine learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## References

- A\. Adigwe, N\. Tits, K\. E\. Haddad, S\. Ostadabbas, and T\. Dutoit \(2018\)The emotional voices database: towards controlling the emotion dimension in voice generation systems\.arXiv preprint arXiv:1806\.09514\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§3\.4](https://arxiv.org/html/2607.02214#S3.SS4.p1.3)\.
- Anonymous \(2025\)KnItLM: weaving knowledge into instruction\-tuned llms via continual pre\-training and merging\.OpenReview preprint\.Note:Under review at ICLR 2026[https://openreview\.net/pdf?id=2uctT30vTS](https://openreview.net/pdf?id=2uctT30vTS)Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- H\. Cao, D\. G\. Cooper, M\. K\. Keutmann, R\. C\. Gur, A\. Nenkova, and R\. Verma \(2014\)Crema\-d: crowd\-sourced emotional multimodal actors dataset\.IEEE transactions on affective computing5\(4\),pp\. 377–390\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- D\. Chen, X\. Zhang, Y\. Wang, K\. Dai, L\. Ma, and Z\. Wu \(2026\)FlexiVoice: enabling flexible style control in zero\-shot tts with natural language instructions\.arXiv preprint arXiv:2601\.04656\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p2.1)\.
- Q\. Chen, L\. Cheng, C\. Deng, X\. Li, J\. Liu, C\. Tan, W\. Wang, J\. Xu, J\. Ye, Q\. Zhang,et al\.\(2025\)Fun\-audio\-chat technical report\.arXiv preprint arXiv:2512\.20156\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§2](https://arxiv.org/html/2607.02214#S2.p4.1),[§3\.1](https://arxiv.org/html/2607.02214#S3.SS1.p4.2),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1),[Table 1](https://arxiv.org/html/2607.02214#S4.T1.9.7.1.m1.1.1.1)\.
- Y\. Chen, X\. Yue, C\. Zhang, X\. Gao, R\. T\. Tan, and H\. Li \(2024\)Voicebench: benchmarking llm\-based voice assistants\.arXiv preprint arXiv:2410\.17196\.Cited by:[§4\.2](https://arxiv.org/html/2607.02214#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.02214#S4.T1.8.6.1.m1.1.1.1)\.
- Z\. Dai, Y\. Chen, J\. Xu, L\. Xie, Y\. Wang, Z\. Yang, B\. Bai, Y\. Gao, W\. Zhou, W\. Zhao,et al\.\(2025\)Deep dubbing: end\-to\-end auto\-audiobook system with text\-to\-timbre and context\-aware instruct\-tts\.arXiv preprint arXiv:2509\.15845\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p2.1)\.
- M\. de Seyssel, A\. D’Avirro, A\. Williams, and E\. Dupoux \(2024\)Emphassess: a prosodic benchmark on assessing emphasis transfer in speech\-to\-speech models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 495–507\.Cited by:[§4\.3](https://arxiv.org/html/2607.02214#S4.SS3.p1.1)\.
- A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour \(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- D\. Ding, Z\. Ju, Y\. Leng, S\. Liu, T\. Liu, Z\. Shang, K\. Shen, W\. Song, X\. Tan, H\. Tang,et al\.\(2025\)Kimi\-audio technical report\.arXiv preprint arXiv:2504\.18425\.Cited by:[§1](https://arxiv.org/html/2607.02214#S1.p6.1),[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1),[§4\.2](https://arxiv.org/html/2607.02214#S4.SS2.p3.1)\.
- Q\. Fang, S\. Guo, Y\. Zhou, Z\. Ma, S\. Zhang, and Y\. Feng \(2025\)Llama\-omni: seamless speech interaction with large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 57607–57624\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p5.1)\.
- Y\. Fathullah, C\. Wu, E\. Lakomkin, K\. Li, J\. Jia, Y\. Shangguan, J\. Mahadeokar, O\. Kalinli, C\. Fuegen, and M\. Seltzer \(2024\)Audiochatllama: towards general\-purpose speech abilities for llms\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5522–5532\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- T\. Feng, J\. Lee, A\. Xu, Y\. Lee, T\. Lertpetchpun, X\. Shi, H\. Wang, T\. Thebaud, L\. Moro\-Velazquez, D\. Byrd,et al\.\(2025\)Vox\-profile: a speech foundation model benchmark for characterizing diverse speaker and speech traits\.arXiv preprint arXiv:2505\.14648\.Cited by:[§A\.2](https://arxiv.org/html/2607.02214#A1.SS2.p7.1)\.
- X\. Geng, Q\. Shao, H\. Xue, S\. Wang, H\. Xie, Z\. Guo, Y\. Zhao, G\. Li, W\. Tian, C\. Wang,et al\.\(2025\)Osum\-echat: enhancing end\-to\-end empathetic spoken chatbot via understanding\-driven spoken dialogue\.arXiv preprint arXiv:2508\.09600\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1),[Table 1](https://arxiv.org/html/2607.02214#S4.T1.9.7.1.m1.1.1.1)\.
- S\. Ghosh, A\. Goel, J\. Kim, S\. Kumar, Z\. Kong, S\. Lee, C\. Yang, R\. Duraiswami, D\. Manocha, R\. Valle,et al\.\(2026\)Audio flamingo 3: advancing audio intelligence with fully open large audio language models\.Advances in Neural Information Processing Systems38,pp\. 41819–41886\.Cited by:[§1](https://arxiv.org/html/2607.02214#S1.p6.1),[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1)\.
- X\. He, S\. N\. Ray, H\. Mallidi, J\. Huang, A\. Bellur, C\. Chandak, M\. Maruf, and V\. Ravichandran \(2025\)Continuous\-token diffusion for speaker\-referenced tts in multimodal llms\.arXiv preprint arXiv:2510\.12995\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p3.1)\.
- hexgrad \(2025\)Kokoro\-82m\.Note:[https://huggingface\.co/hexgrad/Kokoro\-82M](https://huggingface.co/hexgrad/Kokoro-82M)Hugging Face model cardCited by:[Table 2](https://arxiv.org/html/2607.02214#S4.T2.13.13.1.m1.1.1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- S\. Huang, P\. Li, Y\. Hsu, K\. Chen, Y\. T\. Lin, S\. Hsiao, R\. Tsai, and H\. Lee \(2024\)Chat vector: a simple approach to equip llms with instruction following and model alignment in new languages\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10943–10959\.Cited by:[§1](https://arxiv.org/html/2607.02214#S1.p10.1),[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- J\. James, L\. Tian, and C\. I\. Watson \(2018\)An open source emotional speech corpus for human robot interaction applications\.\.InInterspeech,pp\. 2768–2772\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- Z\. Jin, J\. Jia, Q\. Wang, K\. Li, S\. Zhou, S\. Zhou, X\. Qin, and Z\. Wu \(2024\)Speechcraft: a fine\-grained expressive speech dataset with natural language description\.InProceedings of the 32nd ACM International Conference on Multimedia,pp\. 1255–1264\.Cited by:[§A\.2](https://arxiv.org/html/2607.02214#A1.SS2.p5.2)\.
- J\. Kahn, M\. Riviere, W\. Zheng, E\. Kharitonov, Q\. Xu, P\. Mazaré, J\. Karadayi, V\. Liptchinsky, R\. Collobert, C\. Fuegen,et al\.\(2020\)Libri\-light: a benchmark for asr with limited or no supervision\.InICASSP 2020\-2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 7669–7673\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- W\. Kang, J\. Jia, C\. Wu, W\. Zhou, E\. Lakomkin, Y\. Gaur, L\. Sari, S\. Kim, K\. Li, J\. Mahadeokar,et al\.\(2024\)Frozen large language models can perceive paralinguistic aspects of speech\.arXiv preprint arXiv:2410\.01162\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- Y\. Li, R\. Xie, X\. Sun, Y\. Cheng, and Z\. Kang \(2025\)Continuous speech tokenizer in text to speech\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 3341–3347\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p3.1)\.
- R\. Liang, C\. Hsu, C\. Yu, S\. Agrawal, S\. Huang, S\. Chen, K\. Huang, and S\. Sun \(2025\)Adaptive helpfulness\-harmlessness alignment with preference vectors\.arXiv preprint arXiv:2504\.20106\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- H\. Liu, Z\. Zhu, N\. Iwamoto, Y\. Peng, Z\. Li, Y\. Zhou, E\. Bozkurt, and B\. Zheng \(2022\)Beat: a large\-scale semantic and emotional multi\-modal dataset for conversational gestures synthesis\.InEuropean conference on computer vision,pp\. 612–630\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- H\. Lou, H\. Paik, W\. Hu, and L\. Yao \(2025\)ParaStyleTTS: toward efficient and robust paralinguistic style control for expressive text\-to\-speech generation\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 1979–1988\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p2.1)\.
- K\. Lu, Z\. Chen, S\. Fu, C\. H\. Yang, S\. Huang, C\. Yang, C\. Yu, C\. Chen, W\. Chen, C\. Huang,et al\.\(2025\)DeSTA2\. 5\-audio: toward general\-purpose large audio language model with self\-generated cross\-modal alignment\.arXiv preprint arXiv:2507\.02768\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- T\. A\. Nguyen, W\. Hsu, A\. d’Avirro, B\. Shi, I\. Gat, M\. Fazel\-Zarani, T\. Remez, J\. Copet, G\. Synnaeve, M\. Hassid,et al\.\(2023\)Expresso: a benchmark and analysis of discrete expressive speech resynthesis\.arXiv preprint arXiv:2308\.05725\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- OpenAI \(2024\)Hello gpt\-4o\.Note:[https://openai\.com/index/hello\-gpt\-4o/](https://openai.com/index/hello-gpt-4o/)Accessed: 2026\-01\-28Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1)\.
- T\. Pai, J\. Wang, L\. Lu, S\. Sun, H\. Lee, and K\. Chang \(2025\)Billy: steering large language models via merging persona vectors for creative generation\.arXiv preprint arXiv:2510\.10157\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- P\. Pandey, R\. V\. Swaminathan, K\. V\. Girish, A\. Sen, J\. Xie, G\. Strimel, and A\. Schwarz \(2025\)Sift\-50m: a large\-scale multilingual dataset for speech instruction fine\-tuning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13921–13942\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p5.1)\.
- M\. K\. Pichora\-Fuller and K\. Dupuis \(2020\)Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- K\. Qian, X\. Fan, J\. Ni, S\. Shechtman, M\. Hasegawa\-Johnson, C\. Gan, and Y\. Zhang \(2025\)ProsodyLM: uncovering the emerging prosody processing capabilities in speech language models\.arXiv preprint arXiv:2507\.20091\.Cited by:[§A\.3](https://arxiv.org/html/2607.02214#A1.SS3.p5.1),[§2](https://arxiv.org/html/2607.02214#S2.p3.1),[§3\.5](https://arxiv.org/html/2607.02214#S3.SS5.p2.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2023\)Robust speech recognition via large\-scale weak supervision\.InInternational conference on machine learning,pp\. 28492–28518\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- Y\. Ren, J\. Yi, J\. Tao, H\. Sun, Z\. Wen, H\. Gu, L\. Xu, and Y\. Bai \(2026\)OV\-instructtts: towards open\-vocabulary instruct text\-to\-speech\.arXiv preprint arXiv:2601\.01459\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p2.1)\.
- Y\. Shao, W\. Liu, J\. Li, T\. Wang, K\. Wei, M\. Yu, and D\. Yu \(2025\)AzeroS: extending llm to speech with self\-generated instruction\-free tuning\.arXiv preprint arXiv:2601\.06086\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- G\. Sokar, G\. K\. Dziugaite, A\. Arnab, A\. Iscen, P\. S\. Castro, and C\. Schmid \(2025\)Continual learning in vision\-language models via aligned model merging\.arXiv preprint arXiv:2506\.03189\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- D\. Wang, J\. Xu, R\. Chu, Z\. Guo, X\. Wang, J\. Wu, D\. Yang, S\. Ji, and J\. Lin \(2025a\)Inserter: speech instruction following with unsupervised interleaved pre\-training\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 18024–18046\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- K\. Wang, Q\. Wu, L\. Song, Z\. Yang, W\. Wu, C\. Qian, R\. He, Y\. Qiao, and C\. C\. Loy \(2020\)Mead: a large\-scale audio\-visual dataset for emotional talking\-face generation\.InEuropean conference on computer vision,pp\. 700–717\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.
- X\. Wang, Y\. Li, C\. Fu, Y\. Shen, L\. Xie, K\. Li, X\. Sun, and L\. Ma \(2024\)Freeze\-omni: a smart and low latency speech\-to\-speech dialogue model with frozen llm\.arXiv preprint arXiv:2411\.00774\.Cited by:[§1](https://arxiv.org/html/2607.02214#S1.p8.1),[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- Y\. Wang, D\. Chen, X\. Zhang, J\. Zhang, J\. Li, and Z\. Wu \(2025b\)Tadicodec: text\-aware diffusion speech tokenizer for speech language modeling\.arXiv preprint arXiv:2508\.16790\.Cited by:[§1](https://arxiv.org/html/2607.02214#S1.p7.2),[§2](https://arxiv.org/html/2607.02214#S2.p3.1)\.
- H\. Wei, X\. Cao, T\. Dan, and Y\. Chen \(2023\)Rmvpe: a robust model for vocal pitch estimation in polyphonic music\.arXiv preprint arXiv:2306\.15412\.Cited by:[§A\.2](https://arxiv.org/html/2607.02214#A1.SS2.p2.1)\.
- M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith,et al\.\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational conference on machine learning,pp\. 23965–23998\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- B\. Wu, C\. Yan, C\. Hu, C\. Yi, C\. Feng, F\. Tian, F\. Shen, G\. Yu, H\. Zhang, J\. Li,et al\.\(2025\)Step\-audio 2 technical report\.arXiv preprint arXiv:2507\.16632\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1)\.
- Z\. Xie and C\. Wu \(2024\)Mini\-omni: language models can hear, talk while thinking in streaming\.arXiv preprint arXiv:2408\.16725\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p5.1)\.
- J\. Xu, Z\. Guo, J\. He, H\. Hu, T\. He, S\. Bai, K\. Chen, J\. Wang, Y\. Fan, K\. Dang, B\. Zhang, X\. Wang, Y\. Chu, and J\. Lin \(2025a\)Qwen2\.5\-omni technical report\.arXiv preprint arXiv:2503\.20215\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1)\.
- J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.\(2025b\)Qwen3\-omni technical report\.arXiv preprint arXiv:2509\.17765\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1)\.
- R\. Yan, X\. Li, W\. Chen, Z\. Niu, C\. Yang, Z\. Ma, K\. Yu, and X\. Chen \(2025\)Uro\-bench: a comprehensive benchmark for end\-to\-end spoken dialogue models\.arXiv preprint arXiv:2502\.17810\.Cited by:[§4\.2](https://arxiv.org/html/2607.02214#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2607.02214#S4.T1.8.6.1.m1.1.1.1),[Table 2](https://arxiv.org/html/2607.02214#S4.T2.12.12.1.m1.1.1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.6](https://arxiv.org/html/2607.02214#S3.SS6.p2.1)\.
- S\. Yang, B\. Kim, K\. Huang, Q\. Tang, H\. Phan, B\. Lu, H\. Sundar, S\. Ghosh, H\. Lee, C\. Kao,et al\.\(2025b\)Generative audio language modeling with continuous\-valued tokens and masked next\-token prediction\.arXiv preprint arXiv:2507\.09834\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p3.1)\.
- I\. Yosha, D\. Shteyman, and Y\. Adi \(2025\)WHISTRESS: enriching transcriptions with sentence stress detection\.arXiv preprint arXiv:2505\.19103\.Cited by:[§A\.2](https://arxiv.org/html/2607.02214#A1.SS2.p6.1)\.
- Z\. Yu and S\. Ananiadou \(2025\)Locate\-then\-merge: neuron\-level parameter fusion for mitigating catastrophic forgetting in multimodal llms\.arXiv preprint arXiv:2505\.16703\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p4.1)\.
- A\. Zeng, Z\. Du, M\. Liu, K\. Wang, S\. Jiang, L\. Zhao, Y\. Dong, and J\. Tang \(2024\)Glm\-4\-voice: towards intelligent and human\-like end\-to\-end spoken chatbot\.arXiv preprint arXiv:2412\.02612\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p6.1),[footnote 1](https://arxiv.org/html/2607.02214#footnote1)\.
- D\. Zhang, S\. Li, X\. Zhang, J\. Zhan, P\. Wang, Y\. Zhou, and X\. Qiu \(2023\)Speechgpt: empowering large language models with intrinsic cross\-modal conversational abilities\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 15757–15773\.Cited by:[§1](https://arxiv.org/html/2607.02214#S1.p6.1)\.
- Z\. Zhang, H\. Li, J\. Hu, W\. Li, and L\. Xie \(2025\)HiStyle: hierarchical style embedding predictor for text\-prompt\-guided controllable speech synthesis\.InNational Conference on Man\-Machine Speech Communication,pp\. 522–535\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p2.1)\.
- \[59\]X\. Zhao, H\. Xiang, S\. Ye, S\. Li, Z\. Tian, G\. Chen, K\. Ding, and G\. WanLongcat\-audio\-codec: an audio tokenizer and detokenizer solution designed for speech large language models, 2025\.URL https://arxiv\. org/abs/2510\.15227\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p3.1)\.
- X\. Zhao, H\. Xiang, S\. Ye, S\. Li, Z\. Tian, G\. Chen, K\. Ding, and G\. Wan \(2025\)LongCat\-audio\-codec: an audio tokenizer and detokenizer solution designed for speech large language models\.arXiv preprint arXiv:2510\.15227\.Cited by:[§2](https://arxiv.org/html/2607.02214#S2.p3.1)\.
- K\. Zhou, B\. Sisman, R\. Liu, and H\. Li \(2022\)Emotional voice conversion: theory, databases and esd\.Speech Communication137,pp\. 1–18\.Cited by:[§4\.1](https://arxiv.org/html/2607.02214#S4.SS1.p1.4)\.

## Appendix AAdditional Algorithm Details

### A\.1Speech Pre\-training

Special Tokens\.Certain special tokens play a crucial role during instruction\-following, but they do not appear in the data for the speech continuous pre\-training\. As a result,SpeechCombinewill forget the ability to output these special tokens during inference, leading to formatting failures\. These tokens include<\|im\_end\|\>which marks the end of the answer, and</think\>, which marks the outset of the long thinking section333Special tokens that mark the outset of answering sections do not produce as much negative impact, because they are always hard\-coded in the answer template, rather than generated by the model\.\. To prevent forgetting of these tokens, we randomly insert them at the end of each\[text\]section in our training data with probabilitypp\.ppis set to 0\.2 for<\|im\_end\|\>, and 0\.02 for</think\>\.

Data Interleaving\.Our pre\-training data are mostly audiobooks, which consist of very long segments of speech utterances by a single speaker, with consistent prosody styles\. However, during inference, the model is applied to a user\-agent interaction setting, which involves alternating speakers and speaking styles\. To make the model aware of the speaker’s turn\-taking, we chop the audiobooks into small segments containing 5\-7 sentences, and piece together small segments from different audiobooks\. An alternative approach is to use conversational speech corpora instead of audiobooks, which we will explore as a future direction\.

### A\.2Speech Caption

We describe how we derive each attribute below\.

Average Pitch Level\.We use RMVPEWeiet al\.\([2023](https://arxiv.org/html/2607.02214#bib.bib36)\)to extract the fundamental frequency \(F0F\_\{0\}\), and quantize it into three pitch classes:Low,Medium, andHigh\. Since pitch ranges vary substantially across speakers and genders, we define relative thresholds based on median pitch statistics at both the speaker and gender levels\.

Specifically, we compute the median pitchmsm\_\{s\}for each speaker and the median pitchmgm\_\{g\}for each gender group, and apply the following criteria:

- •Low:F0<0\.75​msF\_\{0\}<0\.75m\_\{s\}\(speaker\-level\)orF0<0\.85​mgF\_\{0\}<0\.85m\_\{g\}\(gender\-level\),
- •High:F0\>1\.30​msF\_\{0\}\>1\.30m\_\{s\}\(speaker\-level\)orF0\>1\.20​mgF\_\{0\}\>1\.20m\_\{g\}\(gender\-level\),
- •Medium: otherwise\.

The final pitch class is determined by an OR\-based fusion of the speaker\-level and gender\-level decisions, i\.e\., a frame is labeled asLoworHighif either criterion is satisfied; otherwise, it is labeled asMedium\.

Average Speaking Rate\.FollowingJinet al\.\([2024](https://arxiv.org/html/2607.02214#bib.bib39)\), we use the open\-sourced Python packageg2p\_ento convert transcripts into phonemes and compute the average time per phonemeS​rSr\. We then compute the median valuemm\. For each utterance, we apply the following criteria:

- •Low:S​r\>1\.3​mSr\>1\.3m,
- •High:S​r<0\.85​mSr<0\.85m,
- •Medium: otherwise\.

Emphasized Words\.We utilize WhistressYoshaet al\.\([2025](https://arxiv.org/html/2607.02214#bib.bib52)\)as our emphasized words extractor\.

Emotional arousal\.We use whisper\-large\-v3\-msp\-podcast\-emotion\-dim model introduce inFenget al\.\([2025](https://arxiv.org/html/2607.02214#bib.bib53)\)as our arousal extractor\. Due to the model only support audio input with length of 3 to 15 seconds, we adopt sliding window technique to compute the arousal scoreAAof any unqualified audio\. For each utterance, we apply the following criteria:

- •Low:A<0\.282A<0\.282,
- •High:A\>0\.54A\>0\.54,
- •Medium: otherwise\.

Sampling Strategy\.During the caption generation phase, we uniformly sample one attribute from the full set of available attributes\. If the sampled attribute is*text*, it is dropped with a probability of 90%\. For emotion\-related datasets, the*emotion*attribute is always guaranteed to be sampled\.

### A\.3Prosody Tokenization

For better explanation, consider an example where the content of a speech utterance is“How are you?”The corresponding\[speech\]section takes the following format\.

<speech\><spk f0 med\> <SIL\><Dur\> how<Dur\><f0 range\><f0 med\><f0 slope\><energy\> <SIL\><Dur\> are<Dur\><f0 range\><f0 med\><f0 slope\><energy\> <SIL\><Dur\> you<Dur\><f0 range\><f0 med\><f0 slope\><energy\> <SIL\><Dur\> </speech\>

where<SIL\>represents the silence in between words\.<Dur\>represents the duration of the word \(in frames\) of the word or silence in between words\.<f0 range\>represents the the difference between the 95% and 5% percentiles of the frame\-level Log\-F0 contour of the word\.<f0 med\>represents the median of the frame\-level Log\-F0 contour of the word\.<f0 slope\>represents the slope of the straight line that best fits the Log\-F0 contour of the word\.<energy\>represents the log\-norm of the mel\-spectrogram of the word\.<spk f0 med\>represents the median of the frame\-level Log\-F0 contour the speaker\.

As can be seen, the speech token sequence consists of two alternating patterns: ❶ A<SIL\>followed by a<Dur\>which specifies the duration of the silence in between two words, or at the start/end of the sentence; and ❷ a word followed by five values depicting the prosodic properties of the word\.

All the prosodic properties are extracted by the decoder module of a pre\-trained ProsodyLM\(Qianet al\.,[2025](https://arxiv.org/html/2607.02214#bib.bib2)\), which is modified from a StyleTTS2 TTS synthesizer\. It comes with an F0 contour extractor, an energy extractor, and a force\-aligner \(which can be used to extract duration information\)\. All prosodic properties are quantized into 512 levels, forming new prosdy token vocabulary for size 512\. A complete explanation of the tokenization scheme can be found inQianet al\.\([2025](https://arxiv.org/html/2607.02214#bib.bib2)\)\.

### A\.4Inference

System Prompt\.Since our method is designed for building an audio assistant, a system prompt will help the text\-based LLM better understand its role and task\. We therefore adopt the following system prompt:

You are a helpful assistant\. Always follow the user’s instructions precisely and respond with what is required\. Please read the instructions carefully before answering\. Your responses must strictly align with the user’s intent and constraints\.

Format Forcing\.During inference, we apply several*format\-forcing logits processors*to ensure stable and well\-structured generation:

- •Speaker F0\-Median Processor\.To stabilize speech token generation, we enforce the model to emit the<spk f0 med\>token immediately after<speech\>\. In addition, we constrain the first generated token to be non\-speech\.
- •Thinking Processor\.To ensure that the reasoning phase consists purely of text, we prohibit the model from generating any speech tokens while producing thinking content\.
- •Deep Format\-Forcing Processor\.After the model completes text generation, we enforce the output of speech tokens in a predefined format, which further stabilizes speech generation and ensures structural consistency\.
- •Temperature Processor\.To encourage expressive audio generation, this processor applies different temperature values to the text and speech segments during decoding\.

### A\.5Long Thinking

We empirically observed that even if a<think\>token is provided as the context,SpeechCombinemay not start the reasoning process, because it is unclear whether to perform reasoning, which is inherited from the text LLM’s weight, or to perform text continuation behavior, which is strengthened during speech continuous pre\-training\. To disambiguate that, we will append“Okay”right after<think\>, which is a common starting word of the long thinking process for QWEN models, to enhance the signal for long thinking\.

During the thinking process, we ban the generation from the prosody token vocabulary, as we want the thinking process to be silent\. Generation to verbalized thinking is easily achievable by removing the ban\. Also, to prevent the model from aborting prematurely, we also ban the</think\>token until the thinking reaches a pre\-determined minimum length, which is set to 160 tokens in our experiments\.

## Appendix BAdditional Experiment Details

### B\.1Text\-Oriented Tasks

Datasets\.We provide a brief introduction of the benchmark datasets on text\-oriented tasks\.

- •OpenbookQA \(Voicebench\): A speech\-based multiple\-choice science QA task evaluating knowledge\-intensive reasoning under speech input, with performance measured by answer accuracy\.
- •MMSU \(VoiceBench\): A speech\-based multiple\-choice understanding dataset covering diverse domains, evaluating a model’s multimodal speech comprehension and reasoning ability, measured by answer accuracy\.
- •Gsm8kEval \(URO Bench\): A speech\-based math dataset evaluating multi\-step numerical problem solving from spoken queries, measured using LLM\-as\-a\-judge\.
- •MLCpro\-en \(URO Bench\): A speech logic and reasoning dataset assessing complex reasoning and instruction understanding, measured using LLM\-as\-a\-judge\.
- •TruthfulEval \(URO Bench\): A speech dataset measuring a model’s ability to generate truthful and non\-misleading answers under spoken prompts, measured using LLM\-as\-a\-judge\.
- •MLC \(URO Bench\): A speech\-based benchmark evaluating general reasoning and comprehension, measured using LLM\-as\-a\-judge\.

Task Prompts\.We find that many of the above benchmarks have very rigid evaluation schemes\. If the output format deviates from their pre\-defined templates, the answers will be considered incorrect, even though the output answers are still very reasonable\. For example, on the math benchmarks,3\\sqrt\{3\}is considered an incorrect answer whereas1\.7321\.732is considered correct\. While baseline models, which has been extensively trained to get accustomed to such hidden requirements,SpeechCombineis not familiar with any of such rules\. Therefore, we need to introduce the following task prompt to regulate the output format ofSpeechCombine\.

- •OpenbookQA \(Voicebench\), andMMSU \(VoiceBench\):Please answer the following question\. Please indicate which option \(a, b, c, or d\) is correct\. Please put your answer in the format “The correct answer is: \_\_\_\.”
- •MLCpro\-en \(URO Bench\),TruthfulEval \(URO Bench\), andMLC \(URO Bench\):Please answer the question very carefully and produce complete answers\. If the answer contains any square root, convert it to decimal format\.

### B\.2Speech Understanding Tasks

Datasets\.We provide a brief introduction of the benchmark datasets on speech understanding tasks\.

- •UnderEmotion\-en \(URO Bench\): A speech dataset that assess model’s capability on emotion understanding task, measured using LLM\-as\-a\-judge\.
- •EmphAssessUnderstanding: Since the original EmphAssess contains four different speakers saying the sentences with same emphasized words\. To keep our inference cost tractable, we only keep speaker ex04’s audio for evaluation\. This dataset evaluates model’s capability on emphasized words detection, measured by precision, Recall, and F1 Score\.

Task Prompts\.We adopt the following task prompts for the speech understanding tasks\.

- •UnderEmotion\-en \(URO Bench\):Please answer the question in the second person point of view\. Please use an empathetic tone to answer the question\.
- •EmphAssess Understanding:Answer the question from the user\. Put your answer in the format Emphasized word\(s\):

ForEmphAssess Understanding, the same prompt is applied to*all models*\. ForUnderEmotion\-en, the task prompt provides more information than the format to elicit the prosody reasoning mode, as discussed in Section[4\.3](https://arxiv.org/html/2607.02214#S4.SS3)\.

### B\.3Speech Generation Tasks

Datasets\.We provide a brief introduction of the benchmark datasets on speech generation tasks\.

- •GenEmotion \(URO Bench\): A speech\-based emotion generation benchmark evaluating a model’s ability to generate speech with target emotional characteristics\. The metric is defined as the target emotion probability predicted by anEmotion2Vecclassifier, weighted by\(1−WER\)\(1\-\\mathrm\{WER\}\)to account for content intelligibility\.
- •EmphAssess Generation: Since the originalEmphAssesscontains four different speakers saying the sentences with same emphasized words\. We only keep speaker ex04’s label to construct theEmphAssess Generationdataset\. We then useCosyVoiceto synthesis text questionPlease read the sentence: xxx with emphasis on the word: xxxto audio\. This datasets evluated model’s emphasized word generation\. We use EmphAssess’s evaluation pipeline to compute the overall precision, recall, and F1 Score\.

Task Prompts\.For speech generation tasks, a strict word error rate is computed as part of the evaluation score\. Any deviation from the requested content will be considered incorrect\. For example, when the task requests uttering the sentence “How are you?”\. If the model answers“Sure, here is the utterance…”, which is still a reasonable answer, it will still be considered incorrect\. Therefore, we adopt the following task prompts for the speech generation tasks to regulate the output format\.

- •GenEmotion \(URO Bench\):In this task, you are asked to read a sentence in a given emotion\. You MUST output only the exact sentence that should be spoken\. The arousal level should be high\.
- •EmphAssess Generation:In this task, you are asked to read a sentence with emphasized words\. You MUST output only the exact sentence that should be spoken\. Please normalize number in your output\.

## Appendix CAdditional Experiment Results

### C\.1Ablation Study onλ\\lambdaon more tasks

To further study the performance ofSpeechCombineunder differentλ\\lambdavalues, we evaluate the performance on more tasks, shown in Tables[4](https://arxiv.org/html/2607.02214#A3.T4)and[5](https://arxiv.org/html/2607.02214#A3.T5)\. These results provide insights into the optimal selection ofλ\\lambda\. Specifically –

- •If the deployment goal focuses more on factual knowledge, reasoning, and other text\-oriented skills, we recommend setting aλ\\lambdavalue between 0\.6 and 0\.85\. Aλ\\lambdabelow 0\.6 would result in the model not recognizing the\[speech\]sections in its context; aλ\\lambdaabove 0\.85 would result in the model starting to forget the text LLM’s original knowledge\.
- •If the deployment goal focuses more on speech understanding and generation, we recommend setting aλ\\lambdavalue between 0\.8 to 1\. Aλ\\lambdabelow 0\.8 would result in compromised speech knowledge\.

Table 4:Results on text\-oriented tasks with differentλ\\lambda\.Table 5:Results on speech understanding and generation tasks with differentλ\\lambda\.
### C\.2Removing Format Forcing

To study the effect of removing format forcing, we performed an experiment on three representative tasks where we removed the format forcing\. Note that the deep thinking section tends to degenerate into meaningless prosody tokens without format forcing\. Hence, we disabled the thinking mode and evaluated the output stability of the rest of the output sequence\. Table[6](https://arxiv.org/html/2607.02214#A3.T6)shows the results\.

As can be observed, removing format forcing produces almost no impact on the performance, except for the speech understanding task\. This result suggests that while output instability is an issue for SpeechCombine, it is not devastating for most output segments\. The major instability problem comes from the thinking section, which we will seek to address as a future direction\.

Table 6:Effect of removing format forcing on representative tasks\.
### C\.3Generalization Across Other Model Families

To studySpeechCombine’s generalization across non\-QWEN model families, we performed experiments on \(1\)Olmo\-3\-7B\-Think, \(2\)Llama\-3\.1\-8B\. The results on representative tasks in the three task types are shown in Table[7](https://arxiv.org/html/2607.02214#A3.T7)\.

As can be observed, the generalization to other model families is successful, bounded by the capability constraint of the corresponding text model \(the text models ofOlmoandLlamaare significantly weaker than QWEN\. Hence the corresponding SpeechCombine model underperforms QWEN as well\)\. The only exception is theGenEmoperformance ofOlmo, for which we observed a convergence issue on the prosody token loss during the continuous pre\-training\. Nevertheless, these results confirm the generality of theSpeechCombineframework\.

Table 7:Generalization ofSpeechCombineto different text model families\.