Interleaved Speech Language Models Latently Work In Text
Summary
This paper reveals that interleaved speech-text language models implicitly transcribe speech into text in intermediate layers, then predict in text space before converting back to speech, shedding light on internal modality interaction.
View Cached Full Text
Cached at: 06/30/26, 07:35 AM
Paper page - Interleaved Speech Language Models Latently Work In Text
Source: https://huggingface.co/papers/2606.22473
Abstract
Interleaved speech-text language models exhibit an implicit transcription phase where text tokens become decodable in intermediate layers, followed by text-based prediction before speech domain transformation.
Speech language models(SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach isspeech-text interleavingin which models are trained over sequences containing both speech andtext tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of thelogit lensto provide such insight. We reveal that these models go through an implicit transcription phase in which thetext tokenof the spoken word becomes decodable inintermediate layers, despite not being trained forspeech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates withspoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.22473
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.22473 in a model README.md to link it from this page.
Datasets citing this paper1
#### slprl/common-sense-facts-audio Viewer• Updated5 days ago • 281 • 80 • 2
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.22473 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
This paper proposes Joint Speech-Text Interleaved Pretraining (JSTIP), a pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences to improve ASR entity accuracy and reduce the modality gap between speech and text, showing competitive performance on domain adaptation and zero-shot speech question answering.
The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space
The paper proposes the interlingua hypothesis, suggesting that large language models perform translation by encoding source text into a latent task-agnostic feature space and decoding from it, supported by empirical evidence on variance, causal influence, and monolingual fine-tuning.
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
Proposes TextPro-SLM, a speech large language model that minimizes the modality gap by processing spoken input to resemble prosody-aware text input, achieving strong paralinguistic understanding with low training data.
TextLDM: Language Modeling with Continuous Latent Diffusion
This paper introduces TextLDM, a method that adapts visual latent diffusion transformers for language modeling by mapping discrete tokens to continuous latents. It demonstrates that this approach, enhanced by representation alignment, matches GPT-2 performance and unifies visual and text generation architectures.
Latent-IM: Latent Interaction Management for Speech LLMs
Introduces Latent-IM, a framework for recovering interaction management from frozen speech LLMs using activation-based selection and steering for conversational moves. It improves end-to-end move accuracy by 12.5 points over the unsteered backbone.