Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?

# Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?
Source: [https://arxiv.org/html/2605.22170](https://arxiv.org/html/2605.22170)
Luca Modica1,3,4\*Filip Landin2,3,4\*Mehrdad Farahani3,4\*Livia Qian5 Gabriel Skantze5Richard Johansson3,4 1Zenseact2Unbox AI3Chalmers University of Technology 4University of Gothenburg5KTH Royal Institute of Technology mehrdad\.farahani@chalmers\.se

###### Abstract

In recent years, several Speech Language Models \(SLMs\) that represent speech and written text jointly have been presented\. The question then emerges about how model\-internal mechanisms are*similar*and*different*when operating in the two modalities\. We focus on how these systems encode, store, and retrieve factual knowledge, which has previously been investigated for text\-only models\. To investigate mechanisms behind the storage and recall of factual association in SLMs, we leverage Causal Mediation Analysis, a technique previously applied to text\-based models\.

Initial results usingSpiritLM, a multimodal model integrating discrete speech tokens reveal discrepancies between text\-to\-text and speech\-to\-text results, suggesting that the emergent mechanisms for factual recall are only partially carried over from the text to the speech modality\. These results advance our understanding of how internal mechanisms encode factual associations in SLMs while contributing insights for improving speech\-enabled AI systems\.

Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?

Luca Modica1,3,4\*Filip Landin2,3,4\*Mehrdad Farahani3,4\*Livia Qian5Gabriel Skantze5Richard Johansson3,41Zenseact2Unbox AI3Chalmers University of Technology4University of Gothenburg5KTH Royal Institute of Technologymehrdad\.farahani@chalmers\.se

\*\*footnotetext:Equal contribution\.## 1Introduction

Large Language Models \(LLMs\) have demonstrated exceptional capabilities in various NLP tasks, including answering factual questions such as“the capital of Italy is”by relying on information stored in their parametersPetroniet al\.\([2019](https://arxiv.org/html/2605.22170#bib.bib26)\)\. However, these systems still suffer from hallucination and are prone to committing factual errors, which limits their trustworthiness and usabilityKandpalet al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib31)\): this motivates further investigation into the mechanisms behind knowledge recall and factual memory\. Research using intervention\-based methods reveals that factual knowledge can be*localized*within text\-based LLMs, particularly in mid\-layer feed\-forward networks \(MLP\)Gevaet al\.\([2021](https://arxiv.org/html/2605.22170#bib.bib33)\); Menget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\); Gevaet al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib2)\)\. These findings are being used to develop methods that edit model parameters, allowing for precise intervention on factual associations, representing a step forward in more accurate and steerable modelsMenget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1),[2023](https://arxiv.org/html/2605.22170#bib.bib36)\)\.

Speech\-language models trained directly on audio without text supervision – such as those using GSLM\-style trainingLakhotiaet al\.\([2021](https://arxiv.org/html/2605.22170#bib.bib18)\)– have shown promise in speech understanding tasksLinet al\.\([2025](https://arxiv.org/html/2605.22170#bib.bib45)\); Basuet al\.\([2024](https://arxiv.org/html/2605.22170#bib.bib47)\); Penget al\.\([2026](https://arxiv.org/html/2605.22170#bib.bib19)\); Hassidet al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib29)\); Zhanget al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib25)\)\. Since they do not leverage text\-based knowledge, their factual understanding is more limited\. On the other hand, speech models built on top of LLMs, likeSpiritLMNguyenet al\.\([2025](https://arxiv.org/html/2605.22170#bib.bib5)\), might retain or develop a deeper understanding of knowledge encoded in the text\-based model\. What is less understood is whether this behavior, if it exists, originates from the separate training on speech data or whether it comes from mechanisms learned from text\. This opens up interesting research questions:

- •Are the mechanisms behind factual recall modality\-independent?
- •Does factual localization in speech\-based inputs emerge independently without reliance on the backbone architecture?

In this paper, we investigate where and how factual associations are stored and recalled inSpiritLMby using Causal Tracing \(CT\), one of the intervention\-based techniques used to study the causal effect of components within a neural networkMenget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\)\. We focus on two specific input\-wise settings:

1. 1\.T→\\rightarrowT \(text\-to\-text\): where the model receives and produces text\.
2. 2\.S→\\rightarrowT \(speech\-to\-text\): where the model takes audio as input but still generates a text output\.

By extending CT to analyze factual recall in a multimodal setting, we show that speech input leads to weaker but detectable traces of factual localization\.

## 2Methodology

This section first describes the causal mediation analysis \(CMA\) framework and the mathematical underpinnings of causal tracing, followed by theSpiritLMmodel, dataset preparation, and the experiment design\.

### 2\.1Preliminaries: Causal Mediation Analysis

CMA is a framework for investigating questions about the relative contributions to an overall effect of individual components in a complex systemPearl \([2001](https://arxiv.org/html/2605.22170#bib.bib22)\)\. FollowingViget al\.\([2020](https://arxiv.org/html/2605.22170#bib.bib13)\), it has emerged as part of the standard toolbox for the analysis of LLMs; in mechanistic interpretability, it is also known as*activation patching*Heimersheim and Nanda \([2024](https://arxiv.org/html/2605.22170#bib.bib50)\)\.Menget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\)applied CMA to investigate factual recall in LMs\. Their approach consists of three steps:

#### Clean run\.

The LM is provided a clean promptX=xX=x, producing a probabilityℙx​\[o\]\\mathbb\{P\}\_\{x\}\[o\], whereoodenotes the expected decoded token\. The corresponding hidden states from this inference are cached\.

#### Corrupted run\.

The model receives a corrupted input promptX=x∗X=x^\{\*\}, resulting in a new predicted output probabilityℙx∗​\[o\]\\mathbb\{P\}\_\{x^\{\*\}\}\[o\]\.Menget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\)carried out the corruption intervention by obfuscating the subject tokens with noise proportional to the standard deviation over all input embeddings\.

#### Corrupted\-with\-restoration run\.

The same corrupted promptX=x∗X=x^\{\*\}is passed to the model, but with the activation value of selected componentCiC\_\{i\}restored \(patched\) from theclean run\. The result is denotedℙx∗,clean​Ci​\[o\]\\mathbb\{P\}\_\{x^\{\*\},\\text\{clean \}C\_\{i\}\}\[o\], where "clean​Ci\\text\{clean \}C\_\{i\}" refers to the value of the componentCiC\_\{i\}from the clean inference\.

The results of the three runs allow us to quantify the mediated effects of interventions\. The relative contribution of a hidden\-state mediator is measured by theIndirect Effect\(IE\), defined as the difference between the corrupted\-with\-restoration run and the corrupted run:

IE=ℙx∗,clean​Ci​\[o\]−ℙx∗​\[o\]\.\\text\{IE\}=\\mathbb\{P\}\_\{x^\{\*\},\\text\{clean \}C\_\{i\}\}\[o\]\-\\mathbb\{P\}\_\{x^\{\*\}\}\[o\]\.By averaging over multiple prompts, we obtain Average Indirect Effect \(AIE\) at different levels of the model components and then visualize the contribution results\.

### 2\.2The multimodal large language model under study: TheSpiritLMmodel

Our work examinesSpiritLMNguyenet al\.\([2025](https://arxiv.org/html/2605.22170#bib.bib5)\)as a case of a multimodal \(speech\) language model that can generate text and audio language content\. Furthermore,SpiritLMuses discrete speech tokens and is trained on interleaved speech and text token sequences for better generalization and alignment across modalities, making it well\-suited for the proposed study\.

We illustrate the high\-level architecture ofSpiritLMin Figure[1](https://arxiv.org/html/2605.22170#S2.F1)\. The model handles mixed speech and text inputs using specialmodality declaration tokens\("T" for text, "S" for speech\)\. Audio is discretized into tokens with HuBERTHsuet al\.\([2021](https://arxiv.org/html/2605.22170#bib.bib16)\)and text with the Llama2 tokenizer\. The interleaved sequence, with each portion prefixed by its respective modality tokens, is input to Llama2\. The model predicts the next tokens based on the most recent modality token: a "T" token prompts text generation, while "S" prompts discrete speech tokens\. At inference, speech tokens are decoded with HiFi\-GANKonget al\.\([2020](https://arxiv.org/html/2605.22170#bib.bib49)\)\.

![Refer to caption](https://arxiv.org/html/2605.22170v1/x1.png)Figure 1:The SpiritLM architecture\.The employed speech representation allows targeting and analyzing specific speech tokens in a Causal Mediation Analysis experiment in both a uni\-modal and cross\-modal context\.

### 2\.3Dataset and data preparation

For our study, we use theKnowndatasetMenget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\): It contains almost 1000 factual prompts that the GPT2\-XL model knows,111Link to the collection of factual prompts:[https://rome\.baulab\.info/data/dsets/known\_1000\.json](https://rome.baulab.info/data/dsets/known_1000.json)with the annotated subject and object \(expected correct answer\)\.

Starting from the available text data, we introduce the speech modality counterpart for each information in Known \(prompt, subject, and object\)\. The utterances are generated using the TTS model MeloTTSZhaoet al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib7)\), which is based on architectures that leverage adversarial learning to improve expressive power and high\-quality speech synthesisKimet al\.\([2021](https://arxiv.org/html/2605.22170#bib.bib8)\); Konget al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib9)\)\. We assess the reliability of the curated speech modality through two complementary approaches: manual inspection of challenging samples, particularly prompts, and automatic transcription of the generated audio usingWhisper\-small,222Link to the model checkpoint used:[https://huggingface\.co/openai/whisper\-small](https://huggingface.co/openai/whisper-small)a lightweight ASR model\. The prompt transcription results yield a Word Error Rate of19%19\\%: this demonstrates good performance despite the inherent difficulty of transcribing proper nouns, and the reliability of the TTS model\.

To further ensure the dataset quality for the subsequent experiment, we filter the original collection factual statements based on the model performance in the 2 different modalities as input, resulting in 2 datasets:Known\-t2tandKnown\-s2t\.Known\-t2tincludes datapoints where the model readily generates either an exact correct answer or a close variant in aText→Text\\text\{Text\}\\rightarrow\\text\{Text\}scenario\. For instance, "Rome" is correct for the prompt "The capital of Italy is \_\_\_", while answers like "Rome, Italy" or "the city of Rome" are considered partially correct\.Known\-s2tfollows the same selection criteria, but in aSpeech→Text\\text\{Speech\}\\rightarrow\\text\{Text\}setting\.

### 2\.4Experiment design

Factual associations inSpiritLMare investigated through two CMA experiments, in text and speech domain, in order to determine causal effects of network components: single transformer layers, MLP, and attention sub\-layers\.

Experiments are conducted on prompts from the datasets introduced in[2\.3](https://arxiv.org/html/2605.22170#S2.SS3)\. Similarly toMenget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\), the corrupted run is done by adding noise to the representation of the subject tokens\.

#### Experiment 1: Within\-modality factual recall \(Text→\\rightarrowText\)

In the first experiment, a text prompt is fed into the model and the log probability of predicting the corresponding attribute is computed for each of the three CMA iterations – clean, corrupted, and corrupted\-with\-restoration runs – described in Section[2\.1](https://arxiv.org/html/2605.22170#S2.SS1)\. The IE is aggregated by the position of the token in the sentence:first subject token, middle subject tokens, last subject token, first subsequent token, further tokens, and last token, averaged over all prompts \(AIE\), and presented as log AIE for readability and comparison\.

#### Experiment 2: Cross\-modality factual recall \(Speech→\\rightarrowText\)

The second experiment is similar, but uses the synthesized version of the dataset, where prompts are converted to audio\. Each utterance is encoded and discretized byHuBERT\), and the resulting tokens are fed into the language model, where the CMA pipeline is applied as in the previous experiment\. An additional challenge here is that, in the corrupted run, it is no longer obvious how to localize the subject token\(s\) in the input prompt\. Connectionist Temporal Classification \(CTC\)\-based forced alignmentKürzingeret al\.\([2020](https://arxiv.org/html/2605.22170#bib.bib6)\)is therefore used to find the target time range of the subject in the utterance, and thus the related range of the speech tokens \(see Appendix[A](https://arxiv.org/html/2605.22170#A1)for more details\)\. The same technique admits a mapping between speech tokens and the corresponding text ones, which is used to post\-process the CMA results\. The causal traces of the speech \(HuBERT\) token are aggregated, as for text, by the corresponding text tokens, which facilitates direct comparison and interpretation of causal influence across modalities\. The quality of the forced alignment is validated by manually inspecting the speech segments corresponding to text tokens, ensuring that token boundaries are properly aligned without overlaps or significant gaps throughout the prompt utterance\.

singlepatched layercenter of intervalof 5 patched mlp layerscenter of intervalof 5 patched attention layersT→\\rightarrowT\(LLaMA 2\.7B\)\(a\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x2.png)\(b\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x3.png)\(c\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x4.png)T→\\rightarrowT\(SpiritLM\)\(d\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x5.png)\(e\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x6.png)\(f\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x7.png)S→\\rightarrowT\(SpiritLM\)\(g\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x8.png)\(h\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x9.png)\(i\)![Refer to caption](https://arxiv.org/html/2605.22170v1/x10.png)Figure 2:Log\-scaled AIE across different modules and modalities over 754 prompts\. In each subfigure, the x\-axis represents the layers and the y\-axis shows the tokens of interest\.

## 3Results and Discussion

We begin our experiments at two levels to examine whether factual associations – previously shown to localize around subject tokens in text\-only models – can also be recalled and expressed in other modalities\. In our case, we focus onSpiritLMand its speech modality\. To measure this, we use causal mediation analysis to compute the Average Indirect Effect \(AIE\) for each layer and token across all filtered query prompts\. The AIE captures the marginal contribution of an internal component to the final factual prediction under intervention\. Higher AIE values indicate which layers and positions influence factual recall more\.

As a baseline, we perform CT on the backbone model used inSpiritLM, using text\-to\-text prompts\. As expected from prior work byMenget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\), we observe strong causal signals \(AIE\) centered on subject tokens at early layers, especially in mid\-layer MLPs \(see Figure[2](https://arxiv.org/html/2605.22170#S2.F2)\)\. We also detect notable effects at the final token position in upper layers, where strong causality is typically observed\. Extending the same CT analysis toSpiritLMin the T→\\rightarrowT setting reveals nearly identical behavior: consistent causal signals around subject tokens across hidden States, MLP, and attention layers \(Figure[2](https://arxiv.org/html/2605.22170#S2.F2)\)\. This confirms that text\-processing pathways inSpiritLMpreserve their original capabilities after fine\-tuning on speech\.

On the other hand, we observe different results when the input is speech \(S→\\rightarrowT\)\. The AIE drops drastically, showing a much more diffuse and lower\-magnitude signal; however, we can still observe an effect around the subject tokens in the MLP and attention layers \(see Figure[2](https://arxiv.org/html/2605.22170#S2.F2)\)\.

Our experiments suggest that factual associations in spoken language models likeSpiritLMare not strictly modality\-dependent\. Although the model is capable of retrieving knowledge from both modalities – including, to some extent, speech – the factual recall mechanisms are much more readily activated when the model is provided with text input than with speech\. ForSpiritLM, text serves as a more structured and reliable trigger for recalling facts, suggesting that the speech\-based fine\-tuning in this model does not fully utilize the fact\-recalling mechanisms learned by the text\-based backbone model\. However, we do not have sufficient evidence to conclude whether the partial transfer of factual capabilities from text to speech inSpiritLMarises from noise introduced by controlled conditions or limitations of the mapping between speech and text tokens\. Based on recent studiesXianget al\.\([2025](https://arxiv.org/html/2605.22170#bib.bib57)\); Cuervoet al\.\([2026](https://arxiv.org/html/2605.22170#bib.bib58)\), we also hypothesize that a likely cause is the semantic gap between the two modalities, possibly arising from the post\-training speech adaptation ofSpiritLM’s text\-only backbone: while the two modalities may become increasingly aligned in direction \(cosine similarity\) across deeper layers, a divergence in magnitude \(Euclidean distance\) can still persist, ultimately compromising the transfer of factual knowledge to the speech modality\.

## 4Related Work

Interpretability for speech LLMs remains underexplored compared to their text counterparts: in particular, investigating to what extent multimodal text/speech systems share underlying mechanisms remains unexplored\. Recent worksPasadet al\.\([2024](https://arxiv.org/html/2605.22170#bib.bib41),[2021](https://arxiv.org/html/2605.22170#bib.bib42)\); Shenet al\.\([2024](https://arxiv.org/html/2605.22170#bib.bib51)\)have explored speech model interpretability, considering speech features at different granularity levels \(e\.g\., word boundaries, pronunciation\), finding, for example, that frame\-level representations within each word segment are not all equally informative\. These studies lead to open questions more related to how sentence\-level properties \(e\.g\., subject\) are encoded, a gap our work seeks to fill\. More recently,Glazeret al\.\([2026](https://arxiv.org/html/2605.22170#bib.bib61)\)explored mechanistic interpretability for ASR systems, applying logit lens and activation patching to reveal internal model dynamics responsible for repetition hallucinations and semantic biases within acoustic representations: these findings suggest promising directions for similar investigations in speech LLMs and beyond the ASR setting\.

## 5Conclusion

The study investigates the mechanisms of factual recall in Speech LLMs, focusing on whether this process in the speech modality operates independently or relies on the text modality and the capabilities of the original text model\. By using the CMA framework withSpiritLM, we show that the model preserves the same text\-based computation pathways as its corresponding text\-only counterpart, while the speech modality leads to a considerably weaker causal effect at the level of the MLP and attention layers\. Although the latter does not conclusively prove if knowledge from text is transferred to the speech modality, these preliminary insights hint that speech LLMs are not strictly modality\-dependent when recalling facts\.

## Limitations and Future Directions

Dataset selectivity\.We conducted our experiment on a single synthesized speech dataset –KnownMenget al\.\([2022](https://arxiv.org/html/2605.22170#bib.bib1)\), which might not capture all the nuances of how modality interactions affect factual memory tracing\. Using other datasets, fromSpoken SQuADLiet al\.\([2018](https://arxiv.org/html/2605.22170#bib.bib52)\)to a synthesized version ofPopQAMallenet al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib54)\), can provide a valuable contribution in this field\.

Model generalization\.Although the insights obtained usingSpiritLM, their compatibility with other Speech LLMs that employ discrete speech tokens needs to be exploredZhanget al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib25)\); Rubensteinet al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib27)\)\. Furthermore, replicating our experiments withinSpiritLMthrough different text\-only backbonesYanget al\.\([2025](https://arxiv.org/html/2605.22170#bib.bib59)\); Jianget al\.\([2023](https://arxiv.org/html/2605.22170#bib.bib60)\)or a joint speech\-text training strategy would be crucial to assess the generalizability of our findings\.

Discrete speech tokens limitations\.Using discrete tokens represents an interesting strategy to integrate the modality with text\-based tokens seamlessly; however, recent studies have shown performance limitations on semantic understanding tasks, which might affect results of factual recall studies of speech Large Language ModelsWanget al\.\([2025](https://arxiv.org/html/2605.22170#bib.bib12)\)\. Considering speech LLMs that employ different strategies to convey a richer speech representation, such as training on continuous speech representationPenget al\.\([2026](https://arxiv.org/html/2605.22170#bib.bib19)\); Tanget al\.\([2024](https://arxiv.org/html/2605.22170#bib.bib55)\), can represent a promising future direction of this investigation\.

## Ethical Considerations

This study focuses on the interpretability of speech\-language models\. As part of our research, we do not release any new models or datasets; therefore, we do not implicate any potential risks or concerns related to the misuse of our results\.

## Acknowledgments

This research was funded by the Wallenberg AI, Autonomous Systems and Software Program \(WASP\) funded by the Knut and Alice Wallenberg Foundation\. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden \(NAISS\) at Alvis partly funded by the Swedish Research Council through grant agreement no\. 2022\-06725\. We also acknowledge the Computer Science and Engineering department at Chalmers and the University of Gothenburg, which funds part of the conference costs through the*Lars Pareto travel grant*\.

## References

- Understanding information storage and transfer in multi\-modal large language models\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USAVancouver, BC, Canada\.External Links:ISBN 9798331314385,[Link](https://dl.acm.org/doi/10.5555/3737916.3738153)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1)\.
- S\. Cuervo, S\. Seto, M\. de Seyssel, R\. H\. Bai, Z\. Gu, T\. Likhomanenko, N\. Jaitly, and Z\. Aldeneh \(2026\)Closing the gap between text and speech understanding in LLMs\.InProceedings of the The Fourteenth International Conference on Learning Representations,Rio de Janeiro, Brazil\.External Links:[Link](https://arxiv.org/abs/2510.13632)Cited by:[§3](https://arxiv.org/html/2605.22170#S3.p4.1)\.
- M\. Geva, J\. Bastings, K\. Filippova, and A\. Globerson \(2023\)Dissecting recall of factual associations in auto\-regressive language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12216–12235\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.751/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.751)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.446/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p1.1)\.
- N\. Glazer, Y\. Segal\-Feldman, H\. Segev, A\. Shamsian, A\. Buchnick, G\. Hetz, E\. Fetaya, J\. Keshet, and A\. Navon \(2026\)Beyond transcription: mechanistic interpretability in asr\.InThe Fortieth AAAI Conference on Artificial Intelligence \(AAAI\-26\),Singapore\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/41073/45034)Cited by:[§4](https://arxiv.org/html/2605.22170#S4.p1.1)\.
- M\. Hassid, T\. Remez, T\. A\. Nguyen, I\. Gat, A\. Conneau, F\. Kreuk, J\. Copet, A\. Defossez, G\. Synnaeve, E\. Dupoux, R\. Schwartz, and Y\. Adi \(2023\)Textually pretrained speech language models\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USANew Orleans, LA, USA\.External Links:[Link](https://dl.acm.org/doi/10.5555/3666122.3668893)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1)\.
- S\. Heimersheim and N\. Nanda \(2024\)How to use and interpret activation patching\.arXiv preprint arXiv:2404\.15255\.External Links:[Link](https://arxiv.org/abs/2404.15255)Cited by:[§2\.1](https://arxiv.org/html/2605.22170#S2.SS1.p1.1)\.
- W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed \(2021\)HuBERT: self\-supervised speech representation learning by masked prediction of hidden units\.IEEE/ACM Trans\. Audio, Speech and Lang\. Proc\.29,pp\. 3451–3460\.External Links:ISSN 2329\-9290,[Link](https://doi.org/10.1109/TASLP.2021.3122291),[Document](https://dx.doi.org/10.1109/TASLP.2021.3122291),[Link](https://dl.acm.org/doi/10.1109/TASLP.2021.3122291)Cited by:[Appendix A](https://arxiv.org/html/2605.22170#A1.SS0.SSS0.Px2.p1.1),[§2\.2](https://arxiv.org/html/2605.22170#S2.SS2.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7B\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p2.1)\.
- N\. Kandpal, H\. Deng, A\. Roberts, E\. Wallace, and C\. Raffel \(2023\)Large language models struggle to learn long\-tail knowledge\.InProceedings of the 40th International Conference on Machine Learning,ICML’23,Honolulu, Hawaii, USA\.External Links:[Link](https://dl.acm.org/doi/10.5555/3618408.3619049)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p1.1)\.
- J\. Kim, J\. Kong, and J\. Son \(2021\)Conditional variational autoencoder with adversarial learning for end\-to\-end text\-to\-speech\.InProceedings of the Thirty\-Eighth International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2106.06103)Cited by:[§2\.3](https://arxiv.org/html/2605.22170#S2.SS3.p2.1)\.
- J\. Kong, J\. Kim, and J\. Bae \(2020\)HiFi\-GAN: generative adversarial networks for efficient and high fidelity speech synthesis\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20,Red Hook, NY, USAVancouver, BC, Canada\.External Links:ISBN 9781713829546,[Link](https://dl.acm.org/doi/abs/10.5555/3495724.3497152)Cited by:[§2\.2](https://arxiv.org/html/2605.22170#S2.SS2.p2.1)\.
- J\. Kong, J\. Park, B\. Kim, J\. Kim, D\. Kong, and S\. Kim \(2023\)VITS2: improving quality and efficiency of single\-stage text\-to\-speech with adversarial learning and architecture design\.InProceedings of INTERSPEECH,Dublin, Ireland\.External Links:[Link](https://www.isca-archive.org/interspeech_2023/kong23_interspeech.pdf)Cited by:[§2\.3](https://arxiv.org/html/2605.22170#S2.SS3.p2.1)\.
- L\. Kürzinger, D\. Winkelbauer, L\. Li, T\. Watzel, and G\. Rigoll \(2020\)CTC\-segmentation of large corpora for german end\-to\-end speech recognition\.InSpeech and Computer,pp\. 267–278\.External Links:ISBN 9783030602765,ISSN 1611\-3349,[Link](http://dx.doi.org/10.1007/978-3-030-60276-5_27),[Document](https://dx.doi.org/10.1007/978-3-030-60276-5%5F27)Cited by:[§2\.4](https://arxiv.org/html/2605.22170#S2.SS4.SSS0.Px2.p1.1)\.
- K\. Lakhotia, E\. Kharitonov, W\. Hsu, Y\. Adi, A\. Polyak, B\. Bolte, T\. Nguyen, J\. Copet, A\. Baevski, A\. Mohamed, and E\. Dupoux \(2021\)On generative spoken language modeling from raw audio\.Transactions of the Association for Computational Linguistics9,pp\. 1336–1354\.External Links:[Link](https://aclanthology.org/2021.tacl-1.79/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00430)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1)\.
- C\. Li, S\. Wu, C\. Liu, and H\. Lee \(2018\)Spoken SQuAD: a study of mitigating the impact of speech recognition errors on listening comprehension\.InProceedings of INTERSPEECH,Hyderabad, India\.External Links:[Link](https://www.isca-archive.org/interspeech_2018/lee18d_interspeech.pdf)Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p1.1)\.
- Z\. Lin, S\. Basu, M\. Beigi, V\. Manjunatha, R\. A\. Rossi, Z\. Wang, Y\. Zhou, S\. Balasubramanian, A\. Zarei, K\. Rezaei, Y\. Shen, B\. M\. Yao, Z\. Xu, Q\. Liu, Y\. Zhang, Y\. Sun, S\. Liu, L\. Shen, H\. Li, S\. Feizi, and L\. Huang \(2025\)A survey on mechanistic interpretability for multi\-modal foundation models\.External Links:2502\.17516,[Link](https://arxiv.org/abs/2502.17516)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1)\.
- A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. Hajishirzi \(2023\)When not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 9802–9822\.External Links:[Link](https://aclanthology.org/2023.acl-long.546/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 17359–17372\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p1.1),[§1](https://arxiv.org/html/2605.22170#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.22170#S2.SS1.SSS0.Px2.p1.2),[§2\.1](https://arxiv.org/html/2605.22170#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2605.22170#S2.SS3.p1.1),[§2\.4](https://arxiv.org/html/2605.22170#S2.SS4.p2.1),[§3](https://arxiv.org/html/2605.22170#S3.p2.1),[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p1.1)\.
- K\. Meng, A\. Sen Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau \(2023\)Mass editing memory in a transformer\.InProceedings of the Eleventh International Conference on Learning Representations \(ICLR\),Kigali, Rwanda\.External Links:[Link](https://arxiv.org/abs/2210.07229)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p1.1)\.
- T\. A\. Nguyen, B\. Muller, B\. Yu, M\. R\. Costa\-jussa, M\. Elbayad, S\. Popuri, C\. Ropers, P\. Duquenne, R\. Algayres, R\. Mavlyutov, I\. Gat, M\. Williamson, G\. Synnaeve, J\. Pino, B\. Sagot, and E\. Dupoux \(2025\)SpiRit\-LM: interleaved spoken and written language model\.Transactions of the Association for Computational Linguistics13,pp\. 30–52\.External Links:[Link](https://aclanthology.org/2025.tacl-1.2/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00728)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.22170#S2.SS2.p1.1)\.
- A\. Pasad, C\. Chien, S\. Settle, and K\. Livescu \(2024\)What do self\-supervised speech models know about words?\.Transactions of the Association for Computational Linguistics12,pp\. 372–391\.External Links:[Link](https://aclanthology.org/2024.tacl-1.21/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00656)Cited by:[§4](https://arxiv.org/html/2605.22170#S4.p1.1)\.
- A\. Pasad, J\. Chou, and K\. Livescu \(2021\)Layer\-wise analysis of a self\-supervised speech representation model\.In2021 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),Vol\.,pp\. 914–921\.External Links:[Document](https://dx.doi.org/10.1109/ASRU51503.2021.9688093),[Link](https://ieeexplore.ieee.org/document/9688093)Cited by:[§4](https://arxiv.org/html/2605.22170#S4.p1.1)\.
- J\. Pearl \(2001\)Direct and indirect effects\.InProceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence,UAI’01,Seattle, WashingtonSan Francisco, USA,pp\. 411–420\.External Links:ISBN 1558608001,[Link](https://dl.acm.org/doi/10.5555/2074022.2074073)Cited by:[§2\.1](https://arxiv.org/html/2605.22170#S2.SS1.p1.1)\.
- J\. Peng, Y\. Wang, Y\. Xi, X\. Li, X\. Zhang, and K\. Yu \(2026\)A survey on speech large language models\.IEEE Journal of Selected Topics in Signal Processing20\(1\)\.External Links:[Link](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=11278041)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1),[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p3.1)\.
- F\. Petroni, T\. Rocktäschel, S\. Riedel, P\. Lewis, A\. Bakhtin, Y\. Wu, and A\. Miller \(2019\)Language models as knowledge bases?\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2463–2473\.External Links:[Link](https://aclanthology.org/D19-1250/),[Document](https://dx.doi.org/10.18653/v1/D19-1250)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p1.1)\.
- P\. K\. Rubenstein, C\. Asawaroengchai, D\. D\. Nguyen, A\. Bapna, Z\. Borsos, F\. de Chaumont Quitry, P\. Chen, D\. E\. Badawy, W\. Han, E\. Kharitonov, H\. Muckenhirn, D\. Padfield, J\. Qin, D\. Rozenberg, T\. Sainath, J\. Schalkwyk, M\. Sharifi, M\. T\. Ramanovich, M\. Tagliasacchi, A\. Tudor, M\. Velimirović, D\. Vincent, J\. Yu, Y\. Wang, V\. Zayats, N\. Zeghidour, Y\. Zhang, Z\. Zhang, L\. Zilka, and C\. Frank \(2023\)AudioPaLM: a large language model that can speak and listen\.External Links:2306\.12925,[Link](https://arxiv.org/abs/2306.12925)Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p2.1)\.
- G\. Shen, M\. Watkins, A\. Alishahi, A\. Bisazza, and G\. Chrupała \(2024\)Encoding of lexical tone in self\-supervised models of spoken language\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 4250–4261\.External Links:[Link](https://aclanthology.org/2024.naacl-long.239/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.239)Cited by:[§4](https://arxiv.org/html/2605.22170#S4.p1.1)\.
- C\. Tang, W\. Yu, G\. Sun, X\. Chen, T\. Tan, W\. Li, L\. Lu, Z\. Ma, and C\. Zhang \(2024\)SALMONN: towards generic hearing abilities for large language models\.InProceedings of the Twelfth International Conference on Learning Representations,Vienna, Austria\.External Links:[Link](https://arxiv.org/abs/2310.13289)Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p3.1)\.
- J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, S\. Sakenis, J\. Huang, Y\. Singer, and S\. Shieber \(2020\)Causal mediation analysis for interpreting neural NLP: the case of gender bias\.External Links:2004\.12265,[Link](https://arxiv.org/abs/2004.12265)Cited by:[§2\.1](https://arxiv.org/html/2605.22170#S2.SS1.p1.1)\.
- D\. Wang, J\. Li, M\. Cui, D\. Yang, X\. Chen, and H\. M\. Meng \(2025\)Speech discrete tokens or continuous features? A comparative analysis for spoken language understanding in SpeechLLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 24913–24924\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1266/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1266),ISBN 979\-8\-89176\-332\-6Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p3.1)\.
- B\. Xiang, S\. Zhao, T\. Guo, and W\. Zou \(2025\)Understanding the modality gap: an empirical study on the speech\-text alignment mechanism of large speech language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5187–5202\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.262/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.262),ISBN 979\-8\-89176\-332\-6Cited by:[§3](https://arxiv.org/html/2605.22170#S3.p4.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p2.1)\.
- D\. Zhang, S\. Li, X\. Zhang, J\. Zhan, P\. Wang, Y\. Zhou, and X\. Qiu \(2023\)SpeechGPT: empowering large language models with intrinsic cross\-modal conversational abilities\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 15757–15773\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.1055/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.1055)Cited by:[§1](https://arxiv.org/html/2605.22170#S1.p2.1),[Limitations and Future Directions](https://arxiv.org/html/2605.22170#Sx1.p2.1)\.
- W\. Zhao, X\. Yu, and Z\. Qin \(2023\)MeloTTS: high\-quality multi\-lingual multi\-accent text\-to\-speech\.External Links:[Link](https://github.com/myshell-ai/MeloTTS)Cited by:[§2\.3](https://arxiv.org/html/2605.22170#S2.SS3.p2.1)\.

## Appendix AForced Alignment for Cross\-modal Token Mapping: Implementation Details

#### Text preprocessing for CTC\.

For the transcription to be compatible with the forced alignment, a text preprocessing is necessary to ensure all characters are included in the CTC model vocabulary\. For example, digits and special characters such as "%" are converted to their written format \(e\.g, "0" becomes "zero", or "%" becomes "percent"\)\. On the text token\-level, preprocessing can lead to a longer or shorter sequence of tokens, compared to the original text being tokenized\. We later refer to the preprocessed text tokens asspoken text tokens\. The preprocessing step concludes by joining the spoken text tokens into a single string, using the word boundary character defined by the CTC model as a separator\.

#### Frame\-wise label probability estimation from audio waveform\.

We generate emission probabilities per audio frame, using the pre\-trainedHuBERT\-LARGE333Link to the model checkpoint used:[https://huggingface\.co/facebook/hubert\-large\-ls960\-ft](https://huggingface.co/facebook/hubert-large-ls960-ft)model as a speech tokenizer\. This model is fine\-tuned for automatic speech recognition \(ASR\) with CTC loss, representing a suitable candidate for this use caseHsuet al\.\([2021](https://arxiv.org/html/2605.22170#bib.bib16)\)\.

#### Trellis matrix generation with log\-probability of label alignments at each time step\.

Given an audio input sequence𝐗=\(x1,…,xT\)\\mathbf\{X\}=\(x\_\{1\},\\ldots,x\_\{T\}\)and transcript labels\(c1,…,cN\)\(c\_\{1\},\\ldots,c\_\{N\}\)at the character level, we compute through dynamic programming and map all possible joint probabilities in the trellis diagram matrixK∈ℝT×NK\\in\\mathbb\{R\}^\{T\\times N\};K\(t,j\)K\_\{\(t,j\)\}represents the maximum log\-probability of aligning the first labelsjjup to timett\. To compute the probability at time stept\+1t\+1for labelcj\+1c\_\{j\+1\}, we consider two possible transitions: either we stayed on the same labelcj\+1c\_\{j\+1\}or transitioned fromcjc\_\{j\}tocj\+1c\_\{j\+1\}\. Based on these criteria, the trellis is updated as follows:

K\(t\+1,j\+1\)=max⁡\{K\(t,j\)​p​\(t\+1,cj\+1\)K\(t,j\+1\)​p​\(t\+1,repeat\)K\_\{\(t\+1,j\+1\)\}=\\max\\begin\{cases\}K\_\{\(t,j\)\}\\,p\(t\+1,c\_\{j\+1\}\)\\\\ K\_\{\(t,j\+1\)\}\\,p\(t\+1,\\text\{repeat\}\)\\end\{cases\}
wherep​\(t\+1,cj\+1\)p\(t\+1,c\_\{j\+1\}\)is the probability of emitting labelcj\+1c\_\{j\+1\}at timet\+1t\+1, andp​\(t\+1,repeat\)p\(t\+1,\\text\{repeat\}\)is the probability of emitting no label change\.

#### Find the most likely path from the trellis matrix\.

Once the trellis is generated, we will traverse it following the elements with the highest probability\. Starting from the last label index belonging to the last time step, we progress in the matrix backwards, choosing to keep the current label or move to the previous label based on the highest probability for each time step\. The process ends when we reach the beginning of the sequence, obtaining the most likely path that aligns text and audio\.

#### Merge repetitions and segments into words \(spoken text tokens\)\.

The final step involves postprocessing the output from the optimal path\. Because the path may contain consecutive repetitions of the same label, we merge path points corresponding to repeated characters into a single segment to make it close to the original transcript\.444When merging path points into a single segment, we use the average probability of all frames in that segment\.Similarly, we group segments that correspond to the same spoken text token, using the word boundary character as a guide\. The result is a sequence of segments, each representing a spoken text token from the transcript and annotated with the corresponding range of audio frames and average emission probability\.

From this segmentation, we can derive the time range of each text token, which can also be used to obtain the speech token range, using the token rate of the speech tokenizer\. The last result allows us to directly map text tokens and related speech tokens\. Figure[3](https://arxiv.org/html/2605.22170#A1.F3)illustrates an example of the final output from CTC\-based forced alignment, demonstrating this alignment process\. Beginning with the preprocessed transcript "THE\|CAPITAL\|OF\|ROMAN\|REPUBLIC\|IS," we align each text token with its corresponding segment in the audio\. For each text token, we label the aligned speech segment with the average probability over the merged segment, clearly indicating its position within the utterance as a highlighted segment on the spectrogram, with boundaries marking its start and end\. This segmentation process allows us to determine the precise time range for each text token\.

- •Given the frame range\(fs​t​a​r​t,fe​n​d\)\(f\_\{start\},f\_\{end\}\)of a spoken text token segment, we can first compute its time range in seconds\(ss​t​a​r​t,se​n​d\)\(s\_\{start\},s\_\{end\}\)in the utterance with the following formula: ss​t​a​r​t=⌊r​a​t​i​o⋅fs​t​a​r​t⌋s​r,se​n​d=⌊r​a​t​i​o⋅fe​n​d⌋s​r,s\_\{start\}=\\frac\{\\lfloor ratio\\cdot f\_\{start\}\\rfloor\}\{sr\},s\_\{end\}=\\frac\{\\lfloor ratio\\cdot f\_\{end\}\\rfloor\}\{sr\},\(1\)Wheres​rsrrepresents the sample rate of the original sampled waveformZ=\(z1,…,zM\)Z=\(z\_\{1\},\\ldots,z\_\{M\}\), whiler​a​t​i​o=MTratio=\\frac\{M\}\{T\}represents the number of samples contained in a frame\.
- •Then, considering the token rate of the speech tokenizert​rtrand the previously computed time range\(ss​t​a​r​t,se​n​d\)\(s\_\{start\},s\_\{end\}\), the corresponding speech token range\(s​t​ks​t​a​r​t,s​t​ke​n​d\)\(stk\_\{start\},stk\_\{end\}\)is given by: s​t​ks​t​a​r​t=⌊ss​t​a​r​t⋅t​r⌋,s​t​ke​n​d=⌈se​n​d⋅t​r⌉\.stk\_\{start\}=\\lfloor s\_\{start\}\\cdot tr\\rfloor,stk\_\{end\}=\\lceil s\_\{end\}\\cdot tr\\rceil\.\(2\)

This direct mapping provides a clear correspondence between each text token and its associated speech tokens, linking elements of the transcript to their acoustic realizations in the audio\.

![Refer to caption](https://arxiv.org/html/2605.22170v1/figures/result-alignment.png)Figure 3:Results of the forced alignment for a speech utterance \(transcript: "The capital of Roman Republic is"\)\. The plot on top shows the trellis matrix, with the highlighted optimal path and score for each labeled letter; on the bottom, instead, we show the mel spectrogram of the spoken utterance, with the corresponding boundaries between each \(spoken\) text token\.