Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

# Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions
Source: [https://arxiv.org/html/2609.02940](https://arxiv.org/html/2609.02940)
Chao\-Han Huck Yang††thanks:Work done at NVIDIAAffiliation:NVIDIAEmail:[busso@cmu\.edu](mailto:)Shinji Watanabe‡Hung\-yi LeeCarlos Busso‡‡Carnegie Mellon UniversityAffiliation:National Taiwan University

###### Abstract

Recent automatic speech recognition \(ASR\) systems increasingly integrate large language models \(LLMs\) to leverage their semantic knowledge, either externally through logit fusion or internally through warm initialization\. However, how to effectively combine these two strategies remains underexplored\. In this work, we refine warm\-initialized LLM\-based ASR models by leveraging their own pre\-adaptation base LLMs, focusing on LoRA\-adapted settings where the base LLM is preserved\. To achieve this, we propose Hybrid Search, a targeted correction strategy motivated by two observations\. First, interaction features that characterize the relationship between LLM\-based ASR hidden states and base\-LLM hidden states provide informative signals about a token’s degree of semantic dependence\. Second, selectively refining targeted tokens with high semantic dependence improves ASR performance far beyond naive global LLM\-correction methods including rescoring and late fusion\. Our analysis suggests that, even after semantic knowledge transfer through warm initialization, LLM\-based ASR models can still leverage their base LLM to further improve inference\-time performance\. Demo:[https://huggingface\.co/spaces/Splend1dchan/Listen\-To\-The\-Latent](https://huggingface.co/spaces/Splend1dchan/Listen-To-The-Latent)

## 1Introduction

Recent automatic speech recognition \(ASR\) models with billions of parameters trained on hundreds of thousands to millions of hours of audio have demonstrated strong performance across a wide range of scenarios\([Radford et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib27);[Puvvada et al\., 2024](https://arxiv.org/html/2609.02940#bib.bib28);[Peng et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib29);[Peng et al\., 2024](https://arxiv.org/html/2609.02940#bib.bib30)\)\. However, there remains a substantial data gap between ASR models and large language models \(LLMs\)\. For example,Whisper\-large\-v3was trained on approximately 5 million hours of speech, which roughly corresponds to tens of billions of text tokens in the paired transcriptions[OpenAI \(2023\)](https://arxiv.org/html/2609.02940#bib.bib37)\. This data scale is orders of magnitude smaller than the trillions of tokens used to train modern LLMs\. This discrepancy results in a semantic gap: LLMs possess significantly richer semantic representations and broader world knowledge than ASR models\. Consequently, leveraging LLMs offers a promising direction for improving ASR performance\([Ma et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib2)\), particularly for words that depend more on semantic context, such as named entities and uncommon words\.

![Refer to caption](https://arxiv.org/html/2609.02940v1/figs2/fig_method_v2.png)Figure 1:ASR\-LLM self\-correction framework\. \(a\) Section[3](https://arxiv.org/html/2609.02940#S3): Features are extracted from hidden\-state interactions between the ASR\-LLM and the LLM\. \(b\) Section[4](https://arxiv.org/html/2609.02940#S4): These features govern a gating mechanism that decides whether to incorporate LLM log\-probabilities into prediction\.Existing approaches typically coordinate LLMs with ASR systems in two major ways\. One direction is to treat the entire system as a "single model"\([Hsu et al\., 2025a](https://arxiv.org/html/2609.02940#bib.bib25);[Mittal et al\., 2024](https://arxiv.org/html/2609.02940#bib.bib23);[Hori et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib24)\), where linguistic information from the LLM is typically incorporated through logit\-level late fusion\. Another line of work emphasizes the agentic role of LLMs in ASR\([Yang et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib7);[Chen et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib22);[Radhakrishnan et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib6);[Wan et al\., 2026](https://arxiv.org/html/2609.02940#bib.bib3)\), where their instruction\-following capabilities are used to refine hypotheses from the ASR model\. However, these approaches fundamentally rely on correcting the outputs of a weaker ASR model rather than advancing the model’s underlying recognition ability\.

To leverage the benefits of LLMs in ASR, studies have explored directly integrating LLMs with the ASR backbone, which we refer to as ASR\-LLM throughout this work\. ASR\-LLMs such asPhi\-4\-Multimodal,SALM, andQwen3\-ASR\([Abouelenin et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib31);[Chen et al\., 2024b](https://arxiv.org/html/2609.02940#bib.bib32);[Shi et al\., 2026](https://arxiv.org/html/2609.02940#bib.bib19)\)are typically initialized from pretrained LLMs\([Abouelenin et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib31);[Shoeybi et al\., 2019](https://arxiv.org/html/2609.02940#bib.bib9);[Granite Team, 2024](https://arxiv.org/html/2609.02940#bib.bib8)\)\. The pretrained LLM backbone is combined with a speech encoder and jointly trained on large\-scale speech–language corpora\. Models built in this way achieve performance competitive with, or superior to, ASR systems trained from scratch and post\-hoc LLM correction methods[Srivastav et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib4)\. This performance pattern suggests that LLM semantic knowledge transfers more effectively to ASR through warm initialization than through post\-hoc correction\.

While ASR\-LLMs demonstrate strong empirical performance, it remains unclear how these models utilize the semantic and linguistic knowledge encoded within the integrated LLM\. Prior work has leveraged output distributions or uncertainty signals to improve factuality during decoding\([Chang et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib49);[Hsu et al\., 2025b](https://arxiv.org/html/2609.02940#bib.bib50);[Yang et al\., 2026](https://arxiv.org/html/2609.02940#bib.bib52)\)\. Beyond output\-level signals, layer\-wise representation analysis has long shown that different layers encode complementary information\([Peters et al\., 2018](https://arxiv.org/html/2609.02940#bib.bib47)\)\. More recent work interprets changes across layers as latent trajectories\([Wang et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib34);[Chen et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib33);[Chuang et al\., 2024](https://arxiv.org/html/2609.02940#bib.bib48)\), which can provide useful signals for improving decoding\. For example, the aggregated angle and magnitude features between consecutive layers of latent trajectories has been shown to enable differentiation between correct and incorrect reasoning samples[Wang et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib34)\. We extend prior studies by analyzing the geometric relationship between the hidden states of LoRA\-adapted ASR\-LLMs and those of their original base LLM\. Our analysis shows thatcosine similarityandrelative magnitudebetween these hidden states provide informative signals for identifying tokens with high semantic dependence\.

Building on this observation, we address a central question: can ASR\-LLMs be further improved by reusing their original base LLMs for external guidance during inference? We introduce“Hybrid Search”—a decoding algorithm that first uses hidden\-state interactions to identify tokens whose features fall in regions associated with high semantic dependence, then applies targeted LLM correction to these tokens\. This process is demonstrated in Figure[1](https://arxiv.org/html/2609.02940#S1.F1)\. Under our setting, Hybrid Search is fullyself\-corrective[Grill et al\. \(2020\)](https://arxiv.org/html/2609.02940#bib.bib12);[Chou et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib26), requiring neither additional parameters nor further training\. We focus our current evaluation on ASR, evaluating Hybrid Search under different computational budgets as both a beam\-search\-like decoding strategy[Hsu et al\. \(2025a\)](https://arxiv.org/html/2609.02940#bib.bib25)and a test\-time scaling method[Snell et al\. \(2024\)](https://arxiv.org/html/2609.02940#bib.bib21)\. More broadly, the proposed framework may extend to other tasks or conditions where task\-adapted models retain access to their original base LLM, such as speech translation and audio understanding[Hsu et al\. \(2025b\)](https://arxiv.org/html/2609.02940#bib.bib50);[Lin et al\. \(2026\)](https://arxiv.org/html/2609.02940#bib.bib51)\. Experimental results on ASR show that Hybrid Search improves named\-entity recognition over greedy search, beam search, and naive LLM\-integration methods such as rescoring and late fusion, while maintaining a WER comparable to beam search\. These results suggest that, even after semantic knowledge transfer through warm initialization, an ASR\-LLM can further leverage its original base LLM to improve inference\-time performance\.

## 2Background

LLM\-based ASR\.Previous work has laid the foundation for warm\-initializing LLM\-based ASR across diverse architectural and training paradigms, including fully finetuned systems[Shi et al\. \(2026\)](https://arxiv.org/html/2609.02940#bib.bib19)and parameter\-efficiently adapted models[Abouelenin et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib31);[Saon et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib20);[NVIDIA \(2025\)](https://arxiv.org/html/2609.02940#bib.bib13);[Chen et al\. \(2024b\)](https://arxiv.org/html/2609.02940#bib.bib32)\. These ASR\-LLMs have been shown to be more effective than models trained from scratch[Puvvada et al\. \(2024\)](https://arxiv.org/html/2609.02940#bib.bib28);[Radford et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib27);[Peng et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib29), as well as their LLM\-corrected variants[Hsu et al\. \(2025a\)](https://arxiv.org/html/2609.02940#bib.bib25);[Chen et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib22);[Lin et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib38);[Wan et al\. \(2026\)](https://arxiv.org/html/2609.02940#bib.bib3), on standard ASR benchmarks[Srivastav et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib4)\. In this work, we investigate self\-correction in LoRA\-based ASR\-LLMs through a representative case study ofPhi\-4\-Multimodal, and further validate its generality withGranite\-4\.0\-1B\-Speech\. Both models retain fully functional LLM backbones, enabling direct isolation of the LLM pathway\.111For Phi\-4\-Multimodal, the backbone is Phi\-4\-Mini; for Granite\-4\.0\-1B\-Speech, the backbone is Granite\-4\.0\-1B\-Base\.Furthermore, the ASR capability of both models has already been developed using a substantial amount of speech data \(100k to 2\.3 million hours\), making further data scaling a less attractive path for addressing any shortcomings in semantic knowledge transfer\.

Named Entities in Speech Recognition\.Named entities and domain\-specific rare words often carry disproportionate contextual importance in ASR, motivating extensive work on their recognition[Chen et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib22)\. Prior work has explored domain\-specialized language models[Liu et al\. \(2021\)](https://arxiv.org/html/2609.02940#bib.bib17)and explicit domain tags[Liao et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib18)to improve recognition performance\. Other approaches train embedded named\-entity recognition modules within ASR systems[Ayache et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib16)\. Complementarily, SpeechIQ[Wan et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib5)addresses the limited emphasis that word error rate \(WER\) places on semantically consequential recognition errors by computing LLM\-based similarity scores from embedding representations[Liu et al\. \(2024\)](https://arxiv.org/html/2609.02940#bib.bib14);[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.02940#bib.bib15)\. Following this line of work, we report named\-entity error rate \(NE\-ER\) in addition to WER to better characterize the semantic gains introduced by LLM\-based correction\.

## 3Hidden\-State Interactions Reveal Token Semantic Dependence

### 3\.1Extraction of Hidden State Interactions

Formalization\.An ASR\-LLM model withLLlayers can be represented as a composition of ordered submodules:

f=fhead∘fL∘⋯fl⋯∘f1∘fembf=f\_\{\\mathrm\{head\}\}\\circ f\_\{L\}\\circ\\cdots f\_\{l\}\\cdots\\circ f\_\{1\}\\circ f\_\{\\mathrm\{emb\}\}\\,\(1\)where the embedding modulefembf\_\{\\mathrm\{emb\}\}converts input tokens intodd\-dimensional embeddings\. The intermediate layers\{fl\}l=1L\\\{f\_\{l\}\\\}\_\{l=1\}^\{L\}successively transform these representations, and the classification headfheadf\_\{\\mathrm\{head\}\}maps the final hidden state into the vocabulary spaceVVto produce the output prediction\.

We define the hidden states as the collection of layer\-wise representations before the classification head\. For simplicity, we denote the embedding layerfembf\_\{\\mathrm\{emb\}\}asf0f\_\{0\}\. Given an input token sequence𝐈\\mathbf\{I\}, we define the hidden\-state vector𝐡li\\mathbf\{h\}\_\{l\}^\{i\}at token positioniias the output of the partial compositionfl∘⋯∘f1∘f0\(𝐈\)if\_\{l\}\\circ\\cdots\\circ f\_\{1\}\\circ f\_\{0\}\(\\mathbf\{I\}\)^\{i\}\. We then aggregate these vectors across layers to obtain the layer\-wise hidden states for positionii:

𝐇i=\[\(𝐡0i\)⊤;⋯;\(𝐡Li\)⊤\]∈ℝ\(L\+1\)×d\\mathbf\{H\}^\{i\}=\\big\[\(\\mathbf\{h\}\_\{0\}^\{i\}\)^\{\\top\};\\cdots;\(\\mathbf\{h\}\_\{L\}^\{i\}\)^\{\\top\}\\big\]\\in\\mathbb\{R\}^\{\(L\+1\)\\times d\}\(2\)
In our setting, the model operates in a speech–text regime, where hidden states vary with the input modality\. We denote by𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}the hidden states produced by the ASR\-LLM during ASR decoding at a given step, and by𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}the hidden states obtained by running the base LLM on the corresponding text\-only input\.

𝐇ASR​\-​LLM=f⁡\(𝐈<i,A,\{ϕLLM,ϕLoRA\}\)\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}=f\(\\mathbf\{I\}\_\{<i\};A;\\\{\\phi\_\{\\mathrm\{LLM\}\},\\phi\_\{\\mathrm\{LoRA\}\}\\\}\)\(3\)𝐇LLM=f⁡\(𝐈<i,\{ϕLLM\}\)\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}=f\(\\mathbf\{I\}\_\{<i\};\\\{\\phi\_\{\\mathrm\{LLM\}\}\\\}\)\(4\)
Here,𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}is conditioned on both the audio encoder outputAAand the already decoded context plus the prompt𝐈<i\\mathbf\{I\}\_\{<i\}, under the active ASR\-LLM parameters\{ϕLLM,ϕLoRA\}\\\{\\phi\_\{\\mathrm\{LLM\}\},\\phi\_\{\\mathrm\{LoRA\}\}\\\}\. In contrast,𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}is obtained by running the underlying LLM on only text context𝐈<i\\mathbf\{I\}\_\{<i\}using onlyϕLLM\\phi\_\{\\mathrm\{LLM\}\}\. To analyze hidden\-state interactions, we pair each speech\-conditioned trajectory𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}with its text\-only counterpart𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}, computed under the same text context𝐈<i\\mathbf\{I\}\_\{<i\}\.

\(a\)Layer\-wise averages and standard deviations ofAngleFeat\\mathrm\{AngleFeat\}andMagFeat\\mathrm\{MagFeat\}\.Intermediate Features:finter​\(i\)=13​∑l∈Top310≤l≤28⁡\(f\)f⁡\(l,i\),f∈\{AngleFeat,MagFeat\}f\_\{\\mathrm\{inter\}\}\(i\)=\\frac\{1\}\{3\}\\sum\_\{l\\in\\operatorname\{Top3\}\_\{10\\leq l\\leq 28\}\(f\)\}f\(l,i\),\\quad f\\in\\\{\\mathrm\{AngleFeat\},\\mathrm\{MagFeat\}\\\}Single\-Token WordsMulti\-Token WordsStarting TokenContinuation Token\(s\)Late\-Layer Features:flate​\(i\)=f⁡\(31,i\),f∈\{AngleFeat,MagFeat\}f\_\{\\mathrm\{late\}\}\(i\)=f\(31,i\),\\quad f\\in\\\{\\mathrm\{AngleFeat\},\\mathrm\{MagFeat\}\\\}Single\-Token WordsMulti\-Token WordsStarting TokenContinuation Token\(s\)Non\-named\-entity tokensNamed\-entity tokens\(b\)Gaussian distributions of hidden\-state interaction features fornamed\-entityandnon\-named\-entitytokens\. The upper row aggregates intermediate\-layer features, while the lower row directly selects the features at layer 31\.

Figure 2:Top:Average values and standard deviations ofAngleFeat⁡\(l,i\)\\mathrm\{AngleFeat\}\(l,i\)andMagFeat⁡\(l,i\)\\mathrm\{MagFeat\}\(l,i\)across tokens reveal distinct layer\-wise phases\.Bottom:Hidden\-state interaction feature distributions fornamed\-entityandnon\-named\-entitytokens\. Intermediate\-layer aggregation reveals a clear distribution shift, while the late\-layer features at layer 31 exhibit a further shift toward lower cosine similarity and higher relative norm for named\-entity tokens\.Feature Extraction\.Given paired hidden states from the ASR\-LLM and the base LLM \(𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}and𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}\), the representation associated with each prediction token remains high\-dimensional, with size2×\(L\+1\)×d2\\times\(L\+1\)\\times d\. To obtain compact and interpretable descriptors, prior work reduces the hidden dimensionddby derivingangleandmagnitudefeatures between inter\-layer representation trajectories\([Wang et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib34)\)\. Unlike prior work, we compute them by comparing “cross\-modal” hidden states𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}and𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}at corresponding layers\. This design follows from our goal of characterizing how the pre\-adapted𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}are transformed into the post\-adapted𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}\. Thus, each layer serves as a matched reference point for measuring ASR\-induced representational change\. For each layerlland positionii, we define these features as follows:

AngleFeat⁡\(l,i\):=cos⁡\(𝐡ASR​\-​LLMl,i,𝐡LLMl,i\)\\mathrm\{AngleFeat\}\(l,i\):=\\cos\(\\mathbf\{h\}^\{l,i\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\},\\mathbf\{h\}^\{l,i\}\_\{\\mathrm\{LLM\}\}\)\(5\)MagFeat⁡\(l,i\)=‖𝐡ASR​\-​LLMl,i‖2/‖𝐡LLMl,i‖2\.\\mathrm\{MagFeat\}\(l,i\)=\\\|\\mathbf\{h\}^\{l,i\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}\\\|\_\{2\}/\\\|\\mathbf\{h\}^\{l,i\}\_\{\\mathrm\{LLM\}\}\\\|\_\{2\}\.\(6\)
Under this formulation, each token is characterized by 2 scalar descriptors at each layer\. In the following subsection, we further aggregate these layer\-wise features over selected layers to obtain exactly 2 scalar features for each predicted token in the speech recognition task\. This compact parameterization allows us to fit Gaussian distributions over different token categories, such as whether a token is semantically grounded, and to quantify distributional shifts across these categories\.

### 3\.2How Hidden State Interactions Reveal Token Semantic Dependence

Analysis Setup\.We aim to examine howAngleFeatandMagFeatcharacterize semantic reliance\. To operationalize “semantic dependence,” we treat named\-entity tokens as a representative subset, motivated by empirical evidence from prior work[Chen et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib22)\. Conversely, we use non\-named\-entity tokens as a proxy for tokens whose prediction is more strongly grounded in acoustic evidence\. We perform this analysis on theYODASdataset\([Li et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib35)\), a CC\-licensed audio corpus with substantial acoustic and linguistic diversity\. Named\-entity tokens are identified using a BERT\-based NER tagger222[dslim/bert\-base\-NER\-uncased](https://huggingface.co/dslim/bert-base-NER-uncased)\. We selectPhi\-4\-Multimodalas the ASR\-LLM model, whose base LLM isPhi\-4\-Mini\([Abouelenin et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib31)\)\. We apply teacher forcing using the reference transcription as input, yielding2×N×\(L\+1\)2\\times N\\times\(L\+1\)features for each sample, whereNNdenotes the sequence length\. These per\-sample features are best visualized by heatmaps shown in Appendix[C\.4](https://arxiv.org/html/2609.02940#A3.SS4)\.

We require another feature size reduction\(L\+1\)→1\(L\+1\)\\rightarrow 1for distributional analysis\. To obtain proper aggregations on the layer dimension, we look into the macro trajectories ofAngleFeat\\mathrm\{AngleFeat\}andMagFeat\\mathrm\{MagFeat\}displayed at the top of Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\. We categorize the layers into early \(0\-7\), intermediate \(8\-28\), and late \(29\-32\) phases based on changes in behavior observed in the feature trajectories\. We structure the following discussion around these 3 layer phases to examine potential aggregation strategies\.

Early Layers, Layers 0–7:Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\(a\) shows that bothAngleFeat\\mathrm\{AngleFeat\}andMagFeat\\mathrm\{MagFeat\}remain close to 1 in the early layers, indicating that the ASR\-LLM hidden states deviate only minimally from the corresponding LLM hidden states\. The token\-level standard deviations indicated by the shaded regions are also small, suggesting that there is little room for different token categories to exhibit distinguishable trajectory patterns\.

Intermediate Layers, Layers 8–28:The middle portion of Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\(a\) shows larger standard deviations across tokens, suggesting that feature values vary substantially depending on token identity and context\. Manual inspection shows that named\-entity tokens have higherAngleFeat\\mathrm\{AngleFeat\}andMagFeat\\mathrm\{MagFeat\}within some of the intermediate layers, forming peak\-like patterns\. We thus derive “peak” features from intermediate layers by taking the top\-3 mean ofAngleFeat\\mathrm\{AngleFeat\}andMagFeat\\mathrm\{MagFeat\}from layers 10 to 28\. We refer to the aggregated features asAngleFeati​n​t​e​r\\mathrm\{AngleFeat\}\_\{inter\}andMagFeati​n​t​e​r\\mathrm\{MagFeat\}\_\{inter\}\. We then project each token onto a 2D feature plane using its peak feature values, and estimate the Gaussian feature distributions of named\-entity and non\-named\-entity tokens, as visualized in Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\(b\)\. From Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\(b\), we observe a distributional shift between the Gaussian centroids of named\-entity and non\-named\-entity tokens for single\-token words and continuation tokens\. This suggests that the two token types follow distinct latent trajectories in the ASR\-LLM\. In particular, named\-entity tokens occupy the region characterized by highAngleFeat\\mathrm\{AngleFeat\}and highMagFeat\\mathrm\{MagFeat\}\. We observe that the largest distributional discrepancy is primarily driven by continuation tokens in multi\-token words, which are often collocative with the preceding token; for example, predicting “\-tras” after “orches” to form the word “orchestras\.” The ASR\-LLM leverages the𝐇LLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}trajectory by aligning𝐇ASR​\-​LLM\\mathbf\{H\}\_\{\\mathrm\{ASR\\text\{\-\}LLM\}\}more closely with it in angular space during the intermediate layers\. We demonstrate the generality of this behavior in Appendix[C\.1](https://arxiv.org/html/2609.02940#A3.SS1)withGranite\-4\.0\-1B\-Speech\.

Late Layers, Layers 29–32:In Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\(a\), late layers are characterized by a further decrease inAngleFeat\\mathrm\{AngleFeat\}\. When isolating layer 31, we find that named\-entity tokens shift toward lowerAngleFeat\\mathrm\{AngleFeat\}and higherMagFeat\\mathrm\{MagFeat\}\(Figure[2](https://arxiv.org/html/2609.02940#S3.F2)\(b\)\)\. Since this effect is weaker than the shift observed in the intermediate layers, we do not use it in our method\. We include further analysis in Appendix[B\.1](https://arxiv.org/html/2609.02940#A2.SS1)\.

Summary\.Our analysis shows thatmulti\-token named entitiesexhibit decoding trajectories that are clearly distinct from those of non\-named entities\. This finding echoes the effect of tokenization bias\([Lesci et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib10)\), where probabilities for multi\-token words can be miscalibrated relative to those for single\-token words\. In the following section, we examine how deviations in latent trajectories affect ASR correctness and how these insights can be used to improve recognition performance\.

AngleFeati​n​t​e​r\\mathrm\{AngleFeat\}\_\{inter\}0\.830\.800\.770\.900\.950\.900\.860\.780\.810\.830\.85MagFeati​n​t​e​r\\mathrm\{MagFeat\}\_\{inter\}1\.071\.191\.101\.101\.181\.131\.041\.071\.111\.061\.05Non TargetedTargetedNon TargetedTokens\_Here\_was\_Bela\_Karolyi,\(\_not\)\_not\_just\_upset\_thatoly,\(\_not\)∅\\varnothing\_Caroli,\(\_not\)∅\\varnothing

Figure 3:Visualization of our proposed Hybrid Search\. When features lie in the selected regions \(red border\), beam search is initiated \(shaded region\), backtracking to the beginning\-of\-word token\. Upon reaching a consensus non\-targeted beginning\-of\-word token \("\_not"\), beams are rescored and decoding reverts to greedy search\. The symbol∅\\varnothingdenotes subsequent generated tokens that are omitted\.

## 4Hidden State Interactions Enable ASR Self\-Correction

### 4\.1Targeted Self\-Correction

Formulation\.Prior late\-fusion error\-correction methods interpolate ASR and LLM probabilities for next\-token selection\([Hsu et al\., 2025a](https://arxiv.org/html/2609.02940#bib.bib25);[Chen et al\., 2024a](https://arxiv.org/html/2609.02940#bib.bib36)\)\. When applied to the ASR\-LLM setting, this gives:

\(1−α\)​log⁡pASR\-LLM\+α​log⁡pLLM\(1\-\\alpha\)\\log p\_\{\\textrm\{ASR\\text\{\-\}LLM\}\}\+\\alpha\\log p\_\{\\textrm\{LLM\}\}\(7\)wherelog⁡pM\\log p\_\{\\textrm\{M\}\}denotes the log\-probabilities of the next token produced by the modelM, andα\\alphacontrols the interpolation weight\. While LLM contributions can correct token predictions that rely heavily on semantic context, the lack of audio input also makes LLM\-based fusion susceptible to hallucinations when the decoded token is acoustically dominant\([Hsu et al\., 2025a](https://arxiv.org/html/2609.02940#bib.bib25)\)\. Therefore, we explore a targeted correction framework that applies LLM probability fusion only to targeted, semantically grounded tokens\. The log\-probability score is correspondingly updated to a piecewise function:

\{\(1−α\)​log⁡pASR\-LLM\+α​log⁡pLLM,if targeted,log⁡pASR\-LLMotherwise\.\\begin\{cases\}\(1\-\\alpha\)\\log p\_\{\\textrm\{ASR\\text\{\-\}LLM\}\}\+\\alpha\\log p\_\{\\textrm\{LLM\}\},&\\text\{if targeted\},\\\\\[2\.0pt\] \\log p\_\{\\textrm\{ASR\\text\{\-\}LLM\}\}&\\text\{otherwise\.\}\\end\{cases\}\(8\)
Targeted Token Selection\.We identify named\-entity\-dense regions from the Gaussian distributions introduced in Section[3\.2](https://arxiv.org/html/2609.02940#S3.SS2)\. These regions define a targeted\-token selector𝒯⁡\(⋅\)\\mathcal\{T\}\(\\cdot\):

𝒯\(xi\)=𝟏\[PNE​\(𝐳i\)PnonNE​\(𝐳i\)≥τ\],\\mathcal\{T\}\(x\_\{i\}\)=\\mathbf\{1\}\\left\[\\frac\{P\_\{\\mathrm\{NE\}\}\(\\mathbf\{z\}\_\{i\}\)\}\{P\_\{\\mathrm\{nonNE\}\}\(\\mathbf\{z\}\_\{i\}\)\}\\geq\\tau\\right\],\(9\)where𝐳i=\(AngleFeati,MagFeati\)\\mathbf\{z\}\_\{i\}=\(\\operatorname\{AngleFeat\}\_\{i\},\\operatorname\{MagFeat\}\_\{i\}\)denotes the two\-dimensional feature representation of tokenxix\_\{i\}, andPNEP\_\{\\mathrm\{NE\}\}andPnonNEP\_\{\\mathrm\{nonNE\}\}are the Gaussian density estimates from the YODAS analysis for named\-entity and non\-named\-entity tokens, respectively\. The thresholdτ\\taucontrols the permissiveness of the targeted region\.

We visualize two Gaussian density\-ratio contours for each scenario in Figure[2](https://arxiv.org/html/2609.02940#S3.F2): the equi\-density boundary \(τ=1\\tau=1\), where the estimated named\-entity and non\-named\-entity densities are equal, and a more permissive 1:2 density\-ratio boundary \(τ=12\\tau=\\frac\{1\}\{2\}\), which expands the selected region to include more named\-entity tokens at the cost of lower precision\. For multi\-token words, we only apply the boundary to continuation tokens, as starting tokens do not exhibit clear separation between named\-entity and non\-named\-entity tokens\. The decision boundaries identify an upper\-right region where named\-entity tokens are densely concentrated\. For interpretability, we perform word\-level selection by expanding each targeted token to the full word span containing it\. This means that a multi\-token word is selected if any of its continuation tokens is targeted\. We note that the NER model or ground\-truth entity information is no longer needed after calibration, and the inference algorithm depends only on the hidden\-state interaction features𝐳i\\mathbf\{z\}\_\{i\}\.

### 4\.2Self\-Correction Algorithm

ExperimentGreedy SearchOne\-Pass1×1\\times\+ Hybrid Search\(Decoding\)One\-Pass1\.4×∗1\.4\\times^\{\*\}Beam SearchOne\-Pass5×5\\times\+ Hybrid Search\(Error Correction\)Two\-Pass–∗Amortized effective beam size, averaged over tokens decoded with beam search and greedy search\. \(See Section[7\.4](https://arxiv.org/html/2609.02940#S7.SS4)\)Table 1:Hybrid Search for Decoding provides an efficient alternative within the family of beam\-search methods, whereas Hybrid Search for Error Correction requires a second pass but further improves upon Beam Search \(Table[2](https://arxiv.org/html/2609.02940#S5.T2)\)\.We now instantiate Equation[8](https://arxiv.org/html/2609.02940#S4.E8)as a decoding algorithm\. A challenging aspect of this formula lies in the instability of cross\-beam comparisons between targeted tokens and non\-targeted tokens, as their scores are on different scales\. We proposeHybrid Search, a decoding algorithm that restricts comparisons to hypotheses with aligned targeted and non\-targeted token groups\. Hybrid Search is a mixture between greedy search and beam search, and conducts beam search \(beam sizeBB\) only for targeted words\.

The full decoding process switches state between different beam sizes according to the token type\. The transition from beam size 1 toBBupon encountering a targeted token is trivial\. To reduce the beam sizeBBto 1, we continue decoding until all beams converge on the same non\-targeted token\. We then roll back any beams that may have decoded past the converged token and select the best hypothesis from the resulting candidates\. Hybrid Search greatly reduces decoding cost relative to beam search, since most decoding steps remain in the greedy phase\. We note that this process is made possible by targeting only a subset of tokens for expansion\. Figure[3](https://arxiv.org/html/2609.02940#S3.F3)illustrates hybrid search in action through explored decoding traces\. We provide the full procedure in Algorithm[1](https://arxiv.org/html/2609.02940#alg1)in the Appendix[B\.2](https://arxiv.org/html/2609.02940#A2.SS2)\.

## 5Experimental Settings

We evaluate Hybrid Search in two settings\. First, we look intoHybrid Search for Decoding, directly implementing the procedure in Section[4\.2](https://arxiv.org/html/2609.02940#S4.SS2)\. By applying beam expansion only at selected positions, the method incurs a cost between greedy search and full beam search while remaining a one\-pass decoding algorithm\. Second, we exploreHybrid Search for Error Correction, adopting a two\-pass procedure that starts from the best beam\-search hypothesis and applies Hybrid Search as a refinement step\. For non\-targeted positions, we use teacher forcing on the beam\-search hypothesis rather than rerunning decoding\. The relationship between the two distinct Hybrid Search methods and their decoding complexity is shown in Table[1](https://arxiv.org/html/2609.02940#S4.T1)\.

We evaluate Hybrid Search onPhi\-4\-Multimodalusing the Open ASR Leaderboard benchmark\([Srivastav et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib4)\), following its standard setup of seven audio subsets drawn from diverse sources: AMI\([Carletta et al\., 2005](https://arxiv.org/html/2609.02940#bib.bib40)\), Earnings22\([Del Rio et al\., 2022](https://arxiv.org/html/2609.02940#bib.bib41)\), GigaSpeech\([Chen et al\., 2021](https://arxiv.org/html/2609.02940#bib.bib42)\), LibriSpeech\([Panayotov et al\., 2015](https://arxiv.org/html/2609.02940#bib.bib43)\), SPGISpeech\([O’Neill et al\., 2021](https://arxiv.org/html/2609.02940#bib.bib44)\), TED\-LIUM\([Hernandez et al\., 2018](https://arxiv.org/html/2609.02940#bib.bib45)\), and VoxPopuli\([Wang et al\., 2021](https://arxiv.org/html/2609.02940#bib.bib46)\)\. In addition, we include Common Voice\([Ardila et al\., 2020](https://arxiv.org/html/2609.02940#bib.bib11)\)as an additional evaluation set to assess robustness across diverse speakers\. We report both WER and named\-entity error rate \(NE\-ER\)\. NE\-ER is calculated as the number of unmatched named\-entity words divided by the total number of named\-entity words, where named entities are identified using the BERT\-based NER tagger in Section[3\.2](https://arxiv.org/html/2609.02940#S3.SS2)\. Following beam search conventions, we set the beam size of Hybrid Search toB=5B=5, and use an interpolation weight ofα=0\.2\\alpha=0\.2\. We report results withτ=12\\tau=\\frac\{1\}\{2\}by default, and additionally evaluateτ=1\\tau=1for Hybrid Search for Error Correction\.

Table 2:WER and NE\-ER across the Open ASR Leaderboard and Common Voice \(in %\)\.Top:Phi\-4\-Multimodal Beam Search outperforms the best prior LLM\-corrective methods that incorporate beam search internally on most benchmarks\.Middle and Bottom:Hybrid Search applied to Greedy Search provides a cost\-effective alternative to Beam Search, recovering 25% of its WER improvement and 96% of its NE\-ER improvement\. When applied to Beam Search, Hybrid Search further reduces average NE\-ER by 3\.3% while maintaining comparable WER\. In contrast, naive refinement methods such as rescoring and late fusion noticeably degrade WER\. Values below the refinement results report two\-sided exact McNemar testpp\-values against the Beam Search baseline; significant results \(p<0\.05p<0\.05\) are shown in teal and non\-significant results in gray\.Open ASR LeaderboardCommonVoiceAvg\.AMIEarn\.Giga\.Libri\.SPGI\.TED\.Vox\.Word Error Rate\(A\)GER⇒\\RightarrowParakeet[Chen et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib22)22\.91–12\.104\.843\.986\.097\.49––\(B\)Neko Qwen1\.5\-MoE[Lin et al\. \(2025\)](https://arxiv.org/html/2609.02940#bib.bib38)12\.6011\.829\.952\.321\.943\.205\.80––\(C\)Speech\-Hands⇌\\rightleftharpoonsParakeet[Wan et al\. \(2026\)](https://arxiv.org/html/2609.02940#bib.bib3)11\.20–11\.103\.182\.164\.376\.02––Best of LLM\-corrective \(A–C\)\(Beam Search\)11\.2011\.829\.952\.321\.943\.205\.80––Phi\-4\-Multimodal \(Beam Search\)11\.119\.399\.073\.642\.412\.645\.746\.916\.36Word Error Rate\(McNemarpp\-value below\)Phi\-4\-Multimodal \(Greedy Search\)11\.389\.689\.193\.822\.462\.725\.977\.336\.57\+ Hybrid Search \(τ=12\\tau=\\frac\{1\}\{2\}\)11\.389\.739\.073\.702\.482\.685\.877\.206\.51\- LLM Correction11\.359\.739\.093\.722\.472\.705\.837\.096\.50Phi\-4\-Multimodal \(Beam Search\)11\.119\.399\.073\.642\.412\.645\.746\.916\.36\+ Rescoring12\.5010\.089\.223\.672\.482\.716\.116\.896\.71Late Fusion12\.7912\.0512\.075\.895\.055\.038\.3211\.689\.11Phi\-4\-Multimodal \(Beam Search\)11\.119\.399\.073\.642\.412\.645\.746\.916\.36\+ Hybrid Search \(τ=1\\tau=1\)11\.111\.009\.401\.009\.05<1​e​\-​5<1e\\text\{\-\}53\.64\.7492\.41<1​e​\-​3<1e\\text\{\-\}32\.62\.1255\.73\.6886\.89\.0516\.36<1​e​\-​8<1e\\text\{\-\}8\+ Hybrid Search \(τ=12\\tau=\\frac\{1\}\{2\}\)11\.12\.4519\.40\.6209\.05\.0073\.63\.6712\.40\.0162\.59\.0045\.74\.8646\.88\.0716\.35<1​e​\-​4<1e\\text\{\-\}4\+ Hybrid Search \(τ=0\\tau=0\)11\.169\.569\.063\.642\.412\.655\.836\.946\.41Named\-Entity Error Rate\(McNemarpp\-value below\)Phi\-4\-Multimodal \(Greedy Search\)26\.5731\.7821\.6423\.2715\.469\.915\.1522\.5919\.55\+ Hybrid Search \(τ=12\\tau=\\frac\{1\}\{2\}\)24\.7129\.4420\.3122\.2115\.209\.034\.9121\.2318\.38\- LLM Correction25\.3330\.2020\.7722\.5315\.369\.034\.9121\.6618\.72Phi\-4\-Multimodal \(Beam Search\)24\.7130\.2020\.8222\.1214\.488\.324\.8620\.8318\.29\+ Rescoring25\.4729\.4920\.1122\.9012\.787\.614\.6320\.1017\.89Late Fusion24\.7930\.2721\.1422\.7214\.778\.325\.3422\.4918\.73Phi\-4\-Multimodal \(Beam Search\)24\.7130\.2020\.8222\.1214\.488\.324\.8620\.8318\.29\+ Hybrid Search \(τ=1\\tau=1\)24\.711\.0029\.891\.0020\.46<1​e​\-​6<1e\\text\{\-\}622\.211\.0014\.15<1​e​\-​7<1e\\text\{\-\}77\.79\.254\.72\.2520\.48<1​e​\-​4<1e\\text\{\-\}418\.05<1​e​\-​16<1e\\text\{\-\}16\+ Hybrid Search \(τ=12\\tau=\\frac\{1\}\{2\}\)24\.32\.12529\.82\.38320\.11<1​e​\-​11<1e\\text\{\-\}1122\.53\.29813\.09<1​e​\-​19<1e\\text\{\-\}197\.26\.0704\.43\.01219\.96<1​e​\-​10<1e\\text\{\-\}1017\.69<1​e​\-​37<1e\\text\{\-\}37\+ Hybrid Search \(τ=0\\tau=0\)23\.7829\.7420\.3322\.4913\.688\.144\.6720\.4317\.91

## 6Experimental Results

### 6\.1Baselines

We first examine howPhi\-4\-Multimodalperforms in the broader landscape of LLM\-integrated ASR systems\. The top portion of Table[2](https://arxiv.org/html/2609.02940#S5.T2)comparesPhi\-4\-Multimodalwith the aggregated best results from prior LLM\-based corrective ASR methods on the Open ASR Leaderboard\([Chen et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib22);[Lin et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib38);[Wan et al\., 2026](https://arxiv.org/html/2609.02940#bib.bib3)\)\. Beam search withPhi\-4\-Multimodalachieves the best WER on 5 of the 7 benchmarks and outperforms the ensemble of prior corrective methods overall, which also incorporate beam search within their correction loops\. This performance is achieved with fewer parameters than existing LLM\-based corrective approaches, highlighting the effectiveness of warm\-initialized LLM\-based ASR models and motivating further exploration of self\-correction directly withinPhi\-4\-Multimodal\.

### 6\.2Hybrid Search for Decoding

The rows highlighted in green in Table[2](https://arxiv.org/html/2609.02940#S5.T2)compare Hybrid Search \(τ=12\\tau=\\frac\{1\}\{2\}\) with standard decoding methods\. Hybrid Search recovers 25% of the improvement achieved by beam search over greedy search\. This gain is substantially larger for named entities, where Hybrid Search recovers 96\.4% of the improvement achieved by beam search\. As Hybrid Search combines greedy search and beam search, its computational cost lies between those of the two decoding methods \(Table[1](https://arxiv.org/html/2609.02940#S4.T1)\)\. On average, Hybrid Search targets 10% of tokens for beam expansion\. WithB=5B=5, this gives an amortized beam size of 1\.4 for Hybrid Search, corresponding to approximately1\.4×1\.4\\timesthe greedy\-decoding cost\. These results establish Hybrid Search as an efficient and effective decoding algorithm, especially for improving semantically important tokens\.

How important is the LLM contribution within the Hybrid Search algorithm? To isolate its contribution, we keep the Hybrid Search algorithm unchanged but remove LLM rescoring by settingα=0\\alpha=0\. This variant is denoted as “\- LLM Correction” in Table[2](https://arxiv.org/html/2609.02940#S5.T2)\. While removing the LLM contribution achieves comparable WER, its named\-entity improvement drops to 68\.3% of the beam\-search gain \(from 96\.4%\)\. These results suggest that the targeted regions identify uncertain tokens that benefit not only from additional search, but also from the linguistic information provided by the base LLM\. Since our method is self\-corrective, this result further suggests that speech\-recognition adaptation may induce partial forgetting of semantic knowledge, motivating future work on more effective adaptation strategies for LLM\-to\-ASR\-LLM training\.

### 6\.3Hybrid Search for Error Correction

We combine Hybrid Search with beam search to improve decoding robustness\. Rows highlighted in yellow in Table[2](https://arxiv.org/html/2609.02940#S5.T2)show the results of Hybrid Search compared to Beam Search and naive LLM correction methods such as rescoring and late fusion\. We observe that Hybrid Search improves named\-entity recognition over beam search while maintaining comparable overall WER, whereτ=12\\tau=\\frac\{1\}\{2\}bounds the more favorable targeted region among the two thresholds\. Under this setting, NE\-ER decreases from 18\.29% to 17\.69%\. The largest gains occur on TED\-LIUM and SPGISpeech, where the NE\-ER relative improvements are 12\.7% and 8\.8% respectively\. Two\-sided exact McNemar tests are shown in Table[2](https://arxiv.org/html/2609.02940#S5.T2)below the corresponding results, confirming that the improvements are statistically significant for both NE\-ER \(p=3\.56×10−38p=3\.56\\times 10^\{\-38\}\) and WER \(p=9\.19×10−5p=9\.19\\times 10^\{\-5\}\)\.

The results indicate that our targeted correction method reliably improves difficult predictions, particularly named entities, even when such cases are sparse, without degrading WER\. Nevertheless, because named entities substantially affect semantic fidelity and downstream utility, these gains are particularly meaningful\([Wan et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib5)\)\. We observe that most edits occur within multi\-token words, highlighting a systematic issue in howPhi\-4\-Multimodaladapts to ASR\.

## 7Analysis

### 7\.1Analysis of Correction Examples

Manual inspection of the edits introduced by our proposed Hybrid Search reveals that most edits, whether correct or incorrect, involve semantically specific expressions, particularly named entities\. These include person names \(e\.g\.,Kim Zmeskal\), organizations \(e\.g\.,Cedefop\), and brands \(e\.g\.,Telenor\)\. Foreign and less common lexical items also frequently appear among the edited tokens, such asseñor,Guanyin, andKhan al\-Ahmar\. Overall, Hybrid Search primarily modifies tokens for which linguistic or semantic context is informative, while hallucination\-inducing edits are minimal\. Representative examples are provided in Appendix[C\.3](https://arxiv.org/html/2609.02940#A3.SS3)\.

### 7\.2Is Targeted Correction Necessary?

As an ablation of targeted hybrid search, we include non\-targeted baselines, including naive rescoring from beam search and direct late fusion\. From Table[2](https://arxiv.org/html/2609.02940#S5.T2), it is evident that direct late fusion performs remarkably worse \(WER 6\.36→\\rightarrow9\.11\), as the results frequently introduce hallucinated outputs from undesired LLM guidance\. In contrast, naive rescoring still degrades on WER but can improve on NE\-ER\.

Figure 4:Relative WER and NE\-ER improvements across test sets, measured against beam search\. Dashed lines connect Rescoring and Hybrid Search results, showing that Hybrid Search substantially improves WER while maintaining NE\-ER accuracy\.We visualize the comparison between rescoring, beam search, and Hybrid Search in Figure[4](https://arxiv.org/html/2609.02940#S7.F4)\. Using beam search as the anchor, we plot the relative WER and NE\-ER improvements of other experiments\. Dashed lines connect each “Rescoring” result to its corresponding Hybrid Search result withτ=12\\tau=\\frac\{1\}\{2\}\. The lines reveal that our method substantially improves WER performance while maintaining NE\-ER compared to rescoring\. These results show that targeted correction is necessary to improve NE\-ER beyond beam search without sacrificing overall WER\.

### 7\.3Are Intermediate Interactions Necessary?

We observe from[3\.2](https://arxiv.org/html/2609.02940#S3.SS2)and Section[4\.1](https://arxiv.org/html/2609.02940#S4.SS1)that the distribution shift is largely driven by multi\-token words, and the targeted words are heavily biased towards them\. Since multi\-token words can be determined simply from the surface form of the output tokens, we discuss the necessity of including boundaries that are tied to hidden\-state features\. In Table[2](https://arxiv.org/html/2609.02940#S5.T2), we include a baseline withτ=0\\tau=0that targets only multi\-token words and does not use any intermediate features\. This setting fails to preserve the WER robustness of beam search, demonstrating that the proposed hidden\-state interactions are essential for effective targeting\.

### 7\.4Complexity Analysis

We elaborate on the efficiency of Hybrid Search for Decoding\. The decoding latency ofPhi\-4\-Multimodal, is bounded by the autoregressive decoding ofNNsteps\. In our method, the ASR\-LLM and its internal LLM can be executed in parallel, so the overall latency remains bounded by these sameNNdecoding steps\. The feature extraction cost is negligible compared with a model forward pass\. Specifically, computingc​o​s​\(⋅\)cos\(\\cdot\)andn​o​r​m​\(⋅\)norm\(\\cdot\)together requires approximately10​h​L10hLFLOPs, whereh=3072h=3072is the hidden size andL=33L=33is the total number of layers plus the embedding layer\. This gives10×3072×33≈10610\\times 3072\\times 33\\approx 10^\{6\}FLOPs per decoding position, compared with a model forward pass whose computation is on the order of10910^\{9\}FLOPs\. The decoding complexity thus reduces to the selective beam\-search cost, given byBeff=\(1−ρ\)\+ρ​BB\_\{\\mathrm\{eff\}\}=\(1\-\\rho\)\+\\rho B\. With a beam search expansion selection rate ofρ=0\.1\\rho=0\.1and beam sizeB=5B=5, this equals an amortized cost of1\.4×1\.4\\timesgreedy decoding\.

## 8Conclusions

This paper introduces Hybrid Search, a self\-correction algorithm for ASR\-LLMs that combines targeted refinement with semantic\-based correction\. Using simple latent descriptors, including cosine similarity and relative norm between the ASR\-LLM and its base\-LLM counterpart, we identify clear distributional signals associated with semantically dependent tokens forPhi\-4\-MultimodalandGranite\-4\.0\-1B\-Speech\. Experiments withPhi\-4\-Multimodalfurther show that our method improves NE\-ER while maintaining competitive WER, without requiring external models or additional training\. These results suggest that even for warm\-initialized models, residual knowledge from the base LLM may remain exploitable at inference time\.

## Limitations

While we validate the hidden\-state observations on bothPhi\-4\-MultimodalandGranite\-4\.0\-1B\-Speech, the end\-to\-end Hybrid Search experiments are conducted only onPhi\-4\-Multimodal\. Therefore, the generalization of the full correction method across ASR\-LLM architectures remains to be established\. The exact feature distributions, layer aggregation, and decision thresholds may be model\-dependent and require recalibration, and may not be available for closed\-source systems\. Although we observe generalization across the Open ASR Leaderboard and Common Voice, feature\-space distributions can vary across models and domains\. More adverse settings, such as singing ASR, may therefore require additional calibration or adaptation\.

## References

- Aboueleninet al\.\(2025\)A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen,et al\.Phi\-4\-mini technical report: compact yet powerful multimodal language models via Mixture\-of\-LoRAs\.arXiv preprint arXiv:2503\.01743\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p2.1),[§1](https://arxiv.org/html/2609.02940#S1.p3.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1),[§3\.2](https://arxiv.org/html/2609.02940#S3.SS2.p1.1)\.
- Ardilaet al\.\(2020\)R\. Ardila, M\. Branson, K\. Davis, M\. Kohler, J\. Meyer, M\. Henretty, R\. Morais, L\. Saunders, F\. Tyers, and G\. WeberCommon Voice: a massively\-multilingual speech corpus\.InProceedings of the twelfth language resources and evaluation conference,pp\. 4218–4222\.Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Ayacheet al\.\(2025\)G\. Ayache, M\. Pirchi, A\. Navon, A\. Shamsian, G\. Hetz, and J\. KeshetWhisperNER: unified open named entity and speech recognition\.In2025 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–6\.Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p2.1)\.
- Carlettaet al\.\(2005\)J\. Carletta, S\. Ashby, S\. Bourban, M\. Flynn, M\. Guillemot, T\. Hain, J\. Kadlec, V\. Karaiskos, W\. Kraaij, M\. Kronenthal,et al\.The AMI meeting corpus: a pre\-announcement\.InInternational workshop on machine learning for multimodal interaction,pp\. 28–39\.Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Changet al\.\(2025\)H\. Chang, N\. Peng, M\. Bansal, A\. Ramakrishna, and T\. ChungREAL sampling: boosting factuality and diversity of open\-ended generation by extrapolating the entropy of an infinitely large LM\.Transactions of the Association for Computational Linguistics13,pp\. 760–783\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00757)Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p4.1)\.
- Chenet al\.\(2023\)C\. Chen, Y\. Hu, C\. H\. Yang, S\. M\. Siniscalchi, P\. Chen, and E\. ChngHyPoradise: an open baseline for generative speech recognition with large language models\.Advances in Neural Information Processing Systems36,pp\. 31665–31688\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p3.1),[§1](https://arxiv.org/html/2609.02940#S1.p2.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1),[§2](https://arxiv.org/html/2609.02940#S2.p2.1),[§3\.2](https://arxiv.org/html/2609.02940#S3.SS2.p1.1),[Table 2](https://arxiv.org/html/2609.02940#S5.T2.6.1.4.1),[§6\.1](https://arxiv.org/html/2609.02940#S6.SS1.p1.1)\.
- Chenet al\.\(2024a\)C\. Chen, R\. Li, Y\. Hu, S\. Siniscalchi, P\. Chen, E\. Chng, and C\. H\. YangIt’s never too late: fusing acoustic information into large language models for automatic speech recognition\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 437–453\.Cited by:[§4\.1](https://arxiv.org/html/2609.02940#S4.SS1.p1.1)\.
- Chenet al\.\(2021\)G\. Chen, S\. Chai, G\. Wang, J\. Du, W\. Zhang, C\. Weng, D\. Su, D\. Povey, J\. Trmal, J\. Zhang, M\. Jin, S\. Khudanpur, S\. Watanabe, S\. Zhao, W\. Zou, X\. Li, X\. Yao, Y\. Wang, Z\. You, and Z\. YanGigaSpeech: An Evolving, Multi\-Domain ASR Corpus with 10,000 Hours of Transcribed Audio\.InInterspeech 2021,pp\. 3670–3674\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1965),ISSN 2958\-1796Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Chenet al\.\(2025\)X\. Chen, A\. Zhao, H\. Xia, X\. Lu, H\. Wang, Y\. Chen, W\. Zhang, J\. Wang, W\. Li, and X\. ShenReasoning beyond language: a comprehensive survey on latent chain\-of\-thought reasoning\.arXiv preprint arXiv:2505\.16782\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p5.1),[§1](https://arxiv.org/html/2609.02940#S1.p4.1)\.
- Chenet al\.\(2024b\)Z\. Chen, H\. Huang, A\. Andrusenko, O\. Hrinchuk, K\. C\. Puvvada, J\. Li, S\. Ghosh, J\. Balam, and B\. GinsburgSALM: speech\-augmented language model with in\-context learning for speech recognition and translation\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 13521–13525\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p3.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- Chouet al\.\(2025\)C\. Chou, C\. Hsu, H\. Chung, L\. Tseng, H\. Cheng, Y\. Fu, K\. P\. Huang, and H\. LeeA self\-refining framework for enhancing ASR using TTS\-synthesized data\.In2025 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p5.1)\.
- Chuanget al\.\(2024\)Y\. Chuang, Y\. Xie, H\. Luo, Y\. Kim, J\. R\. Glass, and P\. HeDola: decoding by contrasting layers improves factuality in large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54158–54183\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p4.1)\.
- Del Rioet al\.\(2022\)M\. Del Rio, P\. Ha, Q\. McNamara, C\. Miller, and S\. ChandraEarnings\-22: a practical benchmark for accents in the wild\.arXiv preprint arXiv:2203\.15591\.Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Granite Team \(2024\)I\. Granite TeamGranite 3\.0 language models\.External Links:[Link](https://github.com/ibm-granite/granite-3.0-language-models/)Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p3.1)\.
- Grillet al\.\(2020\)J\. Grill, F\. Strub, F\. Altché, C\. Tallec, P\. Richemond, E\. Buchatskaya, C\. Doersch, B\. Avila Pires, Z\. Guo, M\. Gheshlaghi Azar,et al\.Bootstrap your own latent: a new approach to self\-supervised learning\.Advances in neural information processing systems33,pp\. 21271–21284\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p5.1)\.
- Hernandezet al\.\(2018\)F\. Hernandez, V\. Nguyen, S\. Ghannay, N\. Tomashenko, and Y\. EsteveTED\-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation\.InInternational conference on speech and computer,pp\. 198–208\.Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Horiet al\.\(2025\)T\. Hori, M\. Kocour, A\. Haider, E\. McDermott, and X\. ZhuangDelayed fusion: integrating large language models into first\-pass decoding in end\-to\-end speech recognition\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p4.1),[§1](https://arxiv.org/html/2609.02940#S1.p2.1)\.
- Hsuet al\.\(2025a\)C\. Hsu, Y\. Chen, F\. Liao, P\. Ho, Y\. Wang, P\. Hsu, and D\. ShiuLet’s fuse step by step: a generative fusion decoding algorithm with LLMs for robust and instruction\-aware ASR and OCR\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 24959–24973\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p4.1),[§1](https://arxiv.org/html/2609.02940#S1.p2.1),[§1](https://arxiv.org/html/2609.02940#S1.p5.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.02940#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.02940#S4.SS1.p1.2)\.
- Hsuet al\.\(2025b\)T\. Hsu, K\. Lu, C\. Chiang, and H\. LeeReducing object hallucination in large audio\-language models via audio\-aware decoding\.In2025 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1109/ASRU65441.2025.11434595)Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p4.1),[§1](https://arxiv.org/html/2609.02940#S1.p5.1)\.
- Jianget al\.\(2024\)T\. Jiang, S\. Huang, Z\. Luan, D\. Wang, and F\. ZhuangScaling sentence embeddings with large language models\.InFindings of the association for computational linguistics: EMNLP 2024,pp\. 3182–3196\.Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p2.1)\.
- Lesciet al\.\(2025\)P\. Lesci, C\. Meister, T\. Hofmann, A\. Vlachos, and T\. PimentelCausal estimation of tokenisation bias\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 28325–28340\.Cited by:[§3\.2](https://arxiv.org/html/2609.02940#S3.SS2.p6.1)\.
- Liet al\.\(2023\)X\. Li, S\. Takamichi, T\. Saeki, W\. Chen, S\. Shiota, and S\. WatanabeYODAS: youTube\-oriented dataset for audio and speech\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§3\.2](https://arxiv.org/html/2609.02940#S3.SS2.p1.1)\.
- Liaoet al\.\(2023\)F\. Liao, Y\. Chan, Y\. Chen, C\. Hsu, and D\. ShiuZero\-shot domain\-sensitive speech recognition with prompt\-conditioning fine\-tuning\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p2.1)\.
- Linet al\.\(2026\)T\. Lin, W\. Huang, Y\. Lin, and H\. LeeHow contrastive decoding enhances large audio language models?\.External Links:2603\.09232Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p5.1)\.
- Linet al\.\(2025\)Y\. Lin, Z\. Chen, P\. Zelasko, Z\. Wan, X\. Yang, Z\. Chen, K\. C\. Puvvada, K\. Hu, S\. Fu, J\. W\. Chiu, J\. Balam, B\. Ginsburg, Y\. F\. Wang, and C\. H\. YangNeKo: cross\-modality post\-recognition error correction with tasks\-guided mixture\-of\-experts language model\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),G\. Rehm and Y\. Li \(Eds\.\),Vienna, Austria,pp\. 222–236\.External Links:[Link](https://aclanthology.org/2025.acl-industry.17/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.17)Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p3.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1),[Table 2](https://arxiv.org/html/2609.02940#S5.T2.6.1.5.1),[§6\.1](https://arxiv.org/html/2609.02940#S6.SS1.p1.1)\.
- Liuet al\.\(2021\)L\. Liu, Y\. Gu, A\. Gourav, A\. Gandhe, S\. Kalmane, D\. Filimonov, A\. Rastrow, and I\. BulykoDomain\-aware neural language models for speech recognition\.InICASSP 2021\-2021 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 7373–7377\.Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p2.1)\.
- Liuet al\.\(2024\)Z\. Liu, S\. Kim, and O\. KalinliEvaluating speech recognition performance towards large language model based voice assistants\.\.InINTERSPEECH,Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p2.1)\.
- Maet al\.\(2023\)R\. Ma, M\. J\. Gales, K\. M\. Knill, and M\. QianN\-best T5: robust ASR error correction using multiple input hypotheses and constrained decoding space\.InProc\. Interspeech 2023,pp\. 3267–3271\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p1.1)\.
- Mittalet al\.\(2024\)A\. Mittal, D\. Prabhu, S\. Sarawagi, and P\. JyothiSALSA: speedy ASR\-LLM synchronous aggregation\.InAnnual Conference of the International Speech Communication Association,Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p4.1),[§1](https://arxiv.org/html/2609.02940#S1.p2.1)\.
- NVIDIA \(2025\)NVIDIACanary\-Qwen\-2\.5B\.Note:[https://huggingface\.co/nvidia/canary\-qwen\-2\.5b](https://huggingface.co/nvidia/canary-qwen-2.5b)Hugging Face model cardCited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p2.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- OpenAI \(2023\)OpenAIWhisper large\-v3\.Note:[https://huggingface\.co/openai/whisper\-large\-v3](https://huggingface.co/openai/whisper-large-v3)Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p1.1),[§1](https://arxiv.org/html/2609.02940#S1.p1.1)\.
- O’Neillet al\.\(2021\)P\. K\. O’Neill, V\. Lavrukhin, S\. Majumdar, V\. Noroozi, Y\. Zhang, O\. Kuchaiev, J\. Balam, Y\. Dovzhenko, K\. Freyberg, M\. D\. Shulman, B\. Ginsburg, S\. Watanabe, and G\. KucskoSPGISpeech: 5,000 Hours of Transcribed Financial Audio for Fully Formatted End\-to\-End Speech Recognition\.InInterspeech 2021,pp\. 1434–1438\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1860),ISSN 2958\-1796Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Panayotovet al\.\(2015\)V\. Panayotov, G\. Chen, D\. Povey, and S\. KhudanpurLibriSpeech: an ASR corpus based on public domain audio books\.In2015 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 5206–5210\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP.2015.7178964)Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Penget al\.\(2025\)Y\. Peng, M\. Shakeel, Y\. Sudo, W\. Chen, J\. Tian, C\. Lin, and S\. WatanabeOWSM v4: Improving Open Whisper\-Style Speech Models via Data Scaling and Cleaning\.InInterspeech 2025,pp\. 2225–2229\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p1.1)\.
- Penget al\.\(2024\)Y\. Peng, J\. Tian, W\. Chen, S\. Arora, B\. Yan, Y\. Sudo, M\. Shakeel, K\. Choi, J\. Shi, X\. Chang,et al\.OWSM v3\.1: better and faster open Whisper\-style speech models based on e\-branchformer\.InProc\. Interspeech 2024,pp\. 352–356\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p1.1)\.
- Penget al\.\(2023\)Y\. Peng, J\. Tian, B\. Yan, D\. Berrebbi, X\. Chang, X\. Li, J\. Shi, S\. Arora, W\. Chen, R\. Sharma,et al\.Reproducing whisper\-style training using an open\-source toolkit and publicly available data\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p1.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- Peterset al\.\(2018\)M\. E\. Peters, M\. Neumann, M\. Iyyer, M\. Gardner, C\. Clark, K\. Lee, and L\. ZettlemoyerDeep contextualized word representations\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),pp\. 2227–2237\.External Links:[Document](https://dx.doi.org/10.18653/v1/N18-1202)Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p4.1)\.
- Puvvadaet al\.\(2024\)K\. C\. Puvvada, P\. Żelasko, H\. Huang, O\. Hrinchuk, N\. R\. Koluguri, K\. Dhawan, S\. Majumdar, E\. Rastorgueva, Z\. Chen, V\. Lavrukhin,et al\.Less is more: accurate speech recognition & translation without web\-scale data\.InProc\. Interspeech 2024,pp\. 3964–3968\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p1.1),[§1](https://arxiv.org/html/2609.02940#S1.p1.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- Radfordet al\.\(2023\)A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. SutskeverRobust speech recognition via large\-scale weak supervision\.InInternational conference on machine learning,pp\. 28492–28518\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p1.1),[§1](https://arxiv.org/html/2609.02940#S1.p1.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- Radhakrishnanet al\.\(2023\)S\. Radhakrishnan, C\. H\. Yang, S\. A\. Khan, R\. Kumar, N\. A\. Kiani, D\. Gomez\-Cabrero, and J\. N\. TegnerWhispering LLaMA: a cross\-modal generative error correction framework for speech recognition\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 10007–10016\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p3.1),[§1](https://arxiv.org/html/2609.02940#S1.p2.1)\.
- Saonet al\.\(2025\)G\. Saon, A\. Dekel, A\. Brooks, T\. Nagano, A\. Daniels, A\. Satt, A\. Mittal, B\. Kingsbury, D\. Haws, E\. Morais,et al\.Granite\-Speech: open\-source speech\-aware LLMs with strong English ASR capabilities\.arXiv preprint arXiv:2505\.08699\.Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- Shiet al\.\(2026\)X\. Shi, X\. Wang, Z\. Guo, Y\. Wang, P\. Zhang, X\. Zhang, Z\. Guo, H\. Hao, Y\. Xi, B\. Yang,et al\.Qwen3\-ASR technical report\.arXiv preprint arXiv:2601\.21337\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p3.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1)\.
- Shoeybiet al\.\(2019\)M\. Shoeybi, M\. Patwary, R\. Puri, P\. LeGresley, J\. Casper, and B\. CatanzaroMegatron\-LM: training multi\-billion parameter language models using model parallelism\.arXiv preprint arXiv:1909\.08053\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p3.1)\.
- Snellet al\.\(2024\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p5.1)\.
- Srivastavet al\.\(2025\)V\. Srivastav, S\. Zheng, E\. Bezzam, E\. L\. Bihan, N\. Koluguri, P\. Żelasko, S\. Majumdar, A\. Moumen, and S\. GandhiOpen ASR Leaderboard: towards reproducible and transparent multilingual and long\-form speech recognition evaluation\.External Links:2510\.06961,[Link](https://arxiv.org/abs/2510.06961)Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p3.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1),[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Wanet al\.\(2026\)Z\. Wan, C\. H\. Yang, J\. Tian, H\. Ye, A\. Pasad, S\. Fu, A\. Goel, R\. Hachiuma, S\. Diao, K\. Dhawan,et al\.Speech\-Hands: a self\-reflection voice agentic approach to speech recognition and audio reasoning with omni perception\.arXiv preprint arXiv:2601\.09413\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p2.1),[§2](https://arxiv.org/html/2609.02940#S2.p1.1),[Table 2](https://arxiv.org/html/2609.02940#S5.T2.6.1.6.1),[§6\.1](https://arxiv.org/html/2609.02940#S6.SS1.p1.1)\.
- Wanet al\.\(2025\)Z\. Wan, C\. H\. Yang, Y\. Yu, J\. Tian, S\. Li, K\. Hu, Z\. Chen, S\. Watanabe, F\. Cheng, C\. Chu,et al\.SpeechIQ: speech\-agentic intelligence quotient across cognitive levels in voice understanding by large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 30381–30398\.Cited by:[§2](https://arxiv.org/html/2609.02940#S2.p2.1),[§6\.3](https://arxiv.org/html/2609.02940#S6.SS3.p2.1)\.
- Wanget al\.\(2021\)C\. Wang, M\. Riviere, A\. Lee, A\. Wu, C\. Talnikar, D\. Haziza, M\. Williamson, J\. Pino, and E\. DupouxVoxPopuli: a large\-scale multilingual speech corpus for representation learning, semi\-supervised learning and interpretation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 993–1003\.Cited by:[§5](https://arxiv.org/html/2609.02940#S5.p2.1)\.
- Wanget al\.\(2025\)Y\. Wang, P\. Zhang, B\. Yang, D\. F\. Wong, and R\. WangLatent space chain\-of\-embedding enables output\-free LLM self\-evaluation\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p5.1),[§1](https://arxiv.org/html/2609.02940#S1.p4.1),[§3\.1](https://arxiv.org/html/2609.02940#S3.SS1.p7.1)\.
- Yanget al\.\(2023\)C\. H\. Yang, Y\. Gu, Y\. Liu, S\. Ghosh, I\. Bulyko, and A\. StolckeGenerative speech recognition error correction with large language models and task\-activating prompting\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p2.1)\.
- Yanget al\.\(2025\)C\. H\. Yang, A\. Stolcke, and L\. P\. HeckSpoken conversational agents with large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts,pp\. 7–8\.Cited by:[Appendix A](https://arxiv.org/html/2609.02940#A1.p3.1)\.
- Yanget al\.\(2026\)Z\. Yang, M\. Zhang, F\. Chen, G\. Ding, L\. Hou, X\. Tao, and Y\. ChenLess is more: improving LLM reasoning with minimal test\-time intervention\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 20124–20137\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.921)Cited by:[§1](https://arxiv.org/html/2609.02940#S1.p4.1)\.

## Appendix AExtended Related Work

LLM\-based ASR\.Recent advances in speech foundation models have demonstrated strong generalization and multi\-task capabilities, including multilingual ASR and speech translation, by scaling training data to hundreds of thousands or even millions of hours\. A notable example is Whisper, an encoder–decoder Transformer trained on 680,000 hours of weakly labeled speech data that achieves robust performance across diverse domains and languages[Radford et al\. \(2023\)](https://arxiv.org/html/2609.02940#bib.bib27)\. Later versions further scale the training corpus to 5 million hours[OpenAI \(2023\)](https://arxiv.org/html/2609.02940#bib.bib37)\. Following this direction, the OWSM series\([Peng et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib39)\)introduces fully transparent and reproducible speech foundation models trained on large\-scale open speech datasets while achieving competitive performance with proprietary systems\. Similarly, Canary\([Puvvada et al\., 2024](https://arxiv.org/html/2609.02940#bib.bib28)\)introduces a multilingual ASR and speech translation model based on a FastConformer encoder–decoder architecture that achieves competitive performance while being trained on significantly less data\.

To further improve performance by incorporating the rich linguistic knowledge encoded in LLMs, recent work has explored integrating pretrained speech encoders with LLMs\. These multimodal LLMs are trained on large\-scale speech–text corpora, enabling them to perform various speech\-related tasks while leveraging the semantic capabilities of LLMs\. To preserve the pretrained knowledge of LLMs while maintaining training efficiency, these models often adopt parameter\-efficient adaptation techniques such as Low\-Rank Adaptation \(LoRA\)\. For example, Canary\-Qwen\-2\.5B[NVIDIA \(2025\)](https://arxiv.org/html/2609.02940#bib.bib13)combines the FastConformer speech encoder from the pretrained Canary model with a Qwen\-2\.5B LLM decoder and significantly improves ASR performance compared to the original Canary model\. Similarly, Phi\-4\-Multimodal\([Abouelenin et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib31)\)integrates speech, vision, and text processing within a shared LLM\-based architecture using a Mixture\-of\-LoRAs design, enabling strong ASR performance alongside multimodal reasoning capabilities\. In this work, we adopt Phi\-4\-Multimodal as the ASR–LLM backbone and investigate how its internal semantic representations can be more effectively leveraged to improve ASR performance\.

Generative Error Correction and Agentic Setups\.To leverage the rich semantic and world knowledge encoded in LLMs, GER methods typically operate in a two\-pass pipeline, where an LLM refines the N\-best hypotheses produced by a separate ASR system\. For example, HyPoradise\([Chen et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib22)\)establishes a benchmark for using LLMs to directly predict ground\-truth transcriptions fromNN\-best hypotheses, and Whispering LLaMA\([Radhakrishnan et al\., 2023](https://arxiv.org/html/2609.02940#bib.bib6)\)integrates Whisper\-based acoustic representations with an LLaMA decoder to incorporate both acoustic and linguistic information\. More recently, NeKo\([Lin et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib38)\)explores cross\-modal post\-recognition correction using a mixture\-of\-experts architecture\. However, these agentic approaches\([Yang et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib1)\)rely on a sequential two\-pass inference process, where the ASR model first generates candidate hypotheses and the LLM subsequently performs correction, introducing additional latency\.

To address this limitation, recent work has explored single\-pass, synchronous decoding frameworks that integrate LLMs more tightly into the search process\. For instance, Generative Fusion Decoding\([Hsu et al\., 2025a](https://arxiv.org/html/2609.02940#bib.bib25)\)employs a byte\-level shallow fusion to bridge mismatched token spaces between ASR and LLM decoders without requiring re\-training\. Similarly, SALSA\([Mittal et al\., 2024](https://arxiv.org/html/2609.02940#bib.bib23)\)couples the hidden states of an ASR decoder to an LLM via a trained projection layer for synchronous hypothesis advancement\. To address computational costs, delayed fusion\([Hori et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib24)\)applies LLM scores to ASR hypotheses only after a pruning stage, significantly reducing the number of required LLM inference calls\. While these methods reduce the latency of two\-pass GER pipelines, they still require parallel inference from two separate ASR and LLM models\. In contrast, our proposed approach operates within a unified ASR\-LLM architecture, directly leveraging the internal hidden states of the LLM backbone during a single forward pass to improve ASR performance with relatively small computational overhead\.

Analysis of LLM Hidden States\.With billions of parameters, LLMs learn internal representations that encode rich semantic and functional properties that are closely related to their generation behavior\. Recent studies attempt to analyze these hidden representations to better understand how LLMs perform reasoning and decision\-making\. Specifically, the latent chain\-of\-thought \(CoT\) reasoning suggests that LLMs may perform intermediate inferential computations within their high\-dimensional hidden states, even when no explicit natural\-language reasoning is produced\([Chen et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib33)\)\. Moreover, the Chain\-of\-Embedding \(CoE\) framework\([Wang et al\., 2025](https://arxiv.org/html/2609.02940#bib.bib34)\)demonstrates that the sequence of hidden states across layers can be interpreted as a latent reasoning trajectory\. By analyzing geometric properties of these trajectories, such as magnitude changes and angular differences between layer\-wise embeddings, it becomes possible to estimate prediction correctness or detect semantic uncertainty without requiring ground\-truth supervision\.

Algorithm 1Hybrid Search1:Audio input

AA, ASR\-LLM

pθp\_\{\\theta\}, target detector

𝒯⁡\(⋅\)\\mathcal\{T\}\(\\cdot\), beam size

BB
2:Final decoded sequence

yy
3:

y←∅y\\leftarrow\\emptyset;

ℬ←\{y\}\\mathcal\{B\}\\leftarrow\\\{y\\\};

phase←Greedy\\text\{phase\}\\leftarrow\\textsc\{Greedy\};

K←1K\\leftarrow 1⊳\\trianglerightKK: effective beam size

4:whilenot end\-of\-sequencedo

5:\[Search\]

6:Advance one decoding step withKK

7:

y←y\\leftarrowbest beam in

ℬ\\mathcal\{B\};

yt←y\_\{t\}\\leftarrowlast token in

yy
8:\[State transition\]

9:if

phase=Greedy\\text\{phase\}=\\textsc\{Greedy\}then

10:if

𝒯⁡\(yt\)\\mathcal\{T\}\(y\_\{t\}\)then

11:

K←BK\\leftarrow B;

ℬ←\{y\}B\\mathcal\{B\}\\leftarrow\\\{y\\\}^\{B\};

phase←Beam\\text\{phase\}\\leftarrow\\textsc\{Beam\}
12:endif

13:elseif

phase=Beam\\text\{phase\}=\\textsc\{Beam\}then

14:if

∃y′∈⋂b∈ℬb​s\.t\.​¬𝒯⁡\(y′\)\\exists\\,y^\{\\prime\}\\in\\bigcap\_\{b\\in\\mathcal\{B\}\}b\\;\\text\{s\.t\.\}\\;\\neg\\mathcal\{T\}\(y^\{\\prime\}\)then

15:Roll back each beam to the position ending at

y′y^\{\\prime\}
16:

y←y\\leftarrowbest rolled\-back beam in

ℬ\\mathcal\{B\};

ℬ←\{y\}\\mathcal\{B\}\\leftarrow\\\{y\\\};

K←1K\\leftarrow 1;

phase←Greedy\\text\{phase\}\\leftarrow\\textsc\{Greedy\}
17:endif

18:endif

19:endwhile

20:return

yy

## Appendix BDetailed Experimental Setting

### B\.1Late Layer Feature Discovery

In Figure[2](https://arxiv.org/html/2609.02940#S3.F2), we isolate the𝐀𝐧𝐠𝐥𝐞𝐅𝐞𝐚𝐭\\mathbf\{AngleFeat\}and𝐌𝐚𝐠𝐅𝐞𝐚𝐭\\mathbf\{MagFeat\}from the hidden state before the final layer \(layer 31\)\. For named\-entity tokens, the overall distribution shifts toward greater deviation \(lower cosine, higher norm\)\. In particular, this phenomenon is most pronounced for starting tokens of multi\-token words\. We attribute this behavior to uncertainty in the ASR\-LLM’s final token selection among acoustically similar candidates\. Multi\-token named entities may contain spelling variations that require longer context to resolve, making an immediate decision uncertain \(“Carol \-yn” vs\. “Kar \-ol \-yn”\)\. Constructing an embedding space that supports a more balanced probability mixture over competing candidates requires a larger departure from the original LLM embedding space, as that space is primarily organized around semantic relationships and, therefore, tends to underemphasize acoustic similarity\.

### B\.2Hybrid Search Algorithm

Algorithm[1](https://arxiv.org/html/2609.02940#alg1)presents the Hybrid Search Algorithm\. The algorithm switches between greedy search states and beam search states\. We use the term “Advance one decoding step” to convey both stepping forward in beam search and greedy search depending on the current state of the algorithm\.

![Refer to caption](https://arxiv.org/html/2609.02940v1/figs2/EXP10_GRANITE_-_nonner_-_ner_-_gaussian-with-decision-no-space-only.png)Figure 5:Named\-Entity Distribution Shifts of Granite\-4\.0\-1B\-Speech

## Appendix CAdditional Results and Analysis

### C\.1Generalization of targeted approaches

We show similar distribution shifts inAngleFeati​n​t​e​r\\mathrm\{AngleFeat\}\_\{inter\}andMagFeati​n​t​e​r\\mathrm\{MagFeat\}\_\{inter\}forGranite\-4\.0\-1B\-Speechin Figure[5](https://arxiv.org/html/2609.02940#A2.F5)\.

### C\.2Generalization of targeted region

In Figure[6](https://arxiv.org/html/2609.02940#A3.F6), we highlight the targeted region derived fromYODAS, and show how it generalizes to other datasets\. Applying the region to other datasets in the Open ASR Leaderboard, we observe consistent concentration effects in named\-entity tokens\.

YODAS![Refer to caption](https://arxiv.org/html/2609.02940v1/figs2/EXP10_-_nonner_-_ner_-_gaussian-with-decision-no-space-only.png)⇓\\DownarrowAMICommon VoiceEarnings22![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/0.png)![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/1.png)![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/2.png)GigaSpeechLibriSpeechSPGISpeech![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/3.png)![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/4.png)![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/5.png)TED\-LIUMVoxPopuli![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/6.png)![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/7.png)Figure 6:Generalization of the targeted region derived from theYODASdataset\. Most datasets exhibit a similar distribution shift\. Under the targeted region selection derived from theYODASdataset, named entities are effectively concentrated\.
### C\.3Case Examples of Correction

We list the first few chronological instances from each of the datasets in which Hybrid Search introduces edit, and separate into positive and negative edit categories\. Significant edits mostly involve named entities, while changes to acoustically dominant tokens that risk introducing hallucinations are relatively uncommon\.

\[Common Voice\]Beam Search: assemblyman james silva and former state senatorventranboy scouts of americaHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: assemblyman james silva and former state senatorvan tranboy scouts of americaGold: assemblyman james silva and former state senatorvan tranboy scouts of americaBeam Search: in this performance she sang with examplesigfriedjerusalem and john tomlinsonHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: in this performance she sang with examplesiegfriedjerusalem and john tomlinsonGold: in this performance she sang with examplesiegfriedjerusalem and john tomlinsonBeam Search: it starred hawks jack dee doon mackichan ben miller alistairmcgregorand charlotte pageHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: it starred hawks jack dee doon mackichan ben miller alistairmcgowanand charlotte pageGold: it starred hawks jack dee doon mackichan ben miller alistairmcgowanand charlotte page

\[AMI\]Correct Hybrid editsBeam Search: and i think we should show this prototype to people from various age andsocioeconomicgroups and see about any fine tuning that maybe little things that we have not thought ofHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: and i think we should show this prototype to people from various age andsocio economicgroups and see about any fine tuning that maybe little things that we have not thought ofGold: and i think we should show this prototype to people from various age andsocio economicgroups and see about any fine tuning that maybe little things we have not thought ofBeam Search: the design should minimizeours i and be easy to locate and we were still a slightly ambivalent as to whether to use voice recognition thereHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: the design should minimizers i and be easy to locate and we were still a slightly ambivalent as to whether to use voice recognition thereGold: the design should minimizers i and be easy to locate and we were still slightly ambivalent as to whether to use voice recognition thereBeam Search: it does does an n x t search for the string idunnotranscript or text or whateverHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: it does does an n x t search for the string ido not knowtranscript or text or whateverGold: it does does an n x t search for the string ido not knowtranscript or text or whateverIncorrect Hybrid editsBeam Search: now these features includes the s signal emitting signal it is the m led or l e d theinfraredHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: now these features includes the s signal emitting signal it is the m led or l e d theinfra redGold: now these features includes the s signal emitting signal it is the led or l e d theinfraredBeam Search: actually for for anntext area whatever it is they they define some ha handy highlights on thatHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: actually for for anintext area whatever it is they they define some ha handy highlights on thatGold: actually for for anntext area whatever it is they they defined some ha handy highlights on thatBeam Search: cause in our earlier market research if you would allow me to go to theflat boardsmart boardHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: cause in our earlier market research if you would allow me to go to thewhiteboardsmart boardGold: cause in our earlier market research if you would allow me to go to theflat boardsmartboard

\[Common Voice\]Correct Hybrid editsBeam Search: it was the time of day when all ofspenceslept during the summerHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: it was the time of day when all ofspainslept during the summerGold: it was the time of day when all ofspainslept during the summerBeam Search: due to her symbolization of compassion in east asiaguan yinis associated with vegetarianismHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: due to her symbolization of compassion in east asiaguanyinis associated with vegetarianismGold: due to her symbolization of compassion in east asiaguanyinis associated with vegetarianismBeam Search: from the vietnam era and to presentpurposesbuilt ceremonial knives evolvedHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: from the vietnam era and to presentpurposebuilt ceremonial knives evolvedGold: from the vietnam era and to presentpurposebuilt survival knives evolvedIncorrect Hybrid editsBeam Search: the ivyclimbedup the building and wrapped itself around the chimneyHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: the ivyclamberedup the building and wrapped itself around the chimneyGold: the ivyclimbedup the building and wrapped itself around the chimneyBeam Search: 2 theories exist as to howlocoacquired its nameHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: 2 theories exist as to howlocarnoacquired its nameGold: 2 theories exist as to howlocoacquired its nameBeam Search: after leaving ford shinoda and knudsen co createdrectranswhich built recreational vehiclesHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: after leaving ford shinoda and knudsen co createdrekchampswhich built recreational vehiclesGold: after leaving ford shinoda and knudsen co createdrectranswhich built recreational vehicles

\[Earnings\-22\]Correct Hybrid editsBeam Search: if we look into take this forward then what you can expect is that we you know in the in 22 we will be able as we recover from covid although we still have some weakness in you know the end of january and in january as we speak we will be able to get to a better sustainable commercial rate but not at the level of q 4 not the high level of q 4 because that is a seasonality and not at the low levels we have seen earlier in in 21 where we were very much focused on focused on ebitda in the midst of covidHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: if we look into take this forward then what you can expect is that we you know in theinin 22 we will be able as we recover from covid although we still have some weakness in you know the end of january and in january as we speak we will be able to get to a better sustainable commercial rate but not at the level of q 4 not the high level of q 4 because that is a seasonality and not at the low levels we have seen earlier in in 21 where we were very much focused on focused on ebitda in the midst of covidGold: if we look into take this forward then what you can expect is that we you knowinin 22 we will be able as we recover from covid although we still have some weakness in you know the end of january and in january as we speak we will be able to get to a better sustainable commercial rate but not at the level of q 4 not the high level of q 4 because that is a seasonality and not at the low levels we have seen earlier in in 2021 where we were very much focused on focused on ebitda in the midst of covidBeam Search: good morning i will take the 1st part of your question in relation to what part of the growth on health and nutrition was sort of underlying growth versus a oneoffsand then i will pass it on to lisa in relation to the infant formula in market potentialHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: good morning i will take the 1st part of your question in relation to what part of the growth on health and nutrition was sort of underlying growth versus a oneoffand then i will pass it on to lisa in relation to the infant formula in market potentialGold: i will take the 1st part of your question in relation to what part of the growth on health and nutrition was sort of underlying growth vs a oneoffand then i will pass it on to lisa in relation to the infant formula in market potentialBeam Search: on the marketplace i focus on sweden and in regards to everything spoken about before the positioning in networks and your 5 g position and the jv with\.\.Hybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: on the marketplace i focus on sweden and in regards to everything spoken about before the positioning in networks and your 5 g position and the jv withtelenorGold: on the marketplace i focus on sweden and in regard to everything spoken about before the positioning in networks and your 5 g position and the jv withtelenorIncorrect Hybrid editsBeam Search: so so given that in q 2 and q 4 last yearhmoswere 6 and 7000000 in revenue respectively i just wonder if sort of the underlying run rate forhmoswas in fact weak given that you you you have this big product launch from from abbottHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: so so given that in q 2 and q 4 last yearhmo iswere 6 and 7000000 in revenue respectively i just wonder if sort of the underlying run rate forhmo iswas in fact weak given that you you you have this big product launch from from abbottGold: so so given that in q 2 and q 4 last yearhmoswere 6 and 7000000 in revenue respectively i just wonder if sort of the underlying run rate forhmoswas in fact weak given that you you you have this big product launch fro from abbottBeam Search: excluding extraordinary charges comparable adjusted ebitda near breakeven reaching a loss of3600000improving from losses of 10500000 in the prior quarter and nearly 17000000 in 3rd quarter 2020Hybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: excluding extraordinary charges comparable adjusted ebitda near breakeven reaching a loss of$3600000improving from losses of 10500000 in the prior quarter and nearly 17000000 in 3rd quarter 2020Gold: excluding extraordinary charges comparable adjusted ebitda neared break even reaching a loss of3600000improving from losses of 10 and a half 1000000 in the 3rd quarter and nearly 17000000 in 3rd quarter 2020Beam Search: while this resulted in a lower use of cash of close to 10000000 in 2nd quarter21with better travel conditions in this past quarter we saw a pickup in demand with customers redeeming travel vouchersHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: while this resulted in a lower use of cash of close to 10000000 in 2nd quarter2021with better travel conditions in this past quarter we saw a pickup in demand with customers redeeming travel vouchersGold: while this resulted in a lower use of cash of close to 10000000 in 2nd quarter21with better travel conditions in this past quarter we saw a pick up in demand with customers redeeming travel vouchers

\[GigaSpeech\]Correct Hybrid editsBeam Search: this is heavy metals from 30 for 30 podcasts i am alyssa rohnick gymnastics has been part of my life in some form for as long as i can remember from watching marylou on tv at the 1984 olympics when i was 7 to later being a teenage gymnast obsessing over kimzmesquelis tumbling passesHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: this is heavy metals from 30 for 30 podcasts i am alyssa rohnick gymnastics has been part of my life in some form for as long as i can remember from watching marylou on tv at the 1984 olympics when i was 7 to later being a teenage gymnast obsessing over kimzmeskalis tumbling passesGold: this is heavy medals from 30 for 30 podcasts i am alyssa roenigk gymnastics has been part of my life in some form for as long as i can remember from watching mary lou on t v at the 1984 olympics when i was 7 too later being a teenage gymnast obsessing over kimzmeskalis tumbling passesBeam Search: when i covered my 1st summer olympic games in beijing in 2008 china was accused of having an underage gymnast on the team and i was writing about it i will never forget belakarolyshouting into my digital recorder about how the chinese coaches had just stolen his playbookHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: when i covered my 1st summer olympic games in beijing in 2008 china was accused of having an underage gymnast on the team and i was writing about it i will never forget belakarolyishouting into my digital recorder about how the chinese coaches had just stolen his playbookGold: when i covered my 1st summer olympic games in beijing in 2008 china was accused of having an underaged gymnast on the team and i was writing about it i will never forget belakarolyishouting into my digital recorder about how the chinese coaches had just stolen his playbookBeam Search: and of course most of these critics were pop writers who were only aware of jeannie creillybecause of the pop success of harbor valley pta and whose only other frame of reference for country music in the 60 s and 70 s was probably johnny cashHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: and of course most of these critics were pop writers who were only aware of jeannie crileybecause of the pop success of harbor valley pta and whose only other frame of reference for country music in the 60 s and 70 s was probably johnny cashGold: and of course most of these critics were pop writers who were only aware of jeannie crileybecause of the pop success of harper valley p t a and whose only other frame of reference for country music in the 60s and 70s was probably johnny cashIncorrect Hybrid editsBeam Search: it is 10439493 people watching now simply better showed up and all it is like this is a red carpet event we got all the stars here tonight kg tropical simply better friday fish facts bobcaylorHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: it is 10439493 people watching now simply better showed up and all it is like this is a red carpet event we got all the stars here tonight kg tropical simply better friday fish facts bobkellerGold: it is 10439493 people watching now simply brtta showed up and all that it is like this is a red carpet of me we got all the stars here tonight k g tropical simply betta friday fish facts bobcaylorBeam Search: so that meant that tweens in the philippines or romania or macau could all rock a boxyshirredflounce cuff top or leopard drawstring lounge shorts for less than the price of a movie ticketHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: so that meant that tweens in the philippines or romania or macau could all rock a boxyshoulderedflounce cuff top or leopard drawstring lounge shorts for less than the price of a movie ticketGold: so that meant that tweens in the philippines or romania or macao could all rock a boxyshirredflounce cuff top or leopard drawstring lounge shorts for less than the price of a movie ticketBeam Search: blockbuster andwalmarthowever were a bit late to the party hastings admitted that if they had started 2 years earlier they probably would have won on an end of year conference call hastings declared blockbuster has thrown everything but the kitchen sink at usHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: blockbuster andwal marthowever were a bit late to the party hastings admitted that if they had started 2 years earlier they probably would have won on an end of year conference call hastings declared blockbuster has thrown everything but the kitchen sink at usGold: blockbuster andwalmarthowever were a bit late to the party hastings admitted that if they had started 2 years earlier they probably would have won on an end of year conference call hastings declared blockbuster has thrown everything but the kitchen sink at us

\[LibriSpeech\]Correct Hybrid editsBeam Search: the egyptian obeyed and his master crossed the wide space strewn with sand and approached the stage which had been erected for thefeudalperformances even had his eyes retained the power of sight his blood was coursing so wildly through his veins that he might perhaps have been unable to distinguish the statues around him and the 1000s of spectators who crowded closely together richly garlanded their cheeks glowing with enthusiasm surrounded the arena hermonHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: the egyptian obeyed and his master crossed the wide space strewn with sand and approached the stage which had been erected for thefestalperformances even had his eyes retained the power of sight his blood was coursing so wildly through his veins that he might perhaps have been unable to distinguish the statues around him and the 1000s of spectators who crowded closely together richly garlanded their cheeks glowing with enthusiasm surrounded the arena hermonGold: the egyptian obeyed and his master crossed the wide space strewn with sand and approached the stage which had been erected for thefestalperformances even had his eyes retained the power of sight his blood was coursing so wildly through his veins that he might perhaps have been unable to distinguish the statues around him and the 1000s of spectators who crowded closely together richly garlanded their cheeks glowing with enthusiasm surrounded the arena hermonBeam Search: sandro got up with pain enough in his bones and went after the innkeeper in the dark and meeting the officer who was looking to see what had become of his enemy he said to himseorwhoever you are do us the favor and kindness to give us a little rosemary oil salt and wine for it is water to cure one of our best knights errant on earth who lies on yonder bed wounded by the hands of the enchanted moor that is in this innHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: sandro got up with pain enough in his bones and went after the innkeeper in the dark and meeting the officer who was looking to see what had become of his enemy he said to himsenorwhoever you are do us the favor and kindness to give us a little rosemary oil salt and wine for it is water to cure one of our best knights errant on earth who lies on yonder bed wounded by the hands of the enchanted moor that is in this innGold: sancho got up with pain enough in his bones and went after the innkeeper in the dark and meeting the officer who was looking to see what had become of his enemy he said to himsenorwhoever you are do us the favor and kindness to give us a little rosemary oil salt and wine for it is wanted to cure one of the best knights errant on earth who lies on yonder bed wounded by the hands of the enchanted moor that is in this innBeam Search: true an interesting conversation still had power to charm him but often during its continuance the full consciousness of his misfortune forced itself upon his mind for the majority of the subjects discussed by the artists came to them through the medium of sight and referred to new creations of architecture sculpture and painting from whose enjoyment his blindnessdebaredhimHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: true an interesting conversation still had power to charm him but often during its continuance the full consciousness of his misfortune forced itself upon his mind for the majority of the subjects discussed by the artists came to them through the medium of sight and referred to new creations of architecture sculpture and painting from whose enjoyment his blindnessdebarredhimGold: true an interesting conversation still had power to charm him but often during its continuance the full consciousness of his misfortune forced itself upon his mind for the majority of the subjects discussed by the artists came to them through the medium of sight and referred to new creations of architecture sculpture and painting from whose enjoyment his blindnessdebarredhimIncorrect Hybrid editsBeam Search: he forgot as sergey ivanovitch explained to him afterwards this syllogism that it was necessary for the public good to get rid of the marshal of the province that to get rid of the marshal it was necessary to have a majority of votes that to get a majority of votes it was necessary to secureflerovis right to vote that to secure the recognition offlerovis right to vote they must decide on the interpretation to be put on the actHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: he forgot as sergey ivanovitch explained to him afterwards this syllogism that it was necessary for the public good to get rid of the marshal of the province that to get rid of the marshal it was necessary to have a majority of votes that to get a majority of votes it was necessary to securefleurovis right to vote that to secure the recognition offleurovis right to vote they must decide on the interpretation to be put on the actGold: he forgot as sergey ivanovitch explained to him afterwards this syllogism that it was necessary for the public good to get rid of the marshal of the province that to get rid of the marshal it was necessary to have a majority of votes that to get a majority of votes it was necessary to secureflerovis right to vote that to secure the recognition offlerovis right to vote they must decide on the interpretation to be put on the actBeam Search: why before he looked like the orneryest old rip that ever was but now when he would take off his new white beaver and make a bow and do a smile he looked that grand and good and pious that you would say he had walked right out of the ark and maybe was oldleviticushimselfHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: why before he looked like the orneryest old rip that ever was but now when he would take off his new white beaver and make a bow and do a smile he looked that grand and good and pious that you would say he had walked right out of the ark and maybe was oldleviathushimselfGold: why before he looked like the orneriest old rip that ever was but now when he would take off his new white beaver and make a bow and do a smile he looked that grand and good and pious that you would say he had walked right out of the ark and maybe was oldleviticushimselfBeam Search: she was perfectly safe after writing to basilransomand indeed it was difficult to see what he could have done to her except thank her he was only exceptionally superlative for her letter and assure her that he would come and see her the 1st time his business he was beginning to get a little should take him to bostonHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: she was perfectly safe after writing to basilransomeand indeed it was difficult to see what he could have done to her except thank her he was only exceptionally superlative for her letter and assure her that he would come and see her the 1st time his business he was beginning to get a little should take him to bostonGold: she was perfectly safe after writing to basilransomand indeed it was difficult to see what he could have done to her except thank her he was only exceptionally superlative for her letter and assure her that he would come and see her the 1st time his business he was beginning to get a little should take him to boston

\[SPGISpeech\]Correct Hybrid editsBeam Search: i think what we have found over amulti yearperiod here is that we often come across great ideas that neither provide the liquidity nor the time horizon that allow for us to be able to make it much of an investment in some of the larger pooled assets that we runHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: i think what we have found over amultiyearperiod here is that we often come across great ideas that neither provide the liquidity nor the time horizon that allow for us to be able to make it much of an investment in some of the larger pooled assets that we runGold: i think what we have found over amultiyearperiod here is that we often come across great ideas that neither provide the liquidity nor the time horizon that allow for us to be able to make it much of an investment in some of the larger pooled assets that we runBeam Search:actuarialresults may differ materially from the results expressed or implied in these statements as a result of risks uncertainties and other factors included but not limited to the factors set forth in the company is filings with the securities and exchange commissionHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}:actualresults may differ materially from the results expressed or implied in these statements as a result of risks uncertainties and other factors included but not limited to the factors set forth in the company is filings with the securities and exchange commissionGold:actualresults may differ materially from the results expressed or implied in these statements as a result of risks uncertainties and other factors including but not limited to the factors set forth in the company is filings with the securities and exchange commissionBeam Search: if you take a week of ourusbusiness it is 15000000 of orders now even if you say that we were stopped for the full week and we lost 15000000 of orders which has not happened we were not stopped for the full week we did not lose 15000000 of ordersHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: if you take a week of ouru sbusiness it is 15000000 of orders now even if you say that we were stopped for the full week and we lost 15000000 of orders which has not happened we were not stopped for the full week we did not lose 15000000 of ordersGold: if you take a week of ouru sbusiness it is 15000000 of orders now even if you say that we were stopped for the full week and we lost 15000000 of orders which has not happened we were not stopped for the full week we did not lose 15000000 of ordersIncorrect Hybrid editsBeam Search: for a protocol violation after he received an unrelated vaccination for shingles just before his 2nd dose ofsel212 his uric acid level had been fully controlled over the 1st monthHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: for a protocol violation after he received an unrelated vaccination for shingles just before his 2nd dose ofsele212 his uric acid level had been fully controlled over the 1st monthGold: for a protocol violation after he received an unrelated vaccination for shingles just before his 2nd dose ofsel212 his uric acid level had been fully controlled over the 1st monthBeam Search: the programs have so far generated more than 1300 ideas and initiatives with individual initiatives benefits ranging from 1000s of dollars to multimillions over fiscal year17and beyondHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: the programs have so far generated more than 1300 ideas and initiatives with individual initiatives benefits ranging from 1000s of dollars to multimillions over fiscal year2017and beyondGold: the programs has so far generated more than 1300 ideas and initiatives with individual initiatives benefits ranging from 1000s of dollars to multimillions over fiscal year17and beyondBeam Search: from an ongoing ebit standpoint it is not really going to have an impact and as far as further reducing costs in that business all those costs were really contained inirapuatoand there was very little cost here to deal with that particular issue so i do not think we are going to see anythingHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: from an ongoing ebit standpoint it is not really going to have an impact and as far as further reducing costs in that business all those costs were really contained inrio tintoand there was very little cost here to deal with that particular issue so i do not think we are going to see anythingGold: from an ongoing ebit standpoint it is not really going to have an impact as far as further reducing costs on that business all those costs are really contained inirapuatoand there was very little cost here dealing with that particular issue so i do not think we are going to see anything

\[TED\-LIUM\]Correct Hybrid editsBeam Search: making it can you see it yes good this is actually me making the i am not good at life face this is a piece of graffiti in my old neighborhood in berkeley california where i did myph don why we are better in games than we are in real life and this is a problem that a lot of gamers have we feel that we are not as good in reality as we are in games and i do not mean just good as in successful although that is part of it we do achieve more in game worlds but i also mean good as in motivated to do something that matters inspired to collaborate and to cooperate when we are in game worldsHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: making it can you see it yes good this is actually me making the i am not good at life face this is a piece of graffiti in my old neighborhood in berkeley california where i did myphdon why we are better in games than we are in real life and this is a problem that a lot of gamers have we feel that we are not as good in reality as we are in games and i do not mean just good as in successful although that is part of it we do achieve more in game worlds but i also mean good as in motivated to do something that matters inspired to collaborate and to cooperate when we are in game worldsGold: making it can you see yes good this is actually me making the i am not good at life face this is a piece of graffiti in my old neighborhood in berkeley california where i did myphdon why we are better in games than we are in real life and this is a problem that a lot of gamers have we feel that we are not as good in reality as we are in games and i do not mean just good as in successful although that is part of it we do achieve more in game worlds but i also mean good as in motivated to do something that matters inspired to collaborate and to cooperate and when we are in game worldsBeam Search: and the fundamental lesson i believe is that design truly is a contact sport it demands that we bring all of our senses to the task and that we apply the very best of our thinking our feeling and our doing to the challenge that we have at hand and sometimes a little prototype of this experience is all that it takes to turn us from an ooh 0 moment to atadamoment and that can make a big difference thank you very muchHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: and the fundamental lesson i believe is that design truly is a contact sport it demands that we bring all of our senses to the task and that we apply the very best of our thinking our feeling and our doing to the challenge that we have at hand and sometimes a little prototype of this experience is all that it takes to turn us from an ooh 0 moment to ata damoment and that can make a big difference thank you very muchGold: and the fundamental lesson i believe is that design truly is a contact sport it demands that we bring all of our senses to the task and that we apply the very best of our thinking our feeling and our doing to the challenge that we have at hand and sometimes a little prototype of this experience is all that it takes to turn us from an 0 moment to ata damoment and that can make a big difference thank you very muchBeam Search: bill gross has several companies including one called e solar that has some great solar thermal technology vinodkolstadis investing in dozens of companies that are doing great things and have interesting possibilities and i am i am trying to help back that and nathan meriwald and i actually are backing a companyHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: bill gross has several companies including one called e solar that has some great solar thermal technology vinodkhoslais investing in dozens of companies that are doing great things and have interesting possibilities and i am i am trying to help back that and nathan meriwald and i actually are backing a companyGold: bill gross has several companies including one called esolar that has some great solar thermal technologies vinodkhoslais investing in dozens of companies that are doing great things and have interesting possibilities and i am i am trying to help back that nathan myhrvold and i actually are backing a companyIncorrect Hybrid editsBeam Search: this guy was a hero jonas saw he took one of the worst scourges of mankind away from us no fear no agony poliopuffgoneHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: this guy was a hero jonas saw he took one of the worst scourges of mankind away from us no fear no agony poliopoofgoneGold: this guy was a hero jonas salk he took one of the worst scourges of mankind away from us no fear no agony poliopuffgoneBeam Search:andi am telling a story that many of you know because steve is columns became the basis for a book which was turned into a movieHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: i am telling a story that many of you know because steve is columns became the basis for a book which was turned into a movieGold:andi am telling a story that many of you know because steve is columns became the basis for a book which was turned into a movieBeam Search: i would like to leave you with a poem by a 14th century persian poet namedhafizHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: i would like to leave you with a poem by a 14th century persian poet namedhafezGold: i would like to leave you with a poem by a 14th century persian poet namedhafiz

\[VoxPopuli\]Correct Hybrid editsBeam Search:cedawis role is to contribute to the shaping and implementing of vet but also shaping skills and qualification policies at the union level the agency can do this by providing crucial evidence and services for policy making and knowledge sharing amongst the union and national actors in particular to governments and social partners i believe that the revision of the functioning of this tripartite agency has been a very good one and i fully appreciate the agency is great work in supporting and developing of inclusive and quality vet systems now we have to work together that we put in placeHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}:cedefopis role is to contribute to the shaping and implementing of vet but also shaping skills and qualification policies at the union level the agency can do this by providing crucial evidence and services for policy making and knowledge sharing amongst the union and national actors in particular to governments and social partners i believe that the revision of the functioning of this tripartite agency has been a very good one and i fully appreciate the agency is great work in supporting and developing of inclusive and quality vet systems now we have to work together that we put in placeGold:cedefopis role is to contribute to the shaping and implementing of vt but also to shaping skills and qualification policies at union level the agency can do this by providing crucial evidence and services for policymaking and knowledge sharing among the union and national actors in particular to governments and social partners i believe that the revision of the function of this tripartite agency has been a very good one and i fully appreciate the agency is great work in supporting and developing inclusive and quality vt systems now we have to work together to put in placeBeam Search: eu is intention is to help the yemeni population and this is why we need to support the efforts in identifying a political solution as soon as possible using dialog and negotiations we need to understand that all the parties in the conflict must comply with their obligations under international lawHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}:theeu is intention is to help the yemeni population and this is why we need to support the efforts in identifying a political solution as soon as possible using dialog and negotiations we need to understand that all the parties in the conflict must comply with their obligations under international lawGold:theeu is intention is to help the yemeni population and this is why we need to support the efforts in identifying a political solution as soon as possible using dialog and negotiations we need to understand that all parties in the conflict must comply with their obligations under international lawBeam Search: eu aid worth 1000000s of euros was destroyed in full impunity and with no adequate labeling of settlement products the village ofkanalmachthat i and countless friends of the bedouins have visited is about to be destroyedHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: eu aid worth 1000000s of euros was destroyed in full impunity and with no adequate labeling of settlement products the village ofkhan al ahmarthat i and countless friends of the bedouins have visited is about to be destroyedGold: eu aid worth 1000000s of euros destroyed in full impunity and no adequate labeling of settlement products the village ofkhan al ahmarthat i and countless friends of the bedouins have visited is about to be destroyedIncorrect Hybrid editsBeam Search: but we must also set clear conditions most important of all is the return of the rule of law and political freedoms and there establishmentof civilian rule over the military it must also include a clear path to inclusive and fair elections in 2018 which could also be observed by the european parliamentHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: but we must also set clear conditions most important of all is the return of the rule of law and political freedoms and thereestablishmentof civilian rule over the military it must also include a clear path to inclusive and fair elections in 2018 which could also be observed by the european parliamentGold: but we must also set clear conditions most important of all is the return of the rule of law and political freedoms and there establishmentof civilian rule over the military it must also include a clear path to inclusive and fair elections in 2018 which could also be observed by the european parliament moreoverBeam Search: i would also like to say that the position of the president of the commission in some ways is understandablebuton the other hand there is a huge risk in giving up the community method and coming back to national contributions in special instrumentsHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}: i would also like to say that the position of the president of the commission in some ways is understandableunderstandableon the other hand there is a huge risk in giving up the community method and coming back to national contributions in special instrumentsGold: i would like to say as well that the position of the president of the commission is in some ways understandablebuton the other hand there is a huge risk concerning giving up the community method and going back to national contributions in special instrumentsBeam Search: memory herself has said that marriage is often the end for girls like me but if our leaders will invest in us and give us the chance to be educated we will become women who create a better society for everyoneHybrid Searchτ=12\\tau=\\frac\{1\}\{2\}:madam presidentmemory herself has said that marriage is often the end for girls like me but if our leaders will invest in us and give us the chance to be educated we will become women who create a better society for everyoneGold: memory herself has said marriage is often the end for girls like me but if our leaders will invest in us and give us the chance to be educated we will become women who create a better society for everyone

### C\.4Layer\-wise Features Visualization

We show heatmaps of the layer\-wise features for a Common Voice example in Figure[7](https://arxiv.org/html/2609.02940#A3.F7)and Figure[8](https://arxiv.org/html/2609.02940#A3.F8)\. Named\-entity words such as “Mortimer”, “Adler” and “Randi” exhibit higher cosine similarities and norm ratios in intermediate layers\.

![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/common_voice-cos2.png)Figure 7:Layer\-wise heatmaps forCommon Voiceshowing AngleFeat values\. The text reads: “Nebel also had philosopher Mortimer Adler and James Randi on frequently\.” Named entities “Nebel”, “Mortimer”, “Adler”, “James”, and “Randi” are highlighted by the red box\. Some of the named\-entity tokens exhibit higherAngleFeatvalues\.![Refer to caption](https://arxiv.org/html/2609.02940v1/figs_heatmap/common_voice-norm2.png)Figure 8:Layer\-wise heatmaps forCommon VoiceshowingMagFeatvalues\. The text reads: “Nebel also had philosopher Mortimer Adler and James Randi on frequently\.” Named entities “Nebel”, “Mortimer”, “Adler”, “James”, and “Randi” are highlighted by the red box\. Some of the named\-entity tokens exhibit higherMagFeatvalues\.