Voice Memory for Agentic Speech Recognition

# Voice Memory for Agentic Speech Recognition
Source: [https://arxiv.org/html/2607.26410](https://arxiv.org/html/2607.26410)
Chao\-Han Huck Yang∗Zih\-Ching Chen Piotr Żelasko Zhehuai Chen Jagadeesh Balam Boris Ginsburg NVIDIA Correspondence:huckiyang@ieee\.org\{virginiac, pzelasko, zhehuaic\}@nvidia\.com ∗Work done at NVIDIA\. [Model](https://huggingface.co/huckiyang/voice-memory)[Demo](https://huckiyang.github.io/voice-memory/)[NeMo\-Speech](https://github.com/NVIDIA-NeMo/Speech)[NeMoClaw](https://github.com/NVIDIA/NemoClaw)[Notebook](https://huckiyang.github.io/voice-memory-notebook)

###### Abstract

Abstract We presentVoice Memory, a inference\-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per\-domainmemory\.mdand decides per utterance whether to*act*on the hypothesis or*abstain*and keep the 1\-best\. Asynchronously, a score\-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held\-out score\. Extended from classical ASR\-LM framework, we refer this split the*listener\-thinker*architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable\. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction \(GER\) over\-corrects, breaking correct tokens on up to 64% of its edits on financial news, andVoice Memoryreduces this rate to 35%\. Across ten HyPoradise domains with an open corrector,Voice Memorylowers weighted word error rate from 8\.36% to 7\.52% \(7\.47% with three added in\-context examples\) without regressing any dataset below its 1\-best baseline; gains concentrate where recoverable headroom is largest, including air\-travel commands \(8\.40% to 3\.40%\) and noisy far\-field speech \(CHiME\-4, 12\.69% to 10\.46%\)\. The memory transfers across corrector families and adds zero parameters to the inference path\. A demo and example code are provided for future studies\.

###### keywords:

voice agents, memory modeling, on\-device, language models, test\-time adaptation

## 1Introduction

Modern voice systems have entered a low\-error regime for clean speech recognition\. Decades of progress, from statistical HMM systems\[rabiner1989tutorial,jelinek1997statistical\]to large\-scale neural models\[baevski2020wav2vec,hsu2021hubert,gulati2020conformer,radford2023robust,puvvada2024less\], have brought read speech below 2% word error rate \(WER\); yet decoders still make systematic, domain\-specific errors\[chen2023hyporadise,likhomanenko2021rethinking,szymanski2020wer,ma2023n\]and preferences\[koluguri2026preference,hu2025chain,wu2026speecheq,hu2023scaling\]\. In the classical source\-channel formulation of recognition\[jelinek1997statistical\], which mediates between words and memories\[goldinger1996words,papcun1989long\], these are exactly the failures expected when acoustic evidence is reconciled with a mismatched language prior\. These errors\[szymanski2020wer,stolcke2000dialog\]are consistent within a domain but differ across domains: in financial news, an ASR system renders “Bentsen” as “benson”; in air\-travel queries, it collapses the spelled form “u s air” into “us air\.” A fix tuned for one domain therefore transfers poorly to another\.

![Refer to caption](https://arxiv.org/html/2607.26410v1/x1.png)Figure 1:Overview ofVoice Memory\. At inference, a frozen corrector reads a learnable, human\-readable text memory and decides per utterance whether to*act*on the ASR hypothesis or*abstain*\. Offline, an optimizer distills this instructional memory from training corpora using forward passes only, with no weight updates, so that the learned skill \(i\) generalizes within an acoustic domain and \(ii\) transfers across domains and correctors\.A widely adopted remedy is the generative hypothesis\-rewriting framework: thenn\-best decoding hypotheses \(i\.e\., ASR, OCR, or machine translation\) are passed to an LLM, which rewrites them into a single transcript\[chen2023hyporadise,radhakrishnan2023whispering,thomas2024leveraging,yeo2025mms,hsu2025let,yang2025covoger,carofilis2026text\]\. For instance, GER\[yang2023generative\]extends a long line of language\-model rescoring and correction, from BERT\-based hypothesis scoring\[salazar2020masked\]to edit\-aligned neural correction\[leng2021fastcorrect\]that achieved several state\-of\-the\-art results in annual speech\-translation competitions and on robust ASR benchmarks\. In the low\-error regime, however, a strong LLM tends to*over\-correct*: it rewrites tokens that were already correct, changinghome istohome’sand normalizing digits and spelling to its own conventions\. Each such edit is fluent, yet it moves the output away from the reference transcript; on low\-headroom domains, zero\-shot GER raises WER above the unedited 1\-best \(§[4](https://arxiv.org/html/2607.26410#S4)\)\. The decision facing a corrector has therefore inverted: at sub\-2% WER, the question is no longer whether the model*can*fix an error, but whether it should modify the hypothesis at all\.

We refer to this setting as*agentic speech recognition*, and we make the term precise\. We call an ASR system agentic when it satisfies three conditions\. First,decision: it selects among actions per utterance, in our case whether to*act*on the hypothesis or to*abstain*and keep it, rather than applying one fixed transformation to every input\. Second,persistent state: its decisions are conditioned on an explicit, inspectable state that outlives a single utterance, in our case a text memory rather than activations or weights\. Third,self\-improvement: it revises that state from feedback on its own scored outputs, without human intervention\. Prior work realizes these properties inside the model, fine\-tuning an omni\-model to emit per\-utterance action decisions from audio \(§[2](https://arxiv.org/html/2607.26410#S2)\)\. We study the complementary, test\-time realization: the decision policy lives in an external memory that a frozen corrector reads at inference\.

Definition 1 collects the three conditions into a single format, which we call the*listener\-thinker*\(LT\) architecture\. It extends the generic ASR\-LM cascade, a frozen decoder \(BϕB\_\{\\phi\}\) feeding a frozen language modelMθM\_\{\\theta\}, with exactly the two objects the three conditions require: an explicit action distributionπ\\piover\{act,abstain\}\\\{\\textsf\{act\},\\textsf\{abstain\}\\\}and a persistent text state \(ss\) that a separate, asynchronous optimizer revises\. The listener is the synchronous path that decodes and decides at stream time; the thinker is the background path that improvesssbetween utterances\. Table[1](https://arxiv.org/html/2607.26410#S1.T1)situates the format against the two dominant speech stacks: E2E ASR fixes the map and never decides, SpeechLMs always emit and keep their state in weights and context, and only theLTformat has slots for evidence \(HH\), external state, a per\-utterance actionaa, and self\-improvement\.

Definition 1: The Listener\-Thinker \(LT\) FormatWritex=\(xAudio,xText\)x=\(x\_\{\\mathrm\{Audio\}\},x\_\{\\mathrm\{Text\}\}\)for audio input andy^=\(yText,yAudio\)\\hat\{y\}=\(y\_\{\\mathrm\{Text\}\},y\_\{\\mathrm\{Audio\}\}\)for text output\. An agentic speech system in the*listener\-thinker*is a tuple\(Bϕ,Mθ,π,s,O​p\)\(B\_\{\\phi\},M\_\{\\theta\},\\pi,s,Op\), whose synchronous*listener*\(Bϕ,Mθ,π\)\(B\_\{\\phi\},M\_\{\\theta\},\\pi\)works with an optimizer \(O​pOp\): a frozen decoderBϕB\_\{\\phi\}mapsxxto textual evidenceH=\(h1,…,hn\)H=\(h\_\{1\},\\dots,h\_\{n\}\); a frozenMθM\_\{\\theta\}, holding a persistent, inspectable states∈Σ∗s\\in\\Sigma^\{\*\}, emits per utterance through an actiona∼π\(⋅∣H,s\)a\\sim\\pi\(\\cdot\\mid H,s\), so thatp​\(y^∣x,s\)=∑aπ​\(a∣H,s\)​pθ​\(y^∣H,s,a\)p\(\\hat\{y\}\\mid x,s\)=\\sum\_\{a\}\\pi\(a\\mid H,s\)\\,p\_\{\\theta\}\(\\hat\{y\}\\mid H,s,a\)with:y^=\{Mθ​\(H,s\)a=acth1a=abstain,\\hat\{y\}\\;=\\;\\begin\{cases\}M\_\{\\theta\}\(H,s\)&a=\\textsf\{act\}\\\\\[1\.0pt\] h\_\{1\}&a=\\textsf\{abstain\},\\end\{cases\}while text statessis revised only asynchronously by the score\-gated*thinker*, withθ,ϕ\\theta,\\phifixed\.asr\-lm\(this work\) andspeechlm\-lmare the same type at different signatures:xText=yAudiox\_\{\\mathrm\{Text\}\}=y\_\{\\mathrm\{Audio\}\}at utterance timescale versus the full signature at frame timescale with𝒜⊇\{act,abstain\}\\mathcal\{A\}\\supseteq\\\{\\textsf\{act\},\\textsf\{abstain\}\\\}\(speak, wait, delegate\)\. Constant policies recover the classics:π≡abstain\\pi\\equiv\\textsf\{abstain\}is E2E decoding;π≡act\\pi\\equiv\\textsf\{act\},s=∅s=\\varnothingis a zero\-shot GER case, when the thinker is with fixedθ\\thetaandϕ\\phiparameters with failure rollouts \(FF\) and Success rollouts \(UU\)\.

Remark 1 \(Relation to interaction models\)\.TheLTformat is the speech instance of the interaction/background split now emerging at the frontier\[tml2026interaction,huang2026duplexomni\]: the listener is the synchronous interaction path, the thinker is the asynchronous background path, and the two are coupled*only*through the persistent artifact text state,ss\. The artifact is what makes the pair auditable and portable\.

Table 1:Typology of speech systems\. The listener\-thinker \(LT\) format subsumesasr\-lm\(refer to Definition[1](https://arxiv.org/html/2607.26410#S1), utterance timescale\) andspeechlm\-lm\(full signature, frame timescale\) as one type: only theLTcolumn has slots for evidenceHH, external text statess, a per\-utterance actionaa, and test\-time self\-improvement\.Expressing such a policy with existing adaptation mechanisms is costly\. Fine\-tuning, LoRA\[hu2022lora\], and soft prompts\[lester2021power\]place the adaptation in weights or continuous vectors, which cannot be inspected, transferred across model families, or revised without retraining; few\-shot exemplars\[brown2020language\]recur a token cost on every request\. None yields an adaptation that is simultaneously inference\-only, auditable, and portable across correctors\.

We presentVoice Memory, which stores the correction skill in a text file\. The corrector stays frozen and reads a single human\-readablememory\.mdat test time \(Figure[1](https://arxiv.org/html/2607.26410#S1.F1)\)\. A separate optimizer converts scored rollouts into bounded add/delete/replace edits and accepts an edit only when it strictly improves a held\-out score\[yang2026skillopt\]\. The dominant policy this loop discovers is restraint: from its own scored mistakes, the optimizer writes suppressive rules such as “keep the verbatim ASR token; do not normalize,” with no human in the loop\. Where recoverable headroom exists,Voice Memorycloses most of the gap between the 1\-best and the oracle; a second corrector running the same loop even surpasses thenn\-best oracle by recovering tokens present in no hypothesis\. We will later introduce new measurement metrics of \(i\) recoverable information ratio \(ρ\\rho\) and \(ii\) harmful edit rate on these two scenarios for agentic speech recognition\.

This paper makes four contributions:

1. 1\.Method\.Voice Memory, an inference\-only adaptation scheme for agentic speech recognition: the correction skill resides in a per\-domain text memory that a validation\-gated loop improves through bounded edits, with no weight updates \(§[3](https://arxiv.org/html/2607.26410#S3)\)\.
2. 2\.Formulation and diagnostics\.A formal account of the act/abstain decision as the listener\-thinker format \(Definition 1, Table[1](https://arxiv.org/html/2607.26410#S1.T1)\), together with two measures: the*Recoverable Information Ratio*\(ρ\\rho\) for recovery against thenn\-best oracle, and the*Harmful Edit Rate*\(her, Figure[2](https://arxiv.org/html/2607.26410#S4.F2)\) for over\-correction\. Across nine domains, the gain from correction tracks recoverable headroom \(r=\+0\.90r\{=\}\{\+\}0\.90\), delineating the operating regime in which a corrector should act \(§[4](https://arxiv.org/html/2607.26410#S4)\)\.
3. 3\.Portability\.Evidence that the optimized memory is a transferable, corrector\-agnostic artifact: a memory written by one model family improves a reader from another, and that reader can re\-form its own with the same loop \(§[5](https://arxiv.org/html/2607.26410#S5)\)\.
4. 4\.Noise robustness\.The same restraint policy holds under acoustic noise on CHiME\-4 and NOIZEUS, matching training\-based robust correction without fine\-tuning or latent noise embeddings \(§[6](https://arxiv.org/html/2607.26410#S6)\)\.

Taken together,Voice Memoryoffers a low\-cost adaptation path for cascades of ASR and LLM correction: no training infrastructure, a sub\-10 KB auditable artifact, and a measured account of when a corrector should act and when it should hold still\.

## 2Related Work

#### ASR\-LM and Agentic Setup for ASR\.

Recent work treats audio LLMs as agents that decide how to use internal perception and external evidence, rather than decoding passively\[wan2026speechhands,lin2025neko\]\. Speech\-Hands\[wan2026speechhands\]and Speech\-Mind\[wang2026audio\]are the closest to our work\. These setups learn a per\-utterance action token by fine\-tuning an omni\-model on speech and audio reasoning, so its agency is acoustic, learned in weights, and re\-decided each utterance\.Voice Memorydiffers on each of these axes \(§[1](https://arxiv.org/html/2607.26410#S1)\): it learns one accumulated memory, uses no audio and no training, and runs entirely at inference\. The two methods are the trained and test\-time versions of one idea, an ASR system that decides when to trust its own output\.

#### Generative error correction and contextual biasing\.

Injecting domain\-specific knowledge into ASR output is a long\-standing goal\. Early systems brought contextualnn\-grams into decoding through shallow fusion and on\-the\-fly rescoring\[aleksic2015bringing,zhao2019shallow\]\. End\-to\-end systems then moved biasing into the model: CLAS\[pundak2018deep\]jointly embeds bias phrases and attends to them during decoding, and later work extended this to RNN\-T\[jain2020contextual\]and to trie\-based deep biasing\[le2021contextualized\]\. These methods adapt the acoustic model and require a bias list and a training run\. LLM\-based GER instead recasts correction as a language task overnn\-best hypotheses\[chen2023hyporadise,yang2023generative,radhakrishnan2023whispering\], adapting the corrector through fine\-tuning or in\-context exemplars\.Voice Memorysits at the opposite end of both lines: it leaves the corrector frozen and writes the domain’s conventions into a text memory read at inference, with no bias list, no training, and no weight updates\.

#### Noise\-robust ASR and robust hypotheses\.

One popular ASR\-LM use case is Robust\-ASR\. Acoustic noise makes correction harder because thenn\-best list itself degrades, so a corrector must denoise rather than rerank\. The Robust HyPoradise benchmark\[hu2024large\]pairs noisy Whisper hypotheses with clean references across CHiME\-4, VoiceBank\-DEMAND, LibriSpeech\-FreeSound, and NOIZEUS\. RobustGER distills the diversity of those hypotheses into a language\-space noise embedding and fine\-tunes the corrector against it, which recovers much of the lost accuracy\. Cross\-modality correctors such as NeKo\[lin2025neko\]extend this with task\-guided mixtures of experts\. Both methods read the noise condition in a latent vector and pay for it with a training run\.Voice Memoryuses the same observation, that thenn\-best spread already encodes the noise condition, but writes the prior into a per\-noise\-family text memory that the frozen corrector reads at inference \(§[6](https://arxiv.org/html/2607.26410#S6)\)\.

#### Text\-space optimization\.

Optimizing natural\-language artifacts such as prompts and skills against a score is an alternative to gradient updates\[yuksekgonul2024textgrad,yang2024large,yang2026skillopt\]\. We apply this to ASR correction, where thenn\-best oracle sets a bounded target \(ρ\\rho\) and the main learned behavior is restraint\.

#### WER critique and semantics\.

Prior work questions WER as the only ASR metric\[szymanski2020wer,kim2021semantic\]\. We add two results\. We measure the gap between surface error and meaning \(§[7\.2](https://arxiv.org/html/2607.26410#S7.SS2)\), and we show that a semantic training signal gives lower WER than a traditional WER signal \(§[7\.1](https://arxiv.org/html/2607.26410#S7.SS1)\)\.

Table 2:Adapting a cascade of ASR and LLM correction: weight space adaptation fine\-tuning FT\) versusVoice Memory\.Voice Memoryruns forward passes only on the same frozen endpoint used at deployment, so it needs no training infrastructure and stays energy efficient on device\. Figures are order\-of\-magnitude\.

## 3TheVoice MemoryMethod

The corrector never changes; only its memory does\. We formalize correction as a per\-utterance decision to act or abstain \(§[3\.1](https://arxiv.org/html/2607.26410#S3.SS1)\), define the object we optimize and the inference\-only loop that optimizes it \(§[3\.2](https://arxiv.org/html/2607.26410#S3.SS2)\), and define a measure of over\-correction used in our analysis \(§[3\.3](https://arxiv.org/html/2607.26410#S3.SS3)\)\.

### 3\.1Problem: correction against an oracle ceiling

Let a domain providenn\-best hypothesesH=\(h1,…,hn\)H=\(h\_\{1\},\\dots,h\_\{n\}\)from a frozen acoustic model, withh1h\_\{1\}the 1\-best\. A corrector produces a single transcript fromHH\. At low error rates the corrector’s main decision is not which hypothesis to pick but whether to changeh1h\_\{1\}at all, so we treat correction as a per\-utterance choice to act or abstain\.

Thenn\-best list bounds what reranking can achieve\. The oracle selects, per utterance, the hypothesis inHHwith the lowest WER against the reference, givingWERoracle\\mathrm\{WER\}\_\{\\text\{oracle\}\}\. Reranking cannot go below this bound; correction can, because it may produce tokens that appear in no hypothesis\. We measure how much of the 1\-best\-to\-oracle gap a correction closes with the*Recoverable Information Ratio*\(ρ\\rho\):

ρ=WER1​\-best−WERy^WER1​\-best−WERoracle,\\rho=\\frac\{\\mathrm\{WER\}\_\{1\\text\{\-best\}\}\-\\mathrm\{WER\}\_\{\\hat\{y\}\}\}\{\\mathrm\{WER\}\_\{1\\text\{\-best\}\}\-\\mathrm\{WER\}\_\{\\text\{oracle\}\}\},\(1\)wherey^\\hat\{y\}is the corrected transcript\.ρ=1\\rho\{=\}1closes the gap exactly\.ρ<0\\rho\{<\}0is the damage regime, where correcting is worse than keepingh1h\_\{1\}\.ρ\>1\\rho\{\>\}1means the corrector recovers tokens present in no hypothesis and goes below the oracle bound\.ρ\\rhois the target the method aims at, and §[4](https://arxiv.org/html/2607.26410#S4)reports it per domain\.

To analyze*why*a correction helps, we also label each token edit as helpful \(fixed a wrong token\) or harmful \(broke a correct one\); the Harmful Edit Rate \(her\) is the harmful fraction, and isolates over\-correction\. We reportheron a subset of domains \(Figure[2](https://arxiv.org/html/2607.26410#S4.F2)\)\.

### 3\.2Voice Memory and the optimization loop

A frozen correctorMMreadsHHand an optional memoryss, and emitsy^=M​\(H,s\)\\hat\{y\}=M\(H,s\)\. The memoryssis a short Markdown document, placed in the corrector’s context before each utterance\. Withs=∅s=\\varnothingthis reduces to standard GER\. We keepssin text, external toMM, so that it can be read and edited by an operator, reused across correctors, and added with no change to the inference path\. A learnedssis a small set of rules, for example:

> keep the verbatim ASR token; do not normalize\. place ‘dollars’ after the amount, not ‘$’ before it\.

We optimizesson a domain’s training split with a trajectory feedback loop\[agrawal2025gepa,ni2026trace2skill,yang2026skillopt\], treatingssas the external state of the frozenMM\.

Input:frozenMM, optimizerO​pOp; splitsDtrain,DselD\_\{\\text\{train\}\},D\_\{\\text\{sel\}\}; epochsEE; budget\{ℓe\}\\\{\\ell\_\{e\}\\\}

Output:memory

s⋆s^\{\\star\}
s,s⋆←∅s,s^\{\\star\}\\leftarrow\\varnothing;

R←∅R\\leftarrow\\emptyset;

v⋆←Score​\(M,∅,Dsel\)v^\{\\star\}\\leftarrow\\textsc\{Score\}\(M,\\varnothing,D\_\{\\text\{sel\}\}\)
for*e←1e\\leftarrow 1toEE*do

foreach*batchH⊂D*train*H\\subset D\_\{\\text\{train\}\}*do

Y^←\{M​\(h,s\):h∈H\}\\hat\{Y\}\\leftarrow\\\{M\(h,s\):h\\in H\\\};

F,U←F,U\\leftarrowfailures, successes

//rollout

s′←O​p​\(s,F,U,R,ℓe\)s^\{\\prime\}\\leftarrow Op\(s,F,U,R,\\ell\_\{e\}\);

v′←Score​\(M,s′,Dsel\)v^\{\\prime\}\\leftarrow\\textsc\{Score\}\(M,s^\{\\prime\},D\_\{\\text\{sel\}\}\)
//

≤ℓe\\leq\\ell\_\{e\}edits

if*v′\>v⋆v^\{\\prime\}\>v^\{\\star\}*then

s⋆←s←s′s^\{\\star\}\\leftarrow s\\leftarrow s^\{\\prime\};

v⋆←v′v^\{\\star\}\\leftarrow v^\{\\prime\}
//accept

else

R←R∪\{s′\}R\\leftarrow R\\cup\\\{s^\{\\prime\}\\\}
//reject

return

s⋆s^\{\\star\}

Algorithm 1Voice Memory optimization\. All model calls are forward passes; no gradients, no weight updates\.The three splits stay disjoint: train supplies rollout evidence, selection gates acceptance, and test is used once, for reporting\. At deployment the frozenMMreads the best acceptedss\. No weights change, and no extra model calls occur\. The rules the loop accepts are mostly suppressive; §[4](https://arxiv.org/html/2607.26410#S4)shows the learned policy is restraint\.

### 3\.3Measuring over\-correction

WER reports whether a correction helped, not why\. We add the*Harmful Edit Rate*\(her\)\. We aligny^\\hat\{y\}againsth1h\_\{1\}and the reference, then label each token edit as helpful \(fixed a wrong token\), harmful \(broke a correct token\), or missed\.heris the fraction of all token edits that are harmful\. It isolates over\-correction: a corrector that edits tokens that were already correct has a highher\.

We evaluate on HyPoradise v0\[chen2023hyporadise\], which pairs Whisper 5\-best hypotheses with references across 10 domains under disjoint train/test splits: clean read speech \(ls\_clean\), financial news \(wsj\), voice commands \(atis\), telephone and meeting speech \(swbd,chime4\), and accented or conversational speech \(cv,coraal,lrs2,td3,ls\_other\)\. A single frozen instruction LLM, the open\-source MiniMax\-M3\[lai2026minimax\]with 1M context and∼\\sim428B parameters, serves as both corrector and optimizer\.111We useMiniMax\-M3; the model checkpoint has fixed, open weights:[https://huggingface\.co/MiniMaxAI/MiniMax\-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3)\.Semantic scoring, used to gate some memories and analyzed in §[7](https://arxiv.org/html/2607.26410#S7), runs on an on\-device sentence encoder \(all\-MiniLM\-L6\-v2, 4\-bit;\[reimers2019sentencebert\]\) and adds no API cost\. We report the full agwer python suite, with WER shown alongsideρ\\rhoandher\.222WER andherare computed with our open\-sourceagwerpackage \([https://pypi\.org/project/agwer/](https://pypi.org/project/agwer/)under MIT license\), an efficient implementation that is safe to use as a reward target inside the optimization loop\.

Table 3:Voice Memory\(semantic\-gated\) across HyPoradise full test splits, same corrector \(MiniMax\-M3428B\{\}\_\{\\text\{428B\}\}\)\.Boldmarks the lowest WER among GER0\-shot\{\}\_\{\\text\{0\-shot\}\}, GER5\-shot\{\}\_\{\\text\{5\-shot\}\}, andVMper row\. The gain follows the scaling properties:ρVM\\rho\_\{\\textsc\{VM\}\}is high on high\-WERatis/cvand negative at the near\-floortd3/ls\_clean\.

## 4Over\-Correction and Scaling Properties

#### Over\-correction and restraint\.

The core failure of unconstrained correction is over\-correction: zero\-shot GER edits tokens that were already correct\. In Table[3](https://arxiv.org/html/2607.26410#S3.T3)it raises WER above the 1\-best on the low\-headroom domains \(ls\_clean1\.8→2\.41\.8\\\!\\to\\\!2\.4,ls\_other3\.7→4\.13\.7\\\!\\to\\\!4\.1,td34\.1→4\.44\.1\\\!\\to\\\!4\.4\), and on four labeled domains it is harmful on a large fraction of its edits, up to 0\.64 onwsj\(Figure[2](https://arxiv.org/html/2607.26410#S4.F2)\)\.Voice Memoryoutperforms GER on four tasks\. Moreover, it lowers the harmful\-edit rate on every labeled domain \(to 0\.35 onwsj; Figure[2](https://arxiv.org/html/2607.26410#S4.F2)\), and the deployedwsjmemory is 776 bytes\.

![Refer to caption](https://arxiv.org/html/2607.26410v1/x2.png)Figure 2:Harmful Edit Rate \(HER\) on four HyPoradise domains\. Static GER over\-corrects;Voice Memorylowers HER on every domain\. Lower is better\.
#### The gain scales with WER\.

One benefit of correction is that it scales with recoverable headroom, measured byρ\\rho: largest where 1\-best WER is high, and vanishing at the error floor\. Across the nine domains of Table[3](https://arxiv.org/html/2607.26410#S3.T3),ρ\\rhocorrelates with 1\-best WER atr=\+0\.90r\{=\}\{\+\}0\.90\.Voice Memoryhelps most where headroom is largest: at thels\_cleanfloor \(ρ=−0\.11\\rho\{=\}\{\-\}0\.11\) andtd3\(ρ=−0\.09\\rho\{=\}\{\-\}0\.09\) little remains to recover, while on high\-WERatisit recovers most of the gap \(ρ=0\.96\\rho\{=\}0\.96\)\. Few\-shot prompting is competitive on some domains but pays a token cost per call, whereas theVoice Memorymemory is formed once and read for free\.

## 5Memory Transfer and Portability

Since the memory is in text form, it is transferable to different models\. We show two additional studies: a memory written by one corrector helps a second corrector from a different family, and that second corrector can form its own memory with the same loop\.

Table 4:RobustVoice Memory\(VM\) on the Robust HyPoradise corpora\[hu2024large\]\.Voice Memorylowers WER on the recorded\-noise corpora \(CHiME\-4, NOIZEUS\), where static GER over\-corrects, and is near\-null on read speech with added noise \(VoiceBank\-DEMAND, LS\-FreeSound; discussed in the text\)\.ρvm\\rho\_\{\\textsc\{vm\}\}is the recovered fraction of the 1\-best\-to\-oracle gap; bold marks the best WER per row\.Table 5:A second frozen corrector \(Claude\-4\.6\-Sonnet\) at inference on a 100\-utterance test, varying only the memory\. “\+MiniMax” reads the memory optimized in §[4](https://arxiv.org/html/2607.26410#S4); “\+Claude” reads a memory Claude formed itself with the same validation\-gated loop \(§[3\.2](https://arxiv.org/html/2607.26410#S3.SS2)\)\. Onatisthe self\-formed memory beats the MiniMax one and thenn\-best oracle \(ρ=1\.28\\rho\{=\}1\.28\); on data\-hungrywsjit over\-corrects; at the low\-WER floor no memory is best\.#### A memory transfers to a corrector from a different family\.

We test a second frozen corrector, Anthropic Claude\-4\.6\-Sonnet \(2026 February\), at inference\. Onatis, Claude reads a MiniMax\-written memory and improves over its no\-memory baseline \(6\.68% to 6\.08% WER, Table[5](https://arxiv.org/html/2607.26410#S5.T5)\)\. The reader improves with a memory it did not write, so the memory is a separate component, not a property of the writer\.

#### A corrector can form its own memory\.

Running the same validation\-gated loop with Claude as both corrector and optimizer gives a self\-formed memory that reachesρ=1\.28\\rho\{=\}1\.28onatis, above the MiniMax memory read by Claude and above thenn\-best oracle \(3\.94% vs\. 6\.08% WER, Table[5](https://arxiv.org/html/2607.26410#S5.T5)\)\.

#### Transfer follows the scaling properties\.

The transfer gain concentrates where there is recoverable headroom, as §[4](https://arxiv.org/html/2607.26410#S4)predicts\. Across the four domains of Table[5](https://arxiv.org/html/2607.26410#S5.T5), a memory helps on high\-WERatisbut is neutral to slightly harmful at the clean floor \(ls\_clean,ls\_other\), where little headroom remains\. The learned rules are general restraint rather than a domain\-specific fix, so the memory is reusable and can be applied to different models\.

Table 6:Voice Memory for X→\\toEn translation correction \(BLEU↑\\uparrow\)\.Voice Memoryimproves over the 1\-best in every direction \(ρ\>0\\rho\{\>\}0\); shown are the three directions where the formed memory also beats memory\-free reconciliation \(fr, where reconciliation alone is strongest, is discussed in Appendix[D](https://arxiv.org/html/2607.26410#A4)\)\.ρ\\rhois recoverable BLEU,\(vm−1\-best\)/\(oracle−1\-best\)\(\\textsc\{vm\}\-\\text\{1\-best\}\)/\(\\text\{oracle\}\-\\text\{1\-best\}\)\.
#### The mechanism generalizes beyond ASR\.

The recipe extends beyond speech recognition to speech\-translation error correction \(the GenTranslate setting\[hu2024gentranslate\]\), in which a corrector reconciles thenn\-best English\-translation hypotheses of a non\-English utterance and is scored by BLEU\[li2024investigating\]\. In Table[6](https://arxiv.org/html/2607.26410#S5.T6), we run the identicalVoice Memoryloop across four X→\\toEn directions\.Voice Memoryimproves BLEU over the 1\-best in every direction, recovering 13 to 32% of the oracle headroom, and the optimizer learns the same restraint it learns for ASR: it declines to encode reference content present in no hypothesis, and rejects edits that would raise validation BLEU only by copying a single reference\. A frozen corrector improving by editing a text memory is thus a general recipe fornn\-best language correction, not an ASR trick\. This is a 40\-utterance mini\-study with a single corrector; the full setup is in Appendix[D](https://arxiv.org/html/2607.26410#A4)\.

#### Memory provenance: quality beats coverage\.

Because the voice memory file is editable, its source matters more than its size\. A memory generated by Claude Sonnet 4\.6 reaches the best weighted WER at 7\.17%, ahead of the auto\-generated memory \(7\.52\), the expert hand\-crafted memory \(7\.43\), and the expert\-seeded memory \(7\.32\)\. The Claude memory is not the largest: at 5\.4 KB it is smaller than the 9\.0 KB auto\-generated and 7\.6 KB expert\-seeded variants\. Precise substitution entries and specific formatting rules govern correction quality more than raw coverage\.

## 6Robust Voice Memory for Noisy ASR

The restraint policy of §[4](https://arxiv.org/html/2607.26410#S4)also holds under acoustic noise\. RobustGER\[hu2024large\]makes a corrector noise\-robust by distilling thenn\-best diversity into a language\-space noise embedding and fine\-tuning LLaMA against it, which costs one training run per deployment\.Voice Memorywrites the same prior into a text file\. We run the same pipeline on Whispernn\-best from four noise families \(CHiME\-4, VoiceBank\-DEMAND, LibriSpeech\+FreeSound, NOIZEUS\), optimize one semantic\-gated memory per family, and read it at inference with no weight updates\. Our VoiceBank\-DEMAND 1\-best WERs match RobustGER Table 1 to within0\.10\.1point \(party22\.5922\.59vs\.22\.622\.6, baby7\.987\.98vs\.8\.08\.0\), so the two setups are comparable\.

#### The gain tracks how hard the speech is\.

As the scaling properties of §[4](https://arxiv.org/html/2607.26410#S4)predict,Voice Memoryhelps where 1\-best WER and oracle headroom are large, and does nothing where they are small \(Table[4](https://arxiv.org/html/2607.26410#S5.T4)\)\. It lowers WER on the genuinely degraded corpora, CHiME\-4 and NOIZEUS, where static GER over\-corrects\. On CHiME\-4 dev\-real, static GER raises WER \(7\.75→8\.137\.75\\\!\\to\\\!8\.13\) whileVoice Memorylowers it \(7\.75→7\.317\.75\\\!\\to\\\!7\.31,ρ=0\.31\\rho\{=\}0\.31\); on NOIZEUS\[hu2007subjective\]at55dB,Voice Memoryrecovers14\.51→12\.6014\.51\\\!\\to\\\!12\.60\(ρ=0\.36\\rho\{=\}0\.36; both in Table[4](https://arxiv.org/html/2607.26410#S5.T4)\)\. On VoiceBank\-DEMAND and LS\-FreeSound, which are read speech with noise added rather than recorded, restraint is already the right policy and the memory is near\-null \(partyρ=0\.03\\rho\{=\}0\.03\)\. The result shows that the same restraint policy holds under noise:Voice Memorymatches RobustGER\-style robustness on the degraded corpora with no fine\-tuning and no latent noise vector, and correctly abstains where there is nothing to recover\.

## 7Analysis: Meaning versus Surface Error

We consider that two questions sit underneath the scaling properties of §[4](https://arxiv.org/html/2607.26410#S4)\. First,*which*signal should the optimizer gate on: the metric we report \(i\.e\., WER and HER\), or the quantity we actually care about \(meaning\)? Second,*why*would a meaning\-based signal help at all? We answer both with measurements on the same frozen corrector, and both answers reinforce the central finding that good correction is mostly restraint\.

### 7\.1Meaning Is a Better Training Signal than WER

The validation gate in §[3\.2](https://arxiv.org/html/2607.26410#S3.SS2)can score candidates by any metric\. We compare two: aWER gate\(−WER\-\\mathrm\{WER\}on the selection split\) and asemantic gate\(mean cosine similarity between corrected and reference embeddings from the on\-device encoder\)\. The two gates see the same rollouts, the same candidate edits, and the same acceptance rule; they differ only in the scalar they maximize, which isolates the effect of the objective itself\. We optimize both over two seeds and report on the full test split \(Table[7](https://arxiv.org/html/2607.26410#S7.T7)\)\.

Table 7:Gate ablation, full test, two seeds\. Optimizing the memory for*semantic*preservation beats optimizing it for WER*on WER*, and is∼\\sim4×\\timesmore seed\-stable onls\_clean\(range 0\.03 vs\. 0\.17\)\.The semantic gate wins on WER across both domains and both seeds, and resists the optimization seed far better\. The mechanism follows from §[4](https://arxiv.org/html/2607.26410#S4): the optimal correction policy is mostly restraint, and a semantic objective rewards restraint*without*chasing the orthographic ghosts \(homophones, spelling variants\) that a WER objective is tempted to “fix\.” A WER gate, by construction, credits any edit that removes a surface mismatch\. This includes the meaning\-preserving ones a careful reader would leave alone; the semantic gate withholds that credit, so fewer such edits survive acceptance\. Optimizing the metric you report \(i\.e\., WER\) is thus*worse*than optimizing the quantity you care about \(meaning\), which WER only imperfectly tracks\. We frame this honestly: the margin is modest \(∼\\sim0\.2 points\), yet consistent in direction across domains and seeds, and the stability and meaning\-preservation gains are unambiguous\.

### 7\.2Language Relativity: a Measurable Surface to Meaning Gap

We provide additional study inspired byhoijer1954sapiron Whorf–Sapir hypothesis\. A semantic objective can beat WER on WER because the two quantities live on different axes, and we make this separation quantitative\. For each utterance we measure a*surface*distance and a*semantic*distance \(cos\\cosof sentence embeddings\)\. Regressing semantic on surface distance yields a*decoupling slope*; the fraction of surface\-error mass whose semantic distance falls below a small threshold is the*benign mass*\(Table[8](https://arxiv.org/html/2607.26410#S7.T8)\)\.

Table 8:A measurable surface to meaning gap\. The decoupling slope \(0\.38 to 0\.60,≪1\\ll 1\) means a unit of WER moves meaning by only∼\\simhalf a unit; 53 to 68% of residual error*mass*preserves meaning \(semantic distance<0\.15<0\.15\)\.The slope sits well below 1 in every domain: surface errors move meaning only weakly\. Most residual error mass is semantically benign\. This measured by an independent on\-device model, not asserted\. This is an operational, falsifiable statement of linguistic relativity for ASR: the text string and the underlying message are distinct, and WER conflates them\. It explains why over\-correction is so costly: the corrector trades meaning\-neutral surface forms for a lower apparent error count while leaving the message unchanged\. It also explains why a semantic gate generalizes better\. The practical consequence is a ceiling on how much residual WER is worth pursuing: once an edit moves only the surface form and not the meaning, chasing it trades risk of damage for no communicative gain\.

### 7\.3Error Analysis

The residual afterVoice Memorysplits into two buckets: genuine acoustic confusions \(named entities, rare words\) that no language prior can resolve, and annotation artifacts \(spelling variants, spacing\) that*should not*be “corrected\.” The first bucket is a limit of the cascade rather than of the memory\. The information needed to recover the token is simply absent from thenn\-best list\. The second is exactly where an unconstrained corrector does damage, and where the learned restraint pays off\. Theher/ρ\\rhodecomposition \(§[3\.3](https://arxiv.org/html/2607.26410#S3.SS3)\) separates these, and the benign\-mass measure \(§[7\.2](https://arxiv.org/html/2607.26410#S7.SS2)\) bounds how much of the remaining WER is worth chasing at all\. Appendix[E](https://arxiv.org/html/2607.26410#A5)gives eight representative such cases\.

## 8Conclusion

Voice Memorymakes a frozen corrector better without touching a single weight\. Everything the system learns lives in a small, auditable text memory read at inference time; the only difference between a worse corrector and a better one is that file\. A better agent here is simply a better text file\. The policy the file comes to encode is restraint, knowing which tokens to leave alone, and this is why we cast correction as a per\-utterance decision to act or abstain rather than as a sequence\-to\-sequence rewrite, with a listener that decides at stream time and a thinker that revises the memory asynchronously\.

The pieces compose into one picture\. The gain scales with recoverable headroom: the same policy that repairs far\-field speech holds still on clean read speech\. A memory written by one model family improves a reader from another, because explicit rules travel where weights cannot; the same trust can be trained into weights from audio\[wan2026speechhands\], butVoice Memoryacquires it from inference\-time feedback, with backpropagation\. The cheapest adaptation and personalization we know of for an ASR\-LLM cascade\. We would refocus the task itself: optimize not orthography but meaning, and decide not how often to act but when to hold still\[mullennix2011typicality,sheffert1995effects\]\.

## Limitations

Our main analyses use a single open\-source model with 428B parameters and 23B activated parameters,MiniMax\-M3\. Appendix[B](https://arxiv.org/html/2607.26410#A2)reproduces the core effect with an open Qwen3\-30B\-A3B corrector across ten datasets\.ρ\\rhoand the benign\-mass measure depend on the sentence encoder we use\. We reduce this dependence with a small, public, on\-device model with an average system latency of 610 ms, and by releasing the protocol and audio\-visual memory formation\[kim2026two,ghosh2024lipger\]\.

## Ethics Statement

Voice Memoryproduces a human\-readable artifact, which makes auditing and bias review easier than inspecting weight updates\. Domain memories can encode dataset\-specific conventions, such as spelling norms, that should not transfer across dialects or populations without review\. The benign\-mass analysis is one way to surface these conventions before deployment\.

## References

## Appendix AReproducibility and Hyperparameters

We default to 4 epochs, rollout batch 24, selection split 80, and textual learning rate 4 with cosine decay \(floor 2\), under the “strictly greater” validation gate\. Each domain uses three disjoint splits, and we report test results once\. All semantic scoring runs on\-device\.

## Appendix BBroad Study with an Open Corrector

The main body establishesVoice Memoryin depth with the MiniMax\-M3 corrector\. The effect is not an artifact of one model: an open, inference\-only corrector reproduces it at full breadth\. We pair Whisper\-v2\-Large\[radford2023robust\]as the frozen decoder with Qwen3\-30B\-A3B\[qwen2025qwen3\]\. A sparse Mixture\-of\-Experts that activates only≈\\approx3B of its 30B parameters per forward pass \(i\.e\., as the corrector\.\) The agent reads a per\-domainmemory\.mdat inference time and never updates a weight\. Across 16,108 test samples from 10 datasets, this open\-corrector breadth study complements the main\-body MiniMax\-M3 depth study\.

Table 9:Main results for the open corrector: full test set WER \(%\) across 10 benchmarks\. Raw ASR is the Whisper\-v2\-Large 1\-best;Voice Memory\(All Combined\) applies negative rules, filtered memory, domain prompts, and oracle gating; Few\-shot adds three in\-context correction examples per dataset\. TheWEIGHTEDrow is a single global error ratio across all datasets\.Table[9](https://arxiv.org/html/2607.26410#A2.T9)confirms the depth\-study pattern at breadth:Voice Memorycuts weighted WER from 8\.36 to 7\.47, and no dataset regresses below its raw baseline\. Gains concentrate where memory supplies domain vocabulary the corrector cannot otherwise reach the performance\.ATISfalls from 8\.40 to 3\.00 andCHiME\-4from 12\.69 to 10\.28, while acoustically ambiguous conversational speech \(CORAAL,SWBD\) moves little\.

The component ablation isolates which interventions earn that gain, and the answer is restraint\. Negative rules are the single load\-bearing component: alone, they drive weighted WER from 7\.66 to 7\.52, more than any other fix\. Their job is to stop the corrector from rewriting tokens that were already correct\. This is suppressing contraction edits, paraphrase, and digit\-word conversion, so the agent’s discovered policy is largely knowing when*not*to act\. Few\-shot examples add a further 0\.05 on top, reaching 7\.47\. Removing the memory entirely regresses the system to 7\.66, the level of having no negative rules at all, which quantifies the memory’s contribution as a knowledge source the corrector’s parametric weights do not supply\. A better agent here is simply a better text file\.

#### The irreducible error floor\.

Residual WER does not imply residual headroom\. We quantify a*tolerated WER*by forgiving two error classes that no text\-only corrector can resolve without the source document: out\-of\-vocabulary proper nouns and British/American spelling variants\. OnLS\-Clean, 42\.9% of residual errors are forgivable under these two rules, and accounting for structurally irreducible errors yields a practical hard floor of 0\.749%\. Most of whatVoice Memoryleaves uncorrected on clean read speech is unreachable in principle, not a failure of the agent’s policy\.

## Appendix CCorrector\-Agnostic Memory: a Claude Case Study

Because the memory is text, it need not be tied to the corrector that wrote it\. We test this with a second, independent frozen corrector applied purely at inference \(no fine\-tuning\) onatis,wsj,ls\_other, andls\_clean\. These runs are small\-sample and illustrative \(50 to 100 utterances\); we read them throughρ\\rho, which is sample\-internal, and leave a full\-test, same\-corrector comparison to future work\.

Three findings emerge \(Table[5](https://arxiv.org/html/2607.26410#S5.T5)\)\.\(i\) The skill ports across correctors\.A memory written by MiniMax lifts a different, stronger frozen reader it never trained with, and that reader*out\-restrains*the writer at the clean floor, with no over\-correction\.\(ii\) The corrector can form its own memory\.Running the validation\-gated loop of §[3\.2](https://arxiv.org/html/2607.26410#S3.SS2)with Claude as both corrector and optimizer yields, onatis, a memory that beats the MiniMax\-written one read by the same Claude \(3\.943\.94vs\.6\.086\.08,ρ=1\.28\\rho\{=\}1\.28, past the oracle\)\. For instance, no privileged optimizer is required when the domain has learnable patterns and headroom\.\(iii\) Transfer is roughly neutral\.Applying each memory across the four domains \(a4×44\{\\times\}4matrix\) leaves WER close to the own\-memory diagonal: the learned rules are general restraint, not domain overfits\. The boundaries are honest that self\-formation loses on number\-formatting\-heavywsj\(data\-hungry\) and at the LibriSpeech floor \(nothing to learn\) but the headline holds: the memory is the portable, corrector\-agnostic asset, and a strong reader can re\-form a better one where the domain affords it\.

## Appendix DX→\\toEn Voice Memory: a Translation Mini\-Study

Is the agentic memory mechanism specific to ASR, or does it carry to othernn\-best language tasks? We test it on*speech translation*error correction, the GenTranslate setting\[hu2024gentranslate,imai2025evaluating\]: a corrector reconciles thenn\-best English\-translation hypotheses of a non\-English source utterance into a single better translation, scored byBLEU\. We run the identical Voice Memory loop \(§[3\.2](https://arxiv.org/html/2607.26410#S3.SS2)\) a fixed LM as both corrector and BLEU\-gated optimizer on four X→\\toEn directions spanning three language families:de,ja\(CoVoST\-2 speech translation\) andzh\(FLEURS machine translation; CoVoST\-2 has no Chinese and FLEURS no French, so we take the best available source per language\)\. Each memory is formed on 10 rollout \+ 20 select items and read on a held\-out 40\-utterance test\.

Table[6](https://arxiv.org/html/2607.26410#S5.T6)shows the mechanism transfers\.Voice Memoryimproves BLEU over the 1\-best in every direction, recovering1313to32%32\\%of the oracle headroom, and the formed memory adds over memory\-free reconciliation on three of four languages\. The familiar boundary reappears onfr, where Claude’s reconciliation alone is already strong \(53\.1\) and the memory slightly over\-applies \(51\.1\) to restraint again\. Most telling are the formation traces: the optimizer repeatedly identified reference content present in*no*hypothesis \(place names, idioms, legal terms\) as unrecoverable and declined to encode guesses, and it*rejected*candidate edits that would have raised select BLEU only by copying a single reference\. The same enforced\-restraint discipline the ASR memories learn\. A frozen corrector improving by editing an auditable text skill is thus not an ASR trick but a general recipe fornn\-best language correction\. This is a 40\-utterance mini\-study with a single corrector; we leave full\-scale, multi\-corrector translation to future work\.

## Appendix EQualitative Case Studies

We show eight representative cases \(Table[10](https://arxiv.org/html/2607.26410#A5.T10)\) spanning theher/ρ\\rhotaxonomy:Voice Memoryfixes genuine errors, suppresses the harmful edits static GER introduces, and in the generative cases recovers a transcript present in no hypothesis\.

Table 10:Qualitative case studies across four ASR domains\. Each block shows the Whisper 1\-best, the static GER correction, theVoice Memorycorrection, and the gold reference \(verbatim, lowercased as scored\)\. The bracketed tag is theVoice Memoryedit category\.Generativecases showVoice Memoryproducing a transcript absent from every n\-best hypothesis \(e\.g\., recovering the spelled\-out forms b/b nfromsb/be in\)\.Harmful\-suppressedcases show static GER over\-correcting a clean 1\-best \(introducingme l, normalizing$66\.6, or deletingwords\) whileVoice Memorycorrectly holds back\.Helpfulcases showVoice Memoryrepairing a genuine error and beating the 1\-best WER\.