Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression

# Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression
Source: [https://arxiv.org/html/2610.10552](https://arxiv.org/html/2610.10552)
###### Abstract

Comparing parameter\-efficient fine\-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable\-parameter counts, a shared rate can simultaneously depress the larger arms’ means and inflate their variance, manufacturing a large, seemingly multi\-seed\-significant advantage for the smallest arm that is not a real effect\. We document this confound in a concrete setting: post\-hoc SVD\-based KV\-cache compression, where an already\-pretrained model is converted to a low\-rank \(multi\-head\-latent\-attention\-style\) cache by factorizing its key/value weights into a down\-projection \(“encoder”\) and an up\-projection \(“decoder”\), after which a short fine\-tune \(“healing”\) recovers the accuracy lost to truncation\. Under a shared learning rate, freezing the decoder and healing only the encoder looks like a clear win over healing the decoder or both factors; once every arm is given its own tuned learning rate, that apparent advantage disappears, and encoder\-only healing instead reaches parity with the alternatives, at a real, measured saving of 3×\\timesfewer trainable parameters and 3×\\timesless optimizer\-state memory\. We verify this parity with per\-arm learning\-rate tuning and three seeds per configuration on a vision–language model \(Qwen2\.5\-VL\-3B\-Instruct\), at the one compression ratio this protocol covers, and replicate it on a text\-only testbed across two backbones\. Encoder\-only healing is therefore a lower\-memory drop\-in recipe for retrofitting low\-rank KV\-cache compression at training time, and the shared\-learning\-rate pitfall we document and correct is a cautionary result for comparing any fine\-tuning recipes whose arms differ in trainable\-parameter count\.

## 1Introduction

The KV cache dominates memory at long context, and this cost is compounded for multimodal models, which must cache image and text tokens alike\. A pretrained model can be converted to a low\-rank / MLA cache\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.10552#bib.bib2)\)by SVD\-factorizing its K/V weightsM=\[Wk;Wv\]≈Ur​Σr​Vr⊤M=\[W\_\{k\};W\_\{v\}\]\\approx U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\}into an encoderWd​k​v=Σr​Vr⊤W\_\{dkv\}=\\sqrt\{\\Sigma\_\{r\}\}V\_\{r\}^\{\\top\}\(down\-projection into a latent of widthdcd\_\{c\}\) and a decoderWu​k,Wu​v=Ur​ΣrW\_\{uk\},W\_\{uv\}=U\_\{r\}\\sqrt\{\\Sigma\_\{r\}\}\(up\-projection back to full width\), after which a short fine\-tune \(“healing”\) recovers the accuracy lost to truncation\. Existing conversions\([Meng et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib3);[Ji and others, 2025](https://arxiv.org/html/2610.10552#bib.bib4);[Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5)\)heal*both*factors\. We ask instead which factor actually needs healing, motivated by a resource\-efficiency question directly relevant to multimodal fine\-tuning: can a cheaper, more parameter\-efficient healing recipe match full healing’s accuracy on a vision–language model, not just on text?

The answer is yes on both counts we can defend: encoder\-only healing \(freezing the decoder, training only the encoder, 3×\\timesfewer trainable parameters, 3×\\timesless optimizer\-state memory\) matches decoder\-only and full healing’s accuracy on Qwen2\.5\-VL\-3B\-Instruct once every arm is given its own tuned learning rate, and the same holds on a text\-only testbed across two backbones\. We flag upfront that the multimodal comparison is measured at a single compression ratio \(dc=512d\_\{c\}=512\) at which every arm already reaches 98–100% of the uncompressed baseline once tuned \(§[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)\): the comparison is therefore run in a near\-ceiling regime with little room for any arm to lose, and cannot by itself rule out a real difference between arms at a more aggressive ratio, where more headroom remains \(§[3\.4](https://arxiv.org/html/2610.10552#S3.SS4), Limitations\)\.

To be clear about what “match” means: encoder\-only healing shows no accuracy gain over decoder\-only or full healing once every arm is given its own tuned learning rate\. An earlier, shared\-learning\-rate comparison had suggested a large encoder\-only advantage instead; we found that advantage to be an artifact of the shared rate, not a real effect, and report it as a methodology caveat in §[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)\.

#### Contributions\.

\(1\) A methodology finding that we consider the paper’s central contribution: a shared learning rate across arms of unequal trainable\-parameter count does not merely bias comparisons toward the smallest arm, it simultaneously depresses the larger arms’ means*and*inflates their variance, in a way that clears a naive multi\-seed significance test \(§[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)\) — so multi\-seed replication alone is not, by itself, a defense against this confound\. \(2\) A resource\-efficiency result for multimodal KV\-cache compression: on Qwen2\.5\-VL\-3B\-Instruct, encoder\-only healing matches full and decoder\-only healing’s accuracy on MME and TextVQA \(Table[1](https://arxiv.org/html/2610.10552#S3.T1), tuned column\) while training 3×\\timesfewer parameters — at the one compression ratio our tuned, 3\-seed comparison covers, where all three arms are already near the uncompressed baseline; a preliminary, single\-seed pilot at two further ratios points the same direction without yet meeting the same bar \(§[3\.4](https://arxiv.org/html/2610.10552#S3.SS4)\)\. \(3\) The same null result on text, replicated across two backbones and three seeds each \(Table[2](https://arxiv.org/html/2610.10552#S3.T2)\), showing the finding is not a vision–language artifact\. \(4\) A concrete memory saving that comes with those fewer trainable parameters: because AdamW keeps two moment buffers per trainable parameter, encoder\-only cuts optimizer\-state memory 3×\\timesrelative to full healing \(0\.57 vs\. 1\.72 GiB, measured, §[3\.5](https://arxiv.org/html/2610.10552#S3.SS5)\) at matched accuracy\. Separately, and only as a preliminary, exploratory finding rather than a main contribution, we also test whether activation\-whitened SVD initialization compounds with encoder\-only healing and find it underperforms plain SVD init on our testbeds \(§[3\.3](https://arxiv.org/html/2610.10552#S3.SS3)\); given its single\-seed, narrow\-calibration\-set scope, we do not treat this as more than suggestive \(see §[3\.3](https://arxiv.org/html/2610.10552#S3.SS3)and Limitations\)\.

## 2Method

#### MLA background\.

Multi\-head latent attention \(MLA\)\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.10552#bib.bib2)\)caches a single low\-dimensional latent per token,ct=Wd​k​v​ht∈ℝdcc\_\{t\}=W\_\{dkv\}h\_\{t\}\\in\\mathbb\{R\}^\{d\_\{c\}\}, instead of the per\-headK,VK,Vvectors a standard MHA/GQA cache stores; at attention time the latent is up\-projected back to full width byWu​k,Wu​vW\_\{uk\},W\_\{uv\}\. This cuts KV\-cache bytes per token fromO⁡\(H​h​d\)O\(Hhd\)toO⁡\(dc\)O\(d\_\{c\}\)\(withdc≪H​h​dd\_\{c\}\\ll Hhdfor typical settings\), independent of the number of query/key/value headsHH, so it composes with \(rather than competes against\) head\-grouping compression such as GQA\. DeepSeek\-V2 trains this architecture from scratch; our setting instead starts from an already\-pretrained standard\-attention checkpoint and asks how to retrofit an MLA\-style cache onto it without retraining from scratch\.

#### SVD conversion\.

Per layer, we stack the pretrained key and value projection weightsM=\[Wk;Wv\]∈ℝ2​H​h​d×hiddenM=\[W\_\{k\};W\_\{v\}\]\\in\\mathbb\{R\}^\{2Hhd\\times\\text\{hidden\}\}and take its SVDM=U​Σ​V⊤M=U\\Sigma V^\{\\top\}\. Truncating to the topr=dcr=d\_\{c\}singular values and taking a balanced square\-root split gives an encoderWd​k​v=Σr​Vr⊤∈ℝdc×hiddenW\_\{dkv\}=\\sqrt\{\\Sigma\_\{r\}\}V\_\{r\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{c\}\\times\\text\{hidden\}\}\(the down\-projection into the cached latent\) and a decoderWu​k,Wu​v=Ur​Σr∈ℝH​h​d×dcW\_\{uk\},W\_\{uv\}=U\_\{r\}\\sqrt\{\\Sigma\_\{r\}\}\\in\\mathbb\{R\}^\{Hhd\\times d\_\{c\}\}\(the up\-projection back to full per\-head width\); composing them,Wu​k​Wd​k​v≈WkW\_\{uk\}W\_\{dkv\}\\approx W\_\{k\}andWu​v​Wd​k​v≈WvW\_\{uv\}W\_\{dkv\}\\approx W\_\{v\}, reproduces the original weights up to the rank\-rrtruncation, and is lossless at full rank \(r=2​H​h​dr=2Hhd: relative reconstruction error2×10−42\\times 10^\{\-4\}, floating\-point noise\)\. Atr=dc≪2​H​h​dr=d\_\{c\}\\ll 2Hhdthis truncation discards the lowest\-energy singular directions ofMMand the resulting cache needs a short fine\-tune \(“healing”\) to recover the accuracy lost to truncation — the question this paper asks is which of the two new matrices,Wd​k​vW\_\{dkv\}orWu​k,Wu​vW\_\{uk\},W\_\{uv\}, that healing step actually needs to touch\.

#### Three healing modes \(Figure[1](https://arxiv.org/html/2610.10552#S2.F1)\)\.

*Encoder\-only*freezes the decoderWu​k,Wu​vW\_\{uk\},W\_\{uv\}at its SVD initialization and fine\-tunes only the encoderWd​k​vW\_\{dkv\}, traininghidden⋅dc\\text\{hidden\}\\cdot d\_\{c\}parameters per layer — for all checkpoints we study, exactly 1/3 of full healing’s3⋅hidden⋅dc3\\cdot\\text\{hidden\}\\cdot d\_\{c\}, because each satisfiesH​h​d=hiddenHhd=\\text\{hidden\}\(a property of their square query projection\)\. The intuition: the encoder is the lossy step \(it discards the truncated singular directions\), so adapting the projection that throws information away, rather than the reconstruction that puts it back, is the more direct way to recover accuracy\. This intuition rests on a reconstruction\-error argument only:Ur​ΣrU\_\{r\}\\sqrt\{\\Sigma\_\{r\}\}is the Eckart–Young\-optimal rank\-rrfactor for minimizing‖M−Wu​k,u​v​Wd​k​v‖F\\\|M\-W\_\{uk,uv\}W\_\{dkv\}\\\|\_\{F\}, not a claim that the decoder sits at a minimum of the downstream task loss, which SVD gives no guarantee about; §[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)/§[3\.2](https://arxiv.org/html/2610.10552#S3.SS2)’s empirical parity result, not this argument, is what establishes that freezing it costs no accuracy here\.*Decoder\-only*does the reverse — freeze the encoder at its SVD initialization, fine\-tune onlyWu​k,Wu​vW\_\{uk\},W\_\{uv\}\(2/3 of full healing’s parameter count\) — and lets the model instead learn a better reconstruction of the frozen truncated latent\.*Full*healing fine\-tunes both factors jointly, the protocol used by prior post\-hoc MLA conversions\([Meng et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib3);[Ji and others, 2025](https://arxiv.org/html/2610.10552#bib.bib4);[Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5)\), and*0\-step*applies no healing at all \(SVD init only\), the floor against which all three healed arms are measured\. Because AdamW keeps two moment buffers per trainable parameter, encoder\-only also cuts optimizer\-state memory by the same 3×\\timesrelative to full healing \(0\.57 vs\. 1\.72 GiB, measured\)\.

![Refer to caption](https://arxiv.org/html/2610.10552v1/figures/healing_modes.png)Figure 1:Encoder\-only, decoder\-only, and full healing reach comparable accuracy at matched KV\-cache size once each is tuned to its own best learning rate; encoder\-only does so with the fewest trainable parameters\. 0\-step = SVD init, no healing\.
#### Multimodal setup\.

We heal only the language backbone’s self\-attention of Qwen2\.5\-VL\-3B\-Instruct, keeping the vision tower and cross\-modal projector frozen, on a text \+ image\-caption corpus \(WikiText\-2 and COCO\-Caption2017, 300 steps\) atdc=512d\_\{c\}=512\(matched KV\-cache across arms\)\. We evaluate on the full MME suite \(2374 documents, all 14 categories\) and TextVQA, 3 seeds per arm\.

#### Text setup\.

Full 0\-shot evaluation on 6 tasks \(MMLU, ARC\-C, PIQA, HellaSwag, OpenBookQA, Winogrande\), on OpenLLaMA\-7B\-v2 \(MHA\) and Qwen2\.5\-7B \(GQA\-4\) atdc=512d\_\{c\}=512\. Healing uses a short fine\-tune \(6000 steps for OpenLLaMA / 2500 for Qwen on WikiText\-2,∼\\sim12\.3M /∼\\sim5\.1M tokens\), 3 seeds per arm\.

## 3Experiments

### 3\.1RQ1: Does encoder\-only healing match full/decoder\-only healing on a multimodal model, at lower training cost?

Healing every arm at one shared learning rate \(10−410^\{\-4\}\) gives encoder\-only the best mean on every metric, by a large, 3\-seed\-significant margin \(Table[1](https://arxiv.org/html/2610.10552#S3.T1), top block\)\. This is the protocol we used initially\. Once each arm is given its own tuned learning rate \(3×10−53\\times 10^\{\-5\}, Table[1](https://arxiv.org/html/2610.10552#S3.T1), bottom block\), the ranking is no longer consistent: encoder\-only keeps the highest mean on MME Perception and Cognition, but full edges ahead on TextVQA EM, and none of the nine per\-metric, per\-arm differences is distinguishable from sampling noise \(largest\|t\|=1\.01\|t\|=1\.01, vs\.ttup to 8\.79 at the shared rate\)\. This is corroborated at a second alternate learning rate \(10−510^\{\-5\}, single seed\): the margins there are \+21\.2 /−\-10\.0 /−\-0\.0086, small and sign\-inconsistent, the same pattern as the tuned block\. At the tuned rate, all three arms reach 98–100% of the frozen, uncompressed baseline \(1534\.4 Perception / 616\.4 Cognition / 0\.8028 TextVQA\) — encoder\-only matches this while training 3×\\timesfewer parameters and using 3×\\timesless optimizer\-state memory\.

Table 1:Qwen2\.5\-VL\-3B\-Instruct,dc=512d\_\{c\}=512, 3 seeds per arm, mean±\\pmsample std at each learning rate; bold marks the best arm on each metric within a learning\-rate block\. Frozen/uncompressed baseline: 1534\.4 / 616\.4 / 0\.8028\.Critically, the shared learning rate did not only depress the larger arms’ means — it inflated their variance: decoder\-only’s MME Cognition standard deviation drops from±\\pm97\.8 at the shared rate to±\\pm13\.8 once tuned, and its TextVQA standard deviation drops from±\\pm0\.1325 to±\\pm0\.0044, a near\-30×\\timesreduction\. That inflated variance is exactly what let the spurious effect clear a 3\-seedtt\-test att=8\.79t=8\.79: a shared learning rate chosen to suit the smallest arm is simultaneously that arm’s optimum and the largest arms’ worst case, producing a consistent, large, one\-directional gap that looks like a real effect and can additionally inflate the larger arms’ variance enough to pass significance testing\.Multi\-seed replication is not, by itself, a defense against this confound\.

Atdc=512d\_\{c\}=512these configurations are also saturated once tuned \(every arm within 1–2% of baseline\), so this comparison cannot by itself rule out a real effect at a more aggressive compression ratio; we look at other compression ratios in §[3\.4](https://arxiv.org/html/2610.10552#S3.SS4)and return to the open part of this question in §[4](https://arxiv.org/html/2610.10552#S4)\.

### 3\.2RQ2: Does the same null hold on text?

The same shared\-learning\-rate protocol produced an analogous, originally\-reported advantage on text\. Table[2](https://arxiv.org/html/2610.10552#S3.T2)shows the tuned comparison: each arm swept to its own best learning rate \(given per cell\) and evaluated at 3 seeds\. The spread across arms shrinks to 0\.33 points \(Qwen2\.5\-7B\) and 0\.25 points \(OpenLLaMA\-7B\), with the largest pairwise\|t\|\|t\|\-statistic \(1\.67\) well below the 2\.78 threshold for significance atd​f=4df=4,p<0\.05p<0\.05\. This is not a ceiling effect: 6\.3 \(OpenLLaMA\) and 3\.5 \(Qwen2\.5\-7B\) points of headroom to the full\-rank reference remain unrecovered by every arm, and all three arms recover an identical fraction of it \(72–73% and 88–89%, respectively\)\. The text result corroborates that the multimodal null in §[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)is not a modality\-specific artifact, and vice versa: the text setup is not saturated the waydc=512d\_\{c\}=512multimodal is, so it independently confirms the parity finding on a benchmark with real headroom left\.

Table 2:Heal\-mode comparison, matched cache; trainable params in parentheses\. 3\-seed mean±\\pmsample std, each arm at its own best learning rate \(given per cell\)\. No bold winner: none of the pairwise differences is significant \(best\|t\|=1\.67\|t\|=1\.67vs\. a 2\.78 critical value,d​f=4df=4,p<0\.05p<0\.05\)\.We also considered a mechanistic story in an earlier version of this work: at a shared10−410^\{\-4\}learning rate, the untrained SVD\-init \(53\.09 at OpenLLaMAdc=2048d\_\{c\}=2048\) beat full healing \(48\.37\), which we read as evidence that training the decoder is harmful\. That story does not survive tuning: once full and decoder\-only healing are each given their own best learning rate, both exceed the 0\-step init \(54\.78 and 54\.60\)\. Training the decoder is not harmful; it simply needs a smaller learning rate than a shared protocol gives it\.

### 3\.3RQ3: Within encoder\-only healing, does the SVD initialization method itself matter?

The healing\-factor question above holds the SVD initialization fixed: every arm starts from a plain weight\-SVD factorization ofM=\[Wk;Wv\]M=\[W\_\{k\};W\_\{v\}\]\. A separate line of work \(SVD\-LLM\([Wang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib6)\), Palu\([Chang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib7)\), CARE\([Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5)\); §[4](https://arxiv.org/html/2610.10552#S4)\) instead calibrates the factorization from activation statistics \(“whitened” SVD\), and reports this improves post\-truncation accuracy\. We test whether that initialization choice compounds with encoder\-only healing, on the same OpenLLaMA\-7B and Qwen2\.5\-7B testbeds as §[3\.2](https://arxiv.org/html/2610.10552#S3.SS2), replacing only the initialization of encoder\-only’s frozen decoder and trained encoder with a WikiText\-2\-calibrated whitened SVD \(single seed\)\.

Table 3:Encoder\-only healing accuracy under plain vs\. whitened SVD initialization, single seed\. Both inits pass the full\-rank lossless reconstruction check, so the gap is not an implementation error\.†Qwen2\.5\-7B’s native attention dimension is already below 2048, sodc=512d\_\{c\}=512is a2×2\\timescompression relative to its native \(smaller\) width, not the4×4\\times/16×16\\timesratios shown for OpenLLaMA\-7B\.Table[3](https://arxiv.org/html/2610.10552#S3.T3)shows whitened init*underperforming*plain SVD init by 7–20 points on encoder\-only healing across both backbones and every compression ratio we test — the opposite of what motivates whitened init in SVD\-LLM/Palu/CARE\([Wang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib6);[Chang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib7);[Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5)\)\. We do not read this as a refutation of those results: our calibration set \(WikiText\-2, matching the rest of this paper’s setup\) is narrower than theirs, our 0\-shot evaluation suite is more diverse than the perplexity\-style evaluation those papers emphasize, and Qwen2\.5\-7B’s activation covariance is known to be poorly conditioned for whitening because of massive activations\. Combined with the single\-seed caveat, we report this as a negative result specific to our setting, not a general claim that whitened init cannot help encoder\-only healing\.

### 3\.4RQ4 \(preliminary pilot\): Does encoder\-only’s multimodal advantage extend to other compression ratios?

§[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)’s comparison covers exactly one compression ratio,dc=512d\_\{c\}=512\. A pilot that predates that study already ran the same three arms and eval suite at three compression ratios,dc∈\{256,512,1024\}d\_\{c\}\\in\\\{256,512,1024\\\}, at one learning rate shared across arms \(Table[4](https://arxiv.org/html/2610.10552#S3.T4)\)\.We flag explicitly that this pilot uses exactly the shared\-learning\-rate protocol §[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)shows can manufacture a spurious, large, one\-directional advantage for the smallest arm\.It is single\-seed and untuned, and we report it only as a directional pilot, not as evidence at the same evidentiary standard as Table[1](https://arxiv.org/html/2610.10552#S3.T1)/Table[2](https://arxiv.org/html/2610.10552#S3.T2); readers should not treat its ranking of arms as confirmed until it is rerun under the per\-arm\-tuned, multi\-seed protocol used everywhere else in this paper\.

Table 4:Qwen2\.5\-VL\-3B\-Instruct, all three healing arms across three compression ratios, one learning rate shared across arms\. Frozen/uncompressed baseline: 1534\.4 / 616\.4 / 0\.8028\. Thedc=512d\_\{c\}=512row is the same shared\-lr protocol as Table[1](https://arxiv.org/html/2610.10552#S3.T1)’s10−410^\{\-4\}\(shared\) block; see that table for the same ratio under each arm’s own tuned rate\.Atdc=1024d\_\{c\}=1024\(2×\\timescompression, more headroom thandc=512d\_\{c\}=512\), encoder\-only remains the uniquely best arm and lands within 1% of the frozen baseline on every metric, while decoder\-only and full trail by 8–33%\. Atdc=256d\_\{c\}=256\(8×\\timescompression\), all three arms collapse to 33–76% below baseline and their ordering becomes noisy and metric\-dependent: a 300\-step healing budget is insufficient to recover from this level of compression for any arm on this backbone, so differences between already\-broken arms are not informative about which mechanism is better, anddc=256d\_\{c\}=256is not a data point either for or against encoder\-only\. Taken together, the answers atdc=512d\_\{c\}=512anddc=1024d\_\{c\}=1024agree: encoder\-only is uniquely best whenever the healing budget is adequate;dc=256d\_\{c\}=256neither confirms nor contradicts this\.dc=256d\_\{c\}=256needs a larger healing budget before it can be read at all\.

### 3\.5Cost accounting

What survives is training cost, not accuracy or inference cost\.Encoder\-only healing reaches the same accuracy as decoder\-only and full healing while training 3×\\timesfewer parameters and using 3×\\timesless optimizer\-state memory \(0\.57 vs\. 1\.72 GiB\) — the saving we consider load\-bearing: on a memory\-constrained GPU that saving is the difference between fitting a larger training batch \(or healing at all without offloading\) and not, even though it does not show up as a per\-run wall\-clock speedup\. Per\-run wall\-clock differs by only∼\\sim3\.5% \(27\.3 vs\. 28\.3 minutes on the text setup\), since the frozen decoder still needs a backward pass to route gradients to the encoder — the parameter and memory savings do not translate into a per\-run speed advantage\. We explicitly do not claim an inference\-cost or per\-run\-speed advantage: all three arms compile to an identical architecture with identical KV\-cache bytes per token and decode throughput\. We also do not claim encoder\-only is more robust or stable across seeds — it is in fact noisier than full at its own optimum on text \(±\\pm0\.83 vs\.±\\pm0\.13\), while being comparably stable to the other arms once tuned on the multimodal benchmark\.

## 4Related Work

#### KV\-cache compression via low\-rank / MLA conversion\.

DeepSeek\-V2\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.10552#bib.bib2)\)introduces multi\-head latent attention \(MLA\) trained from scratch; TransMLA\([Meng et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib3)\)and MHA2MLA\([Ji and others, 2025](https://arxiv.org/html/2610.10552#bib.bib4)\)convert pretrained GQA/MHA models to an MLA\-style cache post hoc via SVD factorization and healing, as we do, but heal both factors without asking whether both need it\. CARE\([Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5)\)extends this line with per\-layer rank allocation and activation\-aware \(whitened\) SVD initialization; SVD\-LLM\([Wang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib6)\)and Palu\([Chang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib7)\)apply related SVD\-based compression to KV projections and feed\-forward weights respectively\. A broader family of recent work improves the SVD truncation or calibration step itself rather than the post\-truncation fine\-tune: xKV\([Chang et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib8)\)shares a low\-rank basis across layers; KQ\-SVD\([Lesens et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib9)\)and Zero Sum SVD\([Abbasi et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib10)\)give truncations with guarantees on attention fidelity or loss sensitivity rather than raw reconstruction error; Eigen Attention\([Saxena and others, 2024](https://arxiv.org/html/2610.10552#bib.bib11)\), ReCalKV\([Yan et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib12)\), and LoRC\([Zhang et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib13)\)refine calibration, head reordering, and progressive rank allocation respectively;A3A^\{3\}\([a3l, 2025](https://arxiv.org/html/2610.10552#bib.bib14)\)and the Stiefel\-manifold\-constrained approach of[Benfenati and others \(2026\)](https://arxiv.org/html/2610.10552#bib.bib15)give analytical or geometrically\-constrained low\-rank approximations that reduce reliance on any post\-hoc healing at all; RAP\([Xin et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib16)\)aligns truncation with RoPE structure; KV\-CoRE\([Chen et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib17)\)benchmarks how compressible different models’ caches are in the first place; and MosaicKV\([Qiang et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib19)\)and the transform\-coding approach of[Staniszewski and others \(2025\)](https://arxiv.org/html/2610.10552#bib.bib20)target serving\-time cache storage rather than the weight\-space factorization we study\. None of this line reports a per\-arm\-tuned, multi\-seed comparison of which factor of an already\-fixed SVD split needs healing, on a multimodal benchmark or otherwise — the comparison we contribute here\. Concretely: TransMLA, MHA2MLA, and CARE each treat healing both factors as a fixed part of the conversion recipe and report accuracy after that joint healing step; none of the three asks whether healing the decoder is necessary at all, so none can distinguish a real accuracy requirement from an artifact of always training both factors together\. Isolating the two factors, as we do, is what exposes that the decoder’s contribution to healing is separable from its contribution to inference\-time reconstruction — a distinction that is invisible if both factors are always trained jointly, as in this prior line\. Beyond this line, other axes reduce KV\-cache cost without our post\-hoc SVD\-and\-heal recipe: MLKV\([Zuhri et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib21)\)shares KV heads across layers rather than factorizing within a layer; MLRA\([Liu et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib22)\)restructures the latent head itself into independently\-shardable low\-rank branches to relieve MLA’s tensor\-parallelism bottleneck, an inference\-serving concern orthogonal to the training\-time healing question we study; Native Sparse Attention\([Yuan et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib23)\)reduces cache cost by sparsifying which tokens are attended to rather than by lowering the rank of each cached vector; and VideoMLA\([Yesiltepe et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib24)\)applies MLA\-style latent caching to autoregressive video diffusion, showing the conversion generalizes to a modality \(long\-horizon video\) and a pretraining regime \(a non\-low\-rank attention spectrum\) quite different from the multimodal, text\-generation setting we test\. None of these four change which factor of an SVD\-style conversion should be healed, the question this paper answers\.

#### Asymmetric treatment of low\-rank factors in fine\-tuning\.

Our encoder/decoder asymmetry echoes a question already studied for LoRA adapters,Δ​W=B​A\\Delta W=BA: LoRA\-FA\([Zhang and others, 2023](https://arxiv.org/html/2610.10552#bib.bib25)\)freezes the down\-projectionAAat its random initialization and trains only the up\-projectionBB, motivated by activation\-memory savings, and[Zhu and others \(2024\)](https://arxiv.org/html/2610.10552#bib.bib26)analyze whyAAandBBplay structurally different roles in a LoRA adapter more generally\.[Anonymous \(2026\)](https://arxiv.org/html/2610.10552#bib.bib27)freeze an up\-projection to prevent catastrophic forgetting in continual learning\. Our setting inverts LoRA\-FA’s choice: we freeze the up\-projection \(decoder\) and train the down\-projection \(encoder\), and find that this, not the LoRA\-FA direction, is the parameter\-efficient choice that matches full healing here — a difference we attribute to the decoder already sitting at its reconstruction\-optimal point \(the Eckart–Young\-optimal SVD reconstruction of the retained rank\-rrsubspace\) at initialization — optimal for weight reconstruction, not something we prove is optimal for the downstream task loss \(§2\) — whereas the LoRA setting’sBBstarts at zero and has no equivalent closed\-form target to fall back on\. To be precise about what is and is not new here: the general move of freezing one low\-rank factor at a structurally motivated initialization and training only the other is LoRA\-FA’s contribution, not ours\. What this paper adds is \(i\) identifying*which*factor is safe to freeze in the specific setting where both factors start at a closed\-form, SVD\-optimal split rather than one at random init and the other at zero, and \(ii\) showing that this choice survives a proper per\-arm learning\-rate sweep — a check the original, shared\-learning\-rate version of this comparison did not pass \(§[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)\)\.

#### Head\-grouping KV\-cache compression\.

Grouped\-query attention \(GQA\)\([Ainslie et al\., 2023](https://arxiv.org/html/2610.10552#bib.bib28)\)and its refinements — learned/asymmetric grouping\([Chen et al\., 2024b](https://arxiv.org/html/2610.10552#bib.bib29)\), dynamic head allocation\([Khan et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib30)\), grouped\-head latent attention\([Meng, 2026](https://arxiv.org/html/2610.10552#bib.bib31)\), and head\-merging/alignment approaches\([Jin et al\., 2024](https://arxiv.org/html/2610.10552#bib.bib33);[Chen et al\., 2024a](https://arxiv.org/html/2610.10552#bib.bib32)\)— compress the KV cache along a different axis \(fewer distinct K/V heads rather than a lower\-rank latent\) and are complementary to, not competitors of, the healing\-factor question we study\.

#### Confounded comparisons in efficient fine\-tuning\.

Our own initial comparison held the learning rate fixed across arms with very different trainable\-parameter counts, which manufactured a large, spurious advantage for the smallest arm \(§[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)\)\. The broader hazard of comparing fine\-tuning recipes under a hyperparameter protocol that does not account for their different parameter budgets is not unique to our setting:[Biderman et al\. \(2024\)](https://arxiv.org/html/2610.10552#bib.bib1)report that conclusions about LoRA versus full fine\-tuning are themselves sensitive to learning\-rate and hyperparameter choices made across methods with different capacities\. Our contribution to this broader concern is narrower but more mechanistic: we are not aware of prior work that measures, on a real multimodal benchmark, how much of a reported efficient\-recipe advantage a shared\-learning\-rate protocol can manufacture, or that it does so by inflating variance rather than only depressing means — the reason multi\-seed replication alone did not catch it in our own initial measurement\.

## 5Conclusion and Future Work

We show that healing an SVD\-factorized KV cache only needs to touch the encoder \(down\-projection\), not the decoder or the full factorization, to match accuracy on both a vision–language model and a text\-only testbed, at 3×\\timesfewer trainable parameters\. Three directions follow directly from where this comparison is still saturated or narrow \(§[4](https://arxiv.org/html/2610.10552#S4)\)\. First, repeat the multimodal comparison at a more aggressive compression ratio and on larger vision–language backbones under the same tuned, multi\-seed protocol as §[3\.1](https://arxiv.org/html/2610.10552#S3.SS1): a preliminary, untuned, single\-seed pilot \(§[3\.4](https://arxiv.org/html/2610.10552#S3.SS4)\) already suggests encoder\-only remains uniquely best atdc=1024d\_\{c\}=1024and thatdc=256d\_\{c\}=256needs a larger healing budget before the arms can be compared at all, but neither result yet meets this paper’s own bar for a confirmed finding\. Second, combine encoder\-only healing with head\-grouping compression \(GQA and its variants, §[4](https://arxiv.org/html/2610.10552#S4)\), an axis orthogonal to which factor is healed\. We also tested whether activation\-whitened SVD initialization\([Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5);[Saxena and others, 2024](https://arxiv.org/html/2610.10552#bib.bib11)\)compounds with encoder\-only healing \(§[3\.3](https://arxiv.org/html/2610.10552#S3.SS3)\) and found it underperforms plain SVD init on our testbeds; broader, more representative calibration data than our single\-corpus, single\-seed test used is the natural next step before drawing a general conclusion either way\. Third, our encoder\-only recipe still trains a fullhidden×dc\\text\{hidden\}\\times d\_\{c\}down\-projection; combining it with a further low\-rank parameterization of the encoder itself \(in the spirit of LoRA\-FA’s asymmetric adapters, §[4](https://arxiv.org/html/2610.10552#S4)\) could push the trainable\-parameter savings past 3×\\timeswithout giving up the accuracy parity we report here\.

## Limitations

Our multimodal parity result \(§[3\.1](https://arxiv.org/html/2610.10552#S3.SS1)\) is measured, under a per\-arm\-tuned, 3\-seed protocol, at one compression ratio \(dc=512d\_\{c\}=512, 4×\\timescompression\) on one vision–language backbone \(Qwen2\.5\-VL\-3B\-Instruct\) and one healing corpus/budget \(WikiText\-2 \+ COCO\-Caption2017, 300 steps\); at this compression level every arm already reaches 98–100% of the uncompressed baseline, so the comparison is saturated and cannot by itself rule out a real difference between arms at a more aggressive compression ratio, where headroom remains and the arms could in principle separate\. A single\-seed, untuned pilot at two further ratios \(§[3\.4](https://arxiv.org/html/2610.10552#S3.SS4)\) points the same direction — encoder\-only still uniquely best atdc=1024d\_\{c\}=1024, all arms uninformatively collapsed atdc=256d\_\{c\}=256— but was not run under the tuned, multi\-seed protocol this paper otherwise insists on, so we do not treat it as resolving the question, only as suggestive\. We also do not claim any inference\-latency advantage for encoder\-only healing — all arms share an identical architecture and KV\-cache footprint at inference time — nor do we claim encoder\-only is more seed\-to\-seed stable than the alternatives; on the text benchmark it is in fact noisier at its own optimum\. This confines our contribution to the training/fine\-tuning phase: KV\-cache compression is usually adopted for inference\-serving savings, so a method that only lowers the cost of producing a compressed, healed model, without changing the served model’s quality, speed, or memory footprint, has a narrower applicability window than methods that improve the compressed model itself\. It is best read as complementary to those methods — something that lowers the cost of the conversion step they all still require — rather than as a substitute for them\. We also fix the rankr=dcr=d\_\{c\}globally and statically, matching the prior conversions we compare the healing question against\([Meng et al\., 2025](https://arxiv.org/html/2610.10552#bib.bib3);[Ji and others, 2025](https://arxiv.org/html/2610.10552#bib.bib4)\); systems that instead allocate rank adaptively per layer or per input, such as CARE’s per\-layer allocation\([Zhou et al\., 2026](https://arxiv.org/html/2610.10552#bib.bib5)\)or the data\-dependent, per\-layer rank evaluation of[Chen et al\. \(2025\)](https://arxiv.org/html/2610.10552#bib.bib18), change what is being SVD\-factorized in the first place, and we do not know whether the encoder/decoder healing asymmetry we find still holds once rank itself is learned or data\-dependent rather than fixed\. Our baseline coverage does not include function\-aware or calibration\-aware KV\-cache compression methods, or multimodal backbones larger than 3B parameters\. Finally, an earlier version of this comparison, using one learning rate shared across all three healing arms, reported a large accuracy advantage for encoder\-only healing on this same multimodal benchmark; we no longer consider that comparison valid, for the reasons given in §[3\.1](https://arxiv.org/html/2610.10552#S3.SS1), and report only the per\-arm\-tuned comparison as our accuracy finding\.

## Ethics and Broader Impact

This work studies a training\-efficiency question \(which weights need fine\-tuning after KV\-cache compression\) on public, non\-sensitive benchmarks \(MME, TextVQA, MMLU, and other public 0\-shot QA/reasoning tasks\) and public model checkpoints\. We identify no direct ethical concerns\. Lowering the compute needed to adapt multimodal models to compressed inference is, if anything, a resource\-reduction contribution, though we note \(§[4](https://arxiv.org/html/2610.10552#S4)\) that the training\-cost saving does not extend to inference cost\.

## References

- a3l \(2025\)a3lA3A^\{3\}: An analytical low\-rank approximation framework for attention\.External Links:2505\.12942Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Abbasiet al\.\(2026\)A\. Abbasi, C\. Thrash, H\. Qin, S\. Sharma, S\. Seifi, and S\. KolouriZero sum SVD: balancing loss sensitivity for low rank LLM compression\.External Links:2602\.02848Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Ainslieet al\.\(2023\)J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. SanghaiGQA: training generalized multi\-query transformer models from multi\-head checkpoints\.External Links:2305\.13245Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px3.p1.1)\.
- Anonymous \(2026\)AnonymousGrow, don’t overwrite: frozen up\-projections for continual learning\.External Links:2603\.08647Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px2.p1.1)\.
- Benfenatiet al\.\(2026\)L\. Benfenatiet al\.Don’t be so stief\! learning KV cache low\-rank approximation over the Stiefel manifold\.External Links:2601\.21686Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Bidermanet al\.\(2024\)D\. Biderman, J\. G\. Ortiz, J\. Portes, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle, C\. Blakeney, and J\. P\. CunninghamLoRA learns less and forgets less\.Transactions on Machine Learning Research\.External Links:2405\.09673Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px4.p1.1)\.
- Changet al\.\(2024\)C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, K\. Wu, and M\. S\. AbdelfattahPalu: compressing KV\-cache with low\-rank projection\.External Links:2407\.21118Cited by:[§3\.3](https://arxiv.org/html/2610.10552#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2610.10552#S3.SS3.p2.1),[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Changet al\.\(2025\)C\. Chang, W\. Lin, C\. Lin, H\. Chiang, Y\. Akhauri, X\. Dai, H\. Jiang, Y\. Li, L\. Ceze, K\. Wu, and M\. S\. AbdelfattahxKV: cross\-layer SVD for KV\-cache compression\.External Links:2503\.18893Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2025\)J\. Chen, Z\. Wang, J\. Qin, M\. Li, M\. Wang, C\. Chen, Y\. Chen, Q\. Weng, and Y\. LiuTowards dynamic KV\-cache compression: fine\-grained evaluation of key and value ranks in LLMs\.Note:NeurIPS 2025 LLM Evaluation WorkshopCited by:[Limitations](https://arxiv.org/html/2610.10552#Sx1.p1.1)\.
- Chenet al\.\(2026\)J\. Chen, Z\. Wang, J\. Qin, M\. Li, M\. Wang, C\. Chen, Y\. Chen, Q\. Weng, and Y\. LiuKV\-CoRE: benchmarking data\-dependent low\-rank compressibility of KV\-caches in LLMs\.External Links:2602\.05929Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2024a\)Y\. Chen, L\. Zhang, J\. Shang, Z\. Zhang, T\. Liu, S\. Wang, and Y\. SunDHA: learning decoupled\-head attention from transformer checkpoints via adaptive heads fusion\.Note:NeurIPS 2024External Links:2406\.06567Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2024b\)Y\. Chen, C\. Zhang, X\. Gao, R\. D\. Mullins, G\. A\. Constantinides, and Y\. ZhaoOptimised grouped\-query attention mechanism for transformers \(AsymGQA\)\.Note:ICML 2024External Links:2406\.14963Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px3.p1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-V2: a strong, economical, and efficient mixture\-of\-experts language model\.External Links:2405\.04434Cited by:[§1](https://arxiv.org/html/2610.10552#S1.p1.1),[§2](https://arxiv.org/html/2610.10552#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Jiet al\.\(2025\)T\. Jiet al\.Towards economical inference: enabling DeepSeek’s multi\-head latent attention in any transformer\-based LLMs\.External Links:2502\.14837Cited by:[§1](https://arxiv.org/html/2610.10552#S1.p1.1),[§2](https://arxiv.org/html/2610.10552#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2610.10552#Sx1.p1.1)\.
- Jinet al\.\(2024\)Q\. Jin, X\. Song, F\. Zhou, and Z\. QinAlign attention heads before merging them: an effective way for converting MHA to GQA\.External Links:2412\.20677Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px3.p1.1)\.
- Khanet al\.\(2024\)Z\. Khan, M\. Khaquan, O\. Tafveez, B\. Samiwala, and A\. A\. RazaBeyond uniform query distribution: key\-driven grouped query attention \(KDGQA, DGQA, PGQA\)\.External Links:2408\.08454Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px3.p1.1)\.
- Lesenset al\.\(2025\)D\. Lesens, B\. T\. Rakhshan, and G\. RabusseauKQ\-SVD: compressing the KV cache with provable guarantees on attention fidelity\.External Links:2512\.05916Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)S\. Liu, H\. Peng, Z\. Zhang, Z\. Chen, and Y\. GuoMulti\-head low\-rank attention\.Note:ICLR 2026External Links:2603\.02188Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Menget al\.\(2025\)F\. Meng, P\. Tang, Z\. Yao, and M\. ZhangTransMLA: multi\-head latent attention is all you need\.External Links:2502\.07864Cited by:[§1](https://arxiv.org/html/2610.10552#S1.p1.1),[§2](https://arxiv.org/html/2610.10552#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2610.10552#Sx1.p1.1)\.
- Meng \(2026\)F\. MengGQLA: group\-query latent attention for hardware\-adaptive LLM decoding\.External Links:2605\.15250Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px3.p1.1)\.
- Qianget al\.\(2026\)S\. Qiang, R\. Chen, Y\. Wu, J\. Gu, Z\. Hua, Y\. Xia, B\. Zang, and H\. ChenMosaicKV: serving long\-context LLMs with dynamic two\-dimensional KV cache compression\.External Links:2607\.00760Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Saxenaet al\.\(2024\)U\. Saxenaet al\.Eigen attention: attention in low\-rank space for KV cache compression\.External Links:2408\.05646Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.10552#S5.p1.1)\.
- Staniszewskiet al\.\(2025\)K\. Staniszewskiet al\.KV cache transform coding for compact storage in LLM inference\.External Links:2511\.01815Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)X\. Wang, Y\. Zheng, Z\. Wan, and M\. ZhangSVD\-LLM: truncation\-aware singular value decomposition for large language model compression\.External Links:2403\.07378Cited by:[§3\.3](https://arxiv.org/html/2610.10552#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2610.10552#S3.SS3.p2.1),[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Xinet al\.\(2026\)J\. Xin, T\. Lyu, D\. Keyes, H\. Ltaief, and M\. CaniniRAP: KV\-cache compression via RoPE\-aligned pruning\.External Links:2602\.02599Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Yanet al\.\(2025\)X\. Yan, Z\. Li, T\. Zhang, H\. Qin, L\. Kong, Y\. Zhang, and X\. YangReCalKV: low\-rank KV cache compression via head reordering and offline calibration\.External Links:2505\.24357Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Yesiltepeet al\.\(2026\)H\. Yesiltepe, J\. Hu, T\. H\. S\. Meral, A\. K\. Akan, K\. Oktay, H\. Eldardiry, and P\. YanardagVideoMLA: low\-rank latent KV cache for minute\-scale autoregressive video diffusion\.External Links:2605\.30351Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2025\)J\. Yuan, H\. Gao, D\. Dai, J\. Luo, L\. Zhao, Z\. Zhang, Z\. Xie, Y\. X\. Wei, L\. Wang, Z\. Xiao, Y\. Wang, C\. Ruan, M\. Zhang, W\. Liang, and W\. ZengNative sparse attention: hardware\-aligned and natively trainable sparse attention\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 23078–23097\.Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2023\)L\. Zhanget al\.LoRA\-FA: memory\-efficient low\-rank adaptation for large language models fine\-tuning\.External Links:2308\.03303Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2024\)R\. Zhang, K\. Wang, L\. Liu, S\. Wang, H\. Cheng, C\. Zhang, and Y\. ShenLoRC: low\-rank compression for LLMs KV cache with a progressive compression strategy\.External Links:2410\.03111Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2026\)Z\. Zhou, F\. Bie, Z\. Chen, Z\. Zhang, Y\. Yang, J\. Wang, B\. Athiwaratkun, X\. Wu, and S\. L\. SongCARE: covariance\-aware and rank\-enhanced decomposition for enabling multi\-head latent attention\.External Links:2603\.17946Cited by:[§1](https://arxiv.org/html/2610.10552#S1.p1.1),[§2](https://arxiv.org/html/2610.10552#S2.SS0.SSS0.Px3.p1.1),[§3\.3](https://arxiv.org/html/2610.10552#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2610.10552#S3.SS3.p2.1),[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.10552#S5.p1.1),[Limitations](https://arxiv.org/html/2610.10552#Sx1.p1.1)\.
- Zhuet al\.\(2024\)J\. Zhuet al\.Asymmetry in low\-rank adapters of foundation models\.External Links:2402\.16842Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px2.p1.1)\.
- Zuhriet al\.\(2024\)Z\. M\. K\. Zuhri, M\. F\. Adilazuarda, A\. Purwarianti, and A\. F\. AjiMLKV: multi\-layer key\-value heads for memory efficient transformer decoding\.External Links:2406\.09297Cited by:[§4](https://arxiv.org/html/2610.10552#S4.SS0.SSS0.Px1.p1.1)\.