Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers

# Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Source: [https://arxiv.org/html/2607.15456](https://arxiv.org/html/2607.15456)
James O’Neilljames\.oneill@intercom\.io Fin AI ResearchFergal Reid Fin AI Research

###### Abstract

Looped, weight\-tied Transformers reduce parameters by reusing a block, but decoding still stores a separate K/V cache for every recurrence step\. We show that this loop\-indexed cache is highly structured\. For a fixed token, layer and head, K/V vectors trace a short low\-rank trajectory across loops, while the head and layer axes remain much flatter\.

We introduce Looped Latent Attention \(LLA\), a post\-training cache codec that stores compact K and V latents and reconstructs loop\-specific K/V vectors only when attention reads them\. The default per\-head codec compresses recurrence, while LLA\-2D also folds heads into one latent for the extreme\-compression regime\. The codec is initialized from SVD of teacher activations and refined with KL and attention\-output distillation\. At matched cache budget, per\-head LLA outperforms head\-axis MLA, cross\-layer sharing, KV quantization and final\-loop reuse, showing that the recurrent cache is low\-rank but not safely collapsible to a single state\. The same axis advantage holds on Ouro\-2\.6B\-Thinking and transfers to Huginn\-3\.5B, where an SVD codec remains near\-lossless to32×32\\timesin decoder\-independent evaluation\. The cache reduction is exact\. On one H200, the latent\-store path increases measured Ouro\-1\.4B batch capacity at 4k context from 32 to 768 sequences at21\.3×21\.3\\timescompression\. For long math rollouts, on\-policy refinement on student\-generated prefixes raises MATH\-500 at4×4\\timesfrom 0\.43 to 0\.66 and reduces no\-answer generations\.

## 1Introduction

At inference, the memory that constrains a Transformer is its KV cache, not its weights\. The weights are loaded once and shared across the whole batch\. The cache is private to each request and grows with the context and with every generated token\. Past a certain context length, this per\-request state, not the parameter count, decides how many requests fit on a device and whether batched serving stays economical\.

Looped, weight\-tied Transformers attack the parameter side of this cost by applying one block repeatedly: Universal Transformers\(Dehghani et al\.,[2019](https://arxiv.org/html/2607.15456#bib.bib5)\)recast depth as recurrence, and recent looped language models such as Ouro/LoopLM\(Zhu et al\.,[2025](https://arxiv.org/html/2607.15456#bib.bib16)\)scale the idea to billions of parameters\. The cache, though, is untied\. Each recurrence step recomputes attention and writes its own keys and values, so the cache is indexed by recurrence step in addition to layer, head and token\. A model withT=4T=4loops overD=24D=24layers therefore carries the KV cache of a 96\-layer decoder while sharing the parameters of a 24\-layer one\.

This cache carries far fewer degrees of freedom than its nominal size\. For a fixed token, layer and head, the per\-loop K/V vectors trace a short, low\-rank trajectory, and the redundancy is specific to the loop axis: at the same budget, the head and layer spectra are much flatter\. Looped Latent Attention \(LLA\) exploits this by replacing theTTper\-loop K/V vectors with a single latent and loop\-specific reconstruction maps\.

The real question is which axis to compress, and what the cache must preserve when it does\. A looped decoder invites a degenerate shortcut: if recurrence merely converges to a fixed point, one could store the final loop and reuse it everywhere\. It also inherits the usual Transformer options — compressing heads, sharing layers, quantizing values\. Our experiments separate these\. We freeze the teacher, leave the attention pattern and recurrence intact, and change only what the cache stores\. Under this controlled swap, final\-loop reuse buys the same4×4\\timesreduction for free yet collapses GSM8K generation to zero, while a small low\-rank trajectory code holds performance at the same budget\.

### Contributions\.

- •We define LLA, a cross\-loop latent K/V codec for weight\-tied looped Transformers\. It stores\(rk\+rv\)​H\(r\_\{k\}\+r\_\{v\}\)Hscalars per token per layer instead of2​T​H​dhead2THd\_\{\\rm head\}, giving compressionρ=2​T​dhead/\(rk\+rv\)\\rho=2Td\_\{\\rm head\}/\(r\_\{k\}\+r\_\{v\}\)\. LLA\-2D extends the same family by folding heads into one latent for the extreme\-compression regime\.
- •We show that recurrence is the most compressible cache axis in Ouro\-1\.4B, Ouro\-2\.6B\-Thinking and Huginn\-3\.5B\. K and V follow different trajectories, motivatingrv\>rkr\_\{v\}\>r\_\{k\}, and matched\-budget comparisons against head\-axis MLA, cross\-layer sharing, KV quantization and final\-loop reuse favor the loop axis\.
- •We tie the representation result to deployment: an explicit on\-policy refinement objective uses student\-generated prefixes to stabilize long rollouts, and the exact scalar reduction translates into measured serving gains, raising Ouro\-1\.4B maximum batch from 32 to 768 sequences at21\.3×21\.3\\timescompression on one H200\.

## 2Related Work

### Looped and Universal Transformers\.

Universal Transformers\(Dehghani et al\.,[2019](https://arxiv.org/html/2607.15456#bib.bib5)\)apply a weight\-tied block for several steps, casting depth as recurrence\. Recent looped language models carry this to billion\-parameter LMs and trade compute for parameter efficiency on reasoning workloads\(Zhu et al\.,[2025](https://arxiv.org/html/2607.15456#bib.bib16)\)\. Mechanistic studies of these models report that per\-loop computation settles onto low\-dimensional trajectories\(Blayney et al\.,[2026](https://arxiv.org/html/2607.15456#bib.bib2)\), and LLA builds a cache codec on exactly that structure\. Recurrence does not address serving cost, where attention still stores a separate K/V state for every loop\.

### Latent and compressed KV caches\.

Multi\-head Latent Attention \(MLA\) introduced in DeepSeek\-V2 compresses the standard Transformer KV cache by storing a low\-rank latent across the head/channel axis and reconstructing per\-head keys and values\(DeepSeek\-AI,[2024](https://arxiv.org/html/2607.15456#bib.bib4)\)\. Cross\-layer sharing targets the layer axis\(Brandon et al\.,[2024](https://arxiv.org/html/2607.15456#bib.bib3); Sun et al\.,[2024](https://arxiv.org/html/2607.15456#bib.bib12); Liu et al\.,[2024](https://arxiv.org/html/2607.15456#bib.bib9)\), quantization targets precision\(Hooper et al\.,[2024](https://arxiv.org/html/2607.15456#bib.bib7)\), and token eviction targets the sequence axis\(Zhang et al\.,[2023](https://arxiv.org/html/2607.15456#bib.bib15); Xiao et al\.,[2024](https://arxiv.org/html/2607.15456#bib.bib14)\)\. All of these operate on single\-pass Transformers\. LLA instead compresses recurrence, an axis that only weight\-tied looped models have\. Because the same block and projections are reused at every step, the loop axis is redundant for a reason the head and layer axes lack\. The head axis, targeted by multi\-query and grouped\-query attention\(Shazeer,[2019](https://arxiv.org/html/2607.15456#bib.bib10); Ainslie et al\.,[2023](https://arxiv.org/html/2607.15456#bib.bib1)\), composes on top of LLA \(LLA\-2D\), and sequence\-axis eviction multiplies with it \(Sec\.[7](https://arxiv.org/html/2607.15456#S7)\)\.

### Memory\-efficient looped Transformers\.

MELT\(Vendrell et al\.,[2026](https://arxiv.org/html/2607.15456#bib.bib13)\)is the closest concurrent work\. It distills a single gated K/V state per layer, whereas LLA keeps the loop trajectory and compresses its rank\. The distinction is not cosmetic\. Our zero\-parameter final\-loop\-reuse control, which collapses the loop to one state, fails at the same cache budget\. The two methods also span different regimes\. A collapsed state fixes the memory reduction at roughly the loop count, whereas a tunable latent can sit below or above that point by choice ofrkr\_\{k\}andrvr\_\{v\}\.

## 3Looped Latent Attention

Looped models save parameters but still materialize K/V states for every recurrence step\. We compress that recurrence axis with a codec that retrofits onto a frozen teacher and leaves the attention computation unchanged except for the representation held in memory, so the experiments isolate whether the loop\-indexed K/V stack is low\-rank\.

We first define the uncompressed recurrent cache, then introduce the cross\-loop latent and its exact compression ratio\. The same construction is used for K and V, but with different ranks\. We then describe the SVD initialization and the matched baselines used to test whether the loop axis is the right one to compress\.

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig1_round2/c04_nested_zoom.png)Figure 1:Looped Latent Attention\.\(a\)LLA sits on the KV\-cache path of a weight\-tied looped Transformer block: the block emits per\-loopK,VK,Vat every recurrence step, and the codec compresses only the loop axis of that cache, leaving the attention pattern and the rest of the block untouched\.\(b\)LLA\-2D \(loop×\\timeshead\), a family variant, factorizes loops*and*heads together with a single latent per token and layer, decoding allHHheads MLA\-style from one vector\. The default per\-head LLA instead keepsHHseparate per\-head latents and compresses only the \(more redundant\) loop axis\.\(c\)The head\-axis baseline \(per\-loop MLA\) compresses heads independently at every recurrence step, with no sharing across loops\. LLA instead reconstructs each loop from one shared cross\-loop latent via a per\-loop up\-projectionWt↑W^\{\\uparrow\}\_\{t\}\.### Teacher cache\.

Consider a weight\-tied looped Transformer withDDlayers,HHattention heads of widthdheadd\_\{\\rm head\}andTTrecurrence steps\. For one layer, one head, one token and one axis \(K or V\), the teacher emitsTTvectors\. We stack them as

x=\[x1;…;xT\]∈ℝT​dhead\.x=\[x\_\{1\};\\ldots;x\_\{T\}\]\\in\\mathbb\{R\}^\{Td\_\{\\rm head\}\}\.\(1\)The uncompressed cache stores both K and V for every loop, head and layer, for a per\-token per\-layer cost of2​T​H​dhead2THd\_\{\\rm head\}scalars\.

### Cross\-loop codec\.

For each layer, head and axis, LLA stores a centered low\-rank latent

c=\(x−μ\)​Wdown,c∈ℝr,c=\(x\-\\mu\)W\_\{\\rm down\},\\qquad c\\in\\mathbb\{R\}^\{r\},\(2\)whereμ∈ℝT​dhead\\mu\\in\\mathbb\{R\}^\{Td\_\{\\rm head\}\}is a calibration\-set mean andWdown∈ℝT​dhead×rW\_\{\\rm down\}\\in\\mathbb\{R\}^\{Td\_\{\\rm head\}\\times r\}\. Loopttis reconstructed by a loop\-specific up\-projection,

x^t=c​Wup,t⊤\+μt,Wup,t∈ℝdhead×r\.\\hat\{x\}\_\{t\}=cW\_\{\{\\rm up\},t\}^\{\\top\}\+\\mu\_\{t\},\\qquad W\_\{\{\\rm up\},t\}\\in\\mathbb\{R\}^\{d\_\{\\rm head\}\\times r\}\.\(3\)K and V have separate latents and ranks,rkr\_\{k\}andrvr\_\{v\}, withrv\>rkr\_\{v\}\>r\_\{k\}because the value cache stabilizes more slowly across loops than the key cache\. The compressed cache stores\(rk\+rv\)​H\(r\_\{k\}\+r\_\{v\}\)Hactivation scalars per token per layer, so

ρ=2​T​H​dhead\(rk\+rv\)​H=2​T​dheadrk\+rv\.\\rho=\\frac\{2THd\_\{\\rm head\}\}\{\(r\_\{k\}\+r\_\{v\}\)H\}=\\frac\{2Td\_\{\\rm head\}\}\{r\_\{k\}\+r\_\{v\}\}\.\(4\)The ratio counts the per\-request cache\. The fixed codec matrices are shared across tokens, contexts and batches and do not scale with sequence length\.

### RoPE and reconstruction\.

The latent is fit to the pre\-RoPE content projection\. In a looped decoder, a token has the same position at every recurrence step, so the rotary phase\(Su et al\.,[2021](https://arxiv.org/html/2607.15456#bib.bib11)\)is shared across loops and can be applied after reconstruction\. This is simpler than standard MLA, which requires an explicit decoupled\-RoPE branch to preserve positional information along the head/channel axis\.

### The LLA\-2D variant \(loop×\\timeshead\)\.

The per\-head codec of Eq\. \([2](https://arxiv.org/html/2607.15456#S3.E2)\) can be pushed further by folding the head axis into the same latent\. LLA\-2D stores a single latent per token and layer and reconstructs*all*HHheads’ K/V from it, MLA\-style, in addition to the per\-loop reconstruction of Eq\. \([3](https://arxiv.org/html/2607.15456#S3.E3)\)\. This lowers the cache to\(rk\+rv\)\(r\_\{k\}\+r\_\{v\}\)scalars per token per layer, a factorHHbelow per\-head LLA, at the cost of coupling heads through one shared code\. LLA\-2D is the extreme\-compression member of the family\. Its down\-projection input is the fullT​H​dheadTHd\_\{\\rm head\}stack rather thanT​dheadTd\_\{\\rm head\}, so its parameter count grows with head count and recurrence length\. This becomes the binding constraint at largeTT\(the Huginn transfer, Section[5](https://arxiv.org/html/2607.15456#S5)\)\. Section[5](https://arxiv.org/html/2607.15456#S5)characterizes the accuracy/compression trade between the two members\. LLA without qualification denotes the per\-head variant\.

### Initialization and conversion\.

For each layer, head and axis we collect teacher K/V activations on a calibration set, center by the per\-loop mean and initializeWdownW\_\{\\rm down\}from the top\-rrright singular vectors of the stacked\-loop matrix\. The initial up\-projections are the corresponding loop slices\. The SVD initialization is then fine\-tuned with the teacher frozen\. The objective is the forward KL from the teacher distribution to the swapped model plus an attention\-output matching term,

ℒ=KL​\(pteacher∥pswap\)\+λattnT​D​∑t,d‖at,d−a^t,d‖22,\\mathcal\{L\}=\\mathrm\{KL\}\\left\(p\_\{\\rm teacher\}\\\|p\_\{\\rm swap\}\\right\)\+\\frac\{\\lambda\_\{\\rm attn\}\}\{TD\}\\sum\_\{t,d\}\\\|a\_\{t,d\}\-\\hat\{a\}\_\{t,d\}\\\|\_\{2\}^\{2\},\(5\)withλattn=0\.5\\lambda\_\{\\rm attn\}=0\.5in the reported runs\. Only codec parameters are updated\. The SVD prior matters more than any other design choice\. At4×4\\times, SVD initialization reaches held\-out KL 0\.105 after the short conversion, while random initialization stays at 0\.771 under the same budget \(Table[10](https://arxiv.org/html/2607.15456#A5.T10), Fig\.[10](https://arxiv.org/html/2607.15456#A5.F10)\)\.

### On\-policy refinement\.

Eq\. \([5](https://arxiv.org/html/2607.15456#S3.E5)\) is teacher\-forced, fitting the codec on prefixes visited by the teacher\. For long autoregressive rollouts, the swapped model instead conditions on prefixes produced by its own codec\. We therefore add an optional second stage after the off\-policy fit\. The swapped model samples completions, the frozen teacher scores those sampled prefixes, and only codec parameters are updated\. No reward or correctness filtering is used\. Section[6](https://arxiv.org/html/2607.15456#S6)writes the off\-policy and on\-policy objectives side by side\. The on\-policy version replaces teacher prefixes with student\-generated prefixes and stops the gradient through sampling\.

### Matched\-axis baselines\.

The main ablations place comparable latent codecs on other cache axes under the same scalar budget, testing whether the loop axis is the right one\. The head\-axis baseline is per\-loop MLA \(no cross\-loop sharing\)\. The layer baseline reuses K/V across adjacent layers\. The precision baseline is low\-bit KV quantization\. We also include a zero\-parameter control that caches only the final\-loop K/V and reuses it for every loop\. That control has the same4×4\\timescache footprint as LLA forT=4T=4and directly tests whether the loop trajectory can be replaced by its endpoint\.

## 4Cross\-loop structure in the teacher cache

Before comparing trained codecs, we ask whether the frozen teacher already contains the structure that LLA assumes\. This is a stricter test than downstream accuracy, since it examines the cache tensor directly, without optimization, decoding heuristics or task\-specific effects\. If the loop axis is not spectrally different from the head or layer axes, then a cross\-loop codec would be an arbitrary design choice\. If it is lower\-rank in the teacher, the later iso\-cache ordering has a mechanistic explanation\.

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_why_struct_1x4.png)Figure 2:Cross\-loop K/V is low\-rank because recurrence stabilizes\.Left two panels: the loop axis has much lower normalized effective rank than the head or layer axes, and a single direction carries a large fraction of the energy\. Right two panels: K and V move toward the final loop while the step\-to\-step change shrinks, and V converges more slowly than K, motivatingrv\>rkr\_\{v\}\>r\_\{k\}\.The loop\-axis advantage is visible before training any codec\. Fig\.[2](https://arxiv.org/html/2607.15456#S4.F2)compares the singular spectra of frozen\-teacher K/V along the loop, head and layer axes\. In Ouro\-1\.4B and Ouro\-2\.6B\-Thinking, the loop axis has normalized effective rank about0\.610\.61to0\.630\.63, with a single direction carrying roughly 60 percent of the energy\. The head and layer axes are close to full\-rank by the same measure\. This explains why a small loop latent preserves more downstream behavior per cached scalar than head or layer compression\.

Cross\-loop dynamics also explain the K/V rank split\. Across recurrence steps, K and V move toward the final loop while step\-to\-step changes shrink\. K stabilizes earlier, while V moves more slowly and carries more cross\-loop variance\. We therefore allocaterv\>rkr\_\{v\}\>r\_\{k\}throughout\. The matched\-budget ablation confirms the benefit, withrk/rv=96/160r\_\{k\}/r\_\{v\}=96/160outperforming a shared\-rank128/128128/128codec at the same scalar count \(Table[10](https://arxiv.org/html/2607.15456#A5.T10)\)\.

A single endpoint is not enough\. Fig\.[6](https://arxiv.org/html/2607.15456#A2.F6)shows that each token\-head unit is dominated by one cross\-loop direction, but the mass does not sit on the rank\-1 line\. Only 6 percent of K variance and 17 percent of V variance lie on the loop axis, and most variation remains content variation\. The right object is therefore a short low\-rank trajectory, not a collapsed state\. This distinction is reinforced by the final\-loop reuse baseline in Table[3](https://arxiv.org/html/2607.15456#S5.T3)\.

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_why_huginn_clean_tight.png)Figure 3:The recurrence\-axis mechanism transfers to Huginn\-3\.5B\.Huginn has up to 32 recurrence steps and a different looped\-LM architecture\. Its loop axis is far more redundant than the head axis, successive steps converge toward a fixed point, and V again converges more slowly than K\.The same mechanism appears in a second looped family\. Huginn\-3\.5B uses a different recurrent\-depth architecture and up to 32 recurrence steps\(Geiping et al\.,[2025](https://arxiv.org/html/2607.15456#bib.bib6)\)\. Its loop axis is far more redundant than its head axis, and its K/V states converge toward a fixed point across recurrence steps \(Fig\.[3](https://arxiv.org/html/2607.15456#S4.F3)\)\. The value cache again converges more slowly than the key cache\. This recurrence\-axis structure supports treating LLA as a property of weight\-tied iteration rather than an Ouro\-specific artifact\. The remaining experiments test that structure through downstream accuracy, matched\-budget baselines, long rollouts and serving capacity\.

## 5Experiments

The core question is which axis to spend rank on\. At a fixed cache budget, we test whether compressing the loop beats compressing heads, layers or precision, and how much rank a converted codec needs before it matches the teacher on a generative task\. Two further experiments ask whether the answer holds at a larger Ouro model and in the unrelated Huginn family\.

### Setup\.

Unless stated otherwise, generative results use the cached reconstruction decoder, which inserts reconstructed K/V before attention\. The latent\-store decoder in Sec\.[7](https://arxiv.org/html/2607.15456#S7)is the memory\-realizing path\. Decoder\-independent metrics evaluate the swapped model on teacher\-provided tokens and are not affected by generation drift\. The main teacher is Ouro\-1\.4B withT=4T=4loops\. Scale experiments use Ouro\-2\.6B\-Thinking, and family\-transfer experiments use Huginn\-3\.5B\.

Table 1:GSM8K accuracy\-compression Pareto for Ouro\-1\.4B\.Six cold\-start LLA codecs are trained with the same recipe and evaluated with the cached reconstruction decoder\. The plateau from4\.0×4\.0\\timesto2\.67×2\.67\\timesbreaks sharply at2\.0×2\.0\\times, and the teacher scores0\.820\.82on the same setting\.
### Accuracy\-compression frontier\.

Table[1](https://arxiv.org/html/2607.15456#S5.T1)shows a cold\-start GSM8K Pareto sweep for Ouro\-1\.4B\. Accuracy is flat around 0\.58 from4×4\\timesto2\.67×2\.67\\times, then rises sharply at2×2\\timesand reaches 0\.795 at1\.33×1\.33\\times, close to the teacher\. The result is a rank effect rather than a method ceiling\. Below the transition, the latent is large enough to preserve the decision tokens that determine final\-answer correctness\. The sharp break shows where the loop trajectory becomes sufficiently resolved for GSM8K generation\.

Table 2:Quality against the Ouro\-1\.4B teacher across the broad suite\.GSM8K is 5\-shot strict EM; HumanEval and MBPP use EvalPlus pass@1 \(base/plus\); BBH and MMLU\-Pro use exact match\. MATH\-500 uses reliable final\-answer extraction and symbolic verification\. Light compression matches the teacher across tasks, while heavier compression is task\-dependent\.Across the broad suite \(Table[2](https://arxiv.org/html/2607.15456#S5.T2)\), light compression matches or exceeds the teacher within evaluation noise\. Code and BBH retain non\-trivial performance at4×4\\times, but lighter ratios are needed to match the teacher\. MATH\-500 is the most sensitive task and is treated separately in Section[6](https://arxiv.org/html/2607.15456#S6), because long generations stress rollout stability rather than only per\-token reconstruction\.

The broad\-suite table also separates three regimes\. Light compression \(1\.331\.33to2×2\\times\) is effectively a drop\-in replacement for the full cache, resolving enough of the loop trajectory that residual differences fall within benchmark variation\. By2\.672\.67to5\.33×5\.33\\times, the representation still carries many likelihood and short\-generation tasks, though long reasoning becomes sensitive to prefix drift\. At the most aggressive ratios the codec is a capacity operating point, not a universal quality\-preserving conversion\. The matched\-axis comparisons below use the middle and high\-compression regimes to test whether recurrence is the right axis to spend the latent dimension on\.

The regimes reflect one underlying variable, generation length\. Robustness to compression falls off as generations grow longer\. Multiple\-choice and short\-generation tasks tolerate the middle regime with little loss, code and BBH sit in between, and long\-form mathematical reasoning breaks first\. A deployment should budget around this ordering rather than any single benchmark\.

Table 3:Iso\-cache comparison across compression axes\.All learned codecs are matched by KV\-cache budget\. The LLA rows use the checkpoint selected under the iso\-cache protocol, whereas Table[1](https://arxiv.org/html/2607.15456#S5.T1)is a separate cold\-start rank sweep\. LLA \(per\-head\) compresses the loop axis\. LLA\-2D \(loop×\\timeshead\) is a family variant that additionally folds the head axis \(all heads from one latent\)\. The head, layer and precision rows are baselines that compress orthogonal axes\. The zero\-parameter final\-loop reuse control has the same4×4\\timescache budget as LLA, but collapses generation \(GSM8K0\.0000\.000\)\. Its reconstruction relative error is0\.350\.35for K and0\.600\.60for V\. Precision quantization \(kv\_quant\) is inference\-only \(no codec training, hence no train\-KL\) and cannot exceed∼\\sim8×\\times\. It is competitive at4×4\\timesbut collapses on GSM8K at5\.33×5\.33\\times\.![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_pareto_1p4b_tight.png)Figure 4:Ouro\-1\.4B performance versus KV\-cache compression\.Each panel plots an evaluation metric against compression for per\-head LLA, the LLA\-2D \(loop×\\timeshead\) variant, and the head, layer and precision baselines\. The dashed line is the uncompressed teacher\. LLA tracks the teacher most closely across the suite, while precision quantization collapses on GSM8K past moderate compression\.
### Iso\-cache axis comparison\.

Table[3](https://arxiv.org/html/2607.15456#S5.T3)compares loop compression against matched\-budget alternatives\. Table[1](https://arxiv.org/html/2607.15456#S5.T1)should be read separately as a cold\-start rank sweep, whereas Table[3](https://arxiv.org/html/2607.15456#S5.T3)uses the checkpoint and evaluation protocol adopted for the baseline comparison\. At4×4\\times, LLA has substantially lower train\-KL than head or layer compression and much higher GSM8K accuracy\. The zero\-parameter final\-loop reuse baseline has the same4×4\\timescache budget and no trainable parameters, but collapses GSM8K to 0\.000\. This is the most direct evidence that the loop trajectory cannot be replaced by its endpoint\.

Within the LLA family, LLA\-2D \(loop×\\timeshead\) is competitive at high compression on teacher\-forced metrics but trails the per\-head variant on generation at21\.3×21\.3\\times\. This matches the spectral picture\. LLA\-2D spends part of its budget on the head axis, which is flatter and more expensive to compress\. The two variants therefore span the operating range, with per\-head LLA for accuracy\-preserving compression and LLA\-2D for the extreme\-compression regime where itsH×H\\timessmaller cache outweighs the accuracy gap\. Fig\.[4](https://arxiv.org/html/2607.15456#S5.F4)gives the per\-task view for the same comparison\. Per\-head LLA does not win by overfitting a single benchmark, and the precision baseline is harmless at mild compression but brittle once the bit budget becomes too small\.

Table 4:Scale generalization on Ouro\-2\.6B\-Thinking\.GSM8K is evaluated with the cached reconstruction decoder, while train\-KL is decoder\-independent\. LLA remains near\-lossless across44to10\.67×10\.67\\times, while the head and layer baselines lose substantially more accuracy\.
### Scale generalization\.

The 2\.6B results in Table[4](https://arxiv.org/html/2607.15456#S5.T4)preserve the ordering\. LLA remains near\-lossless on full GSM8K from4×4\\timesto10\.67×10\.67\\times, while the head baseline loses up to 0\.25 absolute accuracy at4×4\\times\. Fig\.[7](https://arxiv.org/html/2607.15456#A2.F7)and Appendix[F](https://arxiv.org/html/2607.15456#A6)show that the advantage also holds across commonsense and knowledge tasks\. The 2\.6B model is deeper and has a stronger reasoning profile, and preserving the ordering under that change supports the structural interpretation from Section[4](https://arxiv.org/html/2607.15456#S4)\.

### Design ablations\.

Table[10](https://arxiv.org/html/2607.15456#A5.T10)isolates the choices used in the codec\. SVD initialization is the largest factor, reducing held\-out KL by an order of magnitude relative to random initialization under the same conversion budget\. The asymmetric rank split improves over a shared\-rank latent at matched scalar cost, and attention\-output matching gives a small additional gain beyond token KL\. These ablations indicate that LLA mostly exploits teacher geometry rather than learning a new attention mechanism from scratch\.

### Transfer to Huginn\.

A LLA codec also works on Huginn\-3\.5B, where the recurrence length is much larger\. Table[5](https://arxiv.org/html/2607.15456#S5.T5)shows that SVD initialization alone is near\-lossless to32×32\\timesin decoder\-independent fidelity\. KL distillation recovers much of the aggressive64×64\\timesand128×128\\timesrange\. The downstream MC average is unchanged at16×16\\timesand32×32\\timesand degrades gracefully after that\. On Huginn, head\-axis SVD reconstruction is 20 to 30 times worse than LLA at matched budget, and LLA\-2D \(loop×\\timeshead\) becomes parameter\-infeasible at this recurrence length because its down\-projection input dimension grows with head count and loop count\. Per\-head LLA avoids this cost, which is why it is the default\.

This transfer is intentionally harder than reproducing a number on a larger Ouro checkpoint\. Huginn changes the recurrent\-depth architecture, increases the realized recurrence to 32 steps, and is evaluated through decoder\-independent fidelity rather than the Ouro cached generation path\. The result indicates that a simple linear subspace found from teacher activations already captures most of the recurrence trajectory\. Distillation is needed only when the requested budget asks the latent to represent the tail of the trajectory at64×64\\timesor128×128\\times\. This is the behavior expected if weight\-tied iteration itself is the source of the low rank, not a peculiarity of one training run or benchmark\.

Table 5:LLA on Huginn\-3\.5B withR=32R=32recurrence steps\.Decoder\-independent fidelity is measured against the uncompressed model on held\-out text\. A training\-free SVD codec is near\-lossless to32×32\\times, and short KL distillation recovers much of the aggressive6464to128×128\\timesrange\. MC\-avg is a broad downstream loglikelihood suite\.

## 6Long\-generation stability

Teacher\-forced conversion trains the codec on teacher trajectories\. At inference, however, the swapped model conditions on tokens produced by its own codec\. A small early reconstruction error can move later prefixes away from the conversion distribution, and the error then compounds autoregressively\. This failure is most visible on long MATH\-500 generations, where the model can produce long non\-terminating solutions even when teacher\-prefix KL is low\.

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_exposure_length_revised.png)Figure 5:Off\-policy failure compounds with generation length\.At3\.2×3\.2\\times, the uncompressed teacher \(gray\) sets the reference\. Accuracy declines gently with length from problem difficulty, and non\-termination stays low\. On\-policy distillation \(red\) tracks the teacher closely at every length, whereas the off\-policy codec \(blue\) loses accuracy as responses grow longer and its no\-answer rate spikes in the longest bin\. The gap between off\-policy and the teacher, not the teacher’s own mild decline, is the exposure\-bias failure that on\-policy training removes\.Fig\.[5](https://arxiv.org/html/2607.15456#S6.F5)bins3\.2×3\.2\\timescompletions by generated length, with the uncompressed teacher as a reference\. Off\-policy accuracy degrades with length far below the teacher and the no\-answer rate spikes in the longest bin\. On\-policy distillation trains on the codec’s own sampled rollouts and closely tracks the teacher across length bins\. The teacher’s own gentle decline with length reflects intrinsic problem difficulty, not exposure bias\.

The refinement objective makes the distribution shift explicit\. LetpTp\_\{T\}denote the frozen teacher andpSθp^\{\\theta\}\_\{S\}the swapped model with codec parametersθ\\theta:

ℒoff​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\rm off\}\(\\theta\)=𝔼y∼pT\[1\|y\|∑tKL\(pT\(⋅∣y<t\)∥pSθ\(⋅∣y<t\)\)\],\\displaystyle=\\mathbb\{E\}\_\{y\\sim p\_\{T\}\}\\left\[\\frac\{1\}\{\|y\|\}\\sum\_\{t\}\\mathrm\{KL\}\\left\(p\_\{T\}\(\\cdot\\mid y\_\{<t\}\)\\\|p^\{\\theta\}\_\{S\}\(\\cdot\\mid y\_\{<t\}\)\\right\)\\right\],\(6\)ℒon​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\rm on\}\(\\theta\)=𝔼y~∼sg​\(pSθ\)\[1\|y~\|∑tKL\(pT\(⋅∣y~<t\)∥pSθ\(⋅∣y~<t\)\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\\tilde\{y\}\\sim\\mathrm\{sg\}\(p^\{\\theta\}\_\{S\}\)\}\\left\[\\frac\{1\}\{\|\\tilde\{y\}\|\}\\sum\_\{t\}\\mathrm\{KL\}\\left\(p\_\{T\}\(\\cdot\\mid\\tilde\{y\}\_\{<t\}\)\\\|p^\{\\theta\}\_\{S\}\(\\cdot\\mid\\tilde\{y\}\_\{<t\}\)\\right\)\\right\]\.\(7\)Off\-policy distillation uses teacher prefixesyy\. On\-policy distillation instead uses student prefixesy~\\tilde\{y\}sampled from the current compressed model\. The stop\-gradient blocks gradients through sampling, so updates still come only from the KL term\. The teacher remains the target distribution in both objectives\. The on\-policy stage only changes which prefixes are used to tune the codec, turning rollout drift into a reconstruction problem on states the deployed model actually visits\.

Table 6:On\-policy distillation across the rank Pareto\.MATH\-500 is scored by reliable final\-answer extraction plus symbolic verification \(teacher0\.740\.74\)\. On\-policy training improves long\-generation stability at every ratio and gives the largest gains at3\.23\.2to5\.33×5\.33\\times\. “no\-ans” denotes no extractable final answer\.Table[6](https://arxiv.org/html/2607.15456#S6.T6)shows the full rank Pareto\. On\-policy distillation raises MATH\-500 accuracy by0\.160\.16to0\.240\.24at3\.23\.2to5\.33×5\.33\\timesand reduces no\-answer generations at every ratio\. At21\.3×21\.3\\times, it still improves termination but does not improve accuracy, indicating that the smallest latent has already lost task\-relevant information\. Larger re\-evaluations confirm the low\-compression lift in Fig\.[9](https://arxiv.org/html/2607.15456#A4.F9)\. For long rollouts, the conversion data must include states visited by the compressed model itself\.

## 7Serving and compute

The motivation for LLA is deployment\. Reducing a cache tensor should increase the contexts or batches that fit in memory\(Kwon et al\.,[2023](https://arxiv.org/html/2607.15456#bib.bib8)\)\. This section separates the exact memory statement from the implementation\-dependent compute statement\. The memory reduction follows directly from Eq\. \([4](https://arxiv.org/html/2607.15456#S3.E4)\), while the compute path depends on whether one reconstructs K/V explicitly or absorbs the projections into attention\.

Table 7:Memory, capacity and latency\.LLA reduces cached scalars exactly byρ\\rho\. At fixed VRAM and context length 4096, measured maximum batch follows the expected capacity gain\. The full\-memory fast path caches reconstructed K/V and is fast, while the latent\-store capacity path realizes the memory saving and is reconstruction\-compute\-bound in the current implementation\. Combining21\.3×21\.3\\timeswith token eviction reduces the 1M\-context 1\.4B cache further to3636MiB\. “n/a” marks an operating point not characterized for that path rather than a missing measurement\. The fast path is profiled at its intended4×4\\timespoint \(it caches full K/V and does not realize the memory saving at higherρ\\rho\), the capacity path at21\.3×21\.3\\times, and the 2\.6B capacity test was run to4×4\\times\.The scalar reduction in Eq\. \([4](https://arxiv.org/html/2607.15456#S3.E4)\) is exact\. Table[7](https://arxiv.org/html/2607.15456#S7.T7)and Fig\.[8](https://arxiv.org/html/2607.15456#A2.F8)translate it into serving capacity\. At context length 4096 on one H200, the measured maximum batch for Ouro\-1\.4B rises from 32 for the teacher to 128 at4×4\\timesand 768 at21\.3×21\.3\\times\. The same measurement gives 16 to 64 at4×4\\timesfor Ouro\-2\.6B\-Thinking\. These tests fabricate the decode cache and push batch to OOM, and the latent attention path is validated against full reconstruction\.

LLA also composes with sequence\-axis eviction, the one orthogonal axis in Sec\.[2](https://arxiv.org/html/2607.15456#S2)we can stack rather than substitute\. Table[9](https://arxiv.org/html/2607.15456#A3.T9)evaluates StreamingLLM/H2O\-style eviction on GSM8K\. Eviction applied to a near\-lossless codec and eviction stacked on LLA\-4×4\\timesscore within one standard error \(0\.510\.51vs0\.520\.52\), so the recurrence codec adds no quality cost beyond eviction’s own\. Because the cache reductions multiply, LLA\-21\.3×21\.3\\timescombined with a10241024\-token eviction budget caps the 1M\-context 1\.4B cache at3636MiB \(Table[7](https://arxiv.org/html/2607.15456#S7.T7)\)\. Memory is only useful if accuracy survives at length\. Table[8](https://arxiv.org/html/2607.15456#S7.T8)probes exact passkey retrieval from a key hidden at random depth in up to a6464k\-token context\. The uncompressed teacher is perfect throughout\. LLA is near\-lossless for retrieval through5\.33×5\.33\\timesto6464k, degrades gracefully from10\.67×10\.67\\times\(the knee is perfect to1616k and0\.730\.73at6464k\), and collapses at21\.3×21\.3\\times\. Moderate loop compression preserves the exact token identity that retrieval requires, while aggressive compression does not, so the operating point trades retrieval fidelity for cache size\. Reaching6464k required reconstructing the cached decoder’s prefill K/V layer\-by\-layer and freeing each layer’s captures on the fly\. A naive one\-shot reconstruction holds the float32 captures and their full\-width reconstruction and exhausts memory past∼\\sim32k\.

Table 8:Long\-context passkey retrieval under LLA compression \(Ouro\-1\.4B\)\.Exact\-match retrieval of a 5\-digit key hidden at a random depth in filler context\. The uncompressed teacher retrieves perfectly to6464k\. LLA is near\-lossless for retrieval through5\.33×5\.33\\timesacross the whole range, with only the4×4\\times,6464k cell dipping to0\.870\.87\. The10\.67×10\.67\\timespoint is the knee \(perfect to1616k, degrading at3232to6464k\), while21\.33×21\.33\\timesloses the exact token identity a key requires\. LLA rows use the capacity/latent cached decoder, whose two\-pass prefill reaches6464k via streamed layer\-wise reconstruction \(Sec\.[7](https://arxiv.org/html/2607.15456#S7)\)\.There are two implementation regimes\. A full\-memory fast path reconstructs and caches full K/V, preserving throughput and running at1\.16×1\.16\\timesteacher latency in our measurement\. It does not realize the memory reduction\. A latent\-store capacity path caches only the latent and realizes theρ×\\rho\\timesmemory gain\. The current version reconstructs cached tokens during attention and is therefore compute\-bound\. It buys otherwise\-unservable contexts and batches rather than raw decode speed\.

### Per\-head LLA versus LLA\-2D at inference\.

The two family members realize these regimes differently\. Per\-head LLA cachesHHlatents per token and layer \(\(rk\+rv\)​H\(r\_\{k\}\+r\_\{v\}\)Hscalars\) and, on the capacity path, reconstructs each head through its own up\-projection\. LLA\-2D caches a single latent \(\(rk\+rv\)\(r\_\{k\}\+r\_\{v\}\)scalars, a factorHHsmaller\) and reconstructs all heads from one shared up\-projection\. LLA\-2D therefore reaches a strictly smaller cache and a higher capacity ceiling for a given\(rk,rv\)\(r\_\{k\},r\_\{v\}\), while per\-head LLA keeps heads independent and is more accurate at matched loop budget \(Table[3](https://arxiv.org/html/2607.15456#S5.T3)\)\. Both preserve teacher latency on the fast path and are reconstruction\-bound on the capacity path\. Choosing the member is thus an operating\-point decision, favoring LLA\-2D when cache capacity is the hard constraint and per\-head LLA when accuracy is\.

The same structure can become a compute win through an absorbed attention implementation\. Because the attention score is bilinear,q⊤​\(Wup​c\)=\(Wup⊤​q\)⊤​cq^\{\\top\}\(W\_\{\\rm up\}c\)=\(W\_\{\\rm up\}^\{\\top\}q\)^\{\\top\}c, the K up\-projection can be moved to the query side and the attention can read latents directly\. The V side dualizes as∑iai​WupV​ci=WupV​∑iai​ci\\sum\_\{i\}a\_\{i\}W^\{V\}\_\{\\rm up\}c\_\{i\}=W^\{V\}\_\{\\rm up\}\\sum\_\{i\}a\_\{i\}c\_\{i\}\. RoPE prevents exact absorption of the positional part, so we use an MLA\-style split into a NoPE latent content path and a small explicit rotary branch\(DeepSeek\-AI,[2024](https://arxiv.org/html/2607.15456#bib.bib4)\)\. In a looped model the rotary branch is shared across recurrence steps for a token\.

On a random\-weight looped model at the4×4\\timesgeometry, absorbed decode is measured2\.3×2\.3\\timesfaster than reconstruct\-then\-attend at 262k context and matches teacher decode latency with4×4\\timesless cache\. Post\-hoc decoupled RoPE has a fidelity floor, though, since the frozen query was trained for full\-RoPE keys\. A query adapter narrows the gap, but full\-RoPE LLA remains substantially closer to the teacher in the post\-training setting\. We therefore treat absorbed decoupled decode as an implementation path and a pretraining\-native opportunity, not as the main accuracy result\.

## 8Discussion

### Evidence hierarchy\.

The strongest conclusion is axis\-specific\. A looped model creates cache redundancy because the same block and K/V projections are applied repeatedly to a stabilizing recurrent state, and LLA works because it models that trajectory directly\. Four results support this, and they reinforce rather than repeat one another\. The clearest is the final\-loop\-reuse control\. It has the same4×4\\timesfootprint as LLA yet collapses generation, so the loop cannot be replaced by its endpoint\. The frozen\-cache spectra show why, with the loop axis far steeper than the head or layer axes\. The remaining two extend the claim\. Matched\-budget codecs on other axes lose more accuracy, and the pattern reappears when we move from Ouro to Huginn\. Together these rule out the two easy alternatives, that any axis compresses equally well or that recurrence collapses to a single endpoint\.

### Operating regimes\.

Rank sets the deployment point\. Light compression, roughly1\.331\.33to2×2\\timeson Ouro\-1\.4B, is close to a drop\-in replacement for the full cache across the broad suite\. By the middle range, about33to5×5\\times, teacher\-forced fidelity is still useful, but long generations now need the on\-policy stage, because the compressed model visits prefixes the teacher\-forced conversion never saw\. Beyond10×10\\timesthe method is a capacity tool rather than a lossless codec\. It keeps many likelihood tasks and can unlock otherwise unservable batches or contexts, while exact retrieval and long\-form reasoning become rank\-limited\.

### Compositionality\.

The recurrence axis is separate from the axes targeted by prior cache methods\. Head\-axis MLA, cross\-layer sharing, quantization and token eviction change different tensor dimensions, so they can be used as substitutes in an iso\-cache comparison or stacked when the deployment target calls for it\. The results suggest a simple ordering for looped decoders\. Compress recurrence first, since it is the steepest spectrum, then spend additional budget on sequence eviction or head sharing if the application needs smaller caches\.

### From memory compression to native recurrent architectures\.

The post\-training design is a controlled test of the cache structure, not the end of the design space\. A model trained natively with a decoupled latent cache could expose the latent directly to attention, align its query projections with the absorbed implementation, and avoid learning a full\-RoPE cache that is later factorized\. More ambitiously, if the residual\-stream update across loops is itself low\-rank, the recurrence could compute shared content plus cheap per\-loop deltas rather than recomputing the full blockTTtimes\. That would be an architecture change rather than a cache codec\. The evidence here points in that direction, since loops trace a low\-rank path but not a rank\-1 endpoint\.

### What the result does not rely on\.

The main accuracy claims do not rely on the absorbed attention implementation or on a throughput speedup\. They use a reconstruction path that preserves the teacher attention computation except for the cached representation\. The absorbed path is a promising implementation route, especially for native training, but the paper’s central claim is simpler\. Looped decoders materialize a recurrence\-indexed cache whose intrinsic rank is much smaller than its stored width\.

## 9Conclusion

Looped Transformers reduce parameter count by reusing blocks, but their KV cache still scales with recurrence depth\. LLA removes much of that recurrent cache cost by storing a cross\-loop latent trajectory for each token, layer and head\. At matched cache budget, LLA is the strongest codec across the baselines we tested, and the exact scalar reduction translates into measured serving\-capacity gains\. These results make the recurrence axis a practical compression target for looped decoders and a natural target for native latent\-cache training\. More broadly, weight\-tied iteration makes the cache, not the parameters, the quantity that grows with reasoning depth\. As looped and latent\-reasoning models scale, compressing the recurrence axis is what keeps their serving cost bounded\.

## References

- Ainslie et al\. \(2023\)Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\.GQA: Training generalized multi\-query transformer models from multi\-head checkpoints\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.
- Blayney et al\. \(2026\)Hugh Blayney, Alvaro Arroyo, Johan Obando\-Ceron, Pablo Samuel Castro, Aaron Courville, Michael M\. Bronstein, and Xiaowen Dong\.A mechanistic analysis of looped reasoning language models, 2026\.URL[https://arxiv\.org/abs/2604\.11791](https://arxiv.org/abs/2604.11791)\.
- Brandon et al\. \(2024\)William Brandon et al\.Reducing transformer key\-value cache size with cross\-layer attention\.*arXiv preprint arXiv:2405\.12981*, 2024\.
- DeepSeek\-AI \(2024\)DeepSeek\-AI\.Deepseek\-v2: A strong, economical, and efficient mixture\-of\-experts language model\.*arXiv preprint arXiv:2405\.04434*, 2024\.Introduces Multi\-head Latent Attention \(MLA\)\.
- Dehghani et al\. \(2019\)Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser\.Universal transformers\.In*International Conference on Learning Representations \(ICLR\)*, 2019\.
- Geiping et al\. \(2025\)Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R\. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein\.Scaling up test\-time compute with latent reasoning: A recurrent depth approach\.*arXiv preprint arXiv:2502\.05171*, 2025\.
- Hooper et al\. \(2024\)Coleman Hooper et al\.Kvquant: Towards 10 million context length llm inference with kv cache quantization\.*arXiv preprint arXiv:2401\.18079*, 2024\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th ACM Symposium on Operating Systems Principles \(SOSP\)*, 2023\.
- Liu et al\. \(2024\)Akide Liu et al\.Minicache: Kv cache compression in depth dimension for large language models\.*arXiv preprint arXiv:2405\.14366*, 2024\.
- Shazeer \(2019\)Noam Shazeer\.Fast transformer decoding: One write\-head is all you need\.*arXiv preprint arXiv:1911\.02150*, 2019\.
- Su et al\. \(2021\)Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*arXiv preprint arXiv:2104\.09864*, 2021\.
- Sun et al\. \(2024\)Yutao Sun et al\.You only cache once: Decoder\-decoder architectures for language models\.*arXiv preprint arXiv:2405\.05254*, 2024\.
- Vendrell et al\. \(2026\)Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolo Grillo, Jordi Ros\-Giralt, Arash Behboodi, and Fabio Valerio Massoli\.Memory\-efficient looped transformer: Decoupling compute from memory in looped language models\.*arXiv preprint arXiv:2605\.07721*, 2026\.
- Xiao et al\. \(2024\)Guangxuan Xiao et al\.Efficient streaming language models with attention sinks\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Zhang et al\. \(2023\)Zhenyu Zhang et al\.H2O: Heavy\-hitter oracle for efficient generative inference of large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Zhu et al\. \(2025\)Rui\-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian\.Scaling latent reasoning via looped language models\.*arXiv preprint arXiv:2510\.25741*, 2025\.Ouro/LoopLM; models ByteDance/Ouro\-1\.4B, Ouro\-2\.6B\-Thinking\.

## Appendix AExperimental details

### Models\.

All experiments use frozen, publicly released looped/recurrent\-depth Transformers\.Ouro\-1\.4B\(the main teacher\):D=24D=24weight\-tied layers,H=16H=16attention heads of widthdhead=128d\_\{\\rm head\}=128, run atT=4T=4recurrence steps\.Ouro\-2\.6B\-Thinking: deeper Ouro variant, alsoT=4T=4, used for the scale study\.Huginn\-3\.5B: a different recurrent\-depth family, evaluated atR=32R=32recurrence steps for the family\-transfer study\. The teacher weights are never updated; only the K/V codec is trained\.

### LLA codec and conversion\.

For each layer, head and axis \(K, V\) we collect the teacher’s pre\-RoPE per\-loop activations on a fixed calibration set of255255prompts and initializeWdownW\_\{\\rm down\}from the top\-rrright singular vectors of the stacked\-loop matrix \(loop slices give the initialWup,tW\_\{\{\\rm up\},t\}\)\. The SVD\-initialized codec is then fine\-tuned with the teacher frozen usingtrain\_rmla\.py: objective = forward KL\(teacher∥\\\|swapped\)\+λattn=0\.5\+\\;\\lambda\_\{\\rm attn\}\\\!=\\\!0\.5per\-loop attention\-output matching \(Eq\.[5](https://arxiv.org/html/2607.15456#S3.E5)\);30003000steps \(300300for the design ablations of Table[10](https://arxiv.org/html/2607.15456#A5.T10)\), batch size88, sequence length512512, AdamW, learning rate1​e−31\\mathrm\{e\}\{\-3\}with100100\-step warmup and cosine decay to5%5\\%, held\-out KL evaluated every200200steps \(n=32n=32\)\. We select the*best\-KL*checkpoint \(not the final step\); a codec whose final\-step KL exceeds its best is reported at its best checkpoint\. Ranks\(rk,rv\)\(r\_\{k\},r\_\{v\}\)per compressionρ\\rhoare listed in Table[1](https://arxiv.org/html/2607.15456#S5.T1); we userv\>rkr\_\{v\}\>r\_\{k\}throughout \(V converges more slowly, Sec\.[4](https://arxiv.org/html/2607.15456#S4)\)\.LLA\-2Dshares the same recipe but stores one latent per token and layer, reconstructing allHHheads from it\. RoPE is applied*after*reconstruction \(the rotary phase is shared across loops for a token\), so no decoupled\-RoPE branch is needed for the content path\.

### Matched\-axis baselines\.

All learned baselines use the identical conversion recipe above; only the codec architecture changes\.*mla\_head*: per\-loop MLA \(head\-axis latent, no cross\-loop sharing\)\.*cla*: cross\-layer K/V sharing overs∈\{4,6,12,24\}s\\in\\\{4,6,12,24\\\}adjacent layers\.*kv\_quant*: inference\-only low\-bit K/V quantization at\{2,3,4,8\}\\\{2,3,4,8\\\}bits \(no codec training, hence no train\-KL, and it cannot exceed∼\\sim8×\\times\)\.*final\-loop reuse*: a zero\-parameter control that caches only the last loop’s K/V and reuses it for all loops\.

### Evaluation suite and protocols\.

GSM8K: 5\-shot at 1\.4B and 2\-shot at 2\.6B, strict\-match EM,max\_gen\_toks=256\. The 1\.4B teacher is scored on the full13191319\-item test set; the iso\-cache and broad\-suite codecs use a100100\-item subset with the faithful re\-forward decoder, while the tier4 cold\-start sweep \(Table[1](https://arxiv.org/html/2607.15456#S5.T1)\) uses a200200\-item subset with the earlier streaming reconstruction decoder and a100100\-item teacher reference\. At 2\.6B, scored with the faithful decoder, the teacher and the per\-head and head\-axis arms use the full13191319items and the LLA\-2D arms use a100100\-item subset\.MATH\-500: 4\-shot,max\_gen\_toks=2048, batch 1, faithful decoder, scored bymath\_verifysymbolic equivalence; teacher on all500500items, codecs on200200\.BBH: 3\-shot,max\_gen\_toks=1024, batch 1, exact match, faithful decoder; teacher2525items/sub\-task \(675675docs\) and codecs55/sub\-task \(135135docs, identical items across arms\)\.HumanEval/MBPP: EvalPlus pass@1 \(base/plus\) on the full sets \(164164/378378\)\.MMLU\-Pro: exact match on the full set\. The commonsense/knowledge MC suite \(ARC\-e/c, HellaSwag, PIQA, WinoGrande, OpenBookQA, CommonsenseQA, MMLU\) is 0\-shot loglikelihood accuracy on the full sets and is decoder\-independent\. Within each table the codec and baseline arms are scored on identical items; the teacher reference is scored on the full set where the two differ\. Passkey \(Table[8](https://arxiv.org/html/2607.15456#S7.T8)\) hides a 5\-digit key at a random depth in filler context,1515trials/cell \(55keys×\\times33depths\), scored by exact retrieval; contexts22k to6464k\. “train\-KL” is the held\-out teacher\-forced KL of the codec and is decoder\-independent\.

### Decoding and caching\.

Two decoders are used\. The*cached reconstruction decoder*is two\-pass: pass 1 captures the teacher’s pre\-RoPE per\-loop K/V, the codec reconstructs them, and pass 2 re\-forwards forcing the reconstruction so the model’s own RoPE and cache store the post\-RoPE reconstruction\. This matches the training objective and is used for the iso\-cache, broad\-suite, 2\.6B and MATH\-500/BBH numbers; the tier4 cold\-start sweep used an earlier streaming reconstruction decoder\. The*capacity / latent\-store decoder*caches only the latent and reconstructs on read, realizing theρ×\\rho\\timesmemory saving; its two\-pass prefill is streamed layer\-by\-layer so long prompts \(to6464k\) fit in memory\. Two serving regimes \(Sec\.[7](https://arxiv.org/html/2607.15456#S7)\): a full\-memory*fast path*\(caches reconstructed full K/V, teacher\-latency, no memory saving\) and a*capacity path*\(stores latents, reconstruction\-compute\-bound\)\. On\-policy refinement samples the codec’s own rollouts \(temperature sampling, no reward/correctness filtering\) and minimizes the frozen teacher’s per\-token KL to those visited states with a stop\-gradient through sampling\.

### Hardware and precision\.

Codecs are trained and evaluated in bfloat16\. Serving\-capacity and latency measurements \(Table[7](https://arxiv.org/html/2607.15456#S7.T7)\) are on a single H200 at context length40964096; generative evaluations use B200 GPUs \(183 GB\) with per\-process memory fraction0\.990\.99for the long\-context and 2\.6B runs\.

## Appendix BAdditional figures

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_why_trajectory_clean_tight.png)Figure 6:Loop trajectories are low\-rank, not collapsed\.In frozen Ouro\-1\.4B, cross\-loop K/V cosines increase toward the final loop, and one direction captures most but not all cross\-loop variance\. The PCA view shows short, consistent paths rather than a single shared endpoint\.![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_pareto_26b_tight.png)Figure 7:Ouro\-2\.6B\-Thinking performance versus KV\-cache compression\.The layout matches Fig\.[4](https://arxiv.org/html/2607.15456#S5.F4)\. The loop\-axis codec remains closest to the teacher on train\-KL, GSM8K and the commonsense/knowledge suite\. The precision baseline cannot reach the largest ratios without collapse\.![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_capacity_clean_tight.png)Figure 8:The latent cache gives an exact memory reduction and measured serving capacity\.Left: cache memory scales linearly with context and is divided by the compression ratio\. Right: at fixed VRAM and context length 4096, maximum batch size follows the idealρ\\rhoscaling in measured capacity tests\.
## Appendix CToken\-eviction composition table

Table 9:LLA composes with token eviction at no extra quality cost\.GSM8K \(5\-shot, strict EM\) under StreamingLLM/H2O\-style eviction to a260260\-token budget \(sink44\+ window256256\)\. The∼\\sim1k\-token 5\-shot prompts are∼\\sim74% evicted, so the sequence\-axis cost is real \(both rows fall from≈0\.79\\approx 0\.79without eviction; teacher/LLA\-4×4\\timesno\-eviction reference in Table[3](https://arxiv.org/html/2607.15456#S5.T3)\)\. Eviction on a near\-lossless 8\-bit codec isolates the sequence axis, and LLA\-4×4\\timesadds the recurrence axis on top\. The two remain close, confirming that the axes are orthogonal\. LLA reconstructs the*kept*tokens accurately and adds little beyond eviction’s own cost\. Cache size composes exactly \(Table[7](https://arxiv.org/html/2607.15456#S7.T7)\)\. LLA\-21\.3×21\.3\\times\+\+eviction caps the 1M\-context 1\.4B cache at3636MiB\.
## Appendix DOn\-policy rank Pareto \(figure\)

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_onpolicy_pareto_revised.png)Figure 9:On\-policy distillation stabilizes the long\-generation Pareto\.On reliably scored MATH\-500, self\-rollout distillation raises the low\-compression points and reduces no\-answer generations throughout the Pareto\. Hollow markers show independent re\-evaluations with bootstrap 95% confidence intervals\.
## Appendix EDesign\-ablation detail

Table 10:Design\-choice ablations\.Held\-out teacher\-forced KL for a4×4\\timescodec after a matched 300\-step conversion\. The reference is SVD initialization, split ranksrk/rv=96/160r\_\{k\}/r\_\{v\}=96/160and attention\-output matching\.AblationVariantheld\-out KL↓\\downarrowinit priorSVD \(ours\)0\.105init priort=1t=1SVD0\.128init priorrandom0\.771rank splitrk<rvr\_\{k\}<r\_\{v\}\(96/160\)\(96/160\)0\.105rank splitshared rank\(128/128\)\(128/128\)0\.129lossKL\+λattn=0\.5\+\\lambda\_\{\\rm attn\}=0\.50\.105lossKL only0\.110![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_ablation_init_tight.png)Figure 10:SVD initialization dominates random initialization\.Held\-out KL for the4×4\\timescodec falls quickly from the SVD teacher prior\. A random codec does not match SVD initialization within the same conversion budget\.
## Appendix FFull 2\.6B commonsense and knowledge suite

Table 11:Ouro\-2\.6B\-Thinking: full commonsense and knowledge suite\.All tasks are 0\-shot loglikelihood accuracy and decoder\-independent\. Per\-head LLA is near\-lossless at every budget and remains several points above the head and layer baselines and the LLA\-2D variant\.
## Appendix GCross\-loop stabilization at 2\.6B

Fig\.[11](https://arxiv.org/html/2607.15456#A7.F11)repeats the trajectory diagnostic on Ouro\-2\.6B\-Thinking\. The attention K/V stabilizes within the first loop, each token\-head unit has a dominant but not exhaustive loop direction, and the value cache remains the higher\-variance axis\.

![Refer to caption](https://arxiv.org/html/2607.15456v1/figures/fig_why_trajectory_26b_clean_tight.png)Figure 11:Cross\-loop K/V stabilization on Ouro\-2\.6B\-Thinking\.The 2\.6B model reproduces the same pattern as Ouro\-1\.4B\. K/V stabilizes across loops, one direction dominates but does not explain all variance, and the value cache has higher cross\-loop variance than the key cache\.