共享记忆在循环 Transformer 中的惊人有效性

# The Surprising Effectivenessof Shared Memory in Looped Transformers
Source: [https://arxiv.org/html/2610.02383](https://arxiv.org/html/2610.02383)
Keshav RamjiAffiliation:IBM ResearchYousef El\-KurdiAffiliation:IBM ResearchLuis A\. LastrasAffiliation:IBM ResearchYoav Artzi Nathan Godey Ramón Fernandez AstudilloAffiliation:Cornell University

###### Abstract

Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters\. Each recursion, however, writes its own key\-value cache, so memory still grows with compute\. Inference\-time techniques can shrink this cache at a cost in quality\. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own\. Surprisingly, we find that sharing memory does not cost quality and instead improves it\. At 150M–1B parameters, our Looped Prediction Transformer \(lpt\) and its hybrid variant set a new quality–memory frontier for looped models: with five recursions, the hybrid lowers validation perplexity on FineWeb\-Edu by1\.121\.12–1\.821\.82relative to a same\-size standard Transformer while using7676–79%79\\%less context memory\. Through an extensive analysis, we investigate why memory sharing helps\. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion\.

11footnotetext:Work done during an internship at IBM Research\. Correspondence to:giovanni@cs\.cornell\.edu22footnotetext:Equal advisingFigure 1:Shared memory not only saves memory, it also improves quality\.Left:in our Looped Prediction Transformer \(lpt\), later recursions read the KV cache written by the first recursion and keep only a short window of their own\.Right:perplexity and context memory of11B\-parameter models with two to five recursions, against a parameter\-matched standard Transformer \(std\)\. Each of our models adds shared memory to the looped baseline above it:lptto the looped Transformerlt, andh\-lptto the gated DeltaNet hybridlt2\.## 1Introduction

Looped Transformers \(lt\) repeatedly apply the same layers several times before predicting the next token, adding computation without adding parameters\([Dehghani et al\., 2019](https://arxiv.org/html/2610.02383#bib.bib13);[Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. In principle, they trade time for memory, matching the compute of a larger model with far fewer parameters\. In standard formulations, however, context memory still grows with recursion depth, as each iteration keeps a separate key\-value \(KV\) cache\([Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12);[Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. At long contexts, this context memory can grow larger than the weights memory itself, undoing the very reason to use a looped model\.

Sharing caches across recursions during decoding can save substantial memory with little quality loss\([Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12);[Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. So far, this reuse has been studied mainly as a way to save memory\. Yet it also shows that memory written by one recursion can serve the others, even in models never trained to share it\. This raises a question:*if sharing can work untrained, what can it do when models are trained for it?*To answer it, we train looped models with memory sharing from the start, investigating it as an architectural inductive bias\.

We study a simple recipe for*Memory Sharing*in looped Transformers \([Figure 1](https://arxiv.org/html/2610.02383#S0.F1), left\)\. At each full\-attention layer, later recursions attend to both their own KV entries and the cache written by that layer in the first recursion, within the same attention operation\. To further save memory, we truncate the cache of every recursion after the first to a short sliding window\. The resulting Looped Prediction Transformer \(lpt\) keepslt’s sequential recurrence and uses context memory close to a standard \(non\-looped\) Transformer’s\. Applied to the full\-attention layers of a hybrid looped Transformer\([Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\), the same rule yields the Hybrid Looped Prediction Transformer \(h\-lpt\)\. We pretrain all models from scratch at150150M to11B parameters\. Because later recursions give up most of their own memory, one would expect Memory Sharing to cost some quality, as untrained sharing does\.

Surprisingly, Memory Sharing*improves*quality over full\-cachelteven when each later recursion’s own cache covers less than2%2\\%of the context\. Its advantage grows with depth:ltstops improving in perplexity after three recursions, whilelptimproves through five\. Across sizes,lptlowers validation perplexity by up to1\.091\.09nats and raises downstream accuracy by up to4\.84\.8percentage points over a matched\-depthlt, with up to4\.7×4\.7\\timesless context memory and nearly the same inference FLOPs\. Sharing also improves the hybrid baseline: at five recursions,h\-lptoutperforms a same\-size standard Transformer in both language modeling and downstream accuracy while using7676–79%79\\%less context memory, moving the quality–memory frontier for looped models \([Figure 2](https://arxiv.org/html/2610.02383#S4.F2)\)\.

These results raise a further question:*why does memory sharing improve quality?*We find that later recursions rely heavily on the shared memory\.lptoutperformsltat every window size, even when later recursions keep no cache of their own, and later recursions attend mostly to shared memory even when their own cache covers the full context\. Shared and local memory also develop distinct key representations, suggesting that they serve different purposes\. Reading shared memory sends gradients directly to the first recursion, helping it learn what to store\. This gradient highway bypasses the intermediate recursions: inlt, the magnitude of the gradient reaching the first recursion falls sharply with depth, whilelptrecovers much of it through the shared memory\. When we block this path, the model still reads shared memory but no longer matcheslpt’s quality and stability\.

## 2Looped Transformers

A standardltshares its weights across recursions, while each recursion computes its own hidden states and KV entries\. We use Ouro’s full\-stack formulation\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\), which builds on earlier recurrent\-depth architectures\([Dehghani et al\., 2019](https://arxiv.org/html/2610.02383#bib.bib13);[Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12)\)\.

LetℳL=𝒯θ\(L\)∘⋯∘𝒯θ\(1\)\\mathcal\{M\}^\{L\}=\\mathcal\{T\}\_\{\\theta\}^\{\(L\)\}\\circ\\cdots\\circ\\mathcal\{T\}\_\{\\theta\}^\{\(1\)\}denote the shared stack ofLLTransformer blocks\. For a sequence of lengthTTand hidden dimensiondd, lethin\(t\)∈ℝT×dh\_\{\\mathrm\{in\}\}^\{\(t\)\}\\in\\mathbb\{R\}^\{T\\times d\}andh\(t\)h^\{\(t\)\}be the residual streams entering and leaving recursiontt\. The stack is appliedRRtimes:

hin\(1\)\\displaystyle h\_\{\\mathrm\{in\}\}^\{\(1\)\}=emb⁡\(𝐱\),\\displaystyle=\\operatorname\{emb\}\(\\mathbf\{x\}\),\(1\)h\(t\)\\displaystyle h^\{\(t\)\}=ℳL​\(hin\(t\)\),\\displaystyle=\\mathcal\{M\}^\{L\}\\\!\\big\(h\_\{\\mathrm\{in\}\}^\{\(t\)\}\\big\),hin\(t\+1\)\\displaystyle h\_\{\\mathrm\{in\}\}^\{\(t\+1\)\}=𝒩rec​\(h\(t\)\),\\displaystyle=\\mathcal\{N\}\_\{\\mathrm\{rec\}\}\\\!\\big\(h^\{\(t\)\}\\big\),logits\(t\)\\displaystyle\\text\{logits\}^\{\(t\)\}=lm​\_​head\(𝒩out\(h\(t\)\)\),t=1,…,R\.\\displaystyle=\\operatorname\{lm\\\_head\}\\\!\\big\(\\mathcal\{N\}\_\{\\mathrm\{out\}\}\(h^\{\(t\)\}\)\\big\),\\qquad t=1,\\dots,R\.Here𝒩rec\\mathcal\{N\}\_\{\\mathrm\{rec\}\}and𝒩out\\mathcal\{N\}\_\{\\mathrm\{out\}\}are separately parameterized RMSNorms\. Both act onh\(t\)h^\{\(t\)\}: the recurrent norm prepares the next recursion’s input, while the readout norm feeds the language\-modeling head\. The effective depth isL⋅RL\\cdot R, with the parameters of a singleLL\-layer stack\.

Each layer computes its keys and values from the hidden states it receives at the current recursion\. For a fixed layer, let𝒞\(t\)\\mathcal\{C\}^\{\(t\)\}denote the cache written by recursiontt, with entry\(kj\(t\),vj\(t\)\)\(k\_\{j\}^\{\(t\)\},v\_\{j\}^\{\(t\)\}\)at positionjj\. We suppress the layer index throughout: for vanilla LT, each layer reads only its own entries\. A query at positioniireads the causal prefix of its recursion’s cache, giving the attention set

𝒜lt\(t\)​\(i\)=𝒞≤i\(t\)=\{\(kj\(t\),vj\(t\)\):j≤i\}\.\\mathcal\{A\}\_\{\\text\{\{lt\}\}\}^\{\(t\)\}\(i\)=\\mathcal\{C\}\_\{\\leq i\}^\{\(t\)\}=\\\{\(k\_\{j\}^\{\(t\)\},v\_\{j\}^\{\(t\)\}\):j\\leq i\\\}\.\(2\)At inference, each layer therefore retainsRRfull\-context caches, one per recursion, and context memory grows linearly with recursion depth even though the parameter count stays fixed\. Some of these caches can be dropped or merged during decoding\([Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12);[Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\), but the model is still trained with a separate cache for every recursion\.

Hybrid architectures reduce this memory cost by replacing most full\-attention layers with linear attention, whose recurrent state does not grow with context length\([Katharopoulos et al\., 2020](https://arxiv.org/html/2610.02383#bib.bib36);[Schlag et al\., 2021](https://arxiv.org/html/2610.02383#bib.bib37);[Yang et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib38)\)\.lt2\([Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\)loops such a stack, using Gated DeltaNet layers and one full\-attention layer per five\. This reduces the number of full\-attention caches, but each remaining layer still needs a separate full\-context cache at every recursion\.

## 3Looping with Shared Memory

We study*Memory Sharing*, in which later recursions read the memory written by the first recursion, as an inductive bias for looped models\. The first recursion runs as inltand writes a full\-context cache𝒞\(1\)\\mathcal\{C\}^\{\(1\)\}at each layer, and every later recursion attends to this cache together with its own KV entries, using a single softmax\.

To also keep looping memory\-efficient, we restrict each later recursion’s own cache to a short window\. The first recursion therefore acts as a memory phase that builds the model’s only full\-context memory, while later recursions use it to refine the prediction\. We refer to the first recursion as the*state recursion*and to the later ones as*prediction recursions*, and call the resulting model the Looped Prediction Transformer \(lpt;[Figure 1](https://arxiv.org/html/2610.02383#S0.F1)\)\. Likelt,lptapplies the sameLL\-layer stackRRtimes and passes the final output to the language\-modeling head \([Equation 1](https://arxiv.org/html/2610.02383#S2.E1)\)\.

Letω\\omegabe the number of preceding positions kept in each prediction recursion’s window, and write𝒞\[i−ω,i\]\(t\)\\mathcal\{C\}\_\{\[i\-\\omega,i\]\}^\{\(t\)\}for the entries from positionsi−ωi\-\\omegathroughii, including the current entry\. The state recursion reads its own cache, as inlt\([Equation 2](https://arxiv.org/html/2610.02383#S2.E2)\)\. A prediction recursion reads the shared memory together with its own window\. The entries read by a query at positioniiin recursionttare

𝒜lpt\(t\)​\(i\)=\{𝒞≤i\(1\),t=1,𝒞≤i\(1\)∪𝒞\[i−ω,i\]\(t\),t=2,…,R\.\\mathcal\{A\}\_\{\\text\{\{lpt\}\}\}^\{\(t\)\}\(i\)=\\begin\{cases\}\\mathcal\{C\}\_\{\\leq i\}^\{\(1\)\},&t=1,\\\\ \\mathcal\{C\}\_\{\\leq i\}^\{\(1\)\}\\cup\\mathcal\{C\}\_\{\[i\-\\omega,i\]\}^\{\(t\)\},&t=2,\\ldots,R\.\\end\{cases\}\(3\)In prediction recursions, the two sets are attended to using a single softmax\.

Settingω=T\\omega=Tgives every prediction recursion a full\-context cache of its own, as inlt, and leaves access to shared memory as the only difference between the two models; we use this setting in[Section 6](https://arxiv.org/html/2610.02383#S6)to separate the effect of sharing from that of the short window\.

Memory Sharing also applies to looped hybrid Transformers, which interleave linear\-attention and full\-attention layers\([Yang et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib38);[Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\)\. The first recursion runs the hybrid as usual\. In later recursions, each linear\-attention layer keeps its own recurrent state, which is small and fixed in size, while each full\-attention layer applies[Equation 3](https://arxiv.org/html/2610.02383#S3.E3), as inlpt\. Applied tolt2\([Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\), this givesh\-lpt\.

### Efficiency and Costs

In training, we implement[Equation 3](https://arxiv.org/html/2610.02383#S3.E3)as a FlexAttention\([Dong et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib9)\)block mask over the concatenated shared and local entries, which skips the fully masked blocks, so each query attends over the shared prefix and itsω\\omegawindow entries only \([Appendix A](https://arxiv.org/html/2610.02383#A1)\)\.

During decoding, each prediction recursion keeps a sliding buffer of at mostω\\omegapreceding entries per layer; its current entry participates in attention before entering the buffer\. In cached token positions acrossLLfull\-attention layers, the inference\-time KV cache sizes are

MKVlt=L​R​T,MKVlpt=L⁡\[T\+\(R−1\)​ω\],M\_\{\\mathrm\{KV\}\}^\{\\text\{\{lt\}\}\}=L\\,R\\,T,\\qquad M\_\{\\mathrm\{KV\}\}^\{\\text\{\{lpt\}\}\}=L\\left\[T\+\(R\-1\)\\omega\\right\],\(4\)where the common head dimensions and datatype size are omitted\.lptuses the same parameters and recursion count as a matched\-depthlt, but performs slightly more attention because each prediction recursion also reads its local window\. At a full context ofT=4,096T=4\{,\}096withω=64\\omega=64and five recursions, these windows increase the number of attended KV entries per generated token by only\(R−1\)​ω/\(R​T\)=1\.25%\(R\-1\)\\omega/\(RT\)=1\.25\\%, while reducing the full\-attention cache relative toltby4\.7×4\.7\\times\.

Table 1:lptandh\-lptreach lower perplexity than their looped baselines with much smaller caches\.Main results across S, M, and L: validation perplexity, mean five\-task accuracy \(%\), and context memory and FLOPs relative to same\-sizestd\. In thestdrows,R=1R=1is the standard Transformer, andR\>1R\>1isdeepstd\. Elsewhere,RRcounts model passes\.Boldandunderlinemark the best and second\-best within each size and column group; our rows are shaded\. We replicate the S results with three additional seeds and find the gains of Memory Sharing statistically significant \(p<0\.01p<0\.01,[Appendix B](https://arxiv.org/html/2610.02383#A2)\)\.R=1R=1R=2R=2R=3R=3R=5R=5SizeModelPPL↓\\downarrowAcc\.↑\\uparrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowstd16\.8316\.8351\.551\.517\.0717\.0750\.250\.21\.501\.501\.1917\.2717\.2750\.750\.71\.881\.881\.3117\.6917\.6950\.450\.42\.502\.501\.54loopie16\.5816\.5851\.651\.62\.002\.001\.831\.8316\.5016\.5051\.951\.93\.003\.002\.662\.6616\.5616\.5652\.152\.15\.005\.004\.324\.32lt16\.1716\.1751\.551\.52\.002\.001\.831\.8315\.9815\.9852\.152\.13\.003\.002\.662\.6616\.0016\.0052\.452\.45\.005\.004\.324\.32plt15\.8415\.8452\.71\.021\.021\.841\.8415\.5515\.5552\.952\.91\.031\.032\.672\.6715\.4915\.4953\.553\.51\.061\.064\.344\.34lt215\.7651\.951\.90\.351\.2915\.4715\.4752\.552\.50\.521\.8615\.1615\.1653\.753\.70\.872\.98lpt15\.7351\.951\.91\.021\.021\.841\.8415\.2854\.01\.031\.032\.672\.6714\.9154\.91\.061\.064\.344\.34Sh\-lpt15\.7352\.40\.181\.2915\.3953\.10\.191\.8615\.0154\.60\.212\.98std13\.7913\.7956\.156\.114\.2414\.2455\.555\.51\.441\.441\.1314\.4714\.4755\.455\.41\.781\.781\.2414\.8814\.8854\.454\.42\.342\.341\.43loopie13\.6913\.6956\.956\.92\.002\.001\.921\.9213\.7513\.7555\.855\.83\.003\.002\.842\.8413\.7413\.7456\.656\.65\.005\.004\.694\.69lt13\.2813\.2857\.557\.52\.002\.001\.921\.9213\.1313\.1359\.459\.43\.003\.002\.842\.8413\.3413\.3457\.957\.95\.005\.004\.694\.69plt13\.0013\.0059\.11\.021\.021\.931\.9312\.6812\.6860\.060\.01\.031\.032\.852\.8512\.6512\.6560\.560\.51\.061\.064\.714\.71lt212\.8159\.10\.351\.4312\.6312\.6359\.559\.50\.522\.1012\.4312\.4360\.260\.20\.873\.45lpt12\.9112\.9159\.21\.021\.021\.931\.9312\.5560\.21\.031\.032\.852\.8512\.3361\.61\.061\.064\.704\.70Mh\-lpt12\.8058\.558\.50\.181\.4312\.5460\.30\.192\.1012\.3161\.40\.213\.45std11\.3511\.3562\.562\.511\.6211\.6262\.562\.51\.451\.451\.1411\.9611\.9661\.361\.31\.731\.731\.1712\.2712\.2761\.461\.42\.252\.251\.32loopie11\.2611\.2663\.563\.52\.002\.001\.951\.9511\.2811\.2863\.963\.93\.003\.002\.912\.9112\.2612\.2661\.061\.05\.005\.004\.814\.81lt11\.2011\.2064\.264\.22\.002\.001\.951\.9511\.1111\.1163\.863\.83\.003\.002\.912\.9111\.3911\.3963\.463\.45\.005\.004\.814\.81plt10\.9510\.9564\.864\.81\.021\.021\.961\.9610\.7510\.7566\.866\.81\.031\.032\.922\.9210\.5910\.5966\.466\.41\.061\.064\.834\.83lt210\.5966\.50\.401\.5310\.4710\.4765\.865\.80\.602\.2710\.3267\.367\.31\.013\.75lpt10\.7510\.7566\.31\.021\.021\.961\.9610\.4567\.31\.031\.032\.922\.9210\.3268\.21\.061\.064\.834\.83Lh\-lpt10\.5865\.665\.60\.211\.5310\.4166\.90\.222\.2710\.2368\.10\.243\.75

## 4Experimental Setup

### Baselines

To study memory sharing, we comparelptwithltandh\-lptwithlt2\. We also compare with the Parallel Loop Transformer \(plt\)\([Wu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib10)\), which parallelizes recursions and uses gated dual attention over the first recursion’s shared cache and each later recursion’s local cache\. Other baselines are a standard Transformer \(std\),loopie, which repeats each layer before advancing to the next\([Gao et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib25)\), anddeepstd, a deeper, narrower Transformer without weight sharing that matches the looped models in effective depth and approximately in parameters\.

### Architecture

All models use a decoder\-only Transformer backbone with RMSNorm, SwiGLU, rotary positional embeddings, and tied input and output embeddings\. We train S, M, and L models with approximately150150M,450450M, and11B parameters; the main comparisons cover all three sizes, and most ablations and analyses use S and M\. The context length is4,0964\{,\}096\. Unless otherwise specified,lptandh\-lptuse a prediction window ofω=64\\omega=64\. We study two, three, and five recursions\. This range includes the two loop steps ofloopie\([Gao et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib25)\)and surrounds the four recursions of Ouro andlt2\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11);[Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\)\.111Nine recursions bring instability in several model families and limited accuracy gains, consistent with the loss spikes Ouro reports at eight\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. We report these experiments in[Appendix E](https://arxiv.org/html/2610.02383#A5)\.All models use nanoGPT’s default optimizer settings\([Karpathy, 2022](https://arxiv.org/html/2610.02383#bib.bib39)\)\. Model dimensions, normalization, and training details are in[Appendix A](https://arxiv.org/html/2610.02383#A1)\.

### Data and Training

We pretrain on FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.02383#bib.bib29)\)using2020B tokens for S and M, and5050B for L, roughly22–7×7\\timesthe Chinchilla compute\-optimal token budgets\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.02383#bib.bib28)\)\. The main tables report one run per configuration, with the same initialization seed and data order across models\. At S, we also train three more seeds per model and confirm that seed variation is far below the gaps we report \([Appendix B](https://arxiv.org/html/2610.02383#A2)\)\.

We study two training objectives\.*Final loss*computes the next\-token NLL at the final recursion only\.*Adaptive loss*follows Ouro\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\), weighting per\-recursion NLLs with a learned per\-token halting distribution and adding KL regularization\. Unless otherwise specified,ltuses adaptive loss and all other models use final loss\.

### Evaluation

We report perplexity on1010M held\-out FineWeb\-Edu tokens and mean zero\-shot accuracy on ARC\-Easy\([Clark et al\., 2018](https://arxiv.org/html/2610.02383#bib.bib30)\), HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.02383#bib.bib31)\), PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.02383#bib.bib32)\), SciQ\([Welbl et al\., 2017](https://arxiv.org/html/2610.02383#bib.bib33)\), and LAMBADA\([Paperno et al\., 2016](https://arxiv.org/html/2610.02383#bib.bib34)\), using the LM Evaluation Harness\([Gao et al\., 2023](https://arxiv.org/html/2610.02383#bib.bib35)\)\. We compute context memory and decoding FLOPs per generated token analytically at a4,0964\{,\}096\-token context, relative to the same sizestd\. Context memory includes the KV cache and GDN’s recurrent state\.

Figure 2:lptandh\-lptreach higher accuracy with much smaller caches than the looped baselines\.Mean five\-task accuracy vs\. context memory at each size\. Curves follow two, three, and five recursions, with larger markers at five\. Dashed lines mark single\-passstd\.

## 5Results

### Memory Sharing Improves Quality

Restricting the memory written by later recursions could hurt prediction\. Surprisingly, our Memory Sharing approach improves quality\. At matched size and recursion count,lptlowers validation perplexity by0\.380\.38–1\.091\.09and raises accuracy by0\.40\.4–4\.84\.8percentage points overlt, with2\.02\.0–4\.7×4\.7\\timesless KV memory and under1%1\\%more FLOPs \([Table 1](https://arxiv.org/html/2610.02383#S3.T1)\)\. Measured on an H100, the memory savings are the same, solptfits2\.02\.0–4\.8×4\.8\\timesas many sequences in a batch aslt\([Appendix F](https://arxiv.org/html/2610.02383#A6)\)\. The perplexity gain grows with recursion count \([Table 1](https://arxiv.org/html/2610.02383#S3.T1)\):ltstops improving beyond three recursions at every size, whilelptcontinues to improve through five\. At comparable cache sizes,lptalso reaches lower perplexity thanplt’s parallel formulation\.

When it comes to hybrid variants,h\-lptkeeps this memory at0\.180\.18–0\.24×0\.24\\timesstd’s across recursion counts, whilelt2’s remaining full\-attention caches still grow with depth, approachingstd’s context memory at five recursions\. Despite these22–4×4\\timesmemory gains,h\-lptmatches or outperformslt2in terms of quality at every recursion count\. At five recursions, it lowers perplexity by0\.090\.09–0\.150\.15and raises accuracy by0\.80\.8–1\.21\.2points overlt2, with4\.14\.1–4\.2×4\.2\\timesless context memory and about0\.1%0\.1\\%more inference FLOPs\.

Incidentally, we note that for low recursion counts, the hybridlt2improves onltwhile using less memory and compute\. A factor that could explain this gap – and the instability oflt– is Ouro’s normalization, which we use for bothltandlpt\. In[Appendix C](https://arxiv.org/html/2610.02383#A3), we explore other normalizations that makeltmore robust, but none of these variants matcheslpt\.

### More Quality from the Same Parameters

Both memory\-sharing models improve onstdwith little additional memory, or much less forh\-lpt\([Figure 2](https://arxiv.org/html/2610.02383#S4.F2)\)\. At five recursions,lptlowers perplexity by1\.031\.03–1\.921\.92and raises accuracy by3\.43\.4–5\.75\.7points with6\.25%6\.25\\%more context memory\.h\-lptlowers perplexity by1\.121\.12–1\.821\.82and raises accuracy by3\.13\.1–5\.65\.6points with7676–79%79\\%less memory\. This smaller cache letsh\-lptfit3\.63\.6–5\.0×5\.0\\timesas many sequences asstdon one GPU\. With three to five recursions,h\-lptlowers perplexity by0\.940\.94–1\.821\.82while beating or staying close tostd’s throughput \(1\.41\.4–1\.5×1\.5\\timesat three,8181–88%88\\%at five,[Appendix F](https://arxiv.org/html/2610.02383#A6)\)\. Both models’ accuracy gains grow from two to five recursions at every size\.lptalso outperformsloopieanddeepstd, neither of which improves consistently with depth\.

Takeaway 1\.Through Memory Sharing,lptand especiallyh\-lptimproves the quality–memory frontier for looped models \([Figure 2](https://arxiv.org/html/2610.02383#S4.F2)\)\.

## 6Why Memory Sharing Helps

We use S\- and M\-size models with two, three, and five recursions to study why this inductive bias improves quality \([Appendix D](https://arxiv.org/html/2610.02383#A4)\)\.

### Memory Sharing Stabilizes Deep Recursion

We compare final and adaptive loss inltandlpt\([Figure 4](https://arxiv.org/html/2610.02383#S6.F4)\)\. Without intermediate supervision, adding recursions consistently worsenslt’s perplexity, and the model collapses to unigram\-level loss at five recursions\.lptstays stable, and final loss lowers its perplexity by0\.160\.16–0\.500\.50compared with adaptive loss\. Its intermediate recursions can refine the hidden state without having to predict early\.

Takeaway 2\.lptstays stable and trains best with final loss\.

![Refer to caption](https://arxiv.org/html/2610.02383v1/supervision_ablation_compact.png)

Figure 3:lpttrains best with final loss, whileltconsistently worsens with more recursions without adaptive loss\.Validation perplexity \(↓\\downarrow\), collapsed runs in the upper strip on a log scale\. Filled markers are final loss, hollow ones adaptive loss\.Figure 4:A short prediction window is enough\.Validation perplexity oflptrelative tolt\(↓\\downarrow\) vs\. window size,ltat its better objective per depth\. The dotted line marksω=64\\omega=64\.
### Prediction Recursion Needs Local Memory

We trainlptwith prediction windows ranging fromω=0\\omega=0to the full context of4,0964\{,\}096tokens \([Figure 4](https://arxiv.org/html/2610.02383#S6.F4)\)\.lptoutperformsltat every window size, across both sizes and all tested depths\. This includesω=0\\omega=0, where prediction recursions keep no local history and read shared memory alongside only their current KV entry\. At the other extreme,ω=4,096\\omega=4\{,\}096,ltandlptdiffer only in access to shared memory\.lptstill lowers perplexity by0\.240\.24–1\.151\.15relative toltwith the better of its two training objectives at each depth, close to the0\.320\.32–1\.091\.09gain atω=64\\omega=64\. Overall, window size has a modest effect on quality, and what really matters is the access to the shared memory\. To balance quality and efficiency, we adoptω=64\\omega=64as the default for our memory sharing models\.

Takeaway 3\.Shared memory improves quality across prediction\-window sizes, even when later recursions keep no local history\.

### Later Recursions Attend More to Shared Memory

We next measure how much attention is assigned to the shared memory in later recursions\. We evaluate each model on the same4,0964\{,\}096\-token validation sequence and average the attention assigned to shared keys over queries and heads \([Figure 6](https://arxiv.org/html/2610.02383#S6.F6)and[Appendix D](https://arxiv.org/html/2610.02383#A4)\)\. Shared memory receives most attention at every prediction recursion in both sizes\. Its share is5757–71%71\\%in earlier prediction recursions and rises to7373–80%80\\%in the final recursion\.222Additional comparisons in the appendix show that shared memory still receives most attention with full prediction caches, while adaptive loss makes its use more uniform across recursions \([Figure 11](https://arxiv.org/html/2610.02383#A4.F11)\)\.

Takeaway 4\.Later recursions attend more to shared memory than to their own caches, with a stronger preference when getting closer to the prediction\.

![Refer to caption](https://arxiv.org/html/2610.02383v1/lpt_state_mass_final.png)

Figure 5:Prediction recursions attend mostly to shared memory\.Attention mass assigned to shared KV entries inlpt, averaged over queries and heads\. For clarity, only six layers are displayed\. The bottom row averages all layers\.![Refer to caption](https://arxiv.org/html/2610.02383v1/qkv_pca_main.png)

Figure 6:Shared and local keys separate inlpt\.Joint query/key PCA, fitted separately for each panel\. Dots are tokens and large circles their means; darker shades mark later recursions\.
### Shared and Local Memory Develop Different Representations

Shared keys serve every later recursion, whereas local keys serve only the recursion that writes them\. To see whether these roles are reflected in their representations, we compare S\-sizeltandlptwith three recursions, both trained with final loss, the only tested setting with at least three recursions wherelttrains with final loss without a substantial drop in quality \([Figure 4](https://arxiv.org/html/2610.02383#S6.F4)\)\. We evaluate both models on the same4,0964\{,\}096\-token validation sequence and extract queries and keys before rotary encoding\. We concatenate the attention heads and fit principal component analysis \(PCA\) jointly to queries and keys across tokens and recursions per model and layer \([Appendix D](https://arxiv.org/html/2610.02383#A4)\)\.

Shared and local keys consistently form distinct clusters inlpt, whereas keys from different recursions overlap more inlt\([Figure 6](https://arxiv.org/html/2610.02383#S6.F6)\)\. In[Appendix D](https://arxiv.org/html/2610.02383#A4), we show similar separation inlptacross S and M at three and five recursions \([Figure 10](https://arxiv.org/html/2610.02383#A4.F10)\)\. We also show that with full prediction caches, the clusters overlap more but still remain distinguishable, even though shared and local keys cover the same positions \([Figure 9](https://arxiv.org/html/2610.02383#A4.F9)\)\. Prior work attributes the small cost of cache reuse to the similarity of keys and values across recursions\([Bae et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib18)\), as we see inlt\. With Memory Sharing, shared and local keys are instead clearly different\. This shows that representation similarity across layers is not a necessary condition for effective memory sharing\. This also fits the freedom that final loss gives: recursions need not all produce prediction\-ready states, so the first can specialize in building memory and the later ones in using it\. This specialization may help explain the quality gains, similar to how[Monea et al\. \(2026\)](https://arxiv.org/html/2610.02383#bib.bib1)show that Transformers benefit from assigning memory construction and next\-token prediction to separate residual streams\.

Takeaway 5\.Shared and local memory diverge in their local representations, so sharing is not reliant on recursions writing similar memories\.

### Shared Memory Is a Gradient Highway

Reading shared memory gives the final loss a direct gradient path to the first recursion, bypassing the intermediate recursions\. We evaluate thestd,lt, andlptmodels of[Table 1](https://arxiv.org/html/2610.02383#S3.T1)on the same3232validation sequences of4,0964\{,\}096tokens and backpropagate the final loss from3232fixed positions in each\. We measure each recursion’s contribution to the gradient on the Transformer block weights, and split the first recursion’s contribution into its hidden\-state and shared\-KV paths by repeating the backward pass with shared KV detached \([Appendix D](https://arxiv.org/html/2610.02383#A4)\)\.

Inlt, the first\-recursion gradient norm falls sharply with depth\.lptrecovers much of the signal magnitude relative to the single\-passstdreference, with a much smaller decline as depth increases \([Figure 8](https://arxiv.org/html/2610.02383#S6.F8)\)\. At five recursions, most of this gradient comes through the shared KV cache\. The same holds forlptwith adaptive loss or full prediction caches \([Figure 12](https://arxiv.org/html/2610.02383#A4.F12)\)\.

To test whether this highway helps learning, we trainlptfrom scratch with shared KV detached at later reads\. The forward computation and hidden\-state gradient path are unchanged\. Perplexity rises in both sizes and at every depth, and deeper detached models often fail to train \([Figure 8](https://arxiv.org/html/2610.02383#S6.F8)\)\.

![Refer to caption](https://arxiv.org/html/2610.02383v1/shared_feedback_later_recursions_compact.png)

Figure 7:Shared memory restores much of the gradient reaching the first recursion\.Mean gradient norm contributed by each recursion, with single\-passstdas reference\. The firstlptbar is split into shared\-KV \(hatched\) and hidden\-state projections\.ltis trained with adaptive loss, the others with final loss\. Whiskers are95%95\\%sequence\-bootstrap intervals\.![Refer to caption](https://arxiv.org/html/2610.02383v1/detach_ablation_compact.png)

Figure 8:Detaching shared\-cache gradients hurts learning\.Validation perplexity \(↓\\downarrow\), collapsed runs in the upper strip on a log scale\. All models are trained with final loss;[Table 6](https://arxiv.org/html/2610.02383#A4.T6)lists the values\.Takeaway 6\.The memory sharing mechanism creates a gradient highway from the final loss to the first recursion that facilitates learning\.

## 7Related Work

### Separating Memory from Prediction

In a standard Transformer, each hidden state does two jobs\. It predicts the next token, and it writes the keys and values that later positions read\. Several architectures separate these jobs across token streams, as in the two\-stream attention of XLNet\([Yang et al\., 2019](https://arxiv.org/html/2610.02383#bib.bib3)\)\. The State\-Prediction Separation Transformer\([Monea et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib1)\)gives memory construction and next\-token prediction separate residual streams and improves language modeling, and[Langford et al\. \(2026\)](https://arxiv.org/html/2610.02383#bib.bib2)study a variant of this split\. A second family separates them across layers\. The first layers write one cache once, and the remaining layers only read it\([Sun et al\., 2024](https://arxiv.org/html/2610.02383#bib.bib5);[Sun et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib6);[DeepSeek\-AI, 2026](https://arxiv.org/html/2610.02383#bib.bib7);[Capps, 2026](https://arxiv.org/html/2610.02383#bib.bib8)\)\. Instead,lptdecouples memory from prediction across the recursions of a looped Transformer: the first recursion writes the model’s only full\-context memory, and every later recursion reads it while it predicts, keeping just a short window of its own\. Looped models thus offer a third place for the split, with no extra stream or layer, and our analysis finds evidence that the recursions learn these roles \([Section 6](https://arxiv.org/html/2610.02383#S6)\)\.

### Looped Transformers and Memory Sharing

Looped Transformers apply the same layers several times\([Dehghani et al\., 2019](https://arxiv.org/html/2610.02383#bib.bib13);[Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12);[Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\), and recent work differs in which layers repeat and how recursions connect\([Gao et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib25);[Wang et al\., 2026a](https://arxiv.org/html/2610.02383#bib.bib15);[Huang et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib14);[Mozer et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib16);[Wang et al\., 2026b](https://arxiv.org/html/2610.02383#bib.bib17);[Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\)\. In standard formulations, each recursion writes its own KV cache, so context memory grows with depth\([Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12);[Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. Sharing a cache across recursions, like sharing it across layers\([Brandon et al\., 2024](https://arxiv.org/html/2610.02383#bib.bib4)\), has been studied as a way to save memory\. Without retraining, Huginn\([Geiping et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib12)\)reuses caches cyclically with little quality loss, and Ouro\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)can keep only the final or an averaged cache\. Keeping only the first recursion’s cache sharply reduces Ouro’s quality, which suggests that shared memory has to be trained\. Trained variants reuse the first recursion’s keys and values at a slight cost in quality\([Bae et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib18)\), compute them once in a separate prelude\([Capps, 2026](https://arxiv.org/html/2610.02383#bib.bib8)\), or learn gated or compressed caches\([Vendrell et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib19);[O’Neill and Reid, 2026](https://arxiv.org/html/2610.02383#bib.bib20)\)\.plt\([Wu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib10)\)has the closest design to ours, although it focuses on parallelizing recursions\. It shares the first recursion’s cache with gated local caches and runs recursions in parallel, whilelptkeeps them sequential and reads both caches in one softmax\. We find thatlptperforms better\. All of these treat sharing as a cost to contain\. We instead treat it as an inductive bias that separates memory from prediction and show that it improves quality \([Section 5](https://arxiv.org/html/2610.02383#S5)and[Section 6](https://arxiv.org/html/2610.02383#S6)\)\.

## 8Discussion

We investigate memory sharing as an architectural inductive bias for looped models, and find that it not only cuts context memory but also improves quality in both full attention and hybrid models\. Our Memory Sharinglptmodels achieve consistently better perplexity and downstream accuracy than their usual looped counterparts, while matching the KV cache memory of a standard non\-looped model\. Our analyses suggest that sharing helps recursions learn different roles\. The first recursion builds memory that later recursions use to refine the prediction\. Those reads also give the final loss a direct gradient path to the first recursion\. The different representations of shared and local memory support this division of roles, and cutting this gradient path hurts learning\. Shared memory changes how later recursions read information and how the first recursion learns to write it\.

Overall, our findings suggest that shared memory is a useful inductive bias for looped models, helping them make better use of computation within a limited memory budget\.

### Limitations & Future work

Like other looped Transformers,lptis not compute\-optimal: it trades time for quality, but unlikelt, its memory barely grows with depth\. At five recursions it costs4\.34\.3–4\.8×4\.8\\timesthe decoding FLOPs ofstd, about the same aslt, and it spends the same number of recursions on every token\. Choosing that number per token is future work\. All our comparisons are parameter\-matched, includingdeepstd, and we do not train a standard Transformer matched in decoding FLOPs instead, which at five recursions would have several times the parameters\. We train up to11B parameters on one corpus with4,0964\{,\}096\-token contexts and do not test long\-context tasks or, given our limited pretraining, very advanced tasks\. Our analyses use S and M models, since not every configuration they need was trained at L, and only the detach ablation looks at training dynamics\. Both the stability and memory improvements oflptare promising for effectively scaling looped models to higher recursion counts\. More sophisticated Memory Sharing approaches could also be explored in future work, such as increasing the number of state recursions or alternating state recursions and prediction recursions\.

## Acknowledgments

We thank Aviv Ron, Brian Kingsbury, and David Cox for useful feedback on internal presentations of this work\.

## References

- Baeet al\.\(2025\)S\. Bae, Y\. Kim, R\. Bayat, S\. Kim, J\. Ha, T\. Schuster, A\. Fisch, H\. Harutyunyan, Z\. Ji, A\. Courville, and S\. YunMixture\-of\-recursions: learning dynamic recursive depths for adaptive token\-level computation\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/8b08bbf8b420faa6eeb4020720582ec7-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.02383#S6.SS0.SSS0.Px4.p2.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6239)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px4.p1.1)\.
- Brandonet al\.\(2024\)W\. Brandon, M\. Mishra, A\. Nrusimha, R\. Panda, and J\. Ragan\-KelleyReducing transformer key\-value cache size with cross\-layer attention\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/9e23d020c18e4c40d81c6a0fc7a46f68-Abstract-Conference.html)Cited by:[Appendix F](https://arxiv.org/html/2610.02383#A6.SS0.SSS0.Px3.p1.1),[Appendix F](https://arxiv.org/html/2610.02383#A6.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Capps \(2026\)C\. A\. CappsCART: context\-anchored recurrent transformer – a parameter\-efficient architecture with learned stability\.External Links:2606\.01495,[Link](https://arxiv.org/abs/2606.01495)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Chen and Wei \(2026\)C\. Chen and L\. WeiPost\-LayerNorm is back: stable, ExpressivE, and deep\.External Links:2601\.19895,[Link](https://arxiv.org/abs/2601.19895)Cited by:[5th item](https://arxiv.org/html/2610.02383#A3.I1.i5.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? Try ARC, the AI2 Reasoning Challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px4.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4\.1\-Flash: pushing the limits of KV cache compression\.External Links:2609\.19969,[Link](https://arxiv.org/abs/2609.19969)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1)\.
- Dehghaniet al\.\(2019\)M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and Ł\. KaiserUniversal Transformers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HyzdRiR9Y7)Cited by:[§1](https://arxiv.org/html/2610.02383#S1.p1.1),[§2](https://arxiv.org/html/2610.02383#S2.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Denget al\.\(2026\)C\. Deng, Y\. Zhang, R\. Zhu, Y\. Xu, J\. Liu, T\. S\. E\. Ng, and H\. ChenLT2: linear\-time looped transformers\.External Links:2605\.20670,[Link](https://arxiv.org/abs/2605.20670)Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px1.p2.1),[Appendix E](https://arxiv.org/html/2610.02383#A5.p2.1),[§1](https://arxiv.org/html/2610.02383#S1.p3.1),[§2](https://arxiv.org/html/2610.02383#S2.p4.1),[§3](https://arxiv.org/html/2610.02383#S3.p5.1),[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Dinget al\.\(2021\)M\. Ding, Z\. Yang, W\. Hong, W\. Zheng, C\. Zhou, D\. Yin, J\. Lin, X\. Zou, Z\. Shao, H\. Yang, and J\. TangCogView: mastering text\-to\-image generation via transformers\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/hash/a4d92e2cd541fca87e4620aba658316d-Abstract.html)Cited by:[2nd item](https://arxiv.org/html/2610.02383#A3.I1.i2.p1.1)\.
- Donget al\.\(2025\)J\. Dong, B\. Feng, D\. Guessous, Y\. Liang, and H\. HeFlexAttention: a programming model for generating fused attention variants\.InProceedings of Machine Learning and Systems,External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2025/hash/61a9278dfef5f871b5e472389f8d6fa1-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px2.p1.1),[Appendix F](https://arxiv.org/html/2610.02383#A6.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2610.02383#S3.SS0.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouA framework for few\-shot language model evaluation\.Zenodo\.Note:Zenodo, version v0\.4\.0External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px4.p1.1)\.
- Gaoet al\.\(2026\)Z\. Gao, Y\. Chen, Y\. Xiao, X\. Yang, R\. Tao, J\. Zhou, and B\. DaiLoop the Loopies\!\.External Links:2607\.16051,[Link](https://arxiv.org/abs/2607.16051)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Geipinget al\.\(2025\)J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. GoldsteinScaling up test\-time compute with latent reasoning: a recurrent depth approach\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.02383#S1.p1.1),[§1](https://arxiv.org/html/2610.02383#S1.p2.1),[§2](https://arxiv.org/html/2610.02383#S2.p1.1),[§2](https://arxiv.org/html/2610.02383#S2.p3.2),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, O\. Vinyals, J\. W\. Rae, and L\. SifreTraining compute\-optimal large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30016–30030\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px3.p1.1)\.
- Huanget al\.\(2026\)B\. Huang, C\. Shi, J\. Chen, S\. Wen, Z\. Liu, E\. Xing, and X\. MaTowards looped models done right—Part I: topology, input injection, recurrent\-state design\.Note:Institute of Foundation Models technical blogAccessed September 16, 2026External Links:[Link](https://ifm-research.notion.site/3ade511912ec8128987dfeb7a5580043)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Karpathy \(2022\)A\. KarpathynanoGPT\.Note:GitHub repository, commit 325be85External Links:[Link](https://github.com/karpathy/nanoGPT)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px2.p1.1)\.
- Katharopouloset al\.\(2020\)A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. FleuretTransformers are RNNs: fast autoregressive transformers with linear attention\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 5156–5165\.External Links:[Link](https://proceedings.mlr.press/v119/katharopoulos20a.html)Cited by:[§2](https://arxiv.org/html/2610.02383#S2.p4.1)\.
- Langfordet al\.\(2026\)J\. Langford, N\. Godey, G\. Monea, Y\. Artzi, H\. Dong, Y\. Fan, G\. de Rosa, and Z\. ZhanAlmost free state prediction separation\.External Links:2609\.03807,[Link](https://arxiv.org/abs/2609.03807)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1)\.
- Moneaet al\.\(2026\)G\. Monea, N\. Godey, K\. Brantley, and Y\. ArtziThe state\-prediction separation hypothesis\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2607.01218)Cited by:[Appendix E](https://arxiv.org/html/2610.02383#A5.p1.1),[§6](https://arxiv.org/html/2610.02383#S6.SS0.SSS0.Px4.p2.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1)\.
- Mozeret al\.\(2026\)M\. C\. Mozer, S\. A\. Siddiqui, D\. Sawyer, S\. Sanyal, and R\. LiuRecirculation\.External Links:2608\.17981,[Link](https://arxiv.org/abs/2608.17981)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- O’Neill and Reid \(2026\)J\. O’Neill and F\. ReidLooped latent attention: cross\-loop KV compression for looped transformers\.External Links:2607\.15456,[Link](https://arxiv.org/abs/2607.15456)Cited by:[Appendix F](https://arxiv.org/html/2610.02383#A6.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Papernoet al\.\(2016\)D\. Paperno, G\. Kruszewski, A\. Lazaridou, Q\. N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. FernándezThe LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1525–1534\.External Links:[Link](https://aclanthology.org/P16-1144/)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px4.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfThe FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.37,pp\. 30811–30849\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px3.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Technical reportOpenAI\.Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px3.p1.1)\.
- Schlaget al\.\(2021\)I\. Schlag, K\. Irie, and J\. SchmidhuberLinear transformers are secretly fast weight programmers\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 9355–9366\.External Links:[Link](https://proceedings.mlr.press/v139/schlag21a.html)Cited by:[§2](https://arxiv.org/html/2610.02383#S2.p4.1)\.
- Shazeer \(2020\)N\. ShazeerGLU variants improve Transformer\.External Links:2002\.05202,[Link](https://arxiv.org/abs/2002.05202)Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px1.p1.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.External Links:[Document](https://dx.doi.org/10.1016/j.neucom.2023.127063),[Link](https://doi.org/10.1016/j.neucom.2023.127063)Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px1.p1.1)\.
- Sunet al\.\(2026\)Y\. Sun, L\. Dong, T\. Ye, S\. Huang, J\. Wang, and F\. WeiUniversal YOCO for efficient depth scaling\.External Links:2604\.01220,[Link](https://arxiv.org/abs/2604.01220)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1)\.
- Sunet al\.\(2024\)Y\. Sun, L\. Dong, Y\. Zhu, S\. Huang, W\. Wang, S\. Ma, Q\. Zhang, J\. Wang, and F\. WeiYou only cache once: decoder\-decoder architectures for language models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/0df38cd13520747e1e64e5b123a78ef8-Abstract-Conference.html)Cited by:[Appendix F](https://arxiv.org/html/2610.02383#A6.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1)\.
- Vendrellet al\.\(2026\)V\. C\. Vendrell, A\. P\. Masdemont, N\. Grillo, J\. Ros\-Giralt, A\. Behboodi, and F\. V\. MassoliMemory\-efficient looped transformer: decoupling compute from memory in looped language models\.External Links:2605\.07721,[Link](https://arxiv.org/abs/2605.07721)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026a\)S\. Wang, G\. Zhang, K\. Luo, Y\. Wu, S\. Liu, J\. Liu, W\. Huang, S\. Yan, and J\. LiSMELT: scaling laws for compute\-matched MoE looped transformers\.External Links:2609\.01343,[Link](https://arxiv.org/abs/2609.01343)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026b\)X\. Wang, Z\. Cai, Z\. Zhan, H\. Dong, Y\. Fan, G\. de Rosa, T\. Pearce, and J\. LangfordFull\-bandwidth transformer\.External Links:2608\.08888,[Link](https://arxiv.org/abs/2608.08888)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Welblet al\.\(2017\)J\. Welbl, N\. F\. Liu, and M\. GardnerCrowdsourcing multiple choice science questions\.InProceedings of the 3rd Workshop on Noisy User\-generated Text,pp\. 94–106\.External Links:[Link](https://aclanthology.org/W17-4413/)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px4.p1.1)\.
- Wuet al\.\(2025\)B\. Wu, M\. Chen, X\. Luo, S\. Yan, Q\. Yu, F\. Xia, T\. Zhang, H\. Zhan, Z\. Zhong, X\. Zhou, S\. Qiao, and X\. BinParallel loop transformer for efficient test\-time computation scaling\.External Links:2510\.24824,[Link](https://arxiv.org/abs/2510.24824)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1)\.
- Xionget al\.\(2020\)R\. Xiong, Y\. Yang, D\. He, K\. Zheng, S\. Zheng, C\. Xing, H\. Zhang, Y\. Lan, L\. Wang, and T\. LiuOn layer normalization in the transformer architecture\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 10524–10533\.External Links:[Link](https://proceedings.mlr.press/v119/xiong20b.html)Cited by:[1st item](https://arxiv.org/html/2610.02383#A3.I1.i1.p1.1)\.
- Yanget al\.\(2025\)S\. Yang, J\. Kautz, and A\. HatamizadehGated delta networks: improving Mamba2 with delta rule\.InInternational Conference on Learning Representations,pp\. 29687–29707\.External Links:[Link](https://openreview.net/forum?id=r8H7xhYPwz)Cited by:[§2](https://arxiv.org/html/2610.02383#S2.p4.1),[§3](https://arxiv.org/html/2610.02383#S3.p5.1)\.
- Yang and Zhang \(2024\)S\. Yang and Y\. ZhangFLA: a Triton\-based library for hardware\-efficient implementations of linear attention mechanism\.Note:GitHub repositoryExternal Links:[Link](https://github.com/fla-org/flash-linear-attention)Cited by:[Appendix F](https://arxiv.org/html/2610.02383#A6.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2019\)Z\. Yang, Z\. Dai, Y\. Yang, J\. Carbonell, R\. Salakhutdinov, and Q\. V\. LeXLNet: generalized autoregressive pretraining for language understanding\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/dc6a7e655d7e5840e66733e9ee67cc69-Abstract.html)Cited by:[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px1.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/)Cited by:[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px4.p1.1)\.
- Zhang and Sennrich \(2019\)B\. Zhang and R\. SennrichRoot mean square layer normalization\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/1e8a19426224ca89e83cef47f1e7f53b-Abstract.html)Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025\)R\. Zhu, Z\. Wang, K\. Hua, T\. Zhang, Z\. Li, H\. Que, B\. Wei, Z\. Wen, F\. Yin, H\. Xing, L\. Li, J\. Shi, K\. Ma, S\. Li, T\. Kergan, A\. Smith, X\. Qu, M\. Hui, B\. Wu, Q\. Min, H\. Huang, X\. Zhou, W\. Ye, J\. Liu, J\. Yang, Y\. Shi, C\. Lin, E\. Zhao, T\. Cai, G\. Zhang, W\. Huang, Y\. Bengio, and J\. EshraghianScaling latent reasoning via looped language models\.External Links:2510\.25741,[Link](https://arxiv.org/abs/2510.25741)Cited by:[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px1.p2.1),[Appendix A](https://arxiv.org/html/2610.02383#A1.SS0.SSS0.Px4.p2.1),[Appendix C](https://arxiv.org/html/2610.02383#A3.p1.1),[Appendix E](https://arxiv.org/html/2610.02383#A5.p2.1),[§1](https://arxiv.org/html/2610.02383#S1.p1.1),[§1](https://arxiv.org/html/2610.02383#S1.p2.1),[§2](https://arxiv.org/html/2610.02383#S2.p1.1),[§2](https://arxiv.org/html/2610.02383#S2.p3.2),[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.02383#S4.SS0.SSS0.Px3.p2.1),[§7](https://arxiv.org/html/2610.02383#S7.SS0.SSS0.Px2.p1.1),[footnote 1](https://arxiv.org/html/2610.02383#footnote1)\.

## Appendix AArchitecture and Training Details

### Architecture

[Table 2](https://arxiv.org/html/2610.02383#A1.T2)lists the base model dimensions\. The decoder\-only backbone uses RMSNorm pre\-normalization\([Zhang and Sennrich, 2019](https://arxiv.org/html/2610.02383#bib.bib22)\), SwiGLU feed\-forward networks\([Shazeer, 2020](https://arxiv.org/html/2610.02383#bib.bib23)\), no biases in linear layers, and tied input and output embeddings\. Full\-attention layers use rotary positional embeddings\([Su et al\., 2024](https://arxiv.org/html/2610.02383#bib.bib26)\)\.

Table 2:Base model configurations\.Looped models reuse the listedLL\-layer stack\.ltandlptuse Ouro’s normalization and recurrent transition\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\.lt2andh\-lptuse the Full\+GDN layer pattern and gating setup of[Deng et al\. \(2026\)](https://arxiv.org/html/2610.02383#bib.bib24), with one full\-attention layer per five layers\. The full\-attention layers read the complete causal context inlt2\. Inh\-lpt, later recursions read the first recursion’s full\-context cache and their own local prediction window, as in[Equation 3](https://arxiv.org/html/2610.02383#S3.E3)\.

pltuses the same block, recurrent, and readout normalization asltandlpt\. Its recurrent transition adds the input embedding to the previous recursion’s normalized output shifted by one token\.

### Attention Implementation

During training, each prediction recursion concatenates its keys and values with the shared keys and values along the sequence dimension, separately at each full\-attention layer\. A block mask implements[Equation 3](https://arxiv.org/html/2610.02383#S3.E3), allowing full causal attention over shared entries and causal sliding\-window attention over local entries\. FlexAttention\([Dong et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib9)\)skips fully masked blocks\.

### Data Processing

We tokenize with the GPT\-2 tokenizer\([Radford et al\., 2019](https://arxiv.org/html/2610.02383#bib.bib27)\)and pack documents into4,0964\{,\}096\-token sequences, inserting an end\-of\-sequence token between documents\. Attention is masked across document boundaries, and boundary tokens are excluded from the language\-modeling loss\.

### Optimization

We use AdamW withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay0\.10\.1, and gradient clipping at1\.01\.0\. The global batch contains9696sequences of4,0964\{,\}096tokens, and training uses bfloat16 mixed precision\. For L\-size models, AdamW optimizer state is sharded across DDP ranks \(ZeRO\-1\)\. The learning rate increases linearly to6×10−46\\times 10^\{\-4\}over the first5050M tokens, stays constant, then decreases linearly to6×10−56\\times 10^\{\-5\}over the final10%10\\%of training\. These are nanoGPT’s default optimizer settings, used unchanged for every model, size, and objective: we did not tune the learning rate or the warmup for any model, and we did not test whether a lower peak learning rate or a longer warmup changes how final\-losslttrains at depth \([Figure 4](https://arxiv.org/html/2610.02383#S6.F4)\)\.

For adaptive loss, we use Ouro’s adaptive\-exit objective\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. A learned per\-token halting distribution weights the losses across recursions, with a KL regularization weight of0\.10\.1\. Final loss applies the language\-modeling loss only at the last recursion\.

## Appendix BSeed Variance and Statistical Tests

Retraining every configuration at every size with several seeds is prohibitive at pretraining cost, so we run the check at S\. Each S\-size model of[Table 1](https://arxiv.org/html/2610.02383#S3.T1)is trained with four seeds: the run reported in the table and three more, each changing the initialization and the data order together\. All other settings stay at their main\-table values\.[Table 3](https://arxiv.org/html/2610.02383#A2.T3)lists the mean and95%95\\%confidence interval \(Student\-tt\) of the final validation perplexity for every model and recursion count\.[Table 4](https://arxiv.org/html/2610.02383#A2.T4)tests each gap of interest with a one\-sided pairedtt\-test on the same\-seed differences, with the alternative that the first model has lower perplexity\. The largest standard deviation of perplexity across seeds is0\.080\.08, against gaps of0\.400\.40–1\.051\.05betweenlptandlt, and every difference keeps its sign at every seed\.

Table 3:Seed variance at S\.Final validation perplexity, mean±\\pm95%95\\%confidence interval \(Student\-tt\) over four seeds\. In thestdrow,R=1R=1is the standard Transformer andR\>1R\>1the deeper, narrower parameter\-matched Transformer\.spsis the two\-recursion baseline of[Appendix E](https://arxiv.org/html/2610.02383#A5)\.Table 4:Paired tests at S\.Same\-seed perplexity gap of the first model over the second, mean±\\pm95%95\\%confidence interval over four seeds, the gap in units of the standard deviation of the same\-seed differences, and the one\-sided pairedtt\-test for the first model having lower perplexity\.
## Appendix CNormalization between Recursions

Ourltandlptuse the Ouro paper’s normalization\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\. Sandwich norms normalize the output of every sublayer before the residual addition, and the recurrent norm𝒩rec\\mathcal\{N\}\_\{\\mathrm\{rec\}\}normalizes each recursion’s output before the next recursion \([Equation 1](https://arxiv.org/html/2610.02383#S2.E1)\)\. With final loss, thisltdegrades or diverges at depth \([Figure 4](https://arxiv.org/html/2610.02383#S6.F4)\), so the main tables train it with adaptive loss\. We did not tune the normalization of either model\. To test whether Ouro’s normalization holdsltback, we also train S and Mltwith final loss and five other normalizations, each shared across recursions:

- •*Pre\-norm only*\([Xiong et al\., 2020](https://arxiv.org/html/2610.02383#bib.bib41)\)removes both, sohin\(t\+1\)=h\(t\)h\_\{\\mathrm\{in\}\}^\{\(t\+1\)\}=h^\{\(t\)\}and each block keeps only its pre\-norms\.
- •*Sandwich only*keeps the sandwich norms\([Ding et al\., 2021](https://arxiv.org/html/2610.02383#bib.bib21)\)and removes𝒩rec\\mathcal\{N\}\_\{\\mathrm\{rec\}\}\.
- •*Input norm*removes the sandwich norms and keeps𝒩rec\\mathcal\{N\}\_\{\\mathrm\{rec\}\}, which it also applies to the embeddings\.
- •*Update norm*removes the sandwich norms and normalizes only each recursion’s update, so the carried state stays unnormalized:hin\(t\+1\)=hin\(t\)\+𝒩⁡\(h\(t\)−hin\(t\)\)h\_\{\\mathrm\{in\}\}^\{\(t\+1\)\}=h\_\{\\mathrm\{in\}\}^\{\(t\)\}\+\\mathcal\{N\}\\big\(h^\{\(t\)\}\-h\_\{\\mathrm\{in\}\}^\{\(t\)\}\\big\)\.
- •*KEEL*removes the sandwich norms and applies the highway residual of[Chen and Wei \(2026\)](https://arxiv.org/html/2610.02383#bib.bib40)across recursions:hin\(t\+1\)=𝒩2​\(R​hin\(t\)\+ℳL​\(𝒩1​\(hin\(t\)\)\)\)h\_\{\\mathrm\{in\}\}^\{\(t\+1\)\}=\\mathcal\{N\}\_\{2\}\\big\(R\\,h\_\{\\mathrm\{in\}\}^\{\(t\)\}\+\\mathcal\{M\}^\{L\}\(\\mathcal\{N\}\_\{1\}\(h\_\{\\mathrm\{in\}\}^\{\(t\)\}\)\)\\big\)\.

Here𝒩\\mathcal\{N\},𝒩1\\mathcal\{N\}\_\{1\}, and𝒩2\\mathcal\{N\}\_\{2\}are RMSNorms\. Data, seed, batch, and learning\-rate schedule match the main runs\.lptkeeps Ouro’s normalization and final loss\.

Table 5:lptoutperformsltwith the best normalization we tried\.Validation perplexity \(↓\\downarrow\) after2020B tokens\.Boldmarks the best per column andunderlinethe bestlt\. Values above2,0002\{,\}000are unigram\-level\. A dash marks a run that did not finish\.At every size and depth in[Table 5](https://arxiv.org/html/2610.02383#A3.T5),lptwith Ouro’s normalization reaches lower perplexity than the bestlt, and the bestltreaches lower perplexity than our mainlt\. The alternatives are also more robust: every finished run trains with final loss at every size and depth, while Ouro’s normalization degrades or diverges\. No single normalization is best forlt: update norm wins three settings and KEEL the other three\. Choosing the best one lowerslt’s perplexity by0\.130\.13–0\.760\.76relative to our mainlt\.lptstill leads the bestltby0\.310\.31–0\.370\.37at S and by0\.060\.06–0\.430\.43at M\. The smallest gap,0\.060\.06at M with two recursions, is within the seed variation we measure at S \([Appendix B](https://arxiv.org/html/2610.02383#A2)\)\. At S, every alternative keeps improving through five recursions, unlike our mainlt\. At M, the bestltstops improving after three recursions\.

Although this comparison favorslt,lptstill performs best\.ltgets the best of six normalizations, chosen on the same validation perplexity we report, whilelptkeeps Ouro’s\. The same search forlptwould include Ouro’s normalization, so it could only keep or lowerlpt’s perplexity\. Since normalization is not the focus of our investigation, we keep Ouro’s for the mainlt, as in the original paper\.

## Appendix DAdditional Analysis Details

### Attention Representations

For[Figure 6](https://arxiv.org/html/2610.02383#S6.F6), we use S\-size models with three recursions and final loss\. We evaluate each model on the same4,0964\{,\}096\-token validation sequence\. At each layer, we concatenate the attention heads into one query vector and one key vector per token, before rotary encoding\. We pool these vectors across all tokens and recursions, subtract their common mean, and fit PCA by singular value decomposition, separately for each model and layer\. We do not normalize the vectors or rescale their features\. All recursions within a panel therefore share one basis\. Axes are not aligned across models or layers\.

We display the first two principal components\. For readability, the panels show one token in2424, while PCA and the large centroid markers use all tokens\. Lines connect centroids across recursions, separately for queries and keys\. Axis limits are shared across layers within each model, with equal scaling on both axes\.

[Figure 9](https://arxiv.org/html/2610.02383#A4.F9)addslptwith full prediction caches\.[Figure 10](https://arxiv.org/html/2610.02383#A4.F10)compareslptacross sizes and recursion counts, matching layers by relative depth\.[Figure 11](https://arxiv.org/html/2610.02383#A4.F11)compares attention to shared memory across window sizes and training objectives\. Each layer’s attention mass is averaged over queries and heads\. We display six layers at matched relative depths, but compute the mean over all1212layers for S and all2424for M\.

![Refer to caption](https://arxiv.org/html/2610.02383v1/qkv_pca_full_window.png)Figure 9:Shared and local keys remain distinguishable with full prediction caches\.Expanded comparison for[Figure 6](https://arxiv.org/html/2610.02383#S6.F6), includingω=4,096\\omega=4\{,\}096\.![Refer to caption](https://arxiv.org/html/2610.02383v1/lpt_pca_depth.png)Figure 10:Query/key geometry acrosslptmodels\.S and M models at three and five recursions, with final loss andω=64\\omega=64\. Columns match relative layer depth\.![Refer to caption](https://arxiv.org/html/2610.02383v1/lpt_state_mass_blocks_main.png)Figure 11:Attention to shared memory across training objectives and window sizes\.Expanded comparison for[Figure 6](https://arxiv.org/html/2610.02383#S6.F6)\. Mean rows average all layers\.
### Gradient Feedback through Shared Memory

We differentiate the mean cross\-entropy of the final recursion at3232sampled positions in each of3232validation sequences of length4,0964\{,\}096\. The sequence contents and loss positions are identical across conditions\.[Figure 8](https://arxiv.org/html/2610.02383#S6.F8)uses the main\-comparison checkpoints:lttrained with adaptive loss andlptwith final loss\.stdprovides a single\-pass reference, measured through its native forward pass on the same data\.[Figure 12](https://arxiv.org/html/2610.02383#A4.F12)addslptwith full prediction caches andlpttrained with adaptive loss\. Every probe differentiates the loss at the final recursion, regardless of the training objective\. We measure gradients on Transformer block parameters, excluding embeddings, readout, recurrent normalization, and the halting gate\. Each model retains its trained normalization\.

For the backward computation, we give each recursion an independent parameter copy with the same values\. This leaves the forward pass unchanged and isolates each recursion’s contribution to the shared\-parameter gradient\. Detaching shared KV at later reads isolates the first recursion’s hidden\-state path contribution,ghiddeng\_\{\\mathrm\{hidden\}\}\. Subtracting it from the ordinary first\-recursion gradientg\(1\)g^\{\(1\)\}gives the shared\-cache contribution,gKVg\_\{\\mathrm\{KV\}\}\. We verify that contributions from all parameter uses sum to the ordinary shared\-parameter gradient\.

The bars show arithmetic mean gradient norms across sequences\. To split the first bar, we project each path onto its total gradient within each sequence:

pa=⟨ga,g\(1\)⟩∥g\(1\)∥2,a∈\{KV,hidden\},pKV\+phidden=∥g\(1\)∥2\.p\_\{a\}=\\frac\{\\langle g\_\{a\},g^\{\(1\)\}\\rangle\}\{\\lVert g^\{\(1\)\}\\rVert\_\{2\}\},\\qquad a\\in\\\{\\mathrm\{KV\},\\mathrm\{hidden\}\\\},\\qquad p\_\{\\mathrm\{KV\}\}\+p\_\{\\mathrm\{hidden\}\}=\\lVert g^\{\(1\)\}\\rVert\_\{2\}\.\(5\)We then average these projected contributions\. Their sum equals the mean first\-recursion norm\. The two path norms themselves are not additive\. The projections are nonnegative in all plotted measurements, without clipping\. Whiskers are95%95\\%sequence\-bootstrap intervals for the full bar, not for its segments or for variation across training seeds\.

[Figure 12](https://arxiv.org/html/2610.02383#A4.F12)shows that the shared\-cache contribution remains strong with a full prediction window and under adaptive loss\. This is consistent across S and M\. The compact main\-paper comparison therefore does not depend on using final loss or a short prediction window forlpt\.

![Refer to caption](https://arxiv.org/html/2610.02383v1/shared_feedback_later_recursions.png)Figure 12:Shared\-cache feedback persists across training objectives and window sizes\.Expanded version of[Figure 8](https://arxiv.org/html/2610.02383#S6.F8), with the same sequences and final\-loss gradient probe in every condition\.
### Gradient Paths at Initialization

[Figure 13](https://arxiv.org/html/2610.02383#A4.F13)compares the norm ratio∥gKV∥2/∥ghidden∥2\\lVert g\_\{\\mathrm\{KV\}\}\\rVert\_\{2\}/\\lVert g\_\{\\mathrm\{hidden\}\}\\rVert\_\{2\}before and after training, using geometric means over the same3232sequences\. Fresh initialization is measured at two, three, and five recursions for both windows and sizes\. It uses separate random weights, not saved initial checkpoints\. The ratio increases with depth even at initialization, but is larger in the trained models\. More recursions add more shared\-cache reads, so this trend alone does not establish that the feedback helps learning\. The detachment experiment tests that separately\.

![Refer to caption](https://arxiv.org/html/2610.02383v1/shared_feedback_depth_init.png)Figure 13:Shared\-KV feedback relative to the hidden\-state path\.Gradient\-norm ratios at fresh initialization and after training\. Whiskers:95%95\\%sequence\-bootstrap intervals\.
### Shared\-Cache Gradient Ablation

We train S and Mlptmodels withR∈\{2,3,5\}R\\in\\\{2,3,5\\\},ω=64\\omega=64, and final loss, changing only whether the shared KV entries are detached at each read by a later recursion\. The first recursion still receives gradients through the hidden\-state path, and all shared parameters remain trainable\. This removes direct gradient feedback through shared\-cache reads, not all training of the first recursion\.

[Table 6](https://arxiv.org/html/2610.02383#A4.T6)compares final\-losslt, detachedlpt, andlptafter2020B training tokens\. Detachment raises perplexity in every comparison\. Detached S at five recursions finishes well abovelpt, while M at three and five recursions remains near unigram\-level loss\. The experiment supports a role for shared\-cache gradients in learning and stability, but does not distinguish shorter gradient paths from changes in gradient scale or direction\.

Table 6:Detaching shared\-cache gradients hurts learning\.Validation perplexity \(↓\\downarrow\) at S and M, all models trained with final loss; values above1,0001\{,\}000are runs that failed to train\. Bold marks the best per column\.

## Appendix EExperiments at Nine Recursions

[Table 7](https://arxiv.org/html/2610.02383#A5.T7)and[Figure 14](https://arxiv.org/html/2610.02383#A5.F14)extend the comparisons to nine recursions\. We also listsps\([Monea et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib1)\)in theR=2R=2column, since its memory and FLOPs match those oflptwith two recursions\. From five to nine,lptgains0\.70\.7accuracy points at S and0\.20\.2at M, but loses1\.71\.7at L\. Perplexity improves at S and worsens at M and L\.ltandpltalso worsen in perplexity at every size\. Both hybrids continue to improve:h\-lptgains0\.40\.4,0\.20\.2, and0\.60\.6accuracy points at S, M, and L, respectively, and reaches the lowest perplexity of all models at every size\. At nine recursions,lptstill outperformsltat every size, andh\-lptoutperformslt2in both perplexity and accuracy\.

Higher recurrent depth can also complicate training\. Ouro reports loss spikes and gradient oscillations at eight recursions and reduces its recurrent depth to four\([Zhu et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib11)\)\.lt2also uses four recursions\([Deng et al\., 2026](https://arxiv.org/html/2610.02383#bib.bib24)\)\.

Table 7:Quality and efficiency through nine recursions\.One table per size\. Metrics and formatting follow[Table 1](https://arxiv.org/html/2610.02383#S3.T1): in thestdrow,R=1R=1is the standard Transformer andR\>1R\>1isdeepstd\.spsappears in theR=2R=2column, whose memory and FLOPs it matches\.SR=1R=1R=2R=2R=3R=3R=5R=5R=9R=9ModelPPL↓\\downarrowAcc\.↑\\uparrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowstd16\.8316\.8351\.551\.517\.0717\.0750\.250\.21\.501\.501\.1917\.2717\.2750\.750\.71\.881\.881\.3117\.6917\.6950\.450\.42\.502\.501\.5418\.3218\.3249\.149\.13\.383\.381\.81sps16\.1816\.1852\.61\.021\.021\.841\.84loopie16\.5816\.5851\.651\.62\.002\.001\.831\.8316\.5016\.5051\.951\.93\.003\.002\.662\.6616\.5616\.5652\.152\.15\.005\.004\.324\.3216\.9816\.9851\.051\.09\.009\.007\.647\.64lt16\.1716\.1751\.551\.52\.002\.001\.831\.8315\.9815\.9852\.152\.13\.003\.002\.662\.6616\.0016\.0052\.452\.45\.005\.004\.324\.3216\.1316\.1352\.252\.29\.009\.007\.647\.64plt15\.8415\.8452\.71\.021\.021\.841\.8415\.5515\.5552\.952\.91\.031\.032\.672\.6715\.4915\.4953\.553\.51\.061\.064\.344\.3415\.5115\.5153\.253\.21\.127\.687\.68lt215\.7651\.951\.90\.351\.2915\.4715\.4752\.552\.50\.521\.8615\.1615\.1653\.753\.70\.872\.9814\.9614\.9654\.454\.41\.561\.565\.23lpt15\.7351\.951\.91\.021\.021\.841\.8415\.2854\.01\.031\.032\.672\.6714\.9154\.91\.061\.064\.344\.3414\.8055\.61\.127\.687\.68h\-lpt15\.7352\.452\.40\.181\.2915\.3953\.10\.191\.8615\.0154\.60\.212\.9814\.7455\.00\.255\.23

MR=1R=1R=2R=2R=3R=3R=5R=5R=9R=9ModelPPL↓\\downarrowAcc\.↑\\uparrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowstd13\.7913\.7956\.156\.114\.2414\.2455\.555\.51\.441\.441\.1314\.4714\.4755\.455\.41\.781\.781\.2414\.8814\.8854\.454\.42\.342\.341\.4315\.6015\.6053\.453\.43\.093\.091\.63sps12\.9912\.9958\.958\.91\.021\.021\.931\.93loopie13\.6913\.6956\.956\.92\.002\.001\.921\.9213\.7513\.7555\.855\.83\.003\.002\.842\.8413\.7413\.7456\.656\.65\.005\.004\.694\.6915\.8015\.8052\.952\.99\.009\.008\.378\.37lt13\.2813\.2857\.557\.52\.002\.001\.921\.9213\.1313\.1359\.459\.43\.003\.002\.842\.8413\.3413\.3457\.957\.95\.005\.004\.694\.6913\.7213\.7256\.956\.99\.009\.008\.378\.37plt13\.0013\.0059\.11\.021\.021\.931\.9312\.6812\.6860\.060\.01\.031\.032\.852\.8512\.6512\.6560\.560\.51\.061\.064\.714\.7112\.8712\.8759\.559\.51\.128\.418\.41lt212\.8159\.10\.351\.4312\.6312\.6359\.559\.50\.522\.1012\.4312\.4360\.260\.20\.873\.4512\.3160\.460\.41\.561\.566\.14lpt12\.9112\.9159\.21\.021\.021\.931\.9312\.5560\.21\.031\.032\.852\.8512\.3361\.61\.061\.064\.704\.7012\.3912\.3961\.81\.128\.418\.41h\-lpt12\.8058\.558\.50\.181\.4312\.5460\.30\.192\.1012\.3161\.40\.213\.4512\.1461\.60\.256\.14

LR=1R=1R=2R=2R=3R=3R=5R=5R=9R=9ModelPPL↓\\downarrowAcc\.↑\\uparrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowPPL↓\\downarrowAcc\.↑\\uparrowKV↓\\downarrowFLOPs↓\\downarrowstd11\.3511\.3562\.562\.511\.6211\.6262\.562\.51\.451\.451\.1411\.9611\.9661\.361\.31\.731\.731\.1712\.2712\.2761\.461\.42\.252\.251\.3212\.6612\.6659\.859\.83\.153\.151\.63sps10\.7710\.7765\.565\.51\.021\.021\.961\.96loopie11\.2611\.2663\.563\.52\.002\.001\.951\.9511\.2811\.2863\.963\.93\.003\.002\.912\.9112\.2612\.2661\.061\.05\.005\.004\.814\.8111\.5311\.5363\.263\.29\.009\.008\.638\.63lt11\.2011\.2064\.264\.22\.002\.001\.951\.9511\.1111\.1163\.863\.83\.003\.002\.912\.9111\.3911\.3963\.463\.45\.005\.004\.814\.8112\.5212\.5260\.260\.29\.009\.008\.638\.63plt10\.9510\.9564\.864\.81\.021\.021\.961\.9610\.7510\.7566\.866\.81\.031\.032\.922\.9210\.5910\.5966\.466\.41\.061\.064\.834\.8310\.7810\.7866\.166\.11\.128\.668\.66lt210\.5966\.50\.401\.5310\.4710\.4765\.865\.80\.602\.2710\.3267\.367\.31\.013\.7510\.2068\.11\.811\.816\.71lpt10\.7510\.7566\.31\.021\.021\.961\.9610\.4567\.31\.031\.032\.922\.9210\.3268\.21\.061\.064\.834\.8310\.8710\.8766\.566\.51\.128\.668\.66h\-lpt10\.5865\.665\.60\.211\.5310\.4166\.90\.222\.2710\.2368\.10\.243\.7510\.1168\.70\.286\.726\.72

Figure 14:Quality and memory through nine recursions\.Validation perplexity, mean five\-task accuracy, and context memory relative tostd\.### Intermediate Readouts

We apply the model’s readout normalization and language\-modeling head to the output of each recursion\. The measurements use S\-sizelptcheckpoints trained for2020B tokens, averaged over1616validation sequences of4,0964\{,\}096tokens\. With nine recursions and final loss, intermediate\-readout perplexity ranges from8787to1,0121\{,\}012, falling to15\.615\.6at the last recursion\. With adaptive loss, it is already2\.792\.79by the fourth recursion and remains near that value\. Under final loss, the head is trained only on the last recursion: high intermediate perplexity measures poor prediction through this head, not the absence of predictive information in the hidden state\. These probe\-set losses are separate from the full validation\-set perplexity in[Table 7](https://arxiv.org/html/2610.02383#A5.T7)\.

### Complete Quality–Memory Sweep

[Figure 15](https://arxiv.org/html/2610.02383#A5.F15)includes every family, size, and recursion count in[Table 7](https://arxiv.org/html/2610.02383#A5.T7)exceptspsandplt\.[Figure 16](https://arxiv.org/html/2610.02383#A5.F16)compares nine\-recursion models withstdacross sizes\.

Figure 15:Quality versus context memory across model sizes\.Curves vary recursion count\. Marker size denotes recursions, and shade denotes model size\. S/M train for2020B tokens and L for5050B\.Figure 16:Quality and memory across model sizes\.Mean five\-task accuracy and context memory forstdand looped models at nine recursions\.

## Appendix FDecoding Speed and Memory on a GPU

We test whether the analytic memory savings translate into larger batches and higher decoding throughput on a GPU\. A smaller cache lets more sequences share a GPU, which can raise throughput\([Brandon et al\., 2024](https://arxiv.org/html/2610.02383#bib.bib4);[Sun et al\., 2024](https://arxiv.org/html/2610.02383#bib.bib5)\)\. As[O’Neill and Reid \(2026\)](https://arxiv.org/html/2610.02383#bib.bib20)do for a looped model, we measure the largest batch that fits at a fixed context and the throughput at that batch\.

### Setup

We decode on one NVIDIA H100 with8080GB of memory, in bf16, at a4,0964\{,\}096\-token context: each step generates one token per sequence from a cache already filled to4,0684\{,\}068tokens\. All models share one decoding implementation in PyTorch: a preallocated cache per attention layer, FlexAttention\([Dong et al\., 2025](https://arxiv.org/html/2610.02383#bib.bib9)\)for attention, and CUDA graphs that replay a whole decoding step, all recursions and layers included\. A later recursion oflptorh\-lptattends to its shared and local caches in one softmax, which we compute by merging the two FlexAttention outputs by their log\-sum\-exp\. Gated DeltaNet layers update their recurrent state one token at a time, with the kernels of flash\-linear\-attention\([Yang and Zhang, 2024](https://arxiv.org/html/2610.02383#bib.bib42)\)\.stdruns the same code with one recursion, andltruns all its recursions, without early exit\. Weights are random, since speed and memory do not depend on them\. On the same GPU, decoding reproduces the full forward pass to bf16 precision\. The maximum batch is the largest batch whose decoding steps run without running out of memory, and we measure throughput at this batch, or at95%95\\%of it when the CUDA graph does not fit\. These measurements come from one research implementation, not an optimized serving engine, which would change the absolute speeds\. Every model runs the same code, so the comparison holds within this implementation\.

### Capacity

[Table 8](https://arxiv.org/html/2610.02383#A6.T8)shows that the analytic savings carry over to the GPU\.lpt’s cache per sequence is at most6\.25%6\.25\\%larger thanstd’s, so it fits9393–98%98\\%ofstd’s batch\.lt’s cache grows with depth, solptfits2\.02\.0,2\.92\.9, and4\.74\.7–4\.84\.8times as many sequences asltat two, three, and five recursions\.h\-lptholds2020–27%27\\%ofstd’s cache and fits3\.63\.6–5\.05\.0times as many sequences asstd, and1\.81\.8–3\.83\.8times as many aslt2\. The hybrids’ measured caches are33–17%17\\%larger than the analytic values in[Table 1](https://arxiv.org/html/2610.02383#S3.T1), because the kernel keeps the recurrent state in fp32 and we also count the short\-convolution windows\.

### Speed

The larger batches raiselpt’s peak throughput abovelt’s by77–91%91\\%, most at L with five recursions, whereltfits only2121sequences\. At equal batch,lptis33–8%8\\%slower thanlt, since each later recursion still reads the whole shared cache, and its own window too\. As[Brandon et al\. \(2024\)](https://arxiv.org/html/2610.02383#bib.bib4)note for sharing across layers, a shared cache saves memory, not the time to read it\. Both looped models pay for depth in time: at batch11, a decoding step oflttakes about2\.32\.3,3\.43\.4, and5\.75\.7times as long asstd’s at two, three, and five recursions, andlpt’s takes88–12%12\\%longer thanlt’s\. The hybrids read a full\-context cache in only one layer of five, so they decode faster\. At its maximum batch,h\-lptgenerates2\.12\.1–2\.22\.2times as many tokens per second asstdwith two recursions,1\.41\.4–1\.51\.5times with three, and0\.80\.8–0\.90\.9times with five, and it is22–3%3\\%slower thanlt2at equal batch\.

### Throughput and Quality

[Figure 17](https://arxiv.org/html/2610.02383#A6.F17)plots quality against throughput\. At every size and recursion count,lpthas higher throughput, lower perplexity, and higher accuracy thanlt, andh\-lpthas higher throughput and lower perplexity thanlt2\. Againststd,h\-lptlowers perplexity by0\.940\.94–1\.441\.44with three recursions while decoding1\.41\.4–1\.51\.5times as many tokens per second, and by1\.121\.12–1\.821\.82with five, at8181–88%88\\%ofstd’s throughput\.

Table 8:Decoding capacity and speed on one H100\.One table per size\. bf16,4,0964\{,\}096\-token context\. Batch: the largest batch that decodes without running out of the8080GB\. Tok/s: generated tokens per second at that batch\. Lat\.: milliseconds per decoding step at batch11\. Our rows are shaded\.
Figure 17:h\-lptis better thanstdat similar or higher throughput\.Validation perplexity \(top\) and mean five\-task accuracy \(bottom\) against generated tokens per second at each model’s maximum batch, relative to same\-sizestd, on one H100 at a4,0964\{,\}096\-token context\. Curves follow two, three, and five recursions, with larger markers at five\. Dashed lines markstd\.