RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers

# RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers
Source: [https://arxiv.org/html/2608.15062](https://arxiv.org/html/2608.15062)
###### Abstract

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency\. While unique weights across layers preserve functional specialization—from input\-grounding to abstract refinement—they incur a substantial memory footprint\. Conversely, standard depth\-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality\. We introduceRecurrentGPT, a recurrent depth transformer where fixed\-depth prelude and coda blocks bracket a single shared core iteratedRRtimes\. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate—conditioned on the hidden state, the fixed prelude output, and noise resampled at every step—to modulate the recurrent update\. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity\. Under anisoFLOPsconstraint, a 3\-layerRecurrentGPTmatches the accuracy of a 12\-layer GPT\-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy\-tail depth sampling in all nine scale\-by\-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled\. Under anisoParamsconstraint, deeper recurrence achieves a2\.76validation loss versus2\.84for a non\-recurrent counterpart at matched parameter and data budget\. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale,63%63\\%fewer parameters and59%59\\%less peak decoding memory for a10%10\\%increase in compiled generation latency\.

## 1Introduction

Scaling transformer language models has delivered remarkable gains, yet it necessitates a simultaneous increase in parameter count and training compute\[[26](https://arxiv.org/html/2608.15062#bib.bib14)\]\. In standard architectures, depth and parameters are rigidly coupled: every additional layer introduces a fresh set of weights, creating a memory bottleneck that constrains effective depth under fixed hardware budgets\.

Beyond memory efficiency, however, there is a deeper computational motivation rooted in the nature of intelligence itself: Turing’s foundational insight\[[38](https://arxiv.org/html/2608.15062#bib.bib28)\]is that a*finite*set of states and symbols, applied*iteratively*, is sufficient to compute anything computable — suggesting that the power of a reasoning system lies not in its breadth of parameters, but in its depth of iteration\. This principle finds a striking modern echo in the test\-time compute paradigm\[[37](https://arxiv.org/html/2608.15062#bib.bib31)\], where models allocate more compute at inference to improve reasoning\[[31](https://arxiv.org/html/2608.15062#bib.bib29),[21](https://arxiv.org/html/2608.15062#bib.bib30)\]\. While existing approaches realize this compute through chain\-of\-thought steps in the output space—consuming sequence length in the process\[[13](https://arxiv.org/html/2608.15062#bib.bib32)\]—recurrent depth offers a complementary realization: by iterating a shared transformation over the input representation, the model “thinks” more deeply entirely within its hidden states without generating extra tokens or storing extra parameters\.

Weight sharing is the natural response to memory\-depth coupling, allowing effective depth to grow without inflating parameter count\[[30](https://arxiv.org/html/2608.15062#bib.bib1),[9](https://arxiv.org/html/2608.15062#bib.bib8),[15](https://arxiv.org/html/2608.15062#bib.bib2),[41](https://arxiv.org/html/2608.15062#bib.bib11)\]\. Prior work has utilized recurrent depth either as an efficiency tool to match quality at reduced parameter counts\[[24](https://arxiv.org/html/2608.15062#bib.bib24),[2](https://arxiv.org/html/2608.15062#bib.bib25)\]or as a performance tool to improve accuracy at matched parameter counts\[[15](https://arxiv.org/html/2608.15062#bib.bib2),[41](https://arxiv.org/html/2608.15062#bib.bib11)\]\. However, a unified treatment of these perspectives is missing, and a deeper architectural tension remains: standard weight sharing forces an identical transformation on an ever\-evolving representation\. Because the hidden state after the first recurrence differs fundamentally from the eighth, a static transformation collapses the functional diversity that makes depth valuable\[[6](https://arxiv.org/html/2608.15062#bib.bib33)\]\. This raises the question: Can a single fixed transformation serve meaningfully across such diverse representational stages, or does it inevitably collapse the functional diversity that makes depth valuable in the first place\[[6](https://arxiv.org/html/2608.15062#bib.bib33)\]\. Such questions motivate the design ofRecurrentGPT\.

To resolve this tension,RecurrentGPTintroduces a per\-element learned gate that conditions on the current hidden state, a fixed*prelude*representation of the original input\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\], and stochastic noise\. This constructs a distinct, context\-grounded input at every recurrence, allowing a single weight tensor to behave as multiple specialized layers\. Gates are initialized such that training starts with the residual stream passing through nearly unchanged through all recurrences\. As training progresses, gating emerges gradually as the shared block learns to selectively refine representations\. The shared transformation is thus fixed in weights but dynamic in behavior — the model thinks differently at each pass despite reusing the same parameters\. Combined with depth sampling during training\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\], this architecture enables a continuous compute\-quality tradeoff at inference and facilitates evaluation across bothisoFLOPsandisoParamsregimes\.

We summarize our contributions as follows:

- •Architecture: We proposeRecurrentGPT, which uses per\-element gating and stochastic perturbations to prevent representational collapse, allowing a shared core to behave distinctly across recurrence steps\.
- •Unified Evaluation: We evaluateRecurrentGPTunder bothisoFLOPsandisoParamsregimes at three scales\. UnderisoFLOPsit leads MoR and heavy\-tail depth sampling in all nine scale\-by\-budget cells and matches or leads RRT and Ouro at every scale at the standard budget, matches the dense GPT\-2 baseline at small scale while using64%fewer parameters, and closes on it at medium and large scale as the token budget grows\. UnderisoParamsit improves validation loss at matched parameter count, at a higher inference\-FLOP cost\.
- •Emergent Early Exit: Without auxiliary losses,RecurrentGPTnaturally supports inference\-time early exiting, retaining∼\\sim92%accuracy when running only half of its recurrence steps\.

Figure 1:RecurrentGPT overview\.\(a\) A Gated Recurrent Unit \(GRU\) reuses the same weight matrices at every sequence position—recurrence over the*sequence*dimension\. \(b\) A standard transformer stacksLLindependent layers, each with unique weights, applied once to the full sequence\. \(c\)RecurrentGPTintroduces recurrence over the*depth*dimension, applying each recurrence to the full sequence
## 2Related Work

The concept of shared weights across repeated computational steps traces back to the seminal 1986 backpropagation paper of Rumelhart, Hinton, and Williams\[[36](https://arxiv.org/html/2608.15062#bib.bib17)\], who briefly proposed at the end of their paper the synchronous iterative net: a network where each iteration corresponds to a layer with tied weights\.\[[39](https://arxiv.org/html/2608.15062#bib.bib18)\]extended this to temporal sequence modeling via backpropagation through time \(BPTT\), inspiring recurrent architectures — RNNs\[[12](https://arxiv.org/html/2608.15062#bib.bib19)\], LSTMs\[[22](https://arxiv.org/html/2608.15062#bib.bib20)\], and GRUs\[[8](https://arxiv.org/html/2608.15062#bib.bib21)\]— that apply shared weights along the sequence dimension, processing inputs one timestep at a time\. In contrast, this work applies recurrent depth across the full input, iterating a shared\-weight transformation along network depth to progressively refine the input representation\.

In the era of transformers, one of the earliest papers on recurrent depth was ALBERT\[[30](https://arxiv.org/html/2608.15062#bib.bib1)\], which proposed a BERT\[[10](https://arxiv.org/html/2608.15062#bib.bib7)\]model with all transformer layers sharing the same weights\. Universal Transformers\[[9](https://arxiv.org/html/2608.15062#bib.bib8)\]further added a per\-token adaptive halting mechanism\. However, both methods shared the*entire*layer stack, forcing every layer to apply an identical transformation regardless of the representational stage of the input\.\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]relaxed this constraint by assigning distinct roles to prelude, shared core, and coda layers\. Similarly,[28](https://arxiv.org/html/2608.15062#bib.bib10)identified early, middle, and late layers of pretrained LLMs as encoding, reasoning, and decoding stages respectively, and proposed recurring only the middle layers while keeping the rest fixed\. OurRecurrentGPTfollows this principle of selectively recurring middle layers\.

Prior work has shown that parameters in feed\-forward network \(FFN\) layers primarily store factual knowledge\[[17](https://arxiv.org/html/2608.15062#bib.bib23)\], while computational depth drives reasoning ability\[[42](https://arxiv.org/html/2608.15062#bib.bib22)\], motivating recurrence to improve reasoning\. We organize prior work on recurrent depth transformers along two evaluation regimes\. In theisoParamsregime — increasing FLOPs by recurring layers while keeping parameter count fixed, trading compute for improved accuracy —\[[15](https://arxiv.org/html/2608.15062#bib.bib2),[41](https://arxiv.org/html/2608.15062#bib.bib11),[28](https://arxiv.org/html/2608.15062#bib.bib10)\]demonstrate consistent quality gains\. In theisoFLOPsregime — reducing parameter count by recurring layers while matching training and inference compute, trading memory for efficiency —\[[24](https://arxiv.org/html/2608.15062#bib.bib24)\]and\[[2](https://arxiv.org/html/2608.15062#bib.bib25)\]show that depth reuse can achieve comparable quality at a smaller memory footprint\. Our work spans both regimes, withisoFLOPsas our primary contribution and complementaryisoParamsresults reported in Section[4](https://arxiv.org/html/2608.15062#S4)\.

A related line of work adapts the amount of computation dynamically rather than fixing it uniformly across tokens or layers\. Adaptive Computation Time \(ACT\)\[[20](https://arxiv.org/html/2608.15062#bib.bib9)\]halts computation per\-position based on learned termination signals, allowing different tokens to consume different amounts of compute\. Mixture\-of\-Depths\[[35](https://arxiv.org/html/2608.15062#bib.bib13)\]takes a routing approach, dynamically assigning tokens to different subsets of layers rather than processing all tokens through all layers\. More recently, Mixture\-of\-Recursions\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]extends this idea to the recurrent setting, routing subsets of tokens to different numbers of recursion steps\. While all three methods adapt computation at the token level — through halting, layer routing, or recursion routing — they require dedicated termination signals or routing mechanisms\. In contrast,RecurrentGPTuses a simpler gating mechanism that modulates how much of the shared\-block update is absorbed at each recurrence step, without any routing or halting logic\.

Weight\-tied iteration toward a fixed point is the defining feature of deep equilibrium models\[[4](https://arxiv.org/html/2608.15062#bib.bib40)\], which solve for the equilibrium directly and differentiate through it with implicit gradients, avoiding the memory cost of storing an unrolled trajectory\.RecurrentGPTshares the weight\-tied update but not the equilibrium objective: we unroll a fixed number of discrete steps and backpropagate through them, and depth sampling trains each step to serve as an exit rather than as an approach to a single limit\.

Scaling laws\[[26](https://arxiv.org/html/2608.15062#bib.bib14),[23](https://arxiv.org/html/2608.15062#bib.bib15)\]have established how model size and data jointly determine performance, with Pythia\[[5](https://arxiv.org/html/2608.15062#bib.bib16)\]providing controlled comparisons across scales\. Notably,[26](https://arxiv.org/html/2608.15062#bib.bib14)directly evaluated parameter\-sharing transformers and observed that recurrent models perform better at matched parameter count \(isoParams\) but worse at matched compute \(isoFLOPs\)\. The interplay between depth and width has received dedicated attention:\[[40](https://arxiv.org/html/2608.15062#bib.bib26)\]theoretically shows that increasing width can compensate for reduced depth in algorithmic reasoning tasks, while\[[33](https://arxiv.org/html/2608.15062#bib.bib27)\]empirically demonstrates that scaling law prescriptions are sensitive to depth\-width ratio\. In ourRecurrentGPT, we propose a novel recurrent depth architecture that aims to improve the scaling laws of loss versus parameter count and loss versus training FLOPs, demonstrating consistent gains under bothisoFLOPsandisoParamsregimes as model size and training data scale\.

## 3Recurrent Depth Reuse for Language Modeling

### 3\.1Preliminaries

Let𝐗=\(x1,…,xT\)\\mathbf\{X\}=\(x\_\{1\},\\ldots,x\_\{T\}\)be a token sequence of lengthTTdrawn from vocabulary𝒱\\mathcal\{V\}\. A standard autoregressive transformer withLLblocks defines a chain of residual updates:𝐡\(ℓ\)=𝐡\(ℓ−1\)\+Blockℓ​\(𝐡\(ℓ−1\)\)\\mathbf\{h\}^\{\(\\ell\)\}=\\mathbf\{h\}^\{\(\\ell\-1\)\}\+\\mathrm\{Block\}\_\{\\ell\}\(\\mathbf\{h\}^\{\(\\ell\-1\)\}\)forℓ=1,…,L\\ell=1,\\ldots,L, whereddis the embedding dimension and𝐡\(0\)∈ℝT×d\\mathbf\{h\}^\{\(0\)\}\\in\\mathbb\{R\}^\{T\\times d\}are initial token embeddings\. Each block comprises pre\-norm multi\-head self\-attention and a token\-wise MLP\. The objective is to minimize the negative log\-likelihood,ℒ=−∑tlogp\(xt∣x<t\)\\mathcal\{L\}=\-\\sum\_\{t\}\\log p\(x\_\{t\}\\mid x\_\{<t\}\)\. Standard transformers couple depth and parameters rigidly, as every additional layer adds a fresh set of weights\. The total unique parameter count scales asΘ⁡\(L⋅d2\)\\Theta\(L\\cdot d^\{2\}\)\.

### 3\.2The Prelude–Shared–Coda Architecture

RecurrentGPTfollows\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]in partitioning transformer blocks into three sets, denoted by the shorthandnpre​\+​nrec×R​\+​ncodan\_\{\\text\{pre\}\}\\texttt\{\+\}n\_\{\\text\{rec\}\}\\\!\\times\\\!R\\texttt\{\+\}n\_\{\\text\{coda\}\}\. We have observed that this separation of fixed context encoders from a shared recurrent core yields more stable training than uniform weight sharing across all layers\.

- •Prelude \(npren\_\{\\text\{pre\}\}blocks\):Applied once to produce a stable, context\-aware conditioning signal𝐡\(pre\)=Blocknpre∘⋯∘Block1\(𝐡\(0\)\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}=\\mathrm\{Block\}\_\{n\_\{\\text\{pre\}\}\}\\circ\\cdots\\circ\\mathrm\{Block\}\_\{1\}\(\\mathbf\{h\}^\{\(0\)\}\)\.
- •Shared core \(nrecn\_\{\\text\{rec\}\}blocks\):Applied recurrentlyRRtimes\. The first step is initialized with stochastic noiseϵx\\epsilon\_\{x\}conditioned on the prelude output\. In subsequent steps, the core iteratively processes the previous hidden state𝐡\(r−1\)\\mathbf\{h\}^\{\(r\-1\)\}alongside the fixed prelude anchor:𝐡\(R\)=Blocknshared∘⋯∘Block1\[ϵx,𝐡\(pre\)\]\\mathbf\{h\}^\{\(R\)\}=\\mathrm\{Block\}\_\{n\_\{\\text\{shared\}\}\}\\circ\\cdots\\circ\\mathrm\{Block\}\_\{1\}\[\\epsilon\_\{x\},\\,\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}\]\. We provide more details on the operation in the subsequent section\.
- •Coda \(ncodan\_\{\\text\{coda\}\}blocks\):Receives the final refined state𝐡\(R\)\\mathbf\{h\}^\{\(R\)\}and projects to output final hidden state𝐡\(coda\)=Blockncoda∘⋯∘Block1\(𝐡\(R\)\)\\mathbf\{h\}^\{\(\\mathrm\{coda\}\)\}=\\mathrm\{Block\}\_\{n\_\{\\text\{coda\}\}\}\\circ\\cdots\\circ\\mathrm\{Block\}\_\{1\}\(\\mathbf\{h\}^\{\(R\)\}\)that are then fed into the language model to produce logits\.

Total unique parameter count isΘ⁡\(\(npre\+nrec\+ncoda\)⋅d2\)\\Theta\\bigl\(\(n\_\{\\text\{pre\}\}\+n\_\{\\text\{rec\}\}\+n\_\{\\text\{coda\}\}\)\\cdot d^\{2\}\\bigr\), independent ofRR\. A configuration such as2\+5×\\times4\+2visits2\+5⋅4\+2=242\+5\\cdot 4\+2=24block executions per forward pass while storing weights for only2\+5\+2=92\+5\+2=9blocks—2\.6×2\.6\\timesfewer unique blocks than the 24\-layer GPT\-2 medium baseline it is isoFLOP\-matched to\. Figure[2](https://arxiv.org/html/2608.15062#S3.F2)illustrates this layout\.

Per\-token forward\-pass cost for anpre​\+​nrec×R​\+​ncodan\_\{\\text\{pre\}\}\\texttt\{\+\}n\_\{\\text\{rec\}\}\\\!\\times\\\!R\\texttt\{\+\}n\_\{\\text\{coda\}\}configuration at sequence lengthSSis

FLOPs=\(npre\+nrec​R\+ncoda\)​\(24​d2\+4​S​d\)\+R⋅10​d2,\\text\{FLOPs\}=\\bigl\(n\_\{\\text\{pre\}\}\+n\_\{\\text\{rec\}\}R\+n\_\{\\text\{coda\}\}\\bigr\)\\bigl\(24d^\{2\}\+4Sd\\bigr\)\\;\+\\;R\\cdot 10d^\{2\},\(1\)where24​d2\+4​S​d24d^\{2\}\+4Sdis the standard per\-block cost \(attention and MLP projections plus the two attention matmuls\) and10​d210d^\{2\}is the per\-step overhead of the recurrent projectionWprojW\_\{\\mathrm\{proj\}\}and the gate MLPf𝐠f\_\{\\mathbf\{g\}\}\.

Figure 2:RecurrentGPTarchitecture \(2\+5×\\times8\+2layout\)\.*Top:*Two prelude blocks \(gray\) encode a fixed context representation𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}that is held constant across all recurrence steps\. Five shared blocks \(blue\) are appliedR=8R=8times recurrently;𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}conditions the projection and gate at each step\. Two coda blocks \(green\) project to logits\.*Bottom:*Zoom\-in of a single recurrence steprr: the current state𝐡\(r−1\)\\mathbf\{h\}^\{\(r\-1\)\}is projected together with𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}viaWprojW\_\{\\mathrm\{proj\}\}, processed by the shared blocks to produce proposal𝐨\(r\)\\mathbf\{o\}^\{\(r\)\}, and blended back via the elementwise gate𝐠\(r\)∈\[0,1\]T×d\\mathbf\{g\}^\{\(r\)\}\\in\[0,1\]^\{T\\times d\}\.
### 3\.3Recurrent Projection and Elementwise Gate

Merely feeding the output of each recurrence as the input to the next recurrence risks forgetting the original input context, suffers from vanishing or exploding gradients across deep recurrences, and taxes the representational capacity of a single shared weight tensor\.

Inspired by RNNs\[[12](https://arxiv.org/html/2608.15062#bib.bib19)\]in general, and GRUs\[[8](https://arxiv.org/html/2608.15062#bib.bib21)\]in particular, we treat the hidden representation after each recurrence step as a*state*that evolves across recurrences \(as illustrated in Figure[1](https://arxiv.org/html/2608.15062#S1.F1)\)\. Each application of the shared core reads this state, transforms it, and writes an updated state back—analogous to an RNN cell, but operating over depth rather than time\. This framing motivates the gating mechanism below: just as GRU gates control how much of its cell state is overwritten at each timestep,RecurrentGPT’s elementwise gate controls how much of the current depth\-state is replaced by the shared block’s output\.

Hence, we introduce our novel adaptive gating approach that aims to mitigate the 3 aforementioned problems\. At each recurrence steprr,RecurrentGPTprepares the input toℬshared\\mathcal\{B\}\_\{\\text\{shared\}\}by projecting the current state together with the fixed prelude output, then selectively writes the block’s output back to the residual stream via a learned elementwise gate\. The full update is:

𝐡~\(r\)\\displaystyle\\tilde\{\\mathbf\{h\}\}^\{\(r\)\}=Wproj​\[𝐡\(r−1\)\+ϵx,𝐡\(pre\)\],\\displaystyle=W\_\{\\mathrm\{proj\}\}\\,\\bigl\[\\mathbf\{h\}^\{\(r\-1\)\}\+\\epsilon\_\{x\},\\;\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}\\bigr\],\(2\)𝐨\(r\)\\displaystyle\\mathbf\{o\}^\{\(r\)\}=ℬshared​\(𝐡~\(r\)\),\\displaystyle=\\mathcal\{B\}\_\{\\text\{shared\}\}\\\!\\left\(\\tilde\{\\mathbf\{h\}\}^\{\(r\)\}\\right\),\(3\)𝐡\(r\)\\displaystyle\\mathbf\{h\}^\{\(r\)\}=𝐠\(r\)⊙𝐡\(r−1\)\+\(1−𝐠\(r\)\)⊙𝐨\(r\),\\displaystyle=\\mathbf\{g\}^\{\(r\)\}\\odot\\mathbf\{h\}^\{\(r\-1\)\}\+\\bigl\(1\-\\mathbf\{g\}^\{\(r\)\}\\bigr\)\\odot\\mathbf\{o\}^\{\(r\)\},\(4\)where\[⋅,⋅\]\[\\cdot,\\cdot\]denotes concatenation along the feature axis,𝐠\(r\)∈\[0,1\]T×d\\mathbf\{g\}^\{\(r\)\}\\in\[0,1\]^\{T\\times d\}is a learned elementwise gate \(defined in full below in Eq\. \([5](https://arxiv.org/html/2608.15062#S3.E5)\)\),Wproj∈ℝd×2​dW\_\{\\mathrm\{proj\}\}\\in\\mathbb\{R\}^\{d\\times 2d\}is a learned projection,ϵx∼𝒩⁡\(0,σ2​𝐈\)\\epsilon\_\{x\}\\sim\\mathcal\{N\}\(0,\\,\\sigma^\{2\}\\mathbf\{I\}\)is injected additive noise, and⊙\\odotis elementwise multiplication\.

The gate𝐠\(r\)∈\[0,1\]T×d\\mathbf\{g\}^\{\(r\)\}\\in\[0,1\]^\{T\\times d\}is produced by a small feed\-forward network conditioned on the normalised current state and the normalised prelude output:

𝐠\(r\)=σ⁡\(f𝐠​\(\[LN⁡\(𝐡\(r−1\)\),LN⁡\(𝐡\(pre\)\)\]\)/τ\+ϵg\),\\mathbf\{g\}^\{\(r\)\}=\\sigma\\\!\\left\(f\_\{\\mathbf\{g\}\}\\\!\\bigl\(\[\\mathrm\{LN\}\(\\mathbf\{h\}^\{\(r\-1\)\}\),\\;\\mathrm\{LN\}\(\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}\)\]\\bigr\)/\\tau\+\\epsilon\_\{g\}\\right\),\(5\)whereσ⁡\(⋅\)=1/\(1\+e−\(⋅\)\)\\sigma\(\\cdot\)=1/\(1\+e^\{\-\(\\cdot\)\}\)is the sigmoid function,LN\\mathrm\{LN\}denotes layer normalization\[[1](https://arxiv.org/html/2608.15062#bib.bib34)\],τ\\tauis a temperature hyperparameter \(set to1\.01\.0throughout our experiments\),ϵg∼𝒩⁡\(0,σg2\)\\epsilon\_\{g\}\\sim\\mathcal\{N\}\(0,\\,\\sigma\_\{g\}^\{2\}\)is per\-scalar gate noise injected during training, andf𝐠f\_\{\\mathbf\{g\}\}is a two\-layer MLP with SiLU activation and hidden dimensiondgate=dd\_\{\\text\{gate\}\}=d\. The second linear layer off𝐠f\_\{\\mathbf\{g\}\}has its bias initialised to\+4\+4, placing𝐠\(r\)≈0\.98\\mathbf\{g\}^\{\(r\)\}\\approx 0\.98at the start of training: the copy branch of Eq\. \([4](https://arxiv.org/html/2608.15062#S3.E4)\) dominates, the residual stream passes through the recurrence almost unchanged, and the model learns which elements to overwrite as training proceeds\. Positive initialisation of a forget gate is long\-standing practice in recurrent networks\[[16](https://arxiv.org/html/2608.15062#bib.bib41)\], and[25](https://arxiv.org/html/2608.15062#bib.bib42)found a bias of\+1\+1or\+2\+2enough to bring a vanilla LSTM level with the best variants in their search\. We simply apply the same principle over depth rather than time\. We swept across\{−2,0,\+2,\+4\}\\\{\-2,0,\+2,\+4\\\}at the full training horizon the spread is0\.0190\.019nats \(Section[B\.6](https://arxiv.org/html/2608.15062#A2.SS6)\)\.

By defining𝐠t,i\(r\)\\mathbf\{g\}^\{\(r\)\}\_\{t,i\}as theithi^\{\\text\{th\}\}element of the gate for tokenttat recurrencerr,RecurrentGPTachieves a highly granular, per\-element specialization\. When𝐠t,i\(r\)→𝟏\\mathbf\{g\}^\{\(r\)\}\_\{t,i\}\\to\\mathbf\{1\}, the residual stream remains unchanged, whereas𝐠t,i\(r\)→𝟎\\mathbf\{g\}^\{\(r\)\}\_\{t,i\}\\to\\mathbf\{0\}allows the block output to fully replace the state\. This per\-recurrence specialization provides a more memory\-efficient alternative to approaches such as Per\-Layer Embeddings \(PLE\) \(used in recent models like Gemma3n\[[18](https://arxiv.org/html/2608.15062#bib.bib39)\]and Gemma\-4\[[19](https://arxiv.org/html/2608.15062#bib.bib38)\]\)\. While PLE requiresLLunique embedding layers to provide layer\-specific context, our approach conditions the gate on𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}to provide a constant view of the*original*input at every step, while modulating it with𝐡r−1\\mathbf\{h\}^\{r\-1\}and stochasticityϵg\\epsilon\_\{g\}\.

State noiseϵx\\epsilon\_\{x\}\(Eq\. \([2](https://arxiv.org/html/2608.15062#S3.E2)\)\) perturbs the recurrent projection input, discouraging the model from learning brittle exact\-match patterns across steps\. Gate noiseϵg\\epsilon\_\{g\}\(Eq\. \([5](https://arxiv.org/html/2608.15062#S3.E5)\)\) prevents the gate from collapsing to a near\-constant value during training\. Specific noise magnitudes \(σx\\sigma\_\{x\},σg\\sigma\_\{g\}\) are reported in Section[4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1)\.

### 3\.4Recurrence Depth Sampling During Training

At each training step, we sampler∼Uniform​\{1,…,R\}r\\sim\\mathrm\{Uniform\}\\\{1,\\ldots,R\\\}\. This serves two purposes: \(1\) it implicitly trains every exit point, enablingearly exit at inference without auxiliary loss termsor the associated gradient interference common in multi\-exit models\[[7](https://arxiv.org/html/2608.15062#bib.bib36),[11](https://arxiv.org/html/2608.15062#bib.bib35)\]; and \(2\) it acts as stochastic depth regularization, improving final validation loss over fixed\-depth schedules \(as will be shown later in our ablations Section[5](https://arxiv.org/html/2608.15062#S5)\.

We summarize the forward pass ofRecurrentGPTin Listing[A](https://arxiv.org/html/2608.15062#A1)\.

## 4Experiments & Results

#### Experimental Setup

We build on the nanoGPT codebase\[[27](https://arxiv.org/html/2608.15062#bib.bib3)\]for our transformer backbone implementation\. All models are trained on a diverse dataset of text with sequence lengthT=1024T=1024tokens and GPT\-2 BPE tokenisation \(50,257\-token vocabulary\)\. We use AdamW\[[32](https://arxiv.org/html/2608.15062#bib.bib4)\]withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay0\.10\.1, a 2,000\-step linear learning rate warmup, and a cosine learning\-rate schedule from a peak of6×10−46\\times 10^\{\-4\}to6×10−56\\times 10^\{\-5\}\.

Gradient norms are clipped at1\.01\.0\. All runs usedbfloat16mixed precision\. State noiseσx=0\.1\\sigma\_\{x\}=0\.1and gate noiseσg=0\.1\\sigma\_\{g\}=0\.1are used throughout allRecurrentGPTruns; the gate temperature is fixed atτ=1\.0\\tau=1\.0\.

We train for 20,000\-steps and use an effective batch size of≈\\approx491,520 tokens per step \(batch size 8, gradient accumulation 60, sequence length 1024\), totalling approximately9\.8Btokens per run\. Dense baselines \(GPT\-2 small, medium, and large\[[34](https://arxiv.org/html/2608.15062#bib.bib6)\]\) are trained from scratch under the identical setup; no pre\-trained weights are used\. We train four recurrent competitors from scratch under the same recipe:*Mixture\-of\-Recursions*\(MoR\)\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]; a variant following[15](https://arxiv.org/html/2608.15062#bib.bib2)that uses the prelude–coda layout with heavy\-tail Poisson depth sampling;*Relaxed Recursive Transformers*\(RRT\)\[[2](https://arxiv.org/html/2608.15062#bib.bib25)\], which relaxes weight tying with per\-recurrence LoRA adapters; and*Ouro*\[[41](https://arxiv.org/html/2608.15062#bib.bib11)\], which supervises every loop iteration\.

Table 1:Main results \(validation loss\)\.*Upper*:isoFLOPsregime—RecurrentGPTmatches forward\-pass FLOPs of the dense baseline with fewer unique parameters\.Shaded rowsare dense baselines\.*Lower*:isoParamsregime—matched unique parameters, higher FLOPs per step\.Layers= unique transformer blocks\.↓\\downarrowlower is better\.†RRT adds one LoRA adapter per recurrence, so its weights are not identical across steps\.ModelConfigLayersFLOPs/fwdParamsVal Loss↓\\downarrow*isoFLOPsregime \(matched FLOPs per forward pass\)*GPT\-2 SmallBaseline12L121\.84 G124M3\.15MoR\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]1\+1×\\times10\+131\.84 G36M3\.30Heavy\-tail Poisson\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]1\+1×\\times10\+131\.84 G36M3\.23Ouro\[[41](https://arxiv.org/html/2608.15062#bib.bib11)\]1\+1×\\times10\+131\.84 G36M3\.19RRT\[[2](https://arxiv.org/html/2608.15062#bib.bib25)\]†1\+1×\\times10\+131\.84 G36M\+\+LoRA3\.14RecurrentGPT\(ours\)1\+1×\\times10\+131\.84 G36M3\.14GPT\-2 MediumBaseline24L247\.35 G354M2\.84MoR\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]2\+5×\\times4\+297\.35 G124M3\.02Heavy\-tail Poisson\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]2\+5×\\times4\+297\.35 G124M2\.97Ouro\[[41](https://arxiv.org/html/2608.15062#bib.bib11)\]2\+5×\\times4\+297\.35 G124M2\.93RRT\[[2](https://arxiv.org/html/2608.15062#bib.bib25)\]†2\+5×\\times4\+297\.35 G124M\+\+LoRA2\.95RecurrentGPT\(ours\)2\+5×\\times4\+297\.35 G124M2\.89GPT\-2 LargeBaseline36L3621\.1 G774M2\.71MoR\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]1\+5×\\times6\+51121\.1 G288M2\.91Heavy\-tail Poisson\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]1\+5×\\times6\+51121\.1 G288M2\.93Ouro\[[41](https://arxiv.org/html/2608.15062#bib.bib11)\]1\+5×\\times6\+51121\.1 G288M2\.82RRT\[[2](https://arxiv.org/html/2608.15062#bib.bib25)\]†1\+5×\\times6\+51121\.1 G288M\+\+LoRA2\.85RecurrentGPT\(ours\)1\+5×\\times6\+51121\.1 G288M2\.77*isoParamsregime \(matched unique parameter count\)*GPT\-2 MediumBaseline24L247\.35 G354M2\.84MoR\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]2\+20×\\times4\+22426\.4 G354M2\.78Heavy\-tail Poisson\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]2\+20×\\times4\+22426\.4 G354M2\.74RecurrentGPT\(ours\)2\+20×\\times4\+22426\.4 G354M2\.76GPT\-2 LargeBaseline36L3621\.1 G774M2\.71MoR\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]3\+30×\\times6\+336109\.0 G774M2\.69Heavy\-tail Poisson\[[15](https://arxiv.org/html/2608.15062#bib.bib2)\]3\+30×\\times6\+336109\.0 G774M2\.70RecurrentGPT\(ours\)3\+30×\\times6\+336109\.0 G774M2\.65Table 2:Downstream task evaluation \(large scale\)\.Zero\-shot accuracy on standard benchmarks usinglm\-eval\-harness\.↑\\uparrowhigher is better; deltas relative to GPT\-2 Large\.Orange: bestisoFLOPsresult;blue: bestisoParamsresult\. TheisoParamsmodel outperforms the dense baseline on average by\+2\.10 points; theisoFLOPsmodel matches GPT\-2 Large at only 37% of its parameter count\.
#### Main Results

Table[1](https://arxiv.org/html/2608.15062#S4.T1)summarizes performance across model sizes\. In theisoFLOPsregime,RecurrentGPTachieves competitive quality while storing only36–37%of the baseline’s parameters\. At small scale it reaches lower loss than the dense baseline on all three seeds we ran:3\.145±0\.0043\.145\\pm 0\.004against3\.188±0\.0563\.188\\pm 0\.056, our worst seed at3\.1483\.148and the baseline’s best at3\.1543\.154\(Appendix[B\.1](https://arxiv.org/html/2608.15062#A2.SS1)\)\. At medium and large scale the dense baseline leads at the standard budget, by0\.050\.05and0\.060\.06nats respectively, and Section[5](https://arxiv.org/html/2608.15062#S5.SS0.SSS0.Px2)shows both gaps closing as the token budget grows\.

Against the recurrent baselines the margin widens with scale\. RRT is the strongest of the four and reaches parity at small scale,3\.1433\.143against our3\.1413\.141, a difference well inside the±0\.004\\pm 0\.004seed spread; at medium it trails by0\.060\.06nats and at large by0\.080\.08\. MoR and heavy\-tail Poisson trail by0\.080\.08to0\.160\.16nats at every scale\. We attribute the widening gap to what each method holds fixed\. RRT’s LoRA deltas are chosen at training time and applied identically to every input, so the diversity they buy is fixed in advance and does not grow with the number of recurrences, whereas the gate conditions on the state it is about to update and keeps successive states separable where uniform sharing collapses them\. Ouro is competitive at small scale \(3\.193\.19\) but its medium run converged to2\.932\.93and its large run converged to2\.822\.82\. Strict weight tying is also what leavesR∗R^\{\*\}free at inference and produces the early\-exit curve of Figure[3\(a\)](https://arxiv.org/html/2608.15062#S4.F3.sf1), which a fixed set of adapters cannot produce from a single checkpoint\. Appendix[B\.3](https://arxiv.org/html/2608.15062#A2.SS3)tabulates the design differences alongside training cost\.

In theisoParamsregime, increasing recurrenceRRat a fixed parameter budget yields significant gains: a0\.08nat improvement at Medium scale and0\.06at Large scale\. This demonstrates that for a fixed memory footprint, trading inference FLOPs for recurrent depth is a principled strategy for improving model expressivity\.

Table[2](https://arxiv.org/html/2608.15062#S4.T2)reports accuracy across nine benchmarks at large scale usinglm\-eval\-harness\[[14](https://arxiv.org/html/2608.15062#bib.bib37)\]\. TheisoFLOPsvariant matches the dense model on average \(42\.08 vs\. 42\.05\), confirming that parameter efficiency translates across evaluation protocols, while theisoParamsvariant outperforms the dense baseline on eight of nine tasks, consistent with the validation\-loss advantage in Table[1](https://arxiv.org/html/2608.15062#S4.T1)\.

#### Early Exit Analysis

As shown in Fig\.[3\(a\)](https://arxiv.org/html/2608.15062#S4.F3.sf1),RecurrentGPTexhibits superior early\-exit performance as an emergent property: at matched inference FLOPsRecurrentGPTexiting with fewer recurrences has better loss than a dense model exiting at an earlier layer\. Fig\.[3\(b\)](https://arxiv.org/html/2608.15062#S4.F3.sf2)extends the analysis across model scales\. Because uniform depth sampling \(Section[3\.4](https://arxiv.org/html/2608.15062#S3.SS4)\) trains all intermediate recurrent states to predict final losses, the model provides a continuous “compute–quality dial”\. This allows a single checkpoint to transition from fast, shallow inference to high\-quality deep inference without re\-training or auxiliary losses\.

In Figure[4](https://arxiv.org/html/2608.15062#S4.F4), we provide a qualitative analysis of how next\-token predictions evolve across recurrence steps\. Our findings reveal thatRecurrentGPTadapts its computational depth to the semantic demands of the prompt: \(1\)Knowledge\-intensivetokens often stabilize early in the recurrence trajectory; \(2\)Reasoning\-basedtasks demonstrate progressive error correction and sharpening of the probability mass over deeper iterations; and \(3\)Open\-ended or creativeprompts exhibit continuous linguistic refinement, where deeper recurrence enhances local coherence despite the lack of a singular ground truth\. This suggests that the model implicitly utilizes the recurrent core to modulate its internal “thinking time” based on task complexity\.

\(a\)Early\-exit Validation loss vs\. equivalent layer depthp\+k⋅n\+cp\+k\\cdot n\+c\.\(b\)Early\-exit Validation loss vs\. inference FLOPs per token across Small, Medium, and Large scales at varying recurrence depthskk\.
Figure 3:Early\-Exit AnalysisInference FLOPs are measured based on sequence length 1024, batch size 1\.Figure 4:Next\-token probability at each recurrence step\.Each row shows one prompt; the correct answer is highlighted in colour\. Prelude\-only output \(leftmost column\) captures partial signal; by step 2 the correct token dominates in three of four prompts\. Step labels use 1\-indexed notation consistent with the analysis section\.
#### KV Cache Sharing

Naively,RecurrentGPTrequiresRRseparate KV caches during decoding—one per recurrence step—multiplying memory byRRover a standard transformer\. Following\[[41](https://arxiv.org/html/2608.15062#bib.bib11)\], we evaluate three compressed strategies that each reduce recurrent KV memory overRRlayers, equivalent to3\.27×3\.27\\timesreduction across the model: reusing only the*last*step’s cache, only the*first*step’s, or an*averaged*K/V across steps\. As shown in Figure[7](https://arxiv.org/html/2608.15062#S5.F7), we observe that*averaged*leads to best accuracy, while*last*leads to the worst accuracy\.

#### Wall\-Clock Latency and Decoding Memory

RecurrentGPTexecutes the same number of block applications as the dense model it isisoFLOPs\-matched to, distributed over fewer unique weight tensors\. Table[3](https://arxiv.org/html/2608.15062#S4.T3)reports measured generation latency and peak GPU memory on identical hardware \(batch size 4, prompt length 1024, 128 generated tokens\)\. Undertorch\.compile,RecurrentGPT\-Large costs\+10%\+10\\%generation latency for63%63\\%fewer parameters and59%59\\%less peak memory; at medium scale the overhead is\+11%\+11\\%\. In eager mode both overheads are\+23%\+23\\%, so roughly half the gap is kernel\-launch overhead from the additional elementwise operations rather than arithmetic\.

Table 3:Generation latency and peak memory \(isoFLOPsconfigurations\)\.Batch size 4, prompt length 1024, 128 generated tokens, same hardware\. Both arms are measured undertorch\.compileand in eager mode\.TheR×R\\times\-expanded KV cache is a real cost, and whether it erodes the parameter saving depends on batch size\. Table[4](https://arxiv.org/html/2608.15062#S4.T4)reports end\-to\-end decoding memory \(weights\+\+KV, bf16,T=1024T=1024\) for the mediumisoFLOPscheckpoint\. AtB=1B\{=\}1decoding is weight\-dominated and the parameter reduction lands at0\.55×0\.55\\timesdense memory even with the naiveR×R\\timescache; atB=32B\{=\}32the cache dominates and the naive strategy recovers only0\.91×0\.91\\times\. Averaging K/V across recurrence steps brings this to0\.39×0\.39\\timeswhile*improving*HellaSwag accuracy over the full cache \(33\.9033\.90vs\.33\.6533\.65\), which suggests the averaging acts as a mild regulariser at this scale\.

Table 4:End\-to\-end decoding memory \(mediumisoFLOPscheckpoint, bf16,T=1024T=1024\)\.Weights\+\+KV cache\. Parenthesised values are relative to GPT\-2 Medium\.
#### Mechanistic Analysis Summary

Detailed mechanistic analysis of the large1\+5×\\times6\+5checkpoint is provided in Appendix[E](https://arxiv.org/html/2608.15062#A5); we summarise the key findings here\. We observe in Figure[10](https://arxiv.org/html/2608.15062#A5.F10)that gate,g\(r\)g^\{\(r\)\}, behavior evolves across recurrence steps: early recurrences \(r≤2r\\leq 2\) are*write\-heavy*\(gate values near00, block output dominates\), while later steps transition to*copy\-heavy*behavior \(gate near11, residual stream preserved\), consistent with a coarse\-to\-fine computation strategy\. Centered Kernel Alignment\[[29](https://arxiv.org/html/2608.15062#bib.bib5)\]analysis in Figure[12](https://arxiv.org/html/2608.15062#A5.F12)confirms that representations across recurrence steps are more similar to one another than representations across layers in a standard transformer of matched depth, suggesting the shared block learns a broadly applicable transformation rather than depth\-specialized ones\. In Figure[13](https://arxiv.org/html/2608.15062#A5.F13), we observe that tokens that are more difficult for the model accumulate10×10\\timesmore improvement across recurrence steps, demonstrating implicit compute routing toward harder inputs without any explicit difficulty signal\.

## 5Ablations

We study two complementary aspects ofRecurrentGPT: the contribution of each architectural component \(Section[5](https://arxiv.org/html/2608.15062#S5.SS0.SSS0.Px1)\) and the model’s behaviour as the training data budget varies \(Section[5](https://arxiv.org/html/2608.15062#S5.SS0.SSS0.Px2)\)\. All ablations use validation loss as the primary metric\.

#### Component Ablation

Figure 5:Component ablation onRecurrentGPT\-small \(20,000 steps\)\.Each bar adds one component to the one on its left\. The dashed line is the dense 12\-layer baseline at3\.1573\.157, trained identically; only the full model falls below it\. The 2,000\-step counterpart on the medium configuration is in Appendix[B\.2](https://arxiv.org/html/2608.15062#A2.SS2)\.Figure[5](https://arxiv.org/html/2608.15062#S5.F5)traces a sequential ablation on the small model at the full 20,000\-step budget\. Recurrence alone raises validation loss by 0\.107 nats: without a mechanism to differentiate steps, the shared block conflates early and late processing into a single weight tensor\. Adding prelude and coda boundaries \(row 3\) recovers 0\.035 nats; state noise \(row 4\) contributes 0\.018 more, which also isolates the value of injecting noise at every step rather than only at initialisation\. Prelude re\-injection into each recurrence \(row 5\) cuts loss by a further 0\.022 nats\. The elementwise gate \(row 6\) is the single largest contributor at−\-0\.048 nats\.

Figure 6:Validation loss vs\. training data budget\.Validation loss vs\. training data budget at small and large scale\.Figure 7:KV cache sharing\.HellaSwag accuracy \(%\) for the largeisoFLOPscheckpoint\.

#### Data Efficiency

As illustrated in Figure[7](https://arxiv.org/html/2608.15062#S5.F7), we analyze the performance of recurrent depth across varying training data budgets\. At smaller model scales,RecurrentGPTconsistently maintains a superior Pareto frontier compared to the dense baseline across all observed data volumes\. At larger scales, a more complex “crossover” dynamic emerges: while the dense model shows an initial lead under low\-data regimes, the gap narrows significantly as the token budget increases\. For the Large\-scale configuration, the trajectories converge at higher data budgets, suggesting that recurrent sharing is particularly effective at saturating parameter capacity when provided with sufficient data\.

## 6Conclusion

RecurrentGPTdemonstrates that a learned per\-element gate conditioning on the current hidden state, the fixed prelude output, and injected stochastic noise is a principled strategy for decoupling model depth from parameter count\. Under anisoFLOPsconstraint,RecurrentGPTstores only3636–37%37\\%of the dense baseline’s unique parameters while beating both MoR\[[3](https://arxiv.org/html/2608.15062#bib.bib12)\]and heavy\-tail Poisson sampling at every scale; under anisoParamsconstraint, deeper recurrence yields consistent validation\-loss gains \(Table 1\) and downstream improvements of up to\+8\.29\+8\.29points \(Table 2\)\. Three limitations bound the current work: the recurrence depthRRis fixed at inference with no per\-token halting\[[20](https://arxiv.org/html/2608.15062#bib.bib9)\]; the gate bias and noise magnitudes may require re\-tuning beyond the GPT\-2 family; and the scale\-dependent optimal sharing fraction warrants systematic study\. Future directions include dynamic per\-token halting and knowledge distillation from a dense language model into a recurrent student\.

## References

- \[1\]J\. L\. Ba, J\. R\. Kiros, and G\. E\. Hinton\(2016\)Layer normalization\.External Links:1607\.06450,[Link](https://arxiv.org/abs/1607.06450)Cited by:[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p5.2)\.
- \[2\]S\. Bae, A\. Fisch, H\. Harutyunyan, Z\. Ji, S\. Kim, and T\. Schuster\(2025\)Relaxed recursive transformers: effective parameter sharing with layer\-wise loRA\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WwpYSOkkCt)Cited by:[Table 8](https://arxiv.org/html/2608.15062#A2.T8.9.2.5),[§1](https://arxiv.org/html/2608.15062#S1.p3.1),[§2](https://arxiv.org/html/2608.15062#S2.p3.1),[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p3.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.15.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.22.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.8.1.1)\.
- \[3\]S\. Bae, Y\. Kim, R\. Bayat, S\. Kim, J\. Ha, T\. Schuster, A\. Fisch, H\. Harutyunyan, Z\. Ji, A\. Courville, and S\. Yun\(2026\)Mixture\-of\-recursions: learning dynamic recursive depths for adaptive token\-level computation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=QuqsEIVWIG)Cited by:[Table 8](https://arxiv.org/html/2608.15062#A2.T8.9.2.3),[§2](https://arxiv.org/html/2608.15062#S2.p4.1),[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p3.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.12.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.19.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.27.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.32.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.5.1.1),[§6](https://arxiv.org/html/2608.15062#S6.p1.1)\.
- \[4\]S\. Bai, J\. Z\. Kolter, and V\. Koltun\(2019\)Deep equilibrium models\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p5.1)\.
- \[5\]S\. Biderman, H\. Schoelkopf, Q\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. van der Wal\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.External Links:2304\.01373,[Link](https://arxiv.org/abs/2304.01373)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p6.1)\.
- \[6\]D\. Bolyaet al\.\(2025\)Depth as modulation in weight\-sharing transformers\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=wm9jRInse3)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p3.1)\.
- \[7\]Y\. Chen, X\. Pan, Y\. Li, B\. Ding, and J\. Zhou\(2024\)EE\-llm: large\-scale training and inference of early\-exit large language models with 3d parallelism\.InThe Forty\-first International Conference on Machine Learning,Cited by:[§3\.4](https://arxiv.org/html/2608.15062#S3.SS4.p1.1)\.
- \[8\]K\. Cho, B\. van Merriënboer, C\. Gulcehre, D\. Bahdanau, F\. Bougares, H\. Schwenk, and Y\. Bengio\(2014\)Learning phrase representations using RNN encoder–decoder for statistical machine translation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),A\. Moschitti, B\. Pang, and W\. Daelemans \(Eds\.\),Doha, Qatar,pp\. 1724–1734\.External Links:[Link](https://aclanthology.org/D14-1179/),[Document](https://dx.doi.org/10.3115/v1/D14-1179)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p1.1),[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p2.1)\.
- \[9\]M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and L\. Kaiser\(2019\)Universal transformers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HyzdRiR9Y7)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p3.1),[§2](https://arxiv.org/html/2608.15062#S2.p2.1)\.
- \[10\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p2.1)\.
- \[11\]M\. Elhoushi, A\. Shrivastava, D\. Liskovich, B\. Hosmer, B\. Wasti, L\. Lai, A\. Mahmoud, B\. Acun, S\. Agarwal, A\. Roman, A\. Aly, B\. Chen, and C\. Wu\(2024\)LayerSkip: enabling early exit inference and self\-speculative decoding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12622–12642\.External Links:[Link](https://aclanthology.org/2024.acl-long.681/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.681)Cited by:[§3\.4](https://arxiv.org/html/2608.15062#S3.SS4.p1.1)\.
- \[12\]J\. L\. Elman\(1990\)Finding structure in time\.Cognitive Science14\(2\),pp\. 179–211\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1207/s15516709cog1402%5F1),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1402_1),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1207/s15516709cog1402\_1Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p1.1),[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p2.1)\.
- \[13\]G\. Feng, B\. Zhang, Y\. Gu, H\. Ye, D\. He, and L\. Wang\(2023\)Towards revealing the mystery behind chain of thought: a theoretical perspective\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=qHrADgAdYu)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p2.1)\.
- \[14\]L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou\(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px2.p4.1)\.
- \[15\]J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. Goldstein\(2025\)Scaling up test\-time compute with latent reasoning: a recurrent depth approach\.External Links:2502\.05171,[Link](https://arxiv.org/abs/2502.05171)Cited by:[Table 8](https://arxiv.org/html/2608.15062#A2.T8.9.2.4),[§1](https://arxiv.org/html/2608.15062#S1.p3.1),[§1](https://arxiv.org/html/2608.15062#S1.p4.1),[§2](https://arxiv.org/html/2608.15062#S2.p2.1),[§2](https://arxiv.org/html/2608.15062#S2.p3.1),[§3\.2](https://arxiv.org/html/2608.15062#S3.SS2.p1.1),[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p3.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.13.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.20.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.28.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.33.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.6.1.1)\.
- \[16\]F\. A\. Gers, J\. Schmidhuber, and F\. Cummins\(2000\)Learning to forget: continual prediction with LSTM\.Neural Computation12\(10\),pp\. 2451–2471\.Cited by:[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p5.2)\.
- \[17\]M\. Geva, R\. Schuster, J\. Berant, and O\. Levy\(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.446/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p3.1)\.
- \[18\]Google DeepMind\(2025\)Gemma 3n model overview\.Note:[https://ai\.google\.dev/gemma/docs/gemma\-3n](https://ai.google.dev/gemma/docs/gemma-3n)Introduced Per\-Layer Embeddings \(PLE\) for on\-device LLMsCited by:[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p6.1)\.
- \[19\]Google DeepMind\(2026\)Gemma 4 model overview\.Note:[https://ai\.google\.dev/gemma/docs/core](https://ai.google.dev/gemma/docs/core)Accessed: 2026\-05\-06\. Last updated 2026\-04\-16Cited by:[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p6.1)\.
- \[20\]A\. Graves\(2017\)Adaptive computation time for recurrent neural networks\.External Links:1603\.08983,[Link](https://arxiv.org/abs/1603.08983)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p4.1),[§6](https://arxiv.org/html/2608.15062#S6.p1.1)\.
- \[21\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p2.1)\.
- \[22\]S\. Hochreiter and J\. Schmidhuber\(1997\)Long short\-term memory\.Neural Computation9\(8\),pp\. 1735–1780\.External Links:[Document](https://dx.doi.org/10.1162/neco.1997.9.8.1735)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p1.1)\.
- \[23\]J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre\(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p6.1)\.
- \[24\]A\. Jeddi, M\. Ciccone, and B\. Taati\(2026\)LoopFormer: elastic\-depth looped transformers for latent reasoning via shortcut modulation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RzYXb5YWBs)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p3.1),[§2](https://arxiv.org/html/2608.15062#S2.p3.1)\.
- \[25\]R\. Jozefowicz, W\. Zaremba, and I\. Sutskever\(2015\)An empirical exploration of recurrent network architectures\.InProceedings of the 32nd International Conference on Machine Learning,pp\. 2342–2350\.Cited by:[§3\.3](https://arxiv.org/html/2608.15062#S3.SS3.p5.2)\.
- \[26\]J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei\(2020\)Scaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p1.1),[§2](https://arxiv.org/html/2608.15062#S2.p6.1)\.
- \[27\]A\. Karpathy\(2022\)NanoGPT\.Note:[https://github\.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)Cited by:[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p1.1)\.
- \[28\]Y\. Koishekenov, A\. Lipani, and N\. Cancedda\(2025\)Encode, think, decode: scaling test\-time reasoning with recursive latent thoughts\.External Links:2510\.07358,[Link](https://arxiv.org/abs/2510.07358)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p2.1),[§2](https://arxiv.org/html/2608.15062#S2.p3.1)\.
- \[29\]S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton\(2019\)Similarity of neural network representations revisited\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 3519–3529\.Cited by:[§E\.4](https://arxiv.org/html/2608.15062#A5.SS4.p1.1),[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px6.p1.1)\.
- \[30\]Z\. Lan, M\. Chen, S\. Goodman, K\. Gimpel, P\. Sharma, and R\. Soricut\(2020\)ALBERT: a lite BERT for self\-supervised learning of language representations\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p3.1),[§2](https://arxiv.org/html/2608.15062#S2.p2.1)\.
- \[31\]Learning to Reason with LLMs\.\(en\-US\)\.External Links:[Link](https://openai.com/index/learning-to-reason-with-llms/)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p2.1)\.
- \[32\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p1.1)\.
- \[33\]S\. M\. McLeish, J\. Kirchenbauer, D\. Y\. Miller, S\. Singh, A\. Bhatele, M\. Goldblum, A\. Panda, and T\. Goldstein\(2026\)Gemstones: a model suite for multi\-faceted scaling laws\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=iZk78dZ1Ap)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p6.1)\.
- \[34\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language models are unsupervised multitask learners\.Cited by:[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p3.1)\.
- \[35\]D\. Raposo, S\. Ritter, B\. Richards, T\. Lillicrap, P\. C\. Humphreys, and A\. Santoro\(2024\)Mixture\-of\-depths: dynamically allocating compute in transformer\-based language models\.External Links:2404\.02258,[Link](https://arxiv.org/abs/2404.02258)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p4.1)\.
- \[36\]D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. Williams\(1986\)Learning representations by back\-propagating errors\.Nature323,pp\. 533–536\.External Links:[Document](https://dx.doi.org/10.1038/323533a0)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p1.1)\.
- \[37\]C\. V\. Snell, J\. Lee, K\. Xu, and A\. Kumar\(2025\)Scaling LLM test\-time compute optimally can be more effective than scaling parameters for reasoning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4FWAwZtd2n)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p2.1)\.
- \[38\]A\. Turing\(1936\)On computable numbers, with an application to the Entscheidungsproblem\.Proceedings of the London Mathematical Society42\(1\),pp\. 230–265\.External Links:[Document](https://dx.doi.org/10.2307/2268810)Cited by:[§1](https://arxiv.org/html/2608.15062#S1.p2.1)\.
- \[39\]P\.J\. Werbos\(1990\)Backpropagation through time: what it does and how to do it\.Proceedings of the IEEE78\(10\),pp\. 1550–1560\.External Links:[Document](https://dx.doi.org/10.1109/5.58337)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p1.1)\.
- \[40\]G\. Yehudai, C\. Sanford, M\. Bechler\-Speicher, O\. Fischer, R\. Gilad\-Bachrach, and A\. Globerson\(2026\)Depth\-width tradeoffs in algorithmic reasoning of graph tasks with transformers\.External Links:2503\.01805,[Link](https://arxiv.org/abs/2503.01805)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p6.1)\.
- \[41\]R\. Zhu, Z\. Wang, K\. Hua, T\. Zhang, Z\. Li, H\. Que, B\. Wei, Z\. Wen, F\. Yin, H\. Xing, L\. Li, J\. Shi, K\. Ma, S\. Li, T\. Kergan, A\. Smith, X\. Qu, M\. Hui, B\. Wu, Q\. Min, H\. Huang, X\. Zhou, W\. Ye, J\. Liu, J\. Yang, Y\. Shi, C\. Lin, E\. Zhao, T\. Cai, G\. Zhang, W\. Huang, Y\. Bengio, and J\. Eshraghian\(2025\)Scaling latent reasoning via looped language models\.External Links:2510\.25741,[Link](https://arxiv.org/abs/2510.25741)Cited by:[Table 8](https://arxiv.org/html/2608.15062#A2.T8.9.2.6),[§1](https://arxiv.org/html/2608.15062#S1.p3.1),[§2](https://arxiv.org/html/2608.15062#S2.p3.1),[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1.p3.1),[§4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.14.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.21.1.1),[Table 1](https://arxiv.org/html/2608.15062#S4.T1.14.7.1.1)\.
- \[42\]R\. Zhu, H\. Zhang, K\. Li, T\. Shi, Y\. Duan, C\. Wang, T\. Zhou, A\. Banerjee, and Z\. Qin\(2025\)Beyond parameters: exploring virtual logic depth for scaling laws\.External Links:2506\.18233,[Link](https://arxiv.org/abs/2506.18233)Cited by:[§2](https://arxiv.org/html/2608.15062#S2.p3.1)\.

## Appendix AImplementation Details

defrecurrentgpt\(x, model, R, r\_min=1\):\# R = max recurrence depthh = token\_embed\(x\) \+ pos\_embed\(x\)\# \[S, d\]forblockinmodel\.prelude\_blocks:h = h \+ block\(h\)h\_pre = h\# h^\(0\) = h^\(pre\): frozen anchorr = Rifnottrainingelseuniform\_sample\(r\_min, R\)\# depth samplingforstepinrange\(r\):\# r in 0,\.\.\.,R\-1eps = sample\_gaussian\(0, sigma\*\*2\)h\_tilde = W\_proj\( cat\(\[h \+ eps, h\_pre\]\) \)\# concat \-\> project to do = B\_shared\(h\_tilde\)\# same weights every stepeps\_g = sample\_gaussian\(0, sigma\_g\*\*2\)g = sigmoid\(f\_gate\(LN\(h\), LN\(h\_pre\)\) / tau \+ eps\_g\)\# gate in \[0,1\]ˆ\{S x d\}h = g \* h \+ \(1 \- g\) \* o\# gated residual updateforblockinmodel\.coda\_blocks:h = h \+ block\(h\)returnlm\_head\(h\)\# \[S, \|V\|\]Listing\\thechapter\.1RecurrentGPTforward pass\.

### A\.1Hyperparameter Table

Table[5](https://arxiv.org/html/2608.15062#A1.T5)lists the full training configuration for every run reported in the main paper\.

Table 5:Training configurations\.pp: prelude blocks;bb: shared blocks;RR: recurrence steps;cc: coda blocks;dd: embedding dimension;hh: attention heads\. Batch tokens = batch\_size×\\timesgradient\_accumulation×\\timessequence\_length×\\timesnum\_GPUs\. All runs use AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95\), weight decay 0\.1, gradient clip 1\.0, bfloat16 mixed precision, cosine LR schedule\.All recurrent runs use gate bias initialisation\+4\+4\(𝐠≈0\.98\\mathbf\{g\}\\approx 0\.98at init\), state noiseσx=0\.1\\sigma\_\{x\}=0\.1, gate noiseσg=0\.1\\sigma\_\{g\}=0\.1, and gate temperatureτ=1\.0\\tau=1\.0\. During training, recurrence depth is sampled uniformly from\{1,…,R\}\\\{1,\\ldots,R\\\}at each training step\.

## Appendix BAdditional Ablations

### B\.1Run\-to\-Run Variance Across Seeds

Table[6](https://arxiv.org/html/2608.15062#A2.T6)reports three independent training runs \(seeds 1337, 42, 123\) of the smallisoFLOPsconfiguration and of its dense counterpart, all at the full 20,000\-step budget\.RecurrentGPTaverages3\.145±0\.0043\.145\\pm 0\.004against3\.188±0\.0563\.188\\pm 0\.056for the dense baseline\.RecurrentGPT’s worst seed \(3\.1483\.148\) still falls below the dense baseline’s best \(3\.1543\.154\)\. Seed 1337 is the run reported throughout the main paper\.

Table 6:Three\-seed validation loss on the small configuration\(20,000 steps\)\. Both models use theisoFLOPssetting of Table[1](https://arxiv.org/html/2608.15062#S4.T1)\. Lower is better\.
### B\.2Component Ablation at Short Horizon

Table[7](https://arxiv.org/html/2608.15062#A2.T7)reports the sequential ablation of Section[5](https://arxiv.org/html/2608.15062#S5.SS0.SSS0.Px1)run on the*medium*configuration at 2,000 steps\. At this horizon the ordering of the last two rows is inverted relative to the full 20,000\-step run: prelude re\-injection contributes−0\.198\-0\.198nats against the gate’s−0\.115\-0\.115\. We report both because the comparison is itself informative — the structural components land early, whereas the gate is a learned mechanism whose contribution accrues over training\.

Table 7:Component ablation onRecurrentGPT\-medium at 2 000 steps\.Short\-horizon counterpart to Figure[5](https://arxiv.org/html/2608.15062#S5.F5)\. The dense baseline \(row 1\) is a standard 24\-layer GPT trained identically\.
### B\.3Design Comparison of Recurrent\-Depth Methods

Table[8](https://arxiv.org/html/2608.15062#A2.T8)sets the small\-scale results of Table[1](https://arxiv.org/html/2608.15062#S4.T1)beside the design choices that produce them\. MoR and Ouro gate the update, but Ouro supervises every iteration and MoR needs a router; heavy\-tail Poisson shares strictly and updates unconditionally; RRT is input\-independent and depth\-fixed\.RecurrentGPTis the only column combining full sharing, an input\-dependent update, and variable inference depth\.

Table 8:Design properties of recurrent\-depth methods, with small\-scaleisoFLOPsresults for reference\. Training times are wall\-clock for the small configuration at 20,000 steps on identical hardware, and are*not*comparable against the dense column: the dense baseline was trained withtorch\.compileenabled, whereas every recurrent method including our own ran with compilation disabled, which did not work on our hardware\. The dense column is therefore optimistic; the recurrent columns remain comparable to one another\.
### B\.4Gate Temperature Sensitivity

The gate temperatureτ\\tau\(Eq\.[5](https://arxiv.org/html/2608.15062#S3.E5)\) controls the sharpness of the sigmoid\. We sweepτ∈\{0\.5,1\.0,2\.0\}\\tau\\in\\\{0\.5,1\.0,2\.0\\\}on the medium configuration at 5,000 steps\. Validation loss is 3\.62, 3\.60, and 3\.63 respectively, indicating that the optimal gate temperature isτ=1\.0\\tau=1\.0\.

### B\.5Noise Magnitude Sensitivity

We ablate state noiseσx∈\{0\.0,0\.05,0\.1,0\.2\}\\sigma\_\{x\}\\in\\\{0\.0,0\.05,0\.1,0\.2\\\}on the medium configuration at 5,000 steps\. Removing noise entirely \(σx=0\.0\\sigma\_\{x\}=0\.0\) increases validation loss by 0\.031 nats relative toσx=0\.1\\sigma\_\{x\}=0\.1; larger noise \(σx=0\.2\\sigma\_\{x\}=0\.2\) degrades by 0\.018 nats\. The optimal range isσx∈\[0\.05,0\.1\]\\sigma\_\{x\}\\in\[0\.05,0\.1\]; we use 0\.1 throughout\.

### B\.6Gate Bias and State Noise at Full Training Horizon

The sweeps above are at 5,000 steps\. We repeated both at the full 20,000\-step budget on the small configuration to check that the response surface does not sharpen at convergence\. It does not\. Moving the gate bias from the default\+4\+4down to00costs0\.0190\.019nats, and−2\-2recovers to3\.1513\.151; the surface is shallow and non\-monotonic rather than peaked, and no setting produced a loss spike or divergence\. State noise behaves the same way: removing it costs0\.0180\.018nats and doubling it costs0\.0190\.019, a symmetric optimum rather than a value that has to be hit precisely\. This is weaker evidence than a cross\-architecture study, but it does suggest the mechanism does not depend on the exact bias value\.

Table 9:Gate\-bias sensitivity, small config at 20,000 steps\.
Table 10:State\-noise sensitivity, small config at 20,000 steps\.

## Appendix CAdditional Qualitative Results

### C\.1Per\-Token Gate Activations Across Recurrence Steps

Figure[8](https://arxiv.org/html/2608.15062#A3.F8)showsg¯t\(r\)=meand\(gt,:\(r\)\)\\bar\{g\}^\{\(r\)\}\_\{t\}=\\mathrm\{mean\}\_\{d\}\(g^\{\(r\)\}\_\{t,:\}\)— the elementwise gate averaged over the model dimension — for each token positionttand recurrence steprr, on four example prompts\. Brighter \(yellow\-green\) cells indicate write\-heavy positions where the gate is open \(g¯≈0\.70\\bar\{g\}\\approx 0\.70–0\.820\.82\) and the shared block’s proposal is substantially absorbed; darker \(purple\) cells indicate copy\-heavy positions \(g¯≈0\.95\\bar\{g\}\\approx 0\.95–1\.001\.00\) where the hidden state passes through largely unchanged\.

Three consistent observations emerge across all four prompts\. First,the per\-step mean gate rises monotonically from step 2 onward: means fall from∼\\sim0\.87–0\.89 at step 1 to a minimum around step 2 \(write\-heavy phase\), then recover steadily toward∼\\sim0\.88–0\.91 at step 6 \(copy\-heavy phase\)\. Second,write\-heavy positions are structurally consistent across steps: the same columns that light up at step 1 tend to be the same at step 6, indicating that token identity — not recurrence depth — drives the gate primary signal, with the gate progressively dimming those writes over steps rather than shifting which tokens are written\. Third,content words and syntactically load\-bearing tokens receive lower gate values than function words: in “The old man sat by the”,*man*and*sat*are visibly brighter than*the*and*by*; in the code prompt, the operator and identifier tokens show write\-heavy behaviour while keywords likedefandforare copy\-heavy\. This implicit routing — without any explicit per\-token mechanism — arises purely from the gate conditioning on both the evolving hidden statex\(r\)x^\{\(r\)\}and the fixed prelude anchorhh\.

![Refer to caption](https://arxiv.org/html/2608.15062v1/fig_gate_token_heatmap.png)Figure 8:Per\-token gate activationg¯t\(r\)\\bar\{g\}^\{\(r\)\}\_\{t\}across recurrence steps\.Each row of cells corresponds to one recurrence step \(columns = token positions\)\. Colour encodesg¯t\(r\)∈\[0\.70,1\.00\]\\bar\{g\}^\{\(r\)\}\_\{t\}\\in\[0\.70,1\.00\]: bright yellow\-green = write\-heavy \(gate open, block output substantially absorbed\); dark purple = copy\-heavy \(hidden state passed through unchanged\)\. Token labels are shown below each step column\. The per\-step mean \(annotated top\-right of each panel\) rises from step 2 onward, reflecting the write\-to\-copy transition identified in Figure[10](https://arxiv.org/html/2608.15062#A5.F10)\(b\)\. Structurally informative tokens \(content words, operators\) consistently receive lower gate values than function words and punctuation across all steps\.

## Appendix DBroader Impact

Parameter\-efficient language models have direct environmental and accessibility benefits: aRecurrentGPTcheckpoint at 37% of a baseline’s parameter count requires proportionally less GPU memory at serving time, enabling deployment on hardware that cannot accommodate the full dense model and reducing energy consumption per inference step\. Recurrent depth also provides a natural compute\-quality tradeoff at inference time—the early\-exit capability of Section[4](https://arxiv.org/html/2608.15062#S4.SS0.SSS0.Px1)allows a single trained model to serve requests at multiple quality\-FLOPs operating points without retraining\.

The risks associated withRecurrentGPTare not specific to its architecture: as a capable language model trained on web text, it inherits the standard dual\-use risks of any such system, including the potential to generate misinformation or harmful content\. These risks are addressed by standard deployment safeguards \(content filtering, output monitoring, and responsible release practices\) that apply equally to all current language models; the recurrent architecture does not introduce qualitatively new threat vectors\.

## Appendix ERecurrence Dynamics Analysis

Beyond aggregate validation loss, we probe*what*the shared block computes at each recurrence stepr∈\{1,…,6\}r\\in\\\{1,\\ldots,6\\\}using the large1\+5×\\times6\+5checkpoint\. Diagnostics are run on a validation split with inference noise disabled\. We report ten interlocking findings—grouped first into behavioural diagnostics \(Section[E\.1](https://arxiv.org/html/2608.15062#A5.SS1)–Section[E\.6](https://arxiv.org/html/2608.15062#A5.SS6)\) and then into mechanistic analyses of weights and attribution \(Section[E\.7](https://arxiv.org/html/2608.15062#A5.SS7)–Section[E\.10](https://arxiv.org/html/2608.15062#A5.SS10)\)—before drawing them together in Section[E\.11](https://arxiv.org/html/2608.15062#A5.SS11)\.

### E\.1Rapid Convergence of Loss Across Recurrence Steps

Figure[9](https://arxiv.org/html/2608.15062#A5.F9)\(a\) traces the per\-step validation loss as the shared block is applied iteratively\. The prelude output alone yields a loss of5\.295\.29; after a single application of the shared core the loss drops to3\.773\.77—a reduction of1\.521\.52nats \(natural\-log\-scale bits; 1 nat=log2⁡e≈1\.44=\\log\_\{2\}e\\approx 1\.44bits\) in one step\. By step 2 the loss reaches3\.143\.14, and the remaining four steps account for only an additional0\.460\.46nats, converging to the final value of2\.682\.68\. Stated differently, the first two recurrence steps account for roughly 77% of the total loss reduction from the prelude representation to the final output\.

This front\-loaded profile is confirmed by the KL\-divergence trajectory in Figure[9](https://arxiv.org/html/2608.15062#A5.F9)\(b\), which measures residual uncertainty relative to the final\-step output distribution\. The KL falls from2\.612\.61at the prelude to0\.460\.46by step 2 and reaches0\.0140\.014at step 5, indicating that the output distribution is essentially committed after the fourth step\. As a lower bound on the gate’s utility, forcing the gate to zero throughout \(the “no\-recurrence” ablation\) yields a catastrophic loss of12\.0312\.03, confirming that the iterative refinements are not redundant\. The complementary ablation—forcing the gate to one throughout, so that every block proposal is discarded and the hidden state is never updated—yields a loss of5\.265\.26, equivalent to the prelude\-only output \(5\.265\.26at stepr=0r=0\)\. This confirms that recurrence with a fully open gate is vacuous: without selective writing, six recurrence steps reduce to a single prelude pass\. Together, Gate=0 \(12\.0312\.03\) and Gate=1 \(5\.265\.26\) bracket the trained model \(2\.682\.68\): the learned gate neither skips recurrence nor blindly overwrites state, but acquires a fine\-grained elementwise balance between retention and update\.

We also evaluate extending the recurrence depth at inference time beyond theR=6R=6steps seen during training\. Applying the trained shared block forR∈\{8,10,12\}R\\in\\\{8,10,12\\\}steps marginally*degrades*performance \(2\.692\.69,2\.722\.72,2\.742\.74respectively\), suggesting the model has converged representationally by step 6 and that additional steps introduce noise rather than refinement\.

Figure 9:Loss convergence across recurrence steps\.\(a\) Validation loss at each recurrence steprrfor the trained1\+5×\\times6\+5model\. The “Gate=1” baseline \(loss5\.265\.26, green dashed\) equals the prelude\-only output, confirming that selective writing is essential: a fully open gate discards every block proposal, leaving the hidden state unchanged across all six steps\. The “Gate=0” baseline \(loss12\.0312\.03\) confirms that recurrence is essential, not a skip connection\. Extended inference atR∈\{8,10,12\}R\\in\\\{8,10,12\\\}does not improve beyondR=6R=6\(2\.692\.69,2\.722\.72, and2\.742\.74respectively\), indicating representational convergence\. \(b\) KL divergence of each step’s output distribution to the final\-step distribution; 95% of the residual uncertainty is resolved within four recurrence steps\.Takeaway77% of the loss reduction from prelude to final output occurs within the first two recurrence steps; the remaining four steps provide targeted corrections\. The model’s output distribution is essentially committed by step 4\.

### E\.2Gate Behaviour: Write\-Heavy Early, Copy\-Heavy Late

The elementwise gate𝐠\(r\)∈\[0,1\]S×d\\mathbf\{g\}^\{\(r\)\}\\in\[0,1\]^\{S\\times d\}controls how much of the shared block’s proposal is written into the residual stream at each step\. Figure[10](https://arxiv.org/html/2608.15062#A5.F10)\(a\) plots the per\-step mean and standard deviation\. The gate is most open—and most variable—in the middle steps \(steps 2–3, mean≈0\.82\\approx 0\.82, std≈0\.18\\approx 0\.18\), corresponding to the highest information\-writing regime, and tightens progressively toward step 6 \(mean0\.870\.87, std0\.120\.12\)\.

Figure[10](https://arxiv.org/html/2608.15062#A5.F10)\(b\) quantifies this transition via the fraction of dimensions where the gate is near\-saturated\. Copy\-saturated dimensions \(g\>0\.95g\>0\.95\) grow monotonically from 19\.7% at step 1 to 28\.8% at step 6, while write\-saturated dimensions \(g<0\.05g<0\.05\) remain negligibly rare \(<10−4<10^\{\-4\}\) throughout\. This asymmetry—abundant copying, scarce full\-overwriting—indicates the model prefers selective blending over clean state replacement\.

The effective gate openness \(Figure[10](https://arxiv.org/html/2608.15062#A5.F10)\(c\)\), measured as the ratio

ρ\(r\)=‖𝐡\(r\)−𝐡\(r−1\)‖2‖𝐨\(r\)−𝐡\(r−1\)‖2=‖\(1−𝐠\(r\)\)⊙\(𝐨\(r\)−𝐡\(r−1\)\)‖2‖𝐨\(r\)−𝐡\(r−1\)‖2,\\rho^\{\(r\)\}=\\frac\{\\\|\\mathbf\{h\}^\{\(r\)\}\-\\mathbf\{h\}^\{\(r\-1\)\}\\\|\_\{2\}\}\{\\\|\\mathbf\{o\}^\{\(r\)\}\-\\mathbf\{h\}^\{\(r\-1\)\}\\\|\_\{2\}\}=\\frac\{\\\|\(1\-\\mathbf\{g\}^\{\(r\)\}\)\\odot\(\\mathbf\{o\}^\{\(r\)\}\-\\mathbf\{h\}^\{\(r\-1\)\}\)\\\|\_\{2\}\}\{\\\|\\mathbf\{o\}^\{\(r\)\}\-\\mathbf\{h\}^\{\(r\-1\)\}\\\|\_\{2\}\},\(6\)where the numerator is the norm of the*applied update*\(actual change to the hidden state\) and the denominator is the norm of the*proposal*\(the full update the block would write if the gate were fully open\), decreases monotonically from0\.1820\.182at step 1 to0\.0660\.066at step 6\. Taken together, the gate transitions the model from a*write\-heavy*regime in early steps—where large fractions of the proposal are incorporated to rapidly restructure the representation—to a*copy\-heavy*regime in later steps, where the representation is largely preserved and only small targeted corrections are applied\.

Figure 10:Gate statistics across recurrence steps\.\(a\) Gate mean and standard deviation\. \(b\) Fraction of copy\-saturated \(g\>0\.95g\>0\.95\) and write\-saturated \(g<0\.05g<0\.05, scaled×1000\\times 1000\) dimensions per step\. Copy\-saturation grows steadily while write\-saturation stays negligible\. \(c\) Effective gate openness \(applied/proposalL2L\_\{2\}ratio\), declining monotonically—early steps write substantially, late steps refine conservatively\.TakeawayThe gate transitions from a write\-heavy regime \(steps 1–2, mean≈0\.82\\approx 0\.82\) to a copy\-heavy regime \(steps 3–6, mean≈0\.87\\approx 0\.87\), with effective gate openness declining monotonically from0\.180\.18to0\.070\.07\.

### E\.3Update Dynamics: Diminishing Magnitude, Increasing Alignment

Figure[11](https://arxiv.org/html/2608.15062#A5.F11)\(a\) separates two quantities at each step: theL2L\_\{2\}norm of the shared block’s raw proposal𝐨\(r\)\\mathbf\{o\}^\{\(r\)\}, and theL2L\_\{2\}norm of the gate\-applied update that actually enters the residual stream\. The proposal norms are nearly constant across all six steps \(9393–9999\), possibly indicating that the shared block does not itself “know” it is being applied repeatedly—its output magnitude is stationary\. The applied update, by contrast, declines from17\.617\.6at step 1 to6\.56\.5at step 6, driven entirely by the gate becoming more closed\.

Figure[11](https://arxiv.org/html/2608.15062#A5.F11)\(b\) shows cosine similarity between successive representations,cos\_sim​\(𝐡r,𝐡r−1\)=𝐡r⋅𝐡r−1‖𝐡r‖​‖𝐡r−1‖\\text\{cos\\\_sim\}\(\\mathbf\{h\}^\{r\},\\mathbf\{h\}^\{r\-1\}\)=\\frac\{\\mathbf\{h\}^\{r\}\\cdot\\mathbf\{h\}^\{r\-1\}\}\{\\\|\\mathbf\{h\}^\{r\}\\\|\\\|\\mathbf\{h\}^\{r\-1\}\\\|\}\. The proposal cosine similarity grows from0\.190\.19at step 1 to0\.510\.51at step 6, indicating that the block and the hidden state increasingly agree on direction as the representation stabilises\. The applied update cosine similarity is already high at step 1 \(0\.940\.94\) and approaches0\.990\.99by step 6, confirming that late\-step corrections are directionally consistent refinements\.

The per\-step gain estimate \(Figure[11](https://arxiv.org/html/2608.15062#A5.F11)\(c\)\) peaks at step 2 \(10\.410\.4\), declining to4\.74\.7at step 6\. This concave gain profile, in which the most productive computation occurs before representational alignment is achieved, is consistent with the loss convergence plots in Section[E\.1](https://arxiv.org/html/2608.15062#A5.SS1)\.

Figure 11:Update dynamics across recurrence steps\.\(a\)L2L\_\{2\}norms of the block’s raw proposal \(near\-constant,≈94\\approx 94\) and the gate\-applied update \(declining17\.6→6\.517\.6\\to 6\.5\); the divergence is caused entirely by gate closure\. \(b\) Cosine similarity between successive representations\. \(c\) Local step gain estimate, peaking at step 2 and declining thereafter\.
### E\.4Representational Geometry: Two\-Phase Structure

We use Centered Kernel Alignment\[[29](https://arxiv.org/html/2608.15062#bib.bib5), CKA;\]to compare hidden\-state representations across recurrence steps and against the corresponding layer representations of the GPT\-2 Large dense baseline\.

#### Within\-model CKA

\(Figure[12](https://arxiv.org/html/2608.15062#A5.F12)\(a\)\)\. The7×77\\times 7CKA matrix reveals a two\-phase structure\. Representations at step 0 \(prelude\) and step 1 are nearly identical \(CKA=0\.996=0\.996\), while both are sharply dissimilar from all subsequent steps \(CKA≈0\.33\\approx 0\.33–0\.650\.65with steps≥2\\geq 2\)\. Steps 2 through 6 form a tightly cohesive cluster \(pairwise CKA≥0\.907\\geq 0\.907\), within which similarity decreases only gradually with step distance\. This suggests the model undergoes a*representational phase transition*between steps 1 and 2\.

#### Cross\-model CKA

\(Figure[12](https://arxiv.org/html/2608.15062#A5.F12)\(b\)\)\. Prelude and step 1 representations align maximally with the shallow layers of the baseline \(peak at layer 2, CKA=0\.993=0\.993\), while steps 2 through 6 all peak at baseline layer 11 \(CKA=0\.97=0\.97–0\.900\.90declining\)\. No recurrent step aligns strongly with the deep layers of the baseline \(layers\>20\>20, CKA<0\.50<0\.50\), confirming that recurrent depth compresses the computational path but does not replicate the later representational hierarchy of the dense model\.

![Refer to caption](https://arxiv.org/html/2608.15062v1/fig_cka.png)Figure 12:CKA representational geometry\.\(a\) Within\-model CKA; the sharp discontinuity between steps 1–2 and steps 3–6 \(dashed lines\) reveals a two\-phase structure\. \(b\) Cross\-model CKA against GPT\-2 Large; early steps map to shallow baseline layers, all later steps peak at layer≈11\\approx 11\.TakeawayA representational phase transition occurs between steps 1 and 2: early steps resemble shallow baseline layers, while steps 2–6 all align with mid\-depth baseline layer 11, compressing the computational path without replicating the deep\-layer hierarchy\.

### E\.5Differential Routing by Token Difficulty

To test whether the gate allocates computation to where it is most needed, we sort tokens by their loss after the prelude \(step 0\) into ten deciles and track the total loss improvement accumulated over six recurrence steps\. Figure[13](https://arxiv.org/html/2608.15062#A5.F13)\(a\) shows a near\-linear relationship between initial token difficulty and total improvement \(R2=0\.998R^\{2\}=0\.998\): the easiest decile gains only0\.490\.49nats, while the hardest decile gains5\.025\.02nats—a ten\-fold difference\. The gate difficulty correlation—whether tokens with higher loss also receive lower \(more open\) gate values—is mildly negative across all steps \(−0\.07\-0\.07to−0\.04\-0\.04\), consistent with the gate allowing slightly more writing for uncertain tokens\.

![Refer to caption](https://arxiv.org/html/2608.15062v1/fig_token_difficulty.png)Figure 13:Token difficulty routing\.\(a\) Total improvement over six steps vs\. initial difficulty decile \(R2=0\.998R^\{2\}=0\.998\); recurrence implicitly allocates more work to uncertain tokens\. \(b\) Per\-step loss curves for selected deciles\.
### E\.6Fixed\-Point Convergence and Effective Rank

Figure[14](https://arxiv.org/html/2608.15062#A5.F14)\(a\) tracks the effective rank of the hidden\-state tensor at each step\. The rank increases from0\.5460\.546at the prelude to0\.6350\.635by step 3, after which it plateaus, indicating that iterative updates progressively spread information across more hidden dimensions\. The effective rank of the*delta subspace*—the directions along which the hidden state changes in later steps—is0\.690\.69, higher than the rank of the representation itself, suggesting that late\-step updates refine a diverse set of features\.

Figure[14](https://arxiv.org/html/2608.15062#A5.F14)\(b\) plots the normalised fixed\-point convergence metric, defined as

δfp\(r\)=‖𝐡\(r\)−𝐡\(r−1\)‖F‖𝐡\(r−1\)‖F,\\delta\_\{\\text\{fp\}\}^\{\(r\)\}=\\frac\{\\\|\\mathbf\{h\}^\{\(r\)\}\-\\mathbf\{h\}^\{\(r\-1\)\}\\\|\_\{F\}\}\{\\\|\\mathbf\{h\}^\{\(r\-1\)\}\\\|\_\{F\}\},\(7\)the Frobenius\-norm magnitude of the update relative to the current representation, which declines monotonically from0\.3240\.324at step 1 to0\.1120\.112at step 6, confirming that the recurrent dynamics are contractive\.

Table[11](https://arxiv.org/html/2608.15062#A5.T11)reports a sanity check via anchor ablations—swapping the fixed prelude output𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}fed at each recurrence step for alternative anchors while keeping the gate and blocks identical\. Crucially, the “previous hidden state” row uses𝐡\(r−1\)\\mathbf\{h\}^\{\(r\-1\)\}as anchor \(i\.e\. the anchor drifts with the recurrence rather than being held fixed\), while the learned gate remains fully active and adaptive\. The0\.700\.70\-nat gap between this condition \(3\.383\.38\) and the trained model \(2\.682\.68\) therefore quantifies the value of providing a*stable, fixed reference point*at every step, not merely the value of the gate\.

Table 11:Anchor ablation: effect of replacing𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}with alternative anchors at each recurrence step\. The “prev\. hidden state” anchor drifts at each step \(𝐡\(r−1\)\\mathbf\{h\}^\{\(r\-1\)\}\) while the learned adaptive gate remains active throughout\. The0\.700\.70\-nat gap between that condition and the trained model isolates the value of holding the prelude output*fixed*as a stable context reference\.Figure 14:Effective rank and fixed\-point convergence\.\(a\) Effective rank grows with depth; the late\-delta subspace rank \(dashed\) is higher than the representation itself\. \(b\) Fixed\-point convergence \(orange\) declines monotonically;Δ​h\\Delta hcosine similarity \(blue\) increases—both consistent with convergent refinement\.
### E\.7Gate Attribution: Current State Gradually Supersedes the Anchor

The elementwise gate \(Eq\.[5](https://arxiv.org/html/2608.15062#S3.E5)\) reads from two sources simultaneously:𝐱^\(r\)≜LN⁡\(𝐡\(r−1\)\)\\hat\{\\mathbf\{x\}\}^\{\(r\)\}\\triangleq\\mathrm\{LN\}\(\\mathbf\{h\}^\{\(r\-1\)\}\), the LayerNorm\-normalised current hidden state,111We write𝐱^\(r\)\\hat\{\\mathbf\{x\}\}^\{\(r\)\}rather thanxxto avoid confusion with token IDs\(x1,…,xS\)\(x\_\{1\},\\ldots,x\_\{S\}\)introduced in the Preliminaries\.and𝐡^≜LN⁡\(𝐡\(pre\)\)\\hat\{\\mathbf\{h\}\}\\triangleq\\mathrm\{LN\}\(\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}\), the normalised prelude anchor\. The gate MLPf𝐠f\_\{\\mathbf\{g\}\}takes their concatenation as input; we write its first linear layer weight asW𝐠=\[Wx∣Wh\]∈ℝdgate×2​dW\_\{\\mathbf\{g\}\}=\[W\_\{x\}\\mid W\_\{h\}\]\\in\\mathbb\{R\}^\{d\_\{\\text\{gate\}\}\\times 2d\}, whereWx∈ℝdgate×dW\_\{x\}\\in\\mathbb\{R\}^\{d\_\{\\text\{gate\}\}\\times d\}acts on𝐱^\(r\)\\hat\{\\mathbf\{x\}\}^\{\(r\)\}andWh∈ℝdgate×dW\_\{h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{gate\}\}\\times d\}acts on𝐡^\\hat\{\\mathbf\{h\}\}\. To quantify which source drives the gate at each step, we measure how much variance in the actual per\-token gate logit each half explains\. As seen in Figure[15](https://arxiv.org/html/2608.15062#A5.F15)\(a\), at step 1 the split is nearly even: 59% current state, 41% anchor\. By step 6 this shifts substantially—74% current state, 26% anchor\.

Figure[15](https://arxiv.org/html/2608.15062#A5.F15)\(b\) shows the same trend at the token level: the fraction of positions where the state half produces a larger absolute gate logit than the anchor grows from a near\-zero 0\.2% at step 1 to 86% at step 6\. The answer to*why*is present in the weights before any forward pass: we find in Figure[15](https://arxiv.org/html/2608.15062#A5.F15)\(c\) that the leading singular value ofWxW\_\{x\}is 9\.87 versus 6\.85 forWhW\_\{h\}, giving the state half structurally greater sensitivity\. At step 1 the hidden state is close to the prelude output, so the gap is small; asx\(r\)x^\{\(r\)\}diverges fromhhacross steps, this structural asymmetry is amplified into the monotone attribution shift in Figure[15](https://arxiv.org/html/2608.15062#A5.F15)\(a\)\.

Figure 15:Gate attribution across recurrence steps\.\(a\) Fraction of gate logit variance from current statexx\(blue\) vs\. frozen anchorhh\(orange\); the crossover from rough parity at step 1 to 74%/26% at step 6 is explained by growing divergence betweenx\(r\)x^\{\(r\)\}andhh\. \(b\) Token\-levelxx\-dominance andx/hx/hsign\-agreement fractions\. \(c\) Gate weight singular value spectra; the 44% gap in leading singular values \(9\.879\.87vs6\.856\.85\) is the structural origin of the state’s eventual dominance\.TakeawayBy step 6, the gate reads 74% of its signal from the current hidden state and only 26% from the fixed prelude anchor\. This shift is structurally predetermined:WxW\_\{x\}has a 44% larger leading singular value thanWhW\_\{h\}, priming the gate to amplify state\-driven signals as𝐡\(r\)\\mathbf\{h\}^\{\(r\)\}departs from𝐡\(pre\)\\mathbf\{h\}^\{\(\\mathrm\{pre\}\)\}\.

### E\.8Projection Weight Analysis: State and Anchor as Complements

Before the shared blocks run, the recurrent projection,WprojW\_\{\\text\{proj\}\}, maps the concatenated\[x\(r\),h\]\[x^\{\(r\)\},\\,h\]into thedd\-dimensional input the transformer sees\. A natural question is whether this projection simply averages the two sources or treats them as carrying distinct information\. The answer is unambiguous: the mean row\-wise cosine similarity between theWxW\_\{x\}andWhW\_\{h\}halves is−0\.189\\mathbf\{\-0\.189\}, with only0\.16%0\.16\\%of the 1024 rows aligned above a cosine of0\.50\.5\.

Both halves are nearly full\-rank \(effective ranks 1002\.9 and 1004\.3, Frobenius norms 32\.7 and 23\.6\), with flat, slowly\-decaying singular value spectra \(Figure[16](https://arxiv.org/html/2608.15062#A5.F16)\(a\)–\(b\)\)\. The negative mean row cosine means the projection reads the*difference*between current state and anchor rather than their average; we call this the*contrastive projection*property\. Directions large in bothx\(r\)x^\{\(r\)\}andhh—stable features unchanged since the prelude—partially cancel; directions large only inx\(r\)x^\{\(r\)\}are amplified\. This provides a direct mechanistic explanation for why the fixed prelude is helpful: it supplies the reference needed to compute what has changed\.

Figure 16:Recurrent projection weight analysis\.\(a\) Singular value spectra ofWxW\_\{x\}andWhW\_\{h\}; both are nearly full\-rank\. \(b\) Cumulative energy; both halves reach 90% at similar dimensionalities\. \(c\) Frobenius norms of all four weight halves; the state half \(WxW\_\{x\}\) consistently outweighs the anchor half \(WhW\_\{h\}\) in both modules\.
### E\.9Attention Entropy: Stable Head Specialisations, Gradual Sharpening

Because identical weights govern every recurrence step, the shared block cannot self\-modulate its attention patterns—any change must arise purely from changes in𝐡\(r\)\\mathbf\{h\}^\{\(r\)\}\. In Figure[17](https://arxiv.org/html/2608.15062#A5.F17)\(a\), we capture the full\[B,nhead,S,S\]\[B,n\_\{\\text\{head\}\},S,S\]attention tensors at each step \(whereBBis batch size andnheadn\_\{\\text\{head\}\}the number of attention heads\) across all five shared blocks \(100 heads total\) and measure per\-head Shannon entropy and diagonal mass\.

#### Aggregate trend\.

Mean attention entropy declines from 3\.00 nats at step 1 to 2\.77 at step 4, then*rebounds*to 2\.82 at steps 5–6 \(Figure[17](https://arxiv.org/html/2608.15062#A5.F17)\(b\)\)\. This non\-monotone profile mirrors the two\-phase CKA structure from Section[E\.4](https://arxiv.org/html/2608.15062#A5.SS4): steps 1–4 are the refinement phase \(lower entropy, more peaked patterns\), while the rebound at steps 5–6 aligns with the final convergence plateau where small corrective updates scan broadly\. Mean diagonal mass peaks at step 4 \(0\.108\) then drops to 0\.092 at step 6\.

#### Head role stability\.

The full6×1006\\times 100entropy heatmap \(Figure[17](https://arxiv.org/html/2608.15062#A5.F17)\(a\)\) makes the dominant pattern immediately visible: individual heads retain their character across all six steps\. Of 100 heads, 2 are local/sharp \(entropy<1\.0<1\.0\), 3 are previous\-token \(prev\-mass\>0\.35\>0\.35\), 2 are self\-attention \(diagonal mass\>0\.35\>0\.35\), 7 are broad broadcast \(entropy\>4\.0\>4\.0\), and 86 are general\. Here,*prev\-mass*denotes the fraction of total attention weight placed on the immediately preceding token position \(i\.e\. diagonal offset−1\-1of the attention matrix\), averaged over the sequence\. A head with prev\-mass\>0\.35\>0\.35predominantly attends to the token immediately before each query position\. The clearest specialist—Block 0, Head 14—has entropy0\.2980\.298and prev\-mass0\.8780\.878at stepr=1r=1,

![Refer to caption](https://arxiv.org/html/2608.15062v1/fig_attn_entropy.png)Figure 17:Attention entropy across recurrence steps\.\(a\) Full6×1006\\times 100entropy heatmap; white lines separate blocks\.⋆\\star: Block 0 Head 14 \(previous\-token specialist, entropy 0\.30\)\.△\\triangle: Block 0 Head 4 \(broadcast head, entropy 4\.87\)\. Head roles are highly stable\. \(b\) Mean entropy \(blue\) and diagonal mass \(orange\) per step; non\-monotone entropy profile mirrors the two\-phase CKA structure\. \(c\) Head type classification across all 100 heads\.TakeawayHead specialisations are locked in by weight and do not adapt to recurrence depth\. What changes across steps is only the magnitude of each head’s contribution, modulated by the gated update to the hidden state\.

### E\.10Recurrence Preferentially Serves Two Token Regimes

We split tokens into four BPE frequency bands: byte/punctuation \(IDs 0–255,n=5,763n=5\{,\}763\), common \(IDs 256–2,000,n=15,789n=15\{,\}789\), medium \(IDs 2,001–10,000,n=6,778n=6\{,\}778\), and rare \(IDs\>10,000\>10\{,\}000,n=4,438n=4\{,\}438\), wherennrefers to the total occurrences of corresponding token id ranges in the validation set\. Figure[18](https://arxiv.org/html/2608.15062#A5.F18)\(a\) shows that all four follow the familiar front\-loaded loss\-reduction profile, but at very different absolute scales: byte/punctuation tokens improve by just 1\.92 nats while rare tokens improve by 3\.39 nats\.

Figure[18](https://arxiv.org/html/2608.15062#A5.F18)\(b\) plots the per\-step relative improvementΔrrel=\(L\(r−1\)−L\(r\)\)/L\(r−1\)\\Delta\_\{r\}^\{\\mathrm\{rel\}\}=\(L^\{\(r\-1\)\}\-L^\{\(r\)\}\)/L^\{\(r\-1\)\}, i\.e\. the fraction of the*preceding step’s*loss resolved at each step\. At stepr=1r=1, byte/punctuation tokens already resolve 49% of their previous loss, compared to 23–28% for the other bands—recurrence is disproportionately productive for easy tokens first\. Gains diminish uniformly beyond step 3 \(<4<4% for all bands\), consistent with the fixed\-point convergence of Section[E\.6](https://arxiv.org/html/2608.15062#A5.SS6)\. Cumulatively over all six steps, byte/punctuation achieves 67% recovery of its initial loss versus 42% for rare tokens\. We call these two roles*completion*\(sharpening already\-likely token predictions\) and*lexical narrowing*\(progressively concentrating probability mass for hard tokens\)\. Both operate simultaneously in the same forward pass, possible because the elementwise gate allocates computation per\-position rather than globally—no explicit routing mechanism is required\.

Figure 18:Token frequency class breakdown\.\(a\) Per\-step loss by BPE frequency band; rare tokens maintain persistently higher loss throughout\. \(b\) Per\-step relative improvement\(L\(r−1\)−L\(r\)\)/L\(r−1\)\(L^\{\(r\-1\)\}\-L^\{\(r\)\}\)/L^\{\(r\-1\)\}per frequency class; the first recurrence step alone resolves 47% of byte/punctuation loss vs\. 23% for rare tokens—easy tokens converge fastest, hard tokens accumulate improvements gradually\. \(c\) Initial loss vs\. total improvement; the two axes are near\-orthogonal, consistent with dual completion/narrowing roles\.
### E\.11Unified Summary

The ten analyses above converge on a coherent computational and mechanistic account of howRecurrentGPT’s shared block achieves its parameter efficiency\.

Behaviourally, the model operates in two regimes separated by a representational phase transition between steps 1 and 2\. In the*early regime*\(steps 1–2\), the gate is relatively open \(≈0\.82\\approx 0\.82mean, large update norms, high proposal cosine entropy\), the hidden state undergoes a rapid restructuring that resolves∼\\sim77% of predictive uncertainty, and CKA confirms a sharp departure from the prelude representation\. In the*late regime*\(steps 3–6\), the gate progressively closes, updates shrink and align, the representation approaches a fixed point, and the attention patterns partially rebroadcast—consistent with targeted correction rather than coarse restructuring\.

Mechanistically, four findings explain*why*this behaviour emerges from the learned weights\. The*gate weight asymmetry*\(WxW\_\{x\}leading singular values 44% larger thanWhW\_\{h\}\) primes the gate to respond more strongly to changes in the hidden state than to the anchor, an advantage that is dynamically amplified asx\(r\)x^\{\(r\)\}departs fromhhacross steps\. The*contrastive projection*\(mean row cosine−0\.19\-0\.19\) ensures the shared block receives an input that emphasises what has changed since the prelude, giving each step access to a difference signal rather than a raw state\. The*head specialisations*are fixed by weight—the same 14 previous\-token heads, 8 broadcast heads, and 11 self\-attention heads appear at every step—so the block applies the same functional program each time; the gate, not the heads, decides how much of each step’s output to incorporate\. Finally, the*token frequency breakdown*reveals that a single forward pass simultaneously performs completion for common tokens and lexical narrowing for rare ones, made possible because the elementwise gate implicitly routes computation per\-position without any explicit routing mechanism\.