Looped Transformers with Source-Centered State Evolution

# Looped Transformers with Source-Centered State Evolution
Source: [https://arxiv.org/html/2607.27656](https://arxiv.org/html/2607.27656)
Bum Jun Kim\\corresponding, Kohei Hayashi, Shunsuke Kamiya, Masanori Koyama, Yusuke Iwasawa, Yutaka Matsuo

###### Abstract

Looped Transformers create a useful train\- and test\-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count\. However, that shared block must then govern an entire trajectory of varying hidden states over trained and extrapolated depths\. Furthermore, in additive\-injection looped Transformers, an input\-conditioned signal is reintroduced at every recurrent step, so applying the shared transition at an input\-conditioned reference can still move the hidden state\. In this paper, we propose Source\-Centered State Evolution \(SCSE\), which is designed to reconcile input conditioning with reference\-preserving shared recurrence\. Specifically,SCSEretains input dependence through its learned anchor and initial deviation, allows nonzero deviations to drive recurrent computation while mapping zero deviation to zero, and guarantees exact anchor invariance through its zero\-deviation mask\. The designated anchor is thereby a one\-step fixed point by construction\. The zero\-deviation forcing bias is the next deviation produced from the anchor itself and vanishes inSCSE, while nonzero deviations remain active and support state\-dependent recurrent computation\. Our theory shows that the zero\-deviation forcing bias is a design degree of freedom whose task effect can be harmful, neutral, or beneficial;SCSEresolves this choice in favor of exact anchor invariance by setting the bias to zero\. Across WikiText\-2, WikiText\-103, direct web\-corpus pretraining, held\-out web\-text transfer, LAMBADA completion, adaptive\-depth evaluation, context lengths from 128 to 1024, and shared\-block comparisons up to 139\.2M parameters,SCSEimproves the controlled recurrent quality frontier\. Ablation studies identify the learned anchor and the anchor\-coordinate deviation recurrence as the primary contributors to the gain, and a trained\-model case study grounds the anchor\-response diagnostic in observed recurrent motion\.

## Introduction

Modern Transformer language models \(LMs\) build on residual depth and self\-attention\(Heet al\.[2016](https://arxiv.org/html/2607.27656#bib.bib2); Vaswaniet al\.[2017](https://arxiv.org/html/2607.27656#bib.bib1)\)and are usually scaled by increasing parameters, data, or training compute\(Kaplanet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib37); Hoffmannet al\.[2022](https://arxiv.org/html/2607.27656#bib.bib38)\)\. Recent test\-time\-scaling work also studies inference compute as a separate scaling axis\(Snellet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib47)\)\. A complementary architectural line is looped Transformers, which reuse parameters recurrently in latent space, increasing effective depth without increasing recurrent\-block parameters\. This direction is promising because looped Transformers turn depth into a reusable inference\-time compute knob\. At inference time, the model can apply the same trained weights for additional recurrent passes through the same shared block, potentially with adaptive halting or budget selection\. Additional recurrent passes can improve algorithmic generalization, latent reasoning, or length extrapolation without storing a separate parameter set for every layer\.

The reusable depth axis, however, requires a single shared transition to govern an entire trajectory of varying hidden states over trained and extrapolated depths\. Furthermore, additive\-injection looped Transformers reintroduce an input\-conditioned signal whenever the shared block is applied\. We call this recurrently reused input\-conditioned signal the source; repeatedly injecting the source leaves the shared transition’s response at a chosen input\-conditioned reference unconstrained\. Even at that reference, another application of the shared transition can produce a source\-driven update\. Recurrent propagation can contract, cancel, exploit, or coherently accumulate this update over loop depth, thereby yielding a source\-driven degree of freedom\. The possibility of such depth\-varying propagation places an additional consistency burden on shared recurrence: the same transition must remain useful across the states generated along its own trajectory\.

We make this source\-driven degree of freedom precise\. Leteedenote the input representation and leth⋆​\(e\)h^\{\\star\}\(e\)be a reference state computed once fromee\. We callh⋆​\(e\)h^\{\\star\}\(e\)an input\-conditioned anchor\. Its value depends on the input but remains fixed throughout the recurrent unroll, serving as the origin for measuring recurrent motion\. Define the anchor\-relative deviationΔt=ht−h⋆​\(e\)\\Delta\_\{t\}=h\_\{t\}\-h^\{\\star\}\(e\), wherehth\_\{t\}denotes the hidden state at recurrent steptt\. Let𝒯t​\(Δ;e\)\\mathcal\{T\}\_\{t\}\(\\Delta;e\)denote the model’s actual one\-step map in these coordinates\. Evaluating𝒯t\\mathcal\{T\}\_\{t\}at zero deviation, corresponding to the input\-conditioned anchor, gives the next deviation

bt​\(e\):=𝒯t​\(0;e\),b\_\{t\}\(e\):=\\mathcal\{T\}\_\{t\}\(0;e\),\(1\)which we call the zero\-deviation forcing bias\. Thus,bt​\(e\)=0b\_\{t\}\(e\)=0exactly when zero deviation is a fixed point of𝒯t​\(⋅;e\)\\mathcal\{T\}\_\{t\}\(\\cdot;e\), equivalently when the chosen anchor is a one\-step fixed point at recurrent steptt\. The forcing biasbt​\(e\)b\_\{t\}\(e\)exposes a source\-driven degree of freedom in additive recurrence\. The response can be attenuated, canceled, used for computation, or preserved across recurrent depth, while its effect on the task can be harmful, neutral, or beneficial depending on the readout and loss\. The appendix formalizes these regimes through an exact bias\-subtraction counterfactual\.

To resolve this design choice in favor of anchor invariance while retaining useful off\-anchor motion, we propose Source\-Centered State Evolution \(SCSE\)\.SCSEbuilds a learned anchor once, evolves the deviation through a zero\-preserving recurrent core, and uses a mask to make the zero\-deviation threshold boundary exact\. The resulting map satisfies𝒯t​\(0;e\)=0\\mathcal\{T\}\_\{t\}\(0;e\)=0, making the designated anchor a one\-step fixed point by construction, while nonzero deviations drive state\-dependent recurrent computation\.

More generally, this anchor\-response diagnostic is not specific to looped Transformers\. Any repeatedly applied transition that reintroduces an input\-conditioned source can have a nonzero response at a chosen reference\. Yet this design choice matters especially in looped Transformers because the same transition must handle both the anchor state and ordinary off\-anchor states while repeatedly operating on its own outputs across trained and extrapolated depths\. Accordingly, we target the following design objective:

*A single shared transition should leave the chosen anchor unchanged, implement useful state\-dependent motion away from that anchor, and maintain both properties along its self\-generated trajectories within and beyond the training loop\-depth range\.*

By contrast, conventional Transformers with unshared layers do not impose the same consistency condition on any one transition, because different layers can specialize to different depth indices and absorb additive source injection as part of a fixed\-depth computation\. Under shared recurrence, however, the same source\-conditioned transition repeatedly propagates the forcing response, so additive source injection becomes a recurrent forcing term rather than just a layer\-local design choice\. This distinction motivates an explicit source\-centered reparameterization\.

Indeed, recent studies raise related concerns about unreliable dynamics under depth extrapolation\(Prairieet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib8); Yanget al\.[2026](https://arxiv.org/html/2607.27656#bib.bib10); Parket al\.[2026](https://arxiv.org/html/2607.27656#bib.bib11); Sharma and Vu[2026](https://arxiv.org/html/2607.27656#bib.bib9)\)\. To our knowledge, however, they do not define, measure, or architecturally control the shared transition’s response at an input\-conditioned reference\. This paper isolates that quantity as the zero\-deviation forcing biasbt​\(e\)b\_\{t\}\(e\)\.

Empirically, in the main WikiText\-103 comparisons,SCSEattains the lowest shared\-block perplexity \(PPL\) at every evaluated loop depth, inside and beyond the training loop\-depth range, among the looped Transformer baseline and the tuned, capacity\-matched, and recurrent\-step\-conditioned controls\. Figure[3](https://arxiv.org/html/2607.27656#Sx3.F3)and Table[1](https://arxiv.org/html/2607.27656#Sx3.T1)report this comparison\. Matched controls and controlled ablation studies identify the learned anchor and the anchor\-coordinate deviation recurrence as the main contributors to this advantage, as shown in Table[2](https://arxiv.org/html/2607.27656#Sx3.T2)and Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)\. To examine whether the anchor response targeted by the design is observed in trained models, Table[3](https://arxiv.org/html/2607.27656#Sx4.T3)verifies the expected pointwise architectural contrast across trained WikiText\-103 model families\. Additive\-injection models retain nonzero anchor responses, whereasSCSErealizes exact anchor invariance while retaining nonzero anchor\-relative motion\. Complementing this replicated measurement, Figure[4](https://arxiv.org/html/2607.27656#Sx4.F4)provides an anchor\-started WikiText\-2 case study in which forcing\-aligned trajectories and dose\-aligned displacements show that the source channel can steer recurrent motion\. Together, results from the performance comparisons, ablations, and diagnostics support source\-centered recurrence as a useful design principle\.

We make three contributions\. First, we define and measure the anchor\-dependent zero\-deviation forcing bias and theoretically show that the finite\-horizon effect of this design degree of freedom on task loss can be harmful, neutral, or beneficial, depending on recurrent propagation and readout–loss alignment\. Second, we proposeSCSE, which evolves deviations around an input\-conditioned anchor through a zero\-preserving recurrent core, making the anchor a one\-step fixed point while retaining useful off\-anchor recurrent computation\. Third, across diverse evaluation settings, we observe thatSCSEoutperforms strong shared\-block controls, with its clearest gains at deep extra\-loop depths\.

## Source\-Centered State Evolution

This paper studies the full\-state, shared\-block looped Transformer setting where the same block is applied repeatedly to the residual stream\. Letxxbe a token sequence,PϕP\_\{\\phi\}a token\-and\-position embedding map,RθR\_\{\\theta\}a shared recurrent core, andCψC\_\{\\psi\}the output head\. A full\-state looped Transformer computese=Pϕ​\(x\)e=P\_\{\\phi\}\(x\),h0=H0​\(e\)h\_\{0\}=H\_\{0\}\(e\),ht\+1=Rθ​\(ht,e\)h\_\{t\+1\}=R\_\{\\theta\}\(h\_\{t\},e\)fort=0,…,T−1t=0,\\ldots,T\-1, and next\-token logits using root\-mean\-square layer normalization \(RMSNorm\) at readout,Cψ​\(RMSNorm⁡\(hT\)\)C\_\{\\psi\}\(\\operatorname\{RMSNorm\}\(h\_\{T\}\)\)\. For a chosen loop budget, each recurrent step updates the full residual stream rather than using token\-wise routing or a learned halting policy\. Here, the recurrently reused source is distinct from the fixed input\-conditioned anchor\. Under additive injection, the source is reintroduced whenever the shared block is applied\. Thus, even at that anchor, the transition can produce the zero\-deviation forcing biasbt​\(e\)b\_\{t\}\(e\)as a nonzero update whose finite\-horizon task effect may be harmful, neutral, or beneficial\. Our goal is therefore to remove repeated source injection from the recurrent update while retaining input dependence through the fixed anchor and the initial deviation\.

The proposedSCSEparameterization selects anchor invariance as a direct architectural condition\. The anchor supplies a fixed origin for recurrent motion, the bias\-free recurrent core acts on deviations from that origin, and the zero\-deviation mask supplies an exact pointwise boundary condition\. Applying the recurrent core toΔt\\Delta\_\{t\}rather thanhth\_\{t\}defines a source\-centered recurrent vector field in which the anchor remains separate from the evolving state\.

Figure[1](https://arxiv.org/html/2607.27656#Sx2.F1)illustrates the high\-level design choice\. Here,GθG\_\{\\theta\}is the shared Transformer update implemented with causal self\-attention\. Additive source injection leaves a source\-driven anchor response available to recurrent propagation, whereas the source\-centered update selects anchor invariance and allocates computation to active nonzero deviations\.

\\begin\{overpic\}\[width=485\.65779pt\]\{zbfb\_new\.pdf\} \\end\{overpic\}Figure 1:Illustration of the zero\-deviation forcing bias and its removal inSCSE\. Repeated additive injection exposes a source\-driven degree of freedombt​\(e\)b\_\{t\}\(e\)at the anchor\.SCSEuseseeonce to set the anchor and a bias\-free recurrent core for whichGθ​\(0\)=0G\_\{\\theta\}\(0\)=0by construction\. Its zero\-deviation mask additionally ensures that the input\-conditioned anchor \(Δ=0\\Delta=0\) is a fixed point of the recurrent update at every step, while allowing active nonzero deviations to evolve\.### Source Control

We compute an input\-conditioned anchor

h⋆=e\+aω​\(e\),h^\{\\star\}=e\+a\_\{\\omega\}\(e\),\(2\)and evolve deviationsΔt=ht−h⋆\\Delta\_\{t\}=h\_\{t\}\-h^\{\\star\}\. The anchorh⋆h^\{\\star\}is produced once per input sequence by applying the learned anchor projection moduleaωa\_\{\\omega\}to the token\-and\-position representationeeand is then held fixed for all recurrent steps\. Thus, the loop evolves the anchor\-relative displacement of the current hidden state from the input\-conditioned reference instead of re\-adding that reference at every step\. The learned anchor defines the origin for the zero\-deviation condition\.

Input dependence enters through both the anchorh⋆​\(e\)h^\{\\star\}\(e\)and the separately learned initial stateh0=H0​\(e\)h\_\{0\}=H\_\{0\}\(e\), hence also through the initial deviationΔ0=H0​\(e\)−h⋆​\(e\)\\Delta\_\{0\}=H\_\{0\}\(e\)\-h^\{\\star\}\(e\)\. Starting fromΔ0\\Delta\_\{0\},SCSEevolves the deviation forTTrecurrent steps to obtainΔT\\Delta\_\{T\}, reconstructshT=h⋆\+ΔTh\_\{T\}=h^\{\\star\}\+\\Delta\_\{T\}, and then applies the final readout\. Throughout this section, deviation refers to the anchor\-relative displacementΔt=ht−h⋆\\Delta\_\{t\}=h\_\{t\}\-h^\{\\star\}\.

##### Why Use Anchor Coordinates\.

With repeated additive source injection, the shared block can produce a nonzero update even when the state is exactly at the anchor \(Δt=0\\Delta\_\{t\}=0\)\. As the block is applied repeatedly, this source\-driven update may be attenuated, used productively, or accumulated\.SCSEinstead stores the input\-conditioned anchor once and applies a zero\-preserving recurrent update to the current deviation\. Thus, zero deviation remains fixed, while nonzero deviations can still evolve and support useful computation\. Specifically, this separation retains input dependence inh⋆​\(e\)h^\{\\star\}\(e\)andΔ0\\Delta\_\{0\}, while the recurrent core receives onlyΔt\\Delta\_\{t\}, so additional loops evolve the current displacement without reapplying a fixed source\.

Figure[2](https://arxiv.org/html/2607.27656#Sx2.F2)summarizes the source\-centered computation represented by the update equations\. Letqt\(b\)q\_\{t\}^\{\(b\)\}andΔt\(b\)\\Delta\_\{t\}^\{\(b\)\}denote the sequence\-by\-channel matrices for batch elementbbat loop steptt\. The implementation first forms a raw source\-centered updateqtq\_\{t\}and then applies a per\-example zero\-deviation maskmb,tm\_\{b,t\}to obtainq¯t\\bar\{q\}\_\{t\}:

![Refer to caption](https://arxiv.org/html/2607.27656v1/x1.png)Figure 2:Illustration ofSCSE\. The proposed method builds the input\-conditioned anchorh⋆=e\+aω​\(e\)h^\{\\star\}=e\+a\_\{\\omega\}\(e\)once and holds the anchor fixed across all recurrent steps, evolves only the anchor\-relative deviationΔt\\Delta\_\{t\}, and reconstructs the final hidden state ash⋆\+ΔTh^\{\\star\}\+\\Delta\_\{T\}before readout\. The zero\-preserving recurrent blockGθG\_\{\\theta\}acts on the deviation with the residual step scaless, while the gatem=𝟏​\{∥Δt∥F2\>ϵ\}m=\\mathbf\{1\}\\\{\\lVert\\Delta\_\{t\}\\rVert\_\{F\}^\{2\}\>\\epsilon\\\}enforces the exact pointwise zero\-deviation boundary condition\.qt\\displaystyle q\_\{t\}=s​Gθ​\(Δt\),\\displaystyle=sG\_\{\\theta\}\(\\Delta\_\{t\}\),Db,t\\displaystyle D\_\{b,t\}=∥Δt\(b\)∥F2,\\displaystyle=\\lVert\\Delta\_\{t\}^\{\(b\)\}\\rVert\_\{F\}^\{2\},\(3\)mb,t\\displaystyle m\_\{b,t\}=𝟏​\{Db,t\>ϵ\},\\displaystyle=\\mathbf\{1\}\\\{D\_\{b,t\}\>\\epsilon\\\},q¯t\(b\)\\displaystyle\\bar\{q\}\_\{t\}^\{\(b\)\}=mb,t​qt\(b\),\\displaystyle=m\_\{b,t\}q\_\{t\}^\{\(b\)\},\(4\)Δt\+1\(b\)\\displaystyle\\Delta\_\{t\+1\}^\{\(b\)\}=Δt\(b\)\+q¯t\(b\)\.\\displaystyle=\\Delta\_\{t\}^\{\(b\)\}\+\\bar\{q\}\_\{t\}^\{\(b\)\}\.\(5\)
WhenΔt\(b\)=0\\Delta\_\{t\}^\{\(b\)\}=0, we haveDb,t=0D\_\{b,t\}=0, which setsmb,t=0m\_\{b,t\}=0andq¯t\(b\)=0\\bar\{q\}\_\{t\}^\{\(b\)\}=0\. Equation[5](https://arxiv.org/html/2607.27656#Sx2.E5)then givesΔt\+1\(b\)=Δt\(b\)=0\\Delta\_\{t\+1\}^\{\(b\)\}=\\Delta\_\{t\}^\{\(b\)\}=0, equivalentlyht\+1\(b\)=ht\(b\)=\(h⋆\)\(b\)h\_\{t\+1\}^\{\(b\)\}=h\_\{t\}^\{\(b\)\}=\(h^\{\\star\}\)^\{\(b\)\}, thereby yielding𝒯t​\(0;e\)=0\\mathcal\{T\}\_\{t\}\(0;e\)=0\. Thus, the designated anchor is a one\-step fixed point by construction\.

Heressis the residual step scale\. In all reported experiments, the mask threshold isϵ=10−8\\epsilon=10^\{\-8\}\. The per\-example mask is active whenever∥Δt\(b\)∥F2\>ϵ\\lVert\\Delta\_\{t\}^\{\(b\)\}\\rVert\_\{F\}^\{2\}\>\\epsilonand otherwise sets the recurrent increment to exactly zero throughout the small threshold region\. Let𝒯~t\\widetilde\{\\mathcal\{T\}\}\_\{t\}denote the raw deviation map before masking andb~t​\(e\)=𝒯~t​\(0;e\)\\widetilde\{b\}\_\{t\}\(e\)=\\widetilde\{\\mathcal\{T\}\}\_\{t\}\(0;e\)its raw anchor response\. For theSCSErecurrence used in the main experiments,𝒯~t​\(Δ;e\)=Δ\+s​Gθ​\(Δ\)\\widetilde\{\\mathcal\{T\}\}\_\{t\}\(\\Delta;e\)=\\Delta\+sG\_\{\\theta\}\(\\Delta\)\. Because RMSNorm maps zero to zero and every attention and multilayer\-perceptron projection in the core is bias\-free,Gθ​\(0\)=0G\_\{\\theta\}\(0\)=0; henceb~t​\(e\)=0\\widetilde\{b\}\_\{t\}\(e\)=0even before masking\. Therefore, the source\-centered, zero\-preserving core is the primary reparameterization\. The mask supplies the exact pointwise boundary condition even if the underlying core is not zero\-preserving\.

Eqs\.[3](https://arxiv.org/html/2607.27656#Sx2.E3)through[5](https://arxiv.org/html/2607.27656#Sx2.E5)specify one recurrent step\. In the end\-to\-end forward pass, the model computese=Pϕ​\(x\)e=P\_\{\\phi\}\(x\),h⋆=e\+aω​\(e\)h^\{\\star\}=e\+a\_\{\\omega\}\(e\), andΔ0=H0​\(e\)−h⋆\\Delta\_\{0\}=H\_\{0\}\(e\)\-h^\{\\star\}once, unrolls the shared update forTTsteps withh⋆h^\{\\star\}held fixed, reconstructshT=h⋆\+ΔTh\_\{T\}=h^\{\\star\}\+\\Delta\_\{T\}, and predicts next\-token logits fromCψ​\(RMSNorm⁡\(hT\)\)C\_\{\\psi\}\(\\operatorname\{RMSNorm\}\(h\_\{T\}\)\)\. BecauseH0H\_\{0\}andh⋆h^\{\\star\}are learned separately, the ordinarySCSEtrajectory generally starts withΔ0≠0\\Delta\_\{0\}\\neq 0and can compute immediately\. Starting exactly atΔ0=0\\Delta\_\{0\}=0is instead a boundary test for the diagnostic below\. Choosingh⋆=H0​\(e\)h^\{\\star\}=H\_\{0\}\(e\)collapses the initial deviation and, as the anchor ablation in Table[2](https://arxiv.org/html/2607.27656#Sx3.T2)confirms, suppresses useful recurrent motion\.

### Anchor\-Consistency Condition and Diagnostic

The preceding construction specifies howSCSEcomputes and establishes the designated anchor as a one\-step fixed point\. We now turn that condition into a diagnostic that can be evaluated on any trained shared\-block model\. The Mechanistic Analysis section applies this diagnostic across trained model families, as reported in Table[3](https://arxiv.org/html/2607.27656#Sx4.T3)\. Becauseh⋆h^\{\\star\}is fixed during the unroll,Δt=0\\Delta\_\{t\}=0places the hidden state exactly at the anchor\. Here,𝒯t​\(Δ;e\)\\mathcal\{T\}\_\{t\}\(\\Delta;e\)is the actual one\-step deviation map before readout, withht=h⋆\+Δth\_\{t\}=h^\{\\star\}\+\\Delta\_\{t\}andht\+1=h⋆\+𝒯t​\(Δt;e\)h\_\{t\+1\}=h^\{\\star\}\+\\mathcal\{T\}\_\{t\}\(\\Delta\_\{t\};e\)\. The diagnosticbt​\(e\)=𝒯t​\(0;e\)b\_\{t\}\(e\)=\\mathcal\{T\}\_\{t\}\(0;e\)measures the source\-driven response at that anchor\. InSCSE, this response vanishes by construction, as established after Eq\.[5](https://arxiv.org/html/2607.27656#Sx2.E5)\. The quantitybt​\(e\)b\_\{t\}\(e\)tests only the realized one\-step map at the anchor\. The diagnostic should not be conflated with the core’s response before masking,b~t​\(e\)\\widetilde\{b\}\_\{t\}\(e\), and the pointwise value alone does not show how source forcing affects an ordinary trajectory over multiple steps\.

On the active branch wheremb,t=1m\_\{b,t\}=1,SCSEstill differs from the looped Transformer baseline\. The baseline applies the shared transition tohth\_\{t\}, whereasSCSEapplies the shared transition to the anchor\-relative deviationΔt\\Delta\_\{t\}and reconstructs states ash⋆\+Δth^\{\\star\}\+\\Delta\_\{t\}\. This reparameterization combines exact anchor invariance with state\-dependent recurrent motion\.

Additive\-source recurrence can retain nonzerobt​\(e\)=𝒯t​\(0;e\)b\_\{t\}\(e\)=\\mathcal\{T\}\_\{t\}\(0;e\)on the anchor\-response test, whereasSCSEsets both the pointwise term and, for its reported zero\-preserving core, the raw anchor response to zero\. The appendix formalizes when the anchor response remains bounded under contraction, accumulates coherently, or increases task loss under adverse readout–loss alignment\.

## Experiments

### Setup

The appendix provides detailed experimental specifications, including dataset splits, model dimensions, optimizer settings, benchmark protocols, floating\-point operation \(FLOP\) accounting, matched token budgets, aggregation, and stochastic\-replication counts\.

The experiments evaluateSCSEagainst a broad suite of shared\-block controls and unshared references\. We complement these comparisons with anchor\-design ablations, trained\-model diagnostics of anchor response and source\-channel steering, and direct intervention controls\. Additional evaluations in the appendix test the method across model scales, training budgets, adaptive loop budgets, context lengths, corpus shifts, completion benchmarks, and runtime settings\.

During training, we sample the loop depth uniformly from\{1,…,8\}\\\{1,\\ldots,8\\\}\. At evaluation, we apply the same shared block at fixed depthsT∈\{4,8,12,16,24,32,48\}T\\in\\\{4,8,12,16,24,32,48\\\}, where available; separate adaptive\-depth evaluations use an adaptive stopping rule over loop iterations\. We callT≤8T\\leq 8the training loop\-depth range andT\>8T\>8extra\-loop evaluation\. This training protocol exposes all shared\-block methods to multiple depths during optimization\.

All mainSCSEexperiments use the recurrence in Eq\.[5](https://arxiv.org/html/2607.27656#Sx2.E5)with the residual step scale fixed ats=0\.50s=0\.50\. The looped Transformer baseline follows the same loop\-depth sampling, optimizer, sequence length, and evaluation schedule\. The main comparison suite includes tuned adapters, recurrent\-step\-conditioned adapters, parameter\-matched unshared controls, and absolute unshared\-depth references\. The tuned adapter uses the same learned anchor and initialization modules asSCSEbut keeps the additive\-source updateht\+1=ht\+s​ℬθ​\(ht\+α​Win​h⋆\)h\_\{t\+1\}=h\_\{t\}\+s\\mathcal\{B\}\_\{\\theta\}\(h\_\{t\}\+\\alpha W\_\{\\rm in\}h^\{\\star\}\), with no recurrent\-step term and no source\-centered zero\-deviation mask\. Here,α\\alphais that model’s learned scalar source gain\. The reported tuned rows use the selected residual step scale shown in the row label, for examples=0\.35s=0\.35in the main comparisons\.

### Shared\-Block Control Comparisons

##### Strong Shared\-Block Transformer Baselines\.

Appendix Table[5](https://arxiv.org/html/2607.27656#A1.T5)reports the stronger shared\-block comparison on WikiText\-103 at the 22M scale\. The capacity\-matched and tuned adapters are substantially stronger than the looped Transformer baseline, yetSCSEattains lower mean PPL than all three baselines at every loop depth reported in Appendix Table[5](https://arxiv.org/html/2607.27656#A1.T5)\. These comparisons support the benefit of the full source\-centered reparameterization\. Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)separately probes the zero\-deviation mask and subtraction of the anchor response throughout the active update\. Removing the mask from the source\-conditioned anchor\-coordinate \(SC\-Cond\) reference leaves PPL nearly unchanged, whereas the separately trained two\-body subtraction controls improve deep extra\-loop PPL\.

##### Recurrent\-Step\-Conditioned Controls\.

To test whether explicit loop\-step information can account forSCSE’s gains, we construct a recurrent\-step\-conditioned control, drawing on a mechanism used by the Universal Transformer and many modern looped\-depth designs\(Dehghaniet al\.[2019](https://arxiv.org/html/2607.27656#bib.bib3); Fanet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib29); Geipinget al\.[2025](https://arxiv.org/html/2607.27656#bib.bib5); Zhuet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib6); Jeddiet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib61); Chen[2026](https://arxiv.org/html/2607.27656#bib.bib30)\)\. Public looped\-depth systems pair recurrent\-step conditioning with other design choices involving scale, data, loop placement, training objective, adaptive computation, stable injection, and learned depth allocation\. Concretely, the recurrent\-step\-conditioned control adds a learned projection of a sinusoidal loop\-step embedding to the shared block’s input at each recurrence\. All other training settings are unchanged\. The added parameter count is small relative to the model scale\.

ht\+1=ht\+s​ℬθ​\(ht\+α​Win​h⋆\+τ​Pstep​γt\),h\_\{t\+1\}=h\_\{t\}\+s\\mathcal\{B\}\_\{\\theta\}\(h\_\{t\}\+\\alpha W\_\{\\rm in\}h^\{\\star\}\+\\tau P\_\{\\rm step\}\\gamma\_\{t\}\),\(6\)whereγt\\gamma\_\{t\}is the sinusoidal loop\-step embedding used at loop steptt,PstepP\_\{\\rm step\}is a learned recurrent\-step projection, distinct from the embedding mapPϕP\_\{\\phi\}, andτ\\tauis tuned on WikiText\-103 22M\. The best setting in the recurrent\-step\-conditioned control is small, withτ=0\.015\\tau=0\.015\.

Table[1](https://arxiv.org/html/2607.27656#Sx3.T1)shows the resulting comparison\. At 22M, the recurrent\-step\-conditioned adapter is essentially tied with the tuned adapter atT=8T=8, with156\.7±0\.5156\.7\\pm 0\.5PPL compared with156\.4±0\.6156\.4\\pm 0\.6, and trails by1\.21\.2PPL atT=24T=24and2\.12\.1PPL atT=48T=48\.SCSEkeeps the best shared\-block quality frontier atT=8T=8,T=24T=24, andT=48T=48\.

At 50M, the recurrent\-step\-conditioned adapter becomes a genuinely strong recurrent baseline\. The adapter improves the looped Transformer baseline from151\.1151\.1to125\.7125\.7PPL atT=8T=8and from178\.9178\.9to160\.1160\.1atT=48T=48\. At this scale,SCSEattains the lowest PPL among the compared shared\-block methods at every measured loop depth, reaching123\.1123\.1atT=8T=8,135\.5135\.5atT=24T=24, and156\.4156\.4atT=48T=48\.

At 95\.6M, the same control remains competitive and informative\. The control reaches100\.8±0\.5100\.8\\pm 0\.5PPL atT=8T=8,111\.9±0\.3111\.9\\pm 0\.3atT=24T=24, and130\.6±1\.0130\.6\\pm 1\.0atT=48T=48, whileSCSEgives the best PPL at every measured depth\.

Appendix Tables[15](https://arxiv.org/html/2607.27656#A1.T15)and[16](https://arxiv.org/html/2607.27656#A1.T16)further evaluate the same recurrent\-step\-conditioned control on held\-out web\-text transfer and the Language Modeling Broadened to Account for Discourse Aspects \(LAMBADA\) completion benchmark\. The recurrent\-step\-conditioned adapter becomes competitive at larger scales and even achieves the lowestT=8T=8PPL in the 50M held\-out OpenWebText setting, but the adapter remains behindSCSEin the reported deep extra\-loop and completion comparisons\. Thus, the recurrent\-step\-conditioned adapter is a strong baseline, but recurrent\-step conditioning alone does not account for the source\-centered quality advantage\.

Table 1:Comparison ofSCSEwith shared\-block controls on WikiText\-103 under matched training protocols\. The recurrent\-step\-conditioned control adds a small projected sinusoidal loop\-step embedding to the tuned\-adapter protocol\. Rows marked†are explicitly tuned 22M hyperparameter variants\.SCSEremains the best shared\-block quality frontier across the reported loop depths\.

### Depth Response Across Loop Depths

Figure[3](https://arxiv.org/html/2607.27656#Sx3.F3)shows the 95\.6M fixed\-depth response across the measured loop\-depth range\. The comparison evaluates one set of trained weights across evaluation depths rather than training a separate model per depth\. Within the training loop\-depth range, the expected benefit of additional loops in looped Transformers\(Zhuet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib6); Prairieet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib8)\)holds: PPL improves fromT=4T=4toT=8T=8for every method with both depths reported, and each method’s minimum over the measured depths occurs atT=8T=8\. BeyondT=8T=8, WikiText\-103 test PPL increases at each subsequently reported depth for every method, indicating worse in\-domain performance\.SCSEretains the lowest PPL among the compared shared\-block methods at each depth\. FromT=8T=8toT=48T=48, PPL increases by28\.028\.0for the looped Transformer baseline,30\.130\.1for the tuned adapter,29\.829\.8for the recurrent\-step\-conditioned adapter, and29\.029\.0forSCSE\. On held\-out OpenWebText, extra loops beyond the training loop\-depth range instead improveSCSEat every scale while the additive\-source controls degrade, as shown in Appendix Table[15](https://arxiv.org/html/2607.27656#A1.T15)\.

![Refer to caption](https://arxiv.org/html/2607.27656v1/x2.png)Figure 3:Mean test PPL versus evaluation loop depth for 95\.6M WikiText\-103 models\. For each trained run, the same weights are evaluated at all loop depths\. For every method reported at both depths, PPL improves fromT=4T=4toT=8T=8within the training loop\-depth range; stars mark the per\-method minima, and the shaded regionT\>8T\>8denotes depth extrapolation\. BeyondT=8T=8, test PPL increases at each subsequently reported depth for every method, whileSCSEretains the lowest PPL among the compared shared\-block methods at each depth\. Loop depth is plotted on a base\-2 logarithmic axis; the PPL axis uses the shifted\-log transformlog⁡\(PPL−84\.85\)\\log\(\\mathrm\{PPL\}\-84\.85\), with tick labels shown in the original PPL units\.
### Anchor Design Ablation

We vary only the anchor definition withinSCSE; all variants use the recurrence in Eq\.[5](https://arxiv.org/html/2607.27656#Sx2.E5)and the same training protocol\. The learned\-anchor row,h⋆=e\+aω​\(e\)h^\{\\star\}=e\+a\_\{\\omega\}\(e\), corresponds to the mainSCSEmodel\. The ablation compares the learned anchor with the raw embedding anchorh⋆=eh^\{\\star\}=e, an initial\-state anchorh⋆=H0​\(e\)h^\{\\star\}=H\_\{0\}\(e\), and a frozen random projection with the same module shape asaωa\_\{\\omega\}\.

Table 2:Anchor design ablation on WikiText\-103\. The learned anchor has the lowest PPL among the tested source\-centered anchors\. Setting the anchor equal to the initial state collapses the initial deviation to zero and prevents useful recurrent motion under the zero\-deviation constraint, yielding the flat depth curve\.Table[2](https://arxiv.org/html/2607.27656#Sx3.T2)shows that the learned anchor contributes beyond parameter capacity\. The raw embedding anchor and frozen random anchor both have higher PPL than the learned anchor at all reported depths\. Capacity\-matched, tuned, and recurrent\-step\-conditioned adapters instantiate the same learned anchor and initialization modules but use additive recurrent updates, isolating the effect of source\-centered recurrence\. Setting the anchor toh⋆=H0​\(e\)h^\{\\star\}=H\_\{0\}\(e\)collapses the initial deviation to zero:Δ0=H0​\(e\)−h⋆=0\\Delta\_\{0\}=H\_\{0\}\(e\)\-h^\{\\star\}=0\. The zero\-deviation constraint then keepsΔt=0\\Delta\_\{t\}=0at every loop step, so the recurrent branch makes no update and PPL is independent of loop depth\. The result favors a fixed anchor that remains separate from the initial recurrent state, while leaving the design of more effective learned anchor modules open\.

## Mechanistic Analysis

Figure[3](https://arxiv.org/html/2607.27656#Sx3.F3)shows the task\-level depth\-response pattern; however, task\-level PPL alone does not identify the underlying mechanism\. We therefore address two specific mechanism\-level questions: \(i\) whether trained maps exhibit the predicted pointwise anchor response, and \(ii\) whether source\-channel interventions can steer recurrent motion in an anchor\-started rollout\.

\\begin\{overpic\}\[width=500\.83388pt\]\{zdfb\_empirical\.pdf\} \\put\(1\.0,44\.0\)\{\(a\)\} \\put\(51\.0,44\.0\)\{\(b\)\} \\end\{overpic\}Figure 4:Anchor\-start diagnostic of forcing\-aligned motion using trained weights from one WikiText\-2 model; all rollouts setΔ0=0\\Delta\_\{0\}=0\. \(a\) In the additive\-injection model, a nonzero anchor response initiates motion away from the anchor, and the ensuing trajectories remain aligned with the input\-specific forcing rayst​b​\(e\)tb\(e\), visualizing forcing\-aligned anchor drift\. The dotted circle marks the mean anchor distance at the largest trained depth \(t=8t=8\) and is not a stability boundary\. \(b\) Varying only the evaluation\-time injection gain yields collinear injection\-relative displacements whose magnitudes scale with the gain, providing dose\-response evidence that the source channel steers recurrent motion\. Triangles mark the half\-gain and double\-gain endpoints\. The panels use separate principal\-component projections, and the green label marks the architectural contrastbt​\(e\)=0b\_\{t\}\(e\)=0\.Table 3:Anchor\-response forcing\-bias diagnostic using trained WikiText\-103 weights and recurrent unrolling throughT=48T=48\.RtR\_\{t\}is the pointwise bias energy normalized by realized update energy, andEΔ,t:=N−1​∥Δt∥F2E\_\{\\Delta,t\}:=N^\{\-1\}\\lVert\\Delta\_\{t\}\\rVert\_\{F\}^\{2\}is the mean\-squared anchor energy\. Within each model family’s anchor coordinates,SCSEhas zero pointwise response at the reported steps, whereas the additive\-source models retain nonzero late\-step responses\. The nonzeroEΔ,47E\_\{\\Delta,47\}values serve only as a non\-collapse check, showing that ordinarySCSEtrajectories remain far outside the zero\-deviation threshold region;EΔE\_\{\\Delta\}magnitudes depend on each family’s anchor and are not compared across families\.##### Pointwise Anchor\-Response Diagnostic\.

To test the predicted pointwise anchor\-response contrast, we measure one\-step responses at zero deviation\. Each model family uses its own fixed input\-conditioned anchor:h⋆=e\+aω​\(e\)h^\{\\star\}=e\+a\_\{\\omega\}\(e\)for anchored adapters and source\-centered variants, andh⋆=eh^\{\\star\}=efor the additive\-injection baseline\. Accordingly, we apply the pointwise anchor\-response diagnostic separately within each trained model family\.

Recall that𝒯t​\(Δ;e\)\\mathcal\{T\}\_\{t\}\(\\Delta;e\)denotes the one\-step deviation map in anchor coordinates before readout, withΔt\+1=𝒯t​\(Δt;e\)\\Delta\_\{t\+1\}=\\mathcal\{T\}\_\{t\}\(\\Delta\_\{t\};e\)\. We decompose

𝒯t​\(Δ;e\)=Δ\+bt​\(e\)\+at​\(Δ;e\),\\mathcal\{T\}\_\{t\}\(\\Delta;e\)=\\Delta\+b\_\{t\}\(e\)\+a\_\{t\}\(\\Delta;e\),\(7\)where

bt​\(e\)\\displaystyle b\_\{t\}\(e\):=𝒯t​\(0;e\),\\displaystyle=\\mathcal\{T\}\_\{t\}\(0;e\),\(8\)at​\(Δ;e\)\\displaystyle a\_\{t\}\(\\Delta;e\):=𝒯t​\(Δ;e\)−𝒯t​\(0;e\)−Δ\.\\displaystyle=\\mathcal\{T\}\_\{t\}\(\\Delta;e\)\-\\mathcal\{T\}\_\{t\}\(0;e\)\-\\Delta\.The termbt​\(e\)b\_\{t\}\(e\)is the pointwise zero\-deviation forcing bias, abbreviated as the forcing bias in experimental labels\.

LetNNbe the number of scalar entries in a measured batch tensor\. The implemented forcing\-bias energy ratio is

Rt\(e\):=N−1​∥bt​\(e\)∥F2max⁡\{N−1​∥Δt\+1−Δt∥F2,10−12\}\.R\_\{t\}\(e\):=\\frac\{N^\{\-1\}\\lVert b\_\{t\}\(e\)\\rVert\_\{F\}^\{2\}\}\{\\max\\\{N^\{\-1\}\\lVert\\Delta\_\{t\+1\}\-\\Delta\_\{t\}\\rVert\_\{F\}^\{2\},10^\{\-12\}\\\}\}\.\(9\)Thus,Rt​\(e\)R\_\{t\}\(e\)measures pointwise anchor\-response energy relative to realized recurrent\-update energy: zero indicates no measured pointwise response and, when the numerical floor is inactive, one indicates equal energies\. Values above one can reflect cancellation between pointwise and state\-dependent terms and are not causal contribution percentages\.

Table[3](https://arxiv.org/html/2607.27656#Sx4.T3)verifies the architectural contrast across scales\. These results answer the first question: additive\-source models retain measurable pointwise responses across scales \(R47=0\.894R\_\{47\}=0\.894–5\.4365\.436\), whereasSCSEremoves both the pointwise and raw pre\-mask anchor responses while preserving nonzero anchor\-relative motion\.

##### Anchor\-Start Source\-Steering Diagnostic\.

Figure[4](https://arxiv.org/html/2607.27656#Sx4.F4)complements these aggregate measurements with a geometric case study using trained weights from one WikiText\-2 model\. The study uses the 22M shared\-block 1200\-step protocol with one stochastic replication and six held\-out test inputs\. The diagnostic rollouts deliberately start from the anchor rather than the model’s ordinary initialization\. Additive\-injection trajectories follow input\-specific forcing rays with trajectory–ray cosines between0\.780\.78and1\.001\.00\. Rescaling only the evaluation\-time injection gain produces dose\-aligned displacement paths relative to the injection\-removed rollout\. These paths grow monotonically and span25%25\\%to51%51\\%of the full sequence\-mean anchor push att=48t=48\. Their root\-mean\-square magnitude is about36%36\\%of the anchor push\. Their directions are nearly orthogonal to the realized drift\.

More specifically, Figure[4](https://arxiv.org/html/2607.27656#Sx4.F4)\(a\) evaluates the full learned block response at the anchor, whereas Figure[4](https://arxiv.org/html/2607.27656#Sx4.F4)\(b\) provides intervention\-based evidence of source\-channel steering\. Within this anchor\-start diagnostic, the injection\-gain intervention answers the second question affirmatively: changing only the evaluation\-time injection gain steers recurrent motion in a dose\-aligned manner\. The appendix separately provides the exactbtb\_\{t\}\-subtraction analysis and direct forcing\-bias controls relevant to broader drift and PPL attribution\.

## Conclusion

Repeated source injection leaves a source\-driven anchor response as a degree of freedom in shared recurrent dynamics\. The contribution of the anchor response is shaped by recurrent propagation and readout–loss alignment, as characterized by the exact bias\-subtraction counterfactual\.SCSEremoves the source\-driven anchor\-response degree of freedom by imposing the anchor\-consistency condition\. A learned anchor and an initial deviation retain input dependence, while a zero\-preserving deviation core allocates recurrent computation to nonzero deviations\. The resulting architectural separation is a desirable property of shared recurrence: the anchor remains a fixed point, yet nonzero deviations can still support input\-dependent computation\. Across fixed\-depth, extra\-loop, adaptive\-depth, transfer, completion, context\-length, runtime, and scale comparisons,SCSEimproves the shared\-block recurrent quality frontier\. Ablation studies identify the learned anchor and the anchor\-coordinate deviation recurrence as the main contributors, and trained\-model diagnostics connect the anchor response to observed recurrent motion\. These results support source\-centered coordinates as an effective inductive bias for reusable recurrent depth\.

## References

- Path Independent Equilibrium Models Can Better Exploit Test\-Time Computation\.InNeurIPS,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px2.p2.1),[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1),[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx2.p7.5)\.
- L\. J\. Ba, J\. R\. Kiros, and G\. E\. Hinton \(2016\)Layer Normalization\.CoRRabs/1607\.06450\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- T\. Bachlechner, B\. P\. Majumder, H\. H\. Mao, G\. Cottrell, and J\. J\. McAuley \(2021\)ReZero is all you need: fast convergence at large depth\.InUAI,Vol\.161,pp\. 1352–1361\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- S\. Bai, J\. Z\. Kolter, and V\. Koltun \(2019a\)Deep Equilibrium Models\.InNeurIPS,pp\. 688–699\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- S\. Bai, J\. Z\. Kolter, and V\. Koltun \(2019b\)Trellis Networks for Sequence Modeling\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px3.p1.1)\.
- A\. Banino, J\. Balaguer, and C\. Blundell \(2021\)PonderNet: Learning to Ponder\.CoRRabs/2107\.05407\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- A\. Bansal, A\. Schwarzschild, E\. Borgnia, Z\. Emam, F\. Huang, M\. Goldblum, and T\. Goldstein \(2022\)End\-to\-end Algorithm Synthesis with Recurrent Networks: Extrapolation without Overthinking\.InNeurIPS,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1)\.
- H\. Blayney, A\. Arroyo, J\. S\. Obando\-Ceron, P\. S\. Castro, A\. C\. Courville, M\. M\. Bronstein, and X\. Dong \(2026\)A Mechanistic Analysis of Looped Reasoning Language Models\.CoRRabs/2604\.11791\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px2.p2.1),[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx2.p7.5)\.
- B\. Chang, M\. Chen, E\. Haber, and E\. H\. Chi \(2019\)AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- H\. Chen \(2026\)Thinking Deeper, Not Longer: Depth\-Recurrent Transformers for Compositional Generalization\.CoRRabs/2603\.21676\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1),[Recurrent\-Step\-Conditioned Controls\.](https://arxiv.org/html/2607.27656#Sx3.SSx2.SSS0.Px2.p1.7)\.
- T\. Q\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. Duvenaud \(2018\)Neural Ordinary Differential Equations\.InNeurIPS,pp\. 6572–6583\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- R\. Dabre and A\. Fujita \(2019\)Recurrent Stacking of Layers for Compact Neural Machine Translation Models\.InAAAI,pp\. 6292–6299\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px3.p1.1)\.
- M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and L\. Kaiser \(2019\)Universal Transformers\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1),[Recurrent\-Step\-Conditioned Controls\.](https://arxiv.org/html/2607.27656#Sx3.SSx2.SSS0.Px2.p1.7)\.
- M\. Elbayad, J\. Gu, E\. Grave, and M\. Auli \(2020\)Depth\-Adaptive Transformer\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- N\. B\. Erichson, O\. Azencot, A\. F\. Queiruga, L\. Hodgkinson, and M\. W\. Mahoney \(2021\)Lipschitz Recurrent Neural Networks\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- A\. Fan, E\. Grave, and A\. Joulin \(2020\)Reducing Transformer Depth on Demand with Structured Dropout\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- Y\. Fan, Y\. Du, K\. Ramchandran, and K\. Lee \(2025\)Looped Transformers for Length Generalization\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1),[Recurrent\-Step\-Conditioned Controls\.](https://arxiv.org/html/2607.27656#Sx3.SSx2.SSS0.Px2.p1.7)\.
- J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. Goldstein \(2025\)Scaling up Test\-Time Compute with Latent Reasoning: A Recurrent Depth Approach\.CoRRabs/2502\.05171\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1),[Recurrent\-Step\-Conditioned Controls\.](https://arxiv.org/html/2607.27656#Sx3.SSx2.SSS0.Px2.p1.7)\.
- A\. Giannou, S\. Rajput, J\. Sohn, K\. Lee, J\. D\. Lee, and D\. Papailiopoulos \(2023\)Looped Transformers as Programmable Computers\.InICML,Vol\.202,pp\. 11398–11442\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1)\.
- A\. Gokaslan, V\. Cohen, E\. Pavlick, and S\. Tellex \(2019\)Openwebtext corpus\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx4.SSSx1.p1.1)\.
- A\. Graves \(2016\)Adaptive Computation Time for Recurrent Neural Networks\.CoRRabs/1603\.08983\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- K\. Greff, R\. K\. Srivastava, and J\. Schmidhuber \(2017\)Highway and Residual Networks learn Unrolled Iterative Estimation\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- E\. Haber and L\. Ruthotto \(2017\)Stable Architectures for Deep Neural Networks\.CoRRabs/1705\.03341\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep Residual Learning for Image Recognition\.InCVPR,pp\. 770–778\.Cited by:[Introduction](https://arxiv.org/html/2607.27656#Sx1.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre \(2022\)Training Compute\-Optimal Large Language Models\.CoRRabs/2203\.15556\.Cited by:[Introduction](https://arxiv.org/html/2607.27656#Sx1.p1.1)\.
- S\. Jastrzebski, D\. Arpit, N\. Ballas, V\. Verma, T\. Che, and Y\. Bengio \(2018\)Residual Connections Encourage Iterative Inference\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- A\. Jeddi, M\. Ciccone, and B\. Taati \(2026\)LoopFormer: Elastic\-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation\.CoRRabs/2602\.11451\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1),[Recurrent\-Step\-Conditioned Controls\.](https://arxiv.org/html/2607.27656#Sx3.SSx2.SSS0.Px2.p1.7)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling Laws for Neural Language Models\.CoRRabs/2001\.08361\.Cited by:[Introduction](https://arxiv.org/html/2607.27656#Sx1.p1.1)\.
- Z\. Lan, M\. Chen, S\. Goodman, K\. Gimpel, P\. Sharma, and R\. Soricut \(2020\)ALBERT: A Lite BERT for Self\-supervised Learning of Language Representations\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px3.p1.1)\.
- Q\. Liao and T\. A\. Poggio \(2016\)Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex\.CoRRabs/1604\.03640\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- L\. Liu, X\. Liu, J\. Gao, W\. Chen, and J\. Han \(2020\)Understanding the Difficulty of Training Transformers\.InEMNLP \(1\),pp\. 5747–5763\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled Weight Decay Regularization\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx4.SSSx1.Px1.p5.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer Sentinel Mixture Models\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx4.SSSx1.p1.1)\.
- J\. Miller and M\. Hardt \(2019\)Stable Recurrent Models\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- T\. Q\. Nguyen and J\. Salazar \(2019\)Transformers without Tears: Improving the Normalization of Self\-Attention\.InIWSLT,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, Q\. N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández \(2016\)The LAMBADA dataset: Word prediction requiring a broad discourse context\.InACL \(1\),Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx4.SSSx1.Px1.p6.1)\.
- T\. Park, Y\. Lee, D\. Kim, and H\. Bae \(2026\)LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models\.CoRRabs/2605\.11011\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px2.p1.1),[Introduction](https://arxiv.org/html/2607.27656#Sx1.p8.1)\.
- R\. Pascanu, T\. Mikolov, and Y\. Bengio \(2013\)On the difficulty of training recurrent neural networks\.InICML \(3\),Vol\.28,pp\. 1310–1318\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- H\. Prairie, Z\. Novack, T\. Berg\-Kirkpatrick, and D\. Y\. Fu \(2026\)Parcae: Scaling Laws For Stable Looped Language Models\.CoRRabs/2604\.12946\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx11.p1.7),[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px2.p1.1),[Introduction](https://arxiv.org/html/2607.27656#Sx1.p8.1),[Depth Response Across Loop Depths](https://arxiv.org/html/2607.27656#Sx3.SSx3.p1.10)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx4.SSSx1.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer\.J\. Mach\. Learn\. Res\.21,pp\. 140:1–140:67\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx4.SSSx1.p1.1),[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx8.SSSx1.Px1.p3.10)\.
- D\. Raposo, S\. Ritter, B\. A\. Richards, T\. P\. Lillicrap, P\. C\. Humphreys, and A\. Santoro \(2024\)Mixture\-of\-Depths: Dynamically allocating compute in transformer\-based language models\.CoRRabs/2404\.02258\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- N\. Saunshi, N\. Dikkala, Z\. Li, S\. Kumar, and S\. J\. Reddi \(2025\)Reasoning with Latent Thoughts: On the Power of Looped Transformers\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1)\.
- T\. Schuster, A\. Fisch, J\. Gupta, M\. Dehghani, D\. Bahri, V\. Tran, Y\. Tay, and D\. Metzler \(2022\)Confident Adaptive Language Modeling\.InNeurIPS,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px5.p1.1)\.
- A\. Schwarzschild, E\. Borgnia, A\. Gupta, F\. Huang, U\. Vishkin, M\. Goldblum, and T\. Goldstein \(2021\)Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks\.InNeurIPS,pp\. 6695–6706\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1)\.
- R\. Sharma and T\. Vu \(2026\)Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models\.CoRRabs/2606\.24898\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px2.p1.1),[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx2.p7.5),[Introduction](https://arxiv.org/html/2607.27656#Sx1.p8.1)\.
- C\. V\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2025\)Scaling LLM Test\-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning\.InICLR,Cited by:[Introduction](https://arxiv.org/html/2607.27656#Sx1.p1.1)\.
- R\. K\. Srivastava, K\. Greff, and J\. Schmidhuber \(2015\)Highway Networks\.CoRRabs/1505\.00387\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- S\. Takase and S\. Kiyono \(2023\)Lessons on Parameter Sharing across Layers in Transformers\.InSustaiNLP,pp\. 78–90\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px3.p1.1)\.
- H\. Touvron, M\. Cord, A\. Sablayrolles, G\. Synnaeve, and H\. Jégou \(2021\)Going deeper with Image Transformers\.InICCV,pp\. 32–42\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is All you Need\.InNIPS,pp\. 5998–6008\.Cited by:[Introduction](https://arxiv.org/html/2607.27656#Sx1.p1.1)\.
- H\. Wang, S\. Ma, L\. Dong, S\. Huang, D\. Zhang, and F\. Wei \(2024\)DeepNet: Scaling Transformers to 1,000 Layers\.IEEE Trans\. Pattern Anal\. Mach\. Intell\.46\(10\),pp\. 6761–6774\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- R\. Xiong, Y\. Yang, D\. He, K\. Zheng, S\. Zheng, C\. Xing, H\. Zhang, Y\. Lan, L\. Wang, and T\. Liu \(2020\)On Layer Normalization in the Transformer Architecture\.InICML,Vol\.119,pp\. 10524–10533\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- X\. Yang, Z\. Han, X\. Zhang, W\. Wei, J\. Shao, L\. Guo, and Y\. Li \(2026\)Stabilizing Recurrent Dynamics for Test\-Time Scalable Latent Reasoning in Looped Language Models\.CoRRabs/2605\.26733\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px2.p1.1),[Introduction](https://arxiv.org/html/2607.27656#Sx1.p8.1)\.
- A\. Zeitoun, L\. T\. Hennigen, and Y\. Kim \(2026\)Hyperloop Transformers\.CoRRabs/2604\.21254\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1)\.
- B\. Zhang and R\. Sennrich \(2019\)Root Mean Square Layer Normalization\.InNeurIPS,pp\. 12360–12371\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- H\. Zhang, Y\. N\. Dauphin, and T\. Ma \(2019\)Fixup Initialization: Residual Learning Without Normalization\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px4.p1.1)\.
- R\. Zhu, Z\. Wang, K\. Hua, T\. Zhang, Z\. Li, H\. Que, B\. Wei, Z\. Wen, F\. Yin, H\. Xing, L\. Li, J\. Shi, K\. Ma, S\. Li, T\. Kergan, A\. Smith, X\. Qu, M\. Hui, B\. Wu, Q\. Min, H\. Huang, X\. Zhou, W\. Ye, J\. Liu, J\. Yang, Y\. Shi, C\. Lin, E\. Zhao, T\. Cai, G\. Zhang, W\. Huang, Y\. Bengio, and J\. Eshraghian \(2025\)Scaling Latent Reasoning via Looped Language Models\.CoRRabs/2510\.25741\.Cited by:[Appendix A](https://arxiv.org/html/2607.27656#A1.SSx17.SSSx1.Px1.p1.1),[Recurrent\-Step\-Conditioned Controls\.](https://arxiv.org/html/2607.27656#Sx3.SSx2.SSS0.Px2.p1.7),[Depth Response Across Loop Depths](https://arxiv.org/html/2607.27656#Sx3.SSx3.p1.10)\.

## Appendix AAppendix

### List of Notation

Table 4:List of notation\.
### When Zero\-Deviation Forcing Accumulates

The definitionbt​\(e\)=𝒯t​\(0;e\)b\_\{t\}\(e\)=\\mathcal\{T\}\_\{t\}\(0;e\)establishes whether the chosen anchor is a one\-step fixed point, but the pointwise definition does not establish whether that pointwise term changes an ordinary off\-anchor trajectory\. We therefore analyze an explicit counterfactual that subtracts the same anchor response throughout the learned map\. This intervention is a mathematical comparison, not an additional empirical result\. For a discontinuous masked map, the result can instead be applied to a continuously differentiable raw or active map or to a separately defined smooth subtractive construction\. In that case, throughout this subsection𝒯t\\mathcal\{T\}\_\{t\}andbtb\_\{t\}denote the selected differentiable map and its own anchor response\. In particular, selecting the raw map means using𝒯~t\\widetilde\{\\mathcal\{T\}\}\_\{t\}andb~t\\widetilde\{b\}\_\{t\}\. The result cannot be inferred merely from changing the masked branch within the zero\-deviation threshold region\.

Fix an inputeeand an intervalt=t0,…,T−1t=t\_\{0\},\\ldots,T\-1\. Using the exact decomposition in Eq\.[7](https://arxiv.org/html/2607.27656#Sx4.E7), define the original and bias\-subtracted trajectories from the same initial state by

Δt\+1\\displaystyle\\Delta\_\{t\+1\}=Δt\+bt​\(e\)\+at​\(Δt;e\),\\displaystyle=\\Delta\_\{t\}\+b\_\{t\}\(e\)\+a\_\{t\}\(\\Delta\_\{t\};e\),Δ¯t\+1\\displaystyle\\bar\{\\Delta\}\_\{t\+1\}=𝒯t​\(Δ¯t;e\)−bt​\(e\)=Δ¯t\+at​\(Δ¯t;e\),\\displaystyle=\\mathcal\{T\}\_\{t\}\(\\bar\{\\Delta\}\_\{t\};e\)\-b\_\{t\}\(e\)=\\bar\{\\Delta\}\_\{t\}\+a\_\{t\}\(\\bar\{\\Delta\}\_\{t\};e\),Δ¯t0\\displaystyle\\bar\{\\Delta\}\_\{t\_\{0\}\}=Δt0\.\\displaystyle=\\Delta\_\{t\_\{0\}\}\.\(10\)Throughout this subsection, each sequence\-by\-channel deviation is identified with its vectorization inℝL​d\\mathbb\{R\}^\{Ld\}\. Accordingly,⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangleand∥⋅∥2\\lVert\\cdot\\rVert\_\{2\}denote the Euclidean inner product and norm\. Before vectorization, these are the Frobenius inner product and norm\. The Jacobians below act on vectorized states\. LetEt=Δt−Δ¯tE\_\{t\}=\\Delta\_\{t\}\-\\bar\{\\Delta\}\_\{t\}denote the counterfactual forcing response\. Assumeat​\(⋅;e\)a\_\{t\}\(\\cdot;e\)is continuously differentiable on a neighborhood of the line segment joiningΔ¯t\\bar\{\\Delta\}\_\{t\}andΔt\\Delta\_\{t\}, and define the exact secant propagation map

Atsec\\displaystyle A\_\{t\}^\{\\rm sec\}:=I\+∫01JΔ​at​\(Δ¯t\+ξ​Et;e\)​𝑑ξ\\displaystyle=I\+\\int\_\{0\}^\{1\}J\_\{\\Delta\}a\_\{t\}\(\\bar\{\\Delta\}\_\{t\}\+\\xi E\_\{t\};e\)d\\xi\(11\)=∫01JΔ​𝒯t​\(Δ¯t\+ξ​Et;e\)​𝑑ξ\.\\displaystyle=\\int\_\{0\}^\{1\}J\_\{\\Delta\}\\mathcal\{T\}\_\{t\}\(\\bar\{\\Delta\}\_\{t\}\+\\xi E\_\{t\};e\)d\\xi\.Fort\>kt\>k, defineΦE​\(t,k\)=At−1sec​⋯​Aksec\\Phi\_\{E\}\(t,k\)=A\_\{t\-1\}^\{\\rm sec\}\\cdots A\_\{k\}^\{\\rm sec\}andΦE​\(k,k\)=I\\Phi\_\{E\}\(k,k\)=I\.

###### Lemma 1\(Exact one\-step response recursion\)\.

The nonlinear trajectories in Eq\.[10](https://arxiv.org/html/2607.27656#A1.E10)satisfy

Et\+1=Atsec​Et\+bt​\(e\)\.E\_\{t\+1\}=A\_\{t\}^\{\\rm sec\}E\_\{t\}\+b\_\{t\}\(e\)\.\(12\)

###### Theorem 2\(Exact counterfactual forcing propagation\)\.

For the nonlinear trajectories in Eq\.[10](https://arxiv.org/html/2607.27656#A1.E10), the counterfactual forcing response satisfies

ET=∑k=t0T−1ΦE​\(T,k\+1\)​bk​\(e\)\.E\_\{T\}=\\sum\_\{k=t\_\{0\}\}^\{T\-1\}\\Phi\_\{E\}\(T,k\+1\)b\_\{k\}\(e\)\.\(13\)

For compactness, writepT,k=ΦE​\(T,k\+1\)​bk​\(e\)p\_\{T,k\}=\\Phi\_\{E\}\(T,k\+1\)b\_\{k\}\(e\)andn=T−t0n=T\-t\_\{0\}\.

###### Corollary 3\(Contractive forcing response\)\.

Under Theorem[2](https://arxiv.org/html/2607.27656#Thmtheorem2), for any vector norm and its induced matrix norm, if there are constants0≤q<10\\leq q<1andB≥0B\\geq 0such that∥Atsec∥≤q\\lVert A\_\{t\}^\{\\rm sec\}\\rVert\\leq qand∥bt​\(e\)∥≤B\\lVert b\_\{t\}\(e\)\\rVert\\leq Bthroughout the interval, then

∥ET∥≤B​1−qn1−q≤B1−q\.\\lVert E\_\{T\}\\rVert\\leq B\\frac\{1\-q^\{n\}\}\{1\-q\}\\leq\\frac\{B\}\{1\-q\}\.\(14\)

###### Corollary 4\(Coherent forcing accumulation\)\.

Under Theorem[2](https://arxiv.org/html/2607.27656#Thmtheorem2)and the Euclidean norm, if there is a unit vectorvvsuch that⟨v,pT,k⟩≥βk≥0\\langle v,p\_\{T,k\}\\rangle\\geq\\beta\_\{k\}\\geq 0for everykk, then

∥ET∥2≥⟨v,ET⟩≥∑k=t0T−1βk\.\\lVert E\_\{T\}\\rVert\_\{2\}\\geq\\langle v,E\_\{T\}\\rangle\\geq\\sum\_\{k=t\_\{0\}\}^\{T\-1\}\\beta\_\{k\}\.\(15\)In particular, ifβk≥β\>0\\beta\_\{k\}\\geq\\beta\>0for everykkin the interval, then∥ET∥2≥n​β\\lVert E\_\{T\}\\rVert\_\{2\}\\geq n\\betaat this horizon\.

###### Corollary 5\(Constant\-secant spectral response\)\.

Under Theorem[2](https://arxiv.org/html/2607.27656#Thmtheorem2), supposeAtsec=AA\_\{t\}^\{\\rm sec\}=Aandbt​\(e\)=b​\(e\)b\_\{t\}\(e\)=b\(e\)throughout the chosen interval\. If a Euclidean unit vectorvvsatisfiesA⊤​v=ρ​vA^\{\\top\}v=\\rho vfor a realρ\\rho, then

⟨v,ET⟩=⟨v,b​\(e\)⟩​∑ℓ=0n−1ρℓ\.\\langle v,E\_\{T\}\\rangle=\\langle v,b\(e\)\\rangle\\sum\_\{\\ell=0\}^\{n\-1\}\\rho^\{\\ell\}\.\(16\)

Time homogeneity of the learned map alone does not implyAtsec=AA\_\{t\}^\{\\rm sec\}=A\. Even when𝒯t\\mathcal\{T\}\_\{t\}has no explicit step dependence, its exact secant map can vary because the paired trajectory segments vary withtt\.

To state the additional task\-level condition, let

𝒥e​\(Δ\):=ℓ​\(Cψ​\(RMSNorm⁡\(h⋆\+Δ\)\),y\)\\mathcal\{J\}\_\{e\}\(\\Delta\):=\\ell\(C\_\{\\psi\}\(\\operatorname\{RMSNorm\}\(h^\{\\star\}\+\\Delta\)\),y\)\(17\)denote the token\-level or sequence\-aggregated negative log\-likelihood for the targetyy\.

###### Proposition 6\(Sufficient task\-loss bound under adverse readout alignment\)\.

SupposeET≠0E\_\{T\}\\neq 0, setE^T=ET/∥ET∥2\\widehat\{E\}\_\{T\}=E\_\{T\}/\\lVert E\_\{T\}\\rVert\_\{2\}, and assume𝒥e\\mathcal\{J\}\_\{e\}is continuously differentiable on a neighborhood of the segment fromΔ¯T\\bar\{\\Delta\}\_\{T\}toΔT\\Delta\_\{T\}\. If, for someμ\>0\\mu\>0and everyξ∈\[0,1\]\\xi\\in\[0,1\],

⟨∇𝒥e​\(Δ¯T\+ξ​ET\),E^T⟩≥μ,\\langle\\nabla\\mathcal\{J\}\_\{e\}\(\\bar\{\\Delta\}\_\{T\}\+\\xi E\_\{T\}\),\\widehat\{E\}\_\{T\}\\rangle\\geq\\mu,\(18\)then

𝒥e​\(ΔT\)−𝒥e​\(Δ¯T\)≥μ​∥ET∥2\.\\mathcal\{J\}\_\{e\}\(\\Delta\_\{T\}\)\-\\mathcal\{J\}\_\{e\}\(\\bar\{\\Delta\}\_\{T\}\)\\geq\\mu\\lVert E\_\{T\}\\rVert\_\{2\}\.\(19\)

Under the uniform coherent conditionβk≥β\>0\\beta\_\{k\}\\geq\\beta\>0in Corollary[4](https://arxiv.org/html/2607.27656#Thmtheorem4),∥ET∥2≥n​β\\lVert E\_\{T\}\\rVert\_\{2\}\\geq n\\beta, so the loss gap in Proposition[6](https://arxiv.org/html/2607.27656#Thmtheorem6)is at leastμ​n​β\\mu n\\betaat that horizon\. The uniform lower boundμ\\muis sufficient, not necessary\. ForET≠0E\_\{T\}\\neq 0, the exact integral identity in the proof shows that a positive integrated directional derivative is necessary and sufficient for a positive loss gap; a zero integral makes the response invisible to this loss, and a negative integral makes the forcing beneficial relative to the bias\-subtracted path\.

These statements separate pointwise presence, propagated state response, horizon scaling, and task harm\. A nonzerobt​\(e\)b\_\{t\}\(e\)can produce a bounded input\-dependent offset under contraction, be neutralized by cancellation, or support useful early recurrent computation; repeated injection can therefore be compatible with and sometimes helpful for fixed\-point computation\(Anilet al\.[2022](https://arxiv.org/html/2607.27656#bib.bib12); Blayneyet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib7)\)\. A bounded nonzero response can still raise or lower task loss according to its integrated readout–loss alignment\. Coherent accumulation together with uniform adverse alignment yields the stronger depth\-linear lower boundμ​n​β\\mu n\\beta, but neither property follows frombt​\(e\)≠0b\_\{t\}\(e\)\\neq 0alone\. A scale\-invariant readout can hide some forms of state drift\(Sharma and Vu[2026](https://arxiv.org/html/2607.27656#bib.bib9)\), while leaving the trained operating region is not by itself sufficient for degradation\. Conversely, setting the pointwisebt​\(e\)=0b\_\{t\}\(e\)=0does not constrain the state\-dependent termata\_\{t\}, the Jacobian away from the anchor, or global stability\.

### Proofs

#### Counterfactual Forcing Propagation

##### Lemma[1](https://arxiv.org/html/2607.27656#Thmtheorem1)\.

###### Proof\.

Subtracting the two recurrences in Eq\.[10](https://arxiv.org/html/2607.27656#A1.E10)gives

Et\+1=Et\+bt​\(e\)\+at​\(Δt;e\)−at​\(Δ¯t;e\)\.\\displaystyle E\_\{t\+1\}=E\_\{t\}\+b\_\{t\}\(e\)\+a\_\{t\}\(\\Delta\_\{t\};e\)\-a\_\{t\}\(\\bar\{\\Delta\}\_\{t\};e\)\.BecauseΔt=Δ¯t\+Et\\Delta\_\{t\}=\\bar\{\\Delta\}\_\{t\}\+E\_\{t\}, the vector\-valued fundamental theorem of calculus gives

at​\(Δt;e\)−at​\(Δ¯t;e\)\\displaystyle a\_\{t\}\(\\Delta\_\{t\};e\)\-a\_\{t\}\(\\bar\{\\Delta\}\_\{t\};e\)=∫01JΔ​at​\(Δ¯t\+ξ​Et;e\)​Et​𝑑ξ\\displaystyle=\\int\_\{0\}^\{1\}J\_\{\\Delta\}a\_\{t\}\(\\bar\{\\Delta\}\_\{t\}\+\\xi E\_\{t\};e\)E\_\{t\}d\\xi=\(Atsec−I\)​Et\.\\displaystyle=\(A\_\{t\}^\{\\rm sec\}\-I\)E\_\{t\}\.The preceding calculation proves Eq\.[12](https://arxiv.org/html/2607.27656#A1.E12)\. ∎

##### Theorem[2](https://arxiv.org/html/2607.27656#Thmtheorem2)\.

###### Proof\.

SinceEt0=0E\_\{t\_\{0\}\}=0, repeated substitution of Eq\.[12](https://arxiv.org/html/2607.27656#A1.E12)in Lemma[1](https://arxiv.org/html/2607.27656#Thmtheorem1)yields

ET=∑k=t0T−1ΦE​\(T,k\+1\)​bk​\(e\),\\displaystyle E\_\{T\}=\\sum\_\{k=t\_\{0\}\}^\{T\-1\}\\Phi\_\{E\}\(T,k\+1\)b\_\{k\}\(e\),which is Eq\.[13](https://arxiv.org/html/2607.27656#A1.E13)\. ∎

##### Corollary[3](https://arxiv.org/html/2607.27656#Thmtheorem3)\.

###### Proof\.

For the contractive case, compatibility of a vector norm with its induced matrix norm and submultiplicativity give

∥ΦE​\(T,k\+1\)​bk​\(e\)∥≤qT−k−1​B\.\\displaystyle\\lVert\\Phi\_\{E\}\(T,k\+1\)b\_\{k\}\(e\)\\rVert\\leq q^\{T\-k\-1\}B\.Summing the geometric series proves the claim\. ∎

##### Corollary[4](https://arxiv.org/html/2607.27656#Thmtheorem4)\.

###### Proof\.

Cauchy–Schwarz, Theorem[2](https://arxiv.org/html/2607.27656#Thmtheorem2), and the assumed projections give

∥ET∥2≥⟨v,ET⟩=∑k=t0T−1⟨v,pT,k⟩≥∑k=t0T−1βk\.\\displaystyle\\lVert E\_\{T\}\\rVert\_\{2\}\\geq\\langle v,E\_\{T\}\\rangle=\\sum\_\{k=t\_\{0\}\}^\{T\-1\}\\langle v,p\_\{T,k\}\\rangle\\geq\\sum\_\{k=t\_\{0\}\}^\{T\-1\}\\beta\_\{k\}\.Whenβk≥β\>0\\beta\_\{k\}\\geq\\beta\>0for everykk, this sum hasn=T−t0n=T\-t\_\{0\}terms, each at leastβ\\beta, so∥ET∥2≥n​β\\lVert E\_\{T\}\\rVert\_\{2\}\\geq n\\beta\. ∎

##### Corollary[5](https://arxiv.org/html/2607.27656#Thmtheorem5)\.

###### Proof\.

In the homogeneous case,

ET=∑ℓ=0n−1Aℓ​b​\(e\)\.\\displaystyle E\_\{T\}=\\sum\_\{\\ell=0\}^\{n\-1\}A^\{\\ell\}b\(e\)\.The left\-eigenvector condition implies\(Aℓ\)⊤​v=ρℓ​v\(A^\{\\ell\}\)^\{\\top\}v=\\rho^\{\\ell\}v, so taking the inner product withvvgives the spectral formula\. ∎

##### Proposition[6](https://arxiv.org/html/2607.27656#Thmtheorem6)\.

###### Proof\.

WriteΔξseg=Δ¯T\+ξ​ET\\Delta\_\{\\xi\}^\{\\rm seg\}=\\bar\{\\Delta\}\_\{T\}\+\\xi E\_\{T\}\. The scalar fundamental theorem of calculus along the segment joining the two final states gives

𝒥e​\(ΔT\)−𝒥e​\(Δ¯T\)\\displaystyle\\mathcal\{J\}\_\{e\}\(\\Delta\_\{T\}\)\-\\mathcal\{J\}\_\{e\}\(\\bar\{\\Delta\}\_\{T\}\)=∫01⟨∇𝒥e​\(Δξseg\),ET⟩​𝑑ξ\\displaystyle=\\int\_\{0\}^\{1\}\\langle\\nabla\\mathcal\{J\}\_\{e\}\(\\Delta\_\{\\xi\}^\{\\rm seg\}\),E\_\{T\}\\rangle d\\xi=∥ET∥2​∫01⟨∇𝒥e​\(Δξseg\),E^T⟩​𝑑ξ\\displaystyle=\\lVert E\_\{T\}\\rVert\_\{2\}\\int\_\{0\}^\{1\}\\langle\\nabla\\mathcal\{J\}\_\{e\}\(\\Delta\_\{\\xi\}^\{\\rm seg\}\),\\widehat\{E\}\_\{T\}\\rangle d\\xi≥μ​∥ET∥2\.\\displaystyle\\geq\\mu\\lVert E\_\{T\}\\rVert\_\{2\}\.∎

### Reproducibility Details

The reproducibility details summarize how stochastic replications, evaluation identities, aggregation, and verification checks are defined\. This organization keeps the numerical claims tied to semantic evaluation settings rather than local storage paths\.

#### Experimental Setup Details

We train decoder\-only LMs from scratch on WikiText\-2 and WikiText\-103\(Merityet al\.[2017](https://arxiv.org/html/2607.27656#bib.bib14)\), OpenWebText\(Gokaslanet al\.[2019](https://arxiv.org/html/2607.27656#bib.bib67)\), and the English split of the Colossal Clean Crawled Corpus \(C4\)\(Raffelet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib13)\), using the Generative Pre\-trained Transformer 2 \(GPT\-2\) tokenizer\(Radfordet al\.[2019](https://arxiv.org/html/2607.27656#bib.bib68)\)\.

For the direct OpenWebText pretraining evaluation, we use the first 200,000 training examples for training, the next 4,096 examples for validation, and the following 4,096 examples for test\. For the direct C4 pretraining evaluation, we use the first 100,000 training examples for training and two disjoint 4,096\-example slices, one for validation and one for testing\.

The 22M shared\-block model has a hidden size of 384, 6 heads, a Swish\-gated linear unit \(SwiGLU\) width of 1536, a sequence length of 128 unless noted, a dropout probability of 0\.1, tied embeddings, and one recurrent Transformer update block\. The 50M shared\-block model uses a hidden size of 768, 12 heads, and a width of 3072\. The larger WikiText\-103 scale evaluation uses a hidden size of 1280, 20 heads, and a width of 5120, yielding 95\.6M trainable parameters for the shared\-block LM\.

##### Compute and Token\-Budget Accounting\.

We use an explicit per\-token Transformer\-body FLOP proxy with a dense causal\-mask attention count:

FLOPsbody,token​\(T\)=Nbody​\(T\)​\(8​d2\+6​d​dff\+4​L​d\),\\mathrm\{FLOPs\}\_\{\\rm body,token\}\(T\)=N\_\{\\rm body\}\(T\)\(8d^\{2\}\+6dd\_\{\\rm ff\}\+4Ld\),\(20\)whereddis the hidden width,dffd\_\{\\rm ff\}is the SwiGLU feed\-forward width, andLLis the sequence length\. Letcbodyc\_\{\\rm body\}be the number of physical shared\-block body evaluations per logical recurrent step\. For shared\-block looped Transformers,Nbody​\(T\)=cbody​TN\_\{\\rm body\}\(T\)=c\_\{\\rm body\}T\. The main method and ordinary baselines havecbody=1c\_\{\\rm body\}=1, while subtractive controls that evaluate the body both at the current state and at the anchor havecbody=2c\_\{\\rm body\}=2\. For the unshared Transformer baseline,Nbody​\(T\)=min⁡\(T,Nlayers\)N\_\{\\rm body\}\(T\)=\\min\(T,N\_\{\\rm layers\}\)\. Adding the tied vocabulary\-projection matrix multiplication gives

FLOPsbody\+head,token​\(T\)\\displaystyle\\mathrm\{FLOPs\}\_\{\\rm body\+head,token\}\(T\)=FLOPsbody,token​\(T\)\\displaystyle=\\mathrm\{FLOPs\}\_\{\\rm body,token\}\(T\)\(21\)\+2​d​\|𝒱\|,\\displaystyle\+2d\|\\mathcal\{V\}\|,where\|𝒱\|\|\\mathcal\{V\}\|is the GPT\-2 tokenizer vocabulary size\. The4​L​d4Ldterm is the denseL×LL\\times Lcausal\-mask attention accounting term per token, not an ideal triangular\-kernel attention count\. Neither expression is a total end\-to\-end LM FLOP count\. Both omit normalization, softmax, nonlinearities, elementwise operations, and architecture\-specific anchor, initialization, and source\-projection paths\. The first also omits the vocabulary head\. We therefore use these quantities only as dominant matrix\-multiplication proxies and report measured accelerator latency separately\.

Under this accounting, the 22M model has 39\.3M body\-proxy FLOPs per token and 77\.9M body\-plus\-head\-proxy FLOPs per token atT=8T=8\. AtT=48T=48, the 22M model has 235\.9M body\-proxy FLOPs per token and 274\.5M body\-plus\-head\-proxy FLOPs per token\.

The 50M WikiText\-103 long\-training setting uses a batch size of 16 with a gradient\-accumulation factor of 2, so its 5k\-step schedule sees 20\.48M tokens\. This 20\.48M\-token schedule matches the 22M 5k\-step context\-128 training\-token budget\. The 95\.6M setting uses the same effective batch and accumulation schedule, again matching the 20\.48M\-token 5k\-step budget\. The 95\.6M 10k\-step longer\-training comparison doubles the training\-token budget to 40\.96M tokens\. Under the same accounting, the 95\.6M body\-plus\-head proxy is 553\.3M FLOPs per token atT=8T=8\.

We also train two parameter\-matched unshared controls: a 4\-layer 22\.7M Transformer with a width of 320 and a 6\-layer 22\.5M Transformer with a width of 288\. The long\-context, direct web\-corpus pretraining, held\-out web\-text transfer, and 22M LAMBADA evaluations additionally include an unshared Transformer\-8 fixed\-depth reference\. This model is reported as an absolute unshared\-depth reference rather than a parameter\-matched shared\-block control\. Its active depth saturates onceT≥8T\\geq 8\.

We use adaptive moment estimation with decoupled weight decay \(AdamW\)\(Loshchilov and Hutter[2019](https://arxiv.org/html/2607.27656#bib.bib16)\)with\(β1,β2\)=\(0\.9,0\.95\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.95\), a weight decay of 0\.1, gradient clipping at 1\.0, cosine learning\-rate decay, and bfloat16 autocast on graphics processing unit \(GPU\) accelerators\. Larger 95\.6M WikiText\-103 runs use the same effective global batch and gradient\-accumulation schedule as the matched training\-budget comparisons\.

We additionally evaluate the trained WikiText\-103 model weights on held\-out OpenWebText and C4 validation slices and on the LAMBADA last\-word completion benchmark\(Papernoet al\.[2016](https://arxiv.org/html/2607.27656#bib.bib15)\), all with no additional training or adaptation\. For Appendix Table[15](https://arxiv.org/html/2607.27656#A1.T15), the held\-out OpenWebText slice immediately follows the 200,000\-example direct\-pretraining block, and the held\-out C4 slice uses the first 4,096 validation examples\.

For adaptive\-depth evaluation, we evaluate the trained WikiText\-103 model weights under a 48\-loop ceiling\. We calibrate a max\-token hidden\-update stopping rule on validation data to target mean loop budgets of 8, 16, and 24, and report effective active\-depth statistics on the test set\. The adaptive\-depth tables use offline full\-trace accounting\.

For LAMBADA, target\-token completion PPL and target\-token accuracy provide the most informative comparison at the evaluated model scales; exact last\-word accuracy is tracked in the evaluation records\.

#### Seed Protocol and Aggregation

The default stochastic replication set uses three fixed random seeds\. The expanded replication set adds two further fixed random seeds, giving a five\-seed set\. For the 95\.6M WikiText\-103 setting, training offsets stochastic random\-number streams while keeping the sampler seed fixed\. Appendix Table[11](https://arxiv.org/html/2607.27656#A1.T11)uses one representative trained model from the default replication set for each reported method–scale pair\. Its PPL columns come from the full evaluations of those models, while latency, throughput, and peak memory come from short measurements on the profiled accelerator setup\.

The expanded set is used for the primary direct comparisons\. These include the 22M WikiText\-103 direct comparisons between the tuned adapter andSCSE, the secondary leak diagnostics, and the 22M recurrent\-step\-conditioned comparison block\. They also include the 50M and 95\.6M source\-centered and recurrent\-step\-conditioned evaluations, the 95\.6M 10k\-step comparison betweenSCSEand the recurrent\-step\-conditioned adapter, and the context\-1024 baseline andSCSEevaluations\. Direct C4 tuned\-adapter andSCSEpretraining evaluations also use the expanded set, as do the 95\.6M LAMBADASCSEand recurrent\-step\-conditioned evaluations\.

Evaluations outside the expanded\-set list use the default three\-seed set\. In particular, the direct OpenWebText pretraining evaluations, held\-out OpenWebText and C4 transfer evaluations, and the main WikiText\-2 and broad scale evaluations use the default set\.

Evaluation identities are defined by the model configuration, training protocol, dataset and split, evaluation loop depth, seed, evaluation tag, and any adaptive\-runtime or calibration setting that changes the reported metric\. Aggregation keeps one semantic evaluation per identity, so repeated benchmark measurements can update stored values without increasing the effective seed count\. Integrity checks verify that each reported summary uses the required seed set and does not count duplicate evaluations within the same phase, family, seed, loop depth, dataset, split, evaluation tag, and adaptive runtime mode\.

All the reported PPL means are computed after token\-weighted mean negative log\-likelihood aggregation and then exponentiation\. Standard deviations are reported across the available seed set for each evaluation\. For paired comparisons, summaries match evaluations by seed and loop depth before taking deltas\. Reported paired deltas use small\-sample 95%tt\-interval half\-widths when shown, with deterministic paired\-bootstrap percentile intervals as additional robustness checks\. These intervals are paired robustness checks rather than large\-sample significance claims\. Broad sweeps typically use the default three\-seed set, while the direct comparisons listed above use the expanded five\-seed set\.

#### Reference PyTorch Implementation

Listing[1](https://arxiv.org/html/2607.27656#LST1)gives an example implementation of the source\-centered recurrent update used bySCSEand by the secondary leak diagnostic\. The default arguments mirror Eq\.[5](https://arxiv.org/html/2607.27656#Sx2.E5)\. In the main method, the recurrent block receives the deviation alone and is zero\-preserving because its projections are bias\-free\. The per\-example mask additionally makes the pointwise update exactly zero at the anchor\. The optional leak and anchor\-conditioning arguments reproduce the secondary diagnostic\. ForSCSE, the deviation\-leak coefficient is fixed atλleak=0\\lambda\_\{\\rm leak\}=0; a nonzero coefficient appears only in the secondary diagnostic\.

In the experiments,coreis the shared update block with RMSNorm pre\-normalization, causal self\-attention, and SwiGLU updates\.cond\_projimplementsWcW\_\{c\},embed,anchor\_proj, andinit\_delta\_projare learned modules, andreadoutapplies the final RMSNorm and tied output head\. Relative to the mathematical notation,anchor\_scale \* anchor\_projimplementsaωa\_\{\\omega\}, ande \+ init\_delta\_scale \* init\_delta\_proj\(e\)implementsH0​\(e\)H\_\{0\}\(e\)\.

The line computinganchoris executed once before the loop, soanchor\_projdefines the fixed reference point; the loop evolveshin deviation coordinates around that reference rather than repeatedly regenerating or adding the anchor\.

1defscse\_step\(h,anchor,core,\*,step\_scale=0\.5,leak=0\.0,

2cond\_proj=None,kappa=0\.0,eps=1e\-8\):

3delta=h\-anchor

4recurrent\_input=delta

5ifcond\_projisnotNoneandkappa\!=0\.0:

6recurrent\_input=delta\+kappa\*cond\_proj\(anchor\)

7

8q=step\_scale\*core\(recurrent\_input\)

9delta\_norm\_sq=delta\.pow\(2\)\.sum\(dim=\(1,2\),keepdim=True\)

10active=\(delta\_norm\_sq\>eps\)\.to\(q\.dtype\)

11q=q\*active

12

13returnanchor\+\(1\.0\-leak\)\*delta\+q

14

15

16defscse\_unroll\(input\_ids,\*,embed,anchor\_proj,init\_delta\_proj,

17core,readout,loops,anchor\_scale=0\.1,

18init\_delta\_scale=0\.1,\*\*step\_kwargs\):

19e=embed\(input\_ids\)

20anchor=e\+anchor\_scale\*anchor\_proj\(e\)

21h=e\+init\_delta\_scale\*init\_delta\_proj\(e\)

22states=\[h\]

23

24for\_inrange\(loops\):

25h=scse\_step\(h,anchor,core,\*\*step\_kwargs\)

26states\.append\(h\)

27

28returnreadout\(h\),states,anchor

Listing 1:PyTorch code forSCSE\. Setcond\_proj=None,kappa=0, andleak=0forSCSE; for the secondary leak diagnostic, setleak=0\.02, letcond\_projimplementWcW\_\{c\}, and supply that diagnostic’s learned scalar gainκ\\kappa, initialized to the same configured value as the baseline’s learned gainα\\alpha\.
#### Verification Entry Points

We verify the reported evaluations at the level of semantic evaluation identities, seed\-level aggregation, unequal\-batch accounting, and token\-weighted metric aggregation\. Evaluation identities include the model family, dataset, split, loop depth, seed, training protocol, and adaptive\-runtime setting when applicable; repeated measurements are deduplicated before computing reported summaries\.

Large\-data reproduction requires access to the listed corpora and suitable accelerators, while the fixed schedules, aggregation rules, and evaluation identities above define how the reported summaries are constructed\.

### Additional Fixed\-Depth Main Comparisons

##### Strong Shared\-Block Transformer Baselines\.

Appendix Table[5](https://arxiv.org/html/2607.27656#A1.T5)compares stronger shared\-block Transformer baselines on WikiText\-103 at the 22M scale with 5k training steps\. The capacity\-matched and tuned adapters are much stronger than the looped Transformer baseline, with the tuned adapter reaching156\.4±0\.6156\.4\\pm 0\.6PPL atT=8T=8\.

SCSEimproves on the tuned adapter at every reported loop depth, both within the training loop\-depth range and under deep extra\-loop evaluation\. These results support the full source\-centered reparameterization\. Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)provides the more specific forcing intervention\.

Table 5:Strong shared\-block baseline comparison on WikiText\-103, 22M, 5k steps\. Rows marked†are explicitly tuned 22M hyperparameter variants rather than default recipe rows\. Within the training loop\-depth range, the tuned adapter is a stronger non\-SCSEbaseline than the capacity\-matched adapter, whileSCSEhas the lowest PPL among shared\-block methods across the reported depths\.
##### Scale and Training\-Budget Evaluations\.

The larger\-scale comparisons focus onSCSEas the reported source\-centered method\. The direct extra\-loop evaluations below show thatSCSEis stronger at deep evaluation depths\.

For the stronger 50M WikiText\-103 setting with a gradient\-accumulation factor of 2, Appendix Table[6](https://arxiv.org/html/2607.27656#A1.T6)extends evaluation toT=48T=48while keeping the main comparison focused onSCSE\. The paired PPL deltas forSCSErelative to the looped Transformer baseline are−28\.0±2\.6\-28\.0\\pm 2\.6atT=8T=8,−26\.9±3\.4\-26\.9\\pm 3\.4atT=24T=24, and−22\.2±5\.5\-22\.2\\pm 5\.5atT=48T=48\.

Table 6:Extended 50M WikiText\-103 5k\-step comparison with evaluation throughT=48T=48on the same 20\.48M\-token training budget\. The table reports the source\-centered comparison:SCSEimproves on the looped Transformer baseline at every reported loop depth\.At 95\.6M, we test whether the source\-centered advantage over stronger non\-SCSEcontrols persists\. Appendix Table[7](https://arxiv.org/html/2607.27656#A1.T7)shows that a tuned adapter cutsT=8T=8PPL from117\.1±0\.5117\.1\\pm 0\.5to98\.6±0\.598\.6\\pm 0\.5, whileSCSEremains best at96\.9±0\.396\.9\\pm 0\.3\. The same ordering holds atT=16T=16,T=24T=24, andT=32T=32; atT=32T=32,SCSEreaches113\.7±0\.7113\.7\\pm 0\.7compared with116\.1±0\.9116\.1\\pm 0\.9for the tuned adapter\. Within the training loop\-depth range, mean PPL improves fromT=4T=4toT=8T=8for all three reported methods\. Mean PPL decreases from117\.4117\.4to117\.1117\.1for the looped Transformer baseline, from98\.998\.9to98\.698\.6for the tuned adapter, and from97\.397\.3to96\.996\.9forSCSE\. The improvement within the training loop\-depth range grows with the training budget\. Under the 10k\-step protocol of Appendix Table[8](https://arxiv.org/html/2607.27656#A1.T8), the mean improvements betweenT=4T=4andT=8T=8are0\.70\.7PPL for the recurrent\-step\-conditioned adapter and0\.80\.8PPL forSCSE\. Loop\-depth scaling therefore helps up to the boundary of the training loop\-depth range, and the degradation atT\>8T\>8is specific to evaluating the same trained weights at extrapolated depths\.

Table 7:Larger\-scale WikiText\-103 strong\-baseline comparison at the 95\.6M shared\-block scale, with models trained for 5k steps on 20\.48M tokens using the same effective batch and gradient\-accumulation schedule as the matched training\-budget comparisons\. Each entry reports the mean test PPL±\\pmone standard deviation\. Boldface marks the best value in the relevant comparison; when an unshared fixed\-depth reference row is included, boldface is restricted to the shared\-block rows unless stated otherwise\. Unless a caption states otherwise, result tables use the stochastic replication sets defined by the replication protocol\. Among the reported non\-SCSEcontrols, the tuned adapter is strongest within the training loop\-depth range \(T=4,8T=4,8\), butSCSEgives the lowest PPL at every reported depth\. For every method,T=8T=8PPL is lower thanT=4T=4PPL, so loop\-depth scaling helps within the training loop\-depth range, and degradation appears only at the extrapolated depthsT\>8T\>8\. Under the stated accounting, the body\-plus\-head proxy is 553\.3M FLOPs per token atT=8T=8\.

### Longer Training at 95\.6M

Appendix Table[8](https://arxiv.org/html/2607.27656#A1.T8)evaluates the 95\.6MSCSEadvantage under a longer training protocol rather than only under the 5k\-step training\-token budget\. We double the training\-token budget from 20\.48M to 40\.96M tokens, while keeping the same loop\-depth sampler, optimizer, effective batch schedule, and recurrent\-step\-conditioned adapter hyperparameters\.

Both methods improve substantially in absolute PPL\. The recurrent\-step\-conditioned adapter reaches68\.5±0\.268\.5\\pm 0\.2PPL atT=8T=8and103\.4±0\.8103\.4\\pm 0\.8atT=48T=48, which is much stronger than its 5k\-step counterpart\.SCSEnevertheless remains better at every measured depth, reaching66\.7±0\.366\.7\\pm 0\.3atT=8T=8,76\.9±0\.576\.9\\pm 0\.5atT=24T=24, and98\.3±1\.098\.3\\pm 1\.0atT=48T=48\.

The pairedSCSEminus recurrent\-step\-conditioned deltas are−1\.80\-1\.80,−3\.45\-3\.45, and−5\.10\-5\.10PPL atT=8,24,48T=8,24,48, with deterministic paired\-bootstrap intervals\[−2\.13,−1\.49\]\[\-2\.13,\-1\.49\],\[−3\.85,−2\.98\]\[\-3\.85,\-2\.98\], and\[−6\.11,−3\.99\]\[\-6\.11,\-3\.99\]\. The longer\-training comparison reinforces that recurrent\-step conditioning alone does not account for the source\-centered quality advantage\.

Table 8:Longer\-training comparison at 95\.6M on WikiText\-103\. Both methods follow the 95\.6M training protocol in Table[1](https://arxiv.org/html/2607.27656#Sx3.T1), but are trained for 10k steps and 40\.96M tokens instead of 5k steps and 20\.48M tokens\. Each entry reports the mean test PPL±\\pmone standard deviation\.SCSEpreserves its advantage over the recurrent\-step\-conditioned adapter with the larger training\-token budget\.
### Adaptive\-Depth and Runtime

##### Adaptive\-Depth Evaluation\.

A practical advantage of shared recurrent depth is that a single set of trained weights can expose a continuous inference loop\-budget knob rather than a single fixed loop depth\. The adaptive\-depth evaluation tests that knob with inference\-time, example\-level stopping\. Appendix Table[9](https://arxiv.org/html/2607.27656#A1.T9)evaluates this property using a stronger 22M comparison set consisting of the looped Transformer baseline, the tuned adapter, the recurrent\-step\-conditioned adapter, andSCSE\. We calibrate a max\-token hidden\-update stopping rule on validation data to target mean loop budgets of 8, 16, and 24 under a 48\-loop ceiling and then evaluate on the test set\.

The result preserves the fixed\-depth ordering under loop\-budgeted inference\.SCSEis the best shared\-block method at target mean loop budgets of 8, 16, and 24\.SCSEreaches156\.0156\.0,163\.3163\.3, and171\.8171\.8PPL\. The tuned adapter reaches157\.9157\.9,165\.4165\.4, and174\.1174\.1, and the recurrent\-step\-conditioned adapter reaches158\.3158\.3,165\.8165\.8, and175\.1175\.1\.

PPLMethodStaticT=8T=8Adaptive 8Adaptive 16Adaptive 24StaticT=48T=48Looped Transformer baseline179\.8±0\.8179\.8\\pm 0\.8182\.6188\.7195\.3218\.0±0\.8218\.0\\pm 0\.8Tuned adapter,s=0\.35s=0\.35156\.4±0\.6156\.4\\pm 0\.6157\.9165\.4174\.1204\.1±1\.4204\.1\\pm 1\.4Recurrent\-step\-conditioned adapter156\.7±0\.5156\.7\\pm 0\.5158\.3165\.8175\.1206\.1±1\.4206\.1\\pm 1\.4SCSE,s=0\.50s=0\.50155\.1±0\.6\\mathbf\{155\.1\\pm 0\.6\}156\.0\\mathbf\{156\.0\}163\.3\\mathbf\{163\.3\}171\.8\\mathbf\{171\.8\}200\.1±1\.3\\mathbf\{200\.1\\pm 1\.3\}Table 9:Validation\-calibrated adaptive\-depth evaluation on WikiText\-103, 22M, 5k steps, with a 48\-loop ceiling and a simple max\-token hidden\-update stopping rule\. We report the stronger comparison set including tuned and recurrent\-step\-conditioned adapters\. Thresholds are chosen on validation to target mean loop budgets of 8, 16, and 24, and performance is then evaluated on the test set\.SCSEremains the strongest shared\-block choice when the reusable loop\-budget knob is evaluated under per\-example stopping\.The same operating\-regime split holds when we scale the loop\-budgeted evaluation\. Appendix Table[10](https://arxiv.org/html/2607.27656#A1.T10)reports the corresponding adaptive\-depth evaluation on the 50M and 95\.6M comparison sets with target mean loop budgets of 8, 16, and 24 under the same 48\-loop ceiling\. The thresholds remain reasonably calibrated: most achieved mean active depths are within about0\.40\.4loops of the target, and the largest reported deviation is about1\.21\.2loops\.

At 50M,SCSEremains the best shared\-block method at every target mean loop budget\.SCSEreaches123\.5123\.5,128\.9128\.9, and135\.5135\.5PPL at target mean loop budgets of 8, 16, and 24\. The recurrent\-step\-conditioned adapter reaches126\.4126\.4,131\.9131\.9, and138\.6138\.6\. At 95\.6M,SCSEagain remains best\.SCSEreaches97\.197\.1,101\.3101\.3, and106\.6106\.6\. The tuned adapter reaches98\.998\.9,103\.1103\.1, and108\.4108\.4, and the recurrent\-step\-conditioned adapter reaches101\.5101\.5,105\.9105\.9, and111\.0111\.0\.

These scale\-up adaptive\-depth evaluations reinforce the same message as the fixed\-depth results\. Source\-centered coordinates, not recurrent\-step conditioning alone, yield the best quality under a reusable loop\-budget knob\.

PPLMethodStaticT=8T=8Adaptive 8Adaptive 16Adaptive 24StaticT=48T=4850MLooped Transformer baseline151\.1±0\.9151\.1\\pm 0\.9153\.8158\.3162\.8178\.9±1\.3178\.9\\pm 1\.3Recurrent\-step\-conditioned adapter125\.7±0\.4125\.7\\pm 0\.4126\.4131\.9138\.6160\.1±0\.2160\.1\\pm 0\.2SCSE123\.1±0\.2\\mathbf\{123\.1\\pm 0\.2\}123\.5\\mathbf\{123\.5\}128\.9\\mathbf\{128\.9\}135\.5\\mathbf\{135\.5\}156\.4±1\.1\\mathbf\{156\.4\\pm 1\.1\}95\.6MLooped Transformer baseline117\.1±0\.5117\.1\\pm 0\.5117\.8122\.2127\.6145\.1±1\.2145\.1\\pm 1\.2Tuned adapter,s=0\.35s=0\.3598\.6±0\.598\.6\\pm 0\.598\.9103\.1108\.4128\.7±0\.9128\.7\\pm 0\.9Recurrent\-step\-conditioned adapter100\.8±0\.5100\.8\\pm 0\.5101\.5105\.9111\.0130\.6±1\.0130\.6\\pm 1\.0SCSE96\.9±0\.3\\mathbf\{96\.9\\pm 0\.3\}97\.1\\mathbf\{97\.1\}101\.3\\mathbf\{101\.3\}106\.6\\mathbf\{106\.6\}125\.9±0\.9\\mathbf\{125\.9\\pm 0\.9\}Table 10:Scale\-up adaptive\-depth evaluation on WikiText\-103 comparison sets\. All methods use a 48\-loop ceiling and a validation\-calibrated max\-token hidden\-update stopping rule with target mean loop budgets of 8, 16, and 24\. Values are test means\. Most achieved mean active depths are within about 0\.4 loops of the targets, and the largest deviation is about 1\.2 loops\.
##### Accelerator Profiling\.

Appendix Table[11](https://arxiv.org/html/2607.27656#A1.T11)reports GPU accelerator measurements for one trained model per reported method–scale pair at the larger 50M and 95\.6M WikiText\-103 scales\. At 50M,SCSEis the fastest quality\-competitive shared\-block method both at the boundary of the training loop\-depth range,T=8T=8, and at the deep extra\-loop depthT=48T=48\. AtT=8T=8,SCSEreaches123\.3123\.3PPL at 17\.6 milliseconds \(ms\) per batch, compared with125\.3125\.3at 19\.2 ms for the recurrent\-step\-conditioned adapter and150\.1150\.1at 24\.2 ms for the looped Transformer baseline\. AtT=48T=48,SCSEreaches158\.2158\.2PPL at 77\.8 ms compared with160\.4160\.4at 87\.5 ms for the recurrent\-step\-conditioned adapter\.

At 95\.6M,SCSEis the best\-quality shared\-block method and remains close to the fastest measured method\.SCSEreaches96\.496\.4PPL atT=8T=8in 19\.6 ms per batch and124\.3124\.3atT=48T=48in 89\.9 ms per batch\. The tuned adapter reaches98\.098\.0and127\.7127\.7PPL in20\.020\.0and92\.992\.9ms, respectively, and the recurrent\-step\-conditioned adapter reaches100\.7100\.7and130\.0130\.0PPL in21\.221\.2and100\.1100\.1ms, respectively\. Peak allocated accelerator memory, reported in megabytes \(MB\), remains closely clustered at each scale, so the main practical distinction among strong shared\-block methods is quality and loop\-step overhead rather than a large memory gap\.

Table 11:Accelerator profile for one trained model per reported method–scale pair at the larger WikiText\-103 scales\. The looped Transformer baseline is fastest at 50MT=48T=48but has much worse PPL\.SCSEis the fastest quality\-competitive shared\-block method at 50M and remains competitive with the fastest 95\.6M method while also being the best\-quality shared\-block method\.

### Context and Corpus Extension

##### Longer Contexts\.

Appendix Table[12](https://arxiv.org/html/2607.27656#A1.T12)revisits the longer\-context setting under a 5k\-step training protocol at sequence lengths 512 and 1024, using batch sizes 8 and 4, respectively, so that each optimizer step still sees 4096 tokens\. Source\-centered coordinates remain beneficial in the longer\-context setting\.

At a context length of 512, theSCSEminus tuned\-adapter paired PPL deltas are−3\.34±3\.81\-3\.34\\pm 3\.81atT=8T=8,−6\.45±2\.91\-6\.45\\pm 2\.91atT=24T=24, and−9\.41±3\.06\-9\.41\\pm 3\.06atT=48T=48\. At a context length of 1024, the same paired deltas are−4\.15±3\.17\-4\.15\\pm 3\.17,−5\.94±3\.38\-5\.94\\pm 3\.38, and−5\.00±6\.31\-5\.00\\pm 6\.31\. Thus, the longer\-context table focuses the primary shared\-block analysis on comparisons ofSCSEwith the tuned adapter and looped Transformer baseline\.

The unshared Transformer\-8 fixed\-depth reference is included as an absolute unshared\-depth reference rather than a parameter\-matched shared\-block control\. The reference is still the absolute PPL reference atT=8T=8\. The reference reaches153\.7±0\.8153\.7\\pm 0\.8and181\.6±1\.9181\.6\\pm 1\.9at context lengths of 512 and 1024, respectively\. However, the reference requires roughly 77 MB rather than 45 MB of 16\-bit parameter memory and trains more slowly than the shared\-block source\-centered method\.

Table 12:WikiText\-103 longer\-context evaluation, 22M, 5k steps\.SCSEis the best shared\-block method at both context lengths and every reported loop depth\. The Unshared Transformer\-8 row is an absolute fixed\-depth reference rather than a parameter\-matched shared\-block control; the reference is trained with a fixed depth of 8 layers, so results for the reference are reported only fromT=8T=8onward, and its active depth saturates onceT≥8T\\geq 8\.
#### Cross\-Corpus and Out\-of\-Domain Evaluation

##### Direct Web\-Corpus Pretraining\.

To test whether the effect is WikiText\-specific, Appendix Tables[13](https://arxiv.org/html/2607.27656#A1.T13)and[14](https://arxiv.org/html/2607.27656#A1.T14)report the full strong\-baseline suite at the 22M scale with 5k training steps under direct web\-corpus pretraining on two web\-text corpora: OpenWebText and English C4\.

On OpenWebText,SCSEis strongest among reported shared\-block methods at every reported loop depth\. The pairedSCSEminus tuned\-adapter deltas are−1\.96±1\.98\-1\.96\\pm 1\.98,−5\.12±3\.13\-5\.12\\pm 3\.13, and−10\.42±14\.41\-10\.42\\pm 14\.41PPL atT=8,24,48T=8,24,48, so the OpenWebText result is consistent with the main source\-centered comparison but is noisier at the deepest setting\.

On English C4\(Raffelet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib13)\), the same primary comparison is clearer\.SCSEis strongest among reported shared\-block methods atT=8T=8,T=24T=24, andT=48T=48\. The pairedSCSEminus tuned\-adapter deltas are−3\.32±1\.28\-3\.32\\pm 1\.28,−7\.12±1\.30\-7\.12\\pm 1\.30, and−8\.74±2\.36\-8\.74\\pm 2\.36PPL atT=8,24,48T=8,24,48, with paired\-bootstrap intervals\[−3\.97,−2\.40\]\[\-3\.97,\-2\.40\],\[−7\.94,−6\.23\]\[\-7\.94,\-6\.23\], and\[−10\.47,−7\.60\]\[\-10\.47,\-7\.60\]\. C4 therefore provides a cleaner cross\-corpus check of the source\-centered pattern\.

##### Cross\-Corpus and Out\-of\-Domain Tables

##### Direct Web\-Corpus Pretraining Tables\.

Appendix Tables[13](https://arxiv.org/html/2607.27656#A1.T13)and[14](https://arxiv.org/html/2607.27656#A1.T14)provide the tabulated direct web\-corpus pretraining results referenced in the main text\.

Table 13:Direct OpenWebText pretraining, 22M, 5k steps, with the same 20\.48M\-token training budget as the WikiText\-103 22M longer\-training settings\.SCSEis the strongest reported shared\-block method across the reported loop depths, though the deepest paired comparison is noisier than the C4 counterpart\. The Unshared Transformer\-8 row is an absolute fixed\-depth reference rather than a parameter\-matched shared\-block control, and its active depth saturates onceT≥8T\\geq 8\.Table 14:Direct English C4 pretraining, 22M, 5k steps, using 100,000 training examples and disjoint 4,096\-example validation and test slices\.SCSEis the strongest reported shared\-block method across the reported loop depths\. The Unshared Transformer\-8 row is an absolute fixed\-depth reference rather than a parameter\-matched shared\-block control, and its active depth saturates onceT≥8T\\geq 8\.
##### Held\-Out Web\-Text Transfer\.

To separate direct web\-corpus pretraining from held\-out web\-text transfer, Appendix Table[15](https://arxiv.org/html/2607.27656#A1.T15)evaluates the trained WikiText\-103 model weights on held\-out OpenWebText and held\-out C4 slices with no additional training\. These benchmarks probe evaluation\-only corpus shift on natural web text with the same GPT\-2 tokenizer\.

On held\-out OpenWebText, the ordering within the training loop\-depth range is mixed across scales, but the deep extra\-loop ordering is consistent\. At 22M,SCSEis best atT=48T=48with3\.87±0\.323\.87\\pm 0\.32k PPL\. At 50M, the recurrent\-step\-conditioned adapter is the best method within the training loop\-depth range atT=8T=8with3\.66±0\.333\.66\\pm 0\.33k PPL, whileSCSEis best atT=48T=48with3\.47±0\.223\.47\\pm 0\.22k PPL\. At 95\.6M, the tuned adapter is the best method within the training loop\-depth range atT=8T=8, whileSCSEis the best deep extra\-loop method atT=48T=48\.

Held\-out C4 has lower absolute PPL values than held\-out OpenWebText in the held\-out transfer setup and gives a more consistent primary ordering\. At 22M,SCSEis best among shared\-block methods atT=48T=48\. At 50M,SCSEis best at both depths among the reported shared\-block methods\. At 95\.6M,SCSEremains best at bothT=8T=8andT=48T=48among the reported shared\-block methods\. Relative to the recurrent\-step\-conditioned adapter, its paired mean deltas are−0\.136\-0\.136k and−0\.284\-0\.284k PPL atT=8T=8andT=48T=48, with paired\-bootstrap intervals\[−0\.207,−0\.054\]\[\-0\.207,\-0\.054\]k and\[−0\.394,−0\.137\]\[\-0\.394,\-0\.137\]k\.

These results support the same source\-centered pattern under direct web\-corpus pretraining and held\-out web\-text transfer, while tuned\-adapter and recurrent\-step\-conditioned controls remain strongest in some settings within the training loop\-depth range\.

##### Held\-Out Web\-Text Transfer Table\.

Appendix Table[15](https://arxiv.org/html/2607.27656#A1.T15)provides the tabulated held\-out web\-text transfer results referenced in the main text\.

Table 15:Held\-out web\-text benchmarks on 4,096\-example OpenWebText and English C4 evaluation slices\. The OpenWebText slice immediately follows the 200,000\-example direct\-pretraining block, and C4 uses the first 4,096 examples of the English validation split\. All methods use their respective trained WikiText\-103 weights with no extra training or adaptation\. The Unshared Transformer\-8 row is an absolute fixed\-depth reference rather than a parameter\-matched shared\-block control\. Each entry reports the mean±\\pmone standard deviation\.
##### Out\-of\-Domain Completion\.

Appendix Table[16](https://arxiv.org/html/2607.27656#A1.T16)adds the LAMBADA last\-word completion benchmark\. Exact last\-word accuracy is near the floor at the evaluated model scales, so we emphasize target\-token completion PPL and target\-token accuracy on the held\-out answer span\.

The 22M suite preserves the main deep extra\-loop advantage:SCSEis the best shared\-block method atT=48T=48\. At 50M and 95\.6M, recurrent\-step conditioning improves the looped Transformer baseline and can approach the tuned adapter on target\-token PPL, butSCSEremains the strongest shared\-block method on the main completion metrics\.

At 95\.6M, the pairedSCSEminus recurrent\-step\-conditioned deltas are−1\.09\-1\.09k target\-token completion PPL atT=8T=8and−2\.87\-2\.87k atT=32T=32, with bootstrap intervals\[−2\.04,−0\.13\]\[\-2\.04,\-0\.13\]k and\[−4\.43,−1\.47\]\[\-4\.43,\-1\.47\]k\.

##### Out\-of\-Domain Completion Table\.

Appendix Table[16](https://arxiv.org/html/2607.27656#A1.T16)provides the tabulated LAMBADA completion results referenced in the main text\.

T=8T=8Reported comparison depthMethodCompletion PPL \(10310^\{3\}\)Accuracy \(%\)DepthCompletion PPL \(10310^\{3\}\)Accuracy \(%\)22M WikiText\-103 5kTuned adapter,s=0\.35s=0\.3523\.8±1\.223\.8\\pm 1\.21\.27±0\.191\.27\\pm 0\.19T=48T=4825\.9±2\.525\.9\\pm 2\.50\.64±0\.160\.64\\pm 0\.16Recurrent\-step\-conditioned adapter30\.3±4\.330\.3\\pm 4\.31\.33±0\.361\.33\\pm 0\.36T=48T=4834\.5±7\.134\.5\\pm 7\.10\.55±0\.150\.55\\pm 0\.15SCSE23\.7±1\.2\\mathbf\{23\.7\\pm 1\.2\}1\.52±0\.11\\mathbf\{1\.52\\pm 0\.11\}T=48T=4824\.2±0\.5\\mathbf\{24\.2\\pm 0\.5\}0\.84±0\.19\\mathbf\{0\.84\\pm 0\.19\}Unshared Transformer\-8, fixed\-depth reference21\.1±0\.921\.1\\pm 0\.91\.33±0\.061\.33\\pm 0\.06T≥8T\\geq 821\.1±0\.921\.1\\pm 0\.91\.33±0\.061\.33\\pm 0\.0650M WikiText\-103 5kLooped Transformer baseline25\.0±1\.225\.0\\pm 1\.20\.97±0\.130\.97\\pm 0\.13T=48T=4826\.5±3\.526\.5\\pm 3\.50\.59±0\.100\.59\\pm 0\.10Recurrent\-step\-conditioned adapter20\.6±2\.020\.6\\pm 2\.02\.37±0\.092\.37\\pm 0\.09T=48T=4824\.3±2\.524\.3\\pm 2\.51\.43±0\.191\.43\\pm 0\.19SCSE18\.2±0\.1\\mathbf\{18\.2\\pm 0\.1\}2\.72±0\.32\\mathbf\{2\.72\\pm 0\.32\}T=48T=4819\.8±0\.8\\mathbf\{19\.8\\pm 0\.8\}1\.69±0\.15\\mathbf\{1\.69\\pm 0\.15\}95\.6M WikiText\-103 5kLooped Transformer baseline23\.0±0\.323\.0\\pm 0\.31\.60±0\.121\.60\\pm 0\.12T=32T=3223\.4±0\.723\.4\\pm 0\.71\.20±0\.251\.20\\pm 0\.25Tuned adapter,s=0\.35s=0\.3518\.5±1\.418\.5\\pm 1\.42\.86±0\.042\.86\\pm 0\.04T=32T=3219\.9±1\.319\.9\\pm 1\.32\.13±0\.082\.13\\pm 0\.08Recurrent\-step\-conditioned adapter18\.0±0\.818\.0\\pm 0\.82\.69±0\.172\.69\\pm 0\.17T=32T=3219\.9±1\.619\.9\\pm 1\.61\.92±0\.081\.92\\pm 0\.08SCSE16\.9±0\.8\\mathbf\{16\.9\\pm 0\.8\}3\.42±0\.16\\mathbf\{3\.42\\pm 0\.16\}T=32T=3217\.1±1\.1\\mathbf\{17\.1\\pm 1\.1\}2\.55±0\.16\\mathbf\{2\.55\\pm 0\.16\}Table 16:Out\-of\-domain LAMBADA last\-word completion benchmark\. We report target\-token completion PPL and target\-token accuracy on the held\-out answer span\. The Depth column shows the comparison depth used in each row\. The Unshared Transformer\-8 row is an absolute fixed\-depth reference rather than a parameter\-matched shared\-block control\. Exact last\-word accuracy stays near the floor at these scales, with group means at most about0\.13%0\.13\\%, so we tabulate the more resolved target\-token completion metrics; exact last\-word accuracy is tracked in the evaluation records\.

### Additional Scale and Corpus Evaluations

The C4 scale runs reuse the selected shared\-block settings without additional large\-corpus retuning, matching the tokenizer, optimizer, context length, loop\-depth sampler, training\-token budget, and evaluation depths\.

The C4 scale evaluations use 136\.5M shared\-block models and 139\.2M recurrent\-step\-conditioned control models withd=1664d=1664, 26 attention heads,dff=6656d\_\{\\rm ff\}=6656, a context length of 128, a batch size of 32, and an AdamW learning rate of2\.5×10−42\.5\\times 10^\{\-4\}\. These evaluations follow the C4 split convention defined in the experimental setup details above\. The 5k\- and 50k\-step C4 runs correspond to training\-token budgets of 20\.48M and 204\.8M tokens per run, respectively, and C4 test\-slice PPL is reported at selected depthsT∈\{4,8,16,24,32,48,96\}T\\in\\\{4,8,16,24,32,48,96\\\}\.

#### Longer\-Training and Web\-Corpus Evaluations

To test whether the source\-centered advantage persists beyond the 5k\-step WikiText\-103 setting, Appendix Table[8](https://arxiv.org/html/2607.27656#A1.T8)reports longer WikiText\-103 training at 95\.6M parameters, while direct 22M OpenWebText and C4 pretraining are reported in Appendix Tables[13](https://arxiv.org/html/2607.27656#A1.T13)and[14](https://arxiv.org/html/2607.27656#A1.T14)\. The additional scale evaluation below trains 136\.5MSCSEmodels on C4 and compares the 136\.5MSCSEmodels with tuned and recurrent\-step\-conditioned adapter controls\. The largest recurrent\-step\-conditioned control has 139\.2M parameters, and no additional large\-corpus retuning is applied\.

Appendix Table[17](https://arxiv.org/html/2607.27656#A1.T17)extends the web\-corpus comparison to the 136\.5M C4 5k\-step setting\.SCSEimproves on the tuned adapter by 5\.83 PPL atT=8T=8, 6\.83 PPL atT=24T=24, and 9\.52 PPL atT=48T=48\. In the same setting,SCSEalso improves on the recurrent\-step\-conditioned adapter by 12\.75, 12\.53, and 11\.10 PPL at the same depths\.

Table 17:Additional C4 scale evaluation with 136\.5M shared\-block models trained for 5k optimization steps\. The recurrent\-step\-conditioned control models have 139\.2M parameters\.SCSEhas lower mean PPL than both adapter controls at every reported loop depth\.Appendix Table[18](https://arxiv.org/html/2607.27656#A1.T18)shows that, in the 50k\-step C4 scale evaluation, the absolute C4 PPL decreases from the 5k\-step regime to the 50k\-step regime\. At the largest reported point in the training loop\-depth range,T=8T=8, the tuned adapter remains competitive, and its PPL is 0\.25 lower thanSCSE’s\. At deeper extra\-loop depths,SCSEhas the lowest mean PPL, improving on the tuned adapter by 1\.38 PPL atT=32T=32, 5\.42 PPL atT=48T=48, and 35\.83 PPL atT=96T=96\. At the same deeper extra\-loop depths,SCSEalso improves on the recurrent\-step\-conditioned adapter by 5\.12, 7\.52, and 0\.72 PPL\. These C4 comparisons use the fixed protocol defined above\.

Table 18:Longer C4 scale pretraining\. All models train for 50k optimization steps, corresponding to 204\.8M training tokens per training run\.SCSEis not uniformly best within the training loop\-depth range\. The tuned adapter is slightly better at the largest reported point in that range,T=8T=8, and the shallow extra\-loop pointT=16T=16is essentially tied\. At deeper extra\-loop depths,SCSEhas the lowest mean PPL atT=32T=32,T=48T=48, andT=96T=96\.
#### When Extra Loops Help on Held\-Out Web Text

In\-domain mean PPL often reaches its minimum near the boundary of the training loop\-depth range, after whichSCSEmaintains the strongest absolute shared\-block quality even though the relative degradation slope is model dependent\. The held\-out OpenWebText evaluation examines whether additional recurrent passes can also improve quality on a natural web\-text distribution for WikiText\-103\-trained weights\. Under this distribution shift, the optimal loop depth can move beyond the training loop\-depth range\.

The held\-out OpenWebText columns in Appendix Table[15](https://arxiv.org/html/2607.27656#A1.T15)show the same distribution\-dependent extra\-loop pattern without retabulating the main transfer benchmark\. Using unrounded means, at 50M,SCSEimproves from3\.685±0\.0673\.685\\pm 0\.067k PPL atT=8T=8to3\.469±0\.2183\.469\\pm 0\.218k PPL atT=48T=48, while the recurrent\-step\-conditioned adapter degrades from3\.656±0\.3323\.656\\pm 0\.332k to4\.015±0\.3614\.015\\pm 0\.361k\. At 95\.6M,SCSEimproves from3\.602±0\.1963\.602\\pm 0\.196k to3\.421±0\.3053\.421\\pm 0\.305k, while PPL increases by0\.0710\.071k for the tuned adapter and by0\.1400\.140k for the recurrent\-step\-conditioned adapter\. Thus, source\-centered recurrence can benefit from extra loops under this web\-text distribution shift, whereas the additive\-source\-conditioning controls degrade\.

Appendix Tables[20](https://arxiv.org/html/2607.27656#A1.T20),[17](https://arxiv.org/html/2607.27656#A1.T17), and[18](https://arxiv.org/html/2607.27656#A1.T18), together with the anchor\-design results in Table[2](https://arxiv.org/html/2607.27656#Sx3.T2)and the held\-out transfer results in Appendix Table[15](https://arxiv.org/html/2607.27656#A1.T15), support the practical relevance of the source\-centered design in these settings\. The learned anchor gives the lowest PPL among the tested source\-centered anchor choices, and capacity\-matched adapter baselines do not close the gap toSCSE\. Among the tested intervention architectures, the separately trained additive forcing\-subtraction control gives the largest deep extra\-loop improvement, while mask\-only changes leave PPL nearly unchanged\. Because the additive subtraction control uses a different injected source than the tuned adapter does, is trained independently, and doubles the body applications per step, we treat these controls as design comparisons rather than mechanistic attributions\.

### Diagnostic WikiText\-2 Tables

Appendix Table[19](https://arxiv.org/html/2607.27656#A1.T19)reports results from the 22M, 1200\-step WikiText\-2 diagnostic\. These results compare stabilization, capacity, and source\-centered controls, while the main text emphasizes stronger baselines on WikiText\-103, larger\-scale evaluations, adaptive\-depth evaluations, and transfer comparisons\.

#### Stabilization and Capacity Controls

Appendix Table[19](https://arxiv.org/html/2607.27656#A1.T19)comparesSCSEwith stabilization and capacity controls atT=8T=8and at deep extra\-loop depths\.

First, periodic normalization reset is a strong baseline within the training loop\-depth range, reaching387\.8±0\.9387\.8\\pm 0\.9PPL atT=8T=8, but the reset baseline degrades sharply at deep extra\-loop depths\. Second,SCSEreaches355\.2±2\.4355\.2\\pm 2\.4atT=8T=8and has the lowest PPL atT=24T=24andT=48T=48\.

The replicated controls show that generic stabilization alone does not account for the source\-centered gain\. Periodic normalization can improve the operating point within the training loop\-depth range while suppressing useful computation at larger recurrent depth, whereas the source\-centered and capacity\-matched rows remain substantially stronger across the reported depths\.

Table 19:Stabilization and capacity controls on WikiText\-2, 22M, 1200 steps\. All shared\-block models use train\-time loop\-depth sampling from 1 to 8\. Periodic normalization reset is a strong baseline within the training loop\-depth range but degrades at deep extra\-loop depths\.SCSEhas the lowest PPL across the reported loop depths in the diagnostic table\.

### Forcing\-Bias Intervention Ablations

The forcing\-bias intervention study compares a masked SC\-Cond reference with three direct controls\. We use SC\-Cond to denote this analysis family, distinct from the mainSCSEmethod, whose masked source\-centered recurrence passes only the deviation to the recurrent block\. The masked SC\-Cond reference keeps the source\-conditioned block input and changes the branch within the zero\-deviation threshold region through the mask\. One SC\-Cond control keeps the same source\-conditioned form but removes that mask\. For the deterministic evaluation map, a subtractive SC\-Cond control uses the raw updateGθ​\(Δt\+csc​\(e\)\)−Gθ​\(csc​\(e\)\)G\_\{\\theta\}\(\\Delta\_\{t\}\+c\_\{\\rm sc\}\(e\)\)\-G\_\{\\theta\}\(c\_\{\\rm sc\}\(e\)\)before applying the residual step scale, thereby removing the raw anchor response throughout the active update rather than only setting the branch within the zero\-deviation threshold region\. Herecsc​\(e\)c\_\{\\rm sc\}\(e\)is a local SC\-Cond conditioning vector\. The additive control uses the embedding sourceeeand subtracts its zero\-deviation update\. The termτt\\tau\_\{t\}denotes an optional recurrent\-step conditioning term and is zero for controls without step conditioning:

Δt\+1\\displaystyle\\Delta\_\{t\+1\}=Δt\+s​utfs,\\displaystyle=\\Delta\_\{t\}\+su\_\{t\}^\{\\mathrm\{fs\}\},utfs\\displaystyle u\_\{t\}^\{\\mathrm\{fs\}\}=ℬθ​\(h⋆\+Δt\+α​Win​e\+τt\)\\displaystyle=\\mathcal\{B\}\_\{\\theta\}\(h^\{\\star\}\+\\Delta\_\{t\}\+\\alpha W\_\{\\rm in\}e\+\\tau\_\{t\}\)−ℬθ​\(h⋆\+α​Win​e\+τt\)\.\\displaystyle\-\\mathcal\{B\}\_\{\\theta\}\(h^\{\\star\}\+\\alpha W\_\{\\rm in\}e\+\\tau\_\{t\}\)\.\(22\)The additive forcing\-subtraction control removes the exact anchor response from its deterministic evaluation map while retaining embedding\-source additive injection\. The control uses the same learned anchor and initialization\-module design as the adapter controls, but the architecture is not a subtraction\-only version of the tuned adapter: Eq\.[22](https://arxiv.org/html/2607.27656#A1.E22)injectsWin​eW\_\{\\rm in\}e, whereas the tuned adapter injectsWin​h⋆W\_\{\\rm in\}h^\{\\star\}\. Moreover, during training, the two body calls use independent dropout realizations, so their stochastic outputs need not cancel exactly atΔt=0\\Delta\_\{t\}=0\. Exact cancellation applies to the dropout\-disabled evaluation map measured in Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)\. The resulting forcing\-subtraction architecture provides a constrained\-injection comparison under the same WikiText\-103 training protocol, related to the stable\-injection direction represented by Parcae\(Prairieet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib8)\)\. The relation is at the level of injected\-source stability rather than the proposed diagnostic: Parcae constrains residual\-stream input injection to stabilize recurrent depth, whereasSCSEreparameterizes the recurrent map to evolve anchor\-relative deviations\. Because the forcing\-subtraction control uses a different injected source than the tuned adapter does, is trained as its own architecture, and costs two shared\-block body applications per recurrent step, the resulting architecture is an analysis comparison rather than an isolated frozen\-weight intervention or a compute\-matched replacement forSCSE\. The first three rows with one shared\-block body application per recurrent step in Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)reuse the matched 22M WikiText\-103 reference PPL values from Appendix Table[5](https://arxiv.org/html/2607.27656#A1.T5)and Table[1](https://arxiv.org/html/2607.27656#Sx3.T1)\. The remaining rows are the additional intervention controls\.

Table 20:Forcing\-bias intervention controls on WikiText\-103\. The “Body applications per step” column reports the number of physical shared\-block body applications per logical recurrent step\. The pointwise\-bias energy ratio isR47​\(e\)R\_\{47\}\(e\)from Eq\.[9](https://arxiv.org/html/2607.27656#Sx4.E9), averaged over three stochastic replications;R47​\(e\)R\_\{47\}\(e\)can exceed one because state\-dependent and pointwise terms can cancel\. The no\-mask SC\-Cond model has a large measured pointwise bias but similar PPL to the masked version\. In the deterministic evaluation map, the separately trained subtractive controls remove the anchor response throughout the active update and improve performance at the deepest reported extra\-loop point, using two body applications per step\.Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)separates masking within the zero\-deviation threshold region from subtracting the anchor response throughout the deterministic active map\. Removing the mask from SC\-Cond leaves a large measured pointwise\-bias ratio but nearly unchanged PPL\. The separately trained subtractive architectures constitute a larger intervention and improve performance in the deep extra\-loop regime\. AtT=48T=48, additive forcing subtraction reaches 160\.46 PPL, compared with 204\.05 for the tuned adapter and 200\.10 forSCSEwith one body application per step\. The different injected source, independent training, and two\-application cost prevent interpreting the tuned\-adapter PPL gap as a subtraction\-only or frozen\-weight mediation estimate\.SCSEinstead uses a zero\-preserving source\-centered core with one body application\.

### Additional Mechanism and Design Controls

We report an additional recurrent\-step\-modulation control for the source\-centered comparison\. The forcing\-bias intervention controls appear in the dedicated ablation above\. Unless a table states otherwise, the WikiText\-103 ablations use the 22M, 5k\-step shared\-block protocol defined in the main WikiText\-103 22M shared\-block comparisons and the experimental setup\. This protocol uses a context length of 128, train\-time loop depths sampled uniformly from11through88, and evaluation atT∈\{4,8,12,16,24,32,48\}T\\in\\\{4,8,12,16,24,32,48\\\}\. The 22M runs use AdamW with a learning rate of3×10−43\\times 10^\{\-4\}, 500 warmup steps, and a weight decay of 0\.1\. Test PPL is reported as the mean±\\pmone standard deviation\. Unless otherwise noted, these ablations use three stochastic replications\. Evaluations shared with the broader numerical suite follow the seed protocol\.

For the non\-SCSEcontrols, we tune the residual step scale, the additive\-source gain, and the recurrent\-step\-modulation hyperparameters under this WikiText\-103 protocol; the selected values are then fixed across stochastic replications\. The stepwise feature\-wise linear modulation \(FiLM\) adapter, denoted Step\-FiLM, uses the modulation scale with the lowest deep extra\-loop validation PPL in that sweep\.

#### Step\-Modulated Adapter Control

The Step\-FiLM adapter control adds explicit recurrent\-step modulation to the tuned additive adapter\. The Step\-FiLM adapter uses a learned recurrent\-step embedding and applies affine FiLM\-style modulation before the shared block\. The modulation scale is selected under the same WikiText\-103 protocol as the other adapter hyperparameters\. This control isolates recurrent\-step modulation under matched parameter scale, data, loop\-depth sampler, and optimizer\.

With the selected modulation scale, the Step\-FiLM adapter reaches157\.07±0\.28157\.07\\pm 0\.28PPL atT=8T=8,178\.20±0\.99178\.20\\pm 0\.99atT=24T=24, and207\.65±1\.65207\.65\\pm 1\.65atT=48T=48\. The corresponding tuned\-adapter, recurrent\-step\-conditioned, andSCSEbaselines are reported in Appendix Table[5](https://arxiv.org/html/2607.27656#A1.T5)and Table[1](https://arxiv.org/html/2607.27656#Sx3.T1)\. Thus, adding either an explicit recurrent\-step signal or a small recurrent\-step\-dependent FiLM path does not match the PPL ofSCSEin this setting\. The tuned adapter remains the lowest\-PPL non\-SCSE22M adapter among the controls with one shared\-block body application per recurrent step\.SCSEimproves on the tuned adapter by 1\.29 PPL atT=8T=8, 3\.28 PPL atT=24T=24, and 3\.95 PPL atT=48T=48\.

### Anchor\-Coordinate Forcing\-Bias Energy Ratios

Appendix Figure[5](https://arxiv.org/html/2607.27656#A1.F5)visualizes the pointwise forcing\-bias energy ratiosRt​\(e\)R\_\{t\}\(e\)reported in Table[3](https://arxiv.org/html/2607.27656#Sx4.T3)across shared\-block scales\. When the numerical floor in Eq\.[9](https://arxiv.org/html/2607.27656#Sx4.E9)is inactive,Rt​\(e\)R\_\{t\}\(e\)is exactly∥bt​\(e\)∥F2/∥Δt\+1−Δt∥F2\\lVert b\_\{t\}\(e\)\\rVert\_\{F\}^\{2\}/\\lVert\\Delta\_\{t\+1\}\-\\Delta\_\{t\}\\rVert\_\{F\}^\{2\}\. We computeRt​\(e\)R\_\{t\}\(e\)for each measured batch with a fixed tensor shape and then average the resulting batch\-level ratios over measured batches and stochastic replications\. The plot emphasizes thatSCSEhas a zero pointwise ratio, while additive\-source baselines retain order\-one or larger late\-step ratios\.

![Refer to caption](https://arxiv.org/html/2607.27656v1/x3.png)Figure 5:Pointwise forcing\-bias energy ratioRt​\(e\)R\_\{t\}\(e\)from Eq\.[9](https://arxiv.org/html/2607.27656#Sx4.E9), using the same method colors as Figure[3](https://arxiv.org/html/2607.27656#Sx3.F3)\. The three plots group the 22M, 50M, and 95\.6M WikiText\-103 model weights\. Bars show the late recurrent stept=47t=47, and open markers show the first recurrent stept=0t=0using the values in Table[3](https://arxiv.org/html/2607.27656#Sx4.T3)\. The dashed line is a unit reference at which the pointwise zero\-deviation forcing bias has the same squared energy as the measured net recurrent update\. The dashed line is not theSCSEvalue\.SCSEhas a ratio of0\.000\.00at both measured steps, while additive\-source baselines retain nonzero pointwise terms whose ratios become order one or larger at late depth\. The ratio diagnoses the local presence of forcing bias relative to the net update, not trajectory accumulation or model quality\.
### Secondary Deviation\-Leak Diagnostic

The mainSCSErecurrence in Eq\.[5](https://arxiv.org/html/2607.27656#Sx2.E5)has no deviation\-leak term \(λleak=0\\lambda\_\{\\rm leak\}=0\)\. The leak diagnostic adds deviation leak and anchor\-conditioned recurrent input to the same masked source\-centered update\. The variant usesλleak=0\.02\\lambda\_\{\\rm leak\}=0\.02and its own learned scalar conditioning gainκ\\kappa, initialized to the same configured value0\.150\.15as the baseline’s separately learned additive\-injection gainα\\alpha\. The two trained gains are not constrained to remain equal\. The variant is retained only to document the lower\-gain regime near the boundary of the training loop\-depth range\. The diagnostic can be slightly better nearT=8T=8, but the variant is consistently weaker once the loop budget is pushed deeper\.

Table 21:Secondary deviation\-leak diagnostic\. The leak variant probes a lower\-gain regime near the boundary of the training loop\-depth range and is essentially tied withSCSEnearT=8T=8but substantially weaker atT=48T=48\.
### Beyond Pointwise Bias: Operating Regimes

For the local\-sensitivity diagnostics, random\-direction gain is the finite\-difference directional estimate

∥𝒯t​\(Δ\+ϵfd​v;e\)−𝒯t​\(Δ;e\)∥Fϵfd,\\displaystyle\\frac\{\\lVert\\mathcal\{T\}\_\{t\}\(\\Delta\+\\epsilon\_\{\\rm fd\}v;e\)\-\\mathcal\{T\}\_\{t\}\(\\Delta;e\)\\rVert\_\{F\}\}\{\\epsilon\_\{\\rm fd\}\},with∥v∥F=1\\lVert v\\rVert\_\{F\}=1andϵfd=10−3\\epsilon\_\{\\rm fd\}=10^\{\-3\}, averaged over four random probe directions, measured batches, and stochastic replications\. Appendix Table[22](https://arxiv.org/html/2607.27656#A1.T22)reports the corresponding 22M diagnostics forSCSEand the tuned\-adapter control\.

The operating\-point diagnostics complement the pointwise\-bias measurement\.SCSEhas zero pointwise bias, remains near\-neutral under the sampled late\-step gain diagnostic, and delivers the best deep extra\-loop quality, while the tuned adapter retains nonzero forcing bias despite its lower anchor energy\.

The same pattern persists across the broader evaluations\. Recurrent\-step\-conditioned adapters become strong baselines at 50M and 95\.6M, but they still trailSCSEat large in\-domain depths and on held\-out OpenWebText, held\-out C4, and LAMBADA\.

Alongside the pointwise\-bias diagnostic, local\-sensitivity measurements compare the learned one\-step maps under the 22M\-parameter, 5k\-step WikiText\-103 protocol\. The looped Transformer baseline starts with at=0t=0random\-direction gain of3\.98±0\.053\.98\\pm 0\.05\. The random\-direction\-gain mean indicates that the sampled perturbations are strongly amplified on average at that operating point; the statistic neither estimates the largest singular value nor certifies expansion in every direction\. The tuned adapter reduces the random\-direction gain to1\.135±0\.0021\.135\\pm 0\.002and the recurrent\-step\-conditioned adapter to1\.135±0\.0041\.135\\pm 0\.004att=0t=0, with both becoming essentially neutral at deeper steps\.

SCSEstarts slightly lower att=0t=0with1\.118±0\.0031\.118\\pm 0\.003and settles to a slightly above\-unit late\-step sampled\-gain regime, reaching1\.00108±0\.000031\.00108\\pm 0\.00003att=47t=47\. Appendix Table[21](https://arxiv.org/html/2607.27656#A1.T21)reports the lower\-gain leak diagnostic separately because the diagnostic is useful mainly near the boundary of the training loop\-depth range and is weaker at larger loop depths\.

These 22M diagnostics compare sampled local directional gains; they neither estimate the Jacobian operator norm nor establish local or global contraction\. They show that the source\-centered reparameterization neutralizes zero\-deviation forcing while retaining nonzero recurrent sensitivity away from the anchor\.

The fixed\-depth evaluations across trained and extra\-loop depths clarify the operating range of the method\.SCSEusually provides the best deep extra\-loop PPL\.

The adaptive\-depth evaluations in Appendix Tables[9](https://arxiv.org/html/2607.27656#A1.T9)and[10](https://arxiv.org/html/2607.27656#A1.T10)sharpen the operating picture:SCSEremains the best reported loop\-budgeted shared\-block method at target mean loop budgets of 8, 16, and 24 across the 22M, 50M, and 95\.6M comparisons\.

### Operating\-Point Diagnostics

Appendix Table[22](https://arxiv.org/html/2607.27656#A1.T22)reports the diagnostics referenced in the main mechanistic analysis\. The table keeps the source\-centered comparison focused on the reportedSCSEmethod\.

Table 22:Mechanism diagnostics on WikiText\-103, 22M, 5k steps\. The source\-centered row has zero pointwise response and a zero\-preserving raw core, so the remaining comparison is governed by the sampled random\-direction gain regime and how much state\-dependent computation survives at late recurrent steps\. Anchor energy is the mean\-squared anchor\-relative deviation at the late state, not a lower\-is\-better quality metric\.
### Related Work

##### Looped and recurrent\-depth Transformers\.

Dehghaniet al\.\([2019](https://arxiv.org/html/2607.27656#bib.bib3)\)introduced recurrent self\-attention with optional adaptive computation, while studies of looped Transformers and recurrent algorithms examine recurrence as a programmable, latent\-reasoning, or easy\-to\-hard extrapolation mechanism\(Giannouet al\.[2023](https://arxiv.org/html/2607.27656#bib.bib59); Schwarzschildet al\.[2021](https://arxiv.org/html/2607.27656#bib.bib57); Bansalet al\.[2022](https://arxiv.org/html/2607.27656#bib.bib58); Fanet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib29); Saunshiet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib60)\)\. Recent looped Transformers and elastic\-depth variants revive recurrent depth as a test\-time compute axis\. Huginn demonstrates arbitrary latent\-depth unrolling\(Geipinget al\.[2025](https://arxiv.org/html/2607.27656#bib.bib5)\)\. Ouro scales up open looped Transformer pretraining with learned depth allocation and entropy regularization\(Zhuet al\.[2025](https://arxiv.org/html/2607.27656#bib.bib6)\)\. LoopFormer studies budget\-conditioned loop trajectories\(Jeddiet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib61)\), and Hyperloop studies parameter\-efficient looped Transformer training\(Zeitounet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib62)\)\. Depth\-recurrent Transformer work on compositional generalization is another nearby algorithmic\-depth reference point\(Chen[2026](https://arxiv.org/html/2607.27656#bib.bib30)\)\.

##### Loop\-depth failures and recurrent stability\.

Several recent studies share our broad concern that hidden\-state dynamics can become unreliable when recurrent depth is extended, but they target different mechanisms\. Parcae models the looped residual stream as a forced dynamical system and constrains repeated injection for stable scaling\(Prairieet al\.[2026](https://arxiv.org/html/2607.27656#bib.bib8)\)\.Yanget al\.\([2026](https://arxiv.org/html/2607.27656#bib.bib10)\)regularize the recurrent Jacobian’s spectral radius, whileParket al\.\([2026](https://arxiv.org/html/2607.27656#bib.bib11)\)learn input\-dependent selective gates to mitigate hidden\-state drift\.Sharma and Vu \([2026](https://arxiv.org/html/2607.27656#bib.bib9)\)identify a readout blind spot through which scale\-invariant supervision can leave the hidden\-state norm uncontrolled\. These approaches share the broad phenomenon of recurrent instability or drift with this paper\. They do not, to our knowledge, define and measure the anchor\-response quantitybt​\(e\)=𝒯t​\(0;e\)b\_\{t\}\(e\)=\\mathcal\{T\}\_\{t\}\(0;e\)or enforce its pointwise removal as the primary architectural condition\. Conversely, the local anchor\-consistency condition is not a general stability theorem; the condition neither constrains the full Jacobian nor subsumes the readout blind spot\.

Input injection itself is not intrinsically harmful\.Blayneyet al\.\([2026](https://arxiv.org/html/2607.27656#bib.bib7)\)find that injection can encourage fixed\-point convergence in looped reasoning models, and path\-independent equilibrium models use input injection to improve test\-time\-compute generalization\(Anilet al\.[2022](https://arxiv.org/html/2607.27656#bib.bib12)\)\. These results are consistent with our contractive case, in which bounded forcing produces a bounded response\. Our narrower claim concerns forcing components that recurrent propagation preserves and state\-dependent updates do not cancel; the pointwise anchor\-response diagnostic and the exact bias\-subtraction counterfactual used to state those conditions are the points of difference\.

##### Cross\-layer parameter sharing\.

Cross\-layer parameter sharing itself is not new\. Cross\-layer\-sharing approaches reduce parameter memory in bidirectional encoder models, recurrently stacked translation layers, Trellis\-style sequence models with weight tying across depth and direct input injection, and broader parameter\-sharing settings\(Lanet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib4); Dabre and Fujita[2019](https://arxiv.org/html/2607.27656#bib.bib27); Baiet al\.[2019b](https://arxiv.org/html/2607.27656#bib.bib46); Takase and Kiyono[2023](https://arxiv.org/html/2607.27656#bib.bib28)\)\. The models studied here instead repeatedly apply one recurrent block to a changing state distribution and then evaluate that same set of trained weights at loop depths beyond the training loop\-depth range\. Applying the same trained weights beyond the training loop\-depth range introduces a dynamical evaluation concern beyond parameter sharing alone\. The same parameters must act on a sequence of hidden\-state distributions whose anchor\-relative deviation, zero\-deviation forcing bias, and measured random\-direction gain can drift with recurrent step\. A deeper unshared Transformer can specialize each layer to a particular depth index, while the shared block in a looped Transformer must learn a single transition rule that remains useful under both trained and extrapolated loop depths\.

##### Residual stabilization and dynamical views\.

Our stabilization view is related to interpretations of highway and residual networks as iterative or recurrent estimators\(Srivastavaet al\.[2015](https://arxiv.org/html/2607.27656#bib.bib48); Liao and Poggio[2016](https://arxiv.org/html/2607.27656#bib.bib49); Greffet al\.[2017](https://arxiv.org/html/2607.27656#bib.bib50); Jastrzebskiet al\.[2018](https://arxiv.org/html/2607.27656#bib.bib51)\)\. The stabilization view is also related to residual scaling and initialization methods such as Fixup, ReZero, DeepNorm, and LayerScale\(Zhanget al\.[2019](https://arxiv.org/html/2607.27656#bib.bib41); Bachlechneret al\.[2021](https://arxiv.org/html/2607.27656#bib.bib18); Wanget al\.[2024](https://arxiv.org/html/2607.27656#bib.bib19); Touvronet al\.[2021](https://arxiv.org/html/2607.27656#bib.bib35)\)\. Other connections include Transformer residual\-dependency analyses\(Liuet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib42)\), normalization choices and pre\-norm analyses\(Baet al\.[2016](https://arxiv.org/html/2607.27656#bib.bib52); Zhang and Sennrich[2019](https://arxiv.org/html/2607.27656#bib.bib24); Nguyen and Salazar[2019](https://arxiv.org/html/2607.27656#bib.bib34); Xionget al\.[2020](https://arxiv.org/html/2607.27656#bib.bib33)\), stable recurrent\-model analyses\(Pascanuet al\.[2013](https://arxiv.org/html/2607.27656#bib.bib45); Miller and Hardt[2019](https://arxiv.org/html/2607.27656#bib.bib36); Changet al\.[2019](https://arxiv.org/html/2607.27656#bib.bib65); Erichsonet al\.[2021](https://arxiv.org/html/2607.27656#bib.bib66)\), and dynamical\-systems and continuous\-depth views of residual networks\(Haber and Ruthotto[2017](https://arxiv.org/html/2607.27656#bib.bib43); Chenet al\.[2018](https://arxiv.org/html/2607.27656#bib.bib44)\)\. These approaches control update scale, normalization, residual dependency, or local spectral behavior\.SCSEis complementary because the method reparameterizes the recurrent vector field around an input\-conditioned anchor rather than directly controlling these stabilization quantities\.

##### Equilibrium and adaptive computation\.

Equilibrium and adaptive\-computation models provide another nearby comparison\. Deep Equilibrium Models solve for an implicit fixed point\(Baiet al\.[2019a](https://arxiv.org/html/2607.27656#bib.bib25)\), and path\-independent equilibrium models regularize trajectories toward input\-conditioned equilibria while targeting generalization across iteration counts\(Anilet al\.[2022](https://arxiv.org/html/2607.27656#bib.bib12)\)\. Adaptive computation, probabilistic halting, depth\-adaptive Transformers, depth\-on\-demand pruning, early\-exit LM methods, and token\-level dynamic\-depth routing decide how much computation to spend per example or token\(Graves[2016](https://arxiv.org/html/2607.27656#bib.bib26); Baninoet al\.[2021](https://arxiv.org/html/2607.27656#bib.bib31); Elbayadet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib22); Fanet al\.[2020](https://arxiv.org/html/2607.27656#bib.bib63); Schusteret al\.[2022](https://arxiv.org/html/2607.27656#bib.bib64); Raposoet al\.[2024](https://arxiv.org/html/2607.27656#bib.bib32)\)\. Here, we study explicit recurrent unrolls, report fixed\-depth and offline loop\-budgeted evaluations, and use adaptive\-depth to show that source\-centered shared blocks expose a reusable inference\-time depth knob\.

##### Positioning of this study\.

All these systems motivate the recurrent\-depth axis while varying in scale, data, loop placement, training objective, adaptive depth allocation, and stability parameterization\. Within this broader recurrent\-depth design space, we isolate the full\-state, shared\-block regime in which every token state is updated by the same block over recurrent depth\. The stability work above shares the broad problem of hidden\-state evolution becoming unreliable with extra loops\. Our narrower contribution is to define the anchor\-dependent pointwise diagnosticbt​\(e\)b\_\{t\}\(e\), distinguish the pointwise diagnostic from the raw pre\-mask response and the finite\-horizon counterfactual response, state sufficient propagation and loss\-alignment conditions under which the latter becomes harmful, and test a source\-centered reparameterization with a zero\-preserving core and an exact masked anchor boundary\. The matched controls cover ingredients most likely to explain the gain, including capacity\-matched adapters, recurrent\-step conditioning, learned step modulation, residual\-scale tuning, parameter\-matched unshared controls, forcing\-subtraction architectures, and absolute unshared\-depth references\.

### Limitations

The experiments isolate controlled shared\-block LM settings rather than frontier\-scale LM pretraining\. The largest source\-centered model has 136\.5M trainable parameters, and the largest matched control has 139\.2M trainable parameters\. The longest direct C4 suite uses 204\.8M training tokens per trained model\. Billion\-parameter scaling\-law behavior and billion\-token web\-corpus training therefore remain open\.

Zero\-deviation forcing bias is defined relative to a chosen, model\-specific anchor\. The pointwise conditionbt​\(e\)=0b\_\{t\}\(e\)=0says that zero deviation is a fixed point of𝒯t​\(⋅;e\)\\mathcal\{T\}\_\{t\}\(\\cdot;e\), equivalently that this anchor is a one\-step fixed point at recurrent steptt\. This pointwise condition is not invariant under changes to the anchor and does not imply that the anchor is an optimal endpoint\. For a masked map, the pointwise response can also differ from the raw active\-branch response, as the no\-mask SC\-Cond control demonstrates\.SCSEremoves both responses in the reported zero\-preserving core, butSCSEalso changes the learned recurrent vector field by applying the shared core to deviations, so the method should be viewed as an architectural reparameterization rather than a lossless coordinate rewrite\.

The exact finite\-horizon theorem, its contraction and coherence corollaries, and the task\-loss proposition give sufficient conditions for a bias\-subtraction response to remain bounded, accumulate, and increase task loss\. These results do not constitute a global theorem for stability or PPL\.SCSEremoves the local pointwise and raw anchor responses in the reported core but does not constrain off\-anchor Jacobians, the state\-dependent residual, or readout sensitivity\. Figure[4](https://arxiv.org/html/2607.27656#Sx4.F4)is a one\-replication geometric case study that uses separate projections for the two plots\. Table[3](https://arxiv.org/html/2607.27656#Sx4.T3)provides replicated pointwise measurements\. Appendix Table[20](https://arxiv.org/html/2607.27656#A1.T20)shows both that a nonzero pointwise response can be benign in one control and that separately trained subtraction architectures can substantially improve deep extra\-loop PPL in others\. For the subtractive controls, exact cancellation refers to the dropout\-disabled evaluation map; during training, independent dropout realizations in the two body calls need not cancel\. Because these controls are retrained, require an extra body application, and, in the additive case, use a different injected source than the tuned adapter does, they do not provide a subtraction\-only or frozen\-weight causal estimate\. Establishing how often the sufficient propagation and readout\-alignment conditions hold across architectures, anchors, and scales remains open\.