DeepLoop: Depth Scaling for Looped Transformers

# DeepLoop: Depth Scaling for Looped Transformers
Source: [https://arxiv.org/html/2607.13491](https://arxiv.org/html/2607.13491)
\\setlist

\[itemize\]leftmargin=\*,topsep=1pt,itemsep=1pt\\setlist\[enumerate\]leftmargin=\*,topsep=1pt,itemsep=1pt

Shuzhen Li1Yifan Zhang1,†Jiacheng Guo1 Quanquan Gu2Mengdi Wang1,† 1Princeton University2University of California

###### Abstract

Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters\. This reuse changes the residual\-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass\. We formalize this tied\-depth effect through a first\-order perturbation bound controlled by a visit\-alignment coefficientκR\\kappa\_\{R\}\. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from1/41/4to1/21/2as loop count grows at fixed physical depth\. The resulting method,DeepLoop, keeps the Post\-LN DeepNorm architecture and setsα=\(2​N\)1/2\\alpha=\(2N\)^\{1/2\}andβ=\(8​N\)−1/2\\beta=\(8N\)^\{\-1/2\}for unrolled depthNN\. On GPT\-style looped language models at GPT\-2 small and GPT\-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated\. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count\.

$\\dagger$$\\dagger$footnotetext:Corresponding authors:yifzhang@princeton\.edu,mengdiw@princeton\.edu\.Project Page:\\urlhttps://github\.com/lszshu/DeepLoop

\(a\) Physical blocks—K=2K\{=\}2shared blocks, stored onceBlock 1 \(ϕ1\\phi\_\{1\}\)attnffnBlock 2 \(ϕ2\\phi\_\{2\}\)attnffnloop×R\\times\\,Rrounds\(b\) Unrolled execution— the same two blocks are revisited in every round𝐱0\\mathbf\{x\}\_\{0\}𝐱N\\mathbf\{x\}\_\{N\}attnffnϕ1\\phi\_\{1\}attnffnϕ2\\phi\_\{2\}attnffnϕ1\\phi\_\{1\}attnffnϕ2\\phi\_\{2\}attnffnϕ1\\phi\_\{1\}attnffnϕ2\\phi\_\{2\}roundr=1r\{=\}1roundr=2r\{=\}2roundr=3r\{=\}3R=3R\{=\}3rounds×\\timesK=2K\{=\}2blocks⇒\\;\\Rightarrow\\;unrolled depthN=K​R=6N\{=\}KR\{=\}6,M=2​N=12M\{=\}2N\{=\}12sublayer visits\(c\) Residual sublayer:𝐱i\+1=Norm​\(α​𝐱i\+fj​\(𝐱i;ϕj\)\)\\mathbf\{x\}\_\{i\+1\}=\\mathrm\{Norm\}\\bigl\(\\alpha\\,\\mathbf\{x\}\_\{i\}\+f\_\{j\}\(\\mathbf\{x\}\_\{i\};\\phi\_\{j\}\)\\bigr\)𝐱i\\mathbf\{x\}\_\{i\}\+\+Norm𝐱i\+1\\mathbf\{x\}\_\{i\+1\}fj​\(⋅;ϕj\)f\_\{j\}\(\\,\\cdot\\,;\\phi\_\{j\}\)skip×α\\times\\,\\alphainit gainβ\\beta\(d\) Scaling ruleDeepLoop\(p=12p=\\tfrac\{1\}\{2\}\)
α=\(2​N\)1/2\\alpha=\(2N\)^\{1/2\}
β=\(8​N\)−1/2\\beta=\(8N\)^\{\-1/2\}
Figure 1:Overview of the DeepLoop framework\.\(a\) Physical blocks:A looped Transformer storesK=2K\{=\}2physical blocks \(each with an attention and an FFN sublayer\), Block 1 \(ϕ1\\phi\_\{1\}\) and Block 2 \(ϕ2\\phi\_\{2\}\)\.\(b\) Unrolled execution:The wholeKK\-block stack is applied once per round forR=3R\{=\}3rounds, givingN=K​R=6N\{=\}KR\{=\}6unrolled blocks andM=2​N=12M\{=\}2N\{=\}12sublayer visits; theϕj\\phi\_\{j\}label shows which physical block is reused\.\(c\) Residual sublayer:Each visit applies𝐱i\+1=Norm​\(α​𝐱i\+fj​\(𝐱i;ϕj\)\)\\mathbf\{x\}\_\{i\+1\}\{=\}\\mathrm\{Norm\}\(\\alpha\\,\\mathbf\{x\}\_\{i\}\+f\_\{j\}\(\\mathbf\{x\}\_\{i\};\\phi\_\{j\}\)\), withα\\alphascaling the skip connection andβ\\betathe per\-matrix initialization gain\.\(d\) Scaling rule:DeepLoop setsα=\(2​N\)1/2\\alpha\{=\}\(2N\)^\{1/2\}andβ=\(8​N\)−1/2\\beta\{=\}\(8N\)^\{\-1/2\}\.## 1Introduction

Depth is one of the most reliable ways to improve Transformer expressivity, but in standard architectures, depth and parameter count grow together: adding a layer also adds a new set of attention and feed\-forward weights\(vaswani2017attention;kaplan2020scaling;hoffmann2022training\)\. Looped Transformers decouple these axes by applying a stack ofKKphysical blocks forRRrounds, yielding unrolled depthN=K​RN=KRwhile storing onlyKKblocks \(Figure[1](https://arxiv.org/html/2607.13491#S0.F1)\)\. This mechanism connects classical depth\-wise sharing in Universal Transformers, ALBERT, and Subformer\(dehghani2018universal;lan2019albert;reid2021subformer\)with recent recurrent\-depth and test\-time\-compute models that spend additional sequential computation without a proportional increase in parameters\(giannou2023looped;yang2023looped;gatmiry2024can;geiping2025scaling;saunshi2025reasoning\)\. For the looped depth to become a practical scaling axis, however, its residual parameterization must remain stable as the same physical blocks are revisited many times\.

The difficulty is that standard residual\-scaling analyses are written for untied depth\. Residual parameterization is decisive for deep Transformer optimization\(xiong2020layer;nguyen2019transformers;liu2020understanding;huang2020improving;bachlechner2021rezero;wang2024deepnet\)\. DeepNorm, in particular, makes very deep Post\-LN Transformers trainable by choosing a skip scaleα\\alphaand a residual\-branch initialization gainβ\\betaso that the first\-order effect of parameter updates remains bounded acrossM=2​NM=2Nresidual sublayer visits\(wang2024deepnet\)\. This calculation assumes that each unrolled residual sublayer owns a distinct parameter tensor and therefore contributes one independent update term\.

Weight sharing violates precisely this assumption\. When a physical sublayer is visitedRRtimes, its optimizer update aggregates the visit\-wise gradients from all rounds\. The updated tensor is then reused by all of those visits in the next linearized forward computation\. Tied depth therefore creates two coupled aggregation paths: the update is written by many visits and then read by many visits\. The size of this effect depends on visit alignment\. If visit\-wise gradients and sensitivities are nearly orthogonal across rounds, a looped model behaves like untied depth up to constants\. If they are coherent, the shared update can acquire an additional factor ofRR\.

This paper makes the tied\-depth effect explicit\. We introduce a visit\-alignment coefficientκR\\kappa\_\{R\}and show that the first\-order stability condition for a looped Transformer becomes

M​κR​\(βα\)2=O​\(1\),M\\kappa\_\{R\}\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=O\(1\),rather than the untied DeepNorm conditionM​\(β/α\)2=O​\(1\)M\(\\beta/\\alpha\)^\{2\}=O\(1\)\. For the scaling familyα=\(c​N\)p\\alpha=\(cN\)^\{p\}andβ=\(d​N\)−p\\beta=\(dN\)^\{\-p\}, the usual DeepNorm thresholdp=1/4p=1/4is recovered when visits decorrelate\. In the conservative aligned regime, whereκR=Θ​\(R\)\\kappa\_\{R\}=\\Theta\(R\)andKKis fixed whileRRgrows, the threshold becomesp=1/2p=1/2\.

The resulting method isDeepLoop: keep the DeepNorm Post\-LN architecture, but use the loop\-aware scaling rule

α=\(2​N\)1/2,β=\(8​N\)−1/2\.\\alpha=\(2N\)^\{1/2\},\\qquad\\beta=\(8N\)^\{\-1/2\}\.DeepLoop introduces no gates, learned residual coefficients, auxiliary losses, or architecture\-specific tuning constants\. It is a deterministic residual parameterization for the regime where effective depth is increased by revisiting shared blocks\.

We evaluate DeepLoop in controlled GPT\-style looped language modeling ablations at GPT\-2 small and GPT\-2 medium scales\. DeepLoop is essentially neutral atR=1R=1, where no physical block is revisited, and consistently improves final validation loss at larger loop counts\. The advantage transfers to an eight\-task language\-model evaluation suite: downstream averages are close atR=1R=1and move in favor of DeepLoop as recurrent depth grows\. These results support the central prediction of the analysis: residual scaling should depend on how depth is realized, not only on the nominal number of unrolled layers\.

Our contributions are:

1. \[leftmargin=\*, itemsep=1pt, topsep=1pt\]
2. 1\.We identify the tied\-depth aggregation mechanism that is absent from untied residual\-scaling analyses: a shared update is accumulated across repeated visits and then read through those same visits;
3. 2\.We derive a loop\-aware first\-order perturbation bound with an explicit visit\-alignment coefficientκR\\kappa\_\{R\}, recovering DeepNorm in the decorrelated regime and yielding ap=1/2p=1/2exponent threshold in the aligned fixed\-physical\-depth regime;
4. 3\.We propose DeepLoop, the scaling ruleα=\(2​N\)1/2\\alpha=\(2N\)^\{1/2\}andβ=\(8​N\)−1/2\\beta=\(8N\)^\{\-1/2\}, as a one\-line conservative parameterization for Post\-LN looped Transformers;
5. 4\.We provide controlled pre\-training and downstream ablations at GPT\-2 small and GPT\-2 medium scale, showing that the correction becomes useful precisely when the loop count is greater than one\.

## 2Background

### 2\.1Looped Transformers and effective depth

A standard depth\-NNTransformer appliesNNdistinct blocks once\. A looped Transformer instead choosesKKphysical blocks and applies them forRRrounds,

𝐱r,k\+1=Bk​\(𝐱r,k;ϕk\),k=1,…,K,r=1,…,R,𝐱r\+1,1=𝐱r,K\+1,\\mathbf\{x\}\_\{r,k\+1\}=B\_\{k\}\(\\mathbf\{x\}\_\{r,k\};\\phi\_\{k\}\),\\qquad k=1,\\ldots,K,\\quad r=1,\\ldots,R,\\qquad\\mathbf\{x\}\_\{r\+1,1\}=\\mathbf\{x\}\_\{r,K\+1\},\(1\)so the effective depth isN=K​RN=KRwhile the block parametersϕ1,…,ϕK\\phi\_\{1\},\\ldots,\\phi\_\{K\}are stored only once\. The Universal Transformer\(dehghani2018universal\)corresponds to the fully recurrent extreme in which the same transition is reused across depth; ALBERT\(lan2019albert\)similarly ties Transformer parameters across layers\. The regime studied here keepsKKfixed or small and increasesRR, thereby increasing test\-time compute and unrolled depth without increasing the number of physical blocks\.

### 2\.2Residual normalization

Letgℓg\_\{\\ell\}denote one residual sublayer, either attention or an MLP\. The two common normalization placements are

Pre\-LN:𝐱ℓ\+1=𝐱ℓ\+gℓ​\(Norm​\(𝐱ℓ\)\),Post\-LN:𝐱ℓ\+1=Norm​\(𝐱ℓ\+gℓ​\(𝐱ℓ\)\)\.\\displaystyle\\text\{Pre\-LN:\}\\quad\\mathbf\{x\}\_\{\\ell\+1\}=\\mathbf\{x\}\_\{\\ell\}\+g\_\{\\ell\}\(\\mathrm\{Norm\}\(\\mathbf\{x\}\_\{\\ell\}\)\),\\qquad\\text\{Post\-LN:\}\\quad\\mathbf\{x\}\_\{\\ell\+1\}=\\mathrm\{Norm\}\\left\(\\mathbf\{x\}\_\{\\ell\}\+g\_\{\\ell\}\(\\mathbf\{x\}\_\{\\ell\}\)\\right\)\.Pre\-LN improves optimization stability in deep Transformers, while Post\-LN can preserve a more expressive residual stream\(xiong2020layer;nguyen2019transformers\)\. DeepNorm\(wang2024deepnet\)modifies the Post\-LN residual path by scaling the skip connection before normalization:

𝐱ℓ\+1=Norm​\(α​𝐱ℓ\+gℓ​\(𝐱ℓ;θℓ\)\)\.\\displaystyle\\mathbf\{x\}\_\{\\ell\+1\}=\\mathrm\{Norm\}\\left\(\\alpha\\,\\mathbf\{x\}\_\{\\ell\}\+g\_\{\\ell\}\(\\mathbf\{x\}\_\{\\ell\};\\theta\_\{\\ell\}\)\\right\)\.\(2\)For an encoder\-only or decoder\-only Transformer withNNblocks andM=2​NM=2Nresidual sublayer applications, DeepNorm sets

α=\(2​N\)1/4,β=\(8​N\)−1/4,\\alpha=\(2N\)^\{1/4\},\\qquad\\beta=\(8N\)^\{\-1/4\},\(3\)Hereβ\\betais an initialization gain, not an additional runtime multiplier on the residual stream\. If𝒮ℓDN\\mathcal\{S\}^\{\\mathrm\{DN\}\}\_\{\\ell\}denotes the residual\-branch matrices scaled by DeepNet in sublayerℓ\\ell, such as the value and output projections in attention and the feed\-forward matrices, then DeepNorm initializes

Wℓ,q\(0\)=βDN​W~ℓ,q\(0\),q∈𝒮ℓDN,βDN=\(8​N\)−1/4,W\_\{\\ell,q\}^\{\(0\)\}=\\beta\_\{\\mathrm\{DN\}\}\\,\\widetilde\{W\}\_\{\\ell,q\}^\{\(0\)\},\\qquad q\\in\\mathcal\{S\}^\{\\mathrm\{DN\}\}\_\{\\ell\},\\qquad\\beta\_\{\\mathrm\{DN\}\}=\(8N\)^\{\-1/4\},\(4\)with the usual unscaled initializer used forW~ℓ,q\(0\)\\widetilde\{W\}\_\{\\ell,q\}^\{\(0\)\}\. The quantity that enters the perturbation argument is therefore

βDNαDN=\(8​N\)−1/4\(2​N\)1/4=12​N\.\\frac\{\\beta\_\{\\mathrm\{DN\}\}\}\{\\alpha\_\{\\mathrm\{DN\}\}\}=\\frac\{\(8N\)^\{\-1/4\}\}\{\(2N\)^\{1/4\}\}=\\frac\{1\}\{2\\sqrt\{N\}\}\.\(5\)The useful way to summarize the DeepNorm calculation is not a bound onα2​β\\alpha^\{2\}\\beta, but the first\-order update condition

M​\(βα\)2=Θ​\(1\)\.M\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=\\Theta\(1\)\.\(6\)Indeed, substituting Eq\. \([5](https://arxiv.org/html/2607.13491#S2.E5)\) gives2​N​\(β/α\)2=1/22N\(\\beta/\\alpha\)^\{2\}=1/2\.

### 2\.3Why weight sharing changes the depth scaling

In a non\-shared depth\-NNTransformer, each sublayer parameter receives one gradient contribution per input\. In a looped Transformer, the same physical parameter is visited once per round, so its gradient is a sum overRRvisits\. If the visit\-wise contributions decorrelate, the loop behaves like untied depth up to constants\. If the visits are aligned, the tied update can beRRtimes larger, and that same update is then read by allRRvisits in the unrolled computation\. DeepLoop is the corresponding conservative correction to Eq\. \([6](https://arxiv.org/html/2607.13491#S2.E6)\): it chooses a smaller update\-to\-residual ratioβ/α\\beta/\\alphaso that the first\-order effect of the tied parameter update remains bounded after allRRvisits are unrolled\.

## 3DeepLoop Transformer

A looped Transformer reuses the sameKKphysical blocks forRRrounds\. Its unrolled depth isN=K​RN=KR, and the number of residual\-sublayer visits isM=2​NM=2N\. DeepLoop keeps the DeepNorm architecture but uses the exponentp=1/2p=1/2rather thanp=1/4p=1/4in the fixed\-physical\-depth, increasing\-loop\-count regime\. The derivation separates two facts: RMSNorm exposes the residual branch through a factor1/α1/\\alpha, and parameter tying changes the first\-order update sum fromMMterms toM​κRM\\kappa\_\{R\}terms, whereκR\\kappa\_\{R\}measures alignment across loop visits\.

### 3\.1Setup: looped post\-normalized block

Each physical block contains two residual sublayers: attention and MLP\. Letj=\(k,s\)j=\(k,s\)index a physical sublayer, withk∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}ands∈\{attn,ffn\}s\\in\\\{\\mathrm\{attn\},\\mathrm\{ffn\}\\\}, and leti=\(r,j\)i=\(r,j\)denote itsrr\-th unrolled visit\. We writeJ=2​KJ=2Kfor the number of physical residual sublayers andM=J​R=2​K​R=2​NM=JR=2KR=2Nfor the number of unrolled visits\. DeepLoop uses

𝐱i\+1=Norm​\(α​𝐱i\+fj​\(𝐱i;ϕj\)\),i=1,…,M,\\displaystyle\\mathbf\{x\}\_\{i\+1\}=\\mathrm\{Norm\}\\left\(\\alpha\\,\\mathbf\{x\}\_\{i\}\+f\_\{j\}\(\\mathbf\{x\}\_\{i\};\\phi\_\{j\}\)\\right\),\\qquad i=1,\\ldots,M,where the same physical parameterϕj\\phi\_\{j\}is reused for allr=1,…,Rr=1,\\ldots,R\. In our implementation,Norm\\mathrm\{Norm\}is RMSNorm\. The gainβ\\betais a per\-matrix initialization gain applied to the residual\-branch matrices specified by DeepNorm\. It should not be read as the standard deviation of the whole sublayer output, since a branch may contain multiple scaled matrices\. Concretely, for each physical sublayerjj, let𝒮j\\mathcal\{S\}\_\{j\}be the set of matrices that receive the DeepNorm gain\. DeepLoop initializes

Wj,q\(0\)=βDL​W~j,q\(0\),q∈𝒮j,βDL=\(8​N\)−1/2\.W\_\{j,q\}^\{\(0\)\}=\\beta\_\{\\mathrm\{DL\}\}\\,\\widetilde\{W\}\_\{j,q\}^\{\(0\)\},\\qquad q\\in\\mathcal\{S\}\_\{j\},\\qquad\\beta\_\{\\mathrm\{DL\}\}=\(8N\)^\{\-1/2\}\.\(7\)The forward recurrence then uses these initialized parameters throughfj​\(𝐱i;ϕj\)f\_\{j\}\(\\mathbf\{x\}\_\{i\};\\phi\_\{j\}\)at every visit;β\\betais not re\-applied as a separate multiplicative factor on each visit\. WithαDL=\(2​N\)1/2\\alpha\_\{\\mathrm\{DL\}\}=\(2N\)^\{1/2\}, the DeepLoop update\-to\-residual ratio is

βDLαDL=\(8​N\)−1/2\(2​N\)1/2=14​N\.\\frac\{\\beta\_\{\\mathrm\{DL\}\}\}\{\\alpha\_\{\\mathrm\{DL\}\}\}=\\frac\{\(8N\)^\{\-1/2\}\}\{\(2N\)^\{1/2\}\}=\\frac\{1\}\{4N\}\.\(8\)
\{assumption\}

\[DeepNorm\-scale local sensitivity\] For each unrolled visiti=\(r,j\)i=\(r,j\), the norm of the linearized output map from a residual\-branch parameter perturbation at that visit and the norm of the corresponding visit\-wise effective update are bothO​\(β/α\)O\(\\beta/\\alpha\)\.

Assumption[3\.1](https://arxiv.org/html/2607.13491#S3.SS1)is the local scaling condition used in the DeepNet perturbation argument\. Constants depending on width, normalization gain, attention heads, learning rate, or optimizer preconditioning are absorbed into theO​\(⋅\)O\(\\cdot\)notation\. The looped analysis below changes only how the visit\-wise terms are aggregated when parameters are tied\.

### 3\.2Forward stability

RMSNorm restores unit RMS at every sublayer, so the forward signal scale is not the binding constraint; the relevant constraint is the first\-order sensitivity of the final output to a residual\-branch parameter update\. A direct expansion ofRMSNorm​\(α​𝐱\+𝐳\)\\mathrm\{RMSNorm\}\(\\alpha\\mathbf\{x\}\+\\mathbf\{z\}\)aroundRMS​\(𝐳\)/α→0\\mathrm\{RMS\}\(\\mathbf\{z\}\)/\\alpha\\to 0shows that the residual branch enters the normalized direction through a factor of1/α1/\\alpha, which is why the relevant DeepNorm ratio isβ/α\\beta/\\alpharather thanβ\\betaalone\. The formal statement and proof are Lemma[A\.1](https://arxiv.org/html/2607.13491#A1.Thmtheorem1)in Appendix[A](https://arxiv.org/html/2607.13491#A1)\.

### 3\.3Depth\-untied DeepNorm bound

The DeepNet perturbation argument bounds the first\-order output changeΔ​F=F​\(𝐱;θ\+δ​θ\)−F​\(𝐱;θ\)\\Delta F=F\(\\mathbf\{x\};\\theta\+\\delta\\theta\)\-F\(\\mathbf\{x\};\\theta\)by summing visit\-wise sensitivities\. Here*depth\-untied*means that residual\-sublayer parameters are not shared across unrolled depth\. This is orthogonal to the standard tying of input and output token embeddings: tied embeddings, when used, are outside theM=2​NM=2Nresidual\-sublayer count and are not among the residual\-branch matrices scaled byβ\\beta\. Under Assumption[3\.1](https://arxiv.org/html/2607.13491#S3.SS1), each visit contributes an output sensitivityO​\(β/α\)O\(\\beta/\\alpha\)and an effective updateO​\(β/α\)O\(\\beta/\\alpha\), so summingM=2​NM=2Ndepth\-untied residual\-sublayer visits gives

‖Δ​F‖≤C′​M​\(βα\)2,\\\|\\Delta F\\\|\\leq C^\{\\prime\}\\,M\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\},\(9\)and a sufficient first\-order stability condition is

M​\(βα\)2=O​\(1\)\.M\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=O\(1\)\.\(10\)The DeepNorm choiceα=\(2​N\)1/4\\alpha=\(2N\)^\{1/4\}andβ=\(8​N\)−1/4\\beta=\(8N\)^\{\-1/4\}satisfiesM​\(β/α\)2=1/2M\(\\beta/\\alpha\)^\{2\}=1/2\. The formal proposition statement and proof are Proposition[A\.3](https://arxiv.org/html/2607.13491#A1.Thmtheorem3)in Appendix[A](https://arxiv.org/html/2607.13491#A1)\.

### 3\.4Loop correction: aligned visits require\\texorpdfstringp=1/2p=1/2p=1/2

Depth\-wise residual\-sublayer weight sharing changes Eq\. \([9](https://arxiv.org/html/2607.13491#S3.E9)\)\. For a physical sublayerjj, letGr,jG\_\{r,j\}be the visit\-wise effective update that would be applied if visit\(r,j\)\(r,j\)had its own depth\-untied residual\-branch parameter, and letUr,jU\_\{r,j\}be the corresponding linearized output\-sensitivity operator\. With tied residual\-sublayer parameters, the update toϕj\\phi\_\{j\}is proportional to∑r=1RGr,j\\sum\_\{r=1\}^\{R\}G\_\{r,j\}, and the output perturbation reads this same update through all visits\. The first\-order tied perturbation therefore has the schematic double\-sum form

Δ​Ftied=−η​∑j=1J\(∑r=1RUr,j\)​\(∑t=1RGt,j\)\+O​\(η2\),\\Delta F\_\{\\mathrm\{tied\}\}=\-\\eta\\sum\_\{j=1\}^\{J\}\\left\(\\sum\_\{r=1\}^\{R\}U\_\{r,j\}\\right\)\\left\(\\sum\_\{t=1\}^\{R\}G\_\{t,j\}\\right\)\+O\(\\eta^\{2\}\),\(11\)where tensor contractions and optimizer\-dependent constants are absorbed intoUr,jU\_\{r,j\}andGr,jG\_\{r,j\}\.

Define the visit\-alignment coefficient

κR:=maxj∈\{1,…,J\}⁡‖∑r=1RUr,j‖​‖∑r=1RGr,j‖R​CU​CG​\(β/α\)2,\\kappa\_\{R\}:=\\max\_\{j\\in\\\{1,\\ldots,J\\\}\}\\frac\{\\left\\\|\\sum\_\{r=1\}^\{R\}U\_\{r,j\}\\right\\\|\\left\\\|\\sum\_\{r=1\}^\{R\}G\_\{r,j\}\\right\\\|\}\{RC\_\{U\}C\_\{G\}\(\\beta/\\alpha\)^\{2\}\},\(12\)where‖Ur,j‖≤CU​β/α\\\|U\_\{r,j\}\\\|\\leq C\_\{U\}\\beta/\\alphaand‖Gr,j‖≤CG​β/α\\\|G\_\{r,j\}\\\|\\leq C\_\{G\}\\beta/\\alpha\. The triangle inequality gives0≤κR≤R0\\leq\\kappa\_\{R\}\\leq R\. Independent or nearly orthogonal visits giveκR=O​\(1\)\\kappa\_\{R\}=O\(1\), while fully aligned visits giveκR=Θ​\(R\)\\kappa\_\{R\}=\\Theta\(R\)\.

Substituting the definition ofκR\\kappa\_\{R\}from Eq\. \([12](https://arxiv.org/html/2607.13491#S3.E12)\) into Eq\. \([11](https://arxiv.org/html/2607.13491#S3.E11)\) and applying submultiplicativity gives the looped version of the DeepNorm bound,

‖Δ​F‖≤C′′​M​κR​\(βα\)2,\\\|\\Delta F\\\|\\leq C^\{\\prime\\prime\}\\,M\\kappa\_\{R\}\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\},\(13\)so a sufficient tied\-depth stability condition is

M​κR​\(βα\)2=O​\(1\)\.M\\kappa\_\{R\}\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=O\(1\)\.\(14\)In the worst\-case aligned regimeκR=Θ​\(R\)\\kappa\_\{R\}=\\Theta\(R\), this reduces toM​R​\(β/α\)2=O​\(1\)MR\(\\beta/\\alpha\)^\{2\}=O\(1\)\. The formal proposition and the aligned\-case corollary are Proposition[A\.5](https://arxiv.org/html/2607.13491#A1.Thmtheorem5)and Corollary[A\.6](https://arxiv.org/html/2607.13491#A1.Thmtheorem6)in Appendix[A](https://arxiv.org/html/2607.13491#A1)\.

Now consider the familyα=\(c​N\)p\\alpha=\(cN\)^\{p\},β=\(d​N\)−p\\beta=\(dN\)^\{\-p\}with constantsc,d\>0c,d\>0\. Its update\-to\-residual ratio is

βα=\(c​d\)−p​N−2​p\.\\frac\{\\beta\}\{\\alpha\}=\(cd\)^\{\-p\}N^\{\-2p\}\.\(15\)
###### Proposition 3\.1\(Exponent threshold\)\.

AssumeKKis fixed,N=K​RN=KR,M=2​NM=2N, andκR=Θ​\(Rγ\)\\kappa\_\{R\}=\\Theta\(R^\{\\gamma\}\)for someγ∈\[0,1\]\\gamma\\in\[0,1\]\. For the scaling familyα=\(c​N\)p\\alpha=\(cN\)^\{p\},β=\(d​N\)−p\\beta=\(dN\)^\{\-p\}, the tied\-depth condition

M​κR​\(β/α\)2=O​\(1\)M\\kappa\_\{R\}\(\\beta/\\alpha\)^\{2\}=O\(1\)holds uniformly asR→∞R\\to\\inftyif and only if

p≥1\+γ4\.p\\geq\\frac\{1\+\\gamma\}\{4\}\.

Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)shows where the new exponent comes from\. When visits decorrelate,γ=0\\gamma=0, and the DeepNorm thresholdp=1/4p=1/4is recovered\. In the conservative aligned case,γ=1\\gamma=1, so the threshold isp=1/2p=1/2\. Equivalently,

M​R​\(βα\)2=2​N​R​\(c​d\)−2​p​N−4​p\.MR\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=2NR\\,\(cd\)^\{\-2p\}N^\{\-4p\}\.\(16\)At fixedKK,R=N/KR=N/K, so Eq\. \([16](https://arxiv.org/html/2607.13491#S3.E16)\) is bounded only whenp≥1/2p\\geq 1/2\. DeepNorm’sp=1/4p=1/4leaves a residual growth ofΘ​\(R\)\\Theta\(R\)in the aligned shared\-loop bound\. The DeepLoop choice is

α=\(2​N\)1/2,β=\(8​N\)−1/2\.\\displaystyle\\alpha=\(2N\)^\{1/2\},\\qquad\\beta=\(8N\)^\{\-1/2\}\.We keep the DeepNorm constants\(c,d\)=\(2,8\)\(c,d\)=\(2,8\)for a two\-sublayer decoder block and change only the exponent\. Equivalently, DeepLoop uses the same per\-matrix definition as Eq\. \([7](https://arxiv.org/html/2607.13491#S3.E7)\) withβ=βDL\\beta=\\beta\_\{\\mathrm\{DL\}\}\. With this choice,

βα=14​N,M​R​\(βα\)2=2​N​R⋅116​N2=18​K,\\frac\{\\beta\}\{\\alpha\}=\\frac\{1\}\{4N\},\\qquad MR\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=2NR\\cdot\\frac\{1\}\{16N^\{2\}\}=\\frac\{1\}\{8K\},so the worst\-case aligned tied\-depth bound remainsO​\(1\)O\(1\)at fixed physical depthKK\. Proofs of Proposition[A\.5](https://arxiv.org/html/2607.13491#A1.Thmtheorem5)and Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)are in Appendix[A](https://arxiv.org/html/2607.13491#A1)\. Appendix[C](https://arxiv.org/html/2607.13491#A3)sweepsppover a single grid atR=3R\{=\}3and finds an empirical training\-stability boundary that lines up with thisp=1/2p\{=\}1/2prediction\.

### 3\.5Summary and comparison to DeepNorm

DeepLoop is therefore a one\-line conservative correction for tied depth: increase the exponent fromp=1/4p=1/4top=1/2p=1/2\. It introduces no gates, learnable residual scalars, or additional tuning constants\. The correction is not a claim that every looped model empirically attains worst\-case alignment\. Rather, it is the minimal exponent for uniform boundedness under aligned visit\-wise contributions at fixed physical depth\.

## 4Application to Hierarchical Recurrent Reasoners

The DeepLoop derivation in Section[3](https://arxiv.org/html/2607.13491#S3)treats a single looped block visited forRRrounds\. Recent recurrent\-reasoning architectures combine looping with two further structural choices: a hierarchy of recurrent modules executed at different rates, and training\-time gradient truncation across the outer recurrence\(wang2025hrm;geiping2025scaling\)\. The Hierarchical Reasoning Model\(wang2025hrm\)instantiates both: a high\-level moduleℋ\\mathcal\{H\}and a low\-level moduleℒ\\mathcal\{L\}are unrolled jointly, and gradients are computed through only the last outer cycle\. This section shows that the framework of Section[3](https://arxiv.org/html/2607.13491#S3)extends to this regime by replacing the unrolled visit countMMwith a gradient\-visible visit countMgM\_\{\\mathrm\{g\}\}and tracking per\-module visit counts separately\. We do not introduce new optimization machinery or a new variant of DeepLoop; we describe how the existing analysis specializes when training\-time depth and forward depth differ\.

### 4\.1Setup: two\-module hierarchical loops

Letℋ\\mathcal\{H\}containKHK\_\{H\}physical blocks \(each with two residual sublayers, attention and MLP\) and letℒ\\mathcal\{L\}containKLK\_\{L\}physical blocks of the same shape\. The hierarchical recurrence runsCCouter cycles; within each outer cycle,ℒ\\mathcal\{L\}iteratesCLC\_\{L\}times before a single update ofℋ\\mathcal\{H\}:

cyclec:𝐳L←ℒ\(CL\)\(𝐳L,𝐳H\),𝐳H←ℋ\(𝐳H,𝐳L\),c=1,…,C\.\\text\{cycle \}c:\\quad\\mathbf\{z\}\_\{L\}\\leftarrow\\mathcal\{L\}^\{\(C\_\{L\}\)\}\(\\mathbf\{z\}\_\{L\},\\mathbf\{z\}\_\{H\}\),\\qquad\\mathbf\{z\}\_\{H\}\\leftarrow\\mathcal\{H\}\(\\mathbf\{z\}\_\{H\},\\mathbf\{z\}\_\{L\}\),\\quad c=1,\\ldots,C\.Counting attention and MLP residual sublayers, the total unrolled visit count is

M=2​C​\(KH\+CL​KL\),M=2\\,C\\,\(K\_\{H\}\+C\_\{L\}K\_\{L\}\),\(17\)which generalizes the single\-module countM=2​K​RM=2KRused in Section[3](https://arxiv.org/html/2607.13491#S3)\. The set of distinct physical residual sublayers has sizeJ=JH\+JLJ=J\_\{H\}\+J\_\{L\}withJH=2​KHJ\_\{H\}=2K\_\{H\}andJL=2​KLJ\_\{L\}=2K\_\{L\}\. Each visiti=\(c,ℓ,j\)i=\(c,\\ell,j\)has an outer\-cycle indexcc, an innerℒ\\mathcal\{L\}\-iterate indexℓ\\ell\(only meaningful forj∈ℒj\\in\\mathcal\{L\}\), and a physical\-sublayer indexjj\.

### 4\.2Loop\-aware bound under one\-step gradient approximation

Hierarchical recurrent reasoners are typically trained with a one\-step gradient approximation: the forward pass uses allCCouter cycles, but the backward pass is computed against the last cycle only, with𝐳H\\mathbf\{z\}\_\{H\}and𝐳L\\mathbf\{z\}\_\{L\}from earlier cycles detached from the autograd graph\. This truncation does not alter the numerical forward values, so Lemma[A\.1](https://arxiv.org/html/2607.13491#A1.Thmtheorem1)applies unchanged locally\. The perturbation bound below is for the truncated training graph: the states entering the last outer cycle are treated as fixed inputs, and both the sensitivity terms and the update terms from earlier cycles are absent\. A perturbation bound for the full untruncated forward map after an optimizer step would require an additional forward\-visible read factor\.

Define the gradient\-visible visit count

Mg:=2​\(KH\+CL​KL\),M\_\{\\mathrm\{g\}\}:=2\\,\(K\_\{H\}\+C\_\{L\}K\_\{L\}\),\(18\)soM=C​MgM=C\\,M\_\{\\mathrm\{g\}\}\. The per\-physical\-sublayer round counts on the gradient\-visible cycle are

Rg\(ℋ\)=1,Rg\(ℒ\)=CL\.R\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\}=1,\\qquad R\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}=C\_\{L\}\.\(19\)Repeating the argument leading to Eq\. \([13](https://arxiv.org/html/2607.13491#S3.E13)\) but restricting the visit\-index sums to the gradient\-visible cycle gives the two\-module loop\-aware bound

‖Δ​F‖≤C′′​\[JH​Rg\(ℋ\)​κg\(ℋ\)\+JL​Rg\(ℒ\)​κg\(ℒ\)\]​\(βα\)2,\\\|\\Delta F\\\|\\leq C^\{\\prime\\prime\}\\\!\\left\[J\_\{H\}\\,R\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\}\\,\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\}\+J\_\{L\}\\,R\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}\\,\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}\\right\]\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\},\(20\)whereκg\(ℋ\),κg\(ℒ\)\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\},\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}are the visit\-alignment coefficients of Eq\. \([12](https://arxiv.org/html/2607.13491#S3.E12)\) restricted to each module’s gradient\-visible visits\. Two structural facts simplify Eq\. \([20](https://arxiv.org/html/2607.13491#S4.E20)\)\. First,κg\(ℋ\)≤1\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\}\\leq 1trivially: withRg\(ℋ\)=1R\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\}=1, each physicalℋ\\mathcal\{H\}\-sublayer contributes a single\-term inner sum, and the alignment coefficient collapses to a constant\. The truncation thus placesℋ\\mathcal\{H\}in the untied\-DeepNorm regime even thoughℋ\\mathcal\{H\}is forward\-visitedCCtimes\. Second,κg\(ℒ\)∈\[0,CL\]\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}\\in\[0,C\_\{L\}\]by the triangle inequality applied to the innerℒ\\mathcal\{L\}\-iterate sum\. Substituting these facts, upper\-boundingκg\(ℋ\)\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{H\}\)\}by one, and applying the per\-module counts in Eq\. \([19](https://arxiv.org/html/2607.13491#S4.E19)\) yields the sufficient stability condition

Mg​κ¯g​\(βα\)2=O​\(1\),κ¯g:=JH\+JL​CL​κg\(ℒ\)Mg,M\_\{\\mathrm\{g\}\}\\,\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}\\,\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=O\(1\),\\qquad\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}:=\\frac\{J\_\{H\}\+J\_\{L\}\\,C\_\{L\}\\,\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}\}\{M\_\{\\mathrm\{g\}\}\},\(21\)which has the same structural form as Eq\. \([14](https://arxiv.org/html/2607.13491#S3.E14)\) withMMreplaced byMgM\_\{\\mathrm\{g\}\}andκR\\kappa\_\{R\}replaced by the gradient\-visible aggregateκ¯g∈\[JH/Mg,1\+JL​CL​\(CL−1\)/Mg\]\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}\\in\[\\,J\_\{H\}/M\_\{\\mathrm\{g\}\},\\,1\+J\_\{L\}C\_\{L\}\(C\_\{L\}\-1\)/M\_\{\\mathrm\{g\}\}\\,\]\.

The decomposition \([20](https://arxiv.org/html/2607.13491#S4.E20)\) also licenses a per\-module scaling family\. Each summand depends only on the\(α,β\)\(\\alpha,\\beta\)used at the corresponding module, so a hierarchical reasoner may be parameterized with\(αℋ,βℋ\)\(\\alpha\_\{\\mathcal\{H\}\},\\beta\_\{\\mathcal\{H\}\}\)and\(αℒ,βℒ\)\(\\alpha\_\{\\mathcal\{L\}\},\\beta\_\{\\mathcal\{L\}\}\)chosen independently\. The admissible exponent at each module is determined by the corresponding summand of Eq\. \([20](https://arxiv.org/html/2607.13491#S4.E20)\), not by the global visit count\.

### 4\.3Predicted exponent regime

LetNg:=Mg/2N\_\{\\mathrm\{g\}\}:=M\_\{\\mathrm\{g\}\}/2denote the gradient\-visible block\-equivalent depth\. Combining Eq\. \([21](https://arxiv.org/html/2607.13491#S4.E21)\) with the same two\-sublayer scaling family used above,α=\(c​Ng\)p\\alpha=\(cN\_\{\\mathrm\{g\}\}\)^\{p\},β=\(d​Ng\)−p\\beta=\(dN\_\{\\mathrm\{g\}\}\)^\{\-p\}, gives, up to constant factors,

Mg​κ¯g​\(c​d\)−2​p​Ng−4​p=Θ​\(\(c​d\)−2​p​κ¯g​Mg1−4​p\),M\_\{\\mathrm\{g\}\}\\,\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}\\,\(cd\)^\{\-2p\}\\,N\_\{\\mathrm\{g\}\}^\{\-4p\}=\\Theta\\\!\\left\(\(cd\)^\{\-2p\}\\,\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}\\,M\_\{\\mathrm\{g\}\}^\{1\-4p\}\\right\),\(22\)and the asymptotic threshold for boundedness depends on howκ¯g\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}andMgM\_\{\\mathrm\{g\}\}co\-scale\. We record the two limit cases that mirror Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)\.

#### Decorrelated inner cycles \(κg\(ℒ\)=O​\(1\)\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}=O\(1\)\)\.

Thenκ¯g=O​\(1\)\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}=O\(1\)and Eq\. \([22](https://arxiv.org/html/2607.13491#S4.E22)\) is bounded asMg→∞M\_\{\\mathrm\{g\}\}\\to\\inftyif and only ifp≥1/4\.p\\geq 1/4\.Both modules sit in the untied\-DeepNorm regime: theℋ\\mathcal\{H\}module by truncation, and theℒ\\mathcal\{L\}module by the inner\-cycle decorrelation\.

#### Aligned inner cycles \(κg\(ℒ\)=Θ​\(CL\)\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}=\\Theta\(C\_\{L\}\)\)\.

Then theℒ\\mathcal\{L\}contribution toκ¯g\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}isΘ​\(CL2​JL/Mg\)\\Theta\(C\_\{L\}^\{2\}\\,J\_\{L\}/M\_\{\\mathrm\{g\}\}\)\. Two sub\-cases follow from howMgM\_\{\\mathrm\{g\}\}is grown\.*\(i\)*IfCLC\_\{L\}is fixed andKH,KL→∞K\_\{H\},K\_\{L\}\\to\\infty, thenκ¯g=Θ​\(1\)\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}=\\Theta\(1\)and the threshold is againp≥1/4p\\geq 1/4\.*\(ii\)*IfKH,KLK\_\{H\},K\_\{L\}are fixed andCL→∞C\_\{L\}\\to\\infty, thenMg=Θ​\(CL\)M\_\{\\mathrm\{g\}\}=\\Theta\(C\_\{L\}\)andκ¯g=Θ​\(CL\)\\bar\{\\kappa\}\_\{\\mathrm\{g\}\}=\\Theta\(C\_\{L\}\), so Eq\. \([22](https://arxiv.org/html/2607.13491#S4.E22)\) is bounded if and only ifp≥1/2,p\\geq 1/2,which recovers the fixed\-physical\-depth tied\-loop threshold of Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)along the inner\-cycle axis\.

For a hierarchical recurrent reasoner as actually built, these two limits are not equally likely possibilities but a binding constraint and a slack one\. The once\-per\-stepℋ\\mathcal\{H\}module is truncated by the one\-step gradient, contributes a single update term, and therefore requires onlyp≥1/4p\\geq 1/4\. Theℒ\\mathcal\{L\}module is the one that realizes the model’s effective depth: it revisits its shared blocks at fixed physical depth, theCLC\_\{L\}axis of sub\-case \(ii\), and weight tying is adopted precisely so that those revisits implement the*same*operation, which aligns their visit\-wise gradients and sensitivities \(κg\(ℒ\)=Θ​\(CL\)\\kappa\_\{\\mathrm\{g\}\}^\{\(\\mathcal\{L\}\)\}=\\Theta\(C\_\{L\}\)\)\. Theℒ\\mathcal\{L\}module therefore requiresp≥1/2p\\geq 1/2\. A single shared residual exponent must satisfy the more demanding module, so the framework predicts

for a hierarchical recurrent reasoner, the same loop\-aware exponent as the single\-module backbone of Section[3](https://arxiv.org/html/2607.13491#S3)\. This is a statement about the architecture as trained, not a conservative envelope: the recurrent depth that makes the model work is grown along the aligned inner\-cycle axis, exactly the regime whose threshold is1/21/2, andp=1/2p=1/2simultaneously satisfies the once\-per\-step module \(for which1/2≥1/41/2\\geq 1/4\)\. Section[5](https://arxiv.org/html/2607.13491#S5)confirms the prediction empirically on ARC\-AGI\.

#### Forward signal at the loop entry\.

A practical detail of hierarchical reasoners is the addition of a learned task or puzzle embedding𝐞task\\mathbf\{e\}\_\{\\mathrm\{task\}\}to the token embedding𝐞tok\\mathbf\{e\}\_\{\\mathrm\{tok\}\}before the loop\. The choice of whether to apply RMSNorm to𝐞tok\+𝐞task\\mathbf\{e\}\_\{\\mathrm\{tok\}\}\+\\mathbf\{e\}\_\{\\mathrm\{task\}\}at the loop entry affects only the forward signal scaleRMS​\(𝐱0\)\\mathrm\{RMS\}\(\\mathbf\{x\}\_\{0\}\)in Lemma[A\.1](https://arxiv.org/html/2607.13491#A1.Thmtheorem1); it does not enter Eq\. \([20](https://arxiv.org/html/2607.13491#S4.E20)\) or the threshold derivation \([22](https://arxiv.org/html/2607.13491#S4.E22)\)\. Input\-side normalization and the residual scaling rule are therefore separable design choices within this framework, even though both can independently influence the relative magnitude of task\-conditioning information across repeated visits\.

### 4\.4Summary

The DeepLoop perturbation argument extends to hierarchical recurrent reasoners by two substitutions\. When training uses a one\-step gradient approximation, the visit count entering the bound is the gradient\-visibleMgM\_\{\\mathrm\{g\}\}from Eq\. \([18](https://arxiv.org/html/2607.13491#S4.E18)\) rather than the forwardMMfrom Eq\. \([17](https://arxiv.org/html/2607.13491#S4.E17)\)\. When the architecture splits the recurrence across modules with different per\-step visit counts, the bound decomposes into per\-module summands and admits asymmetric per\-module\(α,β\)\(\\alpha,\\beta\)assignments\. Because a hierarchical recurrent reasoner grows its effective depth by aligned revisits of the shared inner module at fixed physical depth, the binding threshold from Eq\. \([22](https://arxiv.org/html/2607.13491#S4.E22)\) isp=1/2p=1/2, the same loop\-aware exponent as the single\-module backbone\. Section[5](https://arxiv.org/html/2607.13491#S5)confirms this on ARC\-AGI: the DeepLoop exponentp=1/2p=1/2improves voted accuracy over the vanilla HRM baseline across the full evaluation ladder\.

## 5Experiments

### 5\.1Validation loss

We compare DeepLoop against a baseline that already incorporates the same looped residual\-block sharing, tied input\-output embeddings, and an input\-embedding RMSNorm\. The baseline \(base\) uses this same looped backbone and embedding tying, but keeps the residual and initialization scales atα=1\\alpha=1andβ=1\\beta=1\. The DeepLoop variant additionally enables the scaling rule on every block\. Both configurations share an identical backbone, optimizer, data pipeline, and random seed; the only knob that moves between rows of Table[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)is the loop countR∈\{1,3,5,7\}R\\in\\\{1,3,5,7\\\}, and the only knob that moves between methods is the activation of DeepLoop\.

The GPT\-2 small runs use a GPT\-MHA\-RoPE model trained on FineWeb\-Edu for 50B tokens \(100K optimizer steps, context length 1024, global batch size 480\) on 4×\\timesH200 141GB GPUs\. The GPT\-2 medium runs use the same backbone with hidden size enlarged from768768to10241024and the number of layers doubled from1212to2424, trained on the same data and schedule on 8×\\timesH200 141GB GPUs\. We report the validation cross\-entropy at the final checkpoint for both scales in Table[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)\.

Table 1:Final validation loss at step 100,000 on FineWeb\-Edu 50BT for the GPT\-2 small and GPT\-2 medium backbones, varying loop countR∈\{1,3,5,7\}R\\in\\\{1,3,5,7\\\}\. Bold marks the best cell per column per scale; ties within0\.0020\.002nats are bolded on both rows\. DeepLoop is effectively tied with the baseline atR=1R\{=\}1and improves over it in these single\-seed runs atR=3R\{=\}3,R=5R\{=\}5, andR=7R\{=\}7at both scales\.Figure[2](https://arxiv.org/html/2607.13491#S5.F2)visualizes the same numbers as Table[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)\. At both scales the two curves coincide within noise atR=1R\{=\}1\(\+0\.0004\+0\.0004and\+0\.0011\+0\.0011nats\) and separate as the loop count grows: at the small scale DeepLoop pulls ahead by−0\.016\-0\.016nats atR=3R\{=\}3, the gap peaks at−0\.023\-0\.023nats atR=5R\{=\}5, and remains open at−0\.019\-0\.019nats atR=7R\{=\}7; at the medium scale the gap widens monotonically throughR=7R\{=\}7, reaching−0\.028\-0\.028nats\. Both methods continue to improve monotonically withRRat both scales, and DeepLoop strictly beats the baseline at everyR≥3R\\geq 3\. Multi\-seed runs would be needed to quantify run\-to\-run variance\.

![Refer to caption](https://arxiv.org/html/2607.13491v1/x1.png)Figure 2:Final validation loss \(step 100,000\) against loop countR∈\{1,3,5,7\}R\\in\\\{1,3,5,7\\\}on FineWeb\-Edu 50BT for the GPT\-2 small \(left\) and GPT\-2 medium \(right\) backbones\. DeepLoop matches the baseline atR=1R\{=\}1and improves over it at everyR≥3R\\geq 3at both scales in these single\-seed runs\. Data are the same as Table[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)\.
### 5\.2Downstream evaluation on thelm\-evaluation\-harnesssuite

To test whether the validation\-loss advantage transfers to downstream task accuracy, we evaluate the eight medium\-scale checkpoints from §[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)on the eight\-task zero\- and one\-shot suite used byzhang2026deepin the Deep Delta Learning paper:arc\_challenge,arc\_easy,hellaswag,openbookqa,piqa,sciq,social\_iqa, andwinogrande\. Evaluation uses thelm\-evaluation\-harness\(eval\-harness\)with its defaultaccmetric \(notacc\_norm\), the GPT\-2 tokenizer, a context length of 1024 \(matching training\), and bf16 inference\. The reported*Avg*column is the unweighted arithmetic mean of the eight per\-task accuracies, following the protocol ofzhang2026deep\.

Table[5\.2](https://arxiv.org/html/2607.13491#S5.SS2)reports the per\-task accuracy and Avg for every \(method,RR, shot\) cell at the medium scale\. Figure[3](https://arxiv.org/html/2607.13491#S5.F3)breaks the 1\-shot column out task\-by\-task\. The corresponding GPT\-2 small downstream results, which exhibit the same qualitative pattern, are deferred to Appendix[B](https://arxiv.org/html/2607.13491#A2)\.

Table 2:Downstream accuracy \(%\) for the GPT\-2 medium backbone on FineWeb\-Edu 50BT, mirroring Table[B](https://arxiv.org/html/2607.13491#A2)but extended toR=7R\{=\}7\. Metric isacc; Avg is the unweighted arithmetic mean across the eight tasks\. Bold marks the best cell per column per shot setting\.![Refer to caption](https://arxiv.org/html/2607.13491v1/x2.png)Figure 3:Per\-task 1\-shot accuracy against loop countRRfor base versus DeepLoop on all eight tasks at the GPT\-2 medium scale\. AtR=7R\{=\}7DeepLoop wins on seven of the eight tasks under both shot settings; see Table[5\.2](https://arxiv.org/html/2607.13491#S5.SS2)for exact values\.
Two takeaways\. First, the qualitative picture from the small\-scale ablation \(Appendix Table[B](https://arxiv.org/html/2607.13491#A2)\) carries over to the larger backbone: the two methods are tied atR=1R\{=\}1\(Δ=−0\.01\\Delta=\-0\.01on the 0\-shot Avg,−0\.16\-0\.16on 1\-shot\), DeepLoop opens the gap atR=3R\{=\}3\(\+0\.43\+0\.43/\+0\.32\+0\.32\), and the best Avg cell in the table is DeepLoopR=7R\{=\}7in both shot settings, headlined by55\.20%55\.20\\,\\%on 1\-shot\. On the 0\-shot Avg the gap widens throughR=7R\{=\}7\(\+0\.43\+0\.43,\+1\.06\+1\.06,\+0\.93\+0\.93\); on the 1\-shot Avg the ranking is messier atR=5R\{=\}5\(where the base reaches54\.42%54\.42\\,\\%,0\.230\.23points above DeepLoop\) but DeepLoop recovers the lead by\+0\.58\+0\.58points atR=7R\{=\}7\. Second, atR=7R\{=\}7DeepLoop beats the same\-RRbaseline on seven of the eight individual tasks under both shot settings \(only PIQA goes the other way\), with WinoGrande in particular jumping by\+1\.74\+1\.74points 0\-shot \(59\.0459\.04vs\.57\.3057\.30\)\. The same caveat about single\-seed run\-to\-run variance from §[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)applies, we see no qualitative disagreement with the small\-scale ablation, only a stronger depth signal\.

### 5\.3Reasoning evaluation on ARC\-AGI

To test the prediction of Eq\. \([23](https://arxiv.org/html/2607.13491#S4.E23)\) on a hierarchical recurrent reasoner, we apply the DeepLoop scaling rule withp=1/2p=1/2to the Hierarchical Reasoning Model \(HRM\)\(wang2025hrm\)and evaluate on ARC\-AGI\-1\(chollet2019measure\)\. The only change from the published HRM is the residual parameterization: every reasoning block usesα=\(2​Ng\)1/2\\alpha=\(2N\_\{\\mathrm\{g\}\}\)^\{1/2\}andβ=\(8​Ng\)−1/2\\beta=\(8N\_\{\\mathrm\{g\}\}\)^\{\-1/2\}withMg=24M\_\{\\mathrm\{g\}\}=24gradient\-visible residual\-sublayer visits andNg=Mg/2=12N\_\{\\mathrm\{g\}\}=M\_\{\\mathrm\{g\}\}/2=12block\-equivalent depth \(Eq\.[18](https://arxiv.org/html/2607.13491#S4.E18)\); the backbone, the adaptive\-computation halting, the optimizer \(AdamATan2\), the data pipeline, and the 100K\-epoch schedule are held fixed\. Baseline and DeepLoop runs train on an identical, hash\-verifiedarc\-aug\-1000build and are scored by the same count\-first voting protocol onN=400N=400evaluation puzzles\. As a harness check, the published HRM ARC\-2 checkpoint scores5\.00%5\.00\\%\(K=2K\{=\}2\) on this pipeline, matching the released value\.

Table[3](https://arxiv.org/html/2607.13491#S5.T3)reports voted accuracy at voting budgetsK∈\{1,2,10,100,1000\}K\\in\\\{1,2,10,100,1000\\\}\. DeepLoop improves the paper\-protocol two\-vote accuracy from36\.50%36\.50\\%to39\.75%39\.75\\%\(\+3\.25\+3\.25pp\) and improves every column of the voting ladder\. A four\-seed control places the per\-seedK=2K\{=\}2standard deviation at≈0\.5\\approx 0\.5pp, so the\+3\.25\+3\.25pp gain is roughly a6​σ6\\sigmaeffect rather than a seed draw\. The prediction that a tied hierarchical reasoner sits at the alignedp=1/2p=1/2threshold, the same exponent that is optimal for the single\-module looped backbone \(Table[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)\), is therefore borne out: a single residual\-scaling rule,p=1/2p=1/2, is the correct setting for both the looped language model and the hierarchical recurrent reasoner\.

Table 3:Voted accuracy \(%\) on ARC\-AGI\-1 for the vanilla Hierarchical Reasoning Model and the same model with the DeepLoop residual scaling \(p=1/2p=1/2\), at voting budgetsKK\. Both runs use 100K epochs, AdamATan2, and an identical hash\-verifiedarc\-aug\-1000build;N=400N=400evaluation puzzles\. Bold marks the better cell per column; theK=2K\{=\}2column is the paper\-protocol headline metric\.## 6Related Work

#### Looped and universal Transformers\.

Universal Transformers\(dehghani2018universal\)and ALBERT\(lan2019albert\)introduced depth\-wise parameter sharing as a way to increase effective computation without increasing parameter count\. Later work studied partial sharing\(reid2021subformer\), looped Transformers for in\-context learning and algorithmic computation\(giannou2023looped;yang2023looped;gatmiry2024can\), and recurrent\-depth language models that trade inference compute for quality\(geiping2025scaling\)\. DeepLoop is orthogonal to these architectures: it only changes the residual and initialization scales used when a physical block is revisited\.

#### Adaptive computation and iterative reasoning\.

Looped depth can also be viewed as a differentiable test\-time\-compute mechanism\. Adaptive Computation Time\(graves2016adaptive\), PonderNet\(banino2021pondernet\), and recurrent networks for algorithmic extrapolation\(schwarzschild2021can;bansal2022end\)all exploit repeated application of shared computation\. Recent language\-modeling work similarly adds latent or recurrent computation\(goyal2023think;hao2024training;saunshi2025reasoning\)\. Our analysis targets the stability of this repeated shared computation\.

#### Residual scaling and parameterization\.

Transformer stability has been improved through normalization placement\(xiong2020layer;nguyen2019transformers\), initialization and residual scaling\(zhang2019fixup;huang2020improving;de2020batch;bachlechner2021rezero;liu2020understanding\), and DeepNorm/DeepNet\(wang2024deepnet\), which derivesα=\(2​N\)1/4\\alpha=\(2N\)^\{1/4\}andβ=\(8​N\)−1/4\\beta=\(8N\)^\{\-1/4\}for untied Post\-LN Transformers\. Width\-and\-depth parameterizations such asμ\\muP and Depth\-μ\\muP prescribe complementary scalings for feature learning and hyperparameter transfer\(yang2021tensor;yang2021tuning;yang2023tensor;bordelon2023depthwise\)\. These analyses do not explicitly account for the extra correlation introduced when the same residual\-branch parameters are reused across loop visits\. DeepLoop is a loop\-specific correction to the DeepNorm exponent\.

## 7Conclusion

Looped Transformers turn depth into a controllable compute resource, but repeated parameter reuse changes the residual\-scaling problem\. When a residual branch is revisited, its shared update is accumulated across visits and then read by those same visits in the next linearized forward pass\. We captured this tied\-depth effect with a visit\-alignment coefficientκR\\kappa\_\{R\}, recovering the DeepNorm exponent for decorrelated visits and identifyingp=1/2p=1/2as the conservative exponent for aligned visits at fixed physical depth\.

DeepLoop implements this correction with a one\-line Post\-LN scaling rule,α=\(2​N\)1/2\\alpha=\(2N\)^\{1/2\}andβ=\(8​N\)−1/2\\beta=\(8N\)^\{\-1/2\}\. Empirically, it is neutral when no physical block is revisited and improves looped GPT\-style language models as recurrent depth increases, with gains in validation loss and downstream accuracy\. App\-sweep atR=3R\{=\}3further places the stability boundary nearp=1/2p=1/2\.

Future work should measureκR\\kappa\_\{R\}or cross\-round gradient alignment directly, test whether the same boundary holds at larger scale or under alternative parameterizations, and explore whether training can encourage decorrelated visits to safely use less conservative exponents\.

## Acknowledgement

We thank Zixuan Wang and Eric Song for their constructful feedback and insightful discussion\.

## References

\\appendixpage

\\startcontents

\[section\]\\printcontents\[section\]l1

## Appendix AProofs

###### Lemma A\.1\(RMSNorm exposes the residual branch through1/α1/\\alpha\)\.

LetRMS​\(x\)2=d−1​‖x‖2\\mathrm\{RMS\}\(x\)^\{2\}=d^\{\-1\}\\\|x\\\|^\{2\}, assumeRMS​\(x\)=1\\mathrm\{RMS\}\(x\)=1, and defineℛ​\(y\)=y/RMS​\(y\)\\mathcal\{R\}\(y\)=y/\\mathrm\{RMS\}\(y\)\. IfRMS​\(z\)/α≤c<1\\mathrm\{RMS\}\(z\)/\\alpha\\leq c<1, then

ℛ​\(α​x\+z\)=x\+z−⟨x,z⟩d​xα\+O​\(RMS​\(z\)2α2\),⟨x,z⟩d:=d−1​x⊤​z\.\\mathcal\{R\}\(\\alpha x\+z\)=x\+\\frac\{z\-\\langle x,z\\rangle\_\{d\}x\}\{\\alpha\}\+O\\left\(\\frac\{\\mathrm\{RMS\}\(z\)^\{2\}\}\{\\alpha^\{2\}\}\\right\),\\qquad\\langle x,z\\rangle\_\{d\}:=d^\{\-1\}x^\{\\top\}z\.\(24\)

###### Proof A\.2\(Proof of Lemma[A\.1](https://arxiv.org/html/2607.13491#A1.Thmtheorem1)\)\.

Writem=⟨x,z⟩dm=\\langle x,z\\rangle\_\{d\}andδ2=RMS​\(z\)2\\delta^\{2\}=\\mathrm\{RMS\}\(z\)^\{2\}\. Since\|m\|≤δ\|m\|\\leq\\deltaandδ/α≤c<1\\delta/\\alpha\\leq c<1, the quantity below is bounded away from zero:

RMS​\(α​x\+z\)=α​\(1\+2​m/α\+δ2/α2\)1/2\.\\mathrm\{RMS\}\(\\alpha x\+z\)=\\alpha\\left\(1\+2m/\\alpha\+\\delta^\{2\}/\\alpha^\{2\}\\right\)^\{1/2\}\.A first\-order Taylor expansion of the inverse square root around one gives

\(1\+2​m/α\+δ2/α2\)−1/2=1−m/α\+O​\(δ2/α2\),\\left\(1\+2m/\\alpha\+\\delta^\{2\}/\\alpha^\{2\}\\right\)^\{\-1/2\}=1\-m/\\alpha\+O\(\\delta^\{2\}/\\alpha^\{2\}\),where the remainder is uniform forδ/α≤c\\delta/\\alpha\\leq c\. Multiplying by\(α​x\+z\)/α\(\\alpha x\+z\)/\\alphayields

α​x\+zRMS​\(α​x\+z\)=x\+z−m​xα\+O​\(δ2/α2\),\\frac\{\\alpha x\+z\}\{\\mathrm\{RMS\}\(\\alpha x\+z\)\}=x\+\\frac\{z\-mx\}\{\\alpha\}\+O\(\\delta^\{2\}/\\alpha^\{2\}\),which is Eq\. \([24](https://arxiv.org/html/2607.13491#A1.E24)\)\.

###### Proposition A\.3\(Depth\-untied first\-order stability\)\.

For a depth\-NNpost\-normalized Transformer whose residual\-sublayer parameters are not shared across unrolled depth, withM=2​NM=2Nresidual\-sublayer visits, Assumption[3\.1](https://arxiv.org/html/2607.13491#S3.SS1)implies the first\-order bound Eq\. \([9](https://arxiv.org/html/2607.13491#S3.E9)\), and consequentlyM​\(β/α\)2=O​\(1\)M\(\\beta/\\alpha\)^\{2\}=O\(1\)\(Eq\. \([10](https://arxiv.org/html/2607.13491#S3.E10)\)\) is a sufficient first\-order stability condition\. This statement does not depend on whether the input and output token embeddings are tied\.

###### Proof A\.4\(Proof of Proposition[A\.3](https://arxiv.org/html/2607.13491#A1.Thmtheorem3)\)\.

In the depth\-untied residual stack, every unrolled residual\-sublayer visit has its own residual\-branch parameter tensor\. The first\-order change is therefore a sum ofMMindependent visit\-wise perturbation terms\. By Assumption[3\.1](https://arxiv.org/html/2607.13491#S3.SS1), each term is bounded by an output sensitivityO​\(β/α\)O\(\\beta/\\alpha\)times an effective updateO​\(β/α\)O\(\\beta/\\alpha\)\. Summing overMMvisits gives

‖Δ​F‖≤C′​M​\(β/α\)2,\\\|\\Delta F\\\|\\leq C^\{\\prime\}M\(\\beta/\\alpha\)^\{2\},for a depth\-independent constantC′C^\{\\prime\}, which implies the stated sufficient condition\.

###### Proposition A\.5\(Tied loop perturbation\)\.

Under Assumption[3\.1](https://arxiv.org/html/2607.13491#S3.SS1), a looped Transformer withJ=2​KJ=2Kphysical residual sublayers andRRvisits satisfies the first\-order bound Eq\. \([13](https://arxiv.org/html/2607.13491#S3.E13)\), soM​κR​\(β/α\)2=O​\(1\)M\\kappa\_\{R\}\(\\beta/\\alpha\)^\{2\}=O\(1\)\(Eq\. \([14](https://arxiv.org/html/2607.13491#S3.E14)\)\) is a sufficient tied\-depth stability condition\.

###### Corollary A\.6\(Worst\-case aligned loop bound\)\.

IfκR=Θ​\(R\)\\kappa\_\{R\}=\\Theta\(R\), then Eq\. \([14](https://arxiv.org/html/2607.13491#S3.E14)\) reduces toM​R​\(β/α\)2=O​\(1\)MR\(\\beta/\\alpha\)^\{2\}=O\(1\)\.

###### Proof A\.7\(Proof of Proposition[A\.5](https://arxiv.org/html/2607.13491#A1.Thmtheorem5)\)\.

Using Eq\. \([11](https://arxiv.org/html/2607.13491#S3.E11)\) and submultiplicativity,

‖Δ​Ftied‖≤η​∑j=1J‖∑r=1RUr,j‖​‖∑t=1RGt,j‖\+O​\(η2\)\.\\\|\\Delta F\_\{\\mathrm\{tied\}\}\\\|\\leq\\eta\\sum\_\{j=1\}^\{J\}\\left\\\|\\sum\_\{r=1\}^\{R\}U\_\{r,j\}\\right\\\|\\left\\\|\\sum\_\{t=1\}^\{R\}G\_\{t,j\}\\right\\\|\+O\(\\eta^\{2\}\)\.By the definition ofκR\\kappa\_\{R\}, each physical sublayer contributes at mostR​κR​CU​CG​\(β/α\)2R\\kappa\_\{R\}C\_\{U\}C\_\{G\}\(\\beta/\\alpha\)^\{2\}\. Summing overJJphysical sublayers gives

J​R​κR​CU​CG​\(β/α\)2=M​κR​CU​CG​\(β/α\)2\.JR\\kappa\_\{R\}C\_\{U\}C\_\{G\}\(\\beta/\\alpha\)^\{2\}=M\\kappa\_\{R\}C\_\{U\}C\_\{G\}\(\\beta/\\alpha\)^\{2\}\.Absorbingη\\eta,CUC\_\{U\}, andCGC\_\{G\}into the constant proves Eq\. \([13](https://arxiv.org/html/2607.13491#S3.E13)\)\.

###### Proof A\.8\(Proof of Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)\)\.

Substituting Eq\. \([15](https://arxiv.org/html/2607.13491#S3.E15)\) into the tied\-depth condition gives

M​κR​\(βα\)2=2​N​Θ​\(Rγ\)​\(c​d\)−2​p​N−4​p\.M\\kappa\_\{R\}\\left\(\\frac\{\\beta\}\{\\alpha\}\\right\)^\{2\}=2N\\,\\Theta\(R^\{\\gamma\}\)\\,\(cd\)^\{\-2p\}N^\{\-4p\}\.At fixed physical depthKK,R=N/KR=N/K, so theNN\-dependence isΘ​\(N1\+γ−4​p\)\\Theta\(N^\{1\+\\gamma\-4p\}\)\. This quantity remains uniformly bounded asR→∞R\\to\\inftyif and only if1\+γ−4​p≤01\+\\gamma\-4p\\leq 0, equivalentlyp≥\(1\+γ\)/4p\\geq\(1\+\\gamma\)/4\.

## Appendix BSmall\-scale downstream evaluation

We report the GPT\-2 small version of the downstream evaluation from §[5\.2](https://arxiv.org/html/2607.13491#S5.SS2)\. The architecture is identical to the medium backbone except for hidden size768768\(vs\.10241024\) and1212layers \(vs\.2424\); training uses the same FineWeb\-Edu 50BT data and schedule but on4×4\\timesH200 141 GB GPUs\. The same eight\-tasklm\-evaluation\-harnessprotocol applies\. Table[B](https://arxiv.org/html/2607.13491#A2)reports per\-task accuracy and Avg; Figures[4](https://arxiv.org/html/2607.13491#A2.F4)and[5](https://arxiv.org/html/2607.13491#A2.F5)visualize the 1\-shot Avg and the per\-task 1\-shot accuracy, respectively\.

Table 4:Downstream accuracy \(%\) for the GPT\-2 small backbone\. Metric isacc; Avg is the unweighted arithmetic mean across the eight tasks\. Bold marks the best cell per column per shot setting\.![Refer to caption](https://arxiv.org/html/2607.13491v1/x3.png)Figure 4:Eight\-task 1\-shot Avg accuracy against loop countRRfor the GPT\-2 small backbone, base versus DeepLoop\. Numbers match the 1\-shot Avg column of Table[B](https://arxiv.org/html/2607.13491#A2)\.
![Refer to caption](https://arxiv.org/html/2607.13491v1/x4.png)Figure 5:Per\-task 1\-shot accuracy against loop countRRfor base versus DeepLoop on all eight tasks at the GPT\-2 small scale\. Numbers match Table[B](https://arxiv.org/html/2607.13491#A2)\.
The two methods are within±0\.25\\pm 0\.25points of each other atR=1R\{=\}1on both shot settings, and DeepLoop has higher Avg atR=3R\{=\}3\(by\+1\.31\+1\.31and\+0\.79\+0\.79points on 0\-shot and 1\-shot respectively\) and atR=5R\{=\}5\(by\+0\.15\+0\.15and\+0\.50\+0\.50\)\. The headline small\-scale cell is DeepLoopR=5R\{=\}51\-shot at49\.66%49\.66\\,\\%\. Against the same\-RRbaseline \(baseR=5R\{=\}51\-shot,49\.16%49\.16\\,\\%\), DeepLoop wins on four of the eight tasks \(ARC\-C, ARC\-E, HellaSwag, WinoGrande\), is within0\.110\.11points on three tasks \(PIQA, SciQ, SIQA\), and loses clearly only on OpenBookQA\.

## Appendix CEmpirical\\texorpdfstringppp\-sweep at fixed loop count

Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)predicts a sharpp=1/2p\{=\}1/2training\-stability threshold under the worst\-case aligned regime\. We test this prediction directly with a single\-axis sweep over the exponentppat fixedR=3R\{=\}3on the GPT\-2 small backbone\. All other ingredients, tied embeddings, input\-embedding RMSNorm, FineWeb\-Edu 50BT data pipeline, batch size, learning rate, and seed protocol, are identical to the runs in Table[5\.1](https://arxiv.org/html/2607.13491#S5.SS1); only the residual scalingα=\(2​N\)p\\alpha=\(2N\)^\{p\}and initialization gainβ=\(8​N\)−p\\beta=\(8N\)^\{\-p\}change\. We run a grid of seven values,p∈\{0\.30,0\.35,0\.40,0\.45,0\.50,0\.55,0\.60\}p\\in\\\{0\.30,0\.35,0\.40,0\.45,0\.50,0\.55,0\.60\\\}, with up to five seeds per value\. Each cell is a 90\-minute run on4×4\\timesH200 141 GB GPUs, which under our throughput reaches roughly2,7002\{,\}700optimizer steps; we report validation loss at the common comparison checkpoint of step20002000where available, and use the trailing\-50\-iter mean train loss as a surrogate for runs that do not reach a post\-warmup validation step\.

![Refer to caption](https://arxiv.org/html/2607.13491v1/x5.png)Figure 6:Validation\-loss snapshot at step20002000on FineWeb\-Edu 50BT for the GPT\-2 small backbone,R=3R\{=\}3, sweeping the residual\-scaling exponentp∈\{0\.30,…,0\.60\}p\\in\\\{0\.30,\\ldots,0\.60\\\}\. Each marker is one seed\. Red crosses are seeds that never escape the unigram\-frequency floor \(≈7\.67\\approx 7\.67nats\); blue circles are seeds that train; the blue line connects the per\-pptrained\-seed mean\. Belowp=0\.45p\{=\}0\.45all seeds diverge; atp=0\.45p\{=\}0\.45the boundary is crossed \(some seeds train, some do not\); atp≥0\.50p\\geq 0\.50training is reliable\. The empirical phase boundary lines up with the Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)prediction ofp=1/2p\{=\}1/2\. Conditional on convergence, largerppgives a slightly higher loss \(less aggressive learning\), sop=1/2p\{=\}1/2is also the smallestppthat is reliably stable, supporting it as the default DeepLoop choice\.The picture in Figure[6](https://arxiv.org/html/2607.13491#A3.F6)is consistent with the analysis\. First, no seed atp∈\{0\.30,0\.35,0\.40\}p\\in\\\{0\.30,0\.35,0\.40\\\}ever escapes the unigram floor: these clearly sub\-threshold runs reproduce the divergence that Proposition[3\.1](https://arxiv.org/html/2607.13491#S3.Thmtheorem1)predicts under the aligned worst case\. Second, among seeds that do train, the mean validation loss at the same step rises monotonically asppincreases across the boundary \(3\.703\.70atp=0\.45p\{=\}0\.45,3\.733\.73atp=0\.50p\{=\}0\.50,3\.763\.76atp=0\.55p\{=\}0\.55,3\.803\.80atp=0\.60p\{=\}0\.60\), confirming that more aggressive scaling \(smallerpp\) delivers a stronger learning signal whenever it does not destabilize training\. Putting these two observations together:p=1/2p\{=\}1/2is the smallest exponent that is reliably stable across seeds in this regime\. We adopt it as the DeepLoop default for that reason\. Sub\-threshold choicesp∈\[0\.45,0\.50\)p\\in\[0\.45,0\.50\)produce lower trained\-seed mean loss when they do train, but with≥1/3\\geq 1/3of seeds failing to leave the unigram floor before step20002000, they are not safe defaults\.

Thepp\-sweep is small in scope: it is at one scale \(GPT\-2 small\), one loop depth \(R=3R\{=\}3\), and one optimizer\-step budget\. Whether the same boundary atp=1/2p\{=\}1/2applies at largerKK, at different normalization placements, or at substantially longer training, is left to future work and discussed in the Limitations paragraph \(§[6](https://arxiv.org/html/2607.13491#S6)\)\.

## Appendix DCompute resources

All experiments were run on NVIDIA H200 141 GB GPUs in a SLURM cluster\. GPT\-2 small cells use4×4\\timesH200, and GPT\-2 medium cells use8×8\\timesH200\. Per\-cell wall\-clock time is read from the corresponding WandB run records; aggregate GPU\-hours below are rounded to the nearest hundred\.

#### Per\-cell cost\.

Wall\-clock time scales roughly linearly with the loop countRR:

\\topruleR=1R\{=\}1R=3R\{=\}3R=5R\{=\}5R=7R\{=\}7\\midruleSmall wall\-clock \(h\)99222235355050Small GPU\-hours / cell37379090145145200200\\midruleMedium wall\-clock \(h\)1212323252527777Medium GPU\-hours / cell9595260260420420620620\\bottomrule

#### Aggregate cost\.

The reported FineWeb\-Edu language\-modeling matrix \(Tables[5\.1](https://arxiv.org/html/2607.13491#S5.SS1)and[5\.2](https://arxiv.org/html/2607.13491#S5.SS2): 16 cells\{\\\{base, DeepLoop\}×\{\\\}\\times\\\{small, medium\}×R∈\{1,3,5,7\}\\\}\\times R\\in\\\{1,3,5,7\\\}\) totals approximately𝟑,𝟕𝟎𝟎\\mathbf\{3\{,\}700\}H200 GPU\-hours\. Exploratory experiments not included in the main paper, a range of optimization\-related variants and additional benchmark sweeps we evaluated during the development of the method and ultimately did not report, add approximately𝟕,𝟎𝟎𝟎\\mathbf\{7\{,\}000\}GPU\-hours\.

The total project compute is therefore approximately𝟏𝟎,𝟕𝟎𝟎\\mathbf\{10\{,\}700\}H200 GPU\-hours\.