Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

# Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion
Source: [https://arxiv.org/html/2608.14664](https://arxiv.org/html/2608.14664)
Zeyu Liu1, Jinhao Zhang2, Yunquan Zhang1, Guangming Tan1, Xiang Gao3, Fangming Liu4, Daning Cheng1\\corresponding

###### Abstract

How can we determine whether a trained neural network is already deep enough? We study this under a fixed function\-preserving residual\-growth protocol specifying insertion locations, residual families, zero\-output initializations, and zero\-state first\-order updates\. We define first\-order residual depth saturation as the absence of a strict local decrease from every admissible insertion\. We prove residual non\-degeneracy is necessary and sufficient: additional depth has first\-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space\. This boundary is shared by descent\-compatible zero\-state updates and invariant under regular local reparameterizations preserving that tangent space\. Under residual\-signal realizability, raw activation\-gradient vanishing exactly certifies saturation\. Across ResNets, GPT\-2\-style models, and continued\-pretrained Pythia checkpoints, the maximum activation\-gradient norm decreases toward a low\-signal regime with depth\. Function\-preserving growth also achieves converged performance competitive with training from scratch\. These results support activation\-gradient magnitude as a conservative diagnostic of the remaining empirical first\-order value of residual depth\.

## Introduction

Increasing depth is a standard way to expand neural\-network capacity, yet its marginal benefit does not persist indefinitely: models often improve as layers are added and then enter a regime in which further depth yields little gain in test dataset\. This raises a basic question for model scaling and adaptive growth: given a trained network, how can we determine whether additional residual depth still has useful optimization value? Comparing independently trained models of different depths cannot isolate this question, since their performance differences may reflect initialization, optimization difficulty, training budget, regularization, or finite\-sample variation\. Function\-preserving residual insertion\(Chenet al\.[2015](https://arxiv.org/html/2608.14664#bib.bib3); Weiet al\.[2016](https://arxiv.org/html/2608.14664#bib.bib4)\)provides a cleaner setting: a residual block initialized to produce globally zero output leaves the represented function unchanged, so the marginal value of new depth can be studied at a common initialization\. Existing growth methods, however, provide no exact criterion for when residual growth should terminate\.

Some works on normalized residual networks introduced a residual non\-degeneracy condition and showed that it is sufficient for constructing a locally improving expanded model\(Chenget al\.[2026](https://arxiv.org/html/2608.14664#bib.bib17)\)\. This leaves open a logically prior question: does the condition exactly characterize whether any admissible first\-order residual direction remains? We study this question relative to a fixed*residual\-growth protocol*that specifies, before observing the gradient signal, the admissible insertion candidates, the parameterized residual family with its function\-preserving initialization, and a zero\-state, descent\-compatible first\-order update for the inserted parameters\. A candidate is operationally saturated when this update cannot produce a strict decrease from the function\-preserving initialization for any sufficiently small positive step\. Our main theorem shows that this boundary does not depend on the optimizer chosen within the stated first\-order class\.

The tangent\-space projection is the appropriate parameterization\-independent quantity, because a raw activation gradient may contain components that the selected residual family cannot express\. For standard zero\-output blocks—a feature\-producing subnetwork followed by a trainable output projection—the parameter criterion reduces to the cross\-gradient between the activation gradient and the residual features; holding the feature parameters fixed loses no first\-order directions, since their derivative vanishes at a zero output projection\. We further introduce a checkpoint\-specific*residual\-signal realizability*condition, requiring only that the current conditional activation\-gradient signal lie in this tangent space\. Under realizability the projection preserves the full signal, and raw activation\-gradient vanishing becomes a necessary and sufficient saturation certificate\.

Our experiments combine two complementary forms of evidence\. First, we measure the sample\-wise activation\-gradient norm across depth in ResNets trained on CIFAR\-10, CIFAR\-100, and ImageNet\-100, in Pythia checkpoints continued\-pretrained on FineWeb\-Edu, and in GPT\-2\-style models trained from scratch on the same corpus\. Because orthogonal projection cannot increase norm, this score upper\-bounds the empirical projected residual\-growth value on every fixed sample\. Second, on four controlled CIFAR\-10 ResNet configurations we insert one globally zero\-output residual block at a time, comparing function\-preserving growth with training the same final\-depth architectures from random initialization to control for optimization disadvantages of the growth procedure\. Additional blocks produce realized gains at shallower depths, and these gains disappear once the sample\-wise score enters a stable low\-signal regime\.

Our contributions are fourfold\. First, we introduce an operational notion of first\-order residual depth saturation and prove that its boundary is shared by all zero\-state, descent\-compatible first\-order updates\. Second, we characterize this boundary through the parameterization\-invariant projection of the conditional activation gradient onto the residual tangent space, and derive the cross\-gradient criterion for standard zero\-output blocks\. Third, we establish exact and approximate residual\-signal realizability conditions under which raw activation gradients can replace the projected criterion\. Finally, we combine broad fixed\-sample gradient measurements with controlled one\-block growth and from\-scratch comparisons, supporting low sample\-wise activation\-gradient energy as a conservative practical indicator that little empirical first\-order value of residual depth remains\.

## Related Work

Net2Net and Network Morphism introduced function\-preserving operators that widen or deepen a trained network without changing the represented function\(Chenet al\.[2015](https://arxiv.org/html/2608.14664#bib.bib3); Weiet al\.[2016](https://arxiv.org/html/2608.14664#bib.bib4)\)\. This idea underlies efficient Transformer pre\-training through progressive stacking, parameter reuse and knowledge inheritance, learned or lossless expansion operators, and variance\-transfer initialization\(Gonget al\.[2019](https://arxiv.org/html/2608.14664#bib.bib5); Guet al\.[2020](https://arxiv.org/html/2608.14664#bib.bib19); Duet al\.[2024](https://arxiv.org/html/2608.14664#bib.bib9); Chenet al\.[2022](https://arxiv.org/html/2608.14664#bib.bib6); Qinet al\.[2021](https://arxiv.org/html/2608.14664#bib.bib20); Wanget al\.[2023](https://arxiv.org/html/2608.14664#bib.bib7),[2024](https://arxiv.org/html/2608.14664#bib.bib23); Yuanet al\.[2023](https://arxiv.org/html/2608.14664#bib.bib8)\)\. These works address how to grow efficiently and presuppose that added capacity is useful; none gives an exact criterion for when residual growth stops having local optimization value\.

Deciding when and where to grow is a classical theme of constructive learning\(Fahlman and Lebiere[1990](https://arxiv.org/html/2608.14664#bib.bib24); Ash[1989](https://arxiv.org/html/2608.14664#bib.bib25)\)\. Recent policies schedule depth growth via validation heuristics or fitting risk\(Wenet al\.[2019](https://arxiv.org/html/2608.14664#bib.bib10); Wuet al\.[2024](https://arxiv.org/html/2608.14664#bib.bib11)\), and local scores select beneficial insertions through splitting directions, gradient norms, expressivity bottlenecks, topological derivatives, and natural\-expansion scores\(Wuet al\.[2019](https://arxiv.org/html/2608.14664#bib.bib12),[2021](https://arxiv.org/html/2608.14664#bib.bib13); Evciet al\.[2022](https://arxiv.org/html/2608.14664#bib.bib14); Verbockhavenet al\.[2024](https://arxiv.org/html/2608.14664#bib.bib15); Krishnanunniet al\.[2025](https://arxiv.org/html/2608.14664#bib.bib16); Mitchellet al\.[2023](https://arxiv.org/html/2608.14664#bib.bib27)\)\. These criteria are sufficient conditions or heuristics for beneficial growth; we instead characterize exactly when no admissible function\-preserving residual insertion admits a first\-order improvement\.

Residual networks behave like ensembles of shallow paths, and stochastic depth and layer pruning of large language models reveal substantial redundancy in trained depth\(Veitet al\.[2016](https://arxiv.org/html/2608.14664#bib.bib30); Huanget al\.[2016](https://arxiv.org/html/2608.14664#bib.bib33); Gromovet al\.[2024](https://arxiv.org/html/2608.14664#bib.bib34); Menet al\.[2024](https://arxiv.org/html/2608.14664#bib.bib35)\); zero\-initialized residual branches are benign and trainable\(Zhanget al\.[2019](https://arxiv.org/html/2608.14664#bib.bib36); Bachlechneret al\.[2021](https://arxiv.org/html/2608.14664#bib.bib37)\), supporting our zero\-output initialization\. Unlike scaling\-law comparisons of independently trained models\(Kaplanet al\.[2020](https://arxiv.org/html/2608.14664#bib.bib39); Levineet al\.[2020](https://arxiv.org/html/2608.14664#bib.bib41)\), our function\-preserving setting isolates the marginal first\-order value of additional depth at a common checkpoint\.

Closest to our work,\([2026](https://arxiv.org/html/2608.14664#bib.bib17)\)show that residual non\-degeneracy \(Condition 1\) is sufficient for constructing a locally improving expanded model; we prove that it is also necessary under a fixed zero\-state protocol, yielding a saturation boundary invariant to the first\-order optimizer and to reparameterizations preserving the residual tangent space\.

## Problem Setup, Assumptions, and Notation

### Problem Setup

Let𝒳\\mathcal\{X\}be the input space, let𝒴\\mathcal\{Y\}be the label space, and let𝒟\\mathcal\{D\}be a probability distribution on𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}\. We write\(x,y\)∼𝒟\(x,y\)\\sim\\mathcal\{D\}, wherexxis an input andyyis its label\. Every predictor considered below maps𝒳\\mathcal\{X\}toℝdout\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\}, and the loss is a measurable functionℓ:ℝdout×𝒴→\[0,∞\)\\ell:\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\}\\times\\mathcal\{Y\}\\to\[0,\\infty\)\. For any measurable predictorff, defineR​\(f\):=𝔼\(x,y\)∼𝒟​\[ℓ​\(f​\(x\),y\)\]R\(f\):=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\}\[\\ell\(f\(x\),y\)\]whenever the expectation is finite\. For a fixed sampleS=\{\(xi,yi\)\}i=1MS=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{M\}, defineℒS​\(f\):=M−1​∑i=1Mℓ​\(f​\(xi\),yi\)\\mathcal\{L\}\_\{S\}\(f\):=M^\{\-1\}\\sum\_\{i=1\}^\{M\}\\ell\(f\(x\_\{i\}\),y\_\{i\}\)\.

We follow the residual\-insertion notation of\(Chenget al\.[2026](https://arxiv.org/html/2608.14664#bib.bib17)\)\. Letfold∗f\_\{\\mathrm\{old\}\}^\{\*\}be a trained reference model withR​\(fold∗\)<∞R\(f\_\{\\mathrm\{old\}\}^\{\*\}\)<\\infty\. Before observing any activation\-gradient signal or growth outcome, we fix a*residual\-growth protocol*\. The protocol specifies a finite setℐ\\mathcal\{I\}of admissible insertion candidates\. Each candidatel∈ℐl\\in\\mathcal\{I\}consists of an insertion location, a parameterized residual family, a designated function\-preserving initialization, and a first\-order optimizer direction rule for the newly inserted parameters\.

At candidatell, decompose the reference model asfold∗=ftop\(l\)∘fbot\(l\)f\_\{\\mathrm\{old\}\}^\{\*\}=f\_\{\\mathrm\{top\}\}^\{\(l\)\}\\circ f\_\{\\mathrm\{bot\}\}^\{\(l\)\}and definezl:=fbot\(l\)​\(x\)∈ℝNlz\_\{l\}:=f\_\{\\mathrm\{bot\}\}^\{\(l\)\}\(x\)\\in\\mathbb\{R\}^\{N\_\{l\}\}, whereNlN\_\{l\}is the hidden\-state dimension at candidatell\. Letμl\\mu\_\{l\}denote the distribution ofzlz\_\{l\}induced by\(x,y\)∼𝒟\(x,y\)\\sim\\mathcal\{D\}\. When a single candidate is fixed, we suppresslland writeftopf\_\{\\mathrm\{top\}\},fbotf\_\{\\mathrm\{bot\}\},zz, andμ\\mu\.

The parameterized residual family at candidatellisℱres\(l\):=\{hl,θl:θl∈Θl\}\\mathcal\{F\}\_\{\\mathrm\{res\}\}^\{\(l\)\}:=\\\{h\_\{l,\\theta\_\{l\}\}:\\theta\_\{l\}\\in\\Theta\_\{l\}\\\}, whereΘl⊆ℝpl\\Theta\_\{l\}\\subseteq\\mathbb\{R\}^\{p\_\{l\}\}and0∈int⁡\(Θl\)0\\in\\operatorname\{int\}\(\\Theta\_\{l\}\)\. The designated local origin satisfieshl,0​\(z\)=0h\_\{l,0\}\(z\)=0for everyz∈ℝNlz\\in\\mathbb\{R\}^\{N\_\{l\}\}; equivalently,hl,0≡0h\_\{l,0\}\\equiv 0on the full ambient hidden\-state space\. The notationθl=0\\theta\_\{l\}=0denotes a local coordinate centered at this initialization and does not require every raw parameter in the residual branch to be numerically zero\.

The expanded model isfl,θl​\(x\):=ftop\(l\)​\(zl\+hl,θl​\(zl\)\)f\_\{l,\\theta\_\{l\}\}\(x\):=f\_\{\\mathrm\{top\}\}^\{\(l\)\}\\bigl\(z\_\{l\}\+h\_\{l,\\theta\_\{l\}\}\(z\_\{l\}\)\\bigr\)\. Sincehl,0≡0h\_\{l,0\}\\equiv 0, the insertion preserves the represented function pointwise andfl,0=fold∗f\_\{l,0\}=f\_\{\\mathrm\{old\}\}^\{\*\}\. Define the population and empirical objectives associated with candidatellbyΦl​\(θl\):=R​\(fl,θl\)\\Phi\_\{l\}\(\\theta\_\{l\}\):=R\(f\_\{l,\\theta\_\{l\}\}\)andΦS,l​\(θl\):=ℒS​\(fl,θl\)\\Phi\_\{S,l\}\(\\theta\_\{l\}\):=\\mathcal\{L\}\_\{S\}\(f\_\{l,\\theta\_\{l\}\}\)\.

All components of the residual\-growth protocol are fixed in advance\. The candidates, residual families, parameterizations, designated origins, and optimizer direction rules may not be changed after the activation\-gradient signal has been observed\. All internal hyperparameters that determine each direction rule are fixed; the positive scalar step size used in the local analysis remains free\.

### Assumptions

The assumptions below have distinct roles\. Assumptions[1](https://arxiv.org/html/2608.14664#Thmassumption1)and[2](https://arxiv.org/html/2608.14664#Thmassumption2)support the main saturation theorem\. Assumption[3](https://arxiv.org/html/2608.14664#Thmassumption3)is invoked only when the exact projected criterion is replaced by the raw activation\-gradient criterion\. Assumption[4](https://arxiv.org/html/2608.14664#Thmassumption4)is used only for the supplementary finite\-sample consistency result\.

###### Assumption 1\(First\-order regularity\)\.

For every candidatel∈ℐl\\in\\mathcal\{I\}and for𝒟\\mathcal\{D\}\-almost every\(x,y\)\(x,y\), the mapz↦ℓ​\(ftop\(l\)​\(z\),y\)z\\mapsto\\ell\(f\_\{\\mathrm\{top\}\}^\{\(l\)\}\(z\),y\)is Fréchet differentiable atz=zlz=z\_\{l\}\. Its measurable gradient is denoted byql​\(zl,y\):=∇zlℓ​\(ftop\(l\)​\(zl\),y\)∈ℝNlq\_\{l\}\(z\_\{l\},y\):=\\nabla\_\{z\_\{l\}\}\\ell\(f\_\{\\mathrm\{top\}\}^\{\(l\)\}\(z\_\{l\}\),y\)\\in\\mathbb\{R\}^\{N\_\{l\}\}\. Define the conditional population signal byml​\(z\):=𝔼​\[ql​\(zl,y\)∣zl=z\]m\_\{l\}\(z\):=\\mathbb\{E\}\[q\_\{l\}\(z\_\{l\},y\)\\mid z\_\{l\}=z\]\.

Forμl\\mu\_\{l\}\-almost everyzz, the mapθl↦hl,θl​\(z\)\\theta\_\{l\}\\mapsto h\_\{l,\\theta\_\{l\}\}\(z\)is Fréchet differentiable atθl=0\\theta\_\{l\}=0\. Its measurable Jacobian with respect toθl\\theta\_\{l\}, evaluated at the origin, is denoted byJl​\(z\):=Jθl​hl,0​\(z\)∈ℝNl×plJ\_\{l\}\(z\):=J\_\{\\theta\_\{l\}\}h\_\{l,0\}\(z\)\\in\\mathbb\{R\}^\{N\_\{l\}\\times p\_\{l\}\}, whereJθlJ\_\{\\theta\_\{l\}\}denotes the Jacobian with respect to the active residual coordinateθl\\theta\_\{l\}\.

There exists a neighborhood of the origin on whichΦl\\Phi\_\{l\}is finite\. The mapΦl\\Phi\_\{l\}is Fréchet differentiable at0, and its derivative is obtained by interchanging sample\-wise differentiation and expectation\. For a fixed sampleSS, empirical statements are understood on samples for which all corresponding sample\-wise derivatives exist; an i\.i\.d\. sample has this property almost surely under the preceding conditions\.

We further assume𝔼​‖ql​\(zl,y\)‖22<∞\\mathbb\{E\}\\\|q\_\{l\}\(z\_\{l\},y\)\\\|\_\{2\}^\{2\}<\\inftyand𝔼​‖Jl​\(zl\)‖σ2<∞\\mathbb\{E\}\\\|J\_\{l\}\(z\_\{l\}\)\\\|\_\{\\sigma\}^\{2\}<\\infty, where∥⋅∥σ\\\|\\cdot\\\|\_\{\\sigma\}is the operator norm\. Conditional Jensen’s inequality then givesml∈L2​\(μl;ℝNl\)m\_\{l\}\\in L\_\{2\}\(\\mu\_\{l\};\\mathbb\{R\}^\{N\_\{l\}\}\), and Cauchy–Schwarz ensures that the population residual\-parameter gradient defined below is finite\.

A standard sufficient condition for the interchange in Assumption[1](https://arxiv.org/html/2608.14664#Thmassumption1)is a local integrable Lipschitz envelope: for each candidatell, there existρl\>0\\rho\_\{l\}\>0and an integrable random variableBl​\(x,y\)B\_\{l\}\(x,y\)such that\|ℓ​\(fl,θ​\(x\),y\)−ℓ​\(fl,0​\(x\),y\)\|≤Bl​\(x,y\)​‖θ‖2\|\\ell\(f\_\{l,\\theta\}\(x\),y\)\-\\ell\(f\_\{l,0\}\(x\),y\)\|\\leq B\_\{l\}\(x,y\)\\\|\\theta\\\|\_\{2\}whenever‖θ‖2≤ρl\\\|\\theta\\\|\_\{2\}\\leq\\rho\_\{l\}\. For any sequenceθk→0\\theta\_\{k\}\\to 0, sample\-wise Fréchet differentiability makes the normalized remainder converge pointwise to zero\. The local Lipschitz bound controls both the difference quotient and the norm of its derivative byBlB\_\{l\}, so the normalized remainder is dominated by2​Bl2B\_\{l\}\. Dominated convergence then yields ano​\(‖θk‖2\)o\(\\\|\\theta\_\{k\}\\\|\_\{2\}\)population remainder, which gives Fréchet differentiability ofΦl\\Phi\_\{l\}and the stated differentiation–expectation interchange\. Uniform integrability of the local difference quotients is an alternative sufficient condition\.

##### First\-order residual gradients and Condition 1\.

For a fixed sampleSS, letzl,i:=fbot\(l\)​\(xi\)z\_\{l,i\}:=f\_\{\\mathrm\{bot\}\}^\{\(l\)\}\(x\_\{i\}\)andql,i:=ql​\(zl,i,yi\)q\_\{l,i\}:=q\_\{l\}\(z\_\{l,i\},y\_\{i\}\)\. Definegpop\(l\):=𝔼​\[Jl​\(zl\)⊤​ql​\(zl,y\)\]g\_\{\\mathrm\{pop\}\}^\{\(l\)\}:=\\mathbb\{E\}\[J\_\{l\}\(z\_\{l\}\)^\{\\top\}q\_\{l\}\(z\_\{l\},y\)\]andgS\(l\):=M−1​∑i=1MJl​\(zl,i\)⊤​ql,ig\_\{S\}^\{\(l\)\}:=M^\{\-1\}\\sum\_\{i=1\}^\{M\}J\_\{l\}\(z\_\{l,i\}\)^\{\\top\}q\_\{l,i\}\. For a directionu∈ℝplu\\in\\mathbb\{R\}^\{p\_\{l\}\}, writeD​Φl​\(0\)​\[u\]D\\Phi\_\{l\}\(0\)\[u\]andD​ΦS,l​\(0\)​\[u\]D\\Phi\_\{S,l\}\(0\)\[u\]for the Fréchet directional derivatives at the origin\.

###### Condition 1\(Residual non\-degeneracy\)\.

At candidatell, population residual non\-degeneracy holds if there existsvpop\(l\)∈ℝplv\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\in\\mathbb\{R\}^\{p\_\{l\}\}such thatD​Φl​\(0\)​\[vpop\(l\)\]<0D\\Phi\_\{l\}\(0\)\[v\_\{\\mathrm\{pop\}\}^\{\(l\)\}\]<0\. For a fixed sampleSS, its empirical counterpart holds if there existsvS\(l\)∈ℝplv\_\{S\}^\{\(l\)\}\\in\\mathbb\{R\}^\{p\_\{l\}\}such thatD​ΦS,l​\(0\)​\[vS\(l\)\]<0D\\Phi\_\{S,l\}\(0\)\[v\_\{S\}^\{\(l\)\}\]<0\.

###### Assumption 2\(Fixed zero\-state descent\-compatible first\-order update\)\.

At every candidatel∈ℐl\\in\\mathcal\{I\}, the optimizer direction rule and all internal hyperparameters that determine it are fixed as part of the residual\-growth protocol\. We call the optimizer*zero\-state*when the active residual coordinate is initialized atθl=0\\theta\_\{l\}=0and every optimizer state variable associated withθl\\theta\_\{l\}is initialized at its neutral zero value\. We assume that the optimizer used by the protocol is zero\-state\.

Given the exact population or full\-sample empirical residual\-parameter gradientg∈ℝplg\\in\\mathbb\{R\}^\{p\_\{l\}\}, the local update has the formθl\+=η​dl​\(g\)\\theta\_\{l\}^\{\+\}=\\eta d\_\{l\}\(g\), whereη\>0\\eta\>0is a free scalar step size anddl:ℝpl→ℝpld\_\{l\}:\\mathbb\{R\}^\{p\_\{l\}\}\\to\\mathbb\{R\}^\{p\_\{l\}\}is the fixed direction map\. We assumedl​\(0\)=0d\_\{l\}\(0\)=0andg⊤​dl​\(g\)<0g^\{\\top\}d\_\{l\}\(g\)<0for everyg≠0g\\neq 0\. Since0∈int⁡\(Θl\)0\\in\\operatorname\{int\}\(\\Theta\_\{l\}\), every such direction is feasible for all sufficiently small positiveη\\eta\.

This class includes gradient descent and positive\-definite preconditioned gradient descent\. It also includes the first bias\-corrected Adam or AdamW update at zero state: coordinatewise,dl​\(g\)j=−gj/\(\|gj\|\+ϵAdam\)d\_\{l\}\(g\)\_\{j\}=\-g\_\{j\}/\(\|g\_\{j\}\|\+\\epsilon\_\{\\mathrm\{Adam\}\}\), whereϵAdam\>0\\epsilon\_\{\\mathrm\{Adam\}\}\>0is the numerical\-stability constant\. Hencedl​\(0\)=0d\_\{l\}\(0\)=0andg⊤​dl​\(g\)<0g^\{\\top\}d\_\{l\}\(g\)<0forg≠0g\\neq 0; decoupled weight decay contributes no first\-step drift because the active residual coordinate is zero\. The analysis does not cover Hessian\-based updates, externally injected perturbations, nonzero initial optimizer states, or stochastic escape from a zero full gradient\.

##### Population and empirical first\-order saturation\.

A candidatellis*population first\-order improvable*if there existsη¯l\>0\\bar\{\\eta\}\_\{l\}\>0such that, for everyη∈\(0,η¯l\)\\eta\\in\(0,\\bar\{\\eta\}\_\{l\}\),Φl​\(η​dl​\(gpop\(l\)\)\)<Φl​\(0\)\\Phi\_\{l\}\(\\eta d\_\{l\}\(g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)\)<\\Phi\_\{l\}\(0\)\. It is*population first\-order saturated*otherwise\. The empirical notions are defined by replacingΦl\\Phi\_\{l\}andgpop\(l\)g\_\{\\mathrm\{pop\}\}^\{\(l\)\}withΦS,l\\Phi\_\{S,l\}andgS\(l\)g\_\{S\}^\{\(l\)\}; the corresponding step\-size threshold is denoted byη¯S,l\\bar\{\\eta\}\_\{S,l\}\. The reference model is*first\-order depth\-saturated*relative to the fixed protocol if every candidate is population first\-order saturated\. Unless explicitly qualified, “first\-order saturated” refers to the population notion\.

##### Residual tangent\-space objects\.

For each candidatell, letℋl:=L2​\(μl;ℝNl\)\\mathcal\{H\}\_\{l\}:=L\_\{2\}\(\\mu\_\{l\};\\mathbb\{R\}^\{N\_\{l\}\}\)with inner product⟨a,b⟩ℋl:=𝔼zl∼μl​\[a​\(zl\)⊤​b​\(zl\)\]\\langle a,b\\rangle\_\{\\mathcal\{H\}\_\{l\}\}:=\\mathbb\{E\}\_\{z\_\{l\}\\sim\\mu\_\{l\}\}\[a\(z\_\{l\}\)^\{\\top\}b\(z\_\{l\}\)\]and norm‖a‖ℋl:=⟨a,a⟩ℋl1/2\\\|a\\\|\_\{\\mathcal\{H\}\_\{l\}\}:=\\langle a,a\\rangle\_\{\\mathcal\{H\}\_\{l\}\}^\{1/2\}\. Define the residual tangent operatorAl:ℝpl→ℋlA\_\{l\}:\\mathbb\{R\}^\{p\_\{l\}\}\\to\\mathcal\{H\}\_\{l\}by\(Al​u\)​\(z\):=Jl​\(z\)​u\(A\_\{l\}u\)\(z\):=J\_\{l\}\(z\)u\. SinceRange⁡\(Al\)\\operatorname\{Range\}\(A\_\{l\}\)is finite\-dimensional, it is closed; define𝒯h\(l\):=Range⁡\(Al\)\\mathcal\{T\}\_\{h\}^\{\(l\)\}:=\\operatorname\{Range\}\(A\_\{l\}\)and letΠl\\Pi\_\{l\}be the orthogonal projector onto𝒯h\(l\)\\mathcal\{T\}\_\{h\}^\{\(l\)\}\.

For the fixed sampleSS, define the empirical Hilbert spaceℋS,l:=\(ℝNl\)M\\mathcal\{H\}\_\{S,l\}:=\(\\mathbb\{R\}^\{N\_\{l\}\}\)^\{M\}with⟨a,b⟩S,l:=M−1​∑i=1Mai⊤​bi\\langle a,b\\rangle\_\{S,l\}:=M^\{\-1\}\\sum\_\{i=1\}^\{M\}a\_\{i\}^\{\\top\}b\_\{i\}and‖a‖S,l:=⟨a,a⟩S,l1/2\\\|a\\\|\_\{S,l\}:=\\langle a,a\\rangle\_\{S,l\}^\{1/2\}\. DefineqS,l:=\(ql,1,…,ql,M\)∈ℋS,lq\_\{S,l\}:=\(q\_\{l,1\},\\ldots,q\_\{l,M\}\)\\in\\mathcal\{H\}\_\{S,l\}and its matrix representationQS\(l\):=\[ql,1,…,ql,M\]∈ℝNl×MQ\_\{S\}^\{\(l\)\}:=\[q\_\{l,1\},\\ldots,q\_\{l,M\}\]\\in\\mathbb\{R\}^\{N\_\{l\}\\times M\}\. The stacked tangent operatorAS,l:ℝpl→ℋS,lA\_\{S,l\}:\\mathbb\{R\}^\{p\_\{l\}\}\\to\\mathcal\{H\}\_\{S,l\}isAS,l​u:=\(Jl​\(zl,1\)​u,…,Jl​\(zl,M\)​u\)A\_\{S,l\}u:=\(J\_\{l\}\(z\_\{l,1\}\)u,\\ldots,J\_\{l\}\(z\_\{l,M\}\)u\), andΠS,l\\Pi\_\{S,l\}denotes the orthogonal projector ontoRange⁡\(AS,l\)\\operatorname\{Range\}\(A\_\{S,l\}\)\.

###### Assumption 3\(Residual\-signal realizability\)\.

At every candidate to which the raw activation\-gradient criterion is applied, the current conditional activation\-gradient signal is realizable by the fixed residual tangent space:ml∈𝒯h\(l\)m\_\{l\}\\in\\mathcal\{T\}\_\{h\}^\{\(l\)\}\.

The stronger equality𝒯h\(l\)=ℋl\\mathcal\{T\}\_\{h\}^\{\(l\)\}=\\mathcal\{H\}\_\{l\}is called*population tangent completeness*\. Sincedim𝒯h\(l\)≤pl<∞\\dim\\mathcal\{T\}\_\{h\}^\{\(l\)\}\\leq p\_\{l\}<\\infty, exact population tangent completeness is impossible wheneverℋl\\mathcal\{H\}\_\{l\}is infinite\-dimensional, as is typical for non\-atomic hidden\-state distributions\.

On the fixed sampleSS, residual\-signal realizability meansqS,l∈Range⁡\(AS,l\)q\_\{S,l\}\\in\\operatorname\{Range\}\(A\_\{S,l\}\)\. The stronger condition thatAS,lA\_\{S,l\}be surjective ontoℋS,l\\mathcal\{H\}\_\{S,l\}is called*sample\-wise tangent completeness*\.

###### Assumption 4\(Independent finite\-sample probing\)\.

For the supplementary finite\-sample consistency result, all probabilities are conditional on the fixed reference model and the complete residual\-growth protocol\. LetSnprobe:=\{\(x~i,y~i\)\}i=1n​∼i\.i\.d\.​𝒟nS\_\{n\}^\{\\mathrm\{probe\}\}:=\\\{\(\\widetilde\{x\}\_\{i\},\\widetilde\{y\}\_\{i\}\)\\\}\_\{i=1\}^\{n\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\mathcal\{D\}^\{n\}be independent of the data and randomness used to trainfold∗f\_\{\\mathrm\{old\}\}^\{\*\}\. In that result, all empirical quantities are instantiated withS=SnprobeS=S\_\{n\}^\{\\mathrm\{probe\}\}andM=nM=n\.

For everyl∈ℐl\\in\\mathcal\{I\}, assume𝔼​‖Jl​\(zl\)⊤​ql​\(zl,y\)‖22<∞\\mathbb\{E\}\\\|J\_\{l\}\(z\_\{l\}\)^\{\\top\}q\_\{l\}\(z\_\{l\},y\)\\\|\_\{2\}^\{2\}<\\infty\. Asnnincreases, the reference model and the complete protocol remain fixed; adaptive selection of a new candidate or residual family after observing the probe sample is not covered\.

##### Practical motivation\.

We focus on first\-order optimization because gradient\-based methods are the practical default for modern large\-scale neural networks, whereas exact and structured second\-order methods introduce substantially larger memory and computational costs\(Anilet al\.[2020](https://arxiv.org/html/2608.14664#bib.bib18)\)\. Residual\-signal realizability is motivated by common ResNet and Transformer branches, which typically consist of a feature\-producing subnetwork followed by a trainable terminal output map\(Heet al\.[2016](https://arxiv.org/html/2608.14664#bib.bib1); Vaswaniet al\.[2017](https://arxiv.org/html/2608.14664#bib.bib2)\)\. Zero\-initializing that terminal map preserves the reference function while retaining nonzero upstream features\. An unconstrained output projection alone does not guarantee realizability, however: the upstream features must also span the sample\-dependent variation required by the current activation\-gradient signal\.

### Notation

##### Residual growth values\.

Define the local population and empirical residual growth values by𝒱l:=‖Πl​ml‖ℋl\\mathcal\{V\}\_\{l\}:=\\\|\\Pi\_\{l\}m\_\{l\}\\\|\_\{\\mathcal\{H\}\_\{l\}\}and𝒱S,l:=‖ΠS,l​qS,l‖S,l\\mathcal\{V\}\_\{S,l\}:=\\\|\\Pi\_\{S,l\}q\_\{S,l\}\\\|\_\{S,l\}\. Define their depth\-wide counterparts by𝒱depth:=maxl∈ℐ⁡𝒱l\\mathcal\{V\}\_\{\\mathrm\{depth\}\}:=\\max\_\{l\\in\\mathcal\{I\}\}\\mathcal\{V\}\_\{l\}and𝒱S,depth:=maxl∈ℐ⁡𝒱S,l\\mathcal\{V\}\_\{S,\\mathrm\{depth\}\}:=\\max\_\{l\\in\\mathcal\{I\}\}\\mathcal\{V\}\_\{S,l\}\. Their dependence on the fixed reference model and residual\-growth protocol is suppressed throughout\.

##### Standard zero\-output residual blocks\.

At candidatell, considerhl,Ul,Vl​\(z\):=Vl​ψl,Ul​\(z\)h\_\{l,U\_\{l\},V\_\{l\}\}\(z\):=V\_\{l\}\\psi\_\{l,U\_\{l\}\}\(z\), whereψl,Ul​\(z\)∈ℝrl\\psi\_\{l,U\_\{l\}\}\(z\)\\in\\mathbb\{R\}^\{r\_\{l\}\}andVl∈ℝNl×rlV\_\{l\}\\in\\mathbb\{R\}^\{N\_\{l\}\\times r\_\{l\}\}\. During the local insertion test,Ul=U0,lU\_\{l\}=U\_\{0,l\}is fixed andVlV\_\{l\}is the active parameter, initialized atVl=0V\_\{l\}=0\. We identifyθl=vec⁡\(Vl\)\\theta\_\{l\}=\\operatorname\{vec\}\(V\_\{l\}\),pl=Nl​rlp\_\{l\}=N\_\{l\}r\_\{l\}, andΘl=ℝpl\\Theta\_\{l\}=\\mathbb\{R\}^\{p\_\{l\}\}\. Writingψl​\(z\):=ψl,U0,l​\(z\)\\psi\_\{l\}\(z\):=\\psi\_\{l,U\_\{0,l\}\}\(z\), every matrix perturbationΔ​Vl\\Delta V\_\{l\}satisfiesJl​\(z\)​vec⁡\(Δ​Vl\)=Δ​Vl​ψl​\(z\)J\_\{l\}\(z\)\\operatorname\{vec\}\(\\Delta V\_\{l\}\)=\\Delta V\_\{l\}\\psi\_\{l\}\(z\)\. AllowingUlU\_\{l\}to vary as an additional active coordinate does not enlarge the first\-order tangent space atVl=0V\_\{l\}=0, becauseDUl​hl,Ul,Vl​\(z\)​\[Δ​Ul\]=Vl​DUl​ψl,Ul​\(z\)​\[Δ​Ul\]=0D\_\{U\_\{l\}\}h\_\{l,U\_\{l\},V\_\{l\}\}\(z\)\[\\Delta U\_\{l\}\]=V\_\{l\}D\_\{U\_\{l\}\}\\psi\_\{l,U\_\{l\}\}\(z\)\[\\Delta U\_\{l\}\]=0at the designated origin\. Thus fixingUl=U0,lU\_\{l\}=U\_\{0,l\}is a lossless first\-order simplification\.

Define the Frobenius inner product and norm by⟨B,C⟩F:=tr⁡\(B⊤​C\)\\langle B,C\\rangle\_\{F\}:=\\operatorname\{tr\}\(B^\{\\top\}C\)and‖B‖F:=⟨B,B⟩F1/2\\\|B\\\|\_\{F\}:=\\langle B,B\\rangle\_\{F\}^\{1/2\}\. DefineCpop\(l\):=𝔼​\[ql​\(zl,y\)​ψl​\(zl\)⊤\]C\_\{\\mathrm\{pop\}\}^\{\(l\)\}:=\\mathbb\{E\}\[q\_\{l\}\(z\_\{l\},y\)\\psi\_\{l\}\(z\_\{l\}\)^\{\\top\}\]andCS\(l\):=M−1​∑i=1Mql,i​ψl​\(zl,i\)⊤C\_\{S\}^\{\(l\)\}:=M^\{\-1\}\\sum\_\{i=1\}^\{M\}q\_\{l,i\}\\psi\_\{l\}\(z\_\{l,i\}\)^\{\\top\}\. LetΨS\(l\):=\[ψl​\(zl,1\),…,ψl​\(zl,M\)\]∈ℝrl×M\\Psi\_\{S\}^\{\(l\)\}:=\[\\psi\_\{l\}\(z\_\{l,1\}\),\\ldots,\\psi\_\{l\}\(z\_\{l,M\}\)\]\\in\\mathbb\{R\}^\{r\_\{l\}\\times M\}, so thatCS\(l\)=M−1​QS\(l\)​\(ΨS\(l\)\)⊤C\_\{S\}^\{\(l\)\}=M^\{\-1\}Q\_\{S\}^\{\(l\)\}\(\\Psi\_\{S\}^\{\(l\)\}\)^\{\\top\}\.

LetΠΨ,l:=\(ΨS\(l\)\)⊤​\[ΨS\(l\)​\(ΨS\(l\)\)⊤\]†​ΨS\(l\)\\Pi\_\{\\Psi,l\}:=\(\\Psi\_\{S\}^\{\(l\)\}\)^\{\\top\}\[\\Psi\_\{S\}^\{\(l\)\}\(\\Psi\_\{S\}^\{\(l\)\}\)^\{\\top\}\]^\{\\dagger\}\\Psi\_\{S\}^\{\(l\)\}be the orthogonal projector onto the row space ofΨS\(l\)\\Psi\_\{S\}^\{\(l\)\}, where†is the Moore–Penrose pseudoinverse\. Under the tuple–matrix identification ofℋS,l\\mathcal\{H\}\_\{S,l\}, the matrix representation ofΠS,l​qS,l\\Pi\_\{S,l\}q\_\{S,l\}isQS\(l\)​ΠΨ,lQ\_\{S\}^\{\(l\)\}\\Pi\_\{\\Psi,l\}\. Consequently, finite\-sample residual\-signal realizability is equivalent toQS\(l\)=QS\(l\)​ΠΨ,lQ\_\{S\}^\{\(l\)\}=Q\_\{S\}^\{\(l\)\}\\Pi\_\{\\Psi,l\}\. Sample\-wise tangent completeness is equivalent torank⁡\(ΨS\(l\)\)=M\\operatorname\{rank\}\(\\Psi\_\{S\}^\{\(l\)\}\)=M, in which caseΠΨ,l=IM\\Pi\_\{\\Psi,l\}=I\_\{M\}\.

When one candidate is fixed, we suppresslland writeCpopC\_\{\\mathrm\{pop\}\},CSC\_\{S\},QSQ\_\{S\},ΨS\\Psi\_\{S\}, andΠΨ\\Pi\_\{\\Psi\}\.

## First\-Order Residual Depth Saturation

This section characterizes the exact boundary between residual depth that retains first\-order optimization value and residual depth that cannot be activated from a function\-preserving initialization\.

### Main Theorem: An Exact Saturation Boundary

Condition[1](https://arxiv.org/html/2608.14664#Thmcondition1)was introduced in prior work as a sufficient condition for a locally improving residual insertion\(Chenget al\.[2026](https://arxiv.org/html/2608.14664#bib.bib17)\)\. We show that, under the fixed protocol, it is also necessary for first\-order improvement\.

###### Theorem 1\(Necessary and sufficient characterization of first\-order residual depth saturation\)\.

Suppose Assumptions[1](https://arxiv.org/html/2608.14664#Thmassumption1)and[2](https://arxiv.org/html/2608.14664#Thmassumption2)hold\. At every candidatel∈ℐl\\in\\mathcal\{I\}, the following statements are equivalent:

1. 1\.Condition[1](https://arxiv.org/html/2608.14664#Thmcondition1)holds atll\.
2. 2\.gpop\(l\)≠0g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\neq 0\.
3. 3\.Πl​ml≠0\\Pi\_\{l\}m\_\{l\}\\neq 0\.
4. 4\.𝒱l\>0\\mathcal\{V\}\_\{l\}\>0\.
5. 5\.There existsη¯l\>0\\bar\{\\eta\}\_\{l\}\>0such that, for everyη∈\(0,η¯l\)\\eta\\in\(0,\\bar\{\\eta\}\_\{l\}\),Φl​\(η​dl​\(gpop\(l\)\)\)<Φl​\(0\)\\Phi\_\{l\}\(\\eta d\_\{l\}\(g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)\)<\\Phi\_\{l\}\(0\)\.

Consequently,𝒱depth=0\\mathcal\{V\}\_\{\\mathrm\{depth\}\}=0if and only if every candidate is population first\-order saturated, equivalently, if and only if the reference model is first\-order depth\-saturated relative to the fixed protocol\.

For every fixed sampleSSsatisfying the differentiability conditions in Assumption[1](https://arxiv.org/html/2608.14664#Thmassumption1), the empirical counterparts are also equivalent: empirical Condition[1](https://arxiv.org/html/2608.14664#Thmcondition1),gS\(l\)≠0g\_\{S\}^\{\(l\)\}\\neq 0,ΠS,l​qS,l≠0\\Pi\_\{S,l\}q\_\{S,l\}\\neq 0,𝒱S,l\>0\\mathcal\{V\}\_\{S,l\}\>0, and the existence ofη¯S,l\>0\\bar\{\\eta\}\_\{S,l\}\>0such thatΦS,l​\(η​dl​\(gS\(l\)\)\)<ΦS,l​\(0\)\\Phi\_\{S,l\}\(\\eta d\_\{l\}\(g\_\{S\}^\{\(l\)\}\)\)<\\Phi\_\{S,l\}\(0\)for everyη∈\(0,η¯S,l\)\\eta\\in\(0,\\bar\{\\eta\}\_\{S,l\}\)\.

Items 1–4 characterize the existence of an arbitrary strict first\-order descent direction in the fixed residual parameter space\. Item 5 is the optimizer\-realization statement: it concerns specifically the update obtained by applying the fixed direction mapdld\_\{l\}to the exact residual gradient\. Its reverse implication uses the zero\-state propertydl​\(0\)=0d\_\{l\}\(0\)=0and should not be read as a claim that a stationary point cannot be left by stochastic or higher\-order mechanisms\.

The qualifier*first\-order*is essential\. The theorem does not exclude improvements obtained through higher\-order curvature, externally injected perturbations, stochastic escape, or a different residual\-growth protocol\. The projected activation gradient and the residual\-parameter gradient share the same zero\-versus\-nonzero boundary, but their norms generally differ\. The growth value depends only on the residual tangent subspace and is invariant under regular local reparameterizations that preserve this subspace, whereas the Euclidean parameter\-gradient norm is coordinate\-dependent\.

A finite joint\-insertion extension, under an additional joint Fréchet\-regularity condition, is stated in the supplementary material\. A separate local\-smoothness corollary gives an explicit one\-step decrease for ordinary gradient descent without changing the main equivalence\.

### Residual\-Signal Realizability in Common Architectures

Theorem[1](https://arxiv.org/html/2608.14664#Thmtheorem1)shows that the universal criterion is the activation\-gradient component contained in the fixed residual tangent space\. Residual\-signal realizability permits this projected criterion to be replaced by the raw activation\-gradient signal\.

###### Theorem 2\(Activation\-gradient characterization under residual\-signal realizability\)\.

Suppose Assumptions[1](https://arxiv.org/html/2608.14664#Thmassumption1)and[2](https://arxiv.org/html/2608.14664#Thmassumption2)hold at a fixed candidatell\.

1. 1\.Ifml∈𝒯h\(l\)m\_\{l\}\\in\\mathcal\{T\}\_\{h\}^\{\(l\)\}, then𝒱l=‖ml‖ℋl\\mathcal\{V\}\_\{l\}=\\\|m\_\{l\}\\\|\_\{\\mathcal\{H\}\_\{l\}\}\. Consequently, Condition[1](https://arxiv.org/html/2608.14664#Thmcondition1)holds if and only ifml≠0m\_\{l\}\\neq 0, and the candidate is population first\-order saturated if and only ifml=0m\_\{l\}=0μl\\mu\_\{l\}\-almost everywhere\.
2. 2\.IfqS,l∈Range⁡\(AS,l\)q\_\{S,l\}\\in\\operatorname\{Range\}\(A\_\{S,l\}\), then𝒱S,l=‖qS,l‖S,l\\mathcal\{V\}\_\{S,l\}=\\\|q\_\{S,l\}\\\|\_\{S,l\}\. Consequently, empirical Condition[1](https://arxiv.org/html/2608.14664#Thmcondition1)holds if and only ifqS,l≠0q\_\{S,l\}\\neq 0, equivalentlyQS\(l\)≠0Q\_\{S\}^\{\(l\)\}\\neq 0\.
3. 3\.Independently of Assumption[3](https://arxiv.org/html/2608.14664#Thmassumption3), population Condition[1](https://arxiv.org/html/2608.14664#Thmcondition1)holds if and only ifCpop\(l\)≠0C\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\neq 0, and its empirical counterpart holds if and only ifCS\(l\)≠0C\_\{S\}^\{\(l\)\}\\neq 0\.
4. 4\.For this standard block, sample\-wise tangent completeness is equivalent torank⁡\(ΨS\(l\)\)=M\\operatorname\{rank\}\(\\Psi\_\{S\}^\{\(l\)\}\)=M\. Under this condition,ΠΨ,l=IM\\Pi\_\{\\Psi,l\}=I\_\{M\}, and thereforeCS\(l\)=0C\_\{S\}^\{\(l\)\}=0if and only ifQS\(l\)=0Q\_\{S\}^\{\(l\)\}=0\.

If residual\-signal realizability holds at every candidate, simultaneous activation\-gradient vanishing is equivalent to depth\-wide first\-order saturation\.

Activation\-gradient vanishing is always sufficient for local first\-order saturation, even without realizability\. Realizability is needed only for the reverse implication\. More quantitatively, letεreal∈\[0,1\)\\varepsilon\_\{\\mathrm\{real\}\}\\in\[0,1\)\. Ifml≠0m\_\{l\}\\neq 0anddistℋl⁡\(ml,𝒯h\(l\)\)≤εreal​‖ml‖ℋl\\operatorname\{dist\}\_\{\\mathcal\{H\}\_\{l\}\}\(m\_\{l\},\\mathcal\{T\}\_\{h\}^\{\(l\)\}\)\\leq\\varepsilon\_\{\\mathrm\{real\}\}\\\|m\_\{l\}\\\|\_\{\\mathcal\{H\}\_\{l\}\}, then Pythagoras gives𝒱l≥1−εreal2​‖ml‖ℋl\>0\\mathcal\{V\}\_\{l\}\\geq\\sqrt\{1\-\\varepsilon\_\{\\mathrm\{real\}\}^\{2\}\}\\\|m\_\{l\}\\\|\_\{\\mathcal\{H\}\_\{l\}\}\>0\. The same statement holds on a fixed sample after replacingmlm\_\{l\},𝒯h\(l\)\\mathcal\{T\}\_\{h\}^\{\(l\)\}, and∥⋅∥ℋl\\\|\\cdot\\\|\_\{\\mathcal\{H\}\_\{l\}\}byqS,lq\_\{S,l\},Range⁡\(AS,l\)\\operatorname\{Range\}\(A\_\{S,l\}\), and∥⋅∥S,l\\\|\\cdot\\\|\_\{S,l\}, respectively\.

Many ResNet and Transformer branches contain a feature\-producing subnetwork followed by a trainable output map\. Zero\-initializing the terminal map preserves the reference function while keeping the upstream features nonzero\. Wide and diverse features can therefore make signal\-relative realizability plausible\.

For an arbitrary fixed residual block, the projected activation gradient is the parameterization\-invariant exact criterion; the residual\-parameter gradient has the same zero\-versus\-nonzero boundary\. For a standard zero\-output block, the latter reduces to the activation–feature cross\-gradient\. Under residual\-signal realizability, the raw activation gradient may be used directly\.

### Finite\-Sample Certification

The preceding results characterize saturation through population residual\-parameter gradients, which are not directly observable\. We therefore consider an independent probe sampleSnprobe=\{\(x~i,y~i\)\}i=1nS\_\{n\}^\{\\mathrm\{probe\}\}=\\\{\(\\widetilde\{x\}\_\{i\},\\widetilde\{y\}\_\{i\}\)\\\}\_\{i=1\}^\{n\}drawn after the reference model and the complete residual\-growth protocol have been fixed\.

For each candidatel∈ℐl\\in\\mathcal\{I\}, letξl​\(x,y\):=Jl​\(zl\)⊤​ql​\(zl,y\)\\xi\_\{l\}\(x,y\):=J\_\{l\}\(z\_\{l\}\)^\{\\top\}q\_\{l\}\(z\_\{l\},y\),gpop\(l\):=𝔼​\[ξl\]g\_\{\\mathrm\{pop\}\}^\{\(l\)\}:=\\mathbb\{E\}\[\\xi\_\{l\}\], andg^n\(l\):=1n​∑i=1nξl​\(x~i,y~i\)\\widehat\{g\}\_\{n\}^\{\(l\)\}:=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\xi\_\{l\}\(\\widetilde\{x\}\_\{i\},\\widetilde\{y\}\_\{i\}\)\. DefineΓ:=maxl∈ℐ∥gpop\(l\)∥2\\Gamma:=\\max\_\{l\\in\\mathcal\{I\}\}\\lVert g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\rVert\_\{2\}andΓ^n:=maxl∈ℐ∥g^n\(l\)∥2\\widehat\{\\Gamma\}\_\{n\}:=\\max\_\{l\\in\\mathcal\{I\}\}\\lVert\\widehat\{g\}\_\{n\}^\{\(l\)\}\\rVert\_\{2\}\.

###### Proposition 1\(Finite\-sample error of the saturation score\)\.

Suppose Assumption[4](https://arxiv.org/html/2608.14664#Thmassumption4)holds, and letσl2:=𝔼​∥ξl−gpop\(l\)∥22\\sigma\_\{l\}^\{2\}:=\\mathbb\{E\}\\lVert\\xi\_\{l\}\-g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\rVert\_\{2\}^\{2\}\. Then, for everyt\>0t\>0,

Pr⁡\(\|Γ^n−Γ\|≥t\)≤1n​t2​∑l∈ℐσl2\.\\Pr\\left\(\\left\|\\widehat\{\\Gamma\}\_\{n\}\-\\Gamma\\right\|\\geq t\\right\)\\leq\\frac\{1\}\{nt^\{2\}\}\\sum\_\{l\\in\\mathcal\{I\}\}\\sigma\_\{l\}^\{2\}\.\(1\)
Consequently, fixδ≥0\\delta\\geq 0and consider the separated hypothesesΓ≤δ\\Gamma\\leq\\deltaandΓ≥δ\+2​t\\Gamma\\geq\\delta\+2t\. The decision rule that declares saturation whenΓ^n≤δ\+t\\widehat\{\\Gamma\}\_\{n\}\\leq\\delta\+thas error probability at most the right\-hand side of \([1](https://arxiv.org/html/2608.14664#Sx4.E1)\)\.

## Experiments

We conduct two complementary experiments to evaluate the proposed depth\-saturation criterion\. First, we examine how the maximum per\-example activation\-gradient norm changes with depth across ResNets trained on CIFAR\-10, CIFAR\-100, and ImageNet\-100, GPT\-2\-style models trained on FineWeb\-Edu, and official Pythia checkpoints continued\-pretrained on the same corpus\. Second, we compare function\-preserving growth with training the same final architectures from random initialization to determine whether zero\-output insertion impairs converged solution quality\. The training and model settings are provided in the Appendix\. All models are well\-trained\.

### Activation\-Gradient Norms Across Model Depth

![Refer to caption](https://arxiv.org/html/2608.14664v1/x1.png)

![Refer to caption](https://arxiv.org/html/2608.14664v1/x2.png)

![Refer to caption](https://arxiv.org/html/2608.14664v1/x3.png)

![Refer to caption](https://arxiv.org/html/2608.14664v1/x4.png)

![Refer to caption](https://arxiv.org/html/2608.14664v1/x5.png)

![Refer to caption](https://arxiv.org/html/2608.14664v1/x6.png)

Figure 1:The Loss and Maximum Sample\-Wise Activation\-Gradient Norm Across Model Depth\.We first examine how the fixed\-sample activation\-gradient signal changes as model depth increases\. This analysis covers ResNets trained on CIFAR\-10, CIFAR\-100, and ImageNet\-100, official Pythia checkpoints continued\-pretrained on FineWeb\-Edu, and GPT\-2\-style models trained from scratch on FineWeb\-Edu\.

For each admissible candidatel∈ℐl\\in\\mathcal\{I\}, i\.e\., at the input of an existing residual structure in this experiment, defineAGl:=‖qS,l‖S,l=\(1M​∑i=1M‖ql,i‖22\)1/2,AGmax:=maxl∈ℐ⁡AGl\\mathrm\{AG\}\_\{l\}:=\\\|q\_\{S,l\}\\\|\_\{S,l\}=\\left\(\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\\|q\_\{l,i\}\\\|\_\{2\}^\{2\}\\right\)^\{1/2\},\\mathrm\{AG\}\_\{\\max\}:=\\max\_\{l\\in\\mathcal\{I\}\}\\mathrm\{AG\}\_\{l\}\.

Under sample\-wise residual\-signal realizability,qS,l∈Range⁡\(AS,l\)q\_\{S,l\}\\in\\operatorname\{Range\}\(A\_\{S,l\}\), the empirical activation\-gradient signal lies entirely in the admissible residual tangent space\. Hence,ΠS,l​qS,l=qS,l,VS,l=‖ΠS,l​qS,l‖S,l=AGl\\Pi\_\{S,l\}q\_\{S,l\}=q\_\{S,l\},\\qquad V\_\{S,l\}=\\\|\\Pi\_\{S,l\}q\_\{S,l\}\\\|\_\{S,l\}=\\mathrm\{AG\}\_\{l\}\. If realizability holds at every candidate, thenVS,depth=maxl∈ℐ⁡VS,l=AGmaxV\_\{S,\\mathrm\{depth\}\}=\\max\_\{l\\in\\mathcal\{I\}\}V\_\{S,l\}=\\mathrm\{AG\}\_\{\\max\}\. Thus, under sample\-wise realizability,AGmax\\mathrm\{AG\}\_\{\\max\}exactly equals the empirical first\-order residual\-growth value\. Without verified realizability, we only haveVS,depth≤AGmaxV\_\{S,\\mathrm\{depth\}\}\\leq\\mathrm\{AG\}\_\{\\max\}, soAGmax\\mathrm\{AG\}\_\{\\max\}is the upper bound ofVS,depthV\_\{S,\\mathrm\{depth\}\}\. However, it is enough to use this upper bound to gain the value ofVS,depthV\_\{S,\\mathrm\{depth\}\}because for most of cases,A​GmaxAG\_\{\\max\}is close to zero\.

#### Analysis

Figure[1](https://arxiv.org/html/2608.14664#Sx5.F1)shows a broadly consistent transition from a high\-signal, depth\-beneficial regime to a stable low\-signal regime\. For clearly show resluts, CIFAR 100 results are divideds into two subfigures\. For the controlled ResNet and GPT\-2\-style sweeps,A​GmaxAG\_\{\\max\}decreases rapidly together with task loss at shallow and intermediate depths, and then changes only modestly once the loss improvements begin to diminish\. Pythia exhibits a similar but sharper transition\. ImageNet\-100 is less monotone at intermediate depths, but the overall envelope ofA​GmaxAG\_\{\\max\}still decreases and reaches its minimum at the largest evaluated depth\.

The relevant observation is therefore not strict monotonicity at every checkpoint, but the emergence of a persistent low\-signal plateau\. BecauseA​GmaxAG\_\{\\max\}is the maximum over all admissible insertion locations, a smallA​GmaxAG\_\{\\max\}conservatively implies that the empirical projected residual\-growth value is small at every candidate\. The alignment between this regime and diminishing task\-loss improvements is consistent with the proposed first\-order saturation criterion\. However, a large raw score does not guarantee a realizable residual direction, and the training\-sample diagnostic need not exactly track test loss performance\. These results therefore motivate using a stable low\-signal plateau, rather than a universal threshold or an isolated checkpoint, as the practical indicator of residual depth saturation\.

As shown in Figure[1](https://arxiv.org/html/2608.14664#Sx5.F1), the proposed depth\-saturation indicator exhibits a consistent relationship with the marginal benefit of increasing model depth across CIFAR\-10, CIFAR\-100, and Pythia\. At relatively shallow depths, the indicator decreases rapidly as additional residual structures improve model performance\. However, once the indicator approaches the low\-signal regime—approximately5∗10−35\*10^\{\-3\}, further increases in depth yield little or no additional performance improvement\. This transition occurs at approximately 16 residual structures for the CIFAR\-10 ResNet \(A​Gm​a​x=0\.00381AG\_\{max\}=0\.00381\), 11 residual structures for the CIFAR\-100 ResNet \(A​Gm​a​x=0\.00320AG\_\{max\}=0\.00320\), and 33 residual structures for Pythia \(A​Gm​a​x=0\.0008AG\_\{max\}=0\.0008\)\. The close alignment between the emergence of a near\-zero indicator and the disappearance of measurable performance gains supports the use of the proposed metric as a practical diagnostic of residual depth sufficiency\. In particular, a persistently small value suggests that little first\-order optimization value remains available from adding further residual structures\.

##### Extended\-depth evaluation\.

We conducted independent large\-depth experiments \(Training hundreds of these models is prohibitively expensive\.\)\. For ImageNet\-100, ResNet\-256/257/258 yieldedA​Gm​a​xAG\_\{max\}scores of0\.00200\.0020/0\.00180\.0018/0\.00110\.0011, with training losses decreasing to0\.0170\.017/0\.0150\.015/0\.0110\.011and test losses fluctuating at0\.660\.66/0\.720\.72/0\.590\.59\. Similarly, GPT\-2 configurations \(700/701/702 residual structures\) showedA​Gm​a​xAG\_\{max\}scores of0\.00330\.0033/0\.00290\.0029/0\.00220\.0022, training losses decreasing to2\.672\.67/2\.622\.62/2\.552\.55, and test losses fluctuating at2\.222\.22/2\.832\.83/2\.072\.07\. In both settings,A​Gm​a​xAG\_\{max\}scores remained below approximately5×10−35\\times 10^\{\-3\}\. The change of training and test loss is so small\. Given the probabilistic nature of test loss performance and its sensitivity to finite\-sample and optimization variability, depth saturation need not manifest as monotonically worsening test loss\. Crucially, test performance merely fluctuated within a range without systematic gains from increased depth\. This confirms that onceA​Gm​a​xAG\_\{max\}scores reach a persistent near\-zero level, additional residual structures yield minimal marginal improvements in test loss performance, even if isolated deeper checkpoints occasionally achieve better point estimates\.

### Function\-Preserving Growth versus From\-Scratch Training

For the same four CIFAR\-10 ResNet configurations, we compare the function\-preserving grown model with a model of the same final architecture trained from random initialization\. Both models use the same optimization recipe and are trained to their respective convergence criteria\. This comparison tests whether the disappearance of post\-growth improvement could be explained by an optimization disadvantage caused by the function\-preserving initialization\. It is a convergence\-quality comparison rather than a compute\-matched estimate of training efficiency\. The model in this section is well\-trained and fully converged with enough computational resources\. The Local Effect of Activation\-Gradient\-Matched Insertions experiments is shown in Appendix\.

##### Comparison protocol\.

We evaluate four ResNet configurations on CIFAR\-10\. For each configuration, the*Growth*model is obtained by inserting one globally zero\-output residual block into a converged shallower network\. The insertion preserves the reference function exactly before optimization\. After insertion, the parameters of the deeper model are trained using the classification objective until the prescribed convergence criterion is reached\. The corresponding*Scratch*model has exactly the same final architecture and parameter count, but all of its parameters are initialized randomly and trained from scratch\.

The two procedures use the same dataset, data preprocessing, model architecture, optimizer family, regularization, and convergence criterion\. This experiment compares the quality of the solutions reachable from the two initializations\. It is not a compute\-matched comparison of training efficiency, because the Growth model inherits the optimization already invested in its shallower reference model\.

Table 1:Converged losses for training from scratch and function\-preserving block growth on CIFAR\-10\. Each row compares the same final ResNet architecture under the two initialization procedures\.Train LossTest LossModelScratchGrowthScratchGrowthResNet\-100\.00880\.00700\.30450\.2589ResNet\-110\.00820\.00680\.28740\.2626ResNet\-120\.00590\.00690\.28220\.2580ResNet\-130\.00590\.00570\.29480\.2510
#### Analysis

As shown in Table[1](https://arxiv.org/html/2608.14664#Sx5.T1), function\-preserving growth reaches training losses that are closely comparable to those obtained by training the same final architectures from scratch\. In three of the four configurations—ResNet\-10, ResNet\-11, and ResNet\-13—Growth achieves a lower training loss, whereas ResNet\-12 shows only a small increase relative to Scratch\. Averaged across all four architectures, the training loss is0\.00660\.0066for Growth and0\.00720\.0072for Scratch\. These differences are small in absolute magnitude and do not indicate a systematic optimization disadvantage caused by the zero\-output initialization\. In particular, preserving the original network function at insertion does not appear to trap the expanded model near the inherited solution or prevent the enlarged architecture from reaching a competitive minimum after subsequent task\-loss optimization\. Within the evaluated CIFAR\-10 ResNet configurations, function\-preserving insertion therefore provides a viable initialization for continued training\.

The test loss results are also consistently favorable to Growth\. Growth obtains a lower test loss for all four final architectures, with absolute reductions ranging from0\.02420\.0242for ResNet\-12 to0\.04560\.0456for ResNet\-10\. The average test loss decreases from0\.29220\.2922to0\.25760\.2576, corresponding to a relative reduction of approximately11\.8%11\.8\\%\. This improvement is not accompanied by a systematic increase in training loss, suggesting that it cannot be explained simply by weaker fitting of the training data\. The train–test gap is generally smaller under Growth, especially for ResNet\-12, where Growth has a slightly higher training loss but a lower test loss\. This pattern is consistent with the inherited shallower representation changing the optimization trajectory of the expanded model and potentially inducing different implicit regularization\.

## Conclusion

We developed a first\-order framework for evaluating whether additional residual depth remains locally useful under a fixed function\-preserving growth protocol\. We showed that a residual insertion admits a strict first\-order improvement if and only if the conditional activation gradient has a nonzero projection onto the corresponding residual tangent space\. For standard zero\-output residual blocks, this condition reduces to an activation–feature cross\-gradient criterion\. In the evaluated CIFAR\-10 configurations, function\-preserving growth reaches converged training losses comparable to those obtained from scratch\. Together, these results establish tangent\-space projection as an exact local criterion for the remaining first\-order optimization value of residual depth\.

## References

- R\. Anil, V\. Gupta, T\. Koren, K\. Regan, and Y\. Singer \(2020\)Scalable second order optimization for deep learning\.arXiv preprint arXiv:2002\.09018\.Cited by:[Practical motivation\.](https://arxiv.org/html/2608.14664#Sx3.SSx2.SSS0.Px4.p1.1)\.
- T\. Ash \(1989\)Dynamic node creation in backpropagation networks\.Connection Science1\(4\),pp\. 365–375\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- T\. Bachlechner, B\. P\. Majumder, H\. Mao, G\. Cottrell, and J\. McAuley \(2021\)Rezero is all you need: fast convergence at large depth\.InUncertainty in artificial intelligence,pp\. 1352–1361\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- C\. Chen, Y\. Yin, L\. Shang, X\. Jiang, Y\. Qin, F\. Wang, Z\. Wang, X\. Chen, Z\. Liu, and Q\. Liu \(2022\)Bert2bert: towards reusable pretrained language models\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2134–2148\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- T\. Chen, I\. Goodfellow, and J\. Shlens \(2015\)Net2net: accelerating learning via knowledge transfer\.arXiv preprint arXiv:1511\.05641\.Cited by:[Introduction](https://arxiv.org/html/2608.14664#Sx1.p1.1),[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- D\. Cheng, Z\. Liu, J\. Sun, F\. Xia, B\. Zhang, D\. Liu, and Y\. Zhang \(2026\)A qualitative test\-risk mechanism for scaling behavior in normalized residual networks\.arXiv preprint arXiv:2605\.08297\.Cited by:[Introduction](https://arxiv.org/html/2608.14664#Sx1.p2.1),[Related Work](https://arxiv.org/html/2608.14664#Sx2.p4.1),[Problem Setup](https://arxiv.org/html/2608.14664#Sx3.SSx1.p2.4),[Main Theorem: An Exact Saturation Boundary](https://arxiv.org/html/2608.14664#Sx4.SSx1.p1.1)\.
- W\. Du, T\. Luo, Z\. Qiu, Z\. Huang, Y\. Shen, R\. Cheng, Y\. Guo, and J\. Fu \(2024\)Stacking your transformers: a closer look at model growth for efficient llm pre\-training\.Advances in Neural Information Processing Systems37,pp\. 10491–10540\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- U\. Evci, B\. van Merrienboer, T\. Unterthiner, M\. Vladymyrov, and F\. Pedregosa \(2022\)Gradmax: growing neural networks using gradient information\.arXiv preprint arXiv:2201\.05125\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- S\. E\. Fahlman and C\. Lebiere \(1990\)The cascade\-correlation learning architecture\.InAdvances in Neural Information Processing Systems 2,D\. S\. Touretzky \(Ed\.\),pp\. 524–532\.External Links:[Link](https://papers.nips.cc/paper_files/paper/1989/hash/69adc1e107f7f7d035d7baf04342e1ca-Abstract.html)Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- L\. Gong, D\. He, Z\. Li, T\. Qin, L\. Wang, and T\. Liu \(2019\)Efficient training of bert by progressively stacking\.InInternational conference on machine learning,pp\. 2337–2346\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- A\. Gromov, K\. Tirumala, H\. Shapourian, P\. Glorioso, and D\. A\. Roberts \(2024\)The unreasonable ineffectiveness of the deeper layers\.arXiv preprint arXiv:2403\.17887\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- X\. Gu, L\. Liu, H\. Yu, J\. Li, C\. Chen, and J\. Han \(2020\)On the transformer growth for progressive bert training\.arXiv preprint arXiv:2010\.12562\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[Practical motivation\.](https://arxiv.org/html/2608.14664#Sx3.SSx2.SSS0.Px4.p1.1)\.
- G\. Huang, Y\. Sun, Z\. Liu, D\. Sedra, and K\. Weinberger \(2016\)Deep networks with stochastic depth\.arXiv preprint arXiv:1603\.09382\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- C\. Krishnanunni, T\. Bui\-Thanh, and C\. Dawson \(2025\)Topological derivative approach for deep neural network architecture adaptation\.arXiv preprint arXiv:2502\.06885\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- Y\. Levine, N\. Wies, O\. Sharir, H\. Bata, and A\. Shashua \(2020\)Limits to depth efficiencies of self\-attention\.Advances in Neural Information Processing Systems33,pp\. 22640–22651\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- X\. Men, M\. Xu, Q\. Zhang, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen \(2024\)Shortgpt: layers in large language models are more redundant than you expect\.arXiv preprint arXiv:2403\.03853\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- R\. Mitchell, R\. Menzenbach, K\. Kersting, and M\. Mundt \(2023\)Self\-expanding neural networks\.arXiv preprint arXiv:2307\.04526\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- Y\. Qin, Y\. Lin, J\. Yi, J\. Zhang, X\. Han, Z\. Zhang, Y\. Su, Z\. Liu, P\. Li, M\. Sun,et al\.\(2021\)Knowledge inheritance for pre\-trained language models\.arXiv preprint arXiv:2105\.13880\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[Practical motivation\.](https://arxiv.org/html/2608.14664#Sx3.SSx2.SSS0.Px4.p1.1)\.
- A\. Veit, M\. J\. Wilber, and S\. Belongie \(2016\)Residual networks behave like ensembles of relatively shallow networks\.Advances in neural information processing systems29\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.
- M\. Verbockhaven, S\. Chevallier, G\. Charpiat, and T\. Rudkiewicz \(2024\)Growing tiny networks: spotting expressivity bottlenecks and fixing them optimally\.arXiv preprint arXiv:2405\.19816\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- P\. Wang, R\. Panda, L\. T\. Hennigen, P\. Greengard, L\. Karlinsky, R\. Feris, D\. D\. Cox, Z\. Wang, and Y\. Kim \(2023\)Learning to grow pretrained models for efficient transformer training\.arXiv preprint arXiv:2303\.00980\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- Y\. Wang, J\. Su, H\. Lu, C\. Xie, T\. Liu, J\. Yuan, H\. Lin, R\. Sun, and H\. Yang \(2024\)Lemon: lossless model expansion\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 3338–3367\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- T\. Wei, C\. Wang, Y\. Rui, and C\. W\. Chen \(2016\)Network morphism\.InInternational conference on machine learning,pp\. 564–572\.Cited by:[Introduction](https://arxiv.org/html/2608.14664#Sx1.p1.1),[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- W\. Wen, F\. Yan, Y\. Chen, and H\. Li \(2019\)Autogrow: automatic layer growing in deep convolutional networks\.arXiv preprint arXiv:1906\.02909\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- H\. Wu, W\. Wang, T\. Malepathirana, D\. Senanayake, D\. Oetomo, and S\. Halgamuge \(2024\)When to grow? a fitting risk\-aware policy for layer growing in deep neural networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 5994–6002\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- L\. Wu, B\. Liu, P\. Stone, and Q\. Liu \(2021\)Firefly neural architecture descent: a general approach for growing neural networks\.arXiv preprint arXiv:2102\.08574\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- L\. Wu, D\. Wang, and Q\. Liu \(2019\)Splitting steepest descent for growing neural architectures\.Advances in neural information processing systems32\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p2.1)\.
- X\. Yuan, P\. Savarese, and M\. Maire \(2023\)Accelerated training via incrementally growing neural networks using variance transfer and learning rate adaptation\.Advances in Neural Information Processing Systems36,pp\. 16673–16692\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p1.1)\.
- H\. Zhang, Y\. N\. Dauphin, and T\. Ma \(2019\)Fixup initialization: residual learning without normalization\.arXiv preprint arXiv:1901\.09321\.Cited by:[Related Work](https://arxiv.org/html/2608.14664#Sx2.p3.1)\.

## Appendix AComplete Notation

Tables[2](https://arxiv.org/html/2608.14664#A1.T2)–[4](https://arxiv.org/html/2608.14664#A1.T4)collect the symbols used in the main text and supplementary proofs\. Dependence on the fixed reference model and residual\-growth protocol is suppressed when no ambiguity arises\.

SymbolMeaning𝒳,𝒴,𝒟\\mathcal\{X\},\\mathcal\{Y\},\\mathcal\{D\}Input space, label space, and data distribution on𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}\.doutd\_\{\\mathrm\{out\}\}Predictor output dimension\.S,MS,MA fixed sampleS=\{\(xi,yi\)\}i=1MS=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{M\}and its cardinality\.ℓ,R,ℒS\\ell,R,\\mathcal\{L\}\_\{S\}Loss, population risk, and fixed\-sample empirical risk\.fold∗f\_\{\\mathrm\{old\}\}^\{\*\}Trained reference model\.ℐ,l\\mathcal\{I\},lFixed finite candidate set and one candidate\.fbot\(l\),ftop\(l\)f\_\{\\mathrm\{bot\}\}^\{\(l\)\},f\_\{\\mathrm\{top\}\}^\{\(l\)\}Reference\-model components below and above candidatell\.zl,Nl,μlz\_\{l\},N\_\{l\},\\mu\_\{l\}Hidden state, its dimension, and its induced distribution\.ℱres\(l\)\\mathcal\{F\}\_\{\\mathrm\{res\}\}^\{\(l\)\}Parameterized residual family at candidatell\.Θl,pl,θl\\Theta\_\{l\},p\_\{l\},\\theta\_\{l\}Active residual parameter domain, dimension, and local coordinate\.hl,θl,fl,θlh\_\{l,\\theta\_\{l\}\},f\_\{l,\\theta\_\{l\}\}Residual function and corresponding expanded model\.Φl,ΦS,l\\Phi\_\{l\},\\Phi\_\{S,l\}Population and empirical objectives as functions ofθl\\theta\_\{l\}\.vpop\(l\),vS\(l\)v\_\{\\mathrm\{pop\}\}^\{\(l\)\},v\_\{S\}^\{\(l\)\}Population and empirical directions in the residual non\-degeneracy condition stated in the main paper\.dl,ηd\_\{l\},\\etaFixed first\-order direction map and free positive step\-size scalar\.η¯l,η¯S,l\\bar\{\\eta\}\_\{l\},\\bar\{\\eta\}\_\{S,l\}Population and empirical local descent thresholds\.ϵAdam\\epsilon\_\{\\mathrm\{Adam\}\}Positive numerical\-stability constant in the first Adam/AdamW direction\.Bl,ρlB\_\{l\},\\rho\_\{l\}Integrable local Lipschitz envelope and neighborhood radius in the dominated\-differentiation remark in the main paper\.∥⋅∥σ\\\|\\cdot\\\|\_\{\\sigma\}Matrix operator norm\.Table 2:Model, data, regularity, and optimization notation\.SymbolMeaningql,mlq\_\{l\},m\_\{l\}Sample activation gradient and conditional population activation\-gradient signal\.JlJ\_\{l\}Jacobian ofhl,θlh\_\{l,\\theta\_\{l\}\}with respect toθl\\theta\_\{l\}, evaluated atθl=0\\theta\_\{l\}=0\.gpop\(l\),gS\(l\)g\_\{\\mathrm\{pop\}\}^\{\(l\)\},g\_\{S\}^\{\(l\)\}Population and empirical residual\-parameter gradients\.D​Φl​\(0\)​\[u\],D​ΦS,l​\(0\)​\[u\]D\\Phi\_\{l\}\(0\)\[u\],D\\Phi\_\{S,l\}\(0\)\[u\]Population and empirical Fréchet derivatives applied to directionuu\.ℋl,Al,𝒯h\(l\),Πl\\mathcal\{H\}\_\{l\},A\_\{l\},\\mathcal\{T\}\_\{h\}^\{\(l\)\},\\Pi\_\{l\}Population hidden\-state Hilbert space, tangent operator, tangent range, and orthogonal projector\.ℋS,l,qS,l,QS\(l\)\\mathcal\{H\}\_\{S,l\},q\_\{S,l\},Q\_\{S\}^\{\(l\)\}Empirical Hilbert space and tuple/matrix forms of the sample activation\-gradient signal\.AS,l,ΠS,lA\_\{S,l\},\\Pi\_\{S,l\}Stacked tangent operator and projector onto its range\.𝒱l,𝒱S,l\\mathcal\{V\}\_\{l\},\\mathcal\{V\}\_\{S,l\}Population and empirical local residual growth values\.𝒱depth,𝒱S,depth\\mathcal\{V\}\_\{\\mathrm\{depth\}\},\\mathcal\{V\}\_\{S,\\mathrm\{depth\}\}Maximal local growth values over the fixed candidate set\.Al∗,Gl,G^S,lA\_\{l\}^\{\*\},G\_\{l\},\\widehat\{G\}\_\{S,l\}Tangent adjoint, population tangent Gram operator, and empirical tangent Gram matrix\.distℋl⁡\(a,𝒯\)\\operatorname\{dist\}\_\{\\mathcal\{H\}\_\{l\}\}\(a,\\mathcal\{T\}\)Distance fromaato𝒯\\mathcal\{T\}in theℋl\\mathcal\{H\}\_\{l\}norm;distS,l\\operatorname\{dist\}\_\{S,l\}denotes the empirical analogue\.dim\(⋅\)\\dim\(\\cdot\)Dimension of a finite\-dimensional vector space or vectorized tensor\.εreal\\varepsilon\_\{\\mathrm\{real\}\}Relative residual\-realizability error in\[0,1\)\[0,1\)\.Ll,LS,lL\_\{l\},L\_\{S,l\}Optional population and empirical local smoothness constants in Corollary[1](https://arxiv.org/html/2608.14664#Thmcorollary1)\.Table 3:Activation\-gradient, tangent\-space, and growth\-value notation\.SymbolMeaningUl,U0,l,Vl,ψl,Ul,ψl,rlU\_\{l\},U\_\{0,l\},V\_\{l\},\\psi\_\{l,U\_\{l\}\},\\psi\_\{l\},r\_\{l\}Feature parameters, their designated value, active output projection, feature map, fixed feature map, and feature dimension in the standard block\.Cpop\(l\),CS\(l\)C\_\{\\mathrm\{pop\}\}^\{\(l\)\},C\_\{S\}^\{\(l\)\}Population and empirical activation–feature cross\-gradients\.ΨS\(l\),ΠΨ,l\\Psi\_\{S\}^\{\(l\)\},\\Pi\_\{\\Psi,l\}Residual\-feature matrix and projector onto its row space\.⟨⋅,⋅⟩F,∥⋅∥F\\langle\\cdot,\\cdot\\rangle\_\{F\},\\\|\\cdot\\\|\_\{F\}Frobenius inner product and norm\.,†vec,IM\{\}^\{\\dagger\},\\operatorname\{vec\},I\_\{M\}Moore–Penrose pseudoinverse, vectorization, andM×MM\\times Midentity\.K,\(l1,…,lK\),FjK,\(l\_\{1\},\\ldots,l\_\{K\}\),F\_\{j\}Number of jointly inserted blocks, their distinct locations listed from upstream to downstream, and original network segments between them\.Tlj,θlj,ιjT\_\{l\_\{j\},\\theta\_\{l\_\{j\}\}\},\\iota\_\{j\}Full residual transformationw↦w\+hlj,θlj​\(w\)w\\mapsto w\+h\_\{l\_\{j\},\\theta\_\{l\_\{j\}\}\}\(w\)and canonical injection into joint blockjj\.𝒦,θ𝒦,u𝒦\\mathcal\{K\},\\theta\_\{\\mathcal\{K\}\},u\_\{\\mathcal\{K\}\}Ordered joint candidate tuple, product parameter, and joint direction\.f𝒦,θ𝒦,Φ𝒦f\_\{\\mathcal\{K\},\\theta\_\{\\mathcal\{K\}\}\},\\Phi\_\{\\mathcal\{K\}\}Jointly expanded model and its population objective\.Snprobe,ξl,g^n\(l\)S\_\{n\}^\{\\mathrm\{probe\}\},\\xi\_\{l\},\\widehat\{g\}\_\{n\}^\{\(l\)\}Independent probe sample, one\-sample residual\-gradient contribution, and probe gradient estimate\.Γ,Γ^n,σl2\\Gamma,\\widehat\{\\Gamma\}\_\{n\},\\sigma\_\{l\}^\{2\}Population saturation score, empirical score, and residual\-gradient variance\.τn,Sat^n\\tau\_\{n\},\\widehat\{\\mathrm\{Sat\}\}\_\{n\}Detection threshold and empirical saturation decision\.δ,t,H0,H1\\delta,t,H\_\{0\},H\_\{1\}Practical near\-saturation tolerance, separation radius, and hypothesesH0:Γ≤δH\_\{0\}:\\Gamma\\leq\\deltaandH1:Γ≥δ\+2​tH\_\{1\}:\\Gamma\\geq\\delta\+2t\.𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}Indicator of an event\.𝖠𝖦l,𝖠𝖦max\\mathsf\{AG\}\_\{l\},\\mathsf\{AG\}\_\{\\max\}Experimental scores𝖠𝖦l:=‖qS,l‖S,l=M−1/2​‖QS\(l\)‖F\\mathsf\{AG\}\_\{l\}:=\\\|q\_\{S,l\}\\\|\_\{S,l\}=M^\{\-1/2\}\\\|Q\_\{S\}^\{\(l\)\}\\\|\_\{F\}and𝖠𝖦max:=maxl∈ℐ⁡𝖠𝖦l\\mathsf\{AG\}\_\{\\max\}:=\\max\_\{l\\in\\mathcal\{I\}\}\\mathsf\{AG\}\_\{l\}\.𝖠𝖦~l\\widetilde\{\\mathsf\{AG\}\}\_\{l\}Dimension\-normalized visualization score𝖠𝖦l/Nl\\mathsf\{AG\}\_\{l\}/\\sqrt\{N\_\{l\}\}; it is not used to rank candidates with unequal hidden dimensions\.Table 4:Standard\-block, joint\-insertion, supplementary, and experimental notation\.
## Appendix BDetailed Experimental Settings

This section gives the architecture, data, optimization, checkpoint, and measurement details for the depth\-scaling experiments\. Unless stated otherwise, depth is the controlled architectural variable: width, classification or language\-modeling heads, and the remaining model hyperparameters are held fixed within each sweep\.

### B\.1Depth Variables and Gradient Measurement

##### ResNet depth\.

The ResNet experiments start from a stagewise ResNet\-8 backbone containing one BasicBlock in each of three stages\. We denote byNNthe number of residual blocks added to this fixed backbone, so a model contains3\+N3\+Nresidual blocks in total\. Added blocks are assigned cyclically tolayer1,layer2, andlayer3\. This rule changes depth without changing the stage widths or classification head and avoids placing all added capacity at the end of the network\.

##### GPT\-2 depth\.

For GPT\-2\-style models,LLdenotes the total number of decoder blocks\. The baseline hasL=12L=12, andL=13L=13therefore denotes the baseline architecture with one additional decoder block\. Hidden size, feed\-forward size, number of attention heads, positional encoding, normalization, vocabulary, and the per\-update optimization recipe are fixed asLLvaries; the documented run durations are specified below\.

##### Recorded activation\-gradient statistic\.

For a residual block, decoder block, classifier, or language\-modeling head, letHl,bH\_\{l,b\}be the complete input\-activation tensor at locationllfor analysis batchbb, and letℒb\\mathcal\{L\}\_\{b\}be that batch’s loss\. Each analysis batch requires one forward and one backward pass\. The run\-time diagnostic stored by the experiment scripts is

𝖠𝖦^l:=1B​∑b=1B‖∇Hl,bℒb‖2,\\widehat\{\\mathsf\{AG\}\}\_\{l\}:=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\left\\\|\\nabla\_\{H\_\{l,b\}\}\\mathcal\{L\}\_\{b\}\\right\\\|\_\{2\},\(2\)where the norm is taken once over the full, flattened activation\-gradient tensor and the resulting scalar is averaged overBBanalysis batches\. Inputs to the final fully connected classifier and LM head are recorded as diagnostic endpoints; the residual\-growth candidate set itself contains the residual\- or decoder\-block boundaries\.

Equation \([2](https://arxiv.org/html/2608.14664#A2.E2)\) documents the batch\-level quantity retained in the experimental logs\. The sample\-normalized Hilbert\-space score𝖠𝖦l\\mathsf\{AG\}\_\{l\}used in the theoretical comparisons is defined separately in Remark[1](https://arxiv.org/html/2608.14664#Thmremark1)\. For a fixed batch construction, both statistics vanish exactly when all recorded activation gradients vanish, but their nonzero numerical scales need not agree\. Quantitative comparisons therefore use a fixed analysis\-batch protocol within each model family\.

### B\.2CIFAR\-10 and CIFAR\-100

##### Data and preprocessing\.

CIFAR\-10 and CIFAR\-100 each contain50,00050\{,\}000training images and10,00010\{,\}000test images at resolution32×3232\\times 32\. Training images are randomly cropped after four\-pixel padding and randomly flipped horizontally\. Test images receive no random augmentation\. Images are then normalized using the mean and standard deviation of the corresponding dataset\. CIFAR\-10 and CIFAR\-100 use 10 and 100 output classes, respectively\.

##### Architecture\.

Both datasets use the same stagewise ResNet\-8 backbone\. The base channel count is 16, and the three stage widths are 16, 32, and 64\. Transitions between stages downsample with stride 2\. We sweepN=1,…,100N=1,\\ldots,100, using the cyclic block\-allocation rule described above\.

Configuration itemCIFAR\-10 and CIFAR\-100 settingObjective and optimizerCross\-entropy loss; SGD with momentum\.Initial learning rate0\.10\.1, multiplied by0\.10\.1after epochs 100 and 150\.Momentum and weight decayMomentum0\.90\.9; weight decay5×10−45\\times 10^\{\-4\}\.Batch size and random seedBatch size 128; seed 1\.Maximum duration300 epochs\.Early stoppingEnabled only after at least 180 epochs\. Training stops when the training loss fails to improve by more than10−410^\{\-4\}for 30 consecutive epochs\.Saved checkpointsThe final checkpoint \(last\) and the checkpoint with the lowest training loss \(best\_train\)\. Validation and test results are not used for checkpoint selection\.Table 5:Training protocol for the CIFAR depth sweeps\.

### B\.3ImageNet\-100

##### Subset construction and preprocessing\.

ImageNet\-100 is constructed from ImageNet\-1K and contains128,982128\{,\}982training images and5,0005\{,\}000validation images, with 50 validation images per class\. Training usesRandomResizedCrop\(224\)and random horizontal flipping\. At evaluation time, the shorter image side is resized to 256 pixels and a224×224224\\times 224center crop is taken\. Images are normalized by the standard ImageNet mean and standard deviation\. The data are loaded withtorchvision\.datasets\.ImageFolder, and the number of classes is inferred from the directory structure\.

##### Architecture and run configuration\.

To isolate the effects of dataset scale and depth, the ImageNet\-100 experiments retain the three\-stage CIFAR ResNet\-8 backbone, including its3×33\\times 3, stride\-1 stem\. In particular, they do not introduce the7×77\\times 7, stride\-2 convolution or max pooling used by standard ImageNet ResNets\. The initial sweep usesN∈\{1,5,9,…,49\}N\\in\\\{1,5,9,\\ldots,49\\\}, with base channel count 32\. Each configuration is run for 300 epochs with global batch size 128, automatic mixed precision, and data parallelism across four GPUs\.

### B\.4GPT\-2 Depth Sweep on Ascend

##### Model architecture\.

The GPT\-2 experiments use the MCoreGPTModelimplementation in Megatron\-LM/MindSpeed\. TheL=12L=12baseline follows the main dimensions of GPT\-2 Small: hidden size 768, feed\-forward size 3072, 12 attention heads, and maximum sequence length 1024\. It uses learned absolute positional embeddings, LayerNorm, GELU activations, attention and hidden dropout of 0\.1, and initialization standard deviation 0\.02\. The vocabulary contains50,25750\{,\}257tokens, input and output embeddings are tied, and linear\-layer biases are retained\.

Only the number of decoder blocks changes across the controlled depth sweep; all other architectural dimensions are held fixed\. The parameter count therefore increases with depth\. The extended\-depth configurations reported in the main paper follow the same architectural conventions\.

##### FineWeb\-Edu data\.

The training corpus is a FineWeb\-Edu subset containing approximately310,000,347310\{,\}000\{,\}347GPT\-2 tokens\. The data are divided into 98% training, 1% validation, and 1% test splits\. The training split contains approximately296,715296\{,\}715packed sequences of length 1024\. With global batch size 32, we define one*round*as 9,273 optimizer steps, approximately one complete pass over the training split\.

Configuration itemGPT\-2 depth\-sweep settingNumerical precision and device allocationBF16; one Ascend NPU per model for the initial controlled\-depth runs\.BatchingMicro\-batch size 4, global batch size 32, and eight gradient accumulation steps\.OptimizerAdam withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay 0\.1, and gradient clipping at 1\.0\.Learning\-rate scheduleInitial rate2\.5×10−42\.5\\times 10^\{\-4\}, minimum rate2\.5×10−52\.5\\times 10^\{\-5\}, 1% warmup, and cosine decay\.Random seed1234\.Training durationThe initial controlled\-depth runs are capped at five rounds\. Selected checkpoints are subsequently continued to a total of ten rounds\.Early stoppingThe range of the mean training losses over the most recent ten rounds must be below5×10−55\\times 10^\{\-5\}\. With a ten\-round cap, this criterion can first be evaluated only after the final round\.Saved checkpointsThe lowest\-training\-loss checkpoint \(best\_train\) and the final checkpoint \(last\) at every completed depth\.Table 6:Optimization protocol for the documented GPT\-2 runs\.The completed schedules documented in Table[6](https://arxiv.org/html/2608.14664#A2.T6)cover the controlled\-depth experiments\. The extended\-depth configurations reported in the main paper use the same model definition and per\-update optimization recipe\.

### B\.5Pythia Continued\-Pretraining Analysis

The Pythia study considers six deduplicated pretrained model configurations spanning a range of model sizes\. Each model is initialized from the officialstep143000checkpoint and then continued pretraining on FineWeb\-Edu before activation\-gradient measurement\.

During measurement, model parameters are frozen while gradients are retained for hidden activations\. Activation\-gradient norms are recorded at decoder\-block inputs using the fixed analysis protocol described in Equation \([2](https://arxiv.org/html/2608.14664#A2.E2)\)\. Because the Pythia configurations vary in architectural dimensions in addition to depth, these results provide cross\-model evidence rather than a controlled depth\-only comparison\.

## Appendix CPreliminary Identities

###### Lemma 1\(Population directional derivative\)\.

Under the first\-order regularity assumption stated in the main paper, for everyu∈ℝplu\\in\\mathbb\{R\}^\{p\_\{l\}\},

D​Φl​\(0\)​\[u\]\\displaystyle D\\Phi\_\{l\}\(0\)\[u\]=𝔼​\[ql​\(zl,y\)⊤​Jl​\(zl\)​u\]\\displaystyle=\\mathbb\{E\}\[q\_\{l\}\(z\_\{l\},y\)^\{\\top\}J\_\{l\}\(z\_\{l\}\)u\]=\(gpop\(l\)\)⊤​u=⟨ml,Al​u⟩ℋl\.\\displaystyle=\(g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)^\{\\top\}u=\\langle m\_\{l\},A\_\{l\}u\\rangle\_\{\\mathcal\{H\}\_\{l\}\}\.\(3\)

###### Proof\.

For𝒟\\mathcal\{D\}\-almost every\(x,y\)\(x,y\), the Fréchet chain rule gives

Dθl​ℓ​\(fl,0​\(x\),y\)​\[u\]=ql​\(zl,y\)⊤​Jl​\(zl\)​u\.D\_\{\\theta\_\{l\}\}\\ell\(f\_\{l,0\}\(x\),y\)\[u\]=q\_\{l\}\(z\_\{l\},y\)^\{\\top\}J\_\{l\}\(z\_\{l\}\)u\.
The first\-order regularity assumption stated in the main paper permits differentiation under the expectation, giving the first equality in \([3](https://arxiv.org/html/2608.14664#A3.E3)\)\. The second follows from the definition ofgpop\(l\)g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\. SinceJl​\(zl\)​uJ\_\{l\}\(z\_\{l\}\)uis measurable with respect tozlz\_\{l\}, conditional expectation gives

𝔼​\[ql⊤​Jl​u\]=𝔼​\[ml​\(zl\)⊤​Jl​\(zl\)​u\]=⟨ml,Al​u⟩ℋl\.\\mathbb\{E\}\[q\_\{l\}^\{\\top\}J\_\{l\}u\]=\\mathbb\{E\}\[m\_\{l\}\(z\_\{l\}\)^\{\\top\}J\_\{l\}\(z\_\{l\}\)u\]=\\langle m\_\{l\},A\_\{l\}u\\rangle\_\{\\mathcal\{H\}\_\{l\}\}\.∎

###### Lemma 2\(Bounded tangent operator and adjoint identity\)\.

The operatorAl:ℝpl→ℋlA\_\{l\}:\\mathbb\{R\}^\{p\_\{l\}\}\\to\\mathcal\{H\}\_\{l\}is bounded\. Its Hilbert adjoint satisfiesAl∗​ml=gpop\(l\)A\_\{l\}^\{\*\}m\_\{l\}=g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\. Consequently,

gpop\(l\)=0⟺Πl​ml=0⟺𝒱l=0\.g\_\{\\mathrm\{pop\}\}^\{\(l\)\}=0\\quad\\Longleftrightarrow\\quad\\Pi\_\{l\}m\_\{l\}=0\\quad\\Longleftrightarrow\\quad\\mathcal\{V\}\_\{l\}=0\.\(4\)

###### Proof\.

Foru∈ℝplu\\in\\mathbb\{R\}^\{p\_\{l\}\},

‖Al​u‖ℋl2=𝔼​‖Jl​\(zl\)​u‖22≤𝔼​‖Jl​\(zl\)‖σ2​‖u‖22,\\\|A\_\{l\}u\\\|\_\{\\mathcal\{H\}\_\{l\}\}^\{2\}=\\mathbb\{E\}\\\|J\_\{l\}\(z\_\{l\}\)u\\\|\_\{2\}^\{2\}\\leq\\mathbb\{E\}\\\|J\_\{l\}\(z\_\{l\}\)\\\|\_\{\\sigma\}^\{2\}\\\|u\\\|\_\{2\}^\{2\},soAlA\_\{l\}is bounded\. Lemma[1](https://arxiv.org/html/2608.14664#Thmlemma1)shows⟨ml,Al​u⟩ℋl=\(gpop\(l\)\)⊤​u\\langle m\_\{l\},A\_\{l\}u\\rangle\_\{\\mathcal\{H\}\_\{l\}\}=\(g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)^\{\\top\}ufor everyuu, henceAl∗​ml=gpop\(l\)A\_\{l\}^\{\*\}m\_\{l\}=g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\. NowAl∗​ml=0A\_\{l\}^\{\*\}m\_\{l\}=0exactly whenmlm\_\{l\}is orthogonal toRange⁡\(Al\)=𝒯h\(l\)\\operatorname\{Range\}\(A\_\{l\}\)=\\mathcal\{T\}\_\{h\}^\{\(l\)\}, which is equivalent toΠl​ml=0\\Pi\_\{l\}m\_\{l\}=0\. The final equivalence is the definition of𝒱l\\mathcal\{V\}\_\{l\}\. ∎

###### Lemma 3\(Empirical directional derivative and projection\)\.

For every fixed sample satisfying the differentiability conditions in the main paper’s first\-order regularity assumption and everyu∈ℝplu\\in\\mathbb\{R\}^\{p\_\{l\}\},

D​ΦS,l​\(0\)​\[u\]=\(gS\(l\)\)⊤​u=⟨qS,l,AS,l​u⟩S,l\.D\\Phi\_\{S,l\}\(0\)\[u\]=\(g\_\{S\}^\{\(l\)\}\)^\{\\top\}u=\\langle q\_\{S,l\},A\_\{S,l\}u\\rangle\_\{S,l\}\.\(5\)Moreover,

gS\(l\)=0⟺ΠS,l​qS,l=0⟺𝒱S,l=0\.g\_\{S\}^\{\(l\)\}=0\\quad\\Longleftrightarrow\\quad\\Pi\_\{S,l\}q\_\{S,l\}=0\\quad\\Longleftrightarrow\\quad\\mathcal\{V\}\_\{S,l\}=0\.\(6\)

###### Proof\.

The sample\-wise chain rule and finite summation give

D​ΦS,l​\(0\)​\[u\]=1M​∑i=1Mql,i⊤​Jl​\(zl,i\)​u=\(gS\(l\)\)⊤​u\.D\\Phi\_\{S,l\}\(0\)\[u\]=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}q\_\{l,i\}^\{\\top\}J\_\{l\}\(z\_\{l,i\}\)u=\(g\_\{S\}^\{\(l\)\}\)^\{\\top\}u\.The same expression is⟨qS,l,AS,l​u⟩S,l\\langle q\_\{S,l\},A\_\{S,l\}u\\rangle\_\{S,l\}\. The finite\-dimensional orthogonality argument used in Lemma[2](https://arxiv.org/html/2608.14664#Thmlemma2)gives \([6](https://arxiv.org/html/2608.14664#A3.E6)\)\. ∎

## Appendix DProof of the Main\-Paper Necessary\-and\-Sufficient Saturation Theorem

###### Proof\.

Fixl∈ℐl\\in\\mathcal\{I\}and abbreviateg=gpop\(l\)g=g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\. By Lemma[1](https://arxiv.org/html/2608.14664#Thmlemma1), the residual non\-degeneracy condition stated in the main paper is the existence ofvvwithg⊤​v<0g^\{\\top\}v<0\. This impliesg≠0g\\neq 0\. Conversely, ifg≠0g\\neq 0, choosingv=−gv=\-ggivesg⊤​v=−‖g‖22<0g^\{\\top\}v=\-\\\|g\\\|\_\{2\}^\{2\}<0; the interior\-point condition onΘl\\Theta\_\{l\}makes sufficiently small steps feasible\. Thus statements 1 and 2 are equivalent\. Lemma[2](https://arxiv.org/html/2608.14664#Thmlemma2)gives the equivalence of statements 2, 3, and 4\.

Supposeg≠0g\\neq 0\. The fixed zero\-state descent\-compatible update assumption stated in the main paper and Lemma[1](https://arxiv.org/html/2608.14664#Thmlemma1)give

D​Φl​\(0\)​\[dl​\(g\)\]=g⊤​dl​\(g\)<0\.D\\Phi\_\{l\}\(0\)\[d\_\{l\}\(g\)\]=g^\{\\top\}d\_\{l\}\(g\)<0\.Fréchet differentiability therefore yieldsη¯l\>0\\bar\{\\eta\}\_\{l\}\>0such thatΦl​\(η​dl​\(g\)\)<Φl​\(0\)\\Phi\_\{l\}\(\\eta d\_\{l\}\(g\)\)<\\Phi\_\{l\}\(0\)for everyη∈\(0,η¯l\)\\eta\\in\(0,\\bar\{\\eta\}\_\{l\}\)\. Conversely, ifg=0g=0, zero\-state descent compatibility givesdl​\(g\)=0d\_\{l\}\(g\)=0, so strict decrease is impossible\. This proves statement 5\.

Sinceℐ\\mathcal\{I\}is finite,𝒱depth=0\\mathcal\{V\}\_\{\\mathrm\{depth\}\}=0if and only if every𝒱l=0\\mathcal\{V\}\_\{l\}=0\. The local equivalences identify this with depth\-wide first\-order saturation\. Lemma[3](https://arxiv.org/html/2608.14664#Thmlemma3)gives the empirical result by the same argument, with a thresholdη¯S,l\>0\\bar\{\\eta\}\_\{S,l\}\>0\. ∎

###### Corollary 1\(Quantitative descent under local smoothness\)\.

Fix candidatelland suppose ordinary gradient descent is used, sodl​\(g\)=−gd\_\{l\}\(g\)=\-g\. If∇Φl\\nabla\\Phi\_\{l\}isLlL\_\{l\}\-Lipschitz on a neighborhood containing the segment\{−η​gpop\(l\):0≤η≤1/Ll\}\\\{\-\\eta g\_\{\\mathrm\{pop\}\}^\{\(l\)\}:0\\leq\\eta\\leq 1/L\_\{l\}\\\}, then

Φl​\(−η​gpop\(l\)\)≤Φl​\(0\)−η​\(1−Ll​η2\)​‖gpop\(l\)‖22\\Phi\_\{l\}\\\!\\left\(\-\\eta g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\right\)\\leq\\Phi\_\{l\}\(0\)\-\\eta\\left\(1\-\\frac\{L\_\{l\}\\eta\}\{2\}\\right\)\\\|g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\\|\_\{2\}^\{2\}for everyη∈\(0,1/Ll\]\\eta\\in\(0,1/L\_\{l\}\]\. In particular, atη=1/Ll\\eta=1/L\_\{l\}the one\-step decrease is at least‖gpop\(l\)‖22/\(2​Ll\)\\\|g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\\|\_\{2\}^\{2\}/\(2L\_\{l\}\)\. The same statement holds forΦS,l\\Phi\_\{S,l\}with empirical smoothness constantLS,lL\_\{S,l\}\.

###### Proof\.

The standard descent lemma givesΦl​\(−η​g\)≤Φl​\(0\)−η​‖g‖22\+\(Ll​η2/2\)​‖g‖22\\Phi\_\{l\}\(\-\\eta g\)\\leq\\Phi\_\{l\}\(0\)\-\\eta\\\|g\\\|\_\{2\}^\{2\}\+\(L\_\{l\}\\eta^\{2\}/2\)\\\|g\\\|\_\{2\}^\{2\}\. Substituteg=gpop\(l\)g=g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\. The empirical proof is identical\. ∎

### D\.1Closed Forms for the Growth Values

DefineGl:=Al∗​Al=𝔼​\[Jl​\(zl\)⊤​Jl​\(zl\)\]G\_\{l\}:=A\_\{l\}^\{\*\}A\_\{l\}=\\mathbb\{E\}\[J\_\{l\}\(z\_\{l\}\)^\{\\top\}J\_\{l\}\(z\_\{l\}\)\]andG^S,l:=M−1​∑i=1MJl​\(zl,i\)⊤​Jl​\(zl,i\)\\widehat\{G\}\_\{S,l\}:=M^\{\-1\}\\sum\_\{i=1\}^\{M\}J\_\{l\}\(z\_\{l,i\}\)^\{\\top\}J\_\{l\}\(z\_\{l,i\}\)\.

###### Lemma 4\(Gram\-matrix representation\)\.

The growth values satisfy

𝒱l2=\(gpop\(l\)\)⊤​Gl†​gpop\(l\),𝒱S,l2=\(gS\(l\)\)⊤​G^S,l†​gS\(l\)\.\\mathcal\{V\}\_\{l\}^\{2\}=\(g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)^\{\\top\}G\_\{l\}^\{\\dagger\}g\_\{\\mathrm\{pop\}\}^\{\(l\)\},\\qquad\\mathcal\{V\}\_\{S,l\}^\{2\}=\(g\_\{S\}^\{\(l\)\}\)^\{\\top\}\\widehat\{G\}\_\{S,l\}^\{\\dagger\}g\_\{S\}^\{\(l\)\}\.\(7\)

###### Proof\.

Because𝒯h\(l\)=Range⁡\(Al\)\\mathcal\{T\}\_\{h\}^\{\(l\)\}=\\operatorname\{Range\}\(A\_\{l\}\),Πl=Al​Gl†​Al∗\\Pi\_\{l\}=A\_\{l\}G\_\{l\}^\{\\dagger\}A\_\{l\}^\{\*\}\. Hence

𝒱l2=⟨ml,Al​Gl†​Al∗​ml⟩ℋl=\(gpop\(l\)\)⊤​Gl†​gpop\(l\)\.\\mathcal\{V\}\_\{l\}^\{2\}=\\langle m\_\{l\},A\_\{l\}G\_\{l\}^\{\\dagger\}A\_\{l\}^\{\*\}m\_\{l\}\\rangle\_\{\\mathcal\{H\}\_\{l\}\}=\(g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)^\{\\top\}G\_\{l\}^\{\\dagger\}g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\.The empirical formula follows from the same finite\-dimensional projection identity forAS,lA\_\{S,l\}under the empirical inner product\. ∎

## Appendix EJoint Insertion of Several Residual Blocks

Let𝒦=\(l1,…,lK\)\\mathcal\{K\}=\(l\_\{1\},\\ldots,l\_\{K\}\)be an ordered tuple of candidates at distinct insertion locations, listed from upstream to downstream in the fixed feed\-forward network topology\. Write the original network asfold∗=FK∘FK−1∘⋯∘F0f\_\{\\mathrm\{old\}\}^\{\*\}=F\_\{K\}\\circ F\_\{K\-1\}\\circ\\cdots\\circ F\_\{0\}, where the mapsFjF\_\{j\}are the original network segments between consecutive insertion locations\. DefineTlj,θlj​\(w\):=w\+hlj,θlj​\(w\)T\_\{l\_\{j\},\\theta\_\{l\_\{j\}\}\}\(w\):=w\+h\_\{l\_\{j\},\\theta\_\{l\_\{j\}\}\}\(w\)and

f𝒦,θ𝒦:=FK∘TlK,θlK∘FK−1∘⋯∘Tl1,θl1∘F0,f\_\{\\mathcal\{K\},\\theta\_\{\\mathcal\{K\}\}\}:=F\_\{K\}\\circ T\_\{l\_\{K\},\\theta\_\{l\_\{K\}\}\}\\circ F\_\{K\-1\}\\circ\\cdots\\circ T\_\{l\_\{1\},\\theta\_\{l\_\{1\}\}\}\\circ F\_\{0\},\(8\)withK:=\|𝒦\|K:=\|\\mathcal\{K\}\|,θ𝒦:=\(θl1,…,θlK\)\\theta\_\{\\mathcal\{K\}\}:=\(\\theta\_\{l\_\{1\}\},\\ldots,\\theta\_\{l\_\{K\}\}\)in the product space∏j=1Kℝplj\\prod\_\{j=1\}^\{K\}\\mathbb\{R\}^\{p\_\{l\_\{j\}\}\}, andΦ𝒦​\(θ𝒦\):=R​\(f𝒦,θ𝒦\)\\Phi\_\{\\mathcal\{K\}\}\(\\theta\_\{\\mathcal\{K\}\}\):=R\(f\_\{\\mathcal\{K\},\\theta\_\{\\mathcal\{K\}\}\}\)\.

###### Corollary 2\(Joint insertion of independently parameterized blocks\)\.

Assume that the first\-order regularity assumption stated in the main paper for everyl∈𝒦l\\in\\mathcal\{K\}\. Assume additionally that, for𝒟\\mathcal\{D\}\-almost every\(x,y\)\(x,y\), the joint sample loss is Fréchet differentiable with respect toθ𝒦\\theta\_\{\\mathcal\{K\}\}at the joint origin, and thatΦ𝒦\\Phi\_\{\\mathcal\{K\}\}is Fréchet differentiable there with derivative obtained by interchanging differentiation and expectation\. Suppose the active parameter blocks are disjoint and everyhl,0≡0h\_\{l,0\}\\equiv 0globally\.

A*strict joint first\-order descent direction*is a vectoru𝒦u\_\{\\mathcal\{K\}\}satisfyingD​Φ𝒦​\(0\)​\[u𝒦\]<0D\\Phi\_\{\\mathcal\{K\}\}\(0\)\[u\_\{\\mathcal\{K\}\}\]<0\. Such a direction exists if and only if at least one candidate in𝒦\\mathcal\{K\}satisfies the residual non\-degeneracy condition stated in the main paper\.

###### Proof\.

Letιj:ℝplj→∏r=1Kℝplr\\iota\_\{j\}:\\mathbb\{R\}^\{p\_\{l\_\{j\}\}\}\\to\\prod\_\{r=1\}^\{K\}\\mathbb\{R\}^\{p\_\{l\_\{r\}\}\}be the canonical injection into blockjj\. Because everyhl,0≡0h\_\{l,0\}\\equiv 0, each full residual transformation held at its designated origin isTl,0​\(w\)=wT\_\{l,0\}\(w\)=won the full ambient space\. Therefore, for everyθlj\\theta\_\{l\_\{j\}\},

Φ𝒦​\(ιj​θlj\)=Φlj​\(θlj\);\\Phi\_\{\\mathcal\{K\}\}\(\\iota\_\{j\}\\theta\_\{l\_\{j\}\}\)=\\Phi\_\{l\_\{j\}\}\(\\theta\_\{l\_\{j\}\}\);no differentiability of the intermediate network segments is needed for this pointwise identity\.

LetL:=D​Φ𝒦​\(0\)L:=D\\Phi\_\{\\mathcal\{K\}\}\(0\), which is a continuous linear functional by joint Fréchet differentiability\. RestrictingLLto coordinate blockjjand applying Lemma[1](https://arxiv.org/html/2608.14664#Thmlemma1)gives

L​\[ιj​uj\]=D​Φlj​\(0\)​\[uj\]=\(gpop\(lj\)\)⊤​uj\.L\[\\iota\_\{j\}u\_\{j\}\]=D\\Phi\_\{l\_\{j\}\}\(0\)\[u\_\{j\}\]=\(g\_\{\\mathrm\{pop\}\}^\{\(l\_\{j\}\)\}\)^\{\\top\}u\_\{j\}\.Every joint direction decomposes asu𝒦=∑j=1Kιj​uju\_\{\\mathcal\{K\}\}=\\sum\_\{j=1\}^\{K\}\\iota\_\{j\}u\_\{j\}, so linearity yields

D​Φ𝒦​\(0\)​\[u𝒦\]=∑j=1K\(gpop\(lj\)\)⊤​uj\.D\\Phi\_\{\\mathcal\{K\}\}\(0\)\[u\_\{\\mathcal\{K\}\}\]=\\sum\_\{j=1\}^\{K\}\(g\_\{\\mathrm\{pop\}\}^\{\(l\_\{j\}\)\}\)^\{\\top\}u\_\{j\}\.Thus the joint derivative is represented by the concatenated vector\(gpop\(l1\),…,gpop\(lK\)\)\(g\_\{\\mathrm\{pop\}\}^\{\(l\_\{1\}\)\},\\ldots,g\_\{\\mathrm\{pop\}\}^\{\(l\_\{K\}\)\}\)\. It has a negative direction if and only if at least one component is nonzero, which is equivalent to the residual non\-degeneracy condition stated in the main paper for at least one candidate by the main paper’s necessary\-and\-sufficient saturation theorem\. ∎

## Appendix FProof of the Main\-Paper Activation\-Gradient Characterization Theorem

###### Proof\.

Ifml∈𝒯h\(l\)m\_\{l\}\\in\\mathcal\{T\}\_\{h\}^\{\(l\)\}, thenΠl​ml=ml\\Pi\_\{l\}m\_\{l\}=m\_\{l\}, so𝒱l=‖ml‖ℋl\\mathcal\{V\}\_\{l\}=\\\|m\_\{l\}\\\|\_\{\\mathcal\{H\}\_\{l\}\}\. The main paper’s necessary\-and\-sufficient saturation theorem gives the population equivalence\. IfqS,l∈Range⁡\(AS,l\)q\_\{S,l\}\\in\\operatorname\{Range\}\(A\_\{S,l\}\), thenΠS,l​qS,l=qS,l\\Pi\_\{S,l\}q\_\{S,l\}=q\_\{S,l\}, yielding the empirical equivalence\.

For the standard block, every matrix directionΔ​Vl\\Delta V\_\{l\}satisfiesJl​\(z\)​vec⁡\(Δ​Vl\)=Δ​Vl​ψl​\(z\)J\_\{l\}\(z\)\\operatorname\{vec\}\(\\Delta V\_\{l\}\)=\\Delta V\_\{l\}\\psi\_\{l\}\(z\)\. Therefore,

D​Φl​\(0\)​\[vec⁡\(Δ​Vl\)\]\\displaystyle D\\Phi\_\{l\}\(0\)\[\\operatorname\{vec\}\(\\Delta V\_\{l\}\)\]=𝔼​\[ql​\(zl,y\)⊤​Δ​Vl​ψl​\(zl\)\]\\displaystyle=\\mathbb\{E\}\[q\_\{l\}\(z\_\{l\},y\)^\{\\top\}\\Delta V\_\{l\}\\psi\_\{l\}\(z\_\{l\}\)\]=⟨Cpop\(l\),Δ​Vl⟩F\.\\displaystyle=\\langle C\_\{\\mathrm\{pop\}\}^\{\(l\)\},\\Delta V\_\{l\}\\rangle\_\{F\}\.Hencegpop\(l\)=vec⁡\(Cpop\(l\)\)g\_\{\\mathrm\{pop\}\}^\{\(l\)\}=\\operatorname\{vec\}\(C\_\{\\mathrm\{pop\}\}^\{\(l\)\}\)under the fixed vectorization convention, and the population residual non\-degeneracy condition stated in the main paper holds exactly whenCpop\(l\)≠0C\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\neq 0\. The empirical statement follows identically and does not use residual\-signal realizability\.

IfUlU\_\{l\}andVlV\_\{l\}are both treated as active raw parameters, then atVl=0V\_\{l\}=0the derivative with respect toUlU\_\{l\}isVl​DUl​ψl,Ul​\(z\)=0V\_\{l\}D\_\{U\_\{l\}\}\\psi\_\{l,U\_\{l\}\}\(z\)=0\. Hence the full tangent range is identical to the range obtained by fixingUl=U0,lU\_\{l\}=U\_\{0,l\}and varying onlyVlV\_\{l\}; this reduction loses no first\-order directions\.

The sample tangent range in matrix form is\{Δ​Vl​ΨS\(l\):Δ​Vl∈ℝNl×rl\}\\\{\\Delta V\_\{l\}\\Psi\_\{S\}^\{\(l\)\}:\\Delta V\_\{l\}\\in\\mathbb\{R\}^\{N\_\{l\}\\times r\_\{l\}\}\\\}\. Under the tuple–matrix identification,⟨B,C⟩S,l=M−1​⟨B,C⟩F\\langle B,C\\rangle\_\{S,l\}=M^\{\-1\}\\langle B,C\\rangle\_\{F\}; multiplication by a positive scalar does not change orthogonality or the associated projector\. A matrix belongs to this range exactly when each row belongs to the row space ofΨS\(l\)\\Psi\_\{S\}^\{\(l\)\}\. Hence the matrix representation ofΠS,l​qS,l\\Pi\_\{S,l\}q\_\{S,l\}isQS\(l\)​ΠΨ,lQ\_\{S\}^\{\(l\)\}\\Pi\_\{\\Psi,l\}and𝒱S,l=M−1/2​‖QS\(l\)​ΠΨ,l‖F\\mathcal\{V\}\_\{S,l\}=M^\{\-1/2\}\\\|Q\_\{S\}^\{\(l\)\}\\Pi\_\{\\Psi,l\}\\\|\_\{F\}\. Moreover,CS\(l\)=0C\_\{S\}^\{\(l\)\}=0if and only ifQS\(l\)​ΠΨ,l=0Q\_\{S\}^\{\(l\)\}\\Pi\_\{\\Psi,l\}=0\. The stacked tangent operator is surjective exactly when the row space ofΨS\(l\)\\Psi\_\{S\}^\{\(l\)\}is all ofℝM\\mathbb\{R\}^\{M\}, equivalently whenrank⁡\(ΨS\(l\)\)=M\\operatorname\{rank\}\(\\Psi\_\{S\}^\{\(l\)\}\)=M\. ThenΠΨ,l=IM\\Pi\_\{\\Psi,l\}=I\_\{M\}andCS\(l\)=0C\_\{S\}^\{\(l\)\}=0if and only ifQS\(l\)=0Q\_\{S\}^\{\(l\)\}=0\. Applying the local equivalences at every candidate gives the depth\-wide statement\. ∎

## Appendix GSupplementary Finite\-Sample Consistency Result

Under the independent finite\-sample probing assumption stated in the main paper, instantiate all empirical quantities withS=SnprobeS=S\_\{n\}^\{\\mathrm\{probe\}\}andM=nM=n\. Defineξl​\(x,y\):=Jl​\(zl\)⊤​ql​\(zl,y\)\\xi\_\{l\}\(x,y\):=J\_\{l\}\(z\_\{l\}\)^\{\\top\}q\_\{l\}\(z\_\{l\},y\)and

g^n\(l\)\\displaystyle\\widehat\{g\}\_\{n\}^\{\(l\)\}:=1n​∑i=1nξl​\(x~i,y~i\),\\displaystyle:=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\xi\_\{l\}\(\\widetilde\{x\}\_\{i\},\\widetilde\{y\}\_\{i\}\),Γ\\displaystyle\\Gamma:=maxl∈ℐ⁡‖gpop\(l\)‖2,\\displaystyle:=\\max\_\{l\\in\\mathcal\{I\}\}\\\|g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\\|\_\{2\},Γ^n\\displaystyle\\widehat\{\\Gamma\}\_\{n\}:=maxl∈ℐ⁡‖g^n\(l\)‖2\.\\displaystyle:=\\max\_\{l\\in\\mathcal\{I\}\}\\\|\\widehat\{g\}\_\{n\}^\{\(l\)\}\\\|\_\{2\}\.Letσl2:=𝔼​‖ξl−gpop\(l\)‖22\\sigma\_\{l\}^\{2\}:=\\mathbb\{E\}\\\|\\xi\_\{l\}\-g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\\|\_\{2\}^\{2\}and, for a deterministic thresholdτn\>0\\tau\_\{n\}\>0, defineSat^n:=𝟏​\{Γ^n≤τn\}\\widehat\{\\mathrm\{Sat\}\}\_\{n\}:=\\mathbf\{1\}\\\{\\widehat\{\\Gamma\}\_\{n\}\\leq\\tau\_\{n\}\\\}\. By the main paper’s necessary\-and\-sufficient saturation theorem,

Γ=0\\displaystyle\\Gamma=0⟺𝒱depth=0\\displaystyle\\quad\\Longleftrightarrow\\quad\\mathcal\{V\}\_\{\\mathrm\{depth\}\}=0⟺fold∗​is first\-order depth\-saturated\.\\displaystyle\\quad\\Longleftrightarrow\\quad f\_\{\\mathrm\{old\}\}^\{\*\}\\text\{ is first\-order depth\-saturated\}\.\(9\)
###### Theorem 3\(Consistent finite\-sample saturation detection\)\.

Under the independent finite\-sample probing assumption stated in the main paper, for everyt\>0t\>0,

Pr⁡\(maxl∈ℐ⁡‖g^n\(l\)−gpop\(l\)‖2≥t\)≤1n​t2​∑l∈ℐσl2\.\\Pr\\\!\\left\(\\max\_\{l\\in\\mathcal\{I\}\}\\\|\\widehat\{g\}\_\{n\}^\{\(l\)\}\-g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\\|\_\{2\}\\geq t\\right\)\\leq\\frac\{1\}\{nt^\{2\}\}\\sum\_\{l\\in\\mathcal\{I\}\}\\sigma\_\{l\}^\{2\}\.\(10\)Ifτn→0\\tau\_\{n\}\\to 0andn​τn2→∞n\\tau\_\{n\}^\{2\}\\to\\infty, then

Γ=0\\displaystyle\\Gamma=0⟹Pr⁡\(Sat^n=1\)→1,\\displaystyle\\Longrightarrow\\Pr\(\\widehat\{\\mathrm\{Sat\}\}\_\{n\}=1\)\\to 1,Γ\>0\\displaystyle\\Gamma\>0⟹Pr⁡\(Sat^n=0\)→1\.\\displaystyle\\Longrightarrow\\Pr\(\\widehat\{\\mathrm\{Sat\}\}\_\{n\}=0\)\\to 1\.In view of \([9](https://arxiv.org/html/2608.14664#A7.E9)\), this detector is consistent for first\-order depth saturation relative to the fixed protocol\.

###### Proof\.

Independence and centering give𝔼​‖g^n\(l\)−gpop\(l\)‖22=σl2/n\\mathbb\{E\}\\\|\\widehat\{g\}\_\{n\}^\{\(l\)\}\-g\_\{\\mathrm\{pop\}\}^\{\(l\)\}\\\|\_\{2\}^\{2\}=\\sigma\_\{l\}^\{2\}/n\. Markov’s inequality applied to the squared norm and a union bound over the finite setℐ\\mathcal\{I\}prove \([10](https://arxiv.org/html/2608.14664#A7.E10)\)\.

IfΓ=0\\Gamma=0, then

Pr⁡\(Sat^n=0\)=Pr⁡\(Γ^n\>τn\)≤1n​τn2​∑lσl2→0\.\\Pr\(\\widehat\{\\mathrm\{Sat\}\}\_\{n\}=0\)=\\Pr\(\\widehat\{\\Gamma\}\_\{n\}\>\\tau\_\{n\}\)\\leq\\frac\{1\}\{n\\tau\_\{n\}^\{2\}\}\\sum\_\{l\}\\sigma\_\{l\}^\{2\}\\to 0\.IfΓ\>0\\Gamma\>0, choosel∗l\_\{\*\}with‖gpop\(l∗\)‖2=Γ\\\|g\_\{\\mathrm\{pop\}\}^\{\(l\_\{\*\}\)\}\\\|\_\{2\}=\\Gamma\. For all sufficiently largenn,τn<Γ/2\\tau\_\{n\}<\\Gamma/2\. On the eventΓ^n≤τn\\widehat\{\\Gamma\}\_\{n\}\\leq\\tau\_\{n\},‖g^n\(l∗\)−gpop\(l∗\)‖2\>Γ/2\\\|\\widehat\{g\}\_\{n\}^\{\(l\_\{\*\}\)\}\-g\_\{\\mathrm\{pop\}\}^\{\(l\_\{\*\}\)\}\\\|\_\{2\}\>\\Gamma/2, and thereforePr⁡\(Sat^n=1\)≤4​σl∗2/\(n​Γ2\)→0\\Pr\(\\widehat\{\\mathrm\{Sat\}\}\_\{n\}=1\)\\leq 4\\sigma\_\{l\_\{\*\}\}^\{2\}/\(n\\Gamma^\{2\}\)\\to 0\. ∎

## Appendix HMore experiments

On four converged CIFAR\-10 ResNets, we insert a globally zero\-output residual block and fit only its terminal projection to a small negative activation\-gradient target, testing whether the resulting first\-order direction produces an immediate reduction in the empirical task loss\.

### H\.1Local Effect of Activation\-Gradient\-Matched Insertions

We next examine whether an activation\-gradient\-matched residual block produces the immediate task\-loss change predicted by the fixed\-sample first\-order analysis\. We evaluate four converged CIFAR\-10 ResNet checkpoints with reference depths 8, 14, 20, and 34\.

For four converged ResNet configurations on CIFAR\-10, we directly construct one additional residual block by matching its output to a small negative activation\-gradient perturbation\. At the selected candidate locationll, we freeze the reference model and cache the training\-set hidden representationszl,iz\_\{l,i\}together with their activation gradientsql,iq\_\{l,i\}\. Both quantities are detached and treated as a fixed regression dataset\. The inserted block is a standard ResNet block\. Its feature\-producing parametersUlU\_\{l\}are initialized using the standard ResNet initialization and then held fixed, while the terminal output projection is initialized atVl=0V\_\{l\}=0and is the only optimized parameter\. We fitVlV\_\{l\}by gradient descent on

𝒥match\(l\)​\(Vl\):=12​M​∑i=1M‖Vl​ψl​\(zl,i\)\+α​ql,i‖22,\\mathcal\{J\}\_\{\\mathrm\{match\}\}^\{\(l\)\}\(V\_\{l\}\):=\\frac\{1\}\{2M\}\\sum\_\{i=1\}^\{M\}\\left\\\|V\_\{l\}\\psi\_\{l\}\(z\_\{l,i\}\)\+\\alpha q\_\{l,i\}\\right\\\|\_\{2\}^\{2\},where the plus sign reflects the regression target−α​ql,i\-\\alpha q\_\{l,i\}\. The zero initialization makes the insertion globally function\-preserving before fitting, while training onlyVlV\_\{l\}restricts the construction to the first\-order tangent family generated by the fixed residual features\. After the auxiliary objective has converged, the fitted block is inserted and evaluated immediately; no classification\-loss optimization or joint fine\-tuning is performed\. The matching scale, optimization schedule, and candidate\-selection rule are fixed without using held\-out performance\. We report the matching error, the relative magnitude of the resulting hidden\-state perturbation, and the changes in training loss\.

Table 7:Immediate loss changes after inserting one activation\-gradient\-matched residual block\. The fitted model is evaluated without subsequent classification\-loss training\. Positive gain denotes lower loss after insertion\.ReferenceTrain loss\(ave\)Δtrainmatch\\Delta\_\{\\mathrm\{train\}\}^\{\\mathrm\{match\}\}Test loss\(ave\)Δtestmatch\\Delta\_\{\\mathrm\{test\}\}^\{\\mathrm\{match\}\}ResNet\-80\.0427→0\.04200\.0427\\\!\\rightarrow\\\!0\.0420\+0\.0007\+0\.0007±0\.0003\\pm 0\.00030\.3782→0\.37790\.3782\\\!\\rightarrow\\\!0\.3779\+0\.0003\+0\.0003±0\.0009\\pm 0\.0009ResNet\-140\.0392→0\.03880\.0392\\\!\\rightarrow\\\!0\.0388\+0\.0004\+0\.0004±0\.0002\\pm 0\.00020\.2537→0\.25310\.2537\\\!\\rightarrow\\\!0\.2531\+0\.0006\+0\.0006±0\.0011\\pm 0\.0011ResNet\-200\.0251→0\.02490\.0251\\\!\\rightarrow\\\!0\.0249\+0\.0002\+0\.0002±<0\.0001\\pm<0\.00010\.2015→0\.20170\.2015\\\!\\rightarrow\\\!0\.2017−0\.0002\-0\.0002±0\.0005\\pm 0\.0005ResNet\-340\.0011→0\.00090\.0011\\\!\\rightarrow\\\!0\.0009\+0\.0002\+0\.0002±<0\.0001\\pm<0\.00010\.1891→0\.18950\.1891\\\!\\rightarrow\\\!0\.1895−0\.0004\-0\.0004±0\.0005\\pm 0\.0005

#### Analysis\.

Table[7](https://arxiv.org/html/2608.14664#A8.T7)reports the immediate loss changes after inserting one activation\-gradient\-matched residual block, without subsequent task\-loss optimization or joint fine\-tuning\. The insertion reduces the training loss for all four checkpoints, by7×10−47\\times 10^\{\-4\},4×10−44\\times 10^\{\-4\},2×10−42\\times 10^\{\-4\}, and2×10−42\\times 10^\{\-4\}for ResNet\-8, ResNet\-14, ResNet\-20, and ResNet\-34\. Because the target is constructed from activation gradients on the same training sample, this sign consistency is the main behavior predicted by the fixed\-sample first\-order analysis\. The results are consistent with the fitted block capturing a descending component within the tangent family generated by the fixed features\.

The reductions are small, as expected fromα=10−3\\alpha=10^\{\-3\}and from evaluation before task\-loss fine\-tuning\. Thus, the experiment tests an immediate local improvement rather than the gain after fully optimizing the expanded model\. The reduction decreases from7×10−47\\times 10^\{\-4\}for ResNet\-8 to2×10−42\\times 10^\{\-4\}for ResNet\-20 and ResNet\-34, qualitatively consistent with diminishing empirical first\-order value of additional depth\. However, differing baseline losses and one insertion per checkpoint prevent Table[7](https://arxiv.org/html/2608.14664#A8.T7)from establishing a monotone or calibrated depth–gain relationship\. Test\-loss point estimates improve by3×10−43\\times 10^\{\-4\}and6×10−46\\times 10^\{\-4\}for ResNet\-8 and ResNet\-14, but worsen by2×10−42\\times 10^\{\-4\}and4×10−44\\times 10^\{\-4\}for ResNet\-20 and ResNet\-34\. All changes are smaller than or comparable to the reported uncertainty\.