Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

# Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
Source: [https://arxiv.org/html/2608.11479](https://arxiv.org/html/2608.11479)
Aug 7, 2026

###### Abstract

We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset\. We only assume that the activation functions are Lipschitz smooth, Lipschitz continuous, and linearly bounded— properties that hold for linear, tanh, softplus, and sigmoid activation functions\. For the loss function, we require that it is Lipschitz smooth in the model outputs, which is true for mean\-squared error\. The key theoretical insight is that the Lipschitz properties of the activation functions are partially preserved even through repeated compositions, leading to a novel generalized Lipschitz smoothness condition where the change in gradient is upper bounded by the change in the parameter space, multiplied by polynomial terms of the parameter norms at both endpoints\. This type of condition holds for both the model functionandthe loss function, enabling a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms\. By ensuring that the parameter norms do not grow too quickly to infinity, we prove that the minimum squared gradient norm converges to zero inTTiterations at rateO⁡\(1/T1/L\)O\(1/T^\{1/L\}\)for anLL\-layer neural network\.

## 1Introduction

The behavior of gradient descent on the loss functions of neural networks has evaded a complete theoretical understanding for decades\. These loss landscapes are not only highly nonconvex, but more critically, they do not satisfyLipschitz smoothness, where the change in gradient is linearly bounded by the change in parameter space\. Instead, the local Lipschitz constant, which dictates the sharpness of the landscape, can grow arbitrarily large as the parameters approach infinity\. Because Lipschitz smoothness is essential for proving the convergence of gradient algorithms, a fundamental gap has emerged in machine learning research: popular optimization methods are analyzed under the smoothness assumption, yet empirically validated on neural network landscapes that are not theoretically well\-characterized\.

A rich body of research examines the convergence of gradient descent toglobal minimaof neural networks in highly constrained settings — requiring, for example, a sufficiently wide \(overparametrized\) or infinitely wide network\[[13](https://arxiv.org/html/2608.11479#bib.bib13),[19](https://arxiv.org/html/2608.11479#bib.bib18),[2](https://arxiv.org/html/2608.11479#bib.bib2),[17](https://arxiv.org/html/2608.11479#bib.bib17)\], specific architecture or activation functions\[[3](https://arxiv.org/html/2608.11479#bib.bib4)\], balanced or Gaussian initialization scheme, nondegenerate\[[8](https://arxiv.org/html/2608.11479#bib.bib10)\]or orthogonal\[[12](https://arxiv.org/html/2608.11479#bib.bib12)\]dataset\. Under such conditions, the iterates remain within a bounded set around the initialization, effectively satisfyinglocalLipschitz smoothness\. While these analyses provide valuable insights into how gradient descent can achieve unexpectedly strong performance in deep learning, they require many assumptions and often fail to apply to scenarios beyond their constraints\. For example, it has been shown that the infinite\-width regime does not adequately capture the behavior of training finite\-width neural networks\[[4](https://arxiv.org/html/2608.11479#bib.bib3),[35](https://arxiv.org/html/2608.11479#bib.bib35),[33](https://arxiv.org/html/2608.11479#bib.bib33),[28](https://arxiv.org/html/2608.11479#bib.bib34)\]\.

A separate line of research seeks to identify weaker requirements, such asnon\-uniformorgeneralized Lipschitzsmoothness conditions, that still allow gradient descent to converge\. For twice\-differentiable functions, Lipschitz smoothness is equivalent to requiring that the Hessian norm is bounded by a constant\. One relaxation is \(L0L\_\{0\},L1L\_\{1\}\)\-smoothness\[[36](https://arxiv.org/html/2608.11479#bib.bib36)\], which requires that the Hessian norm is upper bounded by an affine function of the gradient norm\. This condition has been further generalized to any nondecreasing function\[[18](https://arxiv.org/html/2608.11479#bib.bib19)\], and other works consider bounding the Hessian with an affine function of the objective value\[[27](https://arxiv.org/html/2608.11479#bib.bib28)\]\. A related concept isrelative smoothness, where the Hessian is upper bounded by the Hessian of some convex reference function, allowing for convergence of gradient descent or mirror descent\[[6](https://arxiv.org/html/2608.11479#bib.bib6),[20](https://arxiv.org/html/2608.11479#bib.bib20)\]\. However, while these conditions can capture more complex settings, they are not generally satisfied by the loss landscapes of deep neural networks\.

In this work, we provide a fresh analytical framework addressing the limitations of the above two approaches\. We identify a generalized Lipschitz smoothness condition, which we calldouble polynomial smoothness, that fully characterizes feedforward neural networks of any width or depth, as long as the activation functions are Lipschitz smooth, Lipschitz continuous, and linearly bounded, and the loss is Lipschitz smooth in the model outputs\. By recursively relating the Lipschitz properties of one layer to the next, we establish that the change in model function gradient is upper bounded by the change in parameter space, multiplied by polynomial terms of the parameter norms at both endpoints\.111In particular, we use “double” to highlight the dependence on the parameter norms atbothendpoints, and also to distinguish from the use of the term “polynomial smoothness” in adjacent research fields such as the analysis of Approximate Message Passing algorithms\[[24](https://arxiv.org/html/2608.11479#bib.bib24)\]\.This property not only correctly describes the structure of the neural network, but it can also be passed along to the loss function, yielding a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms\. The last step is demonstrating that the parameter norms grow sublinearly to infinity, at rateTL−1LT^\{\\frac\{L\-1\}\{L\}\}\. This allows us to prove that the minimum squared gradient norm converges to zero at rateO⁡\(1/T1/L\)O\(1/T^\{1/L\}\)forLL\-layer neural networks\.

Our convergence analysis apply to a broad range of feedforward neural networks, including models with linear, tanh, softplus, and sigmoid activation functions\. The advantage of our approach is that activation functions are relatively easy to mathematically characterize, and they easily satisfy requirements such as Lipschitz smoothness or continuity that would otherwise be too restrictive to impose on the overall loss function\. We require that the loss is Lipschitz smooth in the model outputs, a mild condition satisfied by mean\-squared error\. We have no special requirements on the width, depth, initialization or dataset; we only assume that the dataset is normalized as a matter of convenience, to streamline some algebraic steps\. As we do not require that the iterates remain in a bounded set, our analysis seamlessly accounts for thefeature learningregime of neural networks, where the parameters may progress far from initialization to learn feature representations of the data\[[11](https://arxiv.org/html/2608.11479#bib.bib9),[30](https://arxiv.org/html/2608.11479#bib.bib30)\]\.

Our contributions can be summarized as follows\.

- •We establish thedouble polynomial smoothnessof feedforward neural networks and their loss functions, proving that the change in gradient is upper bounded by the change in parameter space, multiplied by polynomial terms of the parameter norms of both endpoints\.
- •We prove that gradient descent on the loss function ofLL\-layer neural networks converges at rateO⁡\(1/T1/L\)O\(1/T^\{1/L\}\), provided that the step size is small enough with respect to the parameter norm and loss value\.

## 2Related Work

#### Convergence to global minima of neural network loss functions\.

Due to the enormous success of neural networks, a large body of work focuses on identifying circumstances under which gradient descent converges toglobal minimaof neural network loss functions, where the resulting model achieves zero error on the training set\. These analyses typically rely on a patchwork of assumptions on the model dimension, initialization or dataset in order to demonstrate that some gradient dominance condition persists for all points in the gradient descent trajectory \(although not globally\), driving the loss to zero\. Many such results hold in an over\-parametrized regime where the hidden layers are extremely wide relative to the data dimension or dataset size\[[19](https://arxiv.org/html/2608.11479#bib.bib18),[13](https://arxiv.org/html/2608.11479#bib.bib13),[2](https://arxiv.org/html/2608.11479#bib.bib2),[12](https://arxiv.org/html/2608.11479#bib.bib12),[32](https://arxiv.org/html/2608.11479#bib.bib27)\]\. Still others considerinfinite\-widthneural networks\[[10](https://arxiv.org/html/2608.11479#bib.bib8),[5](https://arxiv.org/html/2608.11479#bib.bib5)\]; in particular, one central framework is the neural tangent kernel \(NTK\) analysis, where as the width of the network goes to infinity, the gradient descent training process reduces to a kernel method\. In these analyses, the iterates can be shown to remain in a bounded set close to initialization\. They also often rely on random initialization; for example, the NTK analysis calls on the fact that a neural network with infinite random initializations converges to a Gaussian process by way of the Central Limit Theorem\[[17](https://arxiv.org/html/2608.11479#bib.bib17)\]\. Finally, still other works require restrictions on the dataset; for example the works\[[3](https://arxiv.org/html/2608.11479#bib.bib4),[9](https://arxiv.org/html/2608.11479#bib.bib11)\]consider gradient descent for deep linear neural networks on whitened data, and the work\[[8](https://arxiv.org/html/2608.11479#bib.bib10)\]analyzes a complementary setting where the neural network dimension is arbitrary, but the dimension of the input data is at least the number of data points\. While these works provide important insights into how neural networks can achieve near\-zero training loss under strict assumptions, they leave the broader behavior of gradient descent beyond such assumptions unexplained\.

At the same time, a general analysis of the convergence rate of gradient descent to astationary pointfor neural networks is conspicuously missing\. A well\-known issue regarding this setting is the lack of a global Lipschitz constant or bound on the Hessian norm as the iterates converge towards infinity\[[25](https://arxiv.org/html/2608.11479#bib.bib25)\]\. By contrast, if training remains within a compact set, the Hessian is necessarily bounded for all iterates, and the function is locally Lipschitz smooth\. Accounting for the potentially unbounded behavior of the parameters is not just a technical challenge; it is critical to understanding the “feature learning” regime of neural networks, where the final iterate moves significantly from initialization as the model learns feature representations of the data\. In contrast, it has been shown that the “lazy training” regime where iterates do not move far from initialization does not lead to models that perform well, adding on to the limitations of the aforementioned works\[[11](https://arxiv.org/html/2608.11479#bib.bib9),[30](https://arxiv.org/html/2608.11479#bib.bib30)\]\.

#### Generalized Lipschitz smoothness\.

In the interest of developing more faithful analyses of the neural network setting, a recent line of work has targeted relaxations of the Lipschitz smoothness condition that can still yield convergence of gradient algorithms, sometimes calledgeneralized Lipschitz smoothnessornon\-uniform smoothness\. The work\[[36](https://arxiv.org/html/2608.11479#bib.bib36)\]proposes the\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness condition, where the Hessian norm is bounded by an affine function of the gradient norm, to explain the performance of gradient clipping\. This condition was then generalized to polynomial functions\[[29](https://arxiv.org/html/2608.11479#bib.bib29),[34](https://arxiv.org/html/2608.11479#bib.bib32)\], later further extended to any nondecreasing functions of the gradient norm\[[18](https://arxiv.org/html/2608.11479#bib.bib19)\]\. Additional works have considered bounding the Hessian norm by polynomial functions of the objective value\[[27](https://arxiv.org/html/2608.11479#bib.bib28),[21](https://arxiv.org/html/2608.11479#bib.bib22)\]\. However, these generalized smoothness conditions are typicallyempiricallymotivated and are only satisfied by some special cases of neural networks— for example, two\-layer neural networks where one layer is frozen\[[27](https://arxiv.org/html/2608.11479#bib.bib28)\], two\-layer neural networks with a strictly increasing activation function on linearly separable data\[[26](https://arxiv.org/html/2608.11479#bib.bib26)\], or deep linear neural networks where the weights remain in a “strongly balanced” subspace at all times\[[1](https://arxiv.org/html/2608.11479#bib.bib1)\]\.

A related area of research isrelative smoothness, motivated by mirror descent analysis, where the Hessian norm is upper bounded by the Hessian of some convex reference function that is used to map iterates to and from a dual space\[[6](https://arxiv.org/html/2608.11479#bib.bib6),[20](https://arxiv.org/html/2608.11479#bib.bib20)\]\. In particular, the works\[[20](https://arxiv.org/html/2608.11479#bib.bib20),[14](https://arxiv.org/html/2608.11479#bib.bib14)\]identify that a two\-layer linear neural network satisfies a relative smoothness condition where the loss Hessian is bounded by a quadratic term in the parameter norms\. This condition is closest to the double polynomial smoothness presented in our work\. However, they only address mirror descent using this reference function, not vanilla gradient descent, and they do not consider general neural networks with nonlinear activation functions\.

Finally, some works attempt to circumvent issues with global smoothness by considering separately the local conditions around the gradient descent trajectory\. The work\[[22](https://arxiv.org/html/2608.11479#bib.bib21)\]proposesdirectional smoothness, where the gradient variation depends solely on the local conditions on the optimization path\. However, this requires solving implicit equations to obtain the learning rate, and the existing analysis only applies to convex functions\. The work\[[15](https://arxiv.org/html/2608.11479#bib.bib15)\]introduces the notion of “glocal” smoothness, where the loss function may have a prohibitively largeglobalLipschitz constant but a smallerlocalLipschitz constant\. This enables the analysis of algorithms on more complex landscapes, but still excludes neural networks, which lack a global Lipschitz constant over an unbounded domain\. Finally, the work\[[7](https://arxiv.org/html/2608.11479#bib.bib7)\]considers the broad scenario where one has access to a local first\-order smoothness oracle that bounds the local Lipschitz smoothness constant within a ball of some radiusRR\. However, a key challenge of this framing lies in choosingRRat each step such that it includes the next iterate\. In contrast, the double polynomial smoothness derived in this work is aglobalcondition that holds for any two points regardless of their distance apart\.

#### Matrix factorization and low\-rank fine\-tuning\.

Many prior works have observed the close connection between matrix factorization and neural network training\[[25](https://arxiv.org/html/2608.11479#bib.bib25),[31](https://arxiv.org/html/2608.11479#bib.bib31)\]\. In particular, we build on the convergence proof structure of the Low\-Rank Adaptation \(LoRA\) algorithm\[[23](https://arxiv.org/html/2608.11479#bib.bib23)\], which can be reinterpreted as training a specific two\-layer linear neural network\. However, the general neural network setting is significantly more complex due to the nonlinearity of the activation functions and the recursive relationships generated by the number of layers\.

## 3Setup

Let⊙\\odotdenote the Hadamard product and∥⋅∥F\\lVert\\cdot\\rVert\_\{F\}denote the Frobenius norm, where for the real matrixX∈ℝm×nX\\in\\mathbb\{R\}^\{m\\times n\}, we have∥X∥F=∑i=1m∑j=1nXi​j2\\lVert X\\rVert\_\{F\}=\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}X\_\{ij\}^\{2\}\}\. We also write∥⋅∥\\lVert\\cdot\\rVertfor∥⋅∥F\\lVert\\cdot\\rVert\_\{F\}throughout\. We denote⟨A,B⟩=tr⁡\(AT​B\)\\langle A,B\\rangle=\\operatorname\{tr\}\(A^\{T\}B\)\.

We consider empirical risk minimization on a dataset ofnntraining samples\{xi,yi\}i=1n\\\{x\_\{i\},y\_\{i\}\\\}^\{n\}\_\{i=1\}, wherexi∈ℝdx\_\{i\}\\in\\mathbb\{R\}^\{d\}andyi∈ℝy\_\{i\}\\in\\mathbb\{R\}\. We model the data withff, a feedforward neural network\. Denote byσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}the activation function, and we overload notation such that forX∈ℝn×mX\\in\\mathbb\{R\}^\{n\\times m\},σ⁡\(X\)∈ℝn×m\\sigma\(X\)\\in\\mathbb\{R\}^\{n\\times m\}denotes the element\-wise application of the activation function\. Then anLL\-layer feedforward neural networkff, predicting on data samplexx, is parametrized byLLweight matricesw1,w2,…,wLw\_\{1\},w\_\{2\},\.\.\.,w\_\{L\}as follows

f⁡\(x,w1,w2,…,wL\)=σ⁡\(wL​σ​\(wL−1​σ​\(…​w2​σ​\(w1​x\)\)\)\),f\(x;w\_\{1\},w\_\{2\},\.\.\.,w\_\{L\}\)=\\sigma\(w\_\{L\}\\sigma\(w\_\{L\-1\}\\sigma\(\.\.\.w\_\{2\}\\sigma\(w\_\{1\}x\)\)\)\),whereL≥2L\\geq 2,wℓ∈ℝdℓ×dℓ−1w\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times d\_\{\\ell\-1\}\},d0=dd\_\{0\}=d, anddL=1d\_\{L\}=1\. We also denote the width of the neural network asdm​a​x=max⁡\{d0,…,dL\}d\_\{max\}=\\max\\\{d\_\{0\},\.\.\.,d\_\{L\}\\\}\. We require the following assumptions on the activation functionσ\\sigma\.

###### Assumption 3\.1\.

The activation functionσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}is continuously differentiable and linearly bounded such that for allw∈ℝw\\in\\mathbb\{R\}, we have

\|σ⁡\(w\)\|≤c0\+c1​\|w\|\.\|\\sigma\(w\)\|\\leq c\_\{0\}\+c\_\{1\}\|w\|\.\(1\)Moreover,σ\\sigmais Lipschitz continuous and Lipschitz smooth with nonnegative constantsc2c\_\{2\},c3c\_\{3\}such that for allw1,w2∈ℝw\_\{1\},w\_\{2\}\\in\\mathbb\{R\}, we have

\|σ⁡\(w1\)−σ⁡\(w2\)\|≤c2​\|w1−w2\|,\|\\sigma\(w\_\{1\}\)\-\\sigma\(w\_\{2\}\)\|\\leq c\_\{2\}\|w\_\{1\}\-w\_\{2\}\|,\(2\)\|σ′​\(w1\)−σ′​\(w2\)\|≤c3​\|w1−w2\|\.\|\\sigma^\{\\prime\}\(w\_\{1\}\)\-\\sigma^\{\\prime\}\(w\_\{2\}\)\|\\leq c\_\{3\}\|w\_\{1\}\-w\_\{2\}\|\.\(3\)

Assumption[3\.1](https://arxiv.org/html/2608.11479#S3.Thmtheorem1)also implies that\|σ′​\(w\)\|≤c2\|\\sigma^\{\\prime\}\(w\)\|\\leq c\_\{2\}for allw∈ℝw\\in\\mathbb\{R\}\. For simplicity, we consider neural networks where each layer has the same activation functionσ\\sigma\. However, since we just bound away the activation function using Assumption[3\.1](https://arxiv.org/html/2608.11479#S3.Thmtheorem1), the analysis can also be applied to neural networks with different activation functions at each layer, as long as they all satisfy Assumption[3\.1](https://arxiv.org/html/2608.11479#S3.Thmtheorem1)\.

###### Assumption 3\.2\.

There exists constantcxc\_\{x\}such that fori=1,…,ni=1,\\dots,n, the input dataxix\_\{i\}are uniformly upper bounded where

∥xi∥≤\\displaystyle\\lVert x\_\{i\}\\rVert\\leqcx​d1/2\.\\displaystyle c\_\{x\}d^\{1/2\}\.

###### Assumption 3\.3\.

We havemax⁡\{cx,c0,c1,c2,c3\}≤1\\max\\\{c\_\{x\},c\_\{0\},c\_\{1\},c\_\{2\},c\_\{3\}\\\}\\leq 1\.

We assume the Lipschitz constants andcxc\_\{x\}are less than or equal to 1\. This is solely to simplify the algebra, and is not required to achieve the main conclusion\. We remark that the coefficientsc0,c1,c2,c3c\_\{0\},c\_\{1\},c\_\{2\},c\_\{3\}are all less than or equal to11for standard tanh, sigmoid, linear, and softplus functions\. Moreover, normalization of the dataset is common in practice\.

We construct a block matrixWWwith the weight matricesw1,…,wLw\_\{1\},\.\.\.,w\_\{L\}on the diagonal\. This matrix has dimensionD2×D1D\_\{2\}\\times D\_\{1\}, whereD1=∑ℓ=0L−1dℓD\_\{1\}=\\sum\_\{\\ell=0\}^\{L\-1\}d\_\{\\ell\}andD2=∑ℓ=1LdℓD\_\{2\}=\\sum\_\{\\ell=1\}^\{L\}d\_\{\\ell\}\.

W=\[w1w2⋱wL\],W∈ℝD2×D1\.W=\\begin\{bmatrix\}w\_\{1\}&&&\\\\ &w\_\{2\}&&\\\\ &&\\ddots&\\\\ &&&w\_\{L\}\\end\{bmatrix\},\\quad W\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}\.
For each layerℓ\\ell, define the matriceseℓ​1∈ℝdℓ×D2e\_\{\\ell 1\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times D\_\{2\}\}andeℓ​2∈ℝD1×dℓ−1e\_\{\\ell 2\}\\in\\mathbb\{R\}^\{D\_\{1\}\\times d\_\{\\ell\-1\}\}as follows

eℓ​1=\\displaystyle e\_\{\\ell 1\}=\[0dℓ×d10dℓ×d2…0dℓ×dℓ−1Idℓ×dℓ0dℓ×dℓ\+1…0dℓ×dL\]\\displaystyle\\begin\{bmatrix\}0\_\{d\_\{\\ell\}\\times d\_\{1\}\}&0\_\{d\_\{\\ell\}\\times d\_\{2\}\}&\\ldots&0\_\{d\_\{\\ell\}\\times d\_\{\\ell\-1\}\}&I\_\{d\_\{\\ell\}\\times d\_\{\\ell\}\}&0\_\{d\_\{\\ell\}\\times d\_\{\\ell\+1\}\}&\\ldots&0\_\{d\_\{\\ell\}\\times d\_\{L\}\}\\end\{bmatrix\}eℓ​2=\\displaystyle e\_\{\\ell 2\}=\[0dℓ−1×d00dℓ−1×d1…0dℓ−1×dℓ−2Idℓ−1×dℓ−10dℓ−1×dℓ…0dℓ−1×dL−1\]T,\\displaystyle\\begin\{bmatrix\}0\_\{d\_\{\\ell\-1\}\\times d\_\{0\}\}&0\_\{d\_\{\\ell\-1\}\\times d\_\{1\}\}&\\ldots&0\_\{d\_\{\\ell\-1\}\\times d\_\{\\ell\-2\}\}&I\_\{d\_\{\\ell\-1\}\\times d\_\{\\ell\-1\}\}&0\_\{d\_\{\\ell\-1\}\\times d\_\{\\ell\}\}&\\ldots&0\_\{d\_\{\\ell\-1\}\\times d\_\{L\-1\}\}\\end\{bmatrix\}^\{T\},such that the linear operatorEℓ​\[W\]E\_\{\\ell\}\[W\]defined as follows extracts theℓ\\ellth diagonal block fromWW, recovering the weight matrixwℓw\_\{\\ell\}of the layer,

Eℓ​\[W\]=eℓ​1​W​eℓ​2=wℓ\.E\_\{\\ell\}\[W\]=e\_\{\\ell 1\}We\_\{\\ell 2\}=w\_\{\\ell\}\.\(4\)We also denote the corresponding adjoint operator on a matrixM∈ℝdℓ×dℓ−1M\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times d\_\{\\ell\-1\}\}as follows,

Eℓ∗​\[M\]=eℓ​1T​M​eℓ​2T\.E^\{\*\}\_\{\\ell\}\[M\]=e^\{T\}\_\{\\ell 1\}Me^\{T\}\_\{\\ell 2\}\.\(5\)
We are ready to writeffin a reparametrized formFFas a function ofW∈ℝD2×D1W\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}\. We have

f\(xi;w1,\.\.,wL\)=F\(xi;W\)=σ\(EL\[W\]σ\(EL−1\[W\]σ\(…E2\[W\]σ\(E1\[W\]xi\)\)\)\)\.f\(x\_\{i\};w\_\{1\},\.\.,w\_\{L\}\)=F\(x\_\{i\};W\)=\\sigma\(E\_\{L\}\[W\]\\sigma\(E\_\{L\-1\}\[W\]\\sigma\(\.\.\.E\_\{2\}\[W\]\\sigma\(E\_\{1\}\[W\]x\_\{i\}\)\)\)\)\.\(6\)
Lety^i=F⁡\(xi,W\)\\hat\{y\}\_\{i\}=F\(x\_\{i\};W\)represent the prediction on the sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\)\. We define the individual loss on the output datayiy\_\{i\}asJi:ℝ→ℝJ\_\{i\}:\\mathbb\{R\}\\to\\mathbb\{R\}\. Then the overall loss functionℒ:ℝD2×D1→ℝ\\mathcal\{L\}:\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}\\to\\mathbb\{R\}is given as

ℒ⁡\(W\)=\\displaystyle\\mathcal\{L\}\(W\)=1n​∑i=1nJi​\(y^i\)=1n​∑i=1nJi​\(F⁡\(xi,W\)\)\.\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}\(\\hat\{y\}\_\{i\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}\(F\(x\_\{i\};W\)\)\.\(7\)
###### Assumption 3\.4\.

The individual lossJiJ\_\{i\}is nonnegative andcJc\_\{J\}\-Lipschitz smooth in the model outputs, such that foruu,v∈ℝv\\in\\mathbb\{R\}representing different model outputs, we have

\|Ji′​\(u\)−Ji′​\(v\)\|≤cJ​\|u−v\|\.\\displaystyle\|J\_\{i\}^\{\\prime\}\(u\)\-J\_\{i\}^\{\\prime\}\(v\)\|\\leq c\_\{J\}\|u\-v\|\.For convenience, we assumecJ≥1c\_\{J\}\\geq 1\.

For example, for mean\-squared error,Ji​\(u\)=\(u−yi\)2J\_\{i\}\(u\)=\(u\-y\_\{i\}\)^\{2\}, andJi′​\(u\)=2​\(u−yi\)J\_\{i\}^\{\\prime\}\(u\)=2\(u\-y\_\{i\}\)\. So mean\-squared error satisfies this assumption withcJ=2c\_\{J\}=2\.

By the chain rule, we have the loss gradient is as follows\.

∇ℒ​\(W\)=\\displaystyle\\nabla\\mathcal\{L\}\(W\)=1n​∑i=1nJi′​\(F⁡\(xi,W\)\)⋅∇WF​\(xi,W\),\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\)\)\\cdot\\nabla\_\{W\}F\(x\_\{i\};W\),and we minimizeℒ⁡\(W\)\\mathcal\{L\}\(W\)using the gradient descent algorithm,

Wt\+1=Wt−ηt∇ℒ\(Wt\)\.\\displaystyle W\_\{t\+1\}=W\_\{t\}\-\\eta\_\{t\}\\nabla\\mathcal\{L\}\(W\_\{t\}\)\.\(8\)The evolution of the block diagonal elements ofWWunder gradient descent matches that of the original weight matricesw1,…,wLw\_\{1\},\.\.\.,w\_\{L\}, while the off diagonal elements exhibit zero change because they do not affect the value ofFForℒ\\mathcal\{L\}\.

## 4Analyses

We first aim to show that for anyW1W\_\{1\},W2W\_\{2\}, we can bound∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVertrelative to∥W2−W1∥\\lVert W\_\{2\}\-W\_\{1\}\\rVert\. We relate the change in loss function gradient to the change in model function valueFFand change in model gradient∇WF\\nabla\_\{W\}F, as stated in Lemma[4\.1](https://arxiv.org/html/2608.11479#S4.Thmtheorem1)\.

###### Lemma 4\.1\.

ForW1,W2∈ℝD2×D1W\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}, we have

∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥≤\\displaystyle\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\leq2n​∑i=1n\|F⁡\(xi,W2\)−F⁡\(xi,W1\)\|​∥∇WF​\(xi,W2\)∥\\displaystyle\\frac\{2\}\{n\}\\sum\_\{i=1\}^\{n\}\|F\(x\_\{i\};W\_\{2\}\)\-F\(x\_\{i\};W\_\{1\}\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\\rVert\+2n∑i=1n\|F\(xi;W1\)−yi\|∥∇WF\(xi;W2\)−∇WF\(xi;W1\)∥\.\\displaystyle\+\\frac\{2\}\{n\}\\sum\_\{i=1\}^\{n\}\|F\(x\_\{i\};W\_\{1\}\)\-y\_\{i\}\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert\.

Now we explicitly compute∇WF​\(xi,W\)\\nabla\_\{W\}F\(x\_\{i\};W\)for some input dataxix\_\{i\}\. We recursively define the intermediate variableszℓ​\(W\)∈ℝdℓz\_\{\\ell\}\(W\)\\in\\mathbb\{R\}^\{d\_\{\\ell\}\}andhℓ​\(W\)∈ℝdℓh\_\{\\ell\}\(W\)\\in\\mathbb\{R\}^\{d\_\{\\ell\}\}as follows

h0​\(W\)=xi,h\_\{0\}\(W\)=x\_\{i\},and forℓ=1,…,L\\ell=1,\.\.\.,L,

zℓ​\(W\)=Eℓ​\[W\]​hℓ−1​\(W\),z\_\{\\ell\}\(W\)=E\_\{\\ell\}\[W\]h\_\{\\ell\-1\}\(W\),\(9\)hℓ​\(W\)=σ⁡\(zℓ​\(W\)\)\.h\_\{\\ell\}\(W\)=\\sigma\(z\_\{\\ell\}\(W\)\)\.\(10\)Then we haveF⁡\(W\)=hL​\(W\)=σ⁡\(zL​\(W\)\)F\(W\)=h\_\{L\}\(W\)=\\sigma\(z\_\{L\}\(W\)\)\. We also define the variableδℓ∈ℝdℓ\\delta\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\}as

δL​\(W\)=σ′​\(zL​\(W\)\),δℓ​\(W\)=σ′​\(zℓ​\(W\)\)⊙\(Eℓ\+1​\[W\]T​δℓ\+1​\(W\)\)\.\\delta\_\{L\}\(W\)=\\sigma^\{\\prime\}\(z\_\{L\}\(W\)\),\\qquad\\delta\_\{\\ell\}\(W\)=\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\)\)\\odot\(E\_\{\\ell\+1\}\[W\]^\{T\}\\delta\_\{\\ell\+1\}\(W\)\)\.\(11\)
###### Lemma 4\.2\.

For some input dataxix\_\{i\}, the gradient ofFFwith respect toWWis given as

∇WF​\(xi,W\)=∑ℓ=1LEℓ∗​\[δℓ​\(W\)​hℓ−1​\(W\)T\]\.\\nabla\_\{W\}F\(x\_\{i\};W\)=\\sum\_\{\\ell=1\}^\{L\}E^\{\*\}\_\{\\ell\}\[\\delta\_\{\\ell\}\(W\)h\_\{\\ell\-1\}\(W\)^\{T\}\]\.

We can decompose the change in∇WF​\(xi,W\)\\nabla\_\{W\}F\(x\_\{i\};W\)in terms of changes inδℓ\\delta\_\{\\ell\}andhℓh\_\{\\ell\}, which can be defined recursively in terms of other layers\. Because of the Lipschitz properties ofσ\\sigmafrom Assumption[3\.1](https://arxiv.org/html/2608.11479#S3.Thmtheorem1), these terms can also be bounded by∥W2−W1∥\\lVert W\_\{2\}\-W\_\{1\}\\rVert\. Collecting these terms together allows us to establish thedouble polynomial smoothnessand continuity ofFF, established in the following lemmas\.

###### Lemma 4\.3\.

For allW1W\_\{1\},W2∈ℝD2×D1W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}, andxix\_\{i\}, we have

∥F⁡\(xi,W2\)−F⁡\(xi,W1\)∥≤dm​a​x1/2​∥W2−W1∥​∑k=0L−1∑j=0L−k−1∥W1∥j​∥W2∥k\.\\displaystyle\\lVert F\(x\_\{i\};W\_\{2\}\)\-F\(x\_\{i\};W\_\{1\}\)\\rVert\\leq d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\sum\_\{j=0\}^\{L\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.

###### Lemma 4\.4\.

For allW1W\_\{1\},W2∈ℝD2×D1W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}, andxix\_\{i\}, we have

∥∇WF​\(xi,W2\)−∇WF​\(xi,W1\)∥≤4​dm​a​x​L3​∥W2−W1∥​∑j=02​L−2∥W1∥j​∑k=02​L−j−2∥W2∥k\.\\displaystyle\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert\\leq 4d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{k=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{k\}\.

By plugging Lemmas[4\.3](https://arxiv.org/html/2608.11479#S4.Thmtheorem3)and[4\.4](https://arxiv.org/html/2608.11479#S4.Thmtheorem4)into Lemma[4\.1](https://arxiv.org/html/2608.11479#S4.Thmtheorem1), we have the following bound establishing the double polynomial smoothness ofℒ\\mathcal\{L\}\.

###### Lemma 4\.5\.

For allW1,W2∈ℝD2×D1W\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}, we have

∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥≤8​cJ​dm​a​x​L3​∥W2−W1∥​\(1\+ℒ​\(W1\)1/2\)​∑j=02​L−2∥W1∥j​∑i=02​L−j−2∥W2∥i\.\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\leq 8c\_\{J\}d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\.

We note that Lemma[4\.5](https://arxiv.org/html/2608.11479#S4.Thmtheorem5)also includes a factor of1\+ℒ​\(W1\)1/21\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}in addition to the polynomials of the parameter norms∥W1∥\\lVert W\_\{1\}\\rVert,∥W2∥\\lVert W\_\{2\}\\rVert\. Becauseℒ⁡\(W1\)\\mathcal\{L\}\(W\_\{1\}\)can also be upper bounded in terms of∥W1∥\\lVert W\_\{1\}\\rVert, we also describe this as double polynomial smoothness\. However, leaving it in this form helps to streamline the convergence analysis by allowing us to leverage descent inℒ\\mathcal\{L\}over time\.

Lemma[4\.5](https://arxiv.org/html/2608.11479#S4.Thmtheorem5)directly leads to the following descent lemma\. In particular, after integrating over the segment betweenW1W\_\{1\}andW2W\_\{2\}, the polynomial terms in∥W2∥\\lVert W\_\{2\}\\rVerttranslate to higher order terms of∥W2−W1∥\\lVert W\_\{2\}\-W\_\{1\}\\rVert\. This departs from many other generalized Lipschitz smoothness conditions, which usually only contain a quadratic term of∥W2−W1∥\\lVert W\_\{2\}\-W\_\{1\}\\rVertin the descent lemma\.

###### Lemma 4\.6\.

For allW1W\_\{1\}andW2∈ℝD2×D1W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}, we have

ℒ⁡\(W2\)−ℒ⁡\(W1\)≤\\displaystyle\\mathcal\{L\}\(W\_\{2\}\)\-\\mathcal\{L\}\(W\_\{1\}\)\\leq⟨∇ℒ​\(W1\),W2−W1⟩\\displaystyle\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\),W\_\{2\}\-W\_\{1\}\\rangle\+C\[\(1\+ℒ\(W1\)1/2\)∑i=02​L−2∥W1∥i\]∑k=02​L−21k\+2∥W2−W1∥k\+2,\\displaystyle\+C\\Big\[\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{i\}\\Big\]\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert^\{k\+2\},whereC=22​L\+2​cJ​dm​a​x​L4C=2^\{2L\+2\}c\_\{J\}d\_\{max\}L^\{4\}\.

Applying this lemma to gradient descent \([8](https://arxiv.org/html/2608.11479#S3.E8)\) yields descent in the loss function value, as long asηt\\eta\_\{t\}is chosen appropriately\. The equation forηt\\eta\_\{t\}only requires access to the current parameter norm∥Wt∥\\lVert W\_\{t\}\\rVertand loss valueℒ⁡\(Wt\)\\mathcal\{L\}\(W\_\{t\}\)and is therefore easily computable without any subroutines or implicit solutions\.

###### Lemma 4\.7\.

For one step of gradient descentWt\+1=Wt−ηt∇ℒ\(Wt\)W\_\{t\+1\}=W\_\{t\}\-\\eta\_\{t\}\\nabla\\mathcal\{L\}\(W\_\{t\}\), if the learning rateηt\\eta\_\{t\}satisfies

ηt=1ρ⁡\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i\\eta\_\{t\}=\\frac\{1\}\{\\rho\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}\(12\)whereρ=22​L\+3​cJ​dm​a​x3/2​L5\\rho=2^\{2L\+3\}c\_\{J\}d\_\{max\}^\{3/2\}L^\{5\}, then the function value descends in one step,

ℒ⁡\(Wt\+1\)−ℒ⁡\(Wt\)≤−ηt2​L​∥∇ℒ​\(Wt\)∥2,\\mathcal\{L\}\(W\_\{t\+1\}\)\-\\mathcal\{L\}\(W\_\{t\}\)\\leq\-\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\},and forTTsteps we obtain the bound

∑t=0T−1ηt​∥∇ℒ​\(Wt\)∥2≤2​L​ℒ​\(W0\)\.\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\leq 2L\\mathcal\{L\}\(W\_\{0\}\)\.\(13\)

To show convergence of∥∇ℒ​\(Wt\)∥2\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}to zero, we require that the learning rate \([12](https://arxiv.org/html/2608.11479#S4.E12)\) not decay too quickly\. By Lemma[4\.7](https://arxiv.org/html/2608.11479#S4.Thmtheorem7), we can boundℒ⁡\(Wt\)≤ℒ⁡\(W0\)\\mathcal\{L\}\(W\_\{t\}\)\\leq\\mathcal\{L\}\(W\_\{0\}\)\. It remains to control the growth of∑i=02​L−2∥Wt∥i\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}, which is dominated by∥Wt∥2​L−2\\lVert W\_\{t\}\\rVert^\{2L\-2\}\. We have forN=2​L−2N=2L\-2,

∥WT∥N≤\\displaystyle\\lVert W\_\{T\}\\rVert^\{N\}\\leq∥W0∥N\+N​∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\+∑t=0T−1∑k=2N\(Nk\)​ηtk/2​∥∇ℒ​\(Wt\)∥k\.\\displaystyle\\lVert W\_\{0\}\\rVert^\{N\}\+N\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}\.By \([13](https://arxiv.org/html/2608.11479#S4.E13)\), it can be shown that the last term is bounded by a constant\. However, the growth of∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVertdepends on the value ofNN\. For two\-layer neural networks,L=2L=2andN=2N=2, and we can directly use \([13](https://arxiv.org/html/2608.11479#S4.E13)\) to show that this term grows asO⁡\(1/T\)O\(1/\\sqrt\{T\}\)\. However, for deeper neural networksL\>2L\>2, we have to use the Cauchy\-Schwarz inequality to bound

∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert=∑t=0T−1ηt1/N−1/2​ηt1/2​∥∇ℒ​\(Wt\)∥≤\(∑t=0T−1ηt2−NN\)1/2​\(∑t=0T−1ηt​∥∇ℒ​\(Wt\)∥2\)1/2\.\\displaystyle=\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\-1/2\}\\eta\_\{t\}^\{1/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\\leq\\Big\(\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{\\frac\{2\-N\}\{N\}\}\\Big\)^\{1/2\}\\Big\(\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\Big\)^\{1/2\}\.We show by induction that forN=2​L−2N=2L\-2,\(∑t=0T−1ηt2−NN\)1/2\(\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{\\frac\{2\-N\}\{N\}\}\)^\{1/2\}and therefore∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVertgrows asO⁡\(TL−1L\)O\(T^\{\\frac\{L\-1\}\{L\}\}\), leading to an overallO⁡\(1/T1/L\)O\(1/T^\{1/L\}\)convergence rate for the minimum squared gradient norm\. We state our main result as follows\.

###### Theorem 4\.8\.

Consider the loss functionℒ\\mathcal\{L\}\([7](https://arxiv.org/html/2608.11479#S3.E7)\) of anLL\-layer neural network \([6](https://arxiv.org/html/2608.11479#S3.E6)\)\. Suppose Assumptions[3\.1](https://arxiv.org/html/2608.11479#S3.Thmtheorem1),[3\.2](https://arxiv.org/html/2608.11479#S3.Thmtheorem2),[3\.3](https://arxiv.org/html/2608.11479#S3.Thmtheorem3), and[3\.4](https://arxiv.org/html/2608.11479#S3.Thmtheorem4)hold\. Then afterTTsteps of gradient descent \([8](https://arxiv.org/html/2608.11479#S3.E8)\), with learning rate set as \([12](https://arxiv.org/html/2608.11479#S4.E12)\), we have

mint=0,…,T−1⁡∥∇ℒ​\(Wt\)∥2=O⁡\(1T1/L\),\\min\_\{t=0,\.\.\.,T\-1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}=O\\Big\(\\frac\{1\}\{T^\{1/L\}\}\\Big\),where theO⁡\(⋅\)O\(\\cdot\)notation hides polynomial dependence oncJc\_\{J\},dm​a​xd\_\{max\},ℒ⁡\(W0\)\\mathcal\{L\}\(W\_\{0\}\), and∥W0∥\\lVert W\_\{0\}\\rVert, and exponential dependence onLL\.

## 5Discussion

In this work, we close an important gap in the theory of deep learning by establishing theO⁡\(1/T1/L\)O\(1/T^\{1/L\}\)upper bound on the convergence rate of gradient descent on neural networks with minimal assumptions\. Our results open up a number of questions regarding the behavior of gradient descent on neural networks\. The most natural question is regarding the tightness of the bounds, and obtaining a lower bound on the convergence rate\. Moreover, gradient descent on Lipschitz smooth functions is famouslydimension\-free, but our analysis includes polynomial dependence on the network widthdm​a​xd\_\{max\}and exponential dependence on the number of layersLL\. Determining whether these dependencies can be removed is an important future direction\. Finally, although we significantly relax the assumptions of prior work, this classical framework still heavily relies on some kind of smoothness — in plain words, that shrinking in parameter space continuously maps to shrinking in the gradient\. This leaves the behavior of gradient descent on ReLU neural networks largely undetermined\.

## References

- \[1\]F\. Alimisis, R\. Islamov, and A\. Lucchi\(2026\)Why do we need warm\-up? a theoretical perspective\.External Links:2510\.03164,[Link](https://arxiv.org/abs/2510.03164)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]Z\. Allen\-Zhu, Y\. Li, and Z\. Song\(2019\)A convergence theory for deep learning via over\-parameterization\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 242–252\.External Links:[Link](https://proceedings.mlr.press/v97/allen-zhu19a.html)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]S\. Arora, N\. Cohen, N\. Golowich, and W\. Hu\(2019\)A convergence analysis of gradient descent for deep linear neural networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkMQg3C5K7)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]S\. Arora, S\. S\. Du, W\. Hu, Z\. Li, R\. Salakhutdinov, and R\. Wang\(2019\)On exact computation with an infinitely wide neural net\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1)\.
- \[5\]R\. Barboni, M\. V\. de Hoop, T\. Furuya, and G\. Peyré\(2026\)Training infinitely deep and wide transformers\.External Links:2605\.17660,[Link](https://arxiv.org/abs/2605.17660)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]H\. H\. Bauschke, J\. Bolte, and M\. Teboulle\(2017\)A descent lemma beyond lipschitz gradient continuity: first\-order methods revisited and applications\.Math\. Oper\. Res\.42\(2\),pp\. 330–348\.External Links:ISSN 0364\-765X,[Link](https://doi.org/10.1287/moor.2016.0817),[Document](https://dx.doi.org/10.1287/moor.2016.0817)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p3.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p2.1)\.
- \[7\]A\. S\. Berahas, L\. Roberts, and F\. Roosta\(2024\)Non\-uniform smoothness for gradient descent\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=17ESEjETbP)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p3.1)\.
- \[8\]S\. Chatterjee\(2026\)Convergence of gradient descent for deep neural networks\.External Links:2203\.16462,[Link](https://arxiv.org/abs/2203.16462)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]K\. Chen, D\. Lin, and Z\. Zhang\(2026\)A non\-local convergence analysis of gradient flow for deep linear networks\.IEEE Transactions on Pattern Analysis and Machine Intelligence\(\),pp\. 1–12\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2026.3704447)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]L\. Chizat, M\. Colombo, X\. Fernández\-Real, and A\. Figalli\(2024\)Infinite\-width limit of deep linear neural networks\.Communications on Pure and Applied Mathematics77\(10\),pp\. 3958–4007\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/cpa.22200),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/cpa.22200),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/cpa\.22200Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]L\. Chizat, E\. Oyallon, and F\. Bach\(2019\)On lazy training in differentiable programming\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p5.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p2.1)\.
- \[12\]L\. Dana, L\. Pillaud\-Vivien, and F\. Bach\(2025\)Convergence of the gradient flow for shallow reLU networks on weakly interacting data\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=dkXDyM66ce)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]S\. S\. Du, X\. Zhai, B\. Poczos, and A\. Singh\(2019\)Gradient descent provably optimizes over\-parameterized neural networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=S1eK3i09YQ)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]I\. Fatkhullin and N\. He\(2024\)Taming nonconvex stochastic mirror descent with general Bregman divergence\.InProceedings of The 27th International Conference on Artificial Intelligence and Statistics,S\. Dasgupta, S\. Mandt, and Y\. Li \(Eds\.\),Proceedings of Machine Learning Research, Vol\.238,pp\. 3493–3501\.External Links:[Link](https://proceedings.mlr.press/v238/fatkhullin24a.html)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p2.1)\.
- \[15\]C\. Fox, A\. Mishkin, S\. Vaswani, and M\. Schmidt\(2026\)Glocal smoothness: line search and adaptive sizes can help in theory too\!\.Transactions on Machine Learning Research\.Note:Expert CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=be9PdukwEL)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p3.1)\.
- \[16\]G\. Garrigos and R\. M\. Gower\(2024\)Handbook of convergence theorems for \(stochastic\) gradient methods\.External Links:2301\.11235,[Link](https://arxiv.org/abs/2301.11235)Cited by:[§A\.5](https://arxiv.org/html/2608.11479#A1.SS5.p5.1.1)\.
- \[17\]A\. Jacot, F\. Gabriel, and C\. Hongler\(2018\)Neural tangent kernel: convergence and generalization in neural networks\.InProceedings of the 32nd International Conference on Neural Information Processing Systems,NIPS’18,Red Hook, NY, USA,pp\. 8580–8589\.Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]H\. Li, J\. Qian, Y\. Tian, A\. Rakhlin, and A\. Jadbabaie\(2023\)Convex and non\-convex optimization under generalized smoothness\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8aunGrXdkl)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p3.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[19\]Y\. Li and Y\. Liang\(2018\)Learning overparameterized neural networks via stochastic gradient descent on structured data\.InProceedings of the 32nd International Conference on Neural Information Processing Systems,NIPS’18,Red Hook, NY, USA,pp\. 8168–8177\.Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]H\. Lu, R\. M\. Freund, and Y\. Nesterov\(2018\)Relatively smooth convex optimization by first\-order methods, and applications\.SIAM Journal on Optimization28\(1\),pp\. 333–354\.External Links:[Document](https://dx.doi.org/10.1137/16M1099546),[Link](https://doi.org/10.1137/16M1099546),https://doi\.org/10\.1137/16M1099546Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p3.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p2.1)\.
- \[21\]J\. Mei, Y\. Gao, B\. Dai, C\. Szepesvari, and D\. Schuurmans\(2021\)Leveraging non\-uniformity in first\-order non\-convex optimization\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 7555–7564\.External Links:[Link](https://proceedings.mlr.press/v139/mei21a.html)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]A\. Mishkin, A\. Khaled, Y\. Wang, A\. Defazio, and R\. M\. Gower\(2024\)Directional smoothness and gradient methods: convergence and adaptivity\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 14810–14848\.External Links:[Document](https://dx.doi.org/10.52202/079017-0473),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/1ac83203e88eb6cf6b30642f0239b932-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p3.1)\.
- \[23\]S\. Mu and D\. Klabjan\(2026\)On the convergence rate of lora gradient descent\.External Links:2512\.18248,[Link](https://arxiv.org/abs/2512.18248)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px3.p1.1)\.
- \[24\]D\. Q\. Nguyen and T\. Kim\(2023\)On the stability of approximate message passing with independent measurement ensembles\.External Links:2305\.08754,[Link](https://arxiv.org/abs/2305.08754)Cited by:[footnote 1](https://arxiv.org/html/2608.11479#footnote1)\.
- \[25\]R\. Sun\(2019\)Optimization for deep learning: theory and algorithms\.External Links:1912\.08957,[Link](https://arxiv.org/abs/1912.08957)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]H\. Taheri and C\. Thrampoulidis\(2023\)Fast convergence in learning two\-layer neural networks with separable data\.External Links:2305\.13471,[Link](https://arxiv.org/abs/2305.13471)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]S\. Vaswani, Y\. Sun, and R\. Babanezhad\(2026\)Convergence of steepest descent and adam under non\-uniform smoothness\.External Links:2605\.30648,[Link](https://arxiv.org/abs/2605.30648)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p3.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]N\. Vyas, Y\. Bansal, and P\. Nakkiran\(2023\)Empirical limitations of the NTK for understanding scaling laws in deep learning\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=Y3saBb7mCE)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1)\.
- \[29\]A\. Wilson, L\. Mackey, and A\. Wibisono\(2019\)Accelerating rescaled gradient descent: fast optimization of smooth functions\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/7a2b33c672ce223b2aa5789171ddde2f-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[30\]B\. Woodworth, S\. Gunasekar, J\. D\. Lee, E\. Moroshko, P\. Savarese, I\. Golan, D\. Soudry, and N\. Srebro\(2020\)Kernel and rich regimes in overparametrized models\.InProceedings of Thirty Third Conference on Learning Theory,J\. Abernethy and S\. Agarwal \(Eds\.\),Proceedings of Machine Learning Research, Vol\.125,pp\. 3635–3673\.External Links:[Link](https://proceedings.mlr.press/v125/woodworth20a.html)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p5.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p2.1)\.
- \[31\]N\. Xiong, L\. Ding, and S\. S\. Du\(2024\)How over\-parameterization slows down gradient descent in matrix sensing: the curses of symmetry and initialization\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xGvPKAiOhq)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px3.p1.1)\.
- \[32\]Z\. Xu, H\. Min, S\. Tarmoun, E\. Mallada, and R\. Vidal\(2023\)Linear convergence of gradient descent for finite width over\-parametrized linear networks with general initialization\.InProceedings of The 26th International Conference on Artificial Intelligence and Statistics,F\. Ruiz, J\. Dy, and J\. van de Meent \(Eds\.\),Proceedings of Machine Learning Research, Vol\.206,pp\. 2262–2284\.External Links:[Link](https://proceedings.mlr.press/v206/xu23c.html)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px1.p1.1)\.
- \[33\]G\. Yang and E\. J\. Hu\(2022\)Feature learning in infinite\-width neural networks\.External Links:2011\.14522,[Link](https://arxiv.org/abs/2011.14522)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1)\.
- \[34\]Y\. Yang, E\. E\. Tripp, Y\. Sun, S\. Zou, and Y\. Zhou\(2025\)Adaptive gradient normalization and independent sampling for \(stochastic\) generalized\-smooth optimization\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=KKSQQMlEfw)Cited by:[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.
- \[35\]G\. Yehudai and O\. Shamir\(2019\)On the power and limitations of random features for understanding neural networks\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/5481b2f34a74e427a2818014b8e103b0-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p2.1)\.
- \[36\]J\. Zhang, T\. He, S\. Sra, and A\. Jadbabaie\(2020\)Why gradient clipping accelerates training: a theoretical justification for adaptivity\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=BJgnXpVYwS)Cited by:[§1](https://arxiv.org/html/2608.11479#S1.p3.1),[§2](https://arxiv.org/html/2608.11479#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AProofs

### A\.1Proof of Lemma[4\.1](https://arxiv.org/html/2608.11479#S4.Thmtheorem1)

###### Proof\.

We have∇ℒ​\(W\)=1n​∑i=1nJi′​\(F⁡\(xi,W\)\)⋅∇WF​\(xi,W\)\\nabla\\mathcal\{L\}\(W\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\)\)\\cdot\\nabla\_\{W\}F\(x\_\{i\};W\)\. The change in gradient is as follows,

∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥=\\displaystyle\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert=∥1n​∑i=1nJi′​\(F⁡\(xi,W2\)\)​∇WF​\(xi,W2\)−1n​∑i=1nJi′​\(F⁡\(xi,W1\)\)​∇WF​\(xi,W1\)∥\\displaystyle\\lVert\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{2\}\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert=\\displaystyle=∥1n∑i=1n\[Ji′\(F\(xi;W2\)\)∇WF\(xi;W2\)−Ji′\(F\(xi;W1\)\)∇WF\(xi;W2\)\\displaystyle\\lVert\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\Big\[J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{2\}\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\+Ji′\(F\(xi;W1\)\)∇WF\(xi;W2\)−Ji′\(F\(xi;W1\)\)∇WF\(xi;W1\)\]∥\\displaystyle\+J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\Big\]\\rVert=\\displaystyle=∥1n∑i=1n\[\(Ji′\(F\(xi;W2\)\)−Ji′\(F\(xi;W1\)\)\)∇WF\(xi;W2\)\\displaystyle\\lVert\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\Big\[\(J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{2\}\)\)\-J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\+Ji′\(F\(xi;W1\)\)\(∇WF\(xi;W2\)−∇WF\(xi;W1\)\)\]∥\\displaystyle\+J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\(\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\)\\Big\]\\rVert≤\\displaystyle\\leq∥1n​∑i=1n\(Ji′​\(F⁡\(xi,W2\)\)−Ji′​\(F⁡\(xi,W1\)\)\)​∇WF​\(xi,W2\)∥\\displaystyle\\lVert\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{2\}\)\)\-J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\)\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\\rVert\+∥1n​∑i=1nJi′​\(F⁡\(xi,W1\)\)​\(∇WF​\(xi,W2\)−∇WF​\(xi,W1\)\)∥\\displaystyle\+\\lVert\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\(\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\)\\rVert≤Assumption[3\.4](https://arxiv.org/html/2608.11479#S3.Thmtheorem4)\\displaystyle\\overset\{\\text\{Assumption \\ref\{assump:lipJ\}\}\}\{\\leq\}1n​∑i=1ncJ​\|F⁡\(xi,W2\)−F⁡\(xi,W1\)\|​∥∇WF​\(xi,W2\)∥\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}c\_\{J\}\|F\(x\_\{i\};W\_\{2\}\)\-F\(x\_\{i\};W\_\{1\}\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\\rVert\+1n∑i=1n\|Ji′\(F\(xi;W1\)\)\|∥∇WF\(xi;W2\)−∇WF\(xi;W1\)∥\.\\displaystyle\+\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert\.∎

### A\.2Proof of Lemma[4\.2](https://arxiv.org/html/2608.11479#S4.Thmtheorem2)

###### Proof\.

For an element\-wise functionσ⁡\(X\)\\sigma\(X\), we have

d​σ​\(X\)=σ′​\(X\)⊙d​X\.d\\sigma\(X\)=\\sigma^\{\\prime\}\(X\)\\odot dX\.So we have the following differentials\.

d​hℓ=\\displaystyle dh\_\{\\ell\}=σ′​\(zℓ​\(W\)\)⊙d​zℓ\\displaystyle\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\)\)\\odot dz\_\{\\ell\}d​zℓ=\\displaystyle dz\_\{\\ell\}=d⁡\(Eℓ​\[W\]\)​hℓ−1\+Eℓ​\[W\]​d​hℓ−1\\displaystyle d\(E\_\{\\ell\}\[W\]\)h\_\{\\ell\-1\}\+E\_\{\\ell\}\[W\]dh\_\{\\ell\-1\}=\\displaystyle=Eℓ​\[d​W\]​hℓ−1\+Eℓ​\[W\]​d​hℓ−1\\displaystyle E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\+E\_\{\\ell\}\[W\]dh\_\{\\ell\-1\}d​z1=\\displaystyle dz\_\{1\}=E1​\[d​W\]​xi\\displaystyle E\_\{1\}\[dW\]x\_\{i\}
We haveδL​\(W\)=σ′​\(zL​\(W\)\)\\delta\_\{L\}\(W\)=\\sigma^\{\\prime\}\(z\_\{L\}\(W\)\), andδℓ−1​\(W\)=σ′​\(zℓ−1​\(W\)\)⊙\(Eℓ​\[W\]T​δℓ​\(W\)\)\\delta\_\{\\ell\-1\}\(W\)=\\sigma^\{\\prime\}\(z\_\{\\ell\-1\}\(W\)\)\\odot\(E\_\{\\ell\}\[W\]^\{T\}\\delta\_\{\\ell\}\(W\)\)\. Then

d​F=⟨δL,d​zL⟩,dF=\\langle\\delta\_\{L\},dz\_\{L\}\\rangle,and forℓ=2,\.\.,L\\ell=2,\.\.,L,

⟨δℓ​\(W\),d​zℓ⟩=\\displaystyle\\langle\\delta\_\{\\ell\}\(W\),dz\_\{\\ell\}\\rangle=⟨δℓ​\(W\),Eℓ​\[d​W\]​hℓ−1⟩\+⟨δℓ​\(W\),Eℓ​\[W\]​d​hℓ−1⟩,\\displaystyle\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\\rangle\+\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[W\]dh\_\{\\ell\-1\}\\rangle,=\\displaystyle=⟨δℓ​\(W\),Eℓ​\[d​W\]​hℓ−1⟩\+⟨δℓ​\(W\),Eℓ​\[W\]​\(σ′​\(zℓ−1​\(W\)\)⊙d​zℓ−1\)⟩,\\displaystyle\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\\rangle\+\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[W\]\(\\sigma^\{\\prime\}\(z\_\{\\ell\-1\}\(W\)\)\\odot dz\_\{\\ell\-1\}\)\\rangle,=Lemma[A\.9](https://arxiv.org/html/2608.11479#A1.Thmtheorem9)\\displaystyle\\overset\{\\text\{Lemma \\ref\{hlemma:innerprod\}\}\}\{=\}⟨δℓ​\(W\),Eℓ​\[d​W\]​hℓ−1⟩\+⟨Eℓ​\[W\]T​δℓ​\(W\),σ′​\(zℓ−1​\(W\)\)⊙d​zℓ−1⟩\\displaystyle\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\\rangle\+\\langle E\_\{\\ell\}\[W\]^\{T\}\\delta\_\{\\ell\}\(W\),\\sigma^\{\\prime\}\(z\_\{\\ell\-1\}\(W\)\)\\odot dz\_\{\\ell\-1\}\\rangle=Lemma[A\.10](https://arxiv.org/html/2608.11479#A1.Thmtheorem10)\\displaystyle\\overset\{\\text\{Lemma \\ref\{hlemma:odot\}\}\}\{=\}⟨δℓ​\(W\),Eℓ​\[d​W\]​hℓ−1⟩\+⟨σ′​\(zℓ−1​\(W\)\)⊙\(Eℓ​\[W\]T​δℓ​\(W\)\),d​zℓ−1​\(W\)⟩\\displaystyle\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\\rangle\+\\langle\\sigma^\{\\prime\}\(z\_\{\\ell\-1\}\(W\)\)\\odot\(E\_\{\\ell\}\[W\]^\{T\}\\delta\_\{\\ell\}\(W\)\),dz\_\{\\ell\-1\}\(W\)\\rangle=\\displaystyle=⟨δℓ​\(W\),Eℓ​\[d​W\]​hℓ−1⟩\+⟨δℓ−1​\(W\),d​zℓ−1⟩\.\\displaystyle\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\\rangle\+\\langle\\delta\_\{\\ell\-1\}\(W\),dz\_\{\\ell\-1\}\\rangle\.Forℓ=1\\ell=1we have

⟨δ1​\(W\),d​z1⟩=⟨δ1​\(W\),E1​\[d​W\]​h0​\(W\)⟩\\displaystyle\\langle\\delta\_\{1\}\(W\),dz\_\{1\}\\rangle=\\langle\\delta\_\{1\}\(W\),E\_\{1\}\[dW\]h\_\{0\}\(W\)\\ranglewhich yields the following equation ford​FdF:

d​F=\\displaystyle dF=∑ℓ=1L⟨δℓ​\(W\),Eℓ​\[d​W\]​hℓ−1​\(W\)⟩\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\langle\\delta\_\{\\ell\}\(W\),E\_\{\\ell\}\[dW\]h\_\{\\ell\-1\}\(W\)\\rangle=\\displaystyle=⟨∑ℓ=1LEℓ∗​\[δℓ​\(W\)​hℓ−1​\(W\)T\],d​W⟩\.\\displaystyle\\langle\\sum\_\{\\ell=1\}^\{L\}E^\{\*\}\_\{\\ell\}\[\\delta\_\{\\ell\}\(W\)h\_\{\\ell\-1\}\(W\)^\{T\}\],dW\\rangle\.We therefore have the following formula for the gradient∇WF​\(xi,W\)\\nabla\_\{W\}F\(x\_\{i\};W\)

∇WF​\(xi,W\)=∑ℓ=1LEℓ∗​\[δℓ​\(W\)​hℓ−1​\(W\)T\]\.\\nabla\_\{W\}F\(x\_\{i\};W\)=\\sum\_\{\\ell=1\}^\{L\}E^\{\*\}\_\{\\ell\}\[\\delta\_\{\\ell\}\(W\)h\_\{\\ell\-1\}\(W\)^\{T\}\]\.∎

### A\.3Proof of Lemma[4\.3](https://arxiv.org/html/2608.11479#S4.Thmtheorem3)

###### Proof\.

∥F⁡\(xi,W2\)−F⁡\(xi,W1\)∥=\\displaystyle\\lVert F\(x\_\{i\};W\_\{2\}\)\-F\(x\_\{i\};W\_\{1\}\)\\rVert=∥σ⁡\(zL​\(W2\)\)−σ⁡\(zL​\(W1\)\)∥\\displaystyle\\lVert\\sigma\(z\_\{L\}\(W\_\{2\}\)\)\-\\sigma\(z\_\{L\}\(W\_\{1\}\)\)\\rVert≤\\displaystyle\\leq∥zL​\(W2\)−zL​\(W1\)∥\\displaystyle\\lVert z\_\{L\}\(W\_\{2\}\)\-z\_\{L\}\(W\_\{1\}\)\\rVert≤Lemma[A\.2](https://arxiv.org/html/2608.11479#A1.Thmtheorem2)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:zchange\}\}\}\{\\leq\}dm​a​x1/2​∥W2−W1∥​∑j=0L−1∑k=0L−j−1∥W1∥j​∥W2∥k,\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{L\-1\}\\sum\_\{k=0\}^\{L\-j\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\},=\\displaystyle=dm​a​x1/2​∥W2−W1∥​∑k=0L−1∑j=0L−k−1∥W1∥j​∥W2∥k\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\sum\_\{j=0\}^\{L\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.∎

### A\.4Proof of Lemma[4\.4](https://arxiv.org/html/2608.11479#S4.Thmtheorem4)

###### Proof\.

We decompose the change in∇WF​\(xi,W\)\\nabla\_\{W\}F\(x\_\{i\};W\)in terms of the change inδ\\deltaandhhas follows

∥∇WF​\(xi,W2\)−∇WF​\(xi,W1\)∥≤\\displaystyle\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert\\leq∑ℓ=1L∥Eℓ∗​\[δℓ​\(W2\)​hℓ−1​\(W2\)T−δℓ​\(W1\)​hℓ−1​\(W1\)T\]∥\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\lVert E\_\{\\ell\}^\{\*\}\[\\delta\_\{\\ell\}\(W\_\{2\}\)h\_\{\\ell\-1\}\(W\_\{2\}\)^\{T\}\-\\delta\_\{\\ell\}\(W\_\{1\}\)h\_\{\\ell\-1\}\(W\_\{1\}\)^\{T\}\]\\rVert≤\\displaystyle\\leq∑ℓ=1L∥δℓ​\(W2\)​hℓ−1​\(W2\)T−δℓ​\(W1\)​hℓ−1​\(W1\)T∥\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)h\_\{\\ell\-1\}\(W\_\{2\}\)^\{T\}\-\\delta\_\{\\ell\}\(W\_\{1\}\)h\_\{\\ell\-1\}\(W\_\{1\}\)^\{T\}\\rVert≤\\displaystyle\\leq∑ℓ=1L\[∥δℓ​\(W2\)∥​∥hℓ−1​\(W2\)−hℓ−1​\(W1\)∥\+∥hℓ−1​\(W1\)∥​∥δℓ​\(W2\)−δℓ​\(W1\)∥\]\.\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\[\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\\rVert\\lVert h\_\{\\ell\-1\}\(W\_\{2\}\)\-h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+\\lVert h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert\]\.
We break up the sum as follows because∥h0​\(W2\)−h0​\(W1\)∥=∥xi−xi∥=0\\lVert h\_\{0\}\(W\_\{2\}\)\-h\_\{0\}\(W\_\{1\}\)\\rVert=\\lVert x\_\{i\}\-x\_\{i\}\\rVert=0, and to handle separately the bound on∥δL​\(W2\)−δL​\(W1\)∥\\lVert\\delta\_\{L\}\(W\_\{2\}\)\-\\delta\_\{L\}\(W\_\{1\}\)\\rVert,

∥∇WF​\(xi,W2\)−∇WF​\(xi,W1\)∥≤\\displaystyle\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert\\leq∑ℓ=2L∥δℓ​\(W2\)∥​∥hℓ−1​\(W2\)−hℓ−1​\(W1\)∥\+∑ℓ=1L∥hℓ−1​\(W1\)∥​∥δℓ​\(W2\)−δℓ​\(W1\)∥,\\displaystyle\\sum\_\{\\ell=2\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\\rVert\\lVert h\_\{\\ell\-1\}\(W\_\{2\}\)\-h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+\\sum\_\{\\ell=1\}^\{L\}\\lVert h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert,≤\([2](https://arxiv.org/html/2608.11479#S3.E2)\)\\displaystyle\\overset\{\(\\ref\{eq:l2\}\)\}\{\\leq\}∑ℓ=2L∥δℓ​\(W2\)∥​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥\+∑ℓ=1L∥hℓ−1​\(W1\)∥​∥δℓ​\(W2\)−δℓ​\(W1\)∥\\displaystyle\\sum\_\{\\ell=2\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\\rVert\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+\\sum\_\{\\ell=1\}^\{L\}\\lVert h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert=\\displaystyle=∑ℓ=2L∥δℓ​\(W2\)∥​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥\+∑ℓ=1L−1∥hℓ−1​\(W1\)∥​∥δℓ​\(W2\)−δℓ​\(W1\)∥\\displaystyle\\sum\_\{\\ell=2\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\\rVert\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+\\sum\_\{\\ell=1\}^\{L\-1\}\\lVert h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert\+∥hL−1​\(W1\)∥​∥δL​\(W2\)−δL​\(W1\)∥\.\\displaystyle\+\\lVert h\_\{L\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{L\}\(W\_\{2\}\)\-\\delta\_\{L\}\(W\_\{1\}\)\\rVert\.
We bound these three terms\. For the first term, we have the sum is bounded as follows,

∑ℓ=2L∥δℓ​\(W2\)∥​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥​≤Lemma[A\.5](https://arxiv.org/html/2608.11479#A1.Thmtheorem5)\\displaystyle\\sum\_\{\\ell=2\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\\rVert\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\\overset\{\\text\{Lemma \\ref\{lemma:deltabound\}\}\}\{\\leq\}∑ℓ=2L∥W2∥L−ℓ​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥,\\displaystyle\\sum\_\{\\ell=2\}^\{L\}\\lVert W\_\{2\}\\rVert^\{L\-\\ell\}\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert,≤Lemma[A\.2](https://arxiv.org/html/2608.11479#A1.Thmtheorem2)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:zchange\}\}\}\{\\leq\}∑ℓ=2L∥W2∥L−ℓ​dm​a​x1/2​∥W2−W1∥​∑k=0ℓ−2∑j=0ℓ−k−2∥W1∥j​∥W2∥k,\\displaystyle\\sum\_\{\\ell=2\}^\{L\}\\lVert W\_\{2\}\\rVert^\{L\-\\ell\}d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{\\ell\-2\}\\sum\_\{j=0\}^\{\\ell\-k\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\},=\\displaystyle=dm​a​x1/2​∥W2−W1∥​∑ℓ=2L∑k=0ℓ−2∥W2∥L−ℓ\+k​∑j=0ℓ−k−2∥W1∥j,\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{\\ell=2\}^\{L\}\\sum\_\{k=0\}^\{\\ell\-2\}\\lVert W\_\{2\}\\rVert^\{L\-\\ell\+k\}\\sum\_\{j=0\}^\{\\ell\-k\-2\}\\lVert W\_\{1\}\\rVert^\{j\},=\\displaystyle=dm​a​x1/2​∥W2−W1∥​∑ℓ=2L∑k=L−ℓL−2∥W2∥k​∑j=0L−k−2∥W1∥j,\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{\\ell=2\}^\{L\}\\sum\_\{k=L\-\\ell\}^\{L\-2\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{j=0\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leqdm​a​x1/2​∥W2−W1∥​L​∑k=0L−2∥W2∥k​∑j=0L−k−2∥W1∥j\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert L\\sum\_\{k=0\}^\{L\-2\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{j=0\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\.We can bound the third term as

∥hL−1​\(W1\)∥​∥δL​\(W2\)−δL​\(W1\)∥​≤Lemma[A\.3](https://arxiv.org/html/2608.11479#A1.Thmtheorem3)\\displaystyle\\lVert h\_\{L\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{L\}\(W\_\{2\}\)\-\\delta\_\{L\}\(W\_\{1\}\)\\rVert\\overset\{\\text\{Lemma \\ref\{lemma:hbound\} \}\}\{\\leq\}dm​a​x1/2​\(∑m=0L−1∥W1∥m\)​∥δL​\(W2\)−δL​\(W1\)∥\\displaystyle d\_\{max\}^\{1/2\}\\Big\(\\sum\_\{m=0\}^\{L\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\lVert\\delta\_\{L\}\(W\_\{2\}\)\-\\delta\_\{L\}\(W\_\{1\}\)\\rVert≤Lemma[A\.6](https://arxiv.org/html/2608.11479#A1.Thmtheorem6)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:delta\_change\}\}\}\{\\leq\}dm​a​x​∥W2−W1∥​\(∑m=0L−1∥W1∥m\)​∑k=0L−1∑j=0L−k−1∥W1∥j​∥W2∥k,\\displaystyle d\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Big\(\\sum\_\{m=0\}^\{L\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\sum\_\{k=0\}^\{L\-1\}\\sum\_\{j=0\}^\{L\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\},=\\displaystyle=dm​a​x​∥W2−W1∥​∑k=0L−1∥W2∥k​∑m=0L−1∑j=0L−k−1∥W1∥j\+m,\\displaystyle d\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=0\}^\{L\-1\}\\sum\_\{j=0\}^\{L\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\+m\},=\\displaystyle=dm​a​x​∥W2−W1∥​∑k=0L−1∥W2∥k​∑m=0L−1∑j=mL−k−1\+m∥W1∥j,\\displaystyle d\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=0\}^\{L\-1\}\\sum\_\{j=m\}^\{L\-k\-1\+m\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leqL​dm​a​x​∥W2−W1∥​∑k=0L−1∥W2∥k​∑j=02​L−2−k∥W1∥j\.\\displaystyle Ld\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{j=0\}^\{2L\-2\-k\}\\lVert W\_\{1\}\\rVert^\{j\}\.For the second term, we have the sum is bounded as

∑ℓ=1L−1∥hℓ−1\(W1\)∥∥δℓ\(W2\)−\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\lVert h\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-δℓ\(W1\)∥≤Lemma[A\.3](https://arxiv.org/html/2608.11479#A1.Thmtheorem3)∑ℓ=1L−1dm​a​x1/2\(∑m=0ℓ−1∥W1∥m\)∥δℓ\(W2\)−δℓ\(W1\)∥\\displaystyle\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert\\overset\{\\text\{Lemma \\ref\{lemma:hbound\}\}\}\{\\leq\}\\sum\_\{\\ell=1\}^\{L\-1\}d\_\{max\}^\{1/2\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert≤Lemma[A\.6](https://arxiv.org/html/2608.11479#A1.Thmtheorem6)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:delta\_change\}\}\}\{\\leq\}∑ℓ=1L−1dm​a​x1/2​\(∑m=0ℓ−1∥W1∥m\)​\(dm​a​x1/2​∥W2−W1∥​∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥jCLOSE\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}d\_\{max\}^\{1/2\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\Bigg\(d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\+∑k=0L−ℓ−1∥W2∥k∥W1∥L−k−ℓ−1∥W2−W1∥\)\\displaystyle\+\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Bigg\)=\\displaystyle=dm​a​x​∥W2−W1∥​∑ℓ=1L−1\(∑m=0ℓ−1∥W1∥m\)​\(∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥jCLOSE\\displaystyle d\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{\\ell=1\}^\{L\-1\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\Bigg\(\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\+∑k=0L−ℓ−1∥W2∥k∥W1∥L−k−ℓ−1\)\\displaystyle\+\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}\\Bigg\)=\\displaystyle=dm​a​x∥W2−W1∥\[∑ℓ=1L−1\(∑m=0ℓ−1∥W1∥m\)\(∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i∑j=L−kL−i\+k−ℓ−1∥W1∥j\)\\displaystyle d\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Bigg\[\\sum\_\{\\ell=1\}^\{L\-1\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\Bigg\(\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\Bigg\)\+∑ℓ=1L−1\(∑m=0ℓ−1∥W1∥m\)∑k=0L−ℓ−1∥W2∥k∥W1∥L−k−ℓ−1\]\\displaystyle\+\\sum\_\{\\ell=1\}^\{L\-1\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}\\Bigg\]
We have the quintuple sum can be bounded and simplified as follows

∑ℓ=1L−1\(∑m=0ℓ−1∥W1∥m\)​\(∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥j\)=\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\Bigg\(\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\Bigg\)=∑ℓ=1L−1∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑m=0ℓ−1∑j=L−kL−i\+k−ℓ−1∥W1∥m\+j,\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{m=0\}^\{\\ell\-1\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\+j\},=\\displaystyle=∑ℓ=1L−1∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑m=0ℓ−1∑j=L−k\+mL−i\+k−ℓ−1\+m∥W1∥j,\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{m=0\}^\{\\ell\-1\}\\sum\_\{j=L\-k\+m\}^\{L\-i\+k\-\\ell\-1\+m\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leq∑ℓ=1L−1∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑m=0ℓ−1∑j=L−kL−i\+k−2∥W1∥j,\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{m=0\}^\{\\ell\-1\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-2\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leqL​∑ℓ=1L−1∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−2∥W1∥j,\\displaystyle L\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-2\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leqL​∑ℓ=1L−1∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=02​L−i−2∥W1∥j,\\displaystyle L\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=0\}^\{2L\-i\-2\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leqL2​∑ℓ=1L−1∑i=02​L−1−ℓ∥W2∥i​∑j=02​L−i−2∥W1∥j,\\displaystyle L^\{2\}\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{i=0\}^\{2L\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=0\}^\{2L\-i\-2\}\\lVert W\_\{1\}\\rVert^\{j\},≤\\displaystyle\\leqL3​∑i=02​L−2∥W2∥i​∑j=02​L−i−2∥W1∥j\.\\displaystyle L^\{3\}\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=0\}^\{2L\-i\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\.
And the triple sum can be bounded as

∑ℓ=1L−1\(∑m=0ℓ−1∥W1∥m\)​∑k=0L−ℓ−1∥W2∥k​∥W1∥L−k−ℓ−1=\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\Big\(\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{m\}\\Big\)\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}=∑ℓ=1L−1∑k=0L−ℓ−1∥W2∥k​∑m=0ℓ−1∥W1∥L−k−ℓ−1\+m\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\+m\}=\\displaystyle=∑ℓ=1L−1∑k=0L−ℓ−1∥W2∥k​∑m=L−k−ℓ−1L−k−2∥W1∥m\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=L\-k\-\\ell\-1\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{m\}≤\\displaystyle\\leq∑ℓ=1L−1∑k=0L−ℓ−1∥W2∥k​∑m=0L−k−2∥W1∥m\\displaystyle\\sum\_\{\\ell=1\}^\{L\-1\}\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=0\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{m\}≤\\displaystyle\\leqL​∑k=0L−2∥W2∥k​∑m=0L−k−2∥W1∥m\.\\displaystyle L\\sum\_\{k=0\}^\{L\-2\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=0\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{m\}\.
Putting it all together and using the fact thatL≥1L\\geq 1,dm​a​x≥1d\_\{max\}\\geq 1, results in

∥∇WF\(xi;W2\)\\displaystyle\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)−∇WF\(xi;W1\)∥\\displaystyle\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert≤\\displaystyle\\leqL3​dm​a​x​∥W2−W1∥​\(∑k=0L−2∥W2∥k​∑j=0L−k−2∥W1∥jCLOSE\\displaystyle L^\{3\}d\_\{max\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Bigg\(\\sum\_\{k=0\}^\{L\-2\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{j=0\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\+∑k=0L−1∥W2∥k∑j=02​L−2−k∥W1∥j\+∑i=02​L−2∥W2∥i∑j=02​L−i−2∥W1∥j\+∑k=0L−2∥W2∥k∑m=0L−k−2∥W1∥m\)\\displaystyle\+\\sum\_\{k=0\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{j=0\}^\{2L\-2\-k\}\\lVert W\_\{1\}\\rVert^\{j\}\+\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=0\}^\{2L\-i\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\+\\sum\_\{k=0\}^\{L\-2\}\\lVert W\_\{2\}\\rVert^\{k\}\\sum\_\{m=0\}^\{L\-k\-2\}\\lVert W\_\{1\}\\rVert^\{m\}\\Bigg\)≤\\displaystyle\\leq4​dm​a​x​L3​∥W2−W1∥​∑i=02​L−2∥W2∥i​∑j=02​L−i−2∥W1∥j,\\displaystyle 4d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=0\}^\{2L\-i\-2\}\\lVert W\_\{1\}\\rVert^\{j\},=\\displaystyle=4​dm​a​x​L3​∥W2−W1∥​∑j=02​L−2∥W1∥j​∑i=02​L−j−2∥W2∥i\.\\displaystyle 4d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\.where in the last step we exchange the summations\.

∎

### A\.5Proof of Lemma[4\.5](https://arxiv.org/html/2608.11479#S4.Thmtheorem5)

###### Proof\.

We decompose the change in gradient∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVertin terms of\|F⁡\(xi,W2\)−F⁡\(xi,W1\)\|\|F\(x\_\{i\};W\_\{2\}\)\-F\(x\_\{i\};W\_\{1\}\)\|and∥∇WF​\(xi,W2\)−∇WF​\(xi,W1\)∥\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVertvia Lemma[4\.1](https://arxiv.org/html/2608.11479#S4.Thmtheorem1)\. In order to leverage Lemma[4\.1](https://arxiv.org/html/2608.11479#S4.Thmtheorem1), we require the following bound on the gradient norm\.

∥∇WF​\(xi,W\)∥≤\\displaystyle\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\)\\rVert\\leq∑ℓ=1L∥δℓ​\(W\)​hℓ−1​\(W\)T∥,\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\)h\_\{\\ell\-1\}\(W\)^\{T\}\\rVert,≤\\displaystyle\\leq∑ℓ=1L∥δℓ​\(W\)∥​∥hℓ−1​\(W\)∥,\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\lVert\\delta\_\{\\ell\}\(W\)\\rVert\\lVert h\_\{\\ell\-1\}\(W\)\\rVert,≤Lemmas[A\.5](https://arxiv.org/html/2608.11479#A1.Thmtheorem5)and[A\.3](https://arxiv.org/html/2608.11479#A1.Thmtheorem3)\\displaystyle\\overset\{\\text\{Lemmas \\ref\{lemma:deltabound\} and \\ref\{lemma:hbound\}\}\}\{\\leq\}∑ℓ=1L∥W∥L−ℓ​dm​a​x1/2​∑k=0ℓ−1∥W∥k,\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\lVert W\\rVert^\{L\-\\ell\}d\_\{max\}^\{1/2\}\\sum\_\{k=0\}^\{\\ell\-1\}\\lVert W\\rVert^\{k\},=\\displaystyle=dm​a​x1/2​∑ℓ=1L∑k=0ℓ−1∥W∥L−ℓ\+k,\\displaystyle d\_\{max\}^\{1/2\}\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{k=0\}^\{\\ell\-1\}\\lVert W\\rVert^\{L\-\\ell\+k\},=\\displaystyle=dm​a​x1/2​∑ℓ=1L∑k=L−ℓL−1∥W∥k\.\\displaystyle d\_\{max\}^\{1/2\}\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{k=L\-\\ell\}^\{L\-1\}\\lVert W\\rVert^\{k\}\.So we have the bound

∥∇WF​\(xi,W\)∥≤dm​a​x1/2​∑ℓ=1L∑k=0L−1∥W∥k=dm​a​x1/2​L​∑k=0L−1∥W∥k\.\\displaystyle\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\)\\rVert\\leq d\_\{max\}^\{1/2\}\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{k=0\}^\{L\-1\}\\lVert W\\rVert^\{k\}=d\_\{max\}^\{1/2\}L\\sum\_\{k=0\}^\{L\-1\}\\lVert W\\rVert^\{k\}\.\(14\)
Then we can use Lemmas[4\.3](https://arxiv.org/html/2608.11479#S4.Thmtheorem3)and[4\.4](https://arxiv.org/html/2608.11479#S4.Thmtheorem4)to achieve an upper bound in terms of∥W2−W1∥\\lVert W\_\{2\}\-W\_\{1\}\\rVert,∥W1∥\\lVert W\_\{1\}\\rVert, and∥W2∥\\lVert W\_\{2\}\\rVert\.

∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥​≤Lemma[4\.1](https://arxiv.org/html/2608.11479#S4.Thmtheorem1)\\displaystyle\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\overset\{\\text\{Lemma \\ref\{lemma:LLsmooth\}\}\}\{\\leq\}1n​∑i=1ncJ​\|F⁡\(xi,W2\)−F⁡\(xi,W1\)\|​∥∇WF​\(xi,W2\)∥\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}c\_\{J\}\|F\(x\_\{i\};W\_\{2\}\)\-F\(x\_\{i\};W\_\{1\}\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\\rVert\+1n∑i=1n\|Ji′\(F\(xi;W1\)\)\|∥∇WF\(xi;W2\)−∇WF\(xi;W1\)∥\\displaystyle\+\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert≤Lemma[4\.3](https://arxiv.org/html/2608.11479#S4.Thmtheorem3)and \([14](https://arxiv.org/html/2608.11479#A1.E14)\)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:lip\_f\} and \(\\ref\{eq:gradfbound\}\)\}\}\{\\leq\}cJdm​a​x1/2∥W2−W1∥∑j=0L−1∑k=0L−j−1∥W1∥j∥W2∥k⋅dm​a​x1/2L∑i=0L−1∥W2∥i\\displaystyle c\_\{J\}d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{L\-1\}\\sum\_\{k=0\}^\{L\-j\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\\cdot d\_\{max\}^\{1/2\}L\\sum\_\{i=0\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{i\}\+1n∑i=1n\|Ji′\(F\(xi;W1\)\)\|∥∇WF\(xi;W2\)−∇WF\(xi;W1\)∥\\displaystyle\+\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert=\\displaystyle=cJ​dm​a​x​L​∥W2−W1∥​∑j=0L−1∥W1∥j​∑k=0L−j−1∑i=kL−1\+k∥W2∥i\\displaystyle c\_\{J\}d\_\{max\}L\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{L\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{k=0\}^\{L\-j\-1\}\\sum\_\{i=k\}^\{L\-1\+k\}\\lVert W\_\{2\}\\rVert^\{i\}\+1n∑i=1n\|Ji′\(F\(xi;W1\)\)\|∥∇WF\(xi;W2\)−∇WF\(xi;W1\)∥\\displaystyle\+\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\|\\lVert\\nabla\_\{W\}F\(x\_\{i\};W\_\{2\}\)\-\\nabla\_\{W\}F\(x\_\{i\};W\_\{1\}\)\\rVert≤Lemma[4\.4](https://arxiv.org/html/2608.11479#S4.Thmtheorem4)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:lip\_nablaf\}\}\}\{\\leq\}cJ​dm​a​x​L​∥W2−W1∥​∑j=0L−1∥W1∥j​∑k=0L−j−1∑i=02​L−2−j∥W2∥i\\displaystyle c\_\{J\}d\_\{max\}L\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{L\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{k=0\}^\{L\-j\-1\}\\sum\_\{i=0\}^\{2L\-2\-j\}\\lVert W\_\{2\}\\rVert^\{i\}\+4dm​a​xL3∥W2−W1∥∑j=02​L−2∥W1∥j∑i=02​L−j−2∥W2∥i⋅1n∑i=1n\|Ji′\(F\(xi;W1\)\)\|\.\\displaystyle\+4d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\\cdot\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\_\{1\}\)\)\|\.
From Lipschitz smoothness ofJJ, we have for allWW\(See Lemma 2\.28 in\[[16](https://arxiv.org/html/2608.11479#bib.bib16)\]\),

\|J′​\(F⁡\(xi,W\)\)\|2≤2​cJ​Ji​\(F⁡\(xi,W\)\),\|J^\{\\prime\}\(F\(x\_\{i\};W\)\)\|^\{2\}\\leq 2c\_\{J\}J\_\{i\}\(F\(x\_\{i\};W\)\),\(15\)1n​∑i=1n\|Ji′​\(F⁡\(xi,W\)\)\|≤\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\)\)\|\\leq1n​∑i=1n\(2​cJ​Ji​\(F⁡\(xi,W\)\)\)1/2\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(2c\_\{J\}J\_\{i\}\(F\(x\_\{i\};W\)\)\)^\{1/2\}≤\\displaystyle\\leq\(1n​∑i=1n2​cJ​Ji​\(F⁡\(xi,W\)\)\)1/2\\displaystyle\(\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}2c\_\{J\}J\_\{i\}\(F\(x\_\{i\};W\)\)\)^\{1/2\}≤\\displaystyle\\leq\(2​cJ​ℒ​\(W\)\)1/2\.\\displaystyle\(2c\_\{J\}\\mathcal\{L\}\(W\)\)^\{1/2\}\.So we have

1n​∑i=1n\|Ji′​\(F⁡\(xi,W\)\)\|≤\(2​cJ​ℒ​\(W\)\)1/2\.\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|J\_\{i\}^\{\\prime\}\(F\(x\_\{i\};W\)\)\|\\leq\(2c\_\{J\}\\mathcal\{L\}\(W\)\)^\{1/2\}\.\(16\)
∥∇ℒ​\(W2\)−∇ℒ​\(W1\)∥​≤\([16](https://arxiv.org/html/2608.11479#A1.E16)\)\\displaystyle\\lVert\\nabla\\mathcal\{L\}\(W\_\{2\}\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\overset\{\(\\ref\{eq:boundloss\}\)\}\{\\leq\}2​dm​a​x​L2​∥W2−W1∥​∑j=0L−1∥W1∥j​∑i=02​L−2−j∥W2∥i\\displaystyle 2d\_\{max\}L^\{2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{L\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-2\-j\}\\lVert W\_\{2\}\\rVert^\{i\}\+4\(2cJℒ\(W1\)\)1/2dm​a​xL3∥W2−W1∥∑j=02​L−2∥W1∥j∑i=02​L−j−2∥W2∥i,\\displaystyle\+4\(2c\_\{J\}\\mathcal\{L\}\(W\_\{1\}\)\)^\{1/2\}d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{i\},≤\\displaystyle\\leq8​cJ​dm​a​x​L3​∥W2−W1∥​\[∑j=0L−1∥W1∥j​∑i=02​L−2−j∥W2∥i\+ℒ​\(W1\)1/2​∑j=02​L−2∥W1∥j​∑i=02​L−j−2∥W2∥i\]\\displaystyle 8c\_\{J\}d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Bigg\[\\sum\_\{j=0\}^\{L\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-2\-j\}\\lVert W\_\{2\}\\rVert^\{i\}\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\\Bigg\]≤\\displaystyle\\leq8​cJ​dm​a​x​L3​∥W2−W1∥​\(1\+ℒ​\(W1\)1/2\)​∑j=02​L−2∥W1∥j​∑i=02​L−j−2∥W2∥i\.\\displaystyle 8c\_\{J\}d\_\{max\}L^\{3\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{2\}\\rVert^\{i\}\.∎

### A\.6Proof of Lemma[4\.6](https://arxiv.org/html/2608.11479#S4.Thmtheorem6)

###### Proof\.

We denotev=W2−W1v=W\_\{2\}\-W\_\{1\}, and parametrizeℒ\\mathcal\{L\}betweenW1W\_\{1\}andW2W\_\{2\}such that fort∈\[0,1\]t\\in\[0,1\], we have

ϕ⁡\(t\)=\\displaystyle\\phi\(t\)=ℒ⁡\(W1\+t​v\),\\displaystyle\\mathcal\{L\}\(W\_\{1\}\+tv\),ϕ′​\(t\)=\\displaystyle\\phi^\{\\prime\}\(t\)=⟨∇ℒ​\(W1\+t​v\),v⟩F\.\\displaystyle\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\+tv\),v\\rangle\_\{F\}\.
By the fundamental theorem of calculus, we have

ℒ⁡\(W2\)−ℒ⁡\(W1\)\\displaystyle\\mathcal\{L\}\(W\_\{2\}\)\-\\mathcal\{L\}\(W\_\{1\}\)=ϕ⁡\(1\)−ϕ⁡\(0\)=∫01ϕ′​\(t\)​𝑑t,\\displaystyle=\\phi\(1\)\-\\phi\(0\)=\\int\_\{0\}^\{1\}\\phi^\{\\prime\}\(t\)dt,=∫01⟨∇ℒ​\(W1\),v⟩F​𝑑t\+∫01⟨∇ℒ​\(W1\+t​v\)−∇ℒ​\(W1\),v⟩F​𝑑t,\\displaystyle=\\int\_\{0\}^\{1\}\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\),v\\rangle\_\{F\}\\;dt\+\\int\_\{0\}^\{1\}\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\+tv\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\),v\\rangle\_\{F\}\\;dt,=⟨∇ℒ​\(W1\),v⟩F\+∫01⟨∇ℒ​\(W1\+t​v\)−∇ℒ​\(W1\),v⟩F​𝑑t,\\displaystyle=\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\),v\\rangle\_\{F\}\+\\int\_\{0\}^\{1\}\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\+tv\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\),v\\rangle\_\{F\}\\;dt,≤⟨∇ℒ​\(W1\),v⟩\+∫01∥∇ℒ​\(W1\+t​v\)−∇ℒ​\(W1\)∥​∥v∥​𝑑t,\\displaystyle\\leq\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\),v\\rangle\+\\int\_\{0\}^\{1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{1\}\+tv\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\lVert v\\rVert dt,where in the last step we use the Cauchy\-Schwarz inequality and the fact that integrals preserve inequalities\. For the integral, we have by Lemma[4\.5](https://arxiv.org/html/2608.11479#S4.Thmtheorem5),

∫01∥∇ℒ​\(W1\+t​v\)−∇ℒ​\(W1\)∥​∥v∥​𝑑t≤\\displaystyle\\int\_\{0\}^\{1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{1\}\+tv\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\lVert v\\rVert dt\\leq8​cJ​dm​a​x​L3​\(1\+ℒ​\(W1\)1/2\)​∥v∥2​∑j=02​L−2∥W1∥j​∫01t​∑i=02​L−j−2∥W1\+t​v∥i​𝑑t,\\displaystyle 8c\_\{J\}d\_\{max\}L^\{3\}\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\lVert v\\rVert^\{2\}\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\int\_\{0\}^\{1\}t\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{1\}\+tv\\rVert^\{i\}dt,where we pull out any terms not dependent onttfrom the integral\. For any nonnegative integerNN, we have by the binomial expansion and reindexing

∫01t​∑i=0N∥W1\+t​v∥i​𝑑t≤\\displaystyle\\int\_\{0\}^\{1\}t\\sum\_\{i=0\}^\{N\}\\lVert W\_\{1\}\+tv\\rVert^\{i\}\\,dt\\leq∫01t​∑i=0N\(∥W1∥\+∥t​v∥\)i​𝑑t\\displaystyle\\int\_\{0\}^\{1\}t\\sum\_\{i=0\}^\{N\}\(\\lVert W\_\{1\}\\rVert\+\\lVert tv\\rVert\)^\{i\}\\,dt=\\displaystyle=∫01t​∑i=0N∑k=0i\(ik\)​∥W1∥i−k​tk​∥v∥k​𝑑t\\displaystyle\\int\_\{0\}^\{1\}t\\sum\_\{i=0\}^\{N\}\\sum\_\{k=0\}^\{i\}\\binom\{i\}\{k\}\\lVert W\_\{1\}\\rVert^\{i\-k\}t^\{k\}\\lVert v\\rVert^\{k\}\\,dt=\\displaystyle=∫01t​∑k=0N∑i=kN\(ik\)​∥W1∥i−k​tk​∥v∥k​𝑑t\\displaystyle\\int\_\{0\}^\{1\}t\\sum\_\{k=0\}^\{N\}\\sum\_\{i=k\}^\{N\}\\binom\{i\}\{k\}\\lVert W\_\{1\}\\rVert^\{i\-k\}t^\{k\}\\lVert v\\rVert^\{k\}\\,dt=\\displaystyle=∑k=0N∑i=kN\(ik\)​∥W1∥i−k​∥v∥k​∫01tk\+1​𝑑t\\displaystyle\\sum\_\{k=0\}^\{N\}\\sum\_\{i=k\}^\{N\}\\binom\{i\}\{k\}\\lVert W\_\{1\}\\rVert^\{i\-k\}\\lVert v\\rVert^\{k\}\\int\_\{0\}^\{1\}t^\{k\+1\}\\,dt=\\displaystyle=∑k=0N∑i=kN\(ik\)​∥W1∥i−k​∥v∥k​1k\+2\\displaystyle\\sum\_\{k=0\}^\{N\}\\sum\_\{i=k\}^\{N\}\\binom\{i\}\{k\}\\lVert W\_\{1\}\\rVert^\{i\-k\}\\lVert v\\rVert^\{k\}\\frac\{1\}\{k\+2\}≤\\displaystyle\\leq2N​∑k=0N1k\+2​∥v∥k​∑i=kN∥W1∥i−k=2N​∑k=0N1k\+2​∥v∥k​∑i=0N−k∥W1∥i,\\displaystyle 2^\{N\}\\sum\_\{k=0\}^\{N\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}\\sum\_\{i=k\}^\{N\}\\lVert W\_\{1\}\\rVert^\{i\-k\}=2^\{N\}\\sum\_\{k=0\}^\{N\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}\\sum\_\{i=0\}^\{N\-k\}\\lVert W\_\{1\}\\rVert^\{i\},where in the last inequality we use the fact that\(ik\)≤2i≤2N\\binom\{i\}\{k\}\\leq 2^\{i\}\\leq 2^\{N\}\. Therefore we have

∑j=02​L−2∥W1∥j​∫01t​∑i=02​L−j−2∥W1\+t​v∥i​𝑑t≤\\displaystyle\\sum\_\{j=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{j\}\\int\_\{0\}^\{1\}t\\sum\_\{i=0\}^\{2L\-j\-2\}\\lVert W\_\{1\}\+tv\\rVert^\{i\}dt\\leq∑j=02​L−222​L−j−2​∑k=02​L−j−21k\+2​∥v∥k​∑i=02​L−j−2−k∥W1∥i\+j\\displaystyle\\sum\_\{j=0\}^\{2L\-2\}2^\{2L\-j\-2\}\\sum\_\{k=0\}^\{2L\-j\-2\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}\\sum\_\{i=0\}^\{2L\-j\-2\-k\}\\lVert W\_\{1\}\\rVert^\{i\+j\}=\\displaystyle=∑j=02​L−222​L−j−2​∑k=02​L−j−21k\+2​∥v∥k​∑i=j2​L−2−k∥W1∥i\\displaystyle\\sum\_\{j=0\}^\{2L\-2\}2^\{2L\-j\-2\}\\sum\_\{k=0\}^\{2L\-j\-2\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}\\sum\_\{i=j\}^\{2L\-2\-k\}\\lVert W\_\{1\}\\rVert^\{i\}≤\\displaystyle\\leq22​L−2​∑j=02​L−2∑i=j2​L−2∥W1∥i​∑k=02​L−j−21k\+2​∥v∥k\\displaystyle 2^\{2L\-2\}\\sum\_\{j=0\}^\{2L\-2\}\\sum\_\{i=j\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{i\}\\sum\_\{k=0\}^\{2L\-j\-2\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}≤\\displaystyle\\leq22​L−2​\(2​L−1\)​∑i=02​L−2∥W1∥i​∑k=02​L−21k\+2​∥v∥k\.\\displaystyle 2^\{2L\-2\}\(2L\-1\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{i\}\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}\.
and

∫01∥∇ℒ\(W1\\displaystyle\\int\_\{0\}^\{1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{1\}\+tv\)−∇ℒ\(W1\)∥∥v∥dt\\displaystyle\+tv\)\-\\nabla\\mathcal\{L\}\(W\_\{1\}\)\\rVert\\lVert v\\rVert dt≤\\displaystyle\\leq8cJdm​a​xL3\(1\+ℒ\(W1\)1/2\)∥v∥2⋅22​L−2\(2L−1\)∑i=02​L−2∥W1∥i∑k=02​L−21k\+2∥v∥k\\displaystyle 8c\_\{J\}d\_\{max\}L^\{3\}\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\lVert v\\rVert^\{2\}\\cdot 2^\{2L\-2\}\(2L\-1\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{i\}\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\}≤\\displaystyle\\leq22​L\+2​cJ​dm​a​x​L4​\(1\+ℒ​\(W1\)1/2\)​∑i=02​L−2∥W1∥i​∑k=02​L−21k\+2​∥v∥k\+2\\displaystyle 2^\{2L\+2\}c\_\{J\}d\_\{max\}L^\{4\}\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{i\}\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert v\\rVert^\{k\+2\}LetC=22​L\+2​cJ​dm​a​x​L4C=2^\{2L\+2\}c\_\{J\}d\_\{max\}L^\{4\}\. Then putting it all together gives

ℒ⁡\(W2\)−ℒ⁡\(W1\)≤\\displaystyle\\mathcal\{L\}\(W\_\{2\}\)\-\\mathcal\{L\}\(W\_\{1\}\)\\leq⟨∇ℒ​\(W1\),W2−W1⟩\\displaystyle\\langle\\nabla\\mathcal\{L\}\(W\_\{1\}\),W\_\{2\}\-W\_\{1\}\\rangle\+C\[\(1\+ℒ\(W1\)1/2\)∑i=02​L−2∥W1∥i\]∑k=02​L−21k\+2∥W2−W1∥k\+2\.\\displaystyle\+C\\Big\[\(1\+\\mathcal\{L\}\(W\_\{1\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{1\}\\rVert^\{i\}\\Big\]\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert^\{k\+2\}\.∎

### A\.7Proof of Lemma[4\.7](https://arxiv.org/html/2608.11479#S4.Thmtheorem7)

Applying the descent lemma toWt\+1W\_\{t\+1\}andWtW\_\{t\}results in sums of powers ofηt\\eta\_\{t\}\. In the following, we setηt\\eta\_\{t\}small enough by considering the coefficients of theηt2\\eta\_\{t\}^\{2\}terms\. We then demonstrate that the terms corresponding to higher powers ofηt\\eta\_\{t\}are also controlled to be small, resulting in descent in function value\.

ForWt\+1=Wt−ηt∇ℒ\(Wt\)W\_\{t\+1\}=W\_\{t\}\-\\eta\_\{t\}\\nabla\\mathcal\{L\}\(W\_\{t\}\), we have by Lemma[4\.6](https://arxiv.org/html/2608.11479#S4.Thmtheorem6)

ℒ⁡\(Wt\+1\)−ℒ⁡\(Wt\)≤\\displaystyle\\mathcal\{L\}\(W\_\{t\+1\}\)\-\\mathcal\{L\}\(W\_\{t\}\)\\leq⟨∇ℒ​\(Wt\),Wt\+1−Wt⟩\+C⁡\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i​∑k=02​L−21k\+2​∥Wt\+1−Wt∥k\+2\\displaystyle\\langle\\nabla\\mathcal\{L\}\(W\_\{t\}\),W\_\{t\+1\}\-W\_\{t\}\\rangle\+C\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert W\_\{t\+1\}\-W\_\{t\}\\rVert^\{k\+2\}=\\displaystyle=−ηt​∥∇ℒ​\(Wt\)∥2\+C⁡\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i​∑k=02​L−21k\+2​∥∇ℒ​\(Wt\)∥k\+2​ηtk\+2\.\\displaystyle\-\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\+C\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\\sum\_\{k=0\}^\{2L\-2\}\\frac\{1\}\{k\+2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\+2\}\\eta\_\{t\}^\{k\+2\}\.We observe the right hand side consists of2​L−12L\-1nonnegative higher\-order terms of the formηtγ​∥∇ℒ​\(Wt\)∥γ\\eta\_\{t\}^\{\\gamma\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{\\gamma\}, forγ=2,…,2​L\\gamma=2,\.\.\.,2L\. We set

ηt=12​dm​a​x1/2​L​C​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i=1ρ⁡\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i\\eta\_\{t\}=\\frac\{1\}\{2d\_\{max\}^\{1/2\}LC\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}=\\frac\{1\}\{\\rho\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}whereρ=2​dm​a​x1/2​L​C=22​L\+3​cJ​dm​a​x3/2​L5\\rho=2d\_\{max\}^\{1/2\}LC=2^\{2L\+3\}c\_\{J\}d\_\{max\}^\{3/2\}L^\{5\}\. We want to demonstrate thatηt\\eta\_\{t\}is small enough such that each term is less than or equal to

ηt2​L​∥∇ℒ​\(Wt\)∥2\.\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\.Theγ=2\\gamma=2case ofηt2​∥∇ℒ​\(Wt\)∥2\\eta\_\{t\}^\{2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}is trivial\. We have

ηt2​∥∇ℒ​\(Wt\)∥2​C2​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i≤\\displaystyle\\eta\_\{t\}^\{2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\frac\{C\}\{2\}\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\\leqηt⋅C2​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i2​L​C​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i​∥∇ℒ​\(Wt\)∥2\\displaystyle\\eta\_\{t\}\\cdot\\frac\{\\frac\{C\}\{2\}\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}\{2LC\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}≤\\displaystyle\\leqηt2​L​∥∇ℒ​\(Wt\)∥2\.\\displaystyle\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\.
Now fork=1,…,2​L−2k=1,\.\.\.,2L\-2, wherek\+2=γk\+2=\\gamma, we want to show the following statement,

ηtk\+2​∥∇ℒ​\(Wt\)∥k\+2​Ck\+2​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i≤\\displaystyle\\eta\_\{t\}^\{k\+2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\+2\}\\frac\{C\}\{k\+2\}\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\\leqηt2​L​∥∇ℒ​\(Wt\)∥2\.\\displaystyle\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\.We have

ηt​∥∇ℒ​\(Wt\)∥​≤\([14](https://arxiv.org/html/2608.11479#A1.E14)\)\\displaystyle\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\\overset\{\(\\ref\{eq:gradfbound\}\)\}\{\\leq\}\(2​cJ​ℒ​\(Wt\)\)1/2​dm​a​x1/2​L​∑k=0L−1∥Wt∥k2​dm​a​x1/2​L​C​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i≤1C,\\displaystyle\\frac\{\(2c\_\{J\}\\mathcal\{L\}\(W\_\{t\}\)\)^\{1/2\}d^\{1/2\}\_\{max\}L\\sum\_\{k=0\}^\{L\-1\}\\lVert W\_\{t\}\\rVert^\{k\}\}\{2d\_\{max\}^\{1/2\}LC\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}\\leq\\frac\{1\}\{C\},\(17\)so we can bound powers of this term,

ηtk\+2∥∇ℒ\(Wt\)\\displaystyle\\eta\_\{t\}^\{k\+2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)∥k\+2Ck\+2\(1\+ℒ\(Wt\)1/2\)∑i=02​L−2∥Wt∥i\\displaystyle\\rVert^\{k\+2\}\\frac\{C\}\{k\+2\}\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}=\\displaystyle=\(ηt​∥∇ℒ​\(Wt\)∥2\)​\(ηt​Ck\+2​\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i\)​ηtk​∥∇ℒ​\(Wt\)∥k\\displaystyle\\Big\(\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\Big\)\\Big\(\\eta\_\{t\}\\frac\{C\}\{k\+2\}\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\\Big\)\\eta\_\{t\}^\{k\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}≤\\displaystyle\\leq\(ηt​∥∇ℒ​\(Wt\)∥2\)​12​L​dm​a​x1/2​ηtk​∥∇ℒ​\(Wt\)∥k\\displaystyle\\Big\(\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\Big\)\\frac\{1\}\{2Ld\_\{max\}^\{1/2\}\}\\eta\_\{t\}^\{k\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}≤\([17](https://arxiv.org/html/2608.11479#A1.E17)\)\\displaystyle\\overset\{\(\\ref\{eq:etagrad\}\)\}\{\\leq\}\(ηt​∥∇ℒ​\(Wt\)∥2\)​12​L​dm​a​x1/2​1Ck≤ηt2​L​∥∇ℒ​\(Wt\)∥2\.\\displaystyle\\Big\(\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\Big\)\\frac\{1\}\{2Ld\_\{max\}^\{1/2\}\}\\frac\{1\}\{C^\{k\}\}\\leq\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\.
So we have shown the statement\. We have

ℒ⁡\(Wt\+1\)−ℒ⁡\(Wt\)≤\\displaystyle\\mathcal\{L\}\(W\_\{t\+1\}\)\-\\mathcal\{L\}\(W\_\{t\}\)\\leq−ηt​∥∇ℒ​\(Wt\)∥2\+ηt​2​L−12​L​∥∇ℒ​\(Wt\)∥2\\displaystyle\-\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\+\\eta\_\{t\}\\frac\{2L\-1\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}=\\displaystyle=−ηt2​L​∥∇ℒ​\(Wt\)∥2\.\\displaystyle\-\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\.Rearranging and summing on both sides yields

∑t=0T−1ηt2​L​∥∇ℒ​\(Wt\)∥2≤\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\frac\{\\eta\_\{t\}\}\{2L\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\leq∑t=0T−1ℒ⁡\(Wt\)−ℒ⁡\(Wt\+1\)\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\mathcal\{L\}\(W\_\{t\}\)\-\\mathcal\{L\}\(W\_\{t\+1\}\)∑t=0T−1ηt​∥∇ℒ​\(Wt\)∥2≤\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\leq2​L​\(ℒ⁡\(W0\)−ℒ⁡\(WT\)\)\.\\displaystyle 2L\(\\mathcal\{L\}\(W\_\{0\}\)\-\\mathcal\{L\}\(W\_\{T\}\)\)\.Sinceℒ⁡\(WT\)≥0\\mathcal\{L\}\(W\_\{T\}\)\\geq 0, we have the following bound, which we will leverage multiple times in the proof:

∑t=0T−1ηt​∥∇ℒ​\(Wt\)∥2≤2​L​ℒ​\(W0\)\.\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\leq 2L\\mathcal\{L\}\(W\_\{0\}\)\.\(18\)We can therefore bound the minimum squared gradient norm as follows\.

mint=0,…,T−1⁡∥∇ℒ​\(Wt\)∥2​∑t=0T−1ηt≤\\displaystyle\\min\_\{t=0,\.\.\.,T\-1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\leq2​L​ℒ​\(W0\),\\displaystyle 2L\\mathcal\{L\}\(W\_\{0\}\),mint=0,…,T−1⁡∥∇ℒ​\(Wt\)∥2≤\\displaystyle\\min\_\{t=0,\.\.\.,T\-1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\leq2​L​ℒ​\(W0\)∑t=0T−1ηt\.\\displaystyle\\frac\{2L\\mathcal\{L\}\(W\_\{0\}\)\}\{\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\}\.

### A\.8Proof of Theorem[4\.8](https://arxiv.org/html/2608.11479#S4.Thmtheorem8)

To show convergence of the gradient norm to zero, we need to show that the sum of learning rates∑t=0T−1ηt\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}diverges\. We achieve this by lower bounding with another series, which we show diverges\. First, we note that sinceℒ⁡\(Wt\)\\mathcal\{L\}\(W\_\{t\}\)is nonincreasing withtt, we have\(ℒ⁡\(Wt\)\)1/2≤\(ℒ⁡\(W0\)\)1/2\(\\mathcal\{L\}\(W\_\{t\}\)\)^\{1/2\}\\leq\(\\mathcal\{L\}\(W\_\{0\}\)\)^\{1/2\}\. Moreover, we have for all0≤j≤N0\\leq j\\leq N,

∥W∥j≤∥W∥N\+1\.\\lVert W\\rVert^\{j\}\\leq\\lVert W\\rVert^\{N\}\+1\.\(19\)So we have forN=2​L−2N=2L\-2,

∑i=02​L−2∥Wt∥i≤\\displaystyle\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\\leq\(2​L−1\)​\(1\+∥Wt∥N\)\\displaystyle\(2L\-1\)\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\)LetDt=ρ⁡\(1\+ℒ​\(W0\)1/2\)​\(2​L−1\)​\(1\+∥Wt∥N\)D\_\{t\}=\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\(2L\-1\)\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\), whereρ=2​dm​a​x1/2​L​C=22​L\+3​dm​a​x3/2​L5\\rho=2d\_\{max\}^\{1/2\}LC=2^\{2L\+3\}d\_\{max\}^\{3/2\}L^\{5\}\. Then we have

ηt=1ρ⁡\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i≥1Dt\\displaystyle\\eta\_\{t\}=\\frac\{1\}\{\\rho\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\}\\geq\\frac\{1\}\{D\_\{t\}\}
To show the divergence of∑t=0∞ηt\\sum\_\{t=0\}^\{\\infty\}\\eta\_\{t\}, it suffices to show the divergence of∑t=0∞1Dt\\sum\_\{t=0\}^\{\\infty\}\\frac\{1\}\{D\_\{t\}\}\. We therefore want to track the growth of∥Wt∥N\\lVert W\_\{t\}\\rVert^\{N\}asttincreases\.

We first observe that for any0≤j≤N≤2​L−20\\leq j\\leq N\\leq 2L\-2, that

∥Wt∥j​ηtjN≤\\displaystyle\\lVert W\_\{t\}\\rVert^\{j\}\\eta\_\{t\}^\{\\frac\{j\}\{N\}\}\\leq∥Wt∥j​\(1ρ​∥Wt∥N\)jN≤1,\\displaystyle\\lVert W\_\{t\}\\rVert^\{j\}\\Big\(\\frac\{1\}\{\\rho\\lVert W\_\{t\}\\rVert^\{N\}\}\\Big\)^\{\\frac\{j\}\{N\}\}\\leq 1,\(20\)where we also use thatρ\>1\\rho\>1\. We can therefore bound for anyN≤2​L−2N\\leq 2L\-2,

∥Wt\+1∥N=\\displaystyle\\lVert W\_\{t\+1\}\\rVert^\{N\}=∥Wt−ηt∇ℒ\(Wt\)∥N\\displaystyle\\lVert W\_\{t\}\-\\eta\_\{t\}\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{N\}≤\\displaystyle\\leq\(∥Wt∥\+ηt​∥∇ℒ​\(Wt\)∥\)N\\displaystyle\(\\lVert W\_\{t\}\\rVert\+\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\)^\{N\}=\\displaystyle=∑k=0N\(Nk\)​∥Wt∥N−k​ηtk​∥∇ℒ​\(Wt\)∥k\\displaystyle\\sum\_\{k=0\}^\{N\}\\binom\{N\}\{k\}\\lVert W\_\{t\}\\rVert^\{N\-k\}\\eta\_\{t\}^\{k\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}=\\displaystyle=∥Wt∥N\+N​∥Wt∥N−1​ηt​∥∇ℒ​\(Wt\)∥\+∑k=2N\(Nk\)​∥Wt∥N−k​ηtk​∥∇ℒ​\(Wt\)∥k\\displaystyle\\lVert W\_\{t\}\\rVert^\{N\}\+N\\lVert W\_\{t\}\\rVert^\{N\-1\}\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\lVert W\_\{t\}\\rVert^\{N\-k\}\\eta\_\{t\}^\{k\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}=\\displaystyle=∥Wt∥N\+N​∥Wt∥N−1​ηtN−1N​ηt1N​∥∇ℒ​\(Wt\)∥\+∑k=2N\(Nk\)​∥Wt∥N−k​ηtk/2​ηtk/2​∥∇ℒ​\(Wt\)∥k\\displaystyle\\lVert W\_\{t\}\\rVert^\{N\}\+N\\lVert W\_\{t\}\\rVert^\{N\-1\}\\eta\_\{t\}^\{\\frac\{N\-1\}\{N\}\}\\eta\_\{t\}^\{\\frac\{1\}\{N\}\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\lVert W\_\{t\}\\rVert^\{N\-k\}\\eta\_\{t\}^\{k/2\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}≤ηt≤1\\displaystyle\\overset\{\\eta\_\{t\}\\leq 1\}\{\\leq\}∥Wt∥N\+N​∥Wt∥N−1​ηtN−1N​ηt1N​∥∇ℒ​\(Wt\)∥\+∑k=2N\(Nk\)​∥Wt∥N−k​ηt​ηtk/2​∥∇ℒ​\(Wt\)∥k\\displaystyle\\lVert W\_\{t\}\\rVert^\{N\}\+N\\lVert W\_\{t\}\\rVert^\{N\-1\}\\eta\_\{t\}^\{\\frac\{N\-1\}\{N\}\}\\eta\_\{t\}^\{\\frac\{1\}\{N\}\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\lVert W\_\{t\}\\rVert^\{N\-k\}\\eta\_\{t\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}≤\([20](https://arxiv.org/html/2608.11479#A1.E20)\)\\displaystyle\\overset\{\(\\ref\{eq:exponentbound\}\)\}\{\\leq\}∥Wt∥N\+N​ηt1/N​∥∇ℒ​\(Wt\)∥\+∑k=2N\(Nk\)​ηtk/2​∥∇ℒ​\(Wt\)∥k\\displaystyle\\lVert W\_\{t\}\\rVert^\{N\}\+N\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}∥Wt\+1∥N−∥Wt∥N≤\\displaystyle\\lVert W\_\{t\+1\}\\rVert^\{N\}\-\\lVert W\_\{t\}\\rVert^\{N\}\\leqN​ηt1/N​∥∇ℒ​\(Wt\)∥\+∑k=2N\(Nk\)​ηtk/2​∥∇ℒ​\(Wt\)∥k\\displaystyle N\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}When we sum fromt=0t=0toT−1T\-1on both sides, we achieve a telescoping sum as follows,

∑t=0T−1∥Wt\+1∥N−∥Wt∥N≤\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\lVert W\_\{t\+1\}\\rVert^\{N\}\-\\lVert W\_\{t\}\\rVert^\{N\}\\leqN​∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\+∑t=0T−1∑k=2N\(Nk\)​ηtk/2​∥∇ℒ​\(Wt\)∥k,\\displaystyle N\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\},∥WT∥N≤\\displaystyle\\lVert W\_\{T\}\\rVert^\{N\}\\leq∥W0∥N\+N​∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\+∑t=0T−1∑k=2N\(Nk\)​ηtk/2​∥∇ℒ​\(Wt\)∥k\.\\displaystyle\\lVert W\_\{0\}\\rVert^\{N\}\+N\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\\eta\_\{t\}^\{k/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{k\}\.The double sum last term can be bounded via \([18](https://arxiv.org/html/2608.11479#A1.E18)\) as follows \(see also, Lemma[A\.11](https://arxiv.org/html/2608.11479#A1.Thmtheorem11)\)

∥WT∥N≤\\displaystyle\\lVert W\_\{T\}\\rVert^\{N\}\\leq∥W0∥N\+N​∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\+∑k=2N\(Nk\)​\(2​L​ℒ​\(W0\)\)k/2\.\\displaystyle\\lVert W\_\{0\}\\rVert^\{N\}\+N\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{k/2\}\.\(21\)The crux of this proof is bounding∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert\. WhenN=2N=2, which is true in the two\-layer case, this term can be directly bounded using \([18](https://arxiv.org/html/2608.11479#A1.E18)\), leading to anO⁡\(T\)O\(\\sqrt\{T\}\)growth rate\. However, forN\>2N\>2, we need to control the cumulative terms more carefully\. ForN=2​L−2N=2L\-2, we can decompose the sum as follows:

∑t=0T−1ηt1/N​∥∇ℒ​\(Wt\)∥\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert=∑t=0T−1ηt1/N−1/2​ηt1/2​∥∇ℒ​\(Wt\)∥=∑t=0T−1ηt2−N2​N​ηt1/2​∥∇ℒ​\(Wt\)∥,\\displaystyle=\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{1/N\-1/2\}\\eta\_\{t\}^\{1/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert=\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{\\frac\{2\-N\}\{2N\}\}\\eta\_\{t\}^\{1/2\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert,≤Cauchy\-Schwarz\\displaystyle\\overset\{\\text\{Cauchy\-Schwarz\}\}\{\\leq\}\(∑t=0T−1ηt2−NN\)1/2​\(∑t=0T−1ηt​∥∇ℒ​\(Wt\)∥2\)1/2\\displaystyle\\Big\(\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{\\frac\{2\-N\}\{N\}\}\\Big\)^\{1/2\}\\Big\(\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\Big\)^\{1/2\}≤\([18](https://arxiv.org/html/2608.11479#A1.E18)\)\\displaystyle\\overset\{\(\\ref\{eq:cumulativeappend\}\)\}\{\\leq\}\(∑t=0T−1ηt2−NN\)1/2​\(2​L​ℒ​\(W0\)\)1/2\.\\displaystyle\\Big\(\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{\\frac\{2\-N\}\{N\}\}\\Big\)^\{1/2\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{1/2\}\.We have

ηt2−NN=\\displaystyle\\eta\_\{t\}^\{\\frac\{2\-N\}\{N\}\}=\(1ηt\)N−2N=\(ρ⁡\(1\+ℒ​\(Wt\)1/2\)​∑i=02​L−2∥Wt∥i\)N−2N\\displaystyle\\Big\(\\frac\{1\}\{\\eta\_\{t\}\}\\Big\)^\{\\frac\{N\-2\}\{N\}\}=\(\\rho\(1\+\\mathcal\{L\}\(W\_\{t\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\)^\{\\frac\{N\-2\}\{N\}\}≤Lemma[4\.7](https://arxiv.org/html/2608.11479#S4.Thmtheorem7)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:onestep\}\}\}\{\\leq\}\(ρ⁡\(1\+ℒ​\(W0\)1/2\)​∑i=02​L−2∥Wt∥i\)N−2N\\displaystyle\(\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\\lVert W\_\{t\}\\rVert^\{i\}\)^\{\\frac\{N\-2\}\{N\}\}≤\([19](https://arxiv.org/html/2608.11479#A1.E19)\)\\displaystyle\\overset\{\(\\ref\{eq:boundoneplus\}\)\}\{\\leq\}\(ρ⁡\(1\+ℒ​\(W0\)1/2\)​∑i=02​L−2\(1\+∥Wt∥N\)\)N−2N\\displaystyle\(\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\\sum\_\{i=0\}^\{2L\-2\}\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\)\)^\{\\frac\{N\-2\}\{N\}\}=\\displaystyle=\(ρ⁡\(1\+ℒ​\(W0\)1/2\)​\(2​L−1\)​\(1\+∥Wt∥N\)\)N−2N\\displaystyle\(\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\(2L\-1\)\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\)\)^\{\\frac\{N\-2\}\{N\}\}Plugging this back into \([21](https://arxiv.org/html/2608.11479#A1.E21)\), we have

∥WT∥N≤\\displaystyle\\lVert W\_\{T\}\\rVert^\{N\}\\leq∥W0∥N\+N​\(∑t=0T−1\(ρ⁡\(1\+ℒ​\(W0\)1/2\)​\(2​L−1\)​\(1\+∥Wt∥N\)\)N−2N\)1/2​\(2​L​ℒ​\(W0\)\)1/2\+∑k=2N\(Nk\)​\(2​L​ℒ​\(W0\)\)k/2\\displaystyle\\lVert W\_\{0\}\\rVert^\{N\}\+N\\Big\(\\sum\_\{t=0\}^\{T\-1\}\(\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\(2L\-1\)\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\)\)^\{\\frac\{N\-2\}\{N\}\}\\Big\)^\{1/2\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{1/2\}\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{k/2\}=\\displaystyle=∥W0∥N\+N​\(ρ⁡\(1\+ℒ​\(W0\)1/2\)​\(2​L−1\)\)N−22​N​\(∑t=0T−1\(1\+∥Wt∥N\)N−2N\)1/2​\(2​L​ℒ​\(W0\)\)1/2\+∑k=2N\(Nk\)​\(2​L​ℒ​\(W0\)\)k/2\.\\displaystyle\\lVert W\_\{0\}\\rVert^\{N\}\+N\(\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\(2L\-1\)\)^\{\\frac\{N\-2\}\{2N\}\}\\Big\(\\sum\_\{t=0\}^\{T\-1\}\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\)^\{\\frac\{N\-2\}\{N\}\}\\Big\)^\{1/2\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{1/2\}\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{k/2\}\.We therefore have that∥WT∥N\\lVert W\_\{T\}\\rVert^\{N\}is bounded as

∥WT∥N≤\\displaystyle\\lVert W\_\{T\}\\rVert^\{N\}\\leqA​\(∑t=0T−1\(1\+∥Wt∥N\)N−2N\)1/2\+B\.\\displaystyle A\\Big\(\\sum\_\{t=0\}^\{T\-1\}\(1\+\\lVert W\_\{t\}\\rVert^\{N\}\)^\{\\frac\{N\-2\}\{N\}\}\\Big\)^\{1/2\}\+B\.whereA=N​\(ρ⁡\(1\+ℒ​\(W0\)1/2\)​\(2​L−1\)\)N−22​N​\(2​L​ℒ​\(W0\)\)1/2A=N\(\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\(2L\-1\)\)^\{\\frac\{N\-2\}\{2N\}\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{1/2\}andB=∥W0∥N\+∑k=2N\(Nk\)​\(2​L​ℒ​\(W0\)\)k/2B=\\lVert W\_\{0\}\\rVert^\{N\}\+\\sum\_\{k=2\}^\{N\}\\binom\{N\}\{k\}\(2L\\mathcal\{L\}\(W\_\{0\}\)\)^\{k/2\}\.

So the growth of∥WT∥N\\lVert W\_\{T\}\\rVert^\{N\}depends on the growth of thet=0,…,T−1t=0,\.\.\.,T\-1iterates before; for example, as long as those can be shown to grow at most linearly, then∥WT∥N\\lVert W\_\{T\}\\rVert^\{N\}will also be linear inTT\. In particular, we can show the growth issublinearby proving with induction that there existsα\>0\\alpha\>0,β\>0\\beta\>0, where for allT\>0T\>0

∥WT∥T≤α​TNN\+2\+β\.\\displaystyle\\lVert W\_\{T\}\\rVert^\{T\}\\leq\\alpha T^\{\\frac\{N\}\{N\+2\}\}\+\\beta\.Letβ=B\\beta=B\. For the base case, we have

∥W0∥N≤B\.\\displaystyle\\lVert W\_\{0\}\\rVert^\{N\}\\leq B\.
For the inductive step, suppose the hypothesis holds fort≤T−1t\\leq T\-1\. We have

∥WT∥N≤\\displaystyle\\lVert W\_\{T\}\\rVert^\{N\}\\leqA​\(∑t=0T−1\(1\+α​tNN\+2\+β\)N−2N\)1/2\+β\\displaystyle A\\Big\(\\sum\_\{t=0\}^\{T\-1\}\(1\+\\alpha t^\{\\frac\{N\}\{N\+2\}\}\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\\Big\)^\{1/2\}\+\\beta≤N−2N<1\\displaystyle\\overset\{\\frac\{N\-2\}\{N\}<1\}\{\\leq\}A​\(∑t=0T−1\(1\+β\)N−2N\+\(α​tNN\+2\)N−2N\)1/2\+β\\displaystyle A\\Big\(\\sum\_\{t=0\}^\{T\-1\}\(1\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\+\(\\alpha t^\{\\frac\{N\}\{N\+2\}\}\)^\{\\frac\{N\-2\}\{N\}\}\\Big\)^\{1/2\}\+\\beta=\\displaystyle=A​\(∑t=0T−1\(\(1\+β\)N−2N\+αN−2N​tN−2N\+2\)\)1/2\+β\\displaystyle A\\Big\(\\sum\_\{t=0\}^\{T\-1\}\(\(1\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\+\\alpha^\{\\frac\{N\-2\}\{N\}\}t^\{\\frac\{N\-2\}\{N\+2\}\}\)\\Big\)^\{1/2\}\+\\beta≤\\displaystyle\\leqA​\(T​\(1\+β\)N−2N\+T⋅αN−2N​TN−2N\+2\)1/2\+β\\displaystyle A\\Big\(T\(1\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\+T\\cdot\\alpha^\{\\frac\{N\-2\}\{N\}\}T^\{\\frac\{N\-2\}\{N\+2\}\}\\Big\)^\{1/2\}\+\\beta=\\displaystyle=A​\(T​\(1\+β\)N−2N\+αN−2N​T2​NN\+2\)1/2\+β\\displaystyle A\\Big\(T\(1\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\+\\alpha^\{\\frac\{N\-2\}\{N\}\}T^\{\\frac\{2N\}\{N\+2\}\}\\Big\)^\{1/2\}\+\\beta≤2​NN\+2\>1\\displaystyle\\overset\{\\frac\{2N\}\{N\+2\}\>1\}\{\\leq\}A​\(\(\(1\+β\)N−2N\+αN−2N\)​T2​NN\+2\)1/2\+β\\displaystyle A\\Big\(\(\(1\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\+\\alpha^\{\\frac\{N\-2\}\{N\}\}\)T^\{\\frac\{2N\}\{N\+2\}\}\\Big\)^\{1/2\}\+\\beta=\\displaystyle=A​\(\(1\+β\)N−2N\+αN−2N\)1/2​TNN\+2\+β\\displaystyle A\(\(1\+\\beta\)^\{\\frac\{N\-2\}\{N\}\}\+\\alpha^\{\\frac\{N\-2\}\{N\}\}\)^\{1/2\}T^\{\\frac\{N\}\{N\+2\}\}\+\\beta≤\\displaystyle\\leqA⁡\(\(1\+β\)N−22​N\+αN−22​N\)​TNN\+2\+β\.\\displaystyle A\(\(1\+\\beta\)^\{\\frac\{N\-2\}\{2N\}\}\+\\alpha^\{\\frac\{N\-2\}\{2N\}\}\)T^\{\\frac\{N\}\{N\+2\}\}\+\\beta\.To achieve the proof, we need to show there existsα\\alphasuch that

A⁡\(\(1\+β\)N−22​N\+αN−22​N\)​TNN\+2\+β≤\\displaystyle A\(\(1\+\\beta\)^\{\\frac\{N\-2\}\{2N\}\}\+\\alpha^\{\\frac\{N\-2\}\{2N\}\}\)T^\{\\frac\{N\}\{N\+2\}\}\+\\beta\\leqα​TNN\+2\+β\\displaystyle\\alpha T^\{\\frac\{N\}\{N\+2\}\}\+\\betaA⁡\(\(1\+β\)N−22​N\+αN−22​N\)≤α\.\\displaystyle A\(\(1\+\\beta\)^\{\\frac\{N\-2\}\{2N\}\}\+\\alpha^\{\\frac\{N\-2\}\{2N\}\}\)\\leq\\alpha\.Since the right hand side grows linearly inα\\alphawhile the left hand side is sublinear, growing asαN−22​N\\alpha^\{\\frac\{N\-2\}\{2N\}\}, we are guaranteed to findα∗\\alpha^\{\*\}large enough to satisfy this condition\. We therefore have that∑t=0T−11Dt\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{D\_\{t\}\}is lower bounded by app\-series that grows asT1/LT^\{1/L\}, which also lower bounds∑t=0T−1ηt\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\.

∑t=0T−1ηt≥∑t=0T−11Dt≥∑t=0T−11ρ⁡\(1\+ℒ​\(W0\)1/2\)​\(2​L−1\)​\(1\+α∗​tL−1L\+β\)=Θ⁡\(T1/L\)\.\\displaystyle\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\\geq\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{D\_\{t\}\}\\geq\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{\\rho\(1\+\\mathcal\{L\}\(W\_\{0\}\)^\{1/2\}\)\(2L\-1\)\(1\+\\alpha^\{\*\}t^\{\\frac\{L\-1\}\{L\}\}\+\\beta\)\}=\\Theta\(T^\{1/L\}\)\.
We therefore have that

mint=0,…,T−1⁡∥∇ℒ​\(Wt\)∥2≤\\displaystyle\\min\_\{t=0,\.\.\.,T\-1\}\\lVert\\nabla\\mathcal\{L\}\(W\_\{t\}\)\\rVert^\{2\}\\leq2​L​ℒ​\(W0\)∑t=0T−1ηt=O⁡\(1T1/L\),\\displaystyle\\frac\{2L\\mathcal\{L\}\(W\_\{0\}\)\}\{\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}\}=O\(\\frac\{1\}\{T^\{1/L\}\}\),which concludes the proof\.

### A\.9Bounds onzℓ​\(W\)z\_\{\\ell\}\(W\),hℓ​\(W\)h\_\{\\ell\}\(W\), andδℓ​\(W\)\\delta\_\{\\ell\}\(W\)

###### Lemma A\.1\.

Forzℓ​\(W\)z\_\{\\ell\}\(W\)defined in \([9](https://arxiv.org/html/2608.11479#S4.E9)\), we have for allW∈ℝD2×D1W\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}

∥zℓ​\(W\)∥≤dm​a​x1/2​∑i=1ℓ∥W∥i\.\\lVert z\_\{\\ell\}\(W\)\\rVert\\leq d\_\{max\}^\{1/2\}\\sum\_\{i=1\}^\{\\ell\}\\lVert W\\rVert^\{i\}\.

###### Proof\.

For the base caseℓ=1\\ell=1, we have

∥z1​\(W\)∥=∥E1​\[W\]​xi∥≤∥W∥​∥xi∥≤∥W∥​cx​d1/2\.\\displaystyle\\lVert z\_\{1\}\(W\)\\rVert=\\lVert E\_\{1\}\[W\]x\_\{i\}\\rVert\\leq\\lVert W\\rVert\\lVert x\_\{i\}\\rVert\\leq\\lVert W\\rVert c\_\{x\}d^\{1/2\}\.For the general case, we have

∥zℓ​\(W\)∥=\\displaystyle\\lVert z\_\{\\ell\}\(W\)\\rVert=∥Eℓ​\[W\]​σ​\(zℓ−1​\(W\)\)∥\\displaystyle\\lVert E\_\{\\ell\}\[W\]\\sigma\(z\_\{\\ell\-1\}\(W\)\)\\rVert≤\\displaystyle\\leq∥W∥​∥σ⁡\(zℓ−1​\(W\)\)∥\\displaystyle\\lVert W\\rVert\\lVert\\sigma\(z\_\{\\ell\-1\}\(W\)\)\\rVert≤\([1](https://arxiv.org/html/2608.11479#S3.E1)\)\\displaystyle\\overset\{\(\\ref\{eq:l0\}\)\}\{\\leq\}∥W∥\(c0dℓ−11/2\+c1∥zℓ−1\(W\)\)∥\\displaystyle\\lVert W\\rVert\(c\_\{0\}d^\{1/2\}\_\{\\ell\-1\}\+c\_\{1\}\\lVert z\_\{\\ell\-1\}\(W\)\)\\rVert≤\\displaystyle\\leqdm​a​x1/2​∑i=1ℓ∥W∥i\.\\displaystyle d\_\{max\}^\{1/2\}\\sum\_\{i=1\}^\{\\ell\}\\lVert W\\rVert^\{i\}\.∎

###### Lemma A\.2\.

For allW1,W2∈ℝD2×D1W\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}andℓ=1,…,L\\ell=1,\.\.\.,L, we have

∥zℓ​\(W2\)−zℓ​\(W1\)∥≤\\displaystyle\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert\\leqdm​a​x1/2​∥W2−W1∥​∑k=0ℓ−1∑j=0ℓ−k−1∥W1∥j​∥W2∥k\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{\\ell\-1\}\\sum\_\{j=0\}^\{\\ell\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.

###### Proof\.

We first show the bound forℓ=1\\ell=1\.

∥z1​\(W2\)−z1​\(W1\)∥=\\displaystyle\\lVert z\_\{1\}\(W\_\{2\}\)\-z\_\{1\}\(W\_\{1\}\)\\rVert=∥E1​\[W2\]​xi−E1​\[W1\]​xi∥\\displaystyle\\lVert E\_\{1\}\[W\_\{2\}\]x\_\{i\}\-E\_\{1\}\[W\_\{1\}\]x\_\{i\}\\rVert≤\\displaystyle\\leqcx​d1/2​∥W2−W1∥≤dm​a​x1/2​∥W2−W1∥\.\\displaystyle c\_\{x\}d^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\leq d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\.
Then for generalℓ=2,…,L\\ell=2,\.\.\.,L, we have

∥zℓ​\(W2\)−zℓ​\(W1\)∥=\\displaystyle\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert=∥Eℓ​\[W2\]​σ​\(zℓ−1​\(W2\)\)−Eℓ​\[W1\]​σ​\(zℓ−1​\(W1\)\)∥\\displaystyle\\lVert E\_\{\\ell\}\[W\_\{2\}\]\\sigma\(z\_\{\\ell\-1\}\(W\_\{2\}\)\)\-E\_\{\\ell\}\[W\_\{1\}\]\\sigma\(z\_\{\\ell\-1\}\(W\_\{1\}\)\)\\rVert≤\\displaystyle\\leq∥W2∥​∥σ⁡\(zℓ−1​\(W2\)\)−σ⁡\(zℓ−1​\(W1\)\)∥\+∥σ⁡\(zℓ−1​\(W1\)\)∥​∥W2−W1∥\\displaystyle\\lVert W\_\{2\}\\rVert\\lVert\\sigma\(z\_\{\\ell\-1\}\(W\_\{2\}\)\)\-\\sigma\(z\_\{\\ell\-1\}\(W\_\{1\}\)\)\\rVert\+\\lVert\\sigma\(z\_\{\\ell\-1\}\(W\_\{1\}\)\)\\rVert\\lVert W\_\{2\}\-W\_\{1\}\\rVert≤\\displaystyle\\leqc2​∥W2∥​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥\+∥σ⁡\(zℓ−1​\(W1\)\)∥​∥W2−W1∥\\displaystyle c\_\{2\}\\lVert W\_\{2\}\\rVert\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+\\lVert\\sigma\(z\_\{\\ell\-1\}\(W\_\{1\}\)\)\\rVert\\lVert W\_\{2\}\-W\_\{1\}\\rVert≤\\displaystyle\\leq∥W2∥​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥\+\(dℓ−11/2\+∥zℓ−1​\(W1\)∥\)​∥W2−W1∥\\displaystyle\\lVert W\_\{2\}\\rVert\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+\(d\_\{\\ell\-1\}^\{1/2\}\+\\lVert z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\)\\lVert W\_\{2\}\-W\_\{1\}\\rVert≤Lemma[A\.1](https://arxiv.org/html/2608.11479#A1.Thmtheorem1)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:zbound\}\}\}\{\\leq\}∥W2∥​∥zℓ−1​\(W2\)−zℓ−1​\(W1\)∥\+dm​a​x1/2​\(∑i=0ℓ−1∥W1∥i\)​∥W2−W1∥\\displaystyle\\lVert W\_\{2\}\\rVert\\lVert z\_\{\\ell\-1\}\(W\_\{2\}\)\-z\_\{\\ell\-1\}\(W\_\{1\}\)\\rVert\+d\_\{max\}^\{1/2\}\\Big\(\\sum\_\{i=0\}^\{\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{i\}\\Big\)\\lVert W\_\{2\}\-W\_\{1\}\\rVert≤\\displaystyle\\leqdm​a​x1/2​∥W2−W1∥​∑k=0ℓ−1∑j=0ℓ−k−1∥W1∥j​∥W2∥k\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{\\ell\-1\}\\sum\_\{j=0\}^\{\\ell\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}
∎

###### Lemma A\.3\.

Forhℓ​\(W\)h\_\{\\ell\}\(W\)defined in \([10](https://arxiv.org/html/2608.11479#S4.E10)\), we have for allW∈ℝD2×D1W\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\},

∥hℓ​\(W\)∥≤dm​a​x1/2​∑k=0ℓ∥W∥k\.\\lVert h\_\{\\ell\}\(W\)\\rVert\\leq d\_\{max\}^\{1/2\}\\sum\_\{k=0\}^\{\\ell\}\\lVert W\\rVert^\{k\}\.

###### Proof\.

For the base caseℓ=0\\ell=0, we have

∥h0​\(W\)∥=∥xi∥≤d1/2​cx≤d1/2\.\\lVert h\_\{0\}\(W\)\\rVert=\\lVert x\_\{i\}\\rVert\\leq d^\{1/2\}c\_\{x\}\\leq d^\{1/2\}\.For the general case, we have by Lemma[A\.1](https://arxiv.org/html/2608.11479#A1.Thmtheorem1),

∥hℓ​\(W\)∥=\\displaystyle\\lVert h\_\{\\ell\}\(W\)\\rVert=∥σ​\(zℓ​\(W\)\)∥\\displaystyle\\lVert\\sigma\(z\_\{\\ell\}\(W\)\)\\rVert≤Lemma[A\.7](https://arxiv.org/html/2608.11479#A1.Thmtheorem7)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:elementwisesigma\}\}\}\{\\leq\}c0​dℓ1/2\+c1​∥zℓ​\(W\)∥,\\displaystyle c\_\{0\}d\_\{\\ell\}^\{1/2\}\+c\_\{1\}\\lVert z\_\{\\ell\}\(W\)\\rVert,≤\\displaystyle\\leqdℓ1/2\+dm​a​x1/2​∑i=1ℓ∥W∥i\.\\displaystyle d\_\{\\ell\}^\{1/2\}\+d\_\{max\}^\{1/2\}\\sum\_\{i=1\}^\{\\ell\}\\lVert W\\rVert^\{i\}\.∎

###### Lemma A\.4\.

For allW1,W2∈ℝD2×D1W\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}andℓ=0\\ell=0, we have

∥h0​\(W2\)−h0​\(W1\)∥=0\.\\lVert h\_\{0\}\(W\_\{2\}\)\-h\_\{0\}\(W\_\{1\}\)\\rVert=0\.Forℓ=1,…,L\\ell=1,\.\.\.,L, we have

∥hℓ​\(W2\)−hℓ​\(W1\)∥≤dm​a​x1/2​∥W2−W1∥​∑k=0ℓ−1∑j=0ℓ−k−1∥W1∥j​∥W2∥k\.\\lVert h\_\{\\ell\}\(W\_\{2\}\)\-h\_\{\\ell\}\(W\_\{1\}\)\\rVert\\leq d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{\\ell\-1\}\\sum\_\{j=0\}^\{\\ell\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.

###### Proof\.

Forℓ=0\\ell=0, we have

∥h0​\(W2\)−h0​\(W1\)∥=0\.\\lVert h\_\{0\}\(W\_\{2\}\)\-h\_\{0\}\(W\_\{1\}\)\\rVert=0\.
Forℓ=1,\.\.,L\\ell=1,\.\.,L

∥hℓ​\(W2\)−hℓ​\(W1\)∥≤\\displaystyle\\lVert h\_\{\\ell\}\(W\_\{2\}\)\-h\_\{\\ell\}\(W\_\{1\}\)\\rVert\\leq∥σ⁡\(zℓ​\(W2\)\)−σ⁡\(zℓ​\(W1\)\)∥,\\displaystyle\\lVert\\sigma\(z\_\{\\ell\}\(W\_\{2\}\)\)\-\\sigma\(z\_\{\\ell\}\(W\_\{1\}\)\)\\rVert,≤\\displaystyle\\leqc2​∥zℓ​\(W2\)−zℓ​\(W1\)∥,\\displaystyle c\_\{2\}\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert,≤Lemma[A\.2](https://arxiv.org/html/2608.11479#A1.Thmtheorem2)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:zchange\}\}\}\{\\leq\}dm​a​x1/2​∥W2−W1∥​∑k=0ℓ−1∑j=0ℓ−k−1∥W1∥j​∥W2∥k\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{\\ell\-1\}\\sum\_\{j=0\}^\{\\ell\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.
∎

###### Lemma A\.5\.

Forδℓ​\(W\)\\delta\_\{\\ell\}\(W\)defined in \([11](https://arxiv.org/html/2608.11479#S4.E11)\), we have for allW∈ℝD2×D1W\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\},

∥δℓ​\(W\)∥≤∥W∥L−ℓ\.\\lVert\\delta\_\{\\ell\}\(W\)\\rVert\\leq\\lVert W\\rVert^\{L\-\\ell\}\.

###### Proof\.

For the base caseℓ=L\\ell=L, we have the following due to Assumption[3\.1](https://arxiv.org/html/2608.11479#S3.Thmtheorem1)anddL=1d\_\{L\}=1,

∥δL​\(W\)∥=\\displaystyle\\lVert\\delta\_\{L\}\(W\)\\rVert=∥σ′​\(zL​\(W\)\)∥≤c2≤1\.\\displaystyle\\lVert\\sigma^\{\\prime\}\(z\_\{L\}\(W\)\)\\rVert\\leq c\_\{2\}\\leq 1\.For generalℓ\\ell, we have

∥δℓ​\(W\)∥=\\displaystyle\\lVert\\delta\_\{\\ell\}\(W\)\\rVert=∥σ′​\(zℓ​\(W\)\)⊙\(Eℓ\+1​\[W\]T​δℓ\+1​\(W\)\)∥,\\displaystyle\\lVert\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\)\)\\odot\(E\_\{\\ell\+1\}\[W\]^\{T\}\\delta\_\{\\ell\+1\}\(W\)\)\\rVert,≤Lemma[A\.14](https://arxiv.org/html/2608.11479#A1.Thmtheorem14)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:hademard\}\}\}\{\\leq\}c2​∥Eℓ\+1​\[W\]T​δℓ\+1​\(W\)∥,\\displaystyle c\_\{2\}\\lVert E\_\{\\ell\+1\}\[W\]^\{T\}\\delta\_\{\\ell\+1\}\(W\)\\rVert,≤\\displaystyle\\leq∥W∥​∥δℓ\+1​\(W\)∥,\\displaystyle\\lVert W\\rVert\\lVert\\delta\_\{\\ell\+1\}\(W\)\\rVert,≤\\displaystyle\\leq∥W∥L−ℓ\.\\displaystyle\\lVert W\\rVert^\{L\-\\ell\}\.∎

###### Lemma A\.6\.

For allW1,W2∈ℝD2×D1W\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}, we have

∥δL​\(W2\)−δL​\(W1\)∥≤\\displaystyle\\lVert\\delta\_\{L\}\(W\_\{2\}\)\-\\delta\_\{L\}\(W\_\{1\}\)\\rVert\\leq∥zL​\(W2\)−zL​\(W1\)∥≤dm​a​x1/2​∥W2−W1∥​∑k=0L−1∑j=0L−k−1∥W1∥j​∥W2∥k\.\\displaystyle\\lVert z\_\{L\}\(W\_\{2\}\)\-z\_\{L\}\(W\_\{1\}\)\\rVert\\leq d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\sum\_\{j=0\}^\{L\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.and forℓ=1,\.\.,L−1\\ell=1,\.\.,L\-1, we have

∥δℓ​\(W2\)−δℓ​\(W1\)∥≤\\displaystyle\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert\\leqdm​a​x1/2​∥W2−W1∥​\[∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥j\+∑k=0L−ℓ−1∥W2∥k​∥W1∥L−k−ℓ−1\]\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Big\[\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\+\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}\\Big\]\.

###### Proof\.

For the base caseℓ=L\\ell=L, we have

∥δL​\(W2\)−δL​\(W1\)∥=\\displaystyle\\lVert\\delta\_\{L\}\(W\_\{2\}\)\-\\delta\_\{L\}\(W\_\{1\}\)\\rVert=∥σ′​\(zL​\(W2\)\)−σ′​\(zL​\(W1\)\)∥\\displaystyle\\lVert\\sigma^\{\\prime\}\(z\_\{L\}\(W\_\{2\}\)\)\-\\sigma^\{\\prime\}\(z\_\{L\}\(W\_\{1\}\)\)\\rVert≤\([3](https://arxiv.org/html/2608.11479#S3.E3)\)\\displaystyle\\overset\{\(\\ref\{eq:l3\}\)\}\{\\leq\}c3​∥zL​\(W2\)−zL​\(W1\)∥,\\displaystyle c\_\{3\}\\lVert z\_\{L\}\(W\_\{2\}\)\-z\_\{L\}\(W\_\{1\}\)\\rVert,≤Lemma[A\.2](https://arxiv.org/html/2608.11479#A1.Thmtheorem2)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:zchange\}\}\}\{\\leq\}dm​a​x1/2​∥W2−W1∥​∑k=0L−1∑j=0L−k−1∥W1∥j​∥W2∥k\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=0\}^\{L\-1\}\\sum\_\{j=0\}^\{L\-k\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{k\}\.Forℓ≤L−1\\ell\\leq L\-1, we have

∥δℓ​\(W2\)−δℓ​\(W1\)∥=\\displaystyle\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert=∥σ′​\(zℓ​\(W2\)\)⊙\(Eℓ\+1​\[W2\]T​δℓ\+1​\(W2\)\)−σ′​\(zℓ​\(W1\)\)⊙\(Eℓ\+1​\[W1\]T​δℓ\+1​\(W1\)\)∥\\displaystyle\\lVert\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{2\}\)\)\\odot\(E\_\{\\ell\+1\}\[W\_\{2\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\)\-\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{1\}\)\)\\odot\(E\_\{\\ell\+1\}\[W\_\{1\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\)\\rVert≤\\displaystyle\\leq∥σ′​\(zℓ​\(W2\)\)⊙\(Eℓ\+1​\[W2\]T​δℓ\+1​\(W2\)−Eℓ\+1​\[W1\]T​δℓ\+1​\(W1\)\)∥\\displaystyle\\lVert\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{2\}\)\)\\odot\(E\_\{\\ell\+1\}\[W\_\{2\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\-E\_\{\\ell\+1\}\[W\_\{1\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\)\\rVert\+∥σ′​\(zℓ​\(W2\)\)−σ′​\(zℓ​\(W1\)\)∥​∥Eℓ\+1​\[W1\]T​δℓ\+1​\(W1\)∥\\displaystyle\+\\lVert\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{2\}\)\)\-\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{1\}\)\)\\rVert\\lVert E\_\{\\ell\+1\}\[W\_\{1\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert≤\\displaystyle\\leqc2​∥Eℓ\+1​\[W2\]T​δℓ\+1​\(W2\)−Eℓ\+1​\[W1\]T​δℓ\+1​\(W1\)∥\\displaystyle c\_\{2\}\\lVert E\_\{\\ell\+1\}\[W\_\{2\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\-E\_\{\\ell\+1\}\[W\_\{1\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert\+∥σ′​\(zℓ​\(W2\)\)−σ′​\(zℓ​\(W1\)\)∥​∥Eℓ\+1​\[W1\]T​δℓ\+1​\(W1\)∥\\displaystyle\+\\lVert\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{2\}\)\)\-\\sigma^\{\\prime\}\(z\_\{\\ell\}\(W\_\{1\}\)\)\\rVert\\lVert E\_\{\\ell\+1\}\[W\_\{1\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert≤\([3](https://arxiv.org/html/2608.11479#S3.E3)\)\\displaystyle\\overset\{\(\\ref\{eq:l3\}\)\}\{\\leq\}c2​∥Eℓ\+1​\[W2\]T​δℓ\+1​\(W2\)−Eℓ\+1​\[W1\]T​δℓ\+1​\(W1\)∥\\displaystyle c\_\{2\}\\lVert E\_\{\\ell\+1\}\[W\_\{2\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\-E\_\{\\ell\+1\}\[W\_\{1\}\]^\{T\}\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert\+c3​∥zℓ​\(W2\)−zℓ​\(W1\)∥​∥W1∥​∥δℓ\+1​\(W1\)∥\\displaystyle\+c\_\{3\}\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert\\lVert W\_\{1\}\\rVert\\lVert\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert≤\\displaystyle\\leq∥δℓ\+1​\(W1\)∥​∥W2−W1∥\+∥W2∥​∥δℓ\+1​\(W2\)−δℓ\+1​\(W1\)∥\\displaystyle\\lVert\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert\\lVert W\_\{2\}\-W\_\{1\}\\rVert\+\\lVert W\_\{2\}\\rVert\\lVert\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\-\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert\+∥zℓ​\(W2\)−zℓ​\(W1\)∥​∥W1∥​∥δℓ\+1​\(W1\)∥\\displaystyle\+\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert\\lVert W\_\{1\}\\rVert\\lVert\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert≤Lemma[A\.5](https://arxiv.org/html/2608.11479#A1.Thmtheorem5)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:deltabound\}\}\}\{\\leq\}∥W1∥L−ℓ−1​∥W2−W1∥\+∥W2∥​∥δℓ\+1​\(W2\)−δℓ\+1​\(W1\)∥\\displaystyle\\lVert W\_\{1\}\\rVert^\{L\-\\ell\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\+\\lVert W\_\{2\}\\rVert\\lVert\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\-\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert\+∥zℓ​\(W2\)−zℓ​\(W1\)∥​∥W1∥​∥W1∥L−ℓ−1\\displaystyle\+\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert\\lVert W\_\{1\}\\rVert\\lVert W\_\{1\}\\rVert^\{L\-\\ell\-1\}=\\displaystyle=∥W1∥L−ℓ−1​∥W2−W1∥\+∥W2∥​∥δℓ\+1​\(W2\)−δℓ\+1​\(W1\)∥\\displaystyle\\lVert W\_\{1\}\\rVert^\{L\-\\ell\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\+\\lVert W\_\{2\}\\rVert\\lVert\\delta\_\{\\ell\+1\}\(W\_\{2\}\)\-\\delta\_\{\\ell\+1\}\(W\_\{1\}\)\\rVert\+∥zℓ​\(W2\)−zℓ​\(W1\)∥​∥W1∥L−ℓ\\displaystyle\+\\lVert z\_\{\\ell\}\(W\_\{2\}\)\-z\_\{\\ell\}\(W\_\{1\}\)\\rVert\\lVert W\_\{1\}\\rVert^\{L\-\\ell\}Solving the recursive relationship leads to

∥δℓ​\(W2\)−δℓ​\(W1\)∥≤\\displaystyle\\lVert\\delta\_\{\\ell\}\(W\_\{2\}\)\-\\delta\_\{\\ell\}\(W\_\{1\}\)\\rVert\\leq∑k=ℓL∥W2∥k−ℓ​∥zk​\(W2\)−zk​\(W1\)∥​∥W1∥L−k\+∑k=ℓL−1∥W2∥k−ℓ​∥W1∥L−k−1​∥W2−W1∥\\displaystyle\\sum\_\{k=\\ell\}^\{L\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\}\\lVert z\_\{k\}\(W\_\{2\}\)\-z\_\{k\}\(W\_\{1\}\)\\rVert\\lVert W\_\{1\}\\rVert^\{L\-k\}\+\\sum\_\{k=\\ell\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\}\\lVert W\_\{1\}\\rVert^\{L\-k\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert≤Lemma[A\.2](https://arxiv.org/html/2608.11479#A1.Thmtheorem2)\\displaystyle\\overset\{\\text\{Lemma \\ref\{lemma:zchange\}\}\}\{\\leq\}dm​a​x1/2​∥W2−W1∥​∑k=ℓL∥W2∥k−ℓ​\(∑i=0k−1∑j=0k−i−1∥W1∥j​∥W2∥i\)​∥W1∥L−k\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=\\ell\}^\{L\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\}\(\\sum\_\{i=0\}^\{k\-1\}\\sum\_\{j=0\}^\{k\-i\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\\lVert W\_\{2\}\\rVert^\{i\}\)\\lVert W\_\{1\}\\rVert^\{L\-k\}\+∑k=ℓL−1∥W2∥k−ℓ∥W1∥L−k−1∥W2−W1∥\\displaystyle\+\\sum\_\{k=\\ell\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\}\\lVert W\_\{1\}\\rVert^\{L\-k\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert=\\displaystyle=dm​a​x1/2​∥W2−W1∥​∑k=ℓL∑i=0k−1∑j=0k−i−1∥W1∥L−k\+j​∥W2∥k−ℓ\+i\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=0\}^\{k\-1\}\\sum\_\{j=0\}^\{k\-i\-1\}\\lVert W\_\{1\}\\rVert^\{L\-k\+j\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\+i\}\+∑k=ℓL−1∥W2∥k−ℓ∥W1∥L−k−1∥W2−W1∥\\displaystyle\+\\sum\_\{k=\\ell\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\}\\lVert W\_\{1\}\\rVert^\{L\-k\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert=\\displaystyle=dm​a​x1/2​∥W2−W1∥​∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥j\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\+∑k=ℓL−1∥W2∥k−ℓ∥W1∥L−k−1∥W2−W1∥\\displaystyle\+\\sum\_\{k=\\ell\}^\{L\-1\}\\lVert W\_\{2\}\\rVert^\{k\-\\ell\}\\lVert W\_\{1\}\\rVert^\{L\-k\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert=\\displaystyle=dm​a​x1/2​∥W2−W1∥​∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥j\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\+∑k=0L−ℓ−1∥W2∥k∥W1∥L−k−ℓ−1∥W2−W1∥\\displaystyle\+\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert≤\\displaystyle\\leqdm​a​x1/2​∥W2−W1∥​\[∑k=ℓL∑i=k−ℓ2​k−1−ℓ∥W2∥i​∑j=L−kL−i\+k−ℓ−1∥W1∥j\+∑k=0L−ℓ−1∥W2∥k​∥W1∥L−k−ℓ−1\]\.\\displaystyle d\_\{max\}^\{1/2\}\\lVert W\_\{2\}\-W\_\{1\}\\rVert\\Big\[\\sum\_\{k=\\ell\}^\{L\}\\sum\_\{i=k\-\\ell\}^\{2k\-1\-\\ell\}\\lVert W\_\{2\}\\rVert^\{i\}\\sum\_\{j=L\-k\}^\{L\-i\+k\-\\ell\-1\}\\lVert W\_\{1\}\\rVert^\{j\}\+\\sum\_\{k=0\}^\{L\-\\ell\-1\}\\lVert W\_\{2\}\\rVert^\{k\}\\lVert W\_\{1\}\\rVert^\{L\-k\-\\ell\-1\}\\Big\]\.
∎

### A\.10Helper Lemmas

In this section, we include some well\-known theorems and identities required for the analysis\.

###### Lemma A\.7\.

Letv∈ℝdv\\in\\mathbb\{R\}^\{d\}and letσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}be a scalar function such that\|σ⁡\(x\)\|≤c0\+c1​\|x\|\|\\sigma\(x\)\|\\leq c\_\{0\}\+c\_\{1\}\|x\|for allx∈ℝx\\in\\mathbb\{R\}, wherec0,c1≥0c\_\{0\},c\_\{1\}\\geq 0\. Ifσ⁡\(v\)\\sigma\(v\)denotes the element\-wise application ofσ\\sigmatovv, then the Euclidean \(L2L\_\{2\}\) norm ofσ⁡\(v\)\\sigma\(v\)satisfies the inequality:

∥σ⁡\(v\)∥2≤c0​d\+c1​∥v∥2\\lVert\\sigma\(v\)\\rVert\_\{2\}\\leq c\_\{0\}\\sqrt\{d\}\+c\_\{1\}\\lVert v\\rVert\_\{2\}

###### Proof\.

We can show this using the triangle inequality\.

∥σ⁡\(v\)∥=∑i=1dσ​\(vi\)2≤∑i=1d\(c0\+c1​\|vi\|\)2≤∑i=1dc02\+∑i=1dc12​\|vi\|2≤c0​d\+c1​∥v∥2\.\\displaystyle\\lVert\\sigma\(v\)\\rVert=\\sqrt\{\\sum\_\{i=1\}^\{d\}\\sigma\(v\_\{i\}\)^\{2\}\}\\leq\\sqrt\{\\sum\_\{i=1\}^\{d\}\(c\_\{0\}\+c\_\{1\}\|v\_\{i\}\|\)^\{2\}\}\\leq\\sqrt\{\\sum\_\{i=1\}^\{d\}c\_\{0\}^\{2\}\}\+\\sqrt\{\\sum\_\{i=1\}^\{d\}c\_\{1\}^\{2\}\|v\_\{i\}\|^\{2\}\}\\leq c\_\{0\}\\sqrt\{d\}\+c\_\{1\}\\lVert v\\rVert\_\{2\}\.∎

###### Lemma A\.8\.

\(Binomial theorem\) For any nonnegative integernnand scalarsx,yx,ywe have

\(x\+y\)n=∑k=0n\(nk\)​xk​yn−k\.\(x\+y\)^\{n\}=\\sum\_\{k=0\}^\{n\}\\binom\{n\}\{k\}x^\{k\}y^\{n\-k\}\.

###### Lemma A\.9\.

For vectorsa∈ℝd1a\\in\\mathbb\{R\}^\{d\_\{1\}\},b∈ℝd2b\\in\\mathbb\{R\}^\{d\_\{2\}\}and matrixM∈ℝd1×d2M\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{2\}\}, we have

⟨a,M​b⟩=⟨MT​a,b⟩\.\\langle a,Mb\\rangle=\\langle M^\{T\}a,b\\rangle\.

###### Proof\.

⟨a,M​b⟩\\displaystyle\\langle a,Mb\\rangle=bT​MT​a=⟨b,MT​a⟩\.\\displaystyle=b^\{T\}M^\{T\}a=\\langle b,M^\{T\}a\\rangle\.∎

###### Lemma A\.10\.

For vectorsa,b,c∈ℝda,b,c\\in\\mathbb\{R\}^\{d\}, we have

⟨a⊙b,c⟩=⟨b,a⊙c⟩\.\\langle a\\odot b,c\\rangle=\\langle b,a\\odot c\\rangle\.

###### Proof\.

Denoted​i​a​g​\(a\)diag\(a\)as the diagonald×dd\\times dmatrix withaaon the diagonal\. Then

a⊙b=\\displaystyle a\\odot b=d​i​a​g​\(a\)​b,\\displaystyle diag\(a\)b,⟨a⊙b,c⟩=\\displaystyle\\langle a\\odot b,c\\rangle=⟨d​i​a​g​\(a\)​b,c⟩,\\displaystyle\\langle diag\(a\)b,c\\rangle,=\\displaystyle=⟨b,d​i​a​g​\(a\)T​c⟩,\\displaystyle\\langle b,diag\(a\)^\{T\}c\\rangle,=\\displaystyle=⟨b,d​i​a​g​\(a\)​c⟩,\\displaystyle\\langle b,diag\(a\)c\\rangle,=\\displaystyle=⟨b,a⊙c⟩\.\\displaystyle\\langle b,a\\odot c\\rangle\.∎

###### Lemma A\.11\.

\(Lp norms\) For a sequence of scalarsa1,\.\.,ana\_\{1\},\.\.,a\_\{n\}, we have forp≥2p\\geq 2,

\(∑i=1n\|ai\|p\)1/p≤\(∑i=1n\|ai\|2\)1/2≤∑i=1n\|ai\|≤n​\(∑i=1n\|ai\|2\)1/2\.\\big\(\\sum\_\{i=1\}^\{n\}\|a\_\{i\}\|^\{p\}\\big\)^\{1/p\}\\leq\\big\(\\sum\_\{i=1\}^\{n\}\|a\_\{i\}\|^\{2\}\\big\)^\{1/2\}\\leq\\sum\_\{i=1\}^\{n\}\|a\_\{i\}\|\\leq\\sqrt\{n\}\\big\(\\sum\_\{i=1\}^\{n\}\|a\_\{i\}\|^\{2\}\\big\)^\{1/2\}\.

###### Lemma A\.12\.

LetEℓE\_\{\\ell\}andEℓ∗E\_\{\\ell\}^\{\*\}be defined as in \([4](https://arxiv.org/html/2608.11479#S3.E4)\) and \([5](https://arxiv.org/html/2608.11479#S3.E5)\)\. For allW∈ℝD2×D1W\\in\\mathbb\{R\}^\{D\_\{2\}\\times D\_\{1\}\}andM∈ℝdℓ×dℓ−1M\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times d\_\{\\ell\-1\}\}, we have

∥Eℓ​\[W\]∥≤∥W∥,\\displaystyle\\lVert E\_\{\\ell\}\[W\]\\rVert\\leq\\lVert W\\rVert,∥Eℓ∗​\[M\]∥≤∥M∥\.\\displaystyle\\lVert E\_\{\\ell\}^\{\*\}\[M\]\\rVert\\leq\\lVert M\\rVert\.

###### Lemma A\.13\.

Suppose a scalar functionσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}is Lipschitz continuous with constantcc, such that for anyw1,w2∈ℝw\_\{1\},w\_\{2\}\\in\\mathbb\{R\}, we have

\|σ⁡\(w1\)−σ⁡\(w2\)\|≤c​\|w1−w2\|\.\|\\sigma\(w\_\{1\}\)\-\\sigma\(w\_\{2\}\)\|\\leq c\|w\_\{1\}\-w\_\{2\}\|\.Then the elementwise extension ofσ\\sigmato matrices is also Lipschitz continuous with the same constant, such that forW,W′∈ℝd1×d2W,W^\{\\prime\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{2\}\}, we have

∥σ⁡\(W\)−σ⁡\(W′\)∥F≤c​∥W−W′∥F\.\\lVert\\sigma\(W\)\-\\sigma\(W^\{\\prime\}\)\\rVert\_\{F\}\\leq c\\lVert W\-W^\{\\prime\}\\rVert\_\{F\}\.

###### Proof\.

∥σ⁡\(W\)−σ⁡\(W′\)∥F=\\displaystyle\\lVert\\sigma\(W\)\-\\sigma\(W^\{\\prime\}\)\\rVert\_\{F\}=∑i,j\(σ⁡\(Wi​j\)−σ⁡\(Wi​j′\)\)2\\displaystyle\\sqrt\{\\sum\_\{i,j\}\(\\sigma\(W\_\{ij\}\)\-\\sigma\(W^\{\\prime\}\_\{ij\}\)\)^\{2\}\}≤\\displaystyle\\leq∑i,jc2​\(Wi​j−Wi​j′\)2\\displaystyle\\sqrt\{\\sum\_\{i,j\}c^\{2\}\(W\_\{ij\}\-W^\{\\prime\}\_\{ij\}\)^\{2\}\}=\\displaystyle=c​∥W−W′∥F\.\\displaystyle c\\lVert W\-W^\{\\prime\}\\rVert\_\{F\}\.∎

###### Lemma A\.14\.

LetA,B∈ℝm×nA,B\\in\\mathbb\{R\}^\{m\\times n\}be real\-valued matrices, and letσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}be an element\-wise function such that\|σ⁡\(x\)\|≤c0\+c1​\|x\|\|\\sigma\(x\)\|\\leq c\_\{0\}\+c\_\{1\}\|x\|for constantsc0,c1≥0c\_\{0\},c\_\{1\}\\geq 0and for allx∈ℝx\\in\\mathbb\{R\}\. Then we have

‖σ⁡\(A\)⊙B‖F≤\(c0\+c1​‖A‖F\)​‖B‖F\\\|\\sigma\(A\)\\odot B\\\|\_\{F\}\\leq\(c\_\{0\}\+c\_\{1\}\\\|A\\\|\_\{F\}\)\\\|B\\\|\_\{F\}

###### Proof\.

We have

‖σ⁡\(A\)⊙B‖F=∑i=1m∑j=1n\|\(σ⁡\(A\)⊙B\)i​j\|2\\displaystyle\\\|\\sigma\(A\)\\odot B\\\|\_\{F\}=\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}\|\(\\sigma\(A\)\\odot B\)\_\{ij\}\|^\{2\}\}=∑i=1m∑j=1n\|σ⁡\(Ai​j\)​Bi​j\|2\\displaystyle=\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}\|\\sigma\(A\_\{ij\}\)B\_\{ij\}\|^\{2\}\}≤\\displaystyle\\leq∑i=1m∑j=1n\(\(c0\+c1​\|Ai​j\|\)​\|Bi​j\|\)2\\displaystyle\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}\(\(c\_\{0\}\+c\_\{1\}\|A\_\{ij\}\|\)\|B\_\{ij\}\|\)^\{2\}\}≤\\displaystyle\\leq∑i=1m∑j=1n\(c0​\|Bi​j\|\+c1​\|Ai​j\|​\|Bi​j\|\)2\\displaystyle\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}\(c\_\{0\}\|B\_\{ij\}\|\+c\_\{1\}\|A\_\{ij\}\|\|B\_\{ij\}\|\)^\{2\}\}≤\\displaystyle\\leq∑i=1m∑j=1nc02​\|Bi​j\|2\+∑i=1m∑j=1nc12​\|Ai​j\|2​\|Bi​j\|2\\displaystyle\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}c\_\{0\}^\{2\}\|B\_\{ij\}\|^\{2\}\}\+\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}c\_\{1\}^\{2\}\|A\_\{ij\}\|^\{2\}\|B\_\{ij\}\|^\{2\}\}≤\\displaystyle\\leqc0​∥B∥\+c1​∥A∥​∥B∥\.\\displaystyle c\_\{0\}\\lVert B\\rVert\+c\_\{1\}\\lVert A\\rVert\\lVert B\\rVert\.
∎

###### Lemma A\.15\.

For a nonnegative valueaa, we have

\(∑i=0nai\)2≤\(n\+1\)​∑i=0na2​i≤\(n\+1\)​∑i=02​nai\.\(\\sum\_\{i=0\}^\{n\}a^\{i\}\)^\{2\}\\leq\(n\+1\)\\sum\_\{i=0\}^\{n\}a^\{2i\}\\leq\(n\+1\)\\sum\_\{i=0\}^\{2n\}a^\{i\}\.

###### Proof\.

By the Cauchy\-Schwarz inequality, we have

\(∑i=0nai\)2=\\displaystyle\(\\sum\_\{i=0\}^\{n\}a^\{i\}\)^\{2\}=\(∑i=0n1⋅ai\)2≤\(∑i=0n12\)​\(∑i=0n\(ai\)2\)=\(n\+1\)​∑i=0na2​i\\displaystyle\(\\sum\_\{i=0\}^\{n\}1\\cdot a^\{i\}\)^\{2\}\\leq\(\\sum\_\{i=0\}^\{n\}1^\{2\}\)\(\\sum\_\{i=0\}^\{n\}\(a^\{i\}\)^\{2\}\)=\(n\+1\)\\sum\_\{i=0\}^\{n\}a^\{2i\}∎