Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway

# Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
Source: [https://arxiv.org/html/2606.05219](https://arxiv.org/html/2606.05219)
###### Abstract

Recent analyses of multi\-pathway Deep Linear Networks use Gradient Flow to predict a “winner\-takes\-all” specialization in which path symmetry breaks and each feature concentrates in a single pathway\. In this work, we show that discrete Gradient Descent \(GD\) with a large step size tells a different story\. We prove that single\-path solutions are sharp minima, whereas distributing signals across pathways reduces sharpness by a factor that decreases with both the number of pathways and depth\. Consequently, while early training reproduces the depth\-driven symmetry breaking predicted by GF, oscillations at the Edge of Stability subsequently override this tendency and drive the network into a re\-balancing phase, where signals redistribute across pathways\. Together, these results clarify how depth shapes pathway competition and explain why large\-step GD favors shared representations rather than persistent single\-pathway dominance\.

Machine Learning, ICML

## 1Introduction

Understanding the training dynamics of Deep Neural Networks remains a central challenge in deep learning theory\. Although deep architectures are highly expressive, how optimization algorithms select specific solutions among many global minima—theimplicit biasof the optimizer—remains poorly understood\(Zhanget al\.,[2017](https://arxiv.org/html/2606.05219#bib.bib1); Neyshaburet al\.,[2017](https://arxiv.org/html/2606.05219#bib.bib3); Gunasekaret al\.,[2017](https://arxiv.org/html/2606.05219#bib.bib4); Soudryet al\.,[2018](https://arxiv.org/html/2606.05219#bib.bib5)\)\. To characterize this bias, a growing body of work analyzes Deep Linear Networks \(DLNs\)\(Saxeet al\.,[2014](https://arxiv.org/html/2606.05219#bib.bib32); Aroraet al\.,[2018](https://arxiv.org/html/2606.05219#bib.bib8); Lampinen and Ganguli,[2019](https://arxiv.org/html/2606.05219#bib.bib25); Shiet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib35)\), under the Gradient Flow \(GF\) approximation, which assumes an infinitesimally small learning rate\.

Recently,Shiet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib35)\)utilized the GF to argue that multi\-pathway DLNs undergo “symmetry breaking,” converging to a sparse “winner\-takes\-all” solution where parallel over\-parameterization becomes redundant\. However, this continuous\-time analysis overlooks the discrete dynamics of Gradient Descent \(GD\) with large learning rates, which characterize practical training at the Edge of Stability\(Cohenet al\.,[2021](https://arxiv.org/html/2606.05219#bib.bib16)\)\. Given that GD actively interacts with the loss curvature to avoid sharp minima, it remains a critical question whether the symmetry\-breaking phenomenon persists under GD with large learning rates\.

In this work, we demonstrate that the answer is no: while GF predicts symmetry breaking, the discreteness of GD induces a fundamentally different phenomenon we termpathway re\-balancing\. By analyzing the geometry of the loss landscape, we prove that sparse, single\-path solutions correspond to sharp minima, whereas solutions balanced across multiple pathways are significantly flatter\. Consequently, when the learning rate is sufficiently large, the implicit regularization of GD drives the network away from unstable, sparse configurations toward stable, balanced configurations\.

Our main contributions are summarized as follows:

- •Sharpness Reduction via Parallelism:We theoretically derive the relationship between the number of pathwaysHH, depthLL, and sharpness of global minima\. We prove that balancing signals acrossHHpathways reduces sharpness byH2/L−1H^\{2/L\-1\}\(Theorem[4\.2](https://arxiv.org/html/2606.05219#S4.Thmtheorem2)\)\.
- •Re\-balancing Dynamics at the Edge of Stability:We identify two distinct phases of training under large step sizes\. Initially, the network exhibits symmetry breaking similar to GF\. However, as the dominant path sharpens, the model enters are\-balancingphase\. Here, violent oscillations force the network to flatten its landscape by distributing signals across multiple paths\.
- •Worst\-Case Return Threshold: Based on adeep linear chainmodel, we derive a depth\-dependent upper bound on learning rate that guarantees the trajectory survives violent oscillations\. The ratio of this threshold to the classical stability limit increases with depth, broadening the window for re\-balancing\.

Beyond the multi\-pathway setting, our results suggest that architectural biases predicted under Gradient Flow—a foundation shared by many recent analyses of DLN\-based models—may require re\-examination once discrete dynamics and finite learning rates are accounted for\. This highlights the essential role of discrete optimization in shaping the final network structure and motivates revisiting GF\-based predictions through the lens of GD\.

## 2Related Work

Deep Linear Networks and Multi\-Pathway Dynamics\.Deep linear networks \(DLNs\) serve as a canonical testbed for isolating optimization and representation learning mechanisms\(Baldi and Hornik,[1989](https://arxiv.org/html/2606.05219#bib.bib11); Saxeet al\.,[2014](https://arxiv.org/html/2606.05219#bib.bib32); Aroraet al\.,[2018](https://arxiv.org/html/2606.05219#bib.bib8)\)\. They have been extensively used to study exact training dynamics and emergent phenomena\(Saxeet al\.,[2019](https://arxiv.org/html/2606.05219#bib.bib33); Lampinen and Ganguli,[2019](https://arxiv.org/html/2606.05219#bib.bib25); J Dominéet al\.,[2023](https://arxiv.org/html/2606.05219#bib.bib23); Namet al\.,[2025](https://arxiv.org/html/2606.05219#bib.bib30)\)\. There has been recent attention to multi\-branch architectures\. Notably,Shiet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib35)\)andSaxeet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib34)\)demonstrate that under continuous\-time Gradient Flow \(GF\), parallel pathways exhibit a “winner\-takes\-all” dynamic\. Specifically, they identify that some pathways dominate the feature learning driven either by small initialization asymmetries\(Shiet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib35)\)or by structural asymmetry such as depth or shared representations across pathways\(Saxeet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib34)\)\.

Implicit Bias of GD and Edge of Stability\.In contrast to GF, discrete\-time Gradient Descent \(GD\) with large learning rates induces distinct dynamical regimes, often operating at the “Edge of Stability” \(EoS\)\(Cohenet al\.,[2021](https://arxiv.org/html/2606.05219#bib.bib16); Wuet al\.,[2018](https://arxiv.org/html/2606.05219#bib.bib38)\)\. Theoretical and empirical studies suggest that instability in this regime acts as an implicit regularizer, driving models toward flatter minima\(Damianet al\.,[2023](https://arxiv.org/html/2606.05219#bib.bib18)\)or inducing transient “catapult” phases that favor generalization\(Lewkowyczet al\.,[2020](https://arxiv.org/html/2606.05219#bib.bib26); Zhuet al\.,[2024](https://arxiv.org/html/2606.05219#bib.bib43)\)\. Recent works have formalized these discrete dynamics through mechanisms like self\-stabilization\(Damianet al\.,[2023](https://arxiv.org/html/2606.05219#bib.bib18)\)and central flows\(Cohenet al\.,[2025](https://arxiv.org/html/2606.05219#bib.bib17)\), highlighting how GD avoids the sharpest directions in the landscape\.

GD in Linear Models\.Recent research has begun to bridge these insights specifically within linear models\.Marion and Chizat \([2024](https://arxiv.org/html/2606.05219#bib.bib29)\)andEvenet al\.\([2023](https://arxiv.org/html/2606.05219#bib.bib19)\)show that GD in linear networks explicitly favors flat minima, differing from the implicit bias of GF\. Most relevant to our work,Ghoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)analyze deep matrix factorization beyond the stability threshold, identifying oscillatory behaviors and convergence properties that violate GF predictions\. Our work extends these findings to the multi\-pathway setting ofShiet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib35)\), demonstrating that the stability constraints of large\-step GD override the symmetry\-breaking tendency of GF, forcing a*re\-balancing*of features across pathways to reduce sharpness\.

## 3Problem Setup

We study a deep linear network withHHparallel pathways indexed byh∈\[H\]=\{1,⋯,H\}h\\in\[H\]=\\\{1,\\cdots,H\\\}\. Pathwayhhhas depthLhL\_\{h\}and weight matricesWh​ℓ∈ℝd×dW\_\{h\\ell\}\\in\\mathbb\{R\}^\{d\\times d\}for layersℓ∈\[Lh\]\\ell\\in\[L\_\{h\}\]\. The architecture follows the multi\-pathway formulation ofShiet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib35)\)\. The end\-to\-end map of pathwayhhis the ordered product

Ωh:=Wh​Lh​Wh​Lh−1​⋯​Wh​1\.\\Omega\_\{h\}\\;:=\\;W\_\{hL\_\{h\}\}W\_\{hL\_\{h\}\-1\}\\cdots W\_\{h1\}\.\(1\)The overall input–output map of the network is the sum of the pathway maps:

M:=∑h=1HΩh\.M\\;:=\\;\\sum\\nolimits\_\{h=1\}^\{H\}\\Omega\_\{h\}\.\(2\)As in prior theoretical work on deep matrix factorization\(Ghoshet al\.,[2025](https://arxiv.org/html/2606.05219#bib.bib21)\), we focus on the square caseWh​ℓ∈ℝd×dW\_\{h\\ell\}\\in\\mathbb\{R\}^\{d\\times d\}for theoretical analysis\.

### 3\.1Optimization Objective

LetΘ=\{Wh​ℓ\}h∈\[H\],ℓ∈\[Lh\]\\Theta=\\\{W\_\{h\\ell\}\\\}\_\{h\\in\[H\],\\ell\\in\[L\_\{h\}\]\}denote all parameters\. We fit a target matrixM⋆∈ℝd×dM\_\{\\star\}\\in\\mathbb\{R\}^\{d\\times d\}by minimizing the squared Frobenius loss

ℒ​\(Θ\)=12​‖M−M⋆‖F2\.\\mathcal\{L\}\(\\Theta\)\\;=\\;\\frac\{1\}\{2\}\\,\\\|M\-M\_\{\\star\}\\\|\_\{F\}^\{2\}\.\(3\)We consider discrete\-time gradient descent updates

Wh​ℓ​\(t\+1\)=Wh​ℓ​\(t\)−η​∇Wh​ℓℒ​\(Θ​\(t\)\),W\_\{h\\ell\}\(t\+1\)\\;=\\;W\_\{h\\ell\}\(t\)\\;\-\\;\\eta\\,\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}\(\\Theta\(t\)\),\(4\)with stepsizeη\>0\\eta\>0\.

### 3\.2Target\-Aligned Parametrization and the SVS Set

Let the target admit the SVD

M⋆=U⋆​Σ⋆​V⋆⊤,Σ⋆=diag​\(σ⋆1,⋯,σ⋆d\)\.M\_\{\\star\}=U\_\{\\star\}\\Sigma\_\{\\star\}V\_\{\\star\}^\{\\top\},\\qquad\\Sigma\_\{\\star\}=\\mathrm\{diag\}\(\\sigma\_\{\\star 1\},\\cdots,\\sigma\_\{\\star d\}\)\.\(5\)A standard simplification in the analysis of deep linear networks aligns each pathway’s end\-to\-end singular vectors with those of the target, so the dynamics decouple across modes\(Saxeet al\.,[2014](https://arxiv.org/html/2606.05219#bib.bib32)\)\. We adopt the multi\-pathway extension of this parametrization, the*singular vector stationary*\(SVS\) set ofGhoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)\. The SVS set consists of parameter configurationsΘ\\Thetafor which each layer factors as

Wh​ℓ=Qh​ℓ\+1​Σh​ℓ​Qh​ℓ⊤,Qh​1=V⋆,Qh​Lh\+1=U⋆,W\_\{h\\ell\}=Q\_\{h\\ell\+1\}\\Sigma\_\{h\\ell\}Q\_\{h\\ell\}^\{\\top\},\\,\\,Q\_\{h1\}=V\_\{\\star\},\\,\\,Q\_\{hL\_\{h\}\+1\}=U\_\{\\star\},\(6\)with fixed orthogonal matrices\{Qh,ℓ\}\\\{Q\_\{h,\\ell\}\\\}and diagonalΣh​ℓ\\Sigma\_\{h\\ell\}\. Under \([6](https://arxiv.org/html/2606.05219#S3.E6)\), adjacent orthogonals cancel in the product, so every pathway map is diagonal in the target basis:

Ωh=U⋆​\(∏ℓ=1LhΣh​ℓ\)​V⋆⊤\.\\Omega\_\{h\}\\;=\\;U\_\{\\star\}\\Big\(\\prod\\nolimits\_\{\\ell=1\}^\{L\_\{h\}\}\\Sigma\_\{h\\ell\}\\Big\)V\_\{\\star\}^\{\\top\}\.\(7\)Gradient descent preserves the SVS set: ifΘ​\(t\)\\Theta\(t\)lies in the set, so doesΘ​\(t\+1\)\\Theta\(t\+1\)\(Appendix[E\.1](https://arxiv.org/html/2606.05219#A5.SS1)\)\. Analogous target\-aligned reductions appear inSaxeet al\.\([2014](https://arxiv.org/html/2606.05219#bib.bib32),[2022](https://arxiv.org/html/2606.05219#bib.bib34)\); Aroraet al\.\([2019](https://arxiv.org/html/2606.05219#bib.bib9)\); Gidelet al\.\([2019](https://arxiv.org/html/2606.05219#bib.bib22)\); Varreet al\.\([2023](https://arxiv.org/html/2606.05219#bib.bib36)\); Chouet al\.\([2024](https://arxiv.org/html/2606.05219#bib.bib15)\); Kwonet al\.\([2024](https://arxiv.org/html/2606.05219#bib.bib24)\)\.

We index modes by the target singular vector pairs\(u⋆i,v⋆i\)\(u\_\{\\star i\},v\_\{\\star i\}\), theii\-th columns ofU⋆U\_\{\\star\}andV⋆V\_\{\\star\}, and track the scalar*mode coefficients*

σh​i​\(t\)\\displaystyle\\sigma\_\{hi\}\(t\):=u⋆i⊤​Ωh​\(t\)​v⋆i,\\displaystyle\\;:=\\;u\_\{\\star i\}^\{\\top\}\\Omega\_\{h\}\(t\)\\,v\_\{\\star i\},\(8\)σi​\(t\)\\displaystyle\\sigma\_\{i\}\(t\):=u⋆i⊤​M​\(t\)​v⋆i=∑h=1Hσh​i​\(t\)\.\\displaystyle\\;:=\\;u\_\{\\star i\}^\{\\top\}M\(t\)\\,v\_\{\\star i\}=\\sum\\nolimits\_\{h=1\}^\{H\}\\sigma\_\{hi\}\(t\)\.\(9\)On the SVS set,σh​i​\(t\)\\sigma\_\{hi\}\(t\)equals theii\-th singular value ofΩh​\(t\)\\Omega\_\{h\}\(t\), and the learning dynamics decouple across modesi∈\[d\]i\\in\[d\]\.

Mode\-wise decoupling on the SVS set decomposes the loss into independent scalar mode losses:

ℒ​\(Θ\)=∑i=1dℒi,ℒi=12​\(∑h=1Hσh​i−σ⋆i\)2\.\\mathcal\{L\}\(\\Theta\)=\\sum\_\{i=1\}^\{d\}\\mathcal\{L\}\_\{i\},\\quad\\mathcal\{L\}\_\{i\}=\\frac\{1\}\{2\}\\Big\(\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\}\\Big\)^\{2\}\.\(10\)

### 3\.3Depth\-Balanced Initialization

We initialize each pathway with

Wh​ℓ​\(0\)=αh1/Lh​Id,W\_\{h\\ell\}\(0\)\\;=\\;\\alpha\_\{h\}^\{1/L\_\{h\}\}\\,I\_\{d\},\(11\)whereαh\>0\\alpha\_\{h\}\>0are small pathway\-dependent scales\. This initialization placesΘ​\(0\)\\Theta\(0\)in the SVS set and equalizes the layerwise singular values within each pathway\. It therefore lies on the*depth\-balanced manifold*, defined by

σh​i\(ℓ\)=σh​i1/Lh,∀h∈\[H\],ℓ∈\[Lh\],i∈\[d\],\\sigma\_\{hi\}^\{\(\\ell\)\}\\;=\\;\\sigma\_\{hi\}^\{1/L\_\{h\}\},\\qquad\\forall\\,h\\in\[H\],\\,\\ell\\in\[L\_\{h\}\],\\,i\\in\[d\],\(12\)whereσh​i\(ℓ\)\\sigma\_\{hi\}^\{\(\\ell\)\}denotes theii\-th singular value of layerℓ\\ellin pathwayhh\. The trajectory remains on this manifold \(and in the SVS set\) throughout training for symmetric targetsM⋆=V⋆​Σ⋆​V⋆⊤M\_\{\\star\}=V\_\{\\star\}\\Sigma\_\{\\star\}V\_\{\\star\}^\{\\top\}\(Proposition[B\.4](https://arxiv.org/html/2606.05219#A2.Thmtheorem4)in Appendix[E\.1](https://arxiv.org/html/2606.05219#A5.SS1)\)\. The scalesαh\\alpha\_\{h\}may differ across pathways\. Such tiny initialization asymmetries trigger the symmetry\-breaking dynamics analyzed in Section[5](https://arxiv.org/html/2606.05219#S5)\.

## 4Sharpness Reduction in Multi\-Pathway Deep Linear Models

In this section, we derive the relationship between the number of pathwaysHHand the loss sharpness at global minima in depth\-balanced manifold\. We assume homogeneous depthLh=LL\_\{h\}=Lin this section\. We demonstrate that distributing a target feature across multiple pathways significantly reduces sharpness\. Detailed proofs for all results are provided in the Appendix[C](https://arxiv.org/html/2606.05219#A3)\.

### 4\.1Sharpness Analysis

The sharpness of the loss is determined by the maximum eigenvalue of the Hessian∇2ℒ​\(Θ\)\\nabla^\{2\}\\mathcal\{L\}\(\\Theta\)at a global minimum whereM=M⋆M=M\_\{\\star\}\. On the SVS set \(Section[3\.2](https://arxiv.org/html/2606.05219#S3.SS2)\), the Hessian decomposes across modes, so we can analyze the Hessian contribution from each modei∈\[d\]i\\in\[d\]independently\.

###### Proposition 4\.1\(Mode\-Wise Dominant Eigenvalue\)\.

At any global minimum within the SVS set restricted to the depth\-balanced manifold, the Hessian contribution from modeiihas a single nonzero eigenvalue given by:

λi=L​∑h=1Hσh​i2−2L\.\\lambda\_\{i\}=L\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}^\{2\-\\frac\{2\}\{L\}\}\.\(13\)

Since the Hessian on this manifold is block\-diagonal across modes with rank\-one blocks, the loss sharpness equalsλmax=maxi∈\[d\]⁡λi\\lambda\_\{\\max\}=\\max\_\{i\\in\[d\]\}\\lambda\_\{i\}\.

This result highlights that sharpness depends on how the total singular valueσ⋆i\\sigma\_\{\\star i\}is distributed among theHHpathways\. We now state the main theorem regarding the optimal distribution of these values\.

η<2/S1\\quad\\eta<2/S\_\{1\}

η\>2/S1\\quad\\eta\>2/S\_\{1\}

η=2/λ1min\\quad\\eta=2/\\lambda\_\{1\}^\{\\min\}

![Refer to caption](https://arxiv.org/html/2606.05219v1/x1.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x2.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x3.png)

Figure 1:Training dynamics of GD with multi\-path DLNs \(L=20L=20,H=2H=2,σ⋆1=1\\sigma\_\{\\star 1\}=1\) across different learning ratesη\\eta\. The evolution of pathway singular values is shown in blue \(σ11,h=1\\sigma\_\{11\},h=1\), light blue \(σ21,h=2\\sigma\_\{21\},h=2\), and gray \(σ1=σ11\+σ21\\sigma\_\{1\}=\\sigma\_\{11\}\+\\sigma\_\{21\}\)\. While smallη\\etaleads to a single\-path solution, GD with a larger learning rate \(η\>2/S1\\eta\>2/S\_\{1\}\) drives the system toward a more balanced configuration where singular values are distributed across pathways\. Consequently, the sharpnessλmax\\lambda\_\{\\max\}\(purple dots\) is suppressed and converges to2/η2/\\eta\.###### Theorem 4\.2\(Sharpness Reduction via Parallelism\)\.

AssumeL\>2L\>2\. Among all global minima satisfying the constraint∑h=1Hσh​i=σ⋆i\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}=\\sigma\_\{\\star i\}, the mode\-wise sharpnessλi\\lambda\_\{i\}is minimized when the target singular value is distributed equally across allHHpathways:

σ1​i=⋯=σH​i=σ⋆iH\.\\sigma\_\{1i\}=\\cdots=\\sigma\_\{Hi\}=\\frac\{\\sigma\_\{\\star i\}\}\{H\}\.\(14\)The minimal eigenvalue on the depth\-balanced manifold is given by:

λimin=L​H2L−1​σ⋆i2−2L\.\\lambda\_\{i\}^\{\\min\}=L\\,H^\{\\frac\{2\}\{L\}\-1\}\\,\\sigma\_\{\\star i\}^\{2\-\\frac\{2\}\{L\}\}\.\(15\)

Theorem[4\.2](https://arxiv.org/html/2606.05219#S4.Thmtheorem2)implies a significant reduction in sharpness compared to a single\-pathway model\. Comparing the balanced multi\-pathway sharpness to that of a single pathway \(λimin​\(H=1\)\\lambda\_\{i\}^\{\\min\}\(H=1\)\), we obtain the reduction factor:

λiminλimin​\(H=1\)=H2L−1<1\.\\frac\{\\lambda\_\{i\}^\{\\min\}\}\{\\lambda\_\{i\}^\{\\min\}\(H=1\)\}=H^\{\\frac\{2\}\{L\}\-1\}<1\.\(16\)Since2/L−1<02/L\-1<0forL\>2L\>2, increasing eitherHHorLLstrictly decreases this ratio\.

Under the conventional orderingσ⋆1≥σ⋆2≥⋯≥σ⋆d\\sigma\_\{\\star 1\}\\geq\\sigma\_\{\\star 2\}\\geq\\cdots\\geq\\sigma\_\{\\star d\}, the largest target mode dominates the sharpness:maxi⁡λimin=λ1min\\max\_\{i\}\\lambda\_\{i\}^\{\\min\}=\\lambda\_\{1\}^\{\\min\}\. The reduction factorH2/L−1H^\{2/L\-1\}applies every mode, so re\-balancing benefits any modeiiwhose single\-path sharpnessSi=L​σ⋆i2−2/LS\_\{i\}=L\\sigma\_\{\\star i\}^\{2\-2/L\}exceeds the stability threshold2/η2/\\eta\(Section[5](https://arxiv.org/html/2606.05219#S5)\)\.

## 5Dynamics of GD Beyond Classical Stability

We now study how multi\-pathway DLNs learn whenη\\etaexceeds the stability threshold of the sharpest minima\. The dynamics reflect two opposing forces: the architectural bias of GF toward “winner\-takes\-all” specialization, and the implicit bias of large\-step GD toward flat minima\. Small learning rates let the former dominate and reproduce GF’s symmetry breaking; large learning rates let the latter override it, driving the network into a distinctre\-balancingphase\.

η<2/S1\\quad\\eta<2/S\_\{1\}η\>2/S1\\quad\\eta\>2/S\_\{1\}η=2/λ1min\\quad\\eta=2/\\lambda\_\{1\}^\{\\min\}
![Refer to caption](https://arxiv.org/html/2606.05219v1/x4.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x5.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x6.png)

Figure 2:Trajectories of\(σ11,σ21\)\(\\sigma\_\{11\},\\sigma\_\{21\}\)for different learning ratesη\\eta\. In the stable regime \(left\), the dominant pathway suppresses others\. Beyond the stability threshold \(middle, right\), the pathways bifurcate into the more balanced minimum \(λmax<S1\\lambda\_\{\\max\}<S\_\{1\}\)\.### 5\.1Two Stages of Training with Large Learning Rate

We focus on modei=1i=1with homogeneous depthLL, whereSi=L​σ⋆i2−2/LS\_\{i\}=L\\sigma\_\{\\star i\}^\{2\-2/L\}is the sharpness of the “winner\-takes\-all” solution carrying the entire target in a single pathway \(σ11=σ⋆1\\sigma\_\{11\}=\\sigma\_\{\\star 1\}andσh​1=0\\sigma\_\{h1\}=0forh≠1h\\neq 1\)\. To trigger symmetry breaking, we initialize each pathway with a slightly different scale \(α1\>α2\>⋯\>αH\\alpha\_\{1\}\>\\alpha\_\{2\}\>\\cdots\>\\alpha\_\{H\}\)\.

Small Learning Rate Regime \(η<2/S1\\eta<2/S\_\{1\}\)\. When the learning rate is small, the discrete GD dynamics closely approximate the continuous GF\. In this regime, small differences in initialization lead to the “winner\-takes\-all” symmetry breaking\(Shiet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib35)\): the relative growth rateσ˙h​i/σ˙k​i=\(σh​i/σk​i\)L−1\\dot\{\\sigma\}\_\{hi\}/\\dot\{\\sigma\}\_\{ki\}=\(\\sigma\_\{hi\}/\\sigma\_\{ki\}\)^\{L\-1\}amplifies infinitesimal initial asymmetries through the depth\-LLexponent, so the pathway with the largest initialization for a given mode grows exponentially faster than the others and suppresses competing gradients\. As shown in the first row of Figure[1](https://arxiv.org/html/2606.05219#S4.F1), the network converges to a dominant\-path solution \(σ11=σ⋆1\\sigma\_\{11\}=\\sigma\_\{\\star 1\}\) while other pathways stay near zero, yielding a sharp minimum \(λmax≈S1\\lambda\_\{\\max\}\\approx S\_\{1\}\)\.

Large Learning Rate Regime \(η\>2/S1\\eta\>2/S\_\{1\}\)\. Training with a learning rate large enough to destabilize the single\-path solution exhibits two distinct phases:

- •Phase 1: Symmetry Breaking\.Early in training, the singular values are small, so local sharpness stays below2/η2/\\eta\. The dynamics track GF, and the same depth\-LLamplification mechanism that drives the small\-LR regime concentrates the signal in the dominant pathway\.
- •Phase 2: Re\-balancing \(Drift via Oscillations\)\.As theσ11\\sigma\_\{11\}approachesσ⋆1\\sigma\_\{\\star 1\}, the sharpness rises nearS1S\_\{1\}and eventually exceeds2/η2/\\eta\. GD can no longer settle into this sharp minimum and enters the Edge of Stability regime, where residuals oscillate around the loss surface rather than decay\. These oscillations are not noise: they drive a systematic transfer of signal from the dominant pathway to the others, reducing sharpness toward the stability threshold\.

Appendix[E](https://arxiv.org/html/2606.05219#A5)identifies the mechanism driving this phase\. On the SVS depth\-balanced manifold, the dynamics reduce to a scalar recursion per mode and pathway\. Under gradient flow,a conserved quantity fixes the leaf coordinatezzthat measures pathway imbalance, and the sharpness at any zero\-loss point grows quadratically in‖z‖\\\|z\\\|\. Large\-step GD at the Edge of Stability breaks the conservation law: under local self\-stabilization, the residual alternates in sign, and‖z‖\\\|z\\\|decays over pairs of steps\. The oscillations thus act as a directed drift toward flatter, more balanced minima\.

This drift acts only while the sharpness exceeds2/η2/\\eta, so the trajectory re\-balances until theσh​i\\sigma\_\{hi\}are distributed enough to enter the stability window

2/S1<η<2/λ1min\.2/S\_\{1\}<\\eta<2/\\lambda\_\{1\}^\{\\min\}\.By Theorem[4\.2](https://arxiv.org/html/2606.05219#S4.Thmtheorem2), this balanced configuration is a significantly flatter minimum,λ1min=H2/L−1​S1\\lambda\_\{1\}^\{\\min\}=H^\{2/L\-1\}S\_\{1\}\. Atη=2/λ1min\\eta=2/\\lambda^\{\\min\}\_\{1\}the stopping boundary coincides with the fully balanced minimum; at intermediate rates the trajectory halts at a partially balanced configuration\. Instability at the Edge of Stability therefore acts not as noise but as a directed force that selects flatter, pathway\-balanced minima \(Figures[2](https://arxiv.org/html/2606.05219#S5.F2)\)\.

![Refer to caption](https://arxiv.org/html/2606.05219v1/x7.png)Figure 3:Symmetry Breaking vs\. Re\-balancing in Nonlinear Networks\.Training dynamics of a two\-pathway MLP \(L=3L=3,H=2H=2, Tanh activation\)\. Training with smallη\\eta\(Top row\) shows symmetry breaking and Training with largerη=2/λ1min\\eta=2/\\lambda\_\{1\}^\{\\min\}\(Bottom row\) shows re\-balancing phase\.Nonlinear Setting\.While our theory assumes linearity, we observe the same transition in multi\-pathway MLPs with Tanh activation \(Figure[3](https://arxiv.org/html/2606.05219#S5.F3)\)\. FollowingShiet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib35)\), we use the canonical basis for inputsxxand initialize the weights from𝒩​\(0,0\.02nin\+nout\)\\mathcal\{N\}\(0,\\frac\{0\.02\}\{n\_\{\\text\{in\}\}\+n\_\{\\text\{out\}\}\}\)\. We monitor learning dynamics via the projectionKh​1=u1⊤​Ωh​v1K\_\{h1\}=u\_\{1\}^\{\\top\}\\Omega\_\{h\}v\_\{1\}\. In the stable regime \(η≈1/S1\\eta\\approx 1/S\_\{1\}\), we recover the “winner\-takes\-all” symmetry breaking of linear networks\. A large learning rate \(η≈2/λ1min\\eta\\approx 2/\\lambda\_\{1\}^\{\\min\}\) instead drives the system toward a flatter, balanced minimum \(K11≈K21K\_\{11\}\\approx K\_\{21\}\)\. suggesting that the re\-balancing mechanism extends to nonlinear architectures\. Figure[7](https://arxiv.org/html/2606.05219#A7.F7)in Appendix[G](https://arxiv.org/html/2606.05219#A7)shows results at additional learning rates; MLPs typically need slightly higher rates than linear networks to fully balance\.

### 5\.2Contrast with Depth\-Wise Balancing

A fundamental distinction exists between the pathway re\-balancing observed in this work and the depth\-wise balancing analyzed in single\-pathway networks\.Ghoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)demonstrate that large learning rates minimize the layer\-wise balancing gap, which is contingent on explicit initialization asymmetries and serves only to rectify unbalanced initial conditions\. In standard Deep Linear Networks \(DLNs\), layers naturally remain aligned throughout Gradient Flow \(GF\) trajectories due to conservation laws \(e\.g\.,Wℓ\+1​Wℓ\+1⊤−Wℓ⊤​Wℓ=constW\_\{\\ell\+1\}W\_\{\\ell\+1\}^\{\\top\}\-W\_\{\\ell\}^\{\\top\}W\_\{\\ell\}=\\text\{const\}\)\. The depth\-wise balancing is only prominent with highly unbalanced initializationsWℓ\+1​Wℓ\+1⊤−Wℓ⊤​Wℓ≫0W\_\{\\ell\+1\}W\_\{\\ell\+1\}^\{\\top\}\-W\_\{\\ell\}^\{\\top\}W\_\{\\ell\}\\gg 0\. However, for balanced or zero initializations \(Wℓ\+1​Wℓ\+1⊤−Wℓ⊤​Wℓ≈0W\_\{\\ell\+1\}W\_\{\\ell\+1\}^\{\\top\}\-W\_\{\\ell\}^\{\\top\}W\_\{\\ell\}\\approx 0\), GF and GD trajectories remain qualitatively indistinguishable regarding their balancing dynamics\.

In contrast, symmetry breaking in multi\-pathway architectures is a spontaneous phenomenon\. As shown byShiet al\.\([2022](https://arxiv.org/html/2606.05219#bib.bib35)\), GF dynamics inherently amplify infinitesimal asymmetries, driving the system toward a dominant\-path solution \(“winner\-takes\-all”\)\. Unlike depth\-wise imbalance, which depends on specific initial conditions, pathway imbalance is the natural attractor of continuous\-time dynamics\. Our results demonstrate that discrete\-time instability actively counteracts this intrinsic bias\. Specifically, it destabilizes the sharp, single\-path minima favored by GF, thereby forcing the network into a balanced configuration that continuous dynamics cannot achieve\.

### 5\.3Heterogeneous Depth between Pathways

![Refer to caption](https://arxiv.org/html/2606.05219v1/x8.png)Figure 4:Trajectories of\(σ11,σ21\)\(\\sigma\_\{11\},\\sigma\_\{21\}\)for heterogeneous depth models with learning rateη∗=2/λ1min\\eta^\{\*\}=2/\\lambda\_\{1\}^\{\\min\}\. See Figure[2](https://arxiv.org/html/2606.05219#S5.F2)together\.So far we assumed homogeneous depthLL, where only initialization differences trigger symmetry breaking\. We now allow heterogeneous depthsLhL\_\{h\}across pathways\. Different depths break symmetry structurally: even under balanced initialization, gradient magnitudes scale with depth, so pathways accelerate at different rates\.

We first extend the sharpness–allocation relationship to heterogeneous depths\.

###### Proposition 5\.1\(Sharpness with Heterogeneous Depth\)\.

Consider a multi\-pathway network where pathwayhhhas depthLhL\_\{h\}\. On the depth\-balanced manifold, the dominant eigenvalue of the Hessian for modeiiis given by:

λi=∑h=1HLh​σh​i2−2Lh\.\\lambda\_\{i\}=\\sum\_\{h=1\}^\{H\}L\_\{h\}\\,\\sigma\_\{hi\}^\{2\-\\frac\{2\}\{L\_\{h\}\}\}\.\(17\)

Unlike the homogeneous case, equal splitting \(σh​i=σ⋆i/H\\sigma\_\{hi\}=\\sigma\_\{\\star i\}/H\) no longer minimizes sharpness\. The Lagrange conditions for minimizingλi\\lambda\_\{i\}subject to∑hσh​i=σ⋆i\\sum\_\{h\}\\sigma\_\{hi\}=\\sigma\_\{\\star i\}give:

2​\(Lh−1\)​σh​i1−2Lh=μ,∀h∈\[H\],2\(L\_\{h\}\-1\)\\sigma\_\{hi\}^\{1\-\\frac\{2\}\{L\_\{h\}\}\}=\\mu,\\quad\\forall h\\in\[H\],\(18\)whereμ\\muis the Lagrange multiplier enforcing the constraint\. The condition equalizes∂λi/∂σh​i\\partial\\lambda\_\{i\}/\\partial\\sigma\_\{hi\}across pathways\. The shallow pathway still carries the largest share ofσ⋆i\\sigma\_\{\\star i\}but the optimum spreads more mass into deeper pathways than the “winner\-takes\-all” solution\. These equations lack a closed form, but we can solve them numerically for any\{Lh\}\\\{L\_\{h\}\\\}to obtain the optimal allocationσh​iopt\\sigma\_\{hi\}^\{\\text\{opt\}\}and minimum sharpnessλimin\\lambda\_\{i\}^\{\\min\}\.

Kinetic bias vs\. stability bias\.Heterogeneous depth provides a stringent test of our theory\. Under GF \(or smallη\\eta\), the shallower pathway learns faster and absorbs the signal\(Saxeet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib34)\), so structural asymmetry alone drives “winner\-takes\-all” toward the shallowest path\. But Proposition 5\.1 shows that concentrating the full signal in a shallow pathway yields a sharper minimum than concentrating it in a deep one, since2−2/Lh2\-2/L\_\{h\}shrinks asLhL\_\{h\}decreases\. GF thus favors shallow paths kinetically, while large\-step GD favors distributed paths for stability—the two forces conflict\.

Training with largeη\\etaresolves this conflict in two phases:

- •Early Phase \(Structural Symmetry Breaking\):Near initialization, the per\-pathway growth rate scales asσh​i\(Lh−1\)/Lh\\sigma\_\{hi\}^\{\(L\_\{h\}\-1\)/L\_\{h\}\}, which is larger for smallerLhL\_\{h\}whenσh​i≪1\\sigma\_\{hi\}\\ll 1\. The shorter pathway \(e\.g\.,L1=3L\_\{1\}=3\) receives larger gradients and grows exponentially faster than the deeper pathway \(e\.g\.,L2=7L\_\{2\}=7\), driving the system toward a shallow\-path\-dominated solution\.
- •Late Phase \(Re\-balancing\):Once the shallow path captures most of the signal, local sharpness climbs past2/η2/\\etaand the path destabilizes\. The induced oscillations \(Section 5\.1\) keep the residual nonzero, which drives the deeper, flatter pathway upward until the configuration reaches the stability window\.

Figure[4](https://arxiv.org/html/2606.05219#S5.F4)shows the dynamics for\(L1,L2\)∈\{3,5,7\}2\(L\_\{1\},L\_\{2\}\)\\in\\\{3,5,7\\\}^\{2\}atη=2/λ1min\\eta=2/\\lambda\_\{1\}^\{\\min\}\(from our numerical solution\)\. We initialize all pathways with the same scale to isolate architectural effects\. The trajectories converge to the numerically predicted equilibrium that equalizes marginal sharpness across pathways\. Large\-step GD therefore overrides structural asymmetry and restores a balanced representation, even when architecture favors a single pathway\. See Figure[9](https://arxiv.org/html/2606.05219#A7.F9)in Appendix[G](https://arxiv.org/html/2606.05219#A7)for results across a wider depth range\(L1,L2\)∈\{3,4,5,6,7\}2\(L\_\{1\},L\_\{2\}\)\\in\\\{3,4,5,6,7\\\}^\{2\}\.

## 6A Worst\-Case Return Threshold from a Deep Linear Chain \(Role of Depth\)

Re\-balancing demands a large step size: to destabilize the sharp single\-path minimum,η\\etamust exceed2/S12/S\_\{1\}, whereS1=L​σ⋆12−2/LS\_\{1\}=L\\sigma\_\{\\star 1\}^\{2\-2/L\}is the dominant mode’s sharpness; pushingη\\etahigher unseats more modes, balancing the toppp\(made precise below\)\. Butη\\etahas a ceiling\. Re\-balancing first passes through a transient single\-path phase, and an overlarge step there drives the dominant pathway’s singular value across zero—a sign flip that breaks the SVS description and aborts re\-balancing\. The failure is worst when the signal concentrates in a single pathway: a*deep linear chain*, from which we derive a depth\-dependent ceilingηWCR\\eta\_\{\\mathrm\{WCR\}\}\. The resulting overshoot\-but\-return window, measured relative to the onset scale, widens with depth, so deeper networks tolerate the large steps required for re\-balancing\.

#### The binding constraint: the transient single pathway\.

During symmetry breaking \(Section[5](https://arxiv.org/html/2606.05219#S5)\), the dominant mode concentrates in one pathway while the others remain near zero\. This dominant pathway is the sharpest component of the landscape and therefore sets the tightest stability constraint\. Hence, a step size large enough to induce re\-balancing can drive it into violent oscillations\. To keep the SVS description valid, the dominant coordinate must remain positive: crossingu⋆1⊤​Ωh​v⋆1≤0u\_\{\\star 1\}^\{\\top\}\\Omega\_\{h\}v\_\{\\star 1\}\\leq 0corresponds to a singular\-vector sign flip across the origin saddle\. Thus, the binding question is whether the dominant coordinate survives its overshoot\.

#### Reduction to a deep linear chain\.

On the SVS depth\-balanced manifold, per\-layer scalarsah​1≥0a\_\{h1\}\\geq 0carry mode11, with end\-to\-end valueσh​1=ah​1L\\sigma\_\{h1\}=a\_\{h1\}^\{L\}\(Appendix[F](https://arxiv.org/html/2606.05219#A6)\)\. When the mode concentrates in a single pathway, the contributions of the other pathways become negligible in the residual, and the dynamics collapse to one scalar—the*deep\-chain map*\.

wt\+1=fη​\(wt\):=wt−η​wtL−1​\(wtL−σ⋆1\),w\_\{t\+1\}=f\_\{\\eta\}\(w\_\{t\}\):=w\_\{t\}\-\\eta\\,w\_\{t\}^\{L\-1\}\\bigl\(w\_\{t\}^\{L\}\-\\sigma\_\{\\star 1\}\\bigr\),\(19\)wherewt≡a11​\(t\)w\_\{t\}\\equiv a\_\{11\}\(t\)with target fixed pointw⋆=σ⋆11/Lw\_\{\\star\}=\\sigma\_\{\\star 1\}^\{1/L\}\. ThewL−1w^\{L\-1\}prefactor makes the update highly sensitive to the current per\-layer scale: once the iterate overshootsw⋆w\_\{\\star\}, the next correction can be large enough to cross the origin\.

#### The worst\-case return threshold\.

We ask that the chain recover from*any*overshoot without crossing the origin\.

###### Definition 6\.1\(Worst\-case return threshold\)\.

Withw⋆=σ⋆11/Lw\_\{\\star\}=\\sigma\_\{\\star 1\}^\{1/L\}andfηf\_\{\\eta\}from \([19](https://arxiv.org/html/2606.05219#S6.E19)\), a stepsizeη\\etais*two\-step return\-safe*if every overshoot from\(0,w⋆\)\(0,w\_\{\\star\}\)returns to the interval\(0,w⋆\)\(0,w\_\{\\star\}\)after one further step:

∀w∈\(0,w⋆\):fη​\(w\)\>w⋆⟹0<fη∘2​\(w\)<w⋆\.\\forall\\,w\\in\(0,w\_\{\\star\}\):\\,f\_\{\\eta\}\(w\)\>w\_\{\\star\}\\ \\Longrightarrow\\ 0<f\_\{\\eta\}^\{\\circ 2\}\(w\)<w\_\{\\star\}\.The*worst\-case return threshold*ηWCR​\(L,σ⋆1\)\\eta\_\{\\mathrm\{WCR\}\}\(L,\\sigma\_\{\\star 1\}\)is the supremum of all return\-safe stepsizes\.

On the relevant overshoot branch, the binding case is the largest overshoot, produced by the maximizer offηf\_\{\\eta\}\(Appendix[F](https://arxiv.org/html/2606.05219#A6)\); if that worst overshoot returns, every smaller one does too\. Figure[5](https://arxiv.org/html/2606.05219#S6.F5)shows the knife\-edge: atη≈0\.97​ηWCR\\eta\\approx 0\.97\\,\\eta\_\{\\mathrm\{WCR\}\}the iterate bounces back into\(0,w⋆\)\(0,w\_\{\\star\}\), while atη≈1\.03​ηWCR\\eta\\approx 1\.03\\,\\eta\_\{\\mathrm\{WCR\}\}it overshoots the origin into the sign\-flip region\.

![Refer to caption](https://arxiv.org/html/2606.05219v1/x9.png)Figure 5:Loss12​\(wL−σ⋆1\)2\\tfrac\{1\}\{2\}\\,\(w^\{L\}\-\\sigma\_\{\\star 1\}\)^\{2\}of the deep linear chain \(L=5L=5,σ⋆1=1\\sigma\_\{\\star 1\}=1;wwis the per\-layer scale, minimizerw⋆=σ⋆11/L=1w\_\{\\star\}=\\sigma\_\{\\star 1\}^\{1/L\}=1\)\. GD follows the map \([19](https://arxiv.org/html/2606.05219#S6.E19)\)\. Fromwtw\_\{t\}, the maximizer offηf\_\{\\eta\}producing the largest overshootwt\+1w\_\{t\+1\}, the second stepwt\+2w\_\{t\+2\}either returns to\(0,w⋆\)\(0,w\_\{\\star\}\)\(blue,η≈0\.97​ηWCR\\eta\\approx 0\.97\\,\\eta\_\{\\mathrm\{WCR\}\}\) or crosses the origin intow<0w<0\(red,η≈1\.03​ηWCR\\eta\\approx 1\.03\\,\\eta\_\{\\mathrm\{WCR\}\}\)\.ηWCR\\eta\_\{\\mathrm\{WCR\}\}is the knife\-edge between the two\.
#### Closed\-form scaling and the role of depth\.

The threshold separates into a curvature scale and a depth factor\. Writingγ:=η​S1\\gamma:=\\eta S\_\{1\}—so the overshoot\-onset scaleη1=1/S1\\eta\_\{1\}=1/S\_\{1\}isγ=1\\gamma=1and the classical stability boundηstable=2/S1\\eta\_\{\\mathrm\{stable\}\}=2/S\_\{1\}isγ=2\\gamma=2—we have

ηWCR​\(L,σ⋆1\)=γWCR​\(L\)S1,\\eta\_\{\\mathrm\{WCR\}\}\(L,\\sigma\_\{\\star 1\}\)=\\frac\{\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\}\{S\_\{1\}\},\(20\)withγWCR​\(L\)\\gamma\_\{\\mathrm\{WCR\}\}\(L\)depending only on depth\.

###### Theorem 6\.2\(Existence and computation\)\.

For every integerL≥2L\\geq 2there is a uniqueγWCR​\(L\)\>2\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\>2for which \([20](https://arxiv.org/html/2606.05219#S6.E20)\) satisfies Definition[6\.1](https://arxiv.org/html/2606.05219#S6.Thmtheorem1)\. It is the root of a single scalar equation, computable by bisection \(Appendix[F](https://arxiv.org/html/2606.05219#A6)\)\.

SinceγWCR​\(L\)\>2\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\>2, the safe ceiling always exceeds the classical boundηstable=2/S1\\eta\_\{\\mathrm\{stable\}\}=2/S\_\{1\}: the chain tolerates steps beyond the classical linear\-stability threshold\. The margin over the onset scale is exactly this constant,

ηWCRη1=γWCR​\(L\),\\frac\{\\eta\_\{\\mathrm\{WCR\}\}\}\{\\eta\_\{1\}\}=\\gamma\_\{\\mathrm\{WCR\}\}\(L\),\(21\)independent of the targetσ⋆1\\sigma\_\{\\star 1\}\.

###### Proposition 6\.3\(Growth with depth\)\.

The normalized WCR ratioγWCR​\(L\)=ηWCR/η1\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=\\eta\_\{\\mathrm\{WCR\}\}/\\eta\_\{1\}grows unboundedly with depth and admits the asymptotic scaling

γWCR​\(L\)=Θ​\(log⁡L\)\.\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=\\Theta\(\\log L\)\.\(22\)In particular, deeper chains admit a strictly larger window of “overshoot\-but\-return” learning rates:

η1<η\\displaystyle\\eta\_\{1\}<\\eta<ηWCR​\(L,σ⋆1\)\\displaystyle<\\eta\_\{\\mathrm\{WCR\}\}\(L,\\sigma\_\{\\star 1\}\)withηWCRη1\\displaystyle\\text\{with\}\\quad\\frac\{\\eta\_\{\\mathrm\{WCR\}\}\}\{\\eta\_\{1\}\}=γWCR​\(L\)↑as​L↑,\\displaystyle=\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\\uparrow\\ \\text\{as \}L\\uparrow,\(23\)so that the maximal safe overshoot amplitude \(in the worst\-case sense of Definition[6\.1](https://arxiv.org/html/2606.05219#S6.Thmtheorem1)\) increases with depth\.

This is also visible in the return\-map view in Appendix[F](https://arxiv.org/html/2606.05219#A6)\(Figure[6](https://arxiv.org/html/2606.05219#A6.F6)\): at a fixed ratioη/η1\\eta/\\eta\_\{1\}, the worst overshoot’s next iterate stays positive asLLgrows, turning a sign flip at small depth into a safe return at large depth\.

#### The admissible window and rank\-ppbalancing\.

We can now identify which modes re\-balance\. On the depth\-balanced manifold, modeiihas single\-path sharpnessSiS\_\{i\}and flattest balanced sharpnessλimin=H2/L−1​Si\\lambda\_\{i\}^\{\\min\}=H^\{2/L\-1\}S\_\{i\}\(Theorem[4\.2](https://arxiv.org/html/2606.05219#S4.Thmtheorem2)\), and leaves the single\-path attractor onceη\>2/Si\\eta\>2/S\_\{i\}\. SinceS1≥⋯≥SdS\_\{1\}\\geq\\cdots\\geq S\_\{d\}, a step in the band

2Sp<η<min⁡\{2Sp\+1,ηmax\},\\displaystyle\\frac\{2\}\{S\_\{p\}\}<\\eta<\\min\\Bigl\\\{\\frac\{2\}\{S\_\{p\+1\}\},\\,\\eta\_\{\\max\}\\Bigr\\\},ηmax:=min⁡\{2λ1min,ηWCR\},\\displaystyle\\eta\_\{\\max\}:=\\min\\Bigl\\\{\\tfrac\{2\}\{\\lambda\_\{1\}^\{\\min\}\},\\,\\eta\_\{\\mathrm\{WCR\}\}\\Bigr\\\},\(24\)re\-balances the topppmodes and leaves the tail in the GF\-like regime; the achievable rank is the largestppwith2/Sp<ηmax2/S\_\{p\}<\\eta\_\{\\max\}\. The two ceilings inηmax\\eta\_\{\\max\}guard different phases:2/λ1min2/\\lambda\_\{1\}^\{\\min\}keeps the*balanced*attractor stable, whileηWCR\\eta\_\{\\mathrm\{WCR\}\}keeps the dominant path alive through the*transient*\. In normalized units they readγ=2​H1−2/L\\gamma=2H^\{1\-2/L\}andγ=γWCR​\(L\)\\gamma=\\gamma\_\{\\mathrm\{WCR\}\}\(L\); both increase with depth, and which one binds depends onHHandLL\. Either way, depth raisesηmax\\eta\_\{\\max\}, and with it the number of modes that re\-balance\.

#### A mechanism for warm\-up\.

The ceiling also hints at why learning\-rate warm\-up helps\. Sharpness peaks early, while the dominant path is still forming, so a step large enough to re\-balance later may exceedηWCR\\eta\_\{\\mathrm\{WCR\}\}during that peak and trigger a sign flip\. warm\-up resolves the tension: a small early step carries the trajectory safely through the sharp transient, and a later increase sustains the drift toward flat, multi\-pathway minima\. In this view, warm\-up does not change the re\-balancing mechanism; it delays the large\-step regime until the trajectory has passed the most fragile single\-path bottleneck\.

## 7Discussion

Large\-step Gradient Descent qualitatively changes the multi\-pathway learning predicted by continuous\-time theories\. Beyond our specific setting, this finding affects how we interpret a broader class of GF\-based architectural bias results\.

Theoretical accounts of over\-parameterized models often predict that training selects sparse, isolated substructures\. GF analyses of multi\-pathway DLNs predict “winner\-takes\-all” specialization\(Shiet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib35); Saxeet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib34)\), and the Lottery Ticket Hypothesis\(Frankle and Carbin,[2019](https://arxiv.org/html/2606.05219#bib.bib20)\)conjectures that trained over\-parameterized networks contain sparse subnetworks carrying most of their function\. Our results suggest that, even within such a subnetwork view, large\-step GD’s preference for low\-curvature minima can reshape which substructures emerge: rather than concentrating signal in a single pathway, GD tends to spread it across parallel ones\. How this flat\-minima bias interacts with the formation of sparse subnetworks under large\-step GD is an open question worth exploring\.

GF\-based analyses also argue that structural asymmetries \(e\.g\., depth differences\) drive the system toward a winner\-takes\-all solution favoring the shallower path\(Saxeet al\.,[2022](https://arxiv.org/html/2606.05219#bib.bib34)\)\. Section[5\.3](https://arxiv.org/html/2606.05219#S5.SS3)shows that large\-step GD overrides such asymmetries within the range of depths we test: once the sharp single\-path minimum becomes GD\-unstable, the system moves toward a flatter, distributed configuration even when GF would predict a strongly asymmetric solution\. In this regime, the optimizer’s implicit bias toward flat minima outweighs the structural bias toward shallow pathways predicted by GF\.

## 8Conclusion

We studied how large\-step Gradient Descent shapes learning in multi\-pathway Deep Linear Networks, and showed that its behavior departs qualitatively from the continuous\-time picture\. First, distributing a target feature equally acrossHHparallel pathways reduces sharpness byH2/L−1H^\{2/L\-1\}relative to a single\-path solution \(Theorem[4\.2](https://arxiv.org/html/2606.05219#S4.Thmtheorem2)\): sparse minima are the sharpest configurations on the depth\-balanced manifold, while balanced ones are the flattest\. Second, training at the Edge of Stability proceeds in two phases\. Symmetry breaking concentrates the signal in a dominant pathway, as GF predicts\. Once local sharpness exceeds2/η2/\\eta, oscillating residuals transfer mass to subordinate pathways until the configuration satisfies the stability constraint—overriding both initialization\-induced and depth\-induced asymmetries\. Third, a Worst\-Case Return threshold derived from a deep linear chain boundsη\\etafrom above to guarantee the trajectory survives the transient oscillations\. The ratioηWCR/ηstable\\eta\_\{\\text\{WCR\}\}/\\eta\_\{\\text\{stable\}\}grows asΘ​\(log⁡L\)\\Theta\(\\log L\), so deeper architectures admit a wider window of re\-balancing learning rates\.

Together, these results show that the symmetry breaking predicted by Gradient Flow does not persist once the infinitesimal\-learning\-rate assumption is relaxed\. Architectural biases derived under continuous\-time dynamics deserve re\-examination through the lens of discrete optimization\. Extending this analysis to nonlinear and modular architectures, such as Mixture\-of\-Experts and multi\-head attention, is a natural next step\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## Acknowledgments

We thank the anonymous reviewers for insightful reviews\. This work was partially supported by Institute of Information & communications Technology Planning & Evaluation \(IITP\) grants \(RS\-2020\-II201373, Artificial Intelligence Graduate School Program \(Hanyang University\); RS\-2023\-002206284, Artificial intelligence for prediction of structure\-based protein interaction reflecting physicochemical principles\); the BK21 FOUR \(Fostering Outstanding Universities for Research\) project; NRF2024S1A5C3A02043653, Socio\-Technological Solutions for Bridging the AI Divide: A Blockchain and Federated Learning\-Based AI Training Data Platform\) and Korea Institute for Advanced Study \(KIAS\) grant funded by the Korean government \(MSIT\)\.

## References

- S\. Arora, N\. Cohen, and E\. Hazan \(2018\)On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization\.InProceedings of the 35th International Conference on Machine Learning,pp\. 244–253\.External Links:ISSN 2640\-3498Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1),[§2](https://arxiv.org/html/2606.05219#S2.p1.1)\.
- S\. Arora, N\. Cohen, W\. Hu, and Y\. Luo \(2019\)Implicit Regularization in Deep Matrix Factorization\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5)\.
- P\. Baldi and K\. Hornik \(1989\)Neural networks and principal component analysis: Learning from examples without local minima\.Neural Networks2\(1\),pp\. 53–58\.External Links:ISSN 0893\-6080,[Document](https://dx.doi.org/10.1016/0893-6080%2889%2990014-2)Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p1.1)\.
- H\. Chou, C\. Gieshoff, J\. Maly, and H\. Rauhut \(2024\)Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank\.Applied and Computational Harmonic Analysis68,pp\. 101595\.External Links:ISSN 1063\-5203,[Document](https://dx.doi.org/10.1016/j.acha.2023.101595)Cited by:[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5)\.
- J\. Cohen, A\. Damian, A\. Talwalkar, J\. Z\. Kolter, and J\. D\. Lee \(2025\)Understanding optimization in deep learning with central flows\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sIE2rI3ZPs)Cited by:[§E\.6](https://arxiv.org/html/2606.05219#A5.SS6.p1.1),[§2](https://arxiv.org/html/2606.05219#S2.p2.1)\.
- J\. Cohen, S\. Kaur, Y\. Li, J\. Z\. Kolter, and A\. Talwalkar \(2021\)Gradient descent on neural networks typically occurs at the edge of stability\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jh-rTtvkGeM)Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p2.1),[§2](https://arxiv.org/html/2606.05219#S2.p2.1)\.
- A\. Damian, E\. Nichani, and J\. D\. Lee \(2023\)Self\-stabilization: the implicit bias of gradient descent at the edge of stability\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nhKHA59gXz)Cited by:[Appendix D](https://arxiv.org/html/2606.05219#A4.SS0.SSS0.Px1.p1.2),[Appendix D](https://arxiv.org/html/2606.05219#A4.SS0.SSS0.Px1.p1.3),[§E\.6](https://arxiv.org/html/2606.05219#A5.SS6.SSS0.Px2.p1.4),[§2](https://arxiv.org/html/2606.05219#S2.p2.1)\.
- M\. Even, S\. Pesme, S\. Gunasekar, and N\. Flammarion \(2023\)\(S\)GD over Diagonal Linear Networks: Implicit bias, Large Stepsizes and Edge of Stability\.Advances in Neural Information Processing Systems36,pp\. 29406–29448\.Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p3.1)\.
- J\. Frankle and M\. Carbin \(2019\)The lottery ticket hypothesis: finding sparse, trainable neural networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJl-b3RcF7)Cited by:[§7](https://arxiv.org/html/2606.05219#S7.p2.1)\.
- A\. Ghosh, S\. M\. Kwon, R\. Wang, S\. Ravishankar, and Q\. Qu \(2025\)Learning dynamics of deep matrix factorization beyond the edge of stability\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=J4Dvxv7WnG)Cited by:[Appendix B](https://arxiv.org/html/2606.05219#A2.p1.1),[§C\.1](https://arxiv.org/html/2606.05219#A3.SS1.p2.5),[§C\.2](https://arxiv.org/html/2606.05219#A3.SS2.p1.4),[§C\.3](https://arxiv.org/html/2606.05219#A3.SS3.SSS0.Px1.p1.4),[§E\.1](https://arxiv.org/html/2606.05219#A5.SS1.p1.2),[§2](https://arxiv.org/html/2606.05219#S2.p3.1),[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.1),[§3](https://arxiv.org/html/2606.05219#S3.p1.8),[§5\.2](https://arxiv.org/html/2606.05219#S5.SS2.p1.3)\.
- G\. Gidel, F\. Bach, and S\. Lacoste\-Julien \(2019\)Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5)\.
- S\. Gunasekar, B\. E\. Woodworth, S\. Bhojanapalli, B\. Neyshabur, and N\. Srebro \(2017\)Implicit regularization in matrix factorization\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1)\.
- C\. C\. J Dominé, L\. Braun, J\. E\. Fitzgerald, and A\. M\. Saxe \(2023\)Exact learning dynamics of deep linear networks with prior knowledge\*\.Journal of Statistical Mechanics: Theory and Experiment2023\(11\),pp\. 114004\.External Links:ISSN 1742\-5468,[Document](https://dx.doi.org/10.1088/1742-5468/ad01b8)Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p1.1)\.
- S\. M\. Kwon, Z\. Zhang, D\. Song, L\. Balzano, and Q\. Qu \(2024\)Efficient Low\-Dimensional Compression of Overparameterized Models\.InProceedings of The 27th International Conference on Artificial Intelligence and Statistics,pp\. 1009–1017\.External Links:ISSN 2640\-3498Cited by:[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5)\.
- A\. K\. Lampinen and S\. Ganguli \(2019\)An analytic theory of generalization dynamics and transfer learning in deep linear networks\.In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019,External Links:[Link](https://openreview.net/forum?id=ryfMLoCqtQ)Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1),[§2](https://arxiv.org/html/2606.05219#S2.p1.1)\.
- A\. Lewkowycz, Y\. Bahri, E\. Dyer, J\. Sohl\-Dickstein, and G\. Gur\-Ari \(2020\)The large learning rate phase of deep learning: the catapult mechanism\.arXiv\.External Links:2003\.02218,[Document](https://dx.doi.org/10.48550/arXiv.2003.02218)Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p2.1)\.
- P\. Marion and L\. Chizat \(2024\)Deep linear networks for regression are implicitly regularized towards flat minima\.Advances in Neural Information Processing Systems37,pp\. 76848–76900\.External Links:[Document](https://dx.doi.org/10.52202/079017-2445)Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p3.1)\.
- Y\. Nam, S\. H\. Lee, C\. C\. J\. Dominé, Y\. Park, C\. London, W\. Choi, N\. A\. Göring, and S\. Lee \(2025\)Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena \(Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking\)\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 81897–81929\.External Links:ISSN 2640\-3498Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p1.1)\.
- B\. Neyshabur, S\. Bhojanapalli, D\. Mcallester, and N\. Srebro \(2017\)Exploring generalization in deep learning\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1)\.
- A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli \(2014\)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.arXiv\.External Links:1312\.6120,[Document](https://dx.doi.org/10.48550/arXiv.1312.6120)Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1),[§2](https://arxiv.org/html/2606.05219#S2.p1.1),[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5)\.
- A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli \(2019\)A mathematical theory of semantic development in deep neural networks\.Proceedings of the National Academy of Sciences116\(23\),pp\. 11537–11546\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1820226116)Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p1.1)\.
- A\. Saxe, S\. Sodhani, and S\. J\. Lewallen \(2022\)The neural race reduction: dynamics of abstraction in gated networks\.InProceedings of the 39th International Conference on Machine Learning,pp\. 19287–19309\.External Links:ISSN 2640\-3498Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p1.1),[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5),[§5\.3](https://arxiv.org/html/2606.05219#S5.SS3.p4.3),[§7](https://arxiv.org/html/2606.05219#S7.p2.1),[§7](https://arxiv.org/html/2606.05219#S7.p3.1)\.
- J\. Shi, E\. Shea\-Brown, and M\. Buice \(2022\)Learning dynamics of deep linear networks with multiple pathways\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 34064–34076\.Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1),[§1](https://arxiv.org/html/2606.05219#S1.p2.1),[§2](https://arxiv.org/html/2606.05219#S2.p1.1),[§2](https://arxiv.org/html/2606.05219#S2.p3.1),[§3](https://arxiv.org/html/2606.05219#S3.p1.7),[§5\.1](https://arxiv.org/html/2606.05219#S5.SS1.p2.5),[§5\.1](https://arxiv.org/html/2606.05219#S5.SS1.p5.6),[§5\.2](https://arxiv.org/html/2606.05219#S5.SS2.p2.1),[§7](https://arxiv.org/html/2606.05219#S7.p2.1)\.
- D\. Soudry, E\. Hoffer, M\. S\. Nacson, S\. Gunasekar, and N\. Srebro \(2018\)The implicit bias of gradient descent on separable data\.Journal of Machine Learning Research19\(70\),pp\. 1–57\.Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1)\.
- A\. V\. Varre, M\. Vladarean, L\. Pillaud\-Vivien, and N\. Flammarion \(2023\)On the spectral bias of two\-layer linear networks\.Advances in Neural Information Processing Systems36,pp\. 64380–64414\.Cited by:[§3\.2](https://arxiv.org/html/2606.05219#S3.SS2.p1.5)\.
- L\. Wu, C\. Ma, and W\. E \(2018\)How SGD Selects the Global Minima in Over\-parameterized Learning: A Dynamical Stability Perspective\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p2.1)\.
- C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals \(2017\)Understanding deep learning requires rethinking generalization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Sy8gdB9xx)Cited by:[§1](https://arxiv.org/html/2606.05219#S1.p1.1)\.
- L\. Zhu, C\. Liu, A\. Radhakrishnan, and M\. Belkin \(2024\)Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning\.InProceedings of the 41st International Conference on Machine Learning,pp\. 62476–62509\.External Links:ISSN 2640\-3498Cited by:[§2](https://arxiv.org/html/2606.05219#S2.p2.1)\.

## Appendix ANotation

The following table summarizes the key mathematical notations used throughout this paper\.

## Appendix BInvariance of Depthwise Balancing under the SVS Assumption

This section justifies the restriction to the depth\-balanced manifold used in our sharpness analysis\. Our argument follows the singular\-vector\-stationary \(SVS\) viewpoint developed for deep matrix factorization in\(Ghoshet al\.,[2025](https://arxiv.org/html/2606.05219#bib.bib21)\)and adapts it to the multi\-pathway architecture\.

#### Setting\.

We consider the multi\-pathway deep linear network in Section[3](https://arxiv.org/html/2606.05219#S3): there areHHparallel pathways\. Pathwayh∈\{1,…,H\}h\\in\\\{1,\\dots,H\\\}has depthLhL\_\{h\}, layer weights\{Wh​ℓ∈ℝd×d\}ℓ=1Lh\\\{W\_\{h\\ell\}\\in\\mathbb\{R\}^\{d\\times d\}\\\}\_\{\\ell=1\}^\{L\_\{h\}\}, end\-to\-end mapΩh:=Wh​Lh​⋯​Wh​1\\Omega\_\{h\}:=W\_\{hL\_\{h\}\}\\cdots W\_\{h1\}, total mapM:=∑h=1HΩhM:=\\sum\_\{h=1\}^\{H\}\\Omega\_\{h\}, and squared Frobenius loss

ℒ​\(Θ\):=12​‖M−M⋆‖F2,M⋆=U⋆​Σ⋆​V⋆⊤,Σ⋆=diag​\(σ1⁣⋆,⋯,σd⁣⋆\)\.\\mathcal\{L\}\(\\Theta\):=\\frac\{1\}\{2\}\\\|M\-M\_\{\\star\}\\\|\_\{F\}^\{2\},\\qquad M\_\{\\star\}=U\_\{\\star\}\\Sigma\_\{\\star\}V\_\{\\star\}^\{\\top\},\\;\\;\\Sigma\_\{\\star\}=\\mathrm\{diag\}\(\\sigma\_\{1\\star\},\\cdots,\\sigma\_\{d\\star\}\)\.\(25\)Throughout, we work in the \(mode\) coordinates induced by the fixed singular vectors\(U⋆,V⋆\)\(U\_\{\\star\},V\_\{\\star\}\)ofM⋆M\_\{\\star\}\.

### B\.1SVS parametrization and mode\-wise decoupling

###### Definition B\.1\(SVS parametrization for each pathway\)\.

We sayΘ\\Thetalies in the SVS set associated withM⋆M\_\{\\star\}if for each pathwayhhthere exist*fixed*orthogonal matrices\{Qh​ℓ\}ℓ=1Lh\+1\\\{Q\_\{h\\ell\}\\\}\_\{\\ell=1\}^\{L\_\{h\}\+1\}such thatQh​1=V⋆Q\_\{h1\}=V\_\{\\star\},Qh​Lh\+1=U⋆Q\_\{hL\_\{h\}\+1\}=U\_\{\\star\}, and each layer admits the decomposition

Wh​ℓ=Qh​ℓ\+1​Σh​ℓ​Qh​ℓ⊤,Σh​ℓ=diag​\(σh​1\(ℓ\),…,σh​d\(ℓ\)\),W\_\{h\\ell\}=Q\_\{h\\ell\+1\}\\,\\Sigma\_\{h\\ell\}\\,Q\_\{h\\ell\}^\{\\top\},\\qquad\\Sigma\_\{h\\ell\}=\\mathrm\{diag\}\(\\sigma\_\{h1\}^\{\(\\ell\)\},\\ldots,\\sigma\_\{hd\}^\{\(\\ell\)\}\),\(26\)for some diagonalΣh​ℓ\\Sigma\_\{h\\ell\}\(with nonnegative diagonal entries, interpreted as layer singular values in the SVS chart\)\.

Under Definition[B\.1](https://arxiv.org/html/2606.05219#A2.Thmtheorem1), adjacent orthogonals cancel in products, so for each pathway

Ωh=Wh​Lh⋯Wh​1=U⋆\(∏ℓ=1LhΣh​ℓ\)V⋆⊤=:U⋆ΣhV⋆⊤,\\Omega\_\{h\}=W\_\{hL\_\{h\}\}\\cdots W\_\{h1\}=U\_\{\\star\}\\Big\(\\prod\_\{\\ell=1\}^\{L\_\{h\}\}\\Sigma\_\{h\\ell\}\\Big\)V\_\{\\star\}^\{\\top\}=:U\_\{\\star\}\\Sigma\_\{h\}V\_\{\\star\}^\{\\top\},\(27\)whereΣh\\Sigma\_\{h\}is diagonal and itsii\-th diagonal entry is the end\-to\-end singular value in modeii:

σh​i:=∏ℓ=1Lhσh​i\(ℓ\)\.\\sigma\_\{hi\}:=\\prod\_\{\\ell=1\}^\{L\_\{h\}\}\\sigma\_\{hi\}^\{\(\\ell\)\}\.\(28\)Consequently,

M=∑h=1HΩh=U⋆​\(∑h=1HΣh\)​V⋆⊤,M=\\sum\_\{h=1\}^\{H\}\\Omega\_\{h\}=U\_\{\\star\}\\Big\(\\sum\_\{h=1\}^\{H\}\\Sigma\_\{h\}\\Big\)V^\{\\star\\top\},\(29\)soMMshares the singular vectors\(U⋆,V⋆\)\(U\_\{\\star\},V\_\{\\star\}\)withM⋆M\_\{\\star\}and the loss decouples across modes\.

###### Lemma B\.2\(Mode\-wise decoupling in the SVS set\)\.

IfΘ\\Thetalies in the SVS set, then the loss decomposes as

ℒ​\(Θ\)=∑i=1dℒi​\(Θ\),ℒi​\(Θ\)=12​\(∑h=1Hσh​i−σ⋆i\)2,\\mathcal\{L\}\(\\Theta\)=\\sum\_\{i=1\}^\{d\}\\mathcal\{L\}\_\{i\}\(\\Theta\),\\qquad\\mathcal\{L\}\_\{i\}\(\\Theta\)=\\frac\{1\}\{2\}\\Big\(\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\}\\Big\)^\{2\},\(30\)whereσh​i\\sigma\_\{hi\}is defined in \([28](https://arxiv.org/html/2606.05219#A2.E28)\)\.

###### Proof\.

SinceM−M⋆=U⋆​\(∑hΣh−Σ⋆\)​V⋆⊤M\-M\_\{\\star\}=U\_\{\\star\}\\big\(\\sum\_\{h\}\\Sigma\_\{h\}\-\\Sigma\_\{\\star\}\\big\)V^\{\\star\\top\}and Frobenius norm is unitarily invariant,

‖M−M⋆‖F2=‖∑h=1HΣh−Σ⋆‖F2=∑i=1d\(∑h=1Hσh​i−σ⋆i\)2,\\\|M\-M\_\{\\star\}\\\|\_\{F\}^\{2\}=\\Big\\\|\\sum\_\{h=1\}^\{H\}\\Sigma\_\{h\}\-\\Sigma\_\{\\star\}\\Big\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{d\}\\Big\(\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\}\\Big\)^\{2\},which yields the claim\. ∎

### B\.2Depthwise\-balanced manifold and its invariance

###### Definition B\.3\(Depthwise\-balanced manifold\)\.

For each pathwayhhand modeii, define the end\-to\-end singular valueσh​i\\sigma\_\{hi\}as in \([28](https://arxiv.org/html/2606.05219#A2.E28)\)\. The depthwise\-balanced manifoldℬ\\mathcal\{B\}is the subset of the SVS set where, for allhhandii,

σh​i\(ℓ\)=σh​i1/Lh∀ℓ∈\{1,…,Lh\},equivalentlyσh​1,i=⋯=σh​Lh,i\.\\sigma\_\{hi\}^\{\(\\ell\)\}=\\sigma\_\{hi\}^\{1/L\_\{h\}\}\\quad\\forall\\ell\\in\\\{1,\\dots,L\_\{h\}\\\},\\qquad\\text\{equivalently\}\\qquad\\sigma\_\{h1,i\}=\\cdots=\\sigma\_\{hL\_\{h\},i\}\.\(31\)

###### Proposition B\.4\(Depthwise balancing is invariant under GD in the SVS set\)\.

Assume the iterates remain in the SVS set and are updated by gradient descent:

Wh​ℓ​\(t\+1\)=Wh​ℓ​\(t\)−η​∇Wh​ℓℒ​\(Θ​\(t\)\),∀h,ℓ\.W\_\{h\\ell\}\(t\+1\)=W\_\{h\\ell\}\(t\)\-\\eta\\,\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}\(\\Theta\(t\)\),\\qquad\\forall h,\\ell\.IfΘ​\(t\)∈ℬ\\Theta\(t\)\\in\\mathcal\{B\}\(Definition[B\.3](https://arxiv.org/html/2606.05219#A2.Thmtheorem3)\) for somett, thenΘ​\(t\+1\)∈ℬ\\Theta\(t\+1\)\\in\\mathcal\{B\}\. Consequently, if the initialization is depthwise\-balanced, thenΘ​\(t\)∈ℬ\\Theta\(t\)\\in\\mathcal\{B\}for allt≥0t\\geq 0\.

Moreover, letting the mode residual be

ri​\(t\):=∑h=1Hσh​i​\(t\)−σ⋆i,r\_\{i\}\(t\):=\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\(t\)\-\\sigma\_\{\\star i\},\(32\)the GD update of each layer singular value \(in the SVS coordinates\) is, for allh,ℓ,ih,\\ell,i,

σh​i\(ℓ\)​\(t\+1\)=σh​i\(ℓ\)​\(t\)−η​ri​\(t\)​∏j∈\{1,…,Lh\},j≠ℓσh,j,i​\(t\)\.\\sigma\_\{hi\}^\{\(\\ell\)\}\(t\+1\)=\\sigma\_\{hi\}^\{\(\\ell\)\}\(t\)\-\\eta\\,r\_\{i\}\(t\)\\\!\\\!\\prod\_\{j\\in\\\{1,\\dots,L\_\{h\}\\\},\\,j\\neq\\ell\}\\\!\\\!\\sigma\_\{h,j,i\}\(t\)\.\(33\)

###### Proof\.

Fix a pathwayhhand layerℓ\\ell\. The standard gradient formula for deep linear networks gives

∇Wh​ℓℒ=Wh​Lh:ℓ\+1⊤​\(M−M⋆\)​Wh​ℓ−1:1⊤,\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}=W\_\{hL\_\{h\}:\\ell\+1\}^\{\\top\}\\,\(M\-M\_\{\\star\}\)\\,W\_\{h\\ell\-1:1\}^\{\\top\},\(34\)whereWh,a:b:=Wh,a​⋯​Wh,bW\_\{h,a:b\}:=W\_\{h,a\}\\cdots W\_\{h,b\}with the conventionWh,a:b=IW\_\{h,a:b\}=Iwhena<ba<b\.

Under Definition[B\.1](https://arxiv.org/html/2606.05219#A2.Thmtheorem1), partial products collapse to the fixed bases with diagonal cores:

Wh,Lh:ℓ\+1⊤=Qh,ℓ\+1​\(∏j=ℓ\+1LhΣh,j\)​U⋆⊤,Wh,ℓ−1:1⊤=V⋆​\(∏j=1ℓ−1Σh,j\)​Qh,ℓ⊤\.W\_\{h,L\_\{h\}:\\ell\+1\}^\{\\top\}=Q\_\{h,\\ell\+1\}\\Big\(\\prod\_\{j=\\ell\+1\}^\{L\_\{h\}\}\\Sigma\_\{h,j\}\\Big\)U^\{\\star\\top\},\\qquad W\_\{h,\\ell\-1:1\}^\{\\top\}=V\_\{\\star\}\\Big\(\\prod\_\{j=1\}^\{\\ell\-1\}\\Sigma\_\{h,j\}\\Big\)Q\_\{h,\\ell\}^\{\\top\}\.Also, sinceM=U⋆​\(∑kΣk\)​V⋆⊤M=U\_\{\\star\}\(\\sum\_\{k\}\\Sigma\_\{k\}\)V^\{\\star\\top\}, we have

M−M⋆=U⋆​diag​\(r1,…,rd\)​V⋆⊤,M\-M\_\{\\star\}=U\_\{\\star\}\\,\\mathrm\{diag\}\(r\_\{1\},\\dots,r\_\{d\}\)\\,V^\{\\star\\top\},withrir\_\{i\}defined in \([32](https://arxiv.org/html/2606.05219#A2.E32)\)\. Substituting these expressions into \([34](https://arxiv.org/html/2606.05219#A2.E34)\) and usingU⋆⊤​U⋆=IU^\{\\star\\top\}U\_\{\\star\}=I,V⋆⊤​V⋆=IV^\{\\star\\top\}V\_\{\\star\}=Iyields

∇Wh​ℓℒ=Qh,ℓ\+1​\(∏j=ℓ\+1LhΣh,j\)​diag​\(r1,…,rd\)​\(∏j=1ℓ−1Σh,j\)​Qh,ℓ⊤\.\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}=Q\_\{h,\\ell\+1\}\\Big\(\\prod\_\{j=\\ell\+1\}^\{L\_\{h\}\}\\Sigma\_\{h,j\}\\Big\)\\mathrm\{diag\}\(r\_\{1\},\\dots,r\_\{d\}\)\\Big\(\\prod\_\{j=1\}^\{\\ell\-1\}\\Sigma\_\{h,j\}\\Big\)Q\_\{h,\\ell\}^\{\\top\}\.All matrices in the middle are diagonal and commute, hence the gradient shares the same left/right orthogonals asWh​ℓW\_\{h\\ell\}and has diagonal entries

\[∇Wh​ℓℒ\]i,i=ri​∏j≠ℓσh,j,i\.\\Big\[\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}\\Big\]\_\{i,i\}=r\_\{i\}\\prod\_\{j\\neq\\ell\}\\sigma\_\{h,j,i\}\.Therefore the GD update preserves the SVS chart and, at the level of diagonal entries, is exactly \([33](https://arxiv.org/html/2606.05219#A2.E33)\)\.

Now assumeΘ​\(t\)∈ℬ\\Theta\(t\)\\in\\mathcal\{B\}\. Then for each fixed\(h,i\)\(h,i\)all layer singular values are equal:σh,1,i\(t\)=⋯=σh,Lh,i\(t\)=:sh​i\(t\)\\sigma\_\{h,1,i\}\(t\)=\\cdots=\\sigma\_\{h,L\_\{h\},i\}\(t\)=:s\_\{hi\}\(t\)\. Hence the multiplicative factor in \([33](https://arxiv.org/html/2606.05219#A2.E33)\) becomes

∏j≠ℓσh,j,i​\(t\)=sh​i​\(t\)Lh−1,\\prod\_\{j\\neq\\ell\}\\sigma\_\{h,j,i\}\(t\)=s\_\{hi\}\(t\)^\{L\_\{h\}\-1\},which is independent ofℓ\\ell\. Thus every layer in the same pathway receives the same update in modeii, implyingσh,1,i​\(t\+1\)=⋯=σh,Lh,i​\(t\+1\)\\sigma\_\{h,1,i\}\(t\+1\)=\\cdots=\\sigma\_\{h,L\_\{h\},i\}\(t\+1\)\. This is preciselyΘ​\(t\+1\)∈ℬ\\Theta\(t\+1\)\\in\\mathcal\{B\}\. The final claim follows by induction ontt\. ∎

#### Implication for our initialization\.

Our depth\-balanced diagonal initialization \(Eq\. \(4\)\) satisfiesσh,1,i​\(0\)=⋯=σh,Lh,i​\(0\)\\sigma\_\{h,1,i\}\(0\)=\\cdots=\\sigma\_\{h,L\_\{h\},i\}\(0\)for each pathwayhhand modeii\. Hence,*conditional on remaining in the SVS set*, Proposition[B\.4](https://arxiv.org/html/2606.05219#A2.Thmtheorem4)justifies restricting the analysis to the depth\-balanced manifold throughout training\.

## Appendix CSharpness Reduction in Multi\-Pathway Linear Models

In this section, we rigorously derive the relationship between the number of parallel pathwaysHHand the loss sharpness at a balanced global minimum\. We demonstrate that distributing a target feature across multiple pathways significantly reduces the network’s sharpness, thereby extending the edge of stability regime\.

### C\.1Problem Formulation and Hessian Structure

Consider a Deep Linear Network \(DLN\) composed ofHHparallel pathways, where the output matrixMMis the sum of the outputs of individual pathwaysΩh\\Omega\_\{h\}\. The objective is to minimize the squared Frobenius loss:

ℒ​\(Θ\)=12​‖M−M⋆‖F2=12​‖∑h=1HΩh​\(Θh\)−M⋆‖F2,\\mathcal\{L\}\(\\Theta\)=\\frac\{1\}\{2\}\\\|M\-M\_\{\\star\}\\\|\_\{F\}^\{2\}=\\frac\{1\}\{2\}\\left\\\|\\sum\_\{h=1\}^\{H\}\\Omega\_\{h\}\(\\Theta\_\{h\}\)\-M\_\{\\star\}\\right\\\|\_\{F\}^\{2\},\(35\)whereΘh=\(Wh,1,…,Wh,L\)\\Theta\_\{h\}=\(W\_\{h,1\},\\dots,W\_\{h,L\}\)represents the parameters of thehh\-th pathway, andΩh=Wh,L​⋯​Wh,1\\Omega\_\{h\}=W\_\{h,L\}\\cdots W\_\{h,1\}\.

To analyze the sharpness, we examine the Hessian of the loss,∇2ℒ​\(Θ\)\\nabla^\{2\}\\mathcal\{L\}\(\\Theta\), at a global minima whereM=M⋆M=M\_\{\\star\}\. Utilizing the vectorization of parameters, the Hessian can be decomposed into blocks corresponding to the interactions between pathwayshhandkk\. Following the derivation ofGhoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)in Appendix C\.3\.2, the\(h,k\)\(h,k\)\-th block of the Hessian is given by:

𝐇h,k=∇vec​\(Θh\)∇vec​\(Θk\)⊤⁡ℒ≈𝐀h⊤​𝐀k,\\mathbf\{H\}\_\{h,k\}=\\nabla\_\{\\text\{vec\}\(\\Theta\_\{h\}\)\}\\nabla\_\{\\text\{vec\}\(\\Theta\_\{k\}\)\}^\{\\top\}\\mathcal\{L\}\\approx\\mathbf\{A\}\_\{h\}^\{\\top\}\\mathbf\{A\}\_\{k\},\(36\)

### C\.2Mode\-Wise Decoupling in the SVS Set

We now refine the argument by leveraging the*singular vector stationary*\(SVS\) structure used inGhoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)\. Assume that training is restricted to the SVS set associated with the target SVD

M⋆=U​Σ⋆​V⊤,Σ⋆=diag​\(σ⋆1,…,σ⋆d\),M\_\{\\star\}=U\\,\\Sigma\_\{\\star\}\\,V^\{\\top\},\\qquad\\Sigma\_\{\\star\}=\\mathrm\{diag\}\(\\sigma\_\{\\star 1\},\\dots,\\sigma\_\{\\star d\}\),so that the singular vectors of all layer matrices remain fixed \(equal to those ofM⋆M\_\{\\star\}\) throughout training\. In particular, for each pathwayhhand layerℓ\\ellwe may write

Wh​ℓ=U​Σh​ℓ​V⊤,Σh​ℓ=diag​\(σh​ℓ,1,…,σh​ℓ,d\),W\_\{h\\ell\}\\;=\\;U\\,\\Sigma\_\{h\\ell\}\\,V^\{\\top\},\\qquad\\Sigma\_\{h\\ell\}=\\mathrm\{diag\}\(\\sigma\_\{h\\ell,1\},\\dots,\\sigma\_\{h\\ell,d\}\),hence each end\-to\-end map is

Ωh\\displaystyle\\Omega\_\{h\}\\;=U​Σh​V⊤,\\displaystyle=\\;U\\,\\Sigma\_\{h\}\\,V^\{\\top\},Σh\\displaystyle\\Sigma\_\{h\}=∏ℓ=1LΣh​ℓ=diag​\(σh​1,…,σh​d\),\\displaystyle=\\prod\_\{\\ell=1\}^\{L\}\\Sigma\_\{h\\ell\}=\\mathrm\{diag\}\(\\sigma\_\{h1\},\\dots,\\sigma\_\{hd\}\),σh​i:\\displaystyle\\quad\\sigma\_\{hi\}:=∏ℓ=1Lσh​i\(ℓ\)\.\\displaystyle=\\prod\_\{\\ell=1\}^\{L\}\\sigma\_\{hi\}^\{\(\\ell\)\}\.
SinceM=∑h=1HΩhM=\\sum\_\{h=1\}^\{H\}\\Omega\_\{h\}and allΩh\\Omega\_\{h\}share the same singular vectors\(U,V\)\(U,V\), the total map is also diagonal in the same basis:

M\\displaystyle M\\;=U​\(∑h=1HΣh\)​V⊤\\displaystyle=\\;U\\Big\(\\sum\_\{h=1\}^\{H\}\\Sigma\_\{h\}\\Big\)V^\{\\top\}⟹σi​\(M\)\\displaystyle\\Longrightarrow\\quad\\sigma\_\{i\}\(M\)=∑h=1Hσh​ifor each mode​i∈\[d\]\.\\displaystyle=\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\\quad\\text\{for each mode \}i\\in\[d\]\.
Therefore the loss decomposes into a sum of independent*mode losses*

ℒ\(Θ\)=12∥M−M⋆∥F2=12∑i=1d\(∑h=1Hσh​i−σ⋆i\)2=:∑i=1dℒi\.\\mathcal\{L\}\(\\Theta\)=\\frac\{1\}\{2\}\\big\\\|M\-M\_\{\\star\}\\big\\\|\_\{F\}^\{2\}=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{d\}\\Big\(\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\}\\Big\)^\{2\}=:\\sum\_\{i=1\}^\{d\}\\mathcal\{L\}\_\{i\}\.\(37\)This SVS decoupling implies that the Hessian decomposes \(up to permutation\) into blocks indexed by modesii, and the dominant positive\-curvature directions can be analyzed mode\-by\-mode\.

### C\.3Mode\-iiHessian Eigenvalue on the Depth\-Balanced Manifold

We refine the Hessian expression starting from

𝐇h,k=∇vec​\(Θh\)∇vec​\(Θk\)⊤⁡ℒ≈Ah⊤​Ak\\mathbf\{H\}\_\{h,k\}=\\nabla\_\{\\mathrm\{vec\}\(\\Theta\_\{h\}\)\}\\nabla\_\{\\mathrm\{vec\}\(\\Theta\_\{k\}\)\}^\{\\top\}\\mathcal\{L\}\\approx A\_\{h\}^\{\\top\}A\_\{k\}\\qquadat a global minimum​\(M=M⋆\)\\text\{at a global minimum \}\(M=M\_\{\\star\}\), whereAhA\_\{h\}is the Jacobian ofvec​\(Ωh\)\\mathrm\{vec\}\(\\Omega\_\{h\}\)w\.r\.t\.vec​\(Θh\)\\mathrm\{vec\}\(\\Theta\_\{h\}\)\. Inside the SVS set, the Jacobian further decomposes across modes\. Concretely, the scalar residual for modeiiis

ri:=∑h=1Hσh​i−σ⋆i,r\_\{i\}:=\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\},soℒi=12​ri2\\mathcal\{L\}\_\{i\}=\\tfrac\{1\}\{2\}r\_\{i\}^\{2\}and, at any global minimum,ri=0r\_\{i\}=0for allii\. Hence the mode\-iiHessian is a Gram matrix of the mode\-iiJacobian:

∇2ℒi\|ri=0=Ji⊤​Ji\.\\nabla^\{2\}\\mathcal\{L\}\_\{i\}\\big\|\_\{r\_\{i\}=0\}=J\_\{i\}^\{\\top\}J\_\{i\}\.
#### Depth\-wise balance\.

We now specialize to the*depth\-balanced*manifold within each pathway \(consistent with balanced initialization and the “balanced minimum” analyzed inGhoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)\):

σh​i\(ℓ\)=σh​i1/L∀h∈\[H\],ℓ∈\[L\],i∈\[d\]\.\\sigma\_\{hi\}^\{\(\\ell\)\}\\;=\\;\\sigma\_\{hi\}^\{1/L\}\\qquad\\forall\\,h\\in\[H\],\\;\\ell\\in\[L\],\\;i\\in\[d\]\.\(38\)That is, depth\-wise the singular value for a fixed modeiiis equally split across layers, while*pathway\-wise*the end\-to\-end valuesσh​i\\sigma\_\{hi\}may be unbalanced acrosshh\.

#### Jacobian in modeii\.

Consider the coordinates\{σh​i\(ℓ\)\}h,ℓ\\\{\\sigma\_\{hi\}^\{\(\\ell\)\}\\\}\_\{h,\\ell\}in modeii\. Sinceσh​i=∏ℓ=1Lσh​i\(ℓ\)\\sigma\_\{hi\}=\\prod\_\{\\ell=1\}^\{L\}\\sigma\_\{hi\}^\{\(\\ell\)\}, we have

∂σh​i∂σh​i\(ℓ\)=σh​iσh​i\(ℓ\)​=\([38](https://arxiv.org/html/2606.05219#A3.E38)\)​σh​iσh​i1/L=σh​i1−1L\.\\frac\{\\partial\\sigma\_\{hi\}\}\{\\partial\\sigma\_\{hi\}^\{\(\\ell\)\}\}=\\frac\{\\sigma\_\{hi\}\}\{\\sigma\_\{hi\}^\{\(\\ell\)\}\}\\overset\{\\eqref\{eq:depth\-balance\}\}\{=\}\\frac\{\\sigma\_\{hi\}\}\{\\sigma\_\{hi\}^\{1/L\}\}=\\sigma\_\{hi\}^\{1\-\\frac\{1\}\{L\}\}\.Becauseri=∑hσh​i−σ⋆ir\_\{i\}=\\sum\_\{h\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\}, the mode\-iiJacobian entries are

∂ri∂σh​i\(ℓ\)=∂σh​i∂σh​i\(ℓ\)=σh​i1−1L\.\\frac\{\\partial r\_\{i\}\}\{\\partial\\sigma\_\{hi\}^\{\(\\ell\)\}\}=\\frac\{\\partial\\sigma\_\{hi\}\}\{\\partial\\sigma\_\{hi\}^\{\(\\ell\)\}\}=\\sigma\_\{hi\}^\{1\-\\frac\{1\}\{L\}\}\.Thus, restricted to these depth\-balanced SVS coordinates, the mode\-iiJacobian vector is

Ji=\(\\displaystyle J\_\{i\}\\;=\\;\\big\(σ1​i1−1/L,…,σ1​i1−1/L⏟L​entries,σ2​i1−1/L,…,σ2​i1−1/L⏟L​entries,…,σH​i1−1/L,…,σH​i1−1/L⏟L​entries\),\\displaystyle\\underbrace\{\\sigma\_\{1i\}^\{1\-1/L\},\\dots,\\sigma\_\{1i\}^\{1\-1/L\}\}\_\{L\\text\{ entries\}\},\\;\\underbrace\{\\sigma\_\{2i\}^\{1\-1/L\},\\dots,\\sigma\_\{2i\}^\{1\-1/L\}\}\_\{L\\text\{ entries\}\},\\dots,\\underbrace\{\\sigma\_\{Hi\}^\{1\-1/L\},\\dots,\\sigma\_\{Hi\}^\{1\-1/L\}\}\_\{L\\text\{ entries\}\}\\big\),
and therefore its squared norm is

‖Ji‖22=∑h=1H∑ℓ=1L\(σh​i1−1/L\)2=L​∑h=1Hσh​i2−2L\.\\\|J\_\{i\}\\\|\_\{2\}^\{2\}\\;=\\;\\sum\_\{h=1\}^\{H\}\\sum\_\{\\ell=1\}^\{L\}\\big\(\\sigma\_\{hi\}^\{1\-1/L\}\\big\)^\{2\}\\;=\\;L\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}^\{\\,2\-\\frac\{2\}\{L\}\}\.\(39\)
###### Lemma C\.1\(Mode\-wise dominant eigenvalue\)\.

At any global minimum within the SVS set, and restricted to the depth\-balanced manifold \([38](https://arxiv.org/html/2606.05219#A3.E38)\), the Hessian contribution from modeiihas a single nonzero eigenvalue given by

λi=L​∑h=1Hσh​i2−2L,\\lambda\_\{i\}\\;=\\;L\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}^\{\\,2\-\\frac\{2\}\{L\}\},\(40\)with eigenvector proportional to the mode\-iiJacobian directionJiJ\_\{i\}\.

###### Proof\.

At a global minimumri=0r\_\{i\}=0, we have∇2ℒi=Ji⊤​Ji\\nabla^\{2\}\\mathcal\{L\}\_\{i\}=J\_\{i\}^\{\\top\}J\_\{i\}, which is rank\-one on the considered coordinates\. Hence its only nonzero eigenvalue equals‖Ji‖22\\\|J\_\{i\}\\\|\_\{2\}^\{2\}, which is \([39](https://arxiv.org/html/2606.05219#A3.E39)\)\. ∎

### C\.4Pathway\-Wise Balancing Minimizes Sharpness in Each Mode

Lemma[C\.1](https://arxiv.org/html/2606.05219#A3.Thmtheorem1)shows that, for a fixed modeii, the sharpness contribution depends on the pathway end\-to\-end singular values\{σh​i\}h=1H\\\{\\sigma\_\{hi\}\\\}\_\{h=1\}^\{H\}through the convex functional∑hσh​ip\\sum\_\{h\}\\sigma\_\{hi\}^\{p\}with

p:=2−2L\.p\\;:=\\;2\-\\frac\{2\}\{L\}\.ForL\>2L\>2, we havep\>1p\>1, sox↦xpx\\mapsto x^\{p\}is convex onℝ\+\\mathbb\{R\}\_\{\+\}\.

At a global minimum, modeiimust satisfy the constraint \(from \([37](https://arxiv.org/html/2606.05219#A3.E37)\)\)

∑h=1Hσh​i=σ⋆i\.\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\\;=\\;\\sigma\_\{\\star i\}\.\(41\)We now optimize \([40](https://arxiv.org/html/2606.05219#A3.E40)\) subject to \([41](https://arxiv.org/html/2606.05219#A3.E41)\)\.

###### Theorem C\.2\(Pathway\-wise equal split minimizes mode\-iisharpness\)\.

AssumeL\>2L\>2\(sop\>1p\>1\)\. Among all global minima satisfying \([41](https://arxiv.org/html/2606.05219#A3.E41)\), the mode\-iieigenvalue \([40](https://arxiv.org/html/2606.05219#A3.E40)\) is minimized when the pathway contributions are equal:

σ1​i=⋯=σH​i=σ⋆iH\.\\sigma\_\{1i\}=\\cdots=\\sigma\_\{Hi\}=\\frac\{\\sigma\_\{\\star i\}\}\{H\}\.In that case,

λimin=L​H2L−1​σ⋆i2−2L\.\\lambda\_\{i\}^\{\\min\}\\;=\\;L\\,H^\{\\frac\{2\}\{L\}\-1\}\\,\\sigma\_\{\\star i\}^\{\\,2\-\\frac\{2\}\{L\}\}\.\(42\)

###### Proof\.

By Jensen’s inequality for the convex functionxpx^\{p\}\(withp\>1p\>1\),

1H​∑h=1Hσh​ip≥\(1H​∑h=1Hσh​i\)p=\(σ⋆iH\)p,\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}^\{p\}\\;\\geq\\;\\left\(\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\\right\)^\{p\}=\\left\(\\frac\{\\sigma\_\{\\star i\}\}\{H\}\\right\)^\{p\},where we used the constraint \([41](https://arxiv.org/html/2606.05219#A3.E41)\)\. Multiplying byL​HLHyields

λi=L​∑h=1Hσh​ip≥L​H​\(σ⋆iH\)p,\\lambda\_\{i\}=L\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}^\{p\}\\;\\geq\\;LH\\left\(\\frac\{\\sigma\_\{\\star i\}\}\{H\}\\right\)^\{p\},with equality iffσ1​i=⋯=σH​i=σ⋆i/H\\sigma\_\{1i\}=\\cdots=\\sigma\_\{Hi\}=\\sigma\_\{\\star i\}/H\. Substitutingp=2−2Lp=2\-\\frac\{2\}\{L\}gives \([42](https://arxiv.org/html/2606.05219#A3.E42)\)\. ∎

#### Improvement over a “single\-path” allocation\.

Consider the extreme \(maximally unbalanced\) allocation that places all of modeiiinto one pathway:

σ1​i=σ⋆i,σ2​i=⋯=σH​i=0\.\\sigma\_\{1i\}=\\sigma\_\{\\star i\},\\qquad\\sigma\_\{2i\}=\\cdots=\\sigma\_\{Hi\}=0\.Then \([40](https://arxiv.org/html/2606.05219#A3.E40)\) gives

λione​path=L​σ⋆i2−2L\.\\lambda\_\{i\}^\{\\mathrm\{one\\ path\}\}=L\\,\\sigma\_\{\\star i\}^\{\\,2\-\\frac\{2\}\{L\}\}\.Comparing with the minimum \([42](https://arxiv.org/html/2606.05219#A3.E42)\), we obtain the reduction factor

λiminλione​path=L​H​\(σ⋆iH\)2−2LL​σ⋆i2−2L=H2L−1\.\\frac\{\\lambda\_\{i\}^\{\\min\}\}\{\\lambda\_\{i\}^\{\\mathrm\{one\\ path\}\}\}=\\frac\{L\\,H\\left\(\\frac\{\\sigma\_\{\\star i\}\}\{H\}\\right\)^\{2\-\\frac\{2\}\{L\}\}\}\{L\\,\\sigma\_\{\\star i\}^\{\\,2\-\\frac\{2\}\{L\}\}\}=H^\{\\frac\{2\}\{L\}\-1\}\.\(43\)Thus, forL≥3L\\geq 3, distributing a mode evenly acrossHHpathways strictly reduces its curvature by a factor\(H\)2L−1\\left\(H\\right\)^\{\\frac\{2\}\{L\}\-1\}relative to concentrating it in a single pathway\.

### C\.5From Mode\-wise Eigenvalues to Global Sharpness

The full Hessian \(restricted to the SVS\) decomposes into independent mode contributions\. Each modeiicontributes a dominant positive eigenvalueλi\\lambda\_\{i\}given by \([40](https://arxiv.org/html/2606.05219#A3.E40)\), while additional eigenvalues correspond to symmetry/gauge directions \(often yielding zero eigenvalues\) and to non\-dominant curvature within each mode\. Consequently, the sharpness satisfies

λmax=maxi∈\[d\]⁡λi\.\\lambda\_\{\\max\}\\;=\\;\\max\_\{i\\in\[d\]\}\\lambda\_\{i\}\.Under the common orderingσ⋆1≥σ⋆2≥⋯≥σ⋆d\\sigma\_\{\\star 1\}\\geq\\sigma\_\{\\star 2\}\\geq\\cdots\\geq\\sigma\_\{\\star d\}, the minimized mode\-wise eigenvalues \([42](https://arxiv.org/html/2606.05219#A3.E42)\) are monotone inσ⋆i\\sigma\_\{\\star i\}, hence the largest eigenvalue is achieved by the top target mode:

λmax=λ1min=L​H2L−1​σ⋆12−2L\.\\lambda\_\{\\max\}=\\lambda\_\{1\}^\{\\min\}=L\\,H^\{\\frac\{2\}\{L\}\-1\}\\,\\sigma\_\{\\star 1\}^\{\\,2\-\\frac\{2\}\{L\}\}\.\(44\)Moreover, the remaining dominant eigenvalues align with the remaining target modes: for eachii, the curvature scale is set byσ⋆i\\sigma\_\{\\star i\}through \([42](https://arxiv.org/html/2606.05219#A3.E42)\)\. In particular, whenL\>2L\>2, increasing the number of pathwaysHHreducesλmax\\lambda\_\{\\max\}asH2L−1H^\{\\frac\{2\}\{L\}\-1\}, thereby extending the edge\-of\-stability regime to larger learning rates\.

### C\.6Proof of Proposition[5\.1](https://arxiv.org/html/2606.05219#S5.Thmtheorem1)

We extend the mode\-wise SVS analysis to the case where pathwayhhhas \(possibly different\) depthLhL\_\{h\}\. Recall that, in modeii, each pathway contributes an end\-to\-end singular value

σh​i=∏ℓ=1Lhσh​i\(ℓ\),\\sigma\_\{hi\}\\;=\\;\\prod\_\{\\ell=1\}^\{L\_\{h\}\}\\sigma\_\{hi\}^\{\(\\ell\)\},and the mode\-iiresidual and loss are

ri:=∑h=1Hσh​i−σ⋆i,ℒi=12​ri2\.r\_\{i\}\\;:=\\;\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\},\\qquad\\mathcal\{L\}\_\{i\}\\;=\\;\\tfrac\{1\}\{2\}r\_\{i\}^\{2\}\.At any global minimum we haveri=0r\_\{i\}=0, hence \(as in the homogeneous\-depth case\)

∇2ℒi\|ri=0=Ji⊤​Ji,\\nabla^\{2\}\\mathcal\{L\}\_\{i\}\\big\|\_\{r\_\{i\}=0\}\\;=\\;J\_\{i\}^\{\\top\}J\_\{i\},whereJiJ\_\{i\}denotes the Jacobian ofrir\_\{i\}with respect to the mode\-iicoordinates\.

#### Depth\-balanced manifold with heterogeneous depth\.

Assume we restrict to the depth\-balanced manifold*within each pathway*:

σh​i\(ℓ\)=σh​i1/Lh∀h∈\[H\],ℓ∈\[Lh\],i∈\[d\]\.\\sigma\_\{hi\}^\{\(\\ell\)\}\\;=\\;\\sigma\_\{hi\}^\{1/L\_\{h\}\}\\qquad\\forall\\,h\\in\[H\],\\;\\ell\\in\[L\_\{h\}\],\\;i\\in\[d\]\.\(45\)That is, for fixed\(h,i\)\(h,i\), the contribution of modeiiis evenly split across theLhL\_\{h\}layers of pathwayhh\.

#### Mode\-iiJacobian entries\.

Fix a pathwayhh\. Sinceσh​i=∏ℓ=1Lhσh​i\(ℓ\)\\sigma\_\{hi\}=\\prod\_\{\\ell=1\}^\{L\_\{h\}\}\\sigma\_\{hi\}^\{\(\\ell\)\}, we have for eachℓ∈\[Lh\]\\ell\\in\[L\_\{h\}\]

∂σh​i∂σh​i\(ℓ\)=σh​iσh​i\(ℓ\)​=\([45](https://arxiv.org/html/2606.05219#A3.E45)\)​σh​iσh​i1/Lh=σh​i1−1Lh\.\\frac\{\\partial\\sigma\_\{hi\}\}\{\\partial\\sigma\_\{hi\}^\{\(\\ell\)\}\}\\;=\\;\\frac\{\\sigma\_\{hi\}\}\{\\sigma\_\{hi\}^\{\(\\ell\)\}\}\\;\\overset\{\\eqref\{eq:depth\-balance\-hetero\}\}\{=\}\\;\\frac\{\\sigma\_\{hi\}\}\{\\sigma\_\{hi\}^\{1/L\_\{h\}\}\}\\;=\\;\\sigma\_\{hi\}^\{\\,1\-\\frac\{1\}\{L\_\{h\}\}\}\.Becauseri=∑h′σh′​i−σ⋆ir\_\{i\}=\\sum\_\{h^\{\\prime\}\}\\sigma\_\{h^\{\\prime\}i\}\-\\sigma\_\{\\star i\}depends onσh​i\\sigma\_\{hi\}only through this additive term, it follows that

∂ri∂σh​i\(ℓ\)=∂σh​i∂σh​i\(ℓ\)=σh​i1−1Lh∀ℓ∈\[Lh\]\.\\frac\{\\partial r\_\{i\}\}\{\\partial\\sigma\_\{hi\}^\{\(\\ell\)\}\}\\;=\\;\\frac\{\\partial\\sigma\_\{hi\}\}\{\\partial\\sigma\_\{hi\}^\{\(\\ell\)\}\}\\;=\\;\\sigma\_\{hi\}^\{\\,1\-\\frac\{1\}\{L\_\{h\}\}\}\\qquad\\forall\\,\\ell\\in\[L\_\{h\}\]\.Therefore, in the coordinate system\{σh​i\(ℓ\)\}h,ℓ\\\{\\sigma\_\{hi\}^\{\(\\ell\)\}\\\}\_\{h,\\ell\}, the mode\-iiJacobian vectorJiJ\_\{i\}consists ofLhL\_\{h\}repeated entries of magnitudeσh​i1−1/Lh\\sigma\_\{hi\}^\{1\-1/L\_\{h\}\}for each pathwayhh\.

#### Rank\-one Hessian and its dominant eigenvalue\.

Since∇2ℒi\|ri=0=Ji⊤​Ji\\nabla^\{2\}\\mathcal\{L\}\_\{i\}\|\_\{r\_\{i\}=0\}=J\_\{i\}^\{\\top\}J\_\{i\}is a Gram matrix of a single vector, it is rank\-one on these coordinates, with the unique nonzero eigenvalue equal to‖Ji‖22\\\|J\_\{i\}\\\|\_\{2\}^\{2\}\. Using the structure above,

‖Ji‖22\\displaystyle\\\|J\_\{i\}\\\|\_\{2\}^\{2\}=∑h=1H∑ℓ=1Lh\(σh​i1−1Lh\)2=∑h=1HLh​σh​i2−2Lh\.\\displaystyle=\\sum\_\{h=1\}^\{H\}\\sum\_\{\\ell=1\}^\{L\_\{h\}\}\\Big\(\\sigma\_\{hi\}^\{\\,1\-\\frac\{1\}\{L\_\{h\}\}\}\\Big\)^\{2\}=\\sum\_\{h=1\}^\{H\}L\_\{h\}\\,\\sigma\_\{hi\}^\{\\,2\-\\frac\{2\}\{L\_\{h\}\}\}\.Hence, on the depth\-balanced manifold, the dominant \(and only nonzero\) eigenvalue contributed by modeiiis

λi=∑h=1HLh​σh​i2−2Lh,\\lambda\_\{i\}\\;=\\;\\sum\_\{h=1\}^\{H\}L\_\{h\}\\,\\sigma\_\{hi\}^\{2\-\\frac\{2\}\{L\_\{h\}\}\},with eigenvector proportional toJiJ\_\{i\}\. This proves Proposition[5\.1](https://arxiv.org/html/2606.05219#S5.Thmtheorem1)\.

## Appendix DStability Constraints Imply Drift Toward Pathway\-Balanced Minima

#### Background: self\-stabilization as constrained minimization\.

Self\-stabilization theory\(Damianet al\.,[2023](https://arxiv.org/html/2606.05219#bib.bib18)\)argues that, near the edge of stability, gradient descent \(GD\) implicitly tracks a projected GD trajectory that solves a sharpness\-constrained problem:

minΘ⁡ℒ​\(Θ\)s\.t\.S​\(Θ\)≤2η,\\min\_\{\\Theta\}\\;\\mathcal\{L\}\(\\Theta\)\\quad\\text\{s\.t\.\}\\quad S\(\\Theta\)\\leq\\frac\{2\}\{\\eta\},\(46\)whereS​\(Θ\)=λmax​\(∇2ℒ​\(Θ\)\)S\(\\Theta\)=\\lambda\_\{\\max\}\(\\nabla^\{2\}\\mathcal\{L\}\(\\Theta\)\)is the sharpness andη\\etais the step size\. \(SeeDamianet al\.\([2023](https://arxiv.org/html/2606.05219#bib.bib18)\)for a formal projected\-dynamics statement\.\)

#### Our setting\.

We consider the SVS regime, where all layers share the singular vectors of the targetM⋆=V​Σ⋆​V⊤M\_\{\\star\}=V\\Sigma\_\{\\star\}V^\{\\top\}, so the dynamics decouple across modesi∈\[d\]i\\in\[d\]and

ℒ\(Θ\)=12∑i=1d\(∑h=1Hσh​i−σ⋆i\)2=:∑i=1dℒi\.\\mathcal\{L\}\(\\Theta\)=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{d\}\\Big\(\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}\-\\sigma\_\{\\star i\}\\Big\)^\{2\}=:\\sum\_\{i=1\}^\{d\}\\mathcal\{L\}\_\{i\}\.\(47\)We also restrict to the depth\-balanced manifold within each pathway \(homogeneous depthLL\):σh​i\(ℓ\)=σh​i1/L\\sigma\_\{hi\}^\{\(\\ell\)\}=\\sigma\_\{hi\}^\{1/L\}for allh,ℓ,ih,\\ell,i\.

### D\.1Local GD stability at a global minimum equals a sharpness bound

LetΘ⋆\\Theta\_\{\\star\}be any global minimum within SVS and depth\-balance, so∑hσh​i=σ⋆i\\sum\_\{h\}\\sigma\_\{hi\}=\\sigma\_\{\\star i\}for allii\. The mode\-iiHessian contribution is rank\-one on the SVS depth\-balanced coordinates, with the single nonzero eigenvalue

λi​\(Θ⋆\)=L​∑h=1Hσh​i2−2L\.\\lambda\_\{i\}\(\\Theta\_\{\\star\}\)=L\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}^\{\\,2\-\\frac\{2\}\{L\}\}\.\(48\)\(Equivalently,∇2ℒi​\(Θ⋆\)=Ji⊤​Ji\\nabla^\{2\}\\mathcal\{L\}\_\{i\}\(\\Theta\_\{\\star\}\)=J\_\{i\}^\{\\top\}J\_\{i\}with‖Ji‖22=λi\\\|J\_\{i\}\\\|\_\{2\}^\{2\}=\\lambda\_\{i\}\.\)

###### Proposition D\.1\(GD stability⇔\\Leftrightarrowsharpness constraint\)\.

Consider full\-batch GDΘt\+1=Θt−η​∇ℒ​\(Θt\)\\Theta\_\{t\+1\}=\\Theta\_\{t\}\-\\eta\\nabla\\mathcal\{L\}\(\\Theta\_\{t\}\)\. At a global minimumΘ⋆\\Theta\_\{\\star\}\(SVS \+ depth\-balance\), the linearized GD map has eigenvalues1−η​λi​\(Θ⋆\)1\-\\eta\\lambda\_\{i\}\(\\Theta\_\{\\star\}\)on each mode\-iicurvature direction and eigenvalue11on the flat directions\. HenceΘ⋆\\Theta\_\{\\star\}is \(linearly\) stable iff

η​λmax​\(Θ⋆\)<2,λmax​\(Θ⋆\):=maxi⁡λi​\(Θ⋆\),\\eta\\,\\lambda\_\{\\max\}\(\\Theta\_\{\\star\}\)<2,\\qquad\\lambda\_\{\\max\}\(\\Theta\_\{\\star\}\):=\\max\_\{i\}\\lambda\_\{i\}\(\\Theta\_\{\\star\}\),\(49\)i\.e\., iffS​\(Θ⋆\)≤2/ηS\(\\Theta\_\{\\star\}\)\\leq 2/\\eta\.

###### Proof\.

Linearizing GD atΘ⋆\\Theta\_\{\\star\}yieldsδ​Θt\+1=\(I−η​∇2ℒ​\(Θ⋆\)\)​δ​Θt\\delta\\Theta\_\{t\+1\}=\(I\-\\eta\\nabla^\{2\}\\mathcal\{L\}\(\\Theta\_\{\\star\}\)\)\\delta\\Theta\_\{t\}\. In SVS \+ depth\-balance, the Hessian is block\-diagonal across modes, and each mode\-iiblock is rank\-one with eigenvalueλi\\lambda\_\{i\}in \([48](https://arxiv.org/html/2606.05219#A4.E48)\)\. Stability on a positive\-curvature direction requires\|1−η​λi\|<1\|1\-\\eta\\lambda\_\{i\}\|<1, i\.e\.,0<η​λi<20<\\eta\\lambda\_\{i\}<2\. Taking the maximum overiigives \([49](https://arxiv.org/html/2606.05219#A4.E49)\)\. ∎

### D\.2Why the stability constraint selects pathway\-balanced minima

For fixed modeiiand homogeneous depthL\>2L\>2, definep:=2−2L\>1p:=2\-\\frac\{2\}\{L\}\>1\. At any global minimum, the pathway end\-to\-end singular values satisfy the conservation constraint∑h=1Hσh​i=σ⋆i\\sum\_\{h=1\}^\{H\}\\sigma\_\{hi\}=\\sigma\_\{\\star i\}, while the mode\-iisharpness is proportional to∑hσh​ip\\sum\_\{h\}\\sigma\_\{hi\}^\{p\}by \([48](https://arxiv.org/html/2606.05219#A4.E48)\)\.

###### Theorem D\.2\(Balanced split is the flattest global minimum in each mode\)\.

AssumeL\>2L\>2\. Among all global minima satisfying∑hσh​i=σ⋆i\\sum\_\{h\}\\sigma\_\{hi\}=\\sigma\_\{\\star i\}, the mode\-iisharpnessλi\\lambda\_\{i\}is minimized uniquely by the equal split

σ1​i=⋯=σH​i=σ⋆iH,λimin=L​H2L−1​\(σ⋆i\)2−2L\.\\sigma\_\{1i\}=\\cdots=\\sigma\_\{Hi\}=\\frac\{\\sigma\_\{\\star i\}\}\{H\},\\qquad\\lambda\_\{i\}^\{\\min\}=L\\,H^\{\\frac\{2\}\{L\}\-1\}\\,\(\\sigma\_\{\\star i\}\)^\{\\,2\-\\frac\{2\}\{L\}\}\.\(50\)

###### Proof\.

Sincep\>1p\>1,x↦xpx\\mapsto x^\{p\}is convex onℝ\+\\mathbb\{R\}\_\{\+\}\. By Jensen,1H​∑hσh​ip≥\(1H​∑hσh​i\)p=\(σ⋆i/H\)p\\frac\{1\}\{H\}\\sum\_\{h\}\\sigma\_\{hi\}^\{p\}\\geq\\big\(\\frac\{1\}\{H\}\\sum\_\{h\}\\sigma\_\{hi\}\\big\)^\{p\}=\(\\sigma\_\{\\star i\}/H\)^\{p\}, with equality iff allσh​i\\sigma\_\{hi\}are equal\. Multiply both sides byL​HLHand substitutep=2−2/Lp=2\-2/L\. ∎

#### Consequence \(stability\-driven drift\)\.

Combine Proposition[D\.1](https://arxiv.org/html/2606.05219#A4.Thmtheorem1)and Theorem[D\.2](https://arxiv.org/html/2606.05219#A4.Thmtheorem2)\. If the stepsize satisfies

2λione\-path<η<2λimin,λione\-path:=L​\(σ⋆i\)2−2L,\\frac\{2\}\{\\lambda\_\{i\}^\{\\text\{one\-path\}\}\}<\\eta<\\frac\{2\}\{\\lambda\_\{i\}^\{\\min\}\},\\qquad\\lambda\_\{i\}^\{\\text\{one\-path\}\}:=L\(\\sigma\_\{\\star i\}\)^\{\\,2\-\\frac\{2\}\{L\}\},\(51\)then any sharp “winner\-takes\-all” minimum \(single\-path allocation\) is unstable, while the equal\-split minimum is stable\. Therefore, once GD enters a neighborhood where modeiidominates the curvature, stability constraints alone rule out convergence to sparse allocations and select balanced allocations as the only stable zero\-loss solutions\. This provides a direct \(assumption\-free\) mechanism linking “stability constraints” to drift toward pathway\-balanced, flatter minima in our multi\-path DLN\.

## Appendix EDecay of a Pathway Balancing Gap at the Edge of Stability

This appendix provides a local mechanism explaining why large\-step GD reduces pathway imbalance near the balanced minimum\. The analysis is carried out on the SVS depth\-balanced manifold, where each singular mode decouples and the multi\-pathway dynamics reduce to scalar recursions for the per\-layer variablesah​i​\(t\)a\_\{hi\}\(t\)\. We first establish the invariance of this reduction and introduce gradient\-flow leaf coordinates that separate the balanced direction from the pathway\-imbalance directions\. In these coordinates, the balanced minimum is the unique local minimizer of terminal sharpness, and deviations from the balanced leaf increase sharpness quadratically\.

The main part of the appendix analyzes how discrete GD moves across these gradient\-flow leaves in the Edge\-of\-Stability regime\. We begin with a sign\-free one\-step contraction result in a strong\-safety neighborhood\. We then prove the central two\-step result: under a local self\-stabilization condition stating that the sharpness remains bracketed near the stability threshold, the residual normal form implies sign alternation and yields a two\-step contraction of the pathway balancing gap, with an explicit quadratic correction\. Thus sign alternation is not imposed as an independent hypothesis; it follows from the local EoS normal form together with sharpness bracketing\. Finally, we translate the two\-step contraction into a formal slow\-time central\-flow description, where the drift is active only when the sharpness excess\(η​λ−2\)\+\(\\eta\\lambda\-2\)\_\{\+\}is positive, and we state precisely which conclusions are rigorous local statements and which remain formal asymptotics\.

### E\.1Invariance of SVS and preservation of depth\-wise balance

We extend the SVS invariance argument ofGhoshet al\.\([2025](https://arxiv.org/html/2606.05219#bib.bib21)\)\(see also their deferred SVS proofs\) to the multi\-pathway architectureM=∑h=1HΩhM=\\sum\_\{h=1\}^\{H\}\\Omega\_\{h\}withΩh=∏ℓ=1LWh​ℓ\\Omega\_\{h\}=\\prod\_\{\\ell=1\}^\{L\}W\_\{h\\ell\}\.

###### Lemma E\.1\(SVS is invariant under diagonal \(SVS\) initialization\)\.

Assume the target isM⋆=V​Σ⋆​V⊤M\_\{\\star\}=V\\Sigma\_\{\\star\}V^\{\\top\}and the initialization satisfiesWh​ℓ​\(0\)=V​Σh​ℓ​\(0\)​V⊤W\_\{h\\ell\}\(0\)=V\\Sigma\_\{h\\ell\}\(0\)V^\{\\top\}with allΣh​ℓ​\(0\)\\Sigma\_\{h\\ell\}\(0\)diagonal \(in particular, scalar multiples of the identity, as in our initialization\)\. Then for all GD iteratest≥0t\\geq 0,

Wh​ℓ​\(t\)=V​Σh​ℓ​\(t\)​V⊤withΣh​ℓ​\(t\)​diagonal,W\_\{h\\ell\}\(t\)=V\\Sigma\_\{h\\ell\}\(t\)V^\{\\top\}\\quad\\text\{with\}\\quad\\Sigma\_\{h\\ell\}\(t\)\\ \\text\{diagonal\},\(52\)so the dynamics remain in the SVS set and decouple on singular values\.

###### Proof\.

Assume inductively thatWh​ℓ​\(t\)=V​Σh​ℓ​\(t\)​V⊤W\_\{h\\ell\}\(t\)=V\\Sigma\_\{h\\ell\}\(t\)V^\{\\top\}with diagonalΣh​ℓ​\(t\)\\Sigma\_\{h\\ell\}\(t\)for allh,ℓh,\\ell\. Each pathway product isΩh​\(t\)=V​\(∏ℓΣh​ℓ​\(t\)\)​V⊤\\Omega\_\{h\}\(t\)=V\\bigl\(\\prod\_\{\\ell\}\\Sigma\_\{h\\ell\}\(t\)\\bigr\)V^\{\\top\}, hence the total map

M​\(t\)=∑h=1HΩh​\(t\)=V​\(∑h=1H∏ℓ=1LΣh​ℓ​\(t\)\)​V⊤M\(t\)=\\sum\_\{h=1\}^\{H\}\\Omega\_\{h\}\(t\)=V\\Bigl\(\\sum\_\{h=1\}^\{H\}\\prod\_\{\\ell=1\}^\{L\}\\Sigma\_\{h\\ell\}\(t\)\\Bigr\)V^\{\\top\}is diagonal in the same basis\. The residualR​\(t\):=M​\(t\)−M⋆R\(t\):=M\(t\)\-M\_\{\\star\}therefore satisfiesR​\(t\)=V​Δ​\(t\)​V⊤R\(t\)=V\\Delta\(t\)V^\{\\top\}with diagonalΔ​\(t\)\\Delta\(t\)\. For the squared Frobenius loss,

∇Wh​ℓℒ​\(Θ​\(t\)\)=Wh,L:ℓ\+1​\(t\)⊤​R​\(t\)​Wh,ℓ−1:1​\(t\)⊤\.\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}\(\\Theta\(t\)\)=W\_\{h,L:\\ell\+1\}\(t\)^\{\\top\}R\(t\)\\,W\_\{h,\\ell\-1:1\}\(t\)^\{\\top\}\.Each factor is diagonal in theVVbasis, so∇Wh​ℓℒ=V​Gh​ℓ​V⊤\\nabla\_\{W\_\{h\\ell\}\}\\mathcal\{L\}=VG\_\{h\\ell\}V^\{\\top\}with diagonalGh​ℓG\_\{h\\ell\}\. The GD update preserves the diagonal SVS form\. ∎

###### Lemma E\.2\(Depth\-wise balance within each pathway is preserved\)\.

Assume homogeneous depthLLandσh​i\(1\)​\(t\)=⋯=σh​i\(L\)​\(t\)\\sigma\_\{hi\}^\{\(1\)\}\(t\)=\\cdots=\\sigma\_\{hi\}^\{\(L\)\}\(t\)for everyh,ih,iat some iteratett\. Then the same equalities hold att\+1t\+1\.

###### Proof\.

By Lemma[E\.1](https://arxiv.org/html/2606.05219#A5.Thmtheorem1), the dynamics reduce to diagonal singular values\. Fix a modeiiand letxh​ℓ:=σh​i\(ℓ\)​\(t\)x\_\{h\\ell\}:=\\sigma\_\{hi\}^\{\(\\ell\)\}\(t\),sh:=∏ℓxh​ℓs\_\{h\}:=\\prod\_\{\\ell\}x\_\{h\\ell\}\. The mode\-iiresidual isr:=∑hsh−σ⋆ir:=\\sum\_\{h\}s\_\{h\}\-\\sigma\_\{\\star i\}andℒi=12​r2\\mathcal\{L\}\_\{i\}=\\tfrac\{1\}\{2\}r^\{2\}\. By the product rule,∂ℒi/∂xh​ℓ=r⋅sh/xh​ℓ=r​∏j≠ℓxh​j\\partial\\mathcal\{L\}\_\{i\}/\\partial x\_\{h\\ell\}=r\\cdot s\_\{h\}/x\_\{h\\ell\}=r\\prod\_\{j\\neq\\ell\}x\_\{hj\}\. Ifxh​1=⋯=xh​Lx\_\{h1\}=\\cdots=x\_\{hL\}, then∏j≠ℓxh​j\\prod\_\{j\\neq\\ell\}x\_\{hj\}is independent ofℓ\\ell, so every layer in pathwayhhreceives the identical mode\-iiupdate\. Depth\-wise balance is preserved\. ∎

### E\.2Mode\-wise scalar dynamics on the depth\-balanced manifold

By Lemma[E\.2](https://arxiv.org/html/2606.05219#A5.Thmtheorem2), for each modeiiand pathwayhhwe can writeσh​i\(ℓ\)​\(t\)=ah​i​\(t\)\\sigma\_\{hi\}^\{\(\\ell\)\}\(t\)=a\_\{hi\}\(t\)for allℓ∈\[L\]\\ell\\in\[L\], so the end\-to\-end pathway singular value isσh​i​\(t\)=ah​i​\(t\)L\\sigma\_\{hi\}\(t\)=a\_\{hi\}\(t\)^\{L\}\. Let

ri​\(t\):=∑h=1Hah​i​\(t\)L−σ⋆i\.r\_\{i\}\(t\):=\\sum\_\{h=1\}^\{H\}a\_\{hi\}\(t\)^\{L\}\-\\sigma\_\{\\star i\}\.\(53\)Then GD on\{ah​i\}\\\{a\_\{hi\}\\\}obeys the exact recursion

ah​i​\(t\+1\)=ah​i​\(t\)−η​ri​\(t\)​ah​i​\(t\)L−1\.a\_\{hi\}\(t\+1\)=a\_\{hi\}\(t\)\-\\eta\\,r\_\{i\}\(t\)\\,a\_\{hi\}\(t\)^\{L\-1\}\.\(54\)
#### Notation for the remainder of this appendix\.

We fix a modeiiand drop the index\. Writeah:=ah​ia\_\{h\}:=a\_\{hi\},r:=rir:=r\_\{i\}, and

a⋆:=\(σ⋆/H\)1/L,a¯​\(t\):=1H​∑hah​\(t\),δh​\(t\):=ah​\(t\)−a¯​\(t\),𝒢​\(t\):=∑hδh​\(t\)2\.a\_\{\\star\}:=\(\\sigma\_\{\\star\}/H\)^\{1/L\},\\qquad\\bar\{a\}\(t\):=\\frac\{1\}\{H\}\\sum\_\{h\}a\_\{h\}\(t\),\\qquad\\delta\_\{h\}\(t\):=a\_\{h\}\(t\)\-\\bar\{a\}\(t\),\\qquad\\mathcal\{G\}\(t\):=\\sum\_\{h\}\\delta\_\{h\}\(t\)^\{2\}\.The balanced equilibrium isah=a⋆a\_\{h\}=a\_\{\\star\}for allhh, at which𝒢=0\\mathcal\{G\}=0\. The sharpness of the balanced minimum isλmin=L​H2/L−1​σ⋆2−2/L=L​H​a⋆2​L−2\\lambda^\{\\min\}=LH^\{2/L\-1\}\\sigma\_\{\\star\}^\{2\-2/L\}=LHa\_\{\\star\}^\{2L\-2\}\.

### E\.3Gradient\-flow foliation and local leaf\-sharpness expansion

The discrete dynamics \([54](https://arxiv.org/html/2606.05219#A5.E54)\) has a gradient\-flow \(GF\) limit

a˙h=−r​ahL−1\.\\dot\{a\}\_\{h\}=\-r\\,a\_\{h\}^\{L\-1\}\.In this subsection we identify the conserved quantities of this GF limit and use them to define leaf coordinates for pathway imbalance\. These coordinates separate motion along a GF leaf from motion across leaves\. We then show that, on the zero\-loss manifold, the terminal sharpness increases quadratically with the leaf\-coordinate norm near the balanced leaf\. Thus the balanced leaf is the unique local sharpness minimizer among nearby GF leaves\.

#### Centered leaf coordinates\.

Define

qh​\(t\):=ah​\(t\)2−L,q¯​\(t\):=1H​∑hqh​\(t\),zh​\(t\):=qh​\(t\)−q¯​\(t\),q\_\{h\}\(t\):=a\_\{h\}\(t\)^\{\\,2\-L\},\\qquad\\bar\{q\}\(t\):=\\frac\{1\}\{H\}\\sum\_\{h\}q\_\{h\}\(t\),\\qquad z\_\{h\}\(t\):=q\_\{h\}\(t\)\-\\bar\{q\}\(t\),so∑hzh​\(t\)=0\\sum\_\{h\}z\_\{h\}\(t\)=0andz​\(t\)∈𝟏⟂⊂ℝHz\(t\)\\in\\mathbf\{1\}^\{\\perp\}\\subset\\mathbb\{R\}^\{H\}\.

###### Proposition E\.3\(GF foliation in centered coordinates\)\.

LetL\>2L\>2\. Under the gradient flowa˙h=−r​ahL−1\\dot\{a\}\_\{h\}=\-r\\,a\_\{h\}^\{L\-1\},

dd​t​qh​\(t\)=\(L−2\)​r​\(t\)for all​h,\\frac\{d\}\{dt\}q\_\{h\}\(t\)=\(L\-2\)\\,r\(t\)\\quad\\text\{for all \}h,so the centered vectorz​\(t\)z\(t\)is conserved\. The phase spaceℝ\>0H\\mathbb\{R\}\_\{\>0\}^\{H\}is therefore foliated by affine level sets inqq\-coordinates: for eachz∈𝟏⟂z\\in\\mathbf\{1\}^\{\\perp\}, the GF leaf is

q=m​1\+z,m\>−minh⁡zh,q=m\\,\\mathbf\{1\}\+z,\\qquad m\>\-\\min\_\{h\}z\_\{h\},\(55\)and its image inaa\-coordinates is the one\-dimensional curveah=\(m\+zh\)−1/\(L−2\)a\_\{h\}=\(m\+z\_\{h\}\)^\{\-1/\(L\-2\)\}\. The unique leaf containing the balanced minimum isz=0z=0\.

###### Proof\.

dd​t​qh=\(2−L\)​ah1−L​a˙h=\(2−L\)​ah1−L​\(−r​ahL−1\)=\(L−2\)​r\\frac\{d\}\{dt\}q\_\{h\}=\(2\-L\)a\_\{h\}^\{1\-L\}\\dot\{a\}\_\{h\}=\(2\-L\)a\_\{h\}^\{1\-L\}\(\-r\\,a\_\{h\}^\{L\-1\}\)=\(L\-2\)r, independent ofhh\. Hencezh​\(t\)=qh​\(t\)−q¯​\(t\)z\_\{h\}\(t\)=q\_\{h\}\(t\)\-\\bar\{q\}\(t\)is constant\. The lower bound onmmin \([55](https://arxiv.org/html/2606.05219#A5.E55)\) ensuresqh\>0q\_\{h\}\>0, equivalentlyah∈ℝ\>0a\_\{h\}\\in\\mathbb\{R\}\_\{\>0\}\. ∎

#### Equivalence ofVVand𝒢\\mathcal\{G\}\.

The quantity naturally controlled by the GF foliation is the variance of the centered leaf coordinate,

V​\(t\):=∑hzh​\(t\)2=∑h\(qh​\(t\)−q¯​\(t\)\)2\.V\(t\):=\\sum\_\{h\}z\_\{h\}\(t\)^\{2\}=\\sum\_\{h\}\\bigl\(q\_\{h\}\(t\)\-\\bar\{q\}\(t\)\\bigr\)^\{2\}\.\(56\)The quantity used in the main text to measure pathway imbalance is

𝒢​\(t\):=∑hδh​\(t\)2,δh​\(t\):=ah​\(t\)−a¯​\(t\)\.\\mathcal\{G\}\(t\):=\\sum\_\{h\}\\delta\_\{h\}\(t\)^\{2\},\\qquad\\delta\_\{h\}\(t\):=a\_\{h\}\(t\)\-\\bar\{a\}\(t\)\.Near the balanced point these two quantities are locally equivalent\. Indeed, ifah=a⋆\+δha\_\{h\}=a\_\{\\star\}\+\\delta\_\{h\}with∑hδh=0\\sum\_\{h\}\\delta\_\{h\}=0, then Taylor expansion ofqh=ah2−Lq\_\{h\}=a\_\{h\}^\{2\-L\}arounda⋆a\_\{\\star\}gives

qh−q¯=\(2−L\)​a⋆1−L​δh\+O​\(ε2\),ε:=maxh⁡\|δh\|\.q\_\{h\}\-\\bar\{q\}=\(2\-L\)a\_\{\\star\}^\{1\-L\}\\delta\_\{h\}\+O\(\\varepsilon^\{2\}\),\\qquad\\varepsilon:=\\max\_\{h\}\|\\delta\_\{h\}\|\.Consequently,

V=\(L−2\)2​a⋆2−2​L​𝒢\+O​\(𝒢3/2\)\.V=\(L\-2\)^\{2\}\\,a\_\{\\star\}^\{\\,2\-2L\}\\,\\mathcal\{G\}\+O\(\\mathcal\{G\}^\{3/2\}\)\.\(57\)Thus contraction of the leaf\-coordinate varianceVVis equivalent, locally, to contraction of the pathway balancing gap𝒢\\mathcal\{G\}up to a positive multiplicative factor\.

###### Proposition E\.4\(Local sharpness expansion across GF leaves\)\.

Leta​\(z\)∈ℳ=\{a:∑hahL=σ⋆\}a\(z\)\\in\\mathcal\{M\}=\\\{a:\\sum\_\{h\}a\_\{h\}^\{L\}=\\sigma\_\{\\star\}\\\}be the zero\-loss point on the GF leaf indexed byz∈𝟏⟂z\\in\\mathbf\{1\}^\{\\perp\}, and define the restricted\-Hessian sharpness

Λ​\(z\):=L​∑hah​\(z\)2​L−2\.\\Lambda\(z\):=L\\sum\_\{h\}a\_\{h\}\(z\)^\{2L\-2\}\.Thenz=0z=0corresponds to the balanced minimumah=a⋆a\_\{h\}=a\_\{\\star\}withΛ​\(0\)=λmin=L​H​a⋆2​L−2\\Lambda\(0\)=\\lambda^\{\\min\}=LHa\_\{\\star\}^\{2L\-2\}, and

Λ​\(z\)=λmin\+L​\(L−1\)L−2​a⋆4​L−6​‖z‖2\+O​\(‖z‖3\)\.\\Lambda\(z\)=\\lambda^\{\\min\}\+\\frac\{L\(L\-1\)\}\{L\-2\}\\,a\_\{\\star\}^\{\\,4L\-6\}\\,\\\|z\\\|^\{2\}\+O\(\\\|z\\\|^\{3\}\)\.\(58\)Equivalently, withPPdenoting the orthogonal projection onto𝟏⟂\\mathbf\{1\}^\{\\perp\},

Hleaf=2​L​\(L−1\)L−2​a⋆4​L−6​P≻0on​𝟏⟂\.H\_\{\\mathrm\{leaf\}\}=\\frac\{2L\(L\-1\)\}\{L\-2\}\\,a\_\{\\star\}^\{\\,4L\-6\}\\,P\\succ 0\\quad\\text\{on \}\\mathbf\{1\}^\{\\perp\}\.\(59\)Hence the balanced leafz=0z=0is the unique local minimizer of the terminal sharpness among nearby GF leaves\.

###### Proof sketch\.

Parametrize the leaf asqh=m\+zhq\_\{h\}=m\+z\_\{h\}\. The constraint∑hahL=σ⋆\\sum\_\{h\}a\_\{h\}^\{L\}=\\sigma\_\{\\star\}fixesm=m​\(z\)m=m\(z\)implicitly; atz=0z=0,m=a⋆2−Lm=a\_\{\\star\}^\{2\-L\}andah=a⋆a\_\{h\}=a\_\{\\star\}for allhh\. A second\-order expansion ofΛ​\(z\)\\Lambda\(z\)usingah=\(m​\(z\)\+zh\)−1/\(L−2\)a\_\{h\}=\(m\(z\)\+z\_\{h\}\)^\{\-1/\(L\-2\)\}together with the implicit derivative ofm​\(z\)m\(z\)yields \([58](https://arxiv.org/html/2606.05219#A5.E58)\)\. The first\-order term inzzvanishes by the Jensen argument of Theorem 4\.2\. ∎

Combining \([58](https://arxiv.org/html/2606.05219#A5.E58)\) with \([57](https://arxiv.org/html/2606.05219#A5.E57)\) gives, on the zero\-loss manifold,

λ​\(a\)=λmin\+Cλ​𝒢\+O​\(𝒢3/2\),Cλ:=L​\(L−1\)​\(L−2\)​a⋆2​L−4\>0\.\\lambda\(a\)=\\lambda^\{\\min\}\+C\_\{\\lambda\}\\,\\mathcal\{G\}\+O\(\\mathcal\{G\}^\{3/2\}\),\\qquad C\_\{\\lambda\}:=L\(L\-1\)\(L\-2\)\\,a\_\{\\star\}^\{\\,2L\-4\}\>0\.\(60\)

### E\.4Sign\-free step\-wise contraction \(strong\-safety regime\)

The change of variablea~h:=ah2−L\\tilde\{a\}\_\{h\}:=a\_\{h\}^\{2\-L\}converts the GD update \([54](https://arxiv.org/html/2606.05219#A5.E54)\) into a coordinate\-wise application of a single one\-dimensional map\.

###### Lemma E\.5\(One\-dimensional reduction\)\.

Seta~h​\(t\):=ah​\(t\)2−L\\tilde\{a\}\_\{h\}\(t\):=a\_\{h\}\(t\)^\{2\-L\}andc​\(t\):=η​r​\(t\)c\(t\):=\\eta\\,r\(t\)\. Then for everyhh,

a~h​\(t\+1\)=Fc​\(t\)​\(a~h​\(t\)\),Fc​\(u\):=uL−1\(u−c\)L−2,\\tilde\{a\}\_\{h\}\(t\+1\)=F\_\{c\(t\)\}\\bigl\(\\tilde\{a\}\_\{h\}\(t\)\\bigr\),\\qquad F\_\{c\}\(u\):=\\frac\{u^\{L\-1\}\}\{\(u\-c\)^\{L\-2\}\},\(61\)providedu−c\>0u\-c\>0\.

###### Proof\.

From \([54](https://arxiv.org/html/2606.05219#A5.E54)\),ah​\(t\+1\)=ah​\(t\)​\(1−η​r​\(t\)​ah​\(t\)L−2\)=ah​\(t\)​\(1−c​\(t\)/a~h​\(t\)\)a\_\{h\}\(t\+1\)=a\_\{h\}\(t\)\(1\-\\eta r\(t\)\\,a\_\{h\}\(t\)^\{L\-2\}\)=a\_\{h\}\(t\)\(1\-c\(t\)/\\tilde\{a\}\_\{h\}\(t\)\)\. Hencea~h​\(t\+1\)=ah​\(t\+1\)2−L=a~h​\(t\)​\(a~h​\(t\)/\(a~h​\(t\)−c\)\)L−2\\tilde\{a\}\_\{h\}\(t\+1\)=a\_\{h\}\(t\+1\)^\{2\-L\}=\\tilde\{a\}\_\{h\}\(t\)\\bigl\(\\tilde\{a\}\_\{h\}\(t\)/\(\\tilde\{a\}\_\{h\}\(t\)\-c\)\\bigr\)^\{L\-2\}, which simplifies to the claimed form\. ∎

The crucial point is thatFc​\(t\)F\_\{c\(t\)\}does*not*depend onhh\. HenceV​\(t\+1\)V\(t\+1\)is determined entirely by the dispersion ofFc​\(t\)F\_\{c\(t\)\}on\{a~h​\(t\)\}h=1H\\\{\\tilde\{a\}\_\{h\}\(t\)\\\}\_\{h=1\}^\{H\}\.

###### Lemma E\.6\(Monotonicity and non\-expansion ofFcF\_\{c\}\)\.

LetL\>2L\>2and suppose either \(i\)c≤0c\\leq 0, or \(ii\)c\>0c\>0withc​maxh⁡ahL−2≤1/\(L−1\)c\\,\\max\_\{h\}a\_\{h\}^\{L\-2\}\\leq 1/\(L\-1\)\. Then onI:=\[minh⁡a~h,maxh⁡a~h\]I:=\[\\min\_\{h\}\\tilde\{a\}\_\{h\},\\max\_\{h\}\\tilde\{a\}\_\{h\}\],

0≤Fc′​\(u\)≤1,0\\leq F^\{\\prime\}\_\{c\}\(u\)\\leq 1,\(62\)with strict inequalityFc′​\(u\)<1F^\{\\prime\}\_\{c\}\(u\)<1wheneverc≠0c\\neq 0\.

###### Proof\.

Fc′​\(u\)=uL−2​\(u−\(L−1\)​c\)/\(u−c\)L−1F^\{\\prime\}\_\{c\}\(u\)=u^\{L\-2\}\\bigl\(u\-\(L\-1\)c\\bigr\)/\(u\-c\)^\{L\-1\}\. The denominator is positive sinceu\>cu\>c\. The factoru−\(L−1\)​cu\-\(L\-1\)cis positive whenc≤0c\\leq 0, and whenc\>0c\>0it is positive iffu≥\(L−1\)​cu\\geq\(L\-1\)c\. In terms ofaha\_\{h\},u=ah2−L≥\(L−1\)​cu=a\_\{h\}^\{2\-L\}\\geq\(L\-1\)crewrites asη​r​ahL−2≤1/\(L−1\)\\eta r\\,a\_\{h\}^\{L\-2\}\\leq 1/\(L\-1\), hypothesis \(ii\)\. For non\-expansion, letg​\(c\):=\(u−c\)L−1−uL−2​\(u−\(L−1\)​c\)g\(c\):=\(u\-c\)^\{L\-1\}\-u^\{L\-2\}\\bigl\(u\-\(L\-1\)c\\bigr\)\. Theng​\(0\)=0g\(0\)=0andg′​\(c\)=\(L−1\)​\[uL−2−\(u−c\)L−2\]g^\{\\prime\}\(c\)=\(L\-1\)\\bigl\[u^\{L\-2\}\-\(u\-c\)^\{L\-2\}\\bigr\]is strictly positive forc\>0c\>0and strictly negative forc<0c<0, henceg​\(c\)\>0g\(c\)\>0for allc≠0c\\neq 0\. This is equivalent toFc′​\(u\)<1F^\{\\prime\}\_\{c\}\(u\)<1\. ∎

###### Definition E\.7\(Strong\-safety condition\)\.

At iteratett,

r​\(t\)≤0,orr​\(t\)\>0​and​η​r​\(t\)​maxh⁡ah​\(t\)L−2≤1L−1\.r\(t\)\\leq 0,\\qquad\\text\{or\}\\qquad r\(t\)\>0\\ \\text\{and\}\\ \\eta\\,r\(t\)\\,\\max\_\{h\}a\_\{h\}\(t\)^\{L\-2\}\\leq\\tfrac\{1\}\{L\-1\}\.\(⋆\\star\)

###### Theorem E\.8\(Sign\-free step\-wise contraction\)\.

LetL\>2L\>2and suppose the strong\-safety condition \([⋆\\star‣E\.7](https://arxiv.org/html/2606.05219#A5.Ex48)\) holds at iteratett\. Then

V​\(t\+1\)≤V​\(t\),V\(t\+1\)\\leq V\(t\),\(63\)with strict inequality wheneverr​\(t\)≠0r\(t\)\\neq 0and theah​\(t\)a\_\{h\}\(t\)are not all equal\.

###### Proof\.

By Lemma[E\.5](https://arxiv.org/html/2606.05219#A5.Thmtheorem5),a~h​\(t\+1\)=Fc​\(t\)​\(a~h​\(t\)\)\\tilde\{a\}\_\{h\}\(t\+1\)=F\_\{c\(t\)\}\(\\tilde\{a\}\_\{h\}\(t\)\)for everyhh\. By Lemma[E\.6](https://arxiv.org/html/2606.05219#A5.Thmtheorem6),Fc​\(t\)F\_\{c\(t\)\}has derivative in\[0,1\]\[0,1\]on the relevant interval, withFc​\(t\)′<1F^\{\\prime\}\_\{c\(t\)\}<1strictly wheneverc​\(t\)≠0c\(t\)\\neq 0\. The mean\-value theorem gives\|Fc​\(t\)​\(a~h\)−Fc​\(t\)​\(a~k\)\|≤\|a~h−a~k\|\|F\_\{c\(t\)\}\(\\tilde\{a\}\_\{h\}\)\-F\_\{c\(t\)\}\(\\tilde\{a\}\_\{k\}\)\|\\leq\|\\tilde\{a\}\_\{h\}\-\\tilde\{a\}\_\{k\}\|, strict unlessa~h=a~k\\tilde\{a\}\_\{h\}=\\tilde\{a\}\_\{k\}\. The varianceVVis proportional to∑h,k\(a~h−a~k\)2\\sum\_\{h,k\}\(\\tilde\{a\}\_\{h\}\-\\tilde\{a\}\_\{k\}\)^\{2\}, so summing pairwise inequalities yieldsV​\(t\+1\)≤V​\(t\)V\(t\+1\)\\leq V\(t\)with the stated strictness\. ∎

#### Scope of Theorem[E\.8](https://arxiv.org/html/2606.05219#A5.Thmtheorem8)\.

The strong\-safety condition \([⋆\\star‣E\.7](https://arxiv.org/html/2606.05219#A5.Ex48)\) is strictly more restrictive than the linear stability boundη​λmin<2\\eta\\lambda^\{\\min\}<2characterizing the basin of the balanced minimum\. In particular, large\-amplitude EoS oscillations generally violate \([⋆\\star‣E\.7](https://arxiv.org/html/2606.05219#A5.Ex48)\)\. Theorem[E\.8](https://arxiv.org/html/2606.05219#A5.Thmtheorem8)therefore guarantees step\-wise monotonicity only inside an inner neighborhood of the balanced minimum, where the strong\-safety condition is automatically satisfied \(sincer→0r\\to 0impliesc→0c\\to 0\)\. To handle the EoS regime itself, we pass to two\-step contraction with explicit quadratic correction; this is the content of the next subsection\.

### E\.5Two\-step contraction from the EoS normal form

The one\-step contraction theorem above applies only in the strong\-safety regime\. To analyze the actual Edge\-of\-Stability regime, where the residual oscillates and the strong\-safety condition may fail, we pass to a two\-step description\. The key input is a local normal form for the scalar residual recursion, retained one order beyond the linear EoS approximation\. This normal form has two consequences: first, sign alternation of the residual follows from sharpness bracketing near the stability threshold; second, the quadratic term in the residual recursion contributes a positive correction to the two\-step contraction coefficient\.

###### Lemma E\.9\(Residual recursion: EoS normal form\)\.

Letλ​\(Θt\):=L​∑hah​\(t\)2​L−2\\lambda\(\\Theta\_\{t\}\):=L\\sum\_\{h\}a\_\{h\}\(t\)^\{2L\-2\}, defineΛ3​\(Θt\):=L​\(L−1\)​∑hah​\(t\)3​L−4\\Lambda\_\{3\}\(\\Theta\_\{t\}\):=L\(L\-1\)\\sum\_\{h\}a\_\{h\}\(t\)^\{3L\-4\}andΛ4​\(Θt\):=L​\(L−1\)​\(L−2\)​∑hah​\(t\)4​L−6\\Lambda\_\{4\}\(\\Theta\_\{t\}\):=L\(L\-1\)\(L\-2\)\\sum\_\{h\}a\_\{h\}\(t\)^\{4L\-6\}, and set

β​\(t\):=2−η​λ​\(Θt\),A​\(t\):=η22​Λ3​\(Θt\)\>0,B​\(t\):=η36​Λ4​\(Θt\)\>0\.\\beta\(t\):=2\-\\eta\\lambda\(\\Theta\_\{t\}\),\\quad A\(t\):=\\tfrac\{\\eta^\{2\}\}\{2\}\\Lambda\_\{3\}\(\\Theta\_\{t\}\)\>0,\\quad B\(t\):=\\tfrac\{\\eta^\{3\}\}\{6\}\\Lambda\_\{4\}\(\\Theta\_\{t\}\)\>0\.Then

r​\(t\+1\)=−r​\(t\)\+β​\(t\)​r​\(t\)\+A​\(t\)​r​\(t\)2−B​\(t\)​r​\(t\)3\+O​\(r​\(t\)4\)\.r\(t\+1\)=\-r\(t\)\+\\beta\(t\)\\,r\(t\)\+A\(t\)\\,r\(t\)^\{2\}\-B\(t\)\\,r\(t\)^\{3\}\+O\\bigl\(r\(t\)^\{4\}\\bigr\)\.\(64\)

###### Proof\.

Expandah​\(t\+1\)L=ah​\(t\)L​\(1−η​r​\(t\)​ah​\(t\)L−2\)La\_\{h\}\(t\+1\)^\{L\}=a\_\{h\}\(t\)^\{L\}\\bigl\(1\-\\eta r\(t\)\\,a\_\{h\}\(t\)^\{L\-2\}\\bigr\)^\{L\}using\(1−x\)L=1−L​x\+\(L2\)​x2−\(L3\)​x3\+O​\(x4\)\(1\-x\)^\{L\}=1\-Lx\+\\binom\{L\}\{2\}x^\{2\}\-\\binom\{L\}\{3\}x^\{3\}\+O\(x^\{4\}\)and sum overhh, identifyingλ\\lambda,Λ3\\Lambda\_\{3\},Λ4\\Lambda\_\{4\}\. ∎

###### Hypothesis E\.10\(Self\-stabilized EoS regime\)\.

There exist a neighborhood𝒩\\mathcal\{N\}of the balanced minimum, constantsε0,ρ0,Cβ\>0\\varepsilon\_\{0\},\\rho\_\{0\},C\_\{\\beta\}\>0, and a timet0t\_\{0\}such that for allt≥t0t\\geq t\_\{0\}withΘt∈𝒩\\Theta\_\{t\}\\in\\mathcal\{N\}:

1. 1\.*Locality:*maxh⁡\|δh​\(t\)\|≤ε0\\max\_\{h\}\|\\delta\_\{h\}\(t\)\|\\leq\\varepsilon\_\{0\}and\|r​\(t\)\|≤ρ0\|r\(t\)\|\\leq\\rho\_\{0\};
2. 2\.*Sharpness bracketing:*\|β​\(t\)\|=\|2−η​λ​\(Θt\)\|≤Cβ​r​\(t\)2\|\\beta\(t\)\|=\|2\-\\eta\\lambda\(\\Theta\_\{t\}\)\|\\leq C\_\{\\beta\}\\,r\(t\)^\{2\}\.

#### Derived sign alternation\.

Hypothesis[E\.10](https://arxiv.org/html/2606.05219#A5.Thmtheorem10)\(ii\), combined with Lemma[E\.9](https://arxiv.org/html/2606.05219#A5.Thmtheorem9), gives

r​\(t\+1\)=−r​\(t\)\+O​\(r​\(t\)2\)\.r\(t\+1\)=\-r\(t\)\+O\(r\(t\)^\{2\}\)\.Therefore, for all sufficiently smallr​\(t\)≠0r\(t\)\\neq 0, the residualsr​\(t\)r\(t\)andr​\(t\+1\)r\(t\+1\)have opposite signs\. In particular, the sign alternation used in the two\-step contraction is not imposed as a separate dynamical assumption; it is a local consequence of the EoS residual normal form together with sharpness bracketing\.

#### Two\-step expansion\.

Under Hypothesis[E\.10](https://arxiv.org/html/2606.05219#A5.Thmtheorem10),

r​\(t\)\+r​\(t\+1\)=A​\(t\)​r​\(t\)2\+O​\(r​\(t\)3\),r​\(t\)​r​\(t\+1\)=−r​\(t\)2\+O​\(r​\(t\)3\)\.r\(t\)\+r\(t\+1\)=A\(t\)\\,r\(t\)^\{2\}\+O\\bigl\(r\(t\)^\{3\}\\bigr\),\\qquad r\(t\)r\(t\+1\)=\-r\(t\)^\{2\}\+O\\bigl\(r\(t\)^\{3\}\\bigr\)\.\(65\)The crucial point — and the source of the corrected contraction coefficient — is thatr​\(t\)\+r​\(t\+1\)r\(t\)\+r\(t\+1\)isO​\(r2\)O\(r^\{2\}\),*not*O​\(r3\)O\(r^\{3\}\)\. The frozen\-linearization two\-step transverse multiplier at the balanced point is

mt=\(1−κ⋆​r​\(t\)\)​\(1−κ⋆​r​\(t\+1\)\)=1−\(κ⋆2\+κ⋆​A​\(t\)\)​r​\(t\)2\+O​\(r​\(t\)3\),m\_\{t\}=\(1\-\\kappa\_\{\\star\}r\(t\)\)\(1\-\\kappa\_\{\\star\}r\(t\+1\)\)=1\-\\bigl\(\\kappa\_\{\\star\}^\{2\}\+\\kappa\_\{\\star\}A\(t\)\\bigr\)r\(t\)^\{2\}\+O\\bigl\(r\(t\)^\{3\}\\bigr\),whereκ⋆:=η​\(L−1\)​a⋆L−2\\kappa\_\{\\star\}:=\\eta\(L\-1\)a\_\{\\star\}^\{L\-2\}, hencemt2=1−2​\(κ⋆2\+κ⋆​A​\(t\)\)​r​\(t\)2\+O​\(r​\(t\)3\)m\_\{t\}^\{2\}=1\-2\(\\kappa\_\{\\star\}^\{2\}\+\\kappa\_\{\\star\}A\(t\)\)r\(t\)^\{2\}\+O\(r\(t\)^\{3\}\)\.

#### Robust formulation\.

For a fully local theorem, the step\-to\-step variation ofκt\\kappa\_\{t\}aroundκ⋆\\kappa\_\{\\star\}should be absorbed into the local error term rather than claimed exactly\. We state the robust form, which is the version we use\.

###### Theorem E\.11\(Robust two\-step contraction from the EoS normal form\)\.

LetL\>2L\>2\. Suppose Hypothesis[E\.10](https://arxiv.org/html/2606.05219#A5.Thmtheorem10)holds at iteratettand the iterates remain on the positive branch\. Then there exist constantsc0\>0c\_\{0\}\>0andC\>0C\>0, depending only on the local neighborhood, such that

𝒢​\(t\+2\)≤\(1−c0​r​\(t\)2\+C​\(\|r​\(t\)\|3\+r​\(t\)2​εt\+r​\(t\)2​εt2\)\)​𝒢​\(t\),\\mathcal\{G\}\(t\+2\)\\leq\\Bigl\(1\-c\_\{0\}\\,r\(t\)^\{2\}\+C\\bigl\(\|r\(t\)\|^\{3\}\+r\(t\)^\{2\}\\varepsilon\_\{t\}\+r\(t\)^\{2\}\\varepsilon\_\{t\}^\{2\}\\bigr\)\\Bigr\)\\,\\mathcal\{G\}\(t\),\(66\)whereεt:=maxh⁡\|δh​\(t\)\|\\varepsilon\_\{t\}:=\\max\_\{h\}\|\\delta\_\{h\}\(t\)\|\. Equivalently, the remainder iso​\(r​\(t\)2\)o\(r\(t\)^\{2\}\)relative to the leading term under the local scaling assumptions\. Moreover, if the transverse linearization is frozen at the balanced point, the leading coefficient admits the explicit form

c0=2​\(κ⋆2\+κ⋆​A⋆\),κ⋆=η​\(L−1\)​a⋆L−2,A⋆=η22​L​\(L−1\)​H​a⋆3​L−4\.c\_\{0\}=2\\bigl\(\\kappa\_\{\\star\}^\{2\}\+\\kappa\_\{\\star\}A\_\{\\star\}\\bigr\),\\qquad\\kappa\_\{\\star\}=\\eta\(L\-1\)a\_\{\\star\}^\{L\-2\},\\qquad A\_\{\\star\}=\\tfrac\{\\eta^\{2\}\}\{2\}L\(L\-1\)H\\,a\_\{\\star\}^\{3L\-4\}\.Whenever the error terms in \([66](https://arxiv.org/html/2606.05219#A5.E66)\) are dominated byc0​r​\(t\)2c\_\{0\}\\,r\(t\)^\{2\}, the balancing gap contracts:𝒢​\(t\+2\)<𝒢​\(t\)\\mathcal\{G\}\(t\+2\)<\\mathcal\{G\}\(t\)\.

###### Proof sketch\.

The relations \([65](https://arxiv.org/html/2606.05219#A5.E65)\) imply the linear part of the two\-step transverse update has multiplier1−c​r​\(t\)2\+O​\(r​\(t\)3\+r​\(t\)2​εt\)1\-c\\,r\(t\)^\{2\}\+O\(r\(t\)^\{3\}\+r\(t\)^\{2\}\\varepsilon\_\{t\}\)for somec\>0c\>0\. Squaring gives the leading factor1−c0​r​\(t\)21\-c\_\{0\}\\,r\(t\)^\{2\}\. The nonlinear transverse contributions from the quadratic Taylor correction of the per\-pathway update are proportional to powers ofr​\(t\)r\(t\)\(since the GD map reduces to the identity atr​\(t\)=0r\(t\)=0\), contributingO​\(\(\|r​\(t\)\|3\+r​\(t\)2​εt\)​𝒢​\(t\)\)O\\bigl\(\(\|r\(t\)\|^\{3\}\+r\(t\)^\{2\}\\varepsilon\_\{t\}\)\\mathcal\{G\}\(t\)\\bigr\)\. This yields \([66](https://arxiv.org/html/2606.05219#A5.E66)\)\. ∎

#### Remark on the remainder\.

The GD updateah\+=ah−η​r​ahL−1a\_\{h\}^\{\+\}=a\_\{h\}\-\\eta r\\,a\_\{h\}^\{L\-1\}reduces to the identity whenr=0r=0, so every term in the two\-step error must carry at least one factor ofr​\(t\)r\(t\)\. This is reflected in \([66](https://arxiv.org/html/2606.05219#A5.E66)\): there is no standaloneεt4\\varepsilon\_\{t\}^\{4\}contribution\. The compact form isR​\(t\)=o​\(r​\(t\)2​𝒢​\(t\)\)R\(t\)=o\(r\(t\)^\{2\}\\,\\mathcal\{G\}\(t\)\)under the EoS scaling\.

#### Relation to the sign\-alternating formulation\.

A weaker way to state the two\-step mechanism is to assume directly that the residual forms a small\-amplitude two\-cycle,

r​\(t\+1\)=−r​\(t\)\+O​\(r​\(t\)2\)\.r\(t\+1\)=\-r\(t\)\+O\(r\(t\)^\{2\}\)\.Under this assumption, the leading part of Theorem[E\.11](https://arxiv.org/html/2606.05219#A5.Thmtheorem11)reduces to the familiar contraction form

𝒢​\(t\+2\)≤\(1−c​r​\(t\)2\)​𝒢​\(t\)\\mathcal\{G\}\(t\+2\)\\leq\\bigl\(1\-c\\,r\(t\)^\{2\}\\bigr\)\\mathcal\{G\}\(t\)for somec\>0c\>0\. The present formulation is stronger in two respects: the small\-amplitude sign alternation is derived from the EoS sharpness bracketing, and the retained quadratic term in the residual normal form contributes the positive correctionκ⋆​A⋆\\kappa\_\{\\star\}A\_\{\\star\}to the leading frozen contraction coefficient\.

### E\.6Formal central\-flow description of the balancing gap

We now translate the local two\-step contraction into a formal slow\-time ODE in the spirit of central flows\(Cohenet al\.,[2025](https://arxiv.org/html/2606.05219#bib.bib17)\)\. This subsection should be read as a leading\-order asymptotic description, not as a finite\-time approximation theorem\. Its purpose is to identify the direction of the slow cross\-leaf drift and the stopping boundary selected by the EoS stability constraint\.

The resulting picture distinguishes two cases\. At the critical learning rateη=2/λmin\\eta=2/\\lambda^\{\\min\}, the formal drift continues until full balance𝒢=0\\mathcal\{G\}=0\. At intermediate learning rates, the drift stops earlier at the one\-sided boundary where the sharpness reaches2/η2/\\eta; the prediction is then partial balance rather than exact balance\.

#### Sign convention\.

Use the sharpness excessχ​\(Θ\):=η​λ​\(Θ\)−2\\chi\(\\Theta\):=\\eta\\lambda\(\\Theta\)\-2\. The EoS regime isχ\>0\\chi\>0, where sustained residual oscillations occur; below the threshold,χ<0\\chi<0, the residual decays exponentially and the central\-flow drift is absent\.

#### Self\-stabilized variance\.

The cubic restoring term in Lemma[E\.9](https://arxiv.org/html/2606.05219#A5.Thmtheorem9)pins the time\-averaged squared residual to scale with the positive part ofχ\\chi:

⟨r2⟩​\(Θ\)=K​\(Θ\)​χ​\(Θ\)\+\+O​\(χ2\),\\langle r^\{2\}\\rangle\(\\Theta\)=K\(\\Theta\)\\,\\chi\(\\Theta\)\_\{\+\}\\,\+O\(\\chi^\{2\}\),\(67\)whereK​\(Θ\)\>0K\(\\Theta\)\>0depends locally onΛ3,Λ4\\Lambda\_\{3\},\\Lambda\_\{4\}via the standard self\-stabilization calculation\(Damianet al\.,[2023](https://arxiv.org/html/2606.05219#bib.bib18), Section 3\)\. Crucially, the positive partχ\+\\chi\_\{\+\}appears: below the EoS threshold there is no oscillation and no drift\.

###### Theorem E\.12\(Formal slow ODE for the balancing gap\)\.

LetL\>2L\>2and suppose Hypothesis[E\.10](https://arxiv.org/html/2606.05219#A5.Thmtheorem10)holds along the trajectory\. Define slow timeτ:=t/2\\tau:=t/2\. To leading order in𝒢\\mathcal\{G\},

d​𝒢d​τ=−K0​\(η​λ​\(𝒢\)−2\)\+​𝒢\+O​\(𝒢2\),\\frac\{d\\mathcal\{G\}\}\{d\\tau\}=\-\\,K\_\{0\}\\,\\bigl\(\\eta\\lambda\(\\mathcal\{G\}\)\-2\\bigr\)\_\{\+\}\\,\\mathcal\{G\}\+O\(\\mathcal\{G\}^\{2\}\),\(68\)whereK0\>0K\_\{0\}\>0collects the per\-step contraction coefficient from Theorem[E\.11](https://arxiv.org/html/2606.05219#A5.Thmtheorem11)and the self\-stabilized residual scale, andλ​\(𝒢\)\\lambda\(\\mathcal\{G\}\)is given by \([60](https://arxiv.org/html/2606.05219#A5.E60)\)\.

#### Stopping boundary, not stable equilibrium\.

The right\-hand side of \([68](https://arxiv.org/html/2606.05219#A5.E68)\) containsχ\+\\chi\_\{\+\}, notχ\\chi\. Hence the drift acts only whenλ​\(𝒢\)\>2/η\\lambda\(\\mathcal\{G\}\)\>2/\\etaand is*identically zero*whenλ​\(𝒢\)≤2/η\\lambda\(\\mathcal\{G\}\)\\leq 2/\\eta\. There is no restoring force in the opposite direction\. The correct dynamical picture is therefore a*one\-sided sliding boundary*:𝒢\\mathcal\{G\}decreases monotonically untilλ​\(𝒢\)\\lambda\(\\mathcal\{G\}\)reaches2/η2/\\eta, at which point the drift switches off and the iterate halts \(at the slow\-ODE level\)\.

Combining with \([60](https://arxiv.org/html/2606.05219#A5.E60)\), the stopping condition is

λ​\(𝒢†\)=2η,𝒢†=2−η​λminη​L​\(L−1\)​\(L−2\)​a⋆2​L−4\+O​\(\(η​λmin−2\)3/2\)\.\\lambda\(\\mathcal\{G\}^\{\\dagger\}\)=\\frac\{2\}\{\\eta\},\\qquad\\mathcal\{G\}^\{\\dagger\}=\\frac\{2\-\\eta\\lambda^\{\\min\}\}\{\\eta\\,L\(L\-1\)\(L\-2\)\\,a\_\{\\star\}^\{2L\-4\}\}\+O\\bigl\(\(\\eta\\lambda^\{\\min\}\-2\)^\{3/2\}\\bigr\)\.\(69\)

#### Three regimes\.

1. 1\.*Intermediate window2/S1<η<2/λmin2/S\_\{1\}<\\eta<2/\\lambda^\{\\min\}\.*Hereη​λmin<2\\eta\\lambda^\{\\min\}<2, so the balanced point𝒢=0\\mathcal\{G\}=0is below the EoS threshold \(χ​\(0\)<0\\chi\(0\)<0\)\. The drift is active when𝒢\>𝒢†\>0\\mathcal\{G\}\>\\mathcal\{G\}^\{\\dagger\}\>0and inactive when𝒢≤𝒢†\\mathcal\{G\}\\leq\\mathcal\{G\}^\{\\dagger\}\. The formal ODE predicts*drift until the sharpness reaches2/η2/\\eta, then stop*; the resulting state is partially balanced, not fully balanced\. This is a stopping boundary, not a two\-sided stable equilibrium\.
2. 2\.*Criticalη=2/λmin\\eta=2/\\lambda^\{\\min\}\.*Hereη​λmin=2\\eta\\lambda^\{\\min\}=2, soχ​\(0\)=0\\chi\(0\)=0and𝒢†=0\\mathcal\{G\}^\{\\dagger\}=0\. The stopping boundary coincides with the balanced center, and the drift drives the trajectory all the way to𝒢=0\\mathcal\{G\}=0\. At this learning rate the formal ODE selects the fully balanced minimum\.
3. 3\.*Super\-criticalη\>2/λmin\\eta\>2/\\lambda^\{\\min\}\.*Hereχ​\(0\)\>0\\chi\(0\)\>0: even the balanced minimum is GD\-unstable in the local model\. The slow ODE has no zero\-loss stopping boundary in𝒢≥0\\mathcal\{G\}\\geq 0; the trajectory must leave𝒩\\mathcal\{N\}or enter a different nonlinear regime, consistent with the worst\-case\-return analysis of Section 6\.

#### Recovery of the stability window \(17\)\.

The window2/S1<η<2/λmin2/S\_\{1\}<\\eta<2/\\lambda^\{\\min\}of the main text is precisely the set ofη\\etafor which \(a\) the sharp single\-path minimum is GD\-unstable \(so the symmetry\-breaking phase cannot terminate there\) and \(b\) the slow ODE admits a nontrivial stopping boundary𝒢†∈\[0,𝒢1\-path\)\\mathcal\{G\}^\{\\dagger\}\\in\[0,\\mathcal\{G\}^\{\\text\{1\-path\}\}\)\. The endpointη=2/λmin\\eta=2/\\lambda^\{\\min\}is the unique learning rate at which the stopping boundary collapses to the fully balanced minimum\.

#### Geometric interpretation\.

Combining Theorem[E\.12](https://arxiv.org/html/2606.05219#A5.Thmtheorem12)with Propositions[E\.3](https://arxiv.org/html/2606.05219#A5.Thmtheorem3)and[E\.4](https://arxiv.org/html/2606.05219#A5.Thmtheorem4): GD oscillations of amplitude⟨r2⟩∝χ\+\\langle r^\{2\}\\rangle\\propto\\chi\_\{\+\}produce a one\-sided cross\-leaf driftd​𝒢/d​τ∝−χ\+​𝒢d\\mathcal\{G\}/d\\tau\\propto\-\\chi\_\{\+\}\\mathcal\{G\}\. Via \([57](https://arxiv.org/html/2606.05219#A5.E57)\), the leaf coordinatezztracks𝒢\\mathcal\{G\}, so the trajectory drifts across GF leaves in the direction of decreasing‖z‖\\\|z\\\|\. The drift switches off precisely at the leaf whose terminal sharpness equals2/η2/\\eta— i\.e\., at the boundary between the EoS regime and the linearly stable regime — and the trajectory stops there\. At intermediate learning rates this stopping leaf is notz=0z=0but a partially balanced leaf; atη=2/λmin\\eta=2/\\lambda^\{\\min\}, the stopping leaf is exactlyz=0z=0\.

### E\.7Scope of the local analysis

We conclude by separating the rigorous local statements from the formal asymptotic interpretation\.

1. 1\.Rigorous reductions and local identities\.We prove SVS invariance and preservation of depth\-wise balance \(Lemmas[E\.1](https://arxiv.org/html/2606.05219#A5.Thmtheorem1)and[E\.2](https://arxiv.org/html/2606.05219#A5.Thmtheorem2)\), the exact scalar recursionah\+=ah−η​r​ahL−1a\_\{h\}^\{\+\}=a\_\{h\}\-\\eta r\\,a\_\{h\}^\{L\-1\}in \([54](https://arxiv.org/html/2606.05219#A5.E54)\), the GF foliation in centered coordinatesz∈𝟏⟂z\\in\\mathbf\{1\}^\{\\perp\}\(Proposition[E\.3](https://arxiv.org/html/2606.05219#A5.Thmtheorem3)\), and the local sharpness expansion across GF leaves in \([58](https://arxiv.org/html/2606.05219#A5.E58)\)\. We also prove one\-step contraction of the leaf\-coordinate varianceVVunder the strong\-safety condition \(Theorem[E\.8](https://arxiv.org/html/2606.05219#A5.Thmtheorem8)\) and the residual normal form \(Lemma[E\.9](https://arxiv.org/html/2606.05219#A5.Thmtheorem9)\)\.
2. 2\.Local EoS contraction under self\-stabilization\.The two\-step contraction theorem \(Theorem[E\.11](https://arxiv.org/html/2606.05219#A5.Thmtheorem11)\) is a local statement near the balanced minimum, conditional on the self\-stabilized EoS bracketing \|2−η​λ​\(Θt\)\|=O​\(r​\(t\)2\)\.\|2\-\\eta\\lambda\(\\Theta\_\{t\}\)\|=O\(r\(t\)^\{2\}\)\.Given this bracketing, the residual normal form implies sign alternation and the pathway balancing gap contracts over two steps whenever the local error terms are dominated by the leading quadratic contraction term\. The bracketing condition itself is imported from the self\-stabilization theory of GD at the EoS and is not proved from first principles here\.
3. 3\.Formal slow\-time interpretation\.The central\-flow equation \([68](https://arxiv.org/html/2606.05219#A5.E68)\) is a leading\-order asymptotic description\. It explains the direction of the slow drift and the one\-sided stopping boundary selected by the stability thresholdη​λ=2\\eta\\lambda=2, but it is not a finite\-time approximation theorem\.
4. 4\.Not proved globally\.We do not prove that every winner\-takes\-all transient enters the local self\-stabilized EoS neighborhood in finite time\. Establishing such a result would turn the local mechanism in this appendix into a full global re\-balancing theorem\. The experiments in Figures 1–4 support this global picture, but the present proof is local\.

## Appendix FWorst\-Case Return Threshold for Deep Chains

![Refer to caption](https://arxiv.org/html/2606.05219v1/x10.png)Figure 6:Deep linear chain update map \([19](https://arxiv.org/html/2606.05219#S6.E19)\) for different values ofη/η1\\eta/\\eta\_\{1\}andLL\. The worst\-case overshoot\(c,fη​\(c\)\)\(c,f\_\{\\eta\}\(c\)\)and the return point\(fη​\(c\),fη​\(fη​\(c\)\)\)\(f\_\{\\eta\}\(c\),f\_\{\\eta\}\(f\_\{\\eta\}\(c\)\)\)illustrate how larger depth permits larger normalized steps before a sign flip occurs\.This appendix provides the proofs and computational details for Section[6](https://arxiv.org/html/2606.05219#S6)\. Our goal is to characterize the largest learning rate for which the deep\-chain GD map cannot cross the origin after the*worst*overshoot from the positive branch\.

### F\.1From one\-path concentration to a deep\-chain map

On the SVS set restricted to the depth\-balanced manifold, each pathway–mode pair admits a per\-layer scalarah​1​\(t\)≥0a\_\{h1\}\(t\)\\geq 0whose end\-to\-end singular value isσh​1​\(t\)=ah​1​\(t\)L\\sigma\_\{h1\}\(t\)=a\_\{h1\}\(t\)^\{L\}\. The corresponding mode\-wise GD recursion is

ah​1​\(t\+1\)\\displaystyle a\_\{h1\}\(t\+1\)=ah​1​\(t\)−η​r1​\(t\)​ah​1​\(t\)L−1,\\displaystyle=a\_\{h1\}\(t\)\-\\eta\\,r\_\{1\}\(t\)\\,a\_\{h1\}\(t\)^\{L\-1\},\(70\)r1​\(t\)\\displaystyle r\_\{1\}\(t\)=∑k=1Hak​1​\(t\)L−σ⋆1\.\\displaystyle=\\sum\_\{k=1\}^\{H\}a\_\{k1\}\(t\)^\{L\}\-\\sigma\_\{\\star 1\}\.If, during symmetry breaking, the dominant mode is concentrated in one pathwayh=1h=1so thata11​\(t\)L≫∑k≠1ak​1​\(t\)La\_\{11\}\(t\)^\{L\}\\gg\\sum\_\{k\\neq 1\}a\_\{k1\}\(t\)^\{L\}, then the residual satisfiesr1​\(t\)≈a11​\(t\)L−σ⋆1r\_\{1\}\(t\)\\approx a\_\{11\}\(t\)^\{L\}\-\\sigma\_\{\\star 1\}\. Substituting this into \([70](https://arxiv.org/html/2606.05219#A6.E70)\) yields the single\-variable deep\-chain map

wt\+1=wt−η​wtL−1​\(wtL−σ⋆1\),wt≡a11​\(t\),w\_\{t\+1\}=w\_\{t\}\-\\eta\\,w\_\{t\}^\{L\-1\}\(w\_\{t\}^\{L\}\-\\sigma\_\{\\star 1\}\),\\qquad w\_\{t\}\\equiv a\_\{11\}\(t\),which is \([19](https://arxiv.org/html/2606.05219#S6.E19)\)\.

### F\.2Normalization and geometric preliminaries

Letw⋆=\(σ⋆1\)1/Lw\_\{\\star\}=\(\\sigma\_\{\\star 1\}\)^\{1/L\}and define normalized variables

u:=ww⋆∈\(0,∞\),α:=η​\(σ⋆1\)2−2/L\.u:=\\frac\{w\}\{w\_\{\\star\}\}\\in\(0,\\infty\),\\qquad\\alpha:=\\eta\\,\(\\sigma\_\{\\star 1\}\)^\{2\-2/L\}\.\(71\)Then \([19](https://arxiv.org/html/2606.05219#S6.E19)\) becomes the dimensionless map

u\+=gα​\(u\):=u−α​uL−1​\(uL−1\)=u\+α​uL−1​\(1−uL\)\.u^\{\+\}=g\_\{\\alpha\}\(u\):=u\-\\alpha\\,u^\{L\-1\}\(u^\{L\}\-1\)=u\+\\alpha\\,u^\{L\-1\}\(1\-u^\{L\}\)\.\(72\)The overshoot\-onset thresholdη1=1/S1\\eta\_\{1\}=1/S\_\{1\}corresponds toα=1/L\\alpha=1/L, sinceS1=L​\(σ⋆1\)2−2/LS\_\{1\}=L\(\\sigma\_\{\\star 1\}\)^\{2\-2/L\}\.

###### Lemma F\.1\(Overshoot regime and monotonicity foru≥1u\\geq 1\)\.

Assumeα\>1/L\\alpha\>1/L\. Then \(i\) there existu∈\(0,1\)u\\in\(0,1\)such thatgα​\(u\)\>1g\_\{\\alpha\}\(u\)\>1\(overshoot occurs\), and \(ii\)gαg\_\{\\alpha\}is strictly decreasing on\[1,∞\)\[1,\\infty\)\.

###### Proof\.

We havegα​\(1\)=1g\_\{\\alpha\}\(1\)=1and

gα′​\(u\)=1−α​uL−2​\(\(2​L−1\)​uL−\(L−1\)\),gα′​\(1\)=1−α​L<0\(α\>1/L\),g\_\{\\alpha\}^\{\\prime\}\(u\)=1\-\\alpha\\,u^\{L\-2\}\\Bigl\(\(2L\-1\)u^\{L\}\-\(L\-1\)\\Bigr\),\\qquad g\_\{\\alpha\}^\{\\prime\}\(1\)=1\-\\alpha L<0\\quad\(\\alpha\>1/L\),sogαg\_\{\\alpha\}exceeds11foru<1u<1sufficiently close to11, proving \(i\)\. For \(ii\), note that for everyu≥1u\\geq 1,

\(2​L−1\)​uL−\(L−1\)≥L,uL−2≥1,\(2L\-1\)u^\{L\}\-\(L\-1\)\\geq L,\\qquad u^\{L\-2\}\\geq 1,hencegα′​\(u\)≤1−α​L<0g\_\{\\alpha\}^\{\\prime\}\(u\)\\leq 1\-\\alpha L<0for allu≥1u\\geq 1\. ∎

### F\.3Worst\-case overshoot and the reduction to a single inequality

Forα\>1/L\\alpha\>1/L, define the worst\-case overshoot value

vmax​\(α\):=maxu∈\[0,1\]⁡gα​\(u\)\.v\_\{\\max\}\(\\alpha\):=\\max\_\{u\\in\[0,1\]\}g\_\{\\alpha\}\(u\)\.\(73\)By continuity, the maximum exists; by Lemma[F\.1](https://arxiv.org/html/2606.05219#A6.Thmtheorem1)\(i\), we havevmax​\(α\)\>1v\_\{\\max\}\(\\alpha\)\>1in the overshoot regime\.

Before stating the reduction, write the two\-step return property in normalized coordinates:

∀u∈\(0,1\):gα\(u\)\>1⟹0<gα∘2\(u\)<1\.\\forall\\,u\\in\(0,1\):\\quad g\_\{\\alpha\}\(u\)\>1\\ \\Longrightarrow\\ 0<g\_\{\\alpha\}^\{\\circ 2\}\(u\)<1\.\(74\)
###### Lemma F\.2\(Two\-step return reduces to positivity atvmaxv\_\{\\max\}\)\.

Assumeα\>1/L\\alpha\>1/L\. Then the WCR property \([74](https://arxiv.org/html/2606.05219#A6.E74)\) is equivalent to

gα​\(vmax​\(α\)\)\>0\.g\_\{\\alpha\}\\bigl\(v\_\{\\max\}\(\\alpha\)\\bigr\)\>0\.\(75\)

###### Proof\.

Fixα\>1/L\\alpha\>1/Land letu∈\(0,1\)u\\in\(0,1\)be such thatgα​\(u\)\>1g\_\{\\alpha\}\(u\)\>1\. By Lemma[F\.1](https://arxiv.org/html/2606.05219#A6.Thmtheorem1)\(ii\),gαg\_\{\\alpha\}is strictly decreasing on\[1,∞\)\[1,\\infty\), hence

gα​\(gα​\(u\)\)≥gα​\(vmax​\(α\)\)\.g\_\{\\alpha\}\\bigl\(g\_\{\\alpha\}\(u\)\\bigr\)\\geq g\_\{\\alpha\}\\bigl\(v\_\{\\max\}\(\\alpha\)\\bigr\)\.Moreover, becausegαg\_\{\\alpha\}is decreasing on\[1,∞\)\[1,\\infty\)andgα​\(1\)=1g\_\{\\alpha\}\(1\)=1, we automatically havegα​\(gα​\(u\)\)<1g\_\{\\alpha\}\(g\_\{\\alpha\}\(u\)\)<1whenevergα​\(u\)\>1g\_\{\\alpha\}\(u\)\>1\. Therefore the only nontrivial part of two\-step return is the lower boundgα​\(gα​\(u\)\)\>0g\_\{\\alpha\}\(g\_\{\\alpha\}\(u\)\)\>0, which is ensured for all overshootinguuiff the worst case is positive:gα​\(vmax​\(α\)\)\>0g\_\{\\alpha\}\(v\_\{\\max\}\(\\alpha\)\)\>0\. ∎

### F\.4Boundary system and the 1D root equation

Letc∈\(0,1\)c\\in\(0,1\)be any maximizer attainingvmax​\(α\)=gα​\(c\)v\_\{\\max\}\(\\alpha\)=g\_\{\\alpha\}\(c\); then necessarilygα′​\(c\)=0g\_\{\\alpha\}^\{\\prime\}\(c\)=0\. Define

y:=cL,A​\(y\):=\(2​L−2\)​y−\(L−2\),B​\(y\):=\(2​L−1\)​y−\(L−1\)\.y:=c^\{L\},\\qquad A\(y\):=\(2L\-2\)y\-\(L\-2\),\\qquad B\(y\):=\(2L\-1\)y\-\(L\-1\)\.A direct substitution yields the standard identities

gα′​\(c\)=0\\displaystyle g\_\{\\alpha\}^\{\\prime\}\(c\)=0⟺α=1cL−2​B​\(y\)=y2/L−1B​\(y\),\\displaystyle\\Longleftrightarrow\\alpha=\\frac\{1\}\{c^\{L\-2\}\\,B\(y\)\}=\\frac\{y^\{2/L\-1\}\}\{B\(y\)\},\(76\)vmax​\(α\)=gα​\(c\)\\displaystyle v\_\{\\max\}\(\\alpha\)=g\_\{\\alpha\}\(c\)=c​A​\(y\)B​\(y\)\.\\displaystyle=c\\,\\frac\{A\(y\)\}\{B\(y\)\}\.\(77\)
###### Definition F\.3\(Normalized WCR constant\)\.

Define

αWCR​\(L\):=sup\{α\>0:gα​\(vmax​\(α\)\)\>0\}\.\\alpha\_\{\\mathrm\{WCR\}\}\(L\):=\\sup\\\{\\alpha\>0:\\ g\_\{\\alpha\}\(v\_\{\\max\}\(\\alpha\)\)\>0\\\}\.\(78\)Equivalently,αWCR​\(L\)\\alpha\_\{\\mathrm\{WCR\}\}\(L\)is the unique boundary value for whichgα​\(vmax​\(α\)\)=0g\_\{\\alpha\}\(v\_\{\\max\}\(\\alpha\)\)=0\.

###### Lemma F\.4\(Boundary system\)\.

Atα=αWCR​\(L\)\\alpha=\\alpha\_\{\\mathrm\{WCR\}\}\(L\), there existsc∈\(0,1\)c\\in\(0,1\)such that

gα′​\(c\)=0,gα​\(gα​\(c\)\)=0\.g\_\{\\alpha\}^\{\\prime\}\(c\)=0,\\qquad g\_\{\\alpha\}\\bigl\(g\_\{\\alpha\}\(c\)\\bigr\)=0\.\(79\)Conversely, any\(α,c\)\(\\alpha,c\)satisfying \([79](https://arxiv.org/html/2606.05219#A6.E79)\) yieldsα=αWCR​\(L\)\\alpha=\\alpha\_\{\\mathrm\{WCR\}\}\(L\)\.

###### Proof\.

By Lemma[F\.2](https://arxiv.org/html/2606.05219#A6.Thmtheorem2), the boundary is characterized bygα​\(vmax​\(α\)\)=0g\_\{\\alpha\}\(v\_\{\\max\}\(\\alpha\)\)=0\. At the maximizerccattainingvmax​\(α\)=gα​\(c\)v\_\{\\max\}\(\\alpha\)=g\_\{\\alpha\}\(c\)we havegα′​\(c\)=0g\_\{\\alpha\}^\{\\prime\}\(c\)=0, hence \([79](https://arxiv.org/html/2606.05219#A6.E79)\)\. Conversely, if \([79](https://arxiv.org/html/2606.05219#A6.E79)\) holds, then the worst overshoot lands exactly at the origin in the next step, so the correspondingα\\alphais the WCR boundary\. ∎

Combining the boundary identities gives the one\-dimensional root equation

B​\(y\)2​L−1=A​\(y\)L−2​\(y​A​\(y\)L−B​\(y\)L\),B\(y\)^\{2L\-1\}=A\(y\)^\{L\-2\}\\Bigl\(y\\,A\(y\)^\{L\}\-B\(y\)^\{L\}\\Bigr\),\(80\)on the interval

y∈\(L−12​L−1,1\)\.y\\in\\Bigl\(\\frac\{L\-1\}\{2L\-1\},\\,1\\Bigr\)\.Once this root is found, the normalized WCR constant is

γWCR​\(L\)=L​y2/L−1B​\(y\)\.\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=L\\,\\frac\{y^\{2/L\-1\}\}\{B\(y\)\}\.\(81\)
###### Proof of Theorem[6\.2](https://arxiv.org/html/2606.05219#S6.Thmtheorem2)\.

Let\(α,c\)\(\\alpha,c\)satisfy \([79](https://arxiv.org/html/2606.05219#A6.E79)\)\. Using \([76](https://arxiv.org/html/2606.05219#A6.E76)\) and \([77](https://arxiv.org/html/2606.05219#A6.E77)\), writey=cLy=c^\{L\}andvmax=c​A/Bv\_\{\\max\}=cA/B\. The conditiongα​\(vmax\)=0g\_\{\\alpha\}\(v\_\{\\max\}\)=0implies

α=1vmaxL−2​\(vmaxL−1\)\.\\alpha=\\frac\{1\}\{v\_\{\\max\}^\{L\-2\}\(v\_\{\\max\}^\{L\}\-1\)\}\.Combining this with \([76](https://arxiv.org/html/2606.05219#A6.E76)\) and substitutingvmaxL=y​\(A/B\)Lv\_\{\\max\}^\{L\}=y\(A/B\)^\{L\}yields, after cancellation and algebra, \([80](https://arxiv.org/html/2606.05219#A6.E80)\)\. Finally,γWCR​\(L\)=ηWCR​S1=L​αWCR​\(L\)\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=\\eta\_\{\\mathrm\{WCR\}\}S\_\{1\}=L\\alpha\_\{\\mathrm\{WCR\}\}\(L\)andαWCR​\(L\)=y2/L−1/B​\(y\)\\alpha\_\{\\mathrm\{WCR\}\}\(L\)=y^\{2/L\-1\}/B\(y\)by \([76](https://arxiv.org/html/2606.05219#A6.E76)\), proving \([81](https://arxiv.org/html/2606.05219#A6.E81)\) and \([20](https://arxiv.org/html/2606.05219#S6.E20)\)\. ∎

### F\.5Numerical computation \(bisection\) and representative values

Equation \([80](https://arxiv.org/html/2606.05219#A6.E80)\) is one\-dimensional on the intervaly∈\(L−12​L−1,1\)y\\in\\bigl\(\\frac\{L\-1\}\{2L\-1\},1\\bigr\)\(which ensuresB​\(y\)\>0B\(y\)\>0\)\. In practice, we solve it via bisection in that interval and then computeγWCR​\(L\)\\gamma\_\{\\mathrm\{WCR\}\}\(L\)by \([81](https://arxiv.org/html/2606.05219#A6.E81)\)\.

Representative values \(computed to high precision\) are:

L2345681020γWCR​\(L\)=ηWCR/η13\.1963\.6613\.9794\.2244\.4274\.7515\.0085\.844\\begin\{array\}\[\]\{c\|cccccccc\}L&2&3&4&5&6&8&10&20\\\\ \\hline\\cr\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=\\eta\_\{\\mathrm\{WCR\}\}/\\eta\_\{1\}&3\.196&3\.661&3\.979&4\.224&4\.427&4\.751&5\.008&5\.844\\end\{array\}\(82\)and the values continue to increase withLL\(e\.g\.,γWCR​\(50\)≈7\.04\\gamma\_\{\\mathrm\{WCR\}\}\(50\)\\approx 7\.04,γWCR​\(100\)≈8\.02\\gamma\_\{\\mathrm\{WCR\}\}\(100\)\\approx 8\.02\)\.

### F\.6Asymptotic growth with depth

In this subsection we justify Proposition[6\.3](https://arxiv.org/html/2606.05219#S6.Thmtheorem3)\. LetγWCR​\(L\)=L​αWCR​\(L\)\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=L\\alpha\_\{\\mathrm\{WCR\}\}\(L\), and lety=y​\(L\)y=y\(L\)be the solution of \([80](https://arxiv.org/html/2606.05219#A6.E80)\)\. Definet:=2​y−1∈\(0,1\)t:=2y\-1\\in\(0,1\)\(soy=\(1\+t\)/2y=\(1\+t\)/2\)\. A convenient form of the boundary algebra \(equivalent to \([80](https://arxiv.org/html/2606.05219#A6.E80)\)\) is

B​\(y\)=r​\(y\)L−2​\(y​r​\(y\)L−1\),r​\(y\):=A​\(y\)B​\(y\)\.B\(y\)=r\(y\)^\{L\-2\}\\bigl\(y\\,r\(y\)^\{L\}\-1\\bigr\),\\qquad r\(y\):=\\frac\{A\(y\)\}\{B\(y\)\}\.\(83\)
For largeLL, the root satisfiest↓0t\\downarrow 0and hencey↓1/2y\\downarrow 1/2\. In that regime, a standard expansion shows

r​\(y\)=1\+12​L​t\+o​\(1L​t\),⇒r​\(y\)L=exp⁡\(12​t\+o​\(1\)\)\.r\(y\)=1\+\\frac\{1\}\{2Lt\}\+o\\\!\\left\(\\frac\{1\}\{Lt\}\\right\),\\qquad\\Rightarrow\\qquad r\(y\)^\{L\}=\\exp\\\!\\left\(\\frac\{1\}\{2t\}\+o\(1\)\\right\)\.\(84\)Moreover, sinceB​\(y\)=L​t\+O​\(1\)B\(y\)=Lt\+O\(1\)andy=1/2\+O​\(t\)y=1/2\+O\(t\), equation \([83](https://arxiv.org/html/2606.05219#A6.E83)\) implies \(to leading order\)

exp⁡\(1t\)≍L​t\.\\exp\\\!\\left\(\\frac\{1\}\{t\}\\right\)\\asymp Lt\.\(85\)UsingγWCR​\(L\)=L​αWCR​\(L\)\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=L\\alpha\_\{\\mathrm\{WCR\}\}\(L\)andαWCR​\(L\)=y2/L−1/B​\(y\)\\alpha\_\{\\mathrm\{WCR\}\}\(L\)=y^\{2/L\-1\}/B\(y\), we haveγWCR​\(L\)∼2/t\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\\sim 2/t\(becausey→1/2y\\to 1/2andB∼L​tB\\sim Lt\)\. Substitutingt∼2/γWCRt\\sim 2/\\gamma\_\{\\mathrm\{WCR\}\}into \([85](https://arxiv.org/html/2606.05219#A6.E85)\) yields

exp⁡\(γWCR​\(L\)2\)≍LγWCR​\(L\),\\exp\\\!\\left\(\\frac\{\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\}\{2\}\\right\)\\asymp\\frac\{L\}\{\\gamma\_\{\\mathrm\{WCR\}\}\(L\)\},\(86\)which impliesγWCR​\(L\)=Θ​\(log⁡L\)\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=\\Theta\(\\log L\)\. More refined manipulations yield the Lambert\-WWscaling

γWCR​\(L\)=2​W​\(Θ​\(L\)\)=2​log⁡L−2​log⁡log⁡L\+O​\(1\),\\gamma\_\{\\mathrm\{WCR\}\}\(L\)=2W\(\\Theta\(L\)\)=2\\log L\-2\\log\\log L\+O\(1\),\(87\)establishing Proposition[6\.3](https://arxiv.org/html/2606.05219#S6.Thmtheorem3)\.

## Appendix GOther Figures

η<2/S1\\quad\\eta<2/S\_\{1\}

η\>2/S1\\quad\\eta\>2/S\_\{1\}

η=2​H2/L−1/S1\\quad\\eta=2H^\{2/L\-1\}/S\_\{1\}

η\>2​H2/L−1/S1\\quad\\eta\>2H^\{2/L\-1\}/S\_\{1\}

![Refer to caption](https://arxiv.org/html/2606.05219v1/x11.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x12.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x13.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x14.png)

Figure 7:Training dynamics of GD with multi path MLP \(L=3L=3,H=2H=2,σ⋆1=1\\sigma\_\{\\star 1\}=1\) according to learning rateη\\eta\. GD with larger learning rateη\\etamitigate sharp minima withλmax\>2/η\\lambda\_\{\\max\}\>2/\\eta\.![Refer to caption](https://arxiv.org/html/2606.05219v1/x15.png)Figure 8:Trajectory of\(σ11,σ21\)\(\\sigma\_\{11\},\\sigma\_\{21\}\)for heterogeneous depth model with learning rateη∗\\eta^\{\*\}\.![Refer to caption](https://arxiv.org/html/2606.05219v1/x16.png)Figure 9:Trajectory ofσh​1\\sigma\_\{h1\}for learning rateη=0\.99​ηWCR\\eta=0\.99\\eta\_\{\\mathrm\{WCR\}\}\. As learning rates getting bigger,σh​1\\sigma\_\{h1\}returns close to zero\.η<2/S1\\quad\\eta<2/S\_\{1\}η\>2/S1\\quad\\eta\>2/S\_\{1\}η=2/λ1min\\quad\\eta=2/\\lambda\_\{1\}^\{\\min\}
![Refer to caption](https://arxiv.org/html/2606.05219v1/x17.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x18.png)
![Refer to caption](https://arxiv.org/html/2606.05219v1/x19.png)

Figure 10:Trajecory ofσh​1\\sigma\_\{h1\}trained withη1=2/λ1min\\eta\_\{1\}=2/\\lambda\_\{1\}^\{\\min\}with 2D unfolded view on minima∑h=13σh​1=σ⋆1\\sum\_\{h=1\}^\{3\}\\sigma\_\{h1\}=\\sigma\_\{\\star 1\}\.