The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

# The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
Source: [https://arxiv.org/html/2605.23087](https://arxiv.org/html/2605.23087)
###### Abstract

Neural collapse \(NC\) describes the structured geometry that emerges in the features and weights of trained classifiers\. Recent theory suggests NC can be suboptimal in deep architectures, attributing this to an explicit low\-rank bias fromL2L\_\{2\}regularization\. We study the deep unconstrained feature model \(UFM\)—equivalent to a deep linear network with orthogonal inputs—trained*without*regularization, to isolate how gradient descent and depth alone shape NC\. We show that depth induces an implicit low\-rank bias: low\-rank matrices propagate norm more efficiently through successive multiplications, promoting low\-rank alternatives to NC\. These alternatives, we argue, correspond to softmax codes: max\-margin solutions previously found in width\-bottlenecked networks\. Analyzing training dynamics under spectral initialization, we identify an early\-time repulsion among singular values that drives low\-rank emergence, and characterize how depth shrinks NC’s basin of attraction\. Finally, we show that some effects act in the opposite direction: for randomly initialized networks, increasing width biases training toward higher\-rank solutions\. Our results provide the first asymptotic and dynamic characterization of implicit bias in deep UFMs trained with unregularized multiclass cross\-entropy\.

Neural Collapse, Gradient Descent, Deep Learning, Low\-Rank Bias\.

## 1Introduction

Deep neural\-network classifiers, when trained beyond interpolation, converge to a highly structured geometric arrangement termed neural collapse\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129)\): class means form a simplex equiangular tight frame \(ETF\), and both features and classifier weights align with this frame\. Interestingly, this configuration arises without any explicit regularization or architectural constraint\.*What biases inherent to gradient\-descent dynamics or the architecture give rise to such geometric regularities?*

A substantial body of work approaches this question using the unconstrained features model \(UFM\)\(Mixonet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib143); Fanget al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib125)\), which treats final\-layer features as free optimization variables jointly optimized with the classifier weights, abstracting away the preceding layers\. The majority of these studies analyze the UFM withL2L\_\{2\}regularization on both features and weights, primarily under mean squared error \(MSE\) loss\(Mixonet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib143); Hanet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib148); Zhouet al\.,[2022a](https://arxiv.org/html/2605.23087#bib.bib144); Tirer and Bruna,[2022](https://arxiv.org/html/2605.23087#bib.bib130); Danget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib29); Tireret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib8)\), and less often under the more practically relevant cross\-entropy \(CE\) loss\(Zhuet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib42); Liet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib85); Zhaoet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib113)\)\. In this framework, regularization becomes essential for NC to emerge\. However, when the UFM is extended to multiple layers—yielding the*deep UFM*—NC and its generalization, deep neural collapse \(DNC\), are no longer globally optimal underL2L\_\{2\}regularization\(Súkeníket al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib28); Garrod and Keating,[2024a](https://arxiv.org/html/2605.23087#bib.bib39)\):L2L\_\{2\}regularization induces a low\-rank bias that suppresses NC in deep architectures\.

Analyses based onL2L\_\{2\}regularization face key limitations\. First, it is not clear whether the regularized model informs us about structures that emerge in unregularized training\(Thrampoulidiset al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib122)\)\. Second, in the deep UFM, despite NC being suboptimal at finite regularization, large regularization values are empirically needed to prevent it from emerging\(Súkeníket al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib28); Garrod and Keating,[2024a](https://arxiv.org/html/2605.23087#bib.bib39)\)\. Third, and perhaps more concerning, the wayL2L\_\{2\}regularization is applied in the UFM—directly on the features—differs markedly from standard training, where regularization acts on network weights\.

These concerns are mitigated when examining UFMs without explicit regularization\.Garrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)shows that the UFM trained with CE loss*without*explicit regularization still converges to NC, demonstrating that NC arises from the implicit bias of CE training alone\. Yet, their analysis is restricted to a single hidden layer\. It remains open how depth impacts the implicit bias:*How do the inductive biases induced by depth affect convergence to NC, both asymptotically and at finite training time? If depth favors alternatives to NC, what geometric structures replace it, and why is NC nonetheless pervasive in practical deep networks?*

### 1\.1Contributions

We provide the first comprehensive characterization of how implicit biases from depth, optimization dynamics, and initialization interact to determine whether NC emerges\. Our setting is the deep UFM trained with CE loss*without*explicit regularization—a model where the loss minimum is achieved only at infinite parameter norm, and many geometric configurations attain it\. The structure that emerges thus reflects purely the implicit biases of optimization and model depth, which we analyze from two complementary perspectives:

Asymptotics: depth induces low\-rank bias\.Asymptotically in time, gradient\-flow \(GF\) dynamics reduce to a non\-convex max\-margin problem\(Lyu and Li,[2019](https://arxiv.org/html/2605.23087#bib.bib108)\)\. By analyzing its landscape, we show that, unlike the shallow case, the deep landscape admits multiple stable local minima, among which NC is strictly suboptimal\. We identify the mechanism behind this non\-benign landscape: low\-rank structures propagate norms more effectively through matrix multiplications, achieving larger logits for a fixed parameter norm\. In the large\-depth limit, we prove the global optimum corresponds to thed=2d=2softmax code\(Jianget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib121)\), connecting depth\-induced bias to explicit width constraints\. Empirical evidence suggests this connection extends to higher\-rank softmax codes at finite depths\.

Dynamics: depth reshapes the trajectory\.Building on the Hadamard framework ofGarrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\), we analyze GF dynamics at finite time\. Unlike the shallow UFM, where NC is the unique stable direction throughout the trajectory, in the multilayer setting we prove this no longer holds: NC becomes unstable near the origin, with the unstable region growing with the number of layers, while alternative low\-rank directions become stable\. We show the mechanism is a “rich\-get\-richer” effect in which larger singular values grow faster, promoting low\-rank structures before exiting the linearized regime\. We further show that the KL divergence to NC, a Lyapunov function for the shallow UFM, can diverge under depth\. Finally, we prove that random initialization induces concentration\-of\-measure effects that, for sufficiently large network widths, drive early dynamics toward NC\. This provides a justification for why low\-rank structures—though theoretically favored by depth—are not a certainty in practice\.

We validate our results experimentally on deep UFMs and neural networks\. Beyond NC, the deep UFM, which is equivalent to a linear network with orthogonal inputs, is the minimal model in which depth\-induced implicit bias can be isolated from explicit regularization\. Our analysis of this canonical setting thus informs the broader question of how depth shapes implicit bias in deep learning\.

### 1\.2Related Work

Here, we review the most closely related works\. See App\.[A](https://arxiv.org/html/2605.23087#A1)for a full review and additional references\.

Deep NC has been explored empirically byHe and Su \([2022](https://arxiv.org/html/2605.23087#bib.bib136)\); Parkeret al\.\([2023](https://arxiv.org/html/2605.23087#bib.bib140)\); Rangamaniet al\.\([2023](https://arxiv.org/html/2605.23087#bib.bib138)\), but theoretical efforts have been mainly based on deep UFMs trained with*MSE loss*\(Tirer and Bruna,[2022](https://arxiv.org/html/2605.23087#bib.bib130); Danget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib29); Súkeníket al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib131); Garrod and Keating,[2024b](https://arxiv.org/html/2605.23087#bib.bib141); Súkeníket al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib28)\)\. Again under MSE loss, a parallel direction examines implicit regularization in more general linear networks, studying the interplay between GD and depth\(Aroraet al\.,[2018a](https://arxiv.org/html/2605.23087#bib.bib61),[b](https://arxiv.org/html/2605.23087#bib.bib38); Bahet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib98); Yaraset al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib100); Tuet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib83)\)and matrix factorization dynamics\(Gunasekaret al\.,[2017](https://arxiv.org/html/2605.23087#bib.bib78); Aroraet al\.,[2019](https://arxiv.org/html/2605.23087#bib.bib34); Liet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib112); Razin and Cohen,[2020](https://arxiv.org/html/2605.23087#bib.bib99)\)\. The more practically relevant CE setting is significantly less studied due to its added complexity\. Only recently,Garrod and Keating \([2024a](https://arxiv.org/html/2605.23087#bib.bib39)\)showed thatexplicitL2L\_\{2\}regularization induces a low\-rank bias in DNC\. Ours is the first direct analysis of the deep UFM with CE loss and without explicit regularization, focusing instead on the implicit biases induced by GF dynamics\.

Within the implicit\-bias literature, the most closely related study isJi and Telgarsky \([2019](https://arxiv.org/html/2605.23087#bib.bib31)\), which analyzed GF dynamics in deep linear networks under binary logistic loss\. While the deep UFM is a special case of a deep linear network with orthogonal inputs, our results are significantly more general as they:*\(i\)*directly apply to*multiclass*settings;*\(ii\)*are conclusive about GF convergence to*explicit geometric configurations*rather than implicit non\-convex characterizations via KKT points\(Lyu and Li,[2019](https://arxiv.org/html/2605.23087#bib.bib108)\);*\(iii\)*go beyond asymptotic convergence to study the*full dynamics*\. GF dynamics for CE loss are significantly more challenging than for MSE: even in shallow UFM, it was only recently thatGarrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)introduced a spectral\-initialization\-based framework to study these dynamics, extending prior work bySaxeet al\.\([2013](https://arxiv.org/html/2605.23087#bib.bib93)\)for MSE\. We generalize this framework to deep models\.

Prior work has shown that depth can bias optimization toward low\-rank solutions, especially in matrix factorization\(Gunasekaret al\.,[2017](https://arxiv.org/html/2605.23087#bib.bib78); Aroraet al\.,[2019](https://arxiv.org/html/2605.23087#bib.bib34); Liet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib112); Chouet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib80)\), with related evidence in nonlinear and homogeneous networks\(Huhet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib71); Jacot,[2023](https://arxiv.org/html/2605.23087#bib.bib81); Timoret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib54)\)\. Our work differs by focusing on a model of CE\-trainedoverparameterizedclassifiers, showing that depth in this setting induces low\-rank representations, reduces normalized margins, and drives a transition from neural collapse to softmax codes\.

## 2Background

\(Deep\) UFM\.We study gradient flow \(GF\) on the deep unconstrained features model \(UFM\) with CE loss forKK\-class classification withnnsamples per class\. The model parameters are matricesWL∈ℝK×dW\_\{L\}\\in\\mathbb\{R\}^\{K\\times d\},WL−1,…,W1∈ℝd×dW\_\{L\-1\},\\ldots,W\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\}, andH1∈ℝd×n​KH\_\{1\}\\in\\mathbb\{R\}^\{d\\times nK\}, whereddis the embedding dimension\. The columns ofH1H\_\{1\}are*trainable*feature vectorshi​c∈ℝdh\_\{ic\}\\in\\mathbb\{R\}^\{d\}for exampleiiof classcc, ordered by class\. The loss is

ℒ​\(Z\)=−∑c=1K∑i=1nlog⁡\(exp⁡\(\(zi​c\)c\)∑c′=1Kexp⁡\(\(zi​c\)c′\)\),\\displaystyle\\mathcal\{L\}\(Z\)=\-\\sum\_\{c=1\}^\{K\}\\sum\_\{i=1\}^\{n\}\\log\\Bigg\(\\frac\{\\exp\(\(z\_\{ic\}\)\_\{c\}\)\}\{\\sum\_\{c^\{\\prime\}=1\}^\{K\}\\exp\(\(z\_\{ic\}\)\_\{c^\{\\prime\}\}\)\}\\Bigg\),\(1\)wherezi​c=WL​WL−1​⋯​W1​hi​cz\_\{ic\}=W\_\{L\}W\_\{L\-1\}\\cdots W\_\{1\}h\_\{ic\}are logit vectors forming the columns of the logit matrixZ∈ℝK×n​KZ\\in\\mathbb\{R\}^\{K\\times nK\}\. WhenL=1L=1\(single hidden layer\), this reduces to the \(shallow\) UFM; our focus is on the deep caseL\>1L\>1\.

The deep UFM serves as an abstraction of overparameterized DNN classifiers, where the featureshi​ch\_\{ic\}represent unconstrained embeddings trained jointly with the weights\(Mixonet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib143); Fanget al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib125); Hanet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib148)\)\. This models a highly expressive feature map, for example a ResNet, with a linear head\. We leave consideration of homogeneous activations in the head for future work\. See App\.[B](https://arxiv.org/html/2605.23087#A2)for more details andGarrod and Keating \([2024b](https://arxiv.org/html/2605.23087#bib.bib141), App\. A\)for justification of the UFM as a modeling tool of deep learning\. That aside, since the deep UFM is equivalent to anLL\-layer linear network with orthogonal inputs \(i\.e\., the simplest architecture exhibiting depth\-dependent dynamics\), it serves as the minimal setting for disentangling explicit regularization from implicit biases induced by optimization and depth\. Linear networks have been studied extensively in deep\-learning theory, but almost exclusively under MSE loss rather than the more practically relevant CE loss\.

\(Deep\) neural collapse\.DNC refers to geometric regularities observed empirically in the final layers of trained networks\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129); Parkeret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib140)\)\. The key property is that the Gram matrix of centered class mean embeddings becomes proportional to the simplex ETF matrixS=IK−1K​1K​1KTS=I\_\{K\}\-\\frac\{1\}\{K\}\{1\}\_\{K\}\{1\}\_\{K\}^\{T\}\. We defer a detailed definition of DNC to App\.[B](https://arxiv.org/html/2605.23087#A2); here we work with the following well\-established equivalent characterization\(Garrod and Keating,[2024a](https://arxiv.org/html/2605.23087#bib.bib39)\)\. Let⊗\\otimesdenote the Kronecker product\.

###### Definition 2\.1\(DNC in the deep UFM\)\.

A DNC solution occurs when the normalized parameter matricesW^L,…,W^1,H^1\\hat\{W\}\_\{L\},\\ldots,\\hat\{W\}\_\{1\},\\hat\{H\}\_\{1\}\(whereX^=X/‖X‖F\\hat\{X\}=X/\\\|X\\\|\_\{F\}\) satisfy the balancedness relations

W^lT​W^l=W^l−1​W^l−1T,l=1,…,L,\\hat\{W\}\_\{l\}^\{T\}\\hat\{W\}\_\{l\}=\\hat\{W\}\_\{l\-1\}\\hat\{W\}\_\{l\-1\}^\{T\},\\quad l=1,\\ldots,L,\(2\)withW^0≔H^1\\hat\{W\}\_\{0\}\\coloneqq\\hat\{H\}\_\{1\}, and the normalized logit matrix is proportional to the simplex:W^L​⋯​W^1​H^1∝S⊗1nT\\hat\{W\}\_\{L\}\\cdots\\hat\{W\}\_\{1\}\\hat\{H\}\_\{1\}\\propto S\\otimes\{1\}\_\{n\}^\{T\}\.

Balancedness emerges asymptotically: the quantityWlT​Wl−Wl−1​Wl−1TW\_\{l\}^\{T\}W\_\{l\}\-W\_\{l\-1\}W\_\{l\-1\}^\{T\}is conserved under GF, and since parameter norms diverge, the normalized matrices satisfy \([2](https://arxiv.org/html/2605.23087#S2.E2)\) in the limit\(Ji and Telgarsky,[2019](https://arxiv.org/html/2605.23087#bib.bib31)\)\.

ForL=1L=1,Jiet al\.\([2022](https://arxiv.org/html/2605.23087#bib.bib145)\)showed that GF converges in direction to a KKT point of a non\-convex max\-margin problem \(Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\)\), with NC as the unique global minimum and all other critical points being saddles\.Garrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)further proved convergence to NC under spectral initialization, showing that the KL divergence from the parameters to the ETF structure decreases monotonically throughout training\. ForL\>1L\>1, both questions—whether the asymptotic landscape remains benign and how the dynamics evolve at finite time—have remained open\.

## 3Implicit Bias in the deep UFM

Here, we characterize the geometric structures that arise as limiting configurations of GF in the deep UFM\. The CE objective admits no finite minimizers: the infimum of zero is achieved only at infinity, and by a wide variety of parameter configurations\. This is the essence of implicit bias: although many directions asymptotically attain zero training loss, the optimization dynamics select structured ones from among them\. We proceed in three steps: we first show that DNC is no longer the unique optimum onceL\>1L\>1, then explain the mechanism behind this—a low\-rank bias induced by depth in such models, and finally obtain an explicit characterization of the optimal low\-rank structures in the large\-depth limit\.

### 3\.1Depth Breaks the Benign Landscape

*Does the benign landscape of the shallow UFM persist with depth?*To answer this we apply the framework ofLyu and Li \([2019](https://arxiv.org/html/2605.23087#bib.bib108)\)to the deep UFM \(Eq\.[1](https://arxiv.org/html/2605.23087#S2.E1)\)\. This shows that GF on CE loss implicitly performs \(non\-convex\) margin maximization\. Concretely, if there exists timet0t\_\{0\}such that the lossℒ​\(t0\)<log⁡\(2\)\\mathcal\{L\}\(t\_\{0\}\)<\\log\(2\), then any limit point of the normalized parameters\(WL^,…,W^1,H1^\)\(\\hat\{W\_\{L\}\},\.\.\.,\\hat\{W\}\_\{1\},\\hat\{H\_\{1\}\}\)lies in the direction of a Karush\-Kuhn\-Tucker \(KKT\) point of the constrained optimization problem

minWL,…,W1,H1⁡‖H1‖F2\+∑l=1L‖Wl‖F2\\displaystyle\\qquad\\qquad\\min\_\{W\_\{L\},\.\.\.,W\_\{1\},H\_\{1\}\}\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}\(3\)s\.t\.​\(zi​c\)c−\(zi​c\)c′≥1,c′≠c=1,…,K,i=1,…,n,\\displaystyle\\textrm\{s\.t\. \}\(z\_\{ic\}\)\_\{c\}\-\(z\_\{ic\}\)\_\{c^\{\\prime\}\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K,\\ i=1,\.\.\.,n,wherezi​cz\_\{ic\}are the columns in class order of the logit matrixZ=WL​…​W1​H1Z=W\_\{L\}\.\.\.W\_\{1\}H\_\{1\}\. Since this problem is non\-convex, KKT points may be global minima, local minima, or saddles\. Our first contribution is to characterize this landscape; the following theorem shows the benign structure of the shallow case shown byJiet al\.\([2022](https://arxiv.org/html/2605.23087#bib.bib145)\)does*not*persist\.

###### Theorem 3\.1\.

Consider the optimization problem in Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\) withd≥Kd\\geq K\. ForL=2,K≥6L=2,K\\geq 6orL≥3,K≥4L\\geq 3,K\\geq 4, DNC is not globally optimal, although it remains a local optimum\. Consequently, the landscape is no longer benign: multiple distinct locally optimal structures exist\.

The proof appears in App\.[D\.1](https://arxiv.org/html/2605.23087#A4.SS1)\. This reveals a fundamental distinction: onceL\>1L\>1, the landscape admits multiple stable configurations and GF may converge to structures other than DNC—bringing it qualitatively closer to practical deep\-learning settings\.

A qualitatively similar phenomenon, i\.e\., suboptimality of DNC at depth, was observed in regularized deep UFMs\(Súkeníket al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib28); Garrod and Keating,[2024a](https://arxiv.org/html/2605.23087#bib.bib39)\), leading to interpretations thatL2L\_\{2\}regularization is the cause\. From a technical standpoint, the two settings differ fundamentally: regularized analyses study finite global minimizers, whereas Theorem[3\.1](https://arxiv.org/html/2605.23087#S3.Thmtheorem1)characterizes the asymptotic hard\-margin regime\. From a conceptual standpoint, our analysis shows the interpretation of previous work is incomplete: Theorem[3\.1](https://arxiv.org/html/2605.23087#S3.Thmtheorem1)establishes that depth alone, without any explicit regularization, suffices to induce the suboptimality of DNC\. The next subsection explains why\.

![Refer to caption](https://arxiv.org/html/2605.23087v1/Experiments/Hypersphere_Illustration/hypersphere_illustration.png)Figure 1:Illustration of low\-rank bias forK=4,L=3,n=1K=4,L=3,n=1, with all parameter matrices normalized to unit Frobenius norm\.μDNC,μDNC′\\mu\_\{\\textrm\{DNC\}\},\\mu\_\{\\textrm\{DNC\}\}^\{\\prime\}denote two feature embeddings of a logit matrixZZunder DNC, whileμLR,μLR′\\mu\_\{\\textrm\{LR\}\},\\mu\_\{\\textrm\{LR\}\}^\{\\prime\}denote two feature embeddings of a logit matrixZZunder the low\-rank geometry of Eq\. \([14](https://arxiv.org/html/2605.23087#A4.E14)\)\. Although DNC yields the larger angle between features, the low\-rank geometry lies on a large\-radius hypersphere and therefore attains a larger Euclidean separation\.
### 3\.2The Mechanism: Low\-Rank Structures Propagate Norm More Efficiently

The mechanism by which depth alters the loss surface in such models is a low\-rank bias: low\-rank matrices propagate norm more effectively through a sequence of matrix multiplications\. To make this precise, consider the logit matrixZZformed as a product of parameter matrices all having the same Frobenius norm, which is true under the balancedness condition \(Eq\. \([2](https://arxiv.org/html/2605.23087#S2.E2)\)\)\. WritingW0:=H1W\_\{0\}:=H\_\{1\}, we decompose

Z=\(∏l=0L‖Wl‖F\)⋅Z~,where​Z~=W^L​⋯​W^1​W^0\.Z=\\big\(\\prod\_\{l=0\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}\\big\)\\cdot\\tilde\{Z\}\\,,\\,\\,\\text\{where\}\\,\\,\\tilde\{Z\}=\\hat\{W\}\_\{L\}\\cdots\\hat\{W\}\_\{1\}\\hat\{W\}\_\{0\}\\,\.\(4\)Once positive margins are achieved, the loss decreases monotonically as the norms grow, and each‖Wl‖F\\\|W\_\{l\}\\\|\_\{F\}increases monotonically under gradient flow\(Lyu and Li,[2019](https://arxiv.org/html/2605.23087#bib.bib108)\)\. Therefore, differences between asymptotic structures are governed byZ~\\tilde\{Z\}\. Crucially, it is clear from Eq\. \([4](https://arxiv.org/html/2605.23087#S3.E4)\) that‖Z~‖F\\\|\\tilde\{Z\}\\\|\_\{F\}contributes to the overall scale ofZZ, so maximizing‖Z~‖F\\\|\\tilde\{Z\}\\\|\_\{F\}is advantageous for reducing the loss by widening the margins\. We thus compare‖Z~‖F\\\|\\tilde\{Z\}\\\|\_\{F\}under DNC versus low\-rank alternatives\.

Under DNC, and takingn=1n=1for simplicity, using Definition[2\.1](https://arxiv.org/html/2605.23087#S2.Thmtheorem1)and a small calculation showsZ~DNC=S^L\+1\\tilde\{Z\}\_\{\\textrm\{DNC\}\}=\\hat\{S\}^\{L\+1\}, yielding‖Z~DNC‖F=\(K−1\)−L/2\.\\\|\\tilde\{Z\}\_\{\\textrm\{DNC\}\}\\\|\_\{F\}=\(K\-1\)^\{\-L/2\}\.By contrast, consider solutions with positive margins whose normalized logits have equal singular values but rankr<K−1r<K\-1111Concrete examples withr=K/2r=K/2are given in App\.[D\.1](https://arxiv.org/html/2605.23087#A4.SS1), andr=2r=2in Theorem[3\.2](https://arxiv.org/html/2605.23087#S3.Thmtheorem2)\.\. For such low\-rank structures, we can compute that‖Z~LR‖F=r−L2,\\\|\\tilde\{Z\}\_\{\\textrm\{LR\}\}\\\|\_\{F\}=r^\{\-\\frac\{L\}\{2\}\},and hence

‖Z~DNC‖F‖Z~LR‖F=exp⁡\(−L2​log⁡\(K−1r\)\)\.\\frac\{\\\|\\tilde\{Z\}\_\{\\textrm\{DNC\}\}\\\|\_\{F\}\}\{\\\|\\tilde\{Z\}\_\{\\textrm\{LR\}\}\\\|\_\{F\}\}=\\exp\\left\(\-\\frac\{L\}\{2\}\\log\\left\(\\frac\{K\-1\}\{r\}\\right\)\\right\)\.The logit norm achievable under DNC is thus exponentially suppressed in depth relative to lower\-rank alternatives\. This occurs because of the basic linear\-algebra effect that low\-rank matrices “propagate norm” more efficiently through repeated matrix multiplication\. This is since fewer singular vector directions means larger individual singular values for a given Frobenius norm, causing less decay under matrix powers, with the effect strengthening with increasingLL\.

![Refer to caption](https://arxiv.org/html/2605.23087v1/x1.png)Figure 2:Effective rank \(Eq\. \([60](https://arxiv.org/html/2605.23087#A5.E60)\), App\.[E](https://arxiv.org/html/2605.23087#A5)\) of the logit matrix at GF convergence forK=10K=10classes and varying network depthLL\. For each value ofLLwe trained deep UFM for five random initializations and computed the effective rank after training\. Mean±\\pmone standard deviation is shown\. Rank decreases with depth\.![Refer to caption](https://arxiv.org/html/2605.23087v1/Experiments/global_minima_example/uniform_circle_features.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x2.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/Experiments/softmax_codes/softmax_codes.png)

Figure 3:Optimal low\-rank structures correspond to softmax codes\.Left:Columns of the gram factorXXdefined in Eq\. \([5](https://arxiv.org/html/2605.23087#S3.E5)\) for one run withL=10L=10that converged to rank 2\.Middle:For eachKK, we trained five deep UFMs that converged to rank 2; shown are the largest and smallest adjacent angle gaps of the columnsxix\_\{i\}of the Gram factorXXover all five runs\. The angle360/K360/Kcorresponds to thed=2d=2softmax code\.Right:Normalized margins for low\-rank solutions versus architecturally bottlenecked solutions\. Dotted lines:L=1L=1UFM with widthd=rd=rwhich is known to yield softmax codes\(Jianget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib121)\)\. Solid lines: networks withL=4L=4andd≥Kd\\geq Kthat converged to rankrr, for varyingKK\.Angular versus Euclidean separation\.For fixed‖Z‖F\\\|Z\\\|\_\{F\}, DNC is the maximum\-margin solution under direct logit optimization\(Thrampoulidiset al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib122); Garrodet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib9)\), and is known to maximize the angular separation between class means\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129)\)—a property closely related to margin maximization\(Jianget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib121)\)\. However, depth distinguishes angular separation from Euclidean separation\. Low\-rank structures can attain larger logit norms, and thus larger Euclidean margins, simply by lying on a larger hypersphere \(even though their angular separation is inferior\)\. Figure[1](https://arxiv.org/html/2605.23087#S3.F1)illustrates this geometry: DNC maximizes the angle between logit means, but the low\-rank configuration achieves greater Euclidean separation by residing on a hypersphere of larger radius\. Figure[2](https://arxiv.org/html/2605.23087#S3.F2)provides empirical confirmation: as depth increases, the effective rank of the converged logit matrix decreases systematically\.

Discrimination versus confidence\.Low\-rank solutions can achieve lower training loss than DNC despite being worse normalized classifiers\. Since diverging logit norms drive softmax outputs toward one\-hot vectors, larger‖Z‖F\\\|Z\\\|\_\{F\}contributes only by increasing prediction confidence\. Maximal angular separation, by contrast, accounts for the discrimination between classes\. Low\-rank solutions thus trade discrimination quality \(encoded byZ^\\hat\{Z\}, which determines the predicted label\) for increased confidence \(encoded by‖Z‖F\\\|Z\\\|\_\{F\}\)\. This suggests they may overfit more and generalize worse than DNC, connecting their rarity in standard training\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129)\)to the broader question of why deep networks generalize despite sufficient capacity to overfit\(Zhanget al\.,[2017](https://arxiv.org/html/2605.23087#bib.bib37)\)\.

### 3\.3Characterizing the Optimal Low\-Rank Structures

Having established that low\-rank structures can outperform DNC and explained the mechanism, we now ask a more challenging question:*what are these optimal structures?*Since explicit characterizations of non\-convex landscapes are difficult to obtain, we focus on a large\-depth limit\.

###### Theorem 3\.2\.

Consider the optimization problem in Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\), with class numberK\>2K\>2and network widthd≥Kd\\geq K\. In the limitL→∞L\\to\\infty, the globally optimal solutions among the set of positive semi\-definite logit matrices satisfying NC1 \(meaningZZcan be written asZ=Z¯⊗1nTZ=\\bar\{Z\}\\otimes 1\_\{n\}^\{T\}\) and equal norm features is of the form

Z=XT​X⊗1nT,X∈ℝ2×K,Z=X^\{T\}X\\otimes 1\_\{n\}^\{T\},\\quad X\\in\\mathbb\{R\}^\{2\\times K\},\(5\)where the columnsxi∈ℝ2x\_\{i\}\\in\\mathbb\{R\}^\{2\}ofXXare uniformly spaced on a circle, meaning for someα,μ∈ℝ\\alpha,\\mu\\in\\mathbb\{R\}, the set\{xi:i=1,…,K\}\\\{x\_\{i\}:i=1,\.\.\.,K\\\}is equal to\{\(μ​cos⁡\(2​π​iK\+α\),μ​sin⁡\(2​π​iK\+α\)\):i∈\[K\]\}\.\\left\\\{\\left\(\\mu\\cos\\left\(\\frac\{2\\pi i\}\{K\}\+\\alpha\\right\),\\mu\\sin\\left\(\\frac\{2\\pi i\}\{K\}\+\\alpha\\right\)\\right\):i\\in\[K\]\\right\\\}\.

The proof appears in App\.[D\.2](https://arxiv.org/html/2605.23087#A4.SS2)\. Figure[3](https://arxiv.org/html/2605.23087#S3.F3)confirms the prediction: for largeLL, when GD converges to rank two solutions they always correspond to the theorem’s structure: class means arranged as vertices of a regularKK\-gon\.

Connection to softmax codes\.The structure identified in Theorem[3\.2](https://arxiv.org/html/2605.23087#S3.Thmtheorem2)coincides with the optimal solution of the single\-layer UFM under an extreme width bottleneckd=2d=2\(Jianget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib121)\)\. More generally,Jianget al\.\([2023](https://arxiv.org/html/2605.23087#bib.bib121)\)show that whend<K−1d<K\-1, NC is geometrically infeasible and alternative structures—which they term*softmax codes*—must arise\. Theorem[3\.2](https://arxiv.org/html/2605.23087#S3.Thmtheorem2)demonstrates that large depths select precisely thed=2d=2softmax code, even when no width constraint is present\. Figure[3](https://arxiv.org/html/2605.23087#S3.F3)\(right\) suggests this connection extends beyond theL→∞L\\to\\inftylimit: the normalized margin achieved by GF on a depth\-LLnetwork that converges to a rank\-rrlogit matrix closely matches that of a single\-layer network with widthrr, suggesting these alternative optima are softmax codes more generally\. This confirms that depth acts as an implicit rank constraint, and intriguingly suggests it selects solutions that would otherwise require explicit width bottlenecks\. Extending Theorem[3\.2](https://arxiv.org/html/2605.23087#S3.Thmtheorem2)to finiteLLis an exciting direction for future work\.

## 4A Dynamical Analysis of Implicit Bias

![Refer to caption](https://arxiv.org/html/2605.23087v1/x3.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x4.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x5.png)

Figure 4:Experiments under Hadamard initialization\.Left:Evolution of the normalized logit singular valuesa^i\\hat\{a\}\_\{i\}under Eq\. \([6](https://arxiv.org/html/2605.23087#S4.E6)\) for depthL=2L=2andK=16K=16classes; singular values initialized with uniform entries in\[0,1\]\[0,1\], then rescaled to haveL1L\_\{1\}norm10−310^\{\-3\}\. Note convergence to low\-rank solution where many modes approach zero\.Middle:Corresponding evolution of the logit norm‖a‖1=∑iai\\\|a\\\|\_\{1\}=\\sum\_\{i\}a\_\{i\}\. By the time norm increases and system exits the linearized regime, the low\-rank structure has already taken hold\.Right: The corresponding KL divergence \(definition in Eq\. \([52](https://arxiv.org/html/2605.23087#A4.E52)\)\.\)\.The previous section characterized implicit bias asymptotically: we identified the KKT points to which GF converges ast→∞t\\to\\inftyand studied the landscape of the corresponding max\-margin problem\. We now take a more ambitious step: understanding the dynamics that GF traverses*before*reaching the asymptotic regime\. This is particularly important because convergence to KKT points of Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\) only becomes accurate at exponentially large times—the convergence rate isO​\(1/log⁡t\)O\(1/\\log t\)\(Lyu and Li,[2019](https://arxiv.org/html/2605.23087#bib.bib108)\)\. Several natural questions arise:*When does DNC become a stable direction along the trajectory? Are there competing stable directions at finite times? What happens near initialization?*

ForL=1L=1,Garrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)recently provided complete answers under spectral initialization: NC is the unique stable direction throughout the entire trajectory, and the KL divergence to NC decreases monotonically for any positive initialization\. In other words, the dynamics not only converge to NC but approach it monotonically \(in KL distance\) at all times\. ForL\>1L\>1, we prove here that this picture changes entirely\.

### 4\.1The Hadamard Framework

To obtain these results, we employ Hadamard initialization\(Garrodet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib9)\), which provides a tractable framework for studying GF dynamics in CE models\. Under this initialization, the singular vectors of all parameter matrices remain fixed throughout training; evolution occurs only in the singular values\. The following theorem shows this remains true for arbitrary depth\. Although this is a special initialization, it suffices for our purposes: it reveals structure in the loss surface, and for instability questions it is enough to exhibit one trajectory along which a given structure is unstable\. Moreover, convergence to non\-DNC solutions under Hadamard initialization implies such alternatives exist in the full landscape\.

We further provide empirical evidence in App\.[E\.4](https://arxiv.org/html/2605.23087#A5.SS4)that both Hadamard initialization and small random display the same qualitative phenomenology\. This is a common observation in the spectral dynamics literature, where structured analytic trajectories serve as indicative paths of a much wider set of initializations\(Saxeet al\.,[2013](https://arxiv.org/html/2605.23087#bib.bib93),[2019](https://arxiv.org/html/2605.23087#bib.bib94); Gidelet al\.,[2019](https://arxiv.org/html/2605.23087#bib.bib102); Garrodet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib9)\)\.

###### Definition 4\.1\(Sylvester Hadamard Matrix\)\.

The Sylvester Hadamard matrices\{Φ2m:m∈ℕ\}\\\{\\Phi\_\{2^\{m\}\}:m\\in\\mathbb\{N\}\\\}are defined recursively by

Φ1=1,Φ2m=\[Φ2m−1Φ2m−1Φ2m−1−Φ2m−1\]∈ℝ2m×2m\.\\Phi\_\{1\}=1,\\quad\\Phi\_\{2^\{m\}\}=\\begin\{bmatrix\}\\Phi\_\{2^\{m\-1\}\}&\\Phi\_\{2^\{m\-1\}\}\\\\ \\Phi\_\{2^\{m\-1\}\}&\-\\Phi\_\{2^\{m\-1\}\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{2^\{m\}\\times 2^\{m\}\}\.

###### Theorem 4\.2\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}, and letU=1K​ΦU=\\frac\{1\}\{\\sqrt\{K\}\}\\Phi, whereΦ\\Phiis theK×KK\\times KSylvester Hadamard matrix\. Consider the deep UFM in Eq\. \([1](https://arxiv.org/html/2605.23087#S2.E1)\), with balanced classes of sizenn\. Initialize the parameter matrices asWL=U​DL​RLTW\_\{L\}=UD\_\{L\}R\_\{L\}^\{T\},Wl=Rl\+1​Dl​RlTW\_\{l\}=R\_\{l\+1\}D\_\{l\}R\_\{l\}^\{T\}forl=1,…,L−1l=1,\.\.\.,L\-1, andH1=R1​D0​VTH\_\{1\}=R\_\{1\}D\_\{0\}V^\{T\}, whereRL,…,R1∈ℝd×dR\_\{L\},\.\.\.,R\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\}are orthogonal, andV=U⊗QV=U\\otimes QwithQQa right singular matrix of1nT1\_\{n\}^\{T\}\. Assume the only non\-zero singular values ofDlD\_\{l\}areα0\(l\),…,αK−1\(l\)\\alpha^\{\(l\)\}\_\{0\},\.\.\.,\\alpha\_\{K\-1\}^\{\(l\)\}forl=0,…,L\+1l=0,\.\.\.,L\+1\. Then under balancedness, and absorbing constants into the time and singular value variables, the gradient flow equations reduce to

d​aid​t=1D​bi​ai2​LL\+1,i=1,…,K−1,\\frac\{da\_\{i\}\}\{dt\}=\\frac\{1\}\{D\}b\_\{i\}a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\},\\quad i=1,\.\.\.,K\-1,\(6\)whereai=∏l=0Lαi\(l\)a\_\{i\}=\\prod\_\{l=0\}^\{L\}\\alpha\_\{i\}^\{\(l\)\}are the logit singular values,bi=∑j=1K−1Ψi​j​e−\(Ψ​a\)jb\_\{i\}=\\sum\_\{j=1\}^\{K\-1\}\\Psi\_\{ij\}e^\{\-\(\\Psi a\)\_\{j\}\},D=1\+∑j=1K−1e−\(Ψ​a\)j,D=1\+\\sum\_\{j=1\}^\{K\-1\}e^\{\-\(\\Psi a\)\_\{j\}\},andΨ∈ℝK−1×K−1\\Psi\\in\\mathbb\{R\}^\{K\-1\\times K\-1\}is the core of the matrix1K​1KT−Φ1\_\{K\}1\_\{K\}^\{T\}\-\\Phi, constructed by deleting its first column and row\.

![Refer to caption](https://arxiv.org/html/2605.23087v1/x6.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x7.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x8.png)

Figure 5:Impact of network width on network training\. Depicted are the mean and one\-standard\-deviation error bars over five runs\.Left:Frobenius distance metricMM\(see \([61](https://arxiv.org/html/2605.23087#A5.E61)\) in App\.\) between the logit\-matrix derivatived​Zd​t​\(0\)\\frac\{dZ\}\{dt\}\(0\)and the simplex ETF\.Middle:Same metric for the logits after training has induced the logit norm to increase by a scale factor\.Right:Effective rank of the logit matrix after training\.The proof appears in App\.[D\.3](https://arxiv.org/html/2605.23087#A4.SS3)\. As in theL=1L=1case\(Garrodet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib9)\), the first singular valuea0a\_\{0\}of the logit matrix does not influence the evolution of the remaining singular values nor evolves itself and can be set to zero; we henceforth drop it from our notation, so thata∈ℝK−1a\\in\\mathbb\{R\}^\{K\-1\}\.

We are ultimately interested in the normalized logit matrix \(the direction\)\. Thus, normalizing with respect to theL1L\_\{1\}norm yields dynamics fora^i=ai/‖a‖1\\hat\{a\}\_\{i\}=a\_\{i\}/\\\|a\\\|\_\{1\}:

d​a^id​t=1D​‖a‖1​\[bi​ai2​LL\+1−a^i​∑jbj​aj2​LL\+1\]\.\\frac\{d\\hat\{a\}\_\{i\}\}\{dt\}=\\frac\{1\}\{D\\\|a\\\|\_\{1\}\}\\Big\[b\_\{i\}a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}\-\\hat\{a\}\_\{i\}\\sum\_\{j\}b\_\{j\}a\_\{j\}^\{\\frac\{2L\}\{L\+1\}\}\\Big\]\.In this context, DNC corresponds toa^∝1K−1\\hat\{a\}\\propto 1\_\{K\-1\}, sinceΦ​diag​\(0,1,…,1\)​ΦT=S\\Phi\\textrm\{diag\}\(0,1,\.\.\.,1\)\\Phi^\{T\}=S, where we recall thata0=0a\_\{0\}=0\. Moreover, we say a directiona^∗\\hat\{a\}^\{\*\}is*stable*if the Euclidean distance froma^\\hat\{a\}toa^∗\\hat\{a\}^\{\*\}decreases when sufficiently small\.

### 4\.2Instability of Neural Collapse

ForL=1L=1,Garrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)proved that the KL divergenceDKL​\(1K−1​1K−1∥a^\)D\_\{\\mathrm\{KL\}\}\(\\frac\{1\}\{K\-1\}1\_\{K\-1\}\\,\\\|\\,\\hat\{a\}\)decreases monotonically, serving as a Lyapunov function that forces all strictly positive initializations to converge to NC\. Since this KL distance diverges at any low\-rank solution, the dynamics must consistently move away from such solutions\. They also showed NC is locally stable throughout the trajectory\. We prove \(see App\.[D\.4](https://arxiv.org/html/2605.23087#A4.SS4)\) that both properties fail forL\>1L\>1:

###### Theorem 4\.3\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}withm\>1m\>1\. Leta​\(t\)∈ℝK−1a\(t\)\\in\\mathbb\{R\}^\{K\-1\}evolve according to Eq\. \([6](https://arxiv.org/html/2605.23087#S4.E6)\) with initializationai​\(0\)\>0a\_\{i\}\(0\)\>0for allii, and letL∈ℕL\\in\\mathbb\{N\}\. Then:

1. \(i\)The directiona^=1K−1​1K−1\\hat\{a\}=\\frac\{1\}\{K\-1\}1\_\{K\-1\}is stable when‖a‖1\>L−1L\+1​\(K−1\)\\\|a\\\|\_\{1\}\>\\frac\{L\-1\}\{L\+1\}\(K\-1\), but unstable below this threshold\.
2. \(ii\)WhenL\>1L\>1, there exists alternative low\-rank structures that are also stable above a threshold in‖a‖1\\\|a\\\|\_\{1\}\.
3. \(iii\)ForL\>1L\>1, the KL\-divergenceDKL​\(1K−1​1K−1∥a^\)D\_\{\\mathrm\{KL\}\}\(\\frac\{1\}\{K\-1\}1\_\{K\-1\}\\,\\\|\\,\\hat\{a\}\)is no longer monotonically decreasing, and there exists gradient\-flow trajectories along which it diverges\.

ForL=1L=1the NC direction is stable for all‖a‖1\\\|a\\\|\_\{1\}, but forL\>1L\>1it is unstable near the origin, with the unstable region growing monotonically inLL\. While DNC becomes stable at larger scales, other stable directions, corresponding to low\-rank structures, coexist\. Thus, in the context of the model, depth induces a low\-rank bias not only in the asymptotic landscape \(Section[3](https://arxiv.org/html/2605.23087#S3)\) but also along finite\-time trajectories\. Crucially, forL\>1L\>1the KL divergence is not only non\-monotone but can diverge along certain GF paths\. This occurs when trajectories converge towards low\-rank solutions, confirming that DNC is not guaranteed as an outcome of training\.

### 4\.3The “Rich\-get\-Richer” in the Linearized Regime

![Refer to caption](https://arxiv.org/html/2605.23087v1/Experiments/Real_nets/real_net_depth_2.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/Experiments/Real_nets/real_net_width.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/Experiments/Real_nets/real_net_generalize_2.png)

Figure 6:Confirmation of theory on the MNIST and CIFAR\-10 datasets using the ResNet\-20 architecture with a ReLU head\. We report the average effective rank of the mean logit matrix at the end of training and one\-standard\-deviation error bars over five runs\.Left:Here the ReLU head has widthd=50d=50and a variety of depthsLL\.Middle:Here the ReLU head has depthL=3L=3for a variety of widths\.Right:Scatter plot of the generalization performance against the effective rank of the mean logit matrix across the forty five MNIST models of the previous two plots\. We also show the line of best fit to show the general trend\.To understand*why*DNC is unstable near the origin forL\>1L\>1, we analyze the small\-norm regime‖a‖1≪1\\\|a\\\|\_\{1\}\\ll 1\.

###### Theorem 4\.4\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}\. Leta​\(t\)∈ℝK−1a\(t\)\\in\\mathbb\{R\}^\{K\-1\}evolve according to Eq\. \([6](https://arxiv.org/html/2605.23087#S4.E6)\) with initializationai​\(0\)\>0a\_\{i\}\(0\)\>0for alli=1,…,K−1i=1,\.\.\.,K\-1, and letL∈ℕL\\in\\mathbb\{N\}\. To linear order in the scale‖a‖1\\\|a\\\|\_\{1\}, the dynamics reduce to:

d​aid​t=ai2​LL\+1​\(1−ai\)\+O​\(‖a‖12\+2​LL\+1\)\.\\frac\{da\_\{i\}\}\{dt\}=a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}\(1\-a\_\{i\}\)\+O\\big\(\\\|a\\\|\_\{1\}^\{2\+\\frac\{2L\}\{L\+1\}\}\\big\)\.As a consequence, when‖a‖1\\\|a\\\|\_\{1\}is small:

- •IfL=1L=1, thendd​t​\(a^ia^j\)\\frac\{d\}\{dt\}\\left\(\\frac\{\\hat\{a\}\_\{i\}\}\{\\hat\{a\}\_\{j\}\}\\right\)has the opposite sign toa^i−a^j\\hat\{a\}\_\{i\}\-\\hat\{a\}\_\{j\}\. Consequently, unequal modes are driven toward each other, and DNC is the only stable direction under the linearized dynamics\.
- •IfL\>1L\>1, thendd​t​\(a^ia^j\)\\frac\{d\}\{dt\}\\left\(\\frac\{\\hat\{a\}\_\{i\}\}\{\\hat\{a\}\_\{j\}\}\\right\)has the same sign asa^i−a^j\\hat\{a\}\_\{i\}\-\\hat\{a\}\_\{j\}\. Consequently, DNC is a critical direction, but is not stable under the linearized dynamics\.

The proof appears in App\.[D\.5](https://arxiv.org/html/2605.23087#A4.SS5)\. The theorem identifies the mechanism behind DNC instability: A “rich\-get\-richer” effect that emerges forL\>1L\>1\. Near initialization, larger singular values grow relatively faster than smaller ones, amplifying initial disparities\. This promotes the development of approximate low\-rank structure in the logit matrix\. By the time the dynamics exit the linearized regime, the system is more likely to lie in the basin of attraction of a low\-rank direction than of DNC\. These small\-norm effects are particularly relevant for CE models due to the lazy dynamics that emerge as the norm grows\(Garrodet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib9)\)\.

Figure[4](https://arxiv.org/html/2605.23087#S4.F4)illustrates this mechanism\. Initially, singular values evolve under this “rich\-get\-richer” effect, and the logit matrix becomes low\-rank\. Aroundt≈5×101t\\approx 5\\times 10^\{1\}, the norm‖a‖1\\\|a\\\|\_\{1\}grows large enough that the linearized dynamics no longer apply, but by then the low\-rank structure has already taken hold, and the trajectory settles into a low\-rank optimum\.

### 4\.4Random Initialization and Large Width is Biased Towards Neural Collapse

The preceding sections established that depth alone—withoutL2L\_\{2\}regularization—suffices to make DNC suboptimal and to create stable low\-rank alternatives\. Yet empirically, DNC is observed ubiquitously in standard training, even with little or no regularization\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129); Parkeret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib140)\)\. Here we highlight how random initialization combines with large network width actually encourages higher rank outcomes\.

Garrod and Keating \([2024a](https://arxiv.org/html/2605.23087#bib.bib39)\)conjecture that DNC becomes more prevalent with increasing width, offering a heuristic explanation for the impact of width\. However, they do not study optimization trajectories or whether DNC is dynamically reachable from typical initializations\. Here we give a dynamical explanation:*standard random initialization, together with large width, biases early training toward DNC*, steering optimization away from regions of the loss surface where low\-rank optima are attractive\.

###### Theorem 4\.5\.

Consider the deep UFM described in Eq\. \([1](https://arxiv.org/html/2605.23087#S2.E1)\) with parameter matrices initialized asWl=ϵ​BlW\_\{l\}=\\epsilon B\_\{l\}forl=1,…,Ll=1,\.\.\.,L, andH1=ϵ​B0H\_\{1\}=\\epsilon B\_\{0\}, where eachBlB\_\{l\}has entries sampled i\.i\.d\. from a Gaussian distribution with mean0and variance1d\\frac\{1\}\{d\}\. Then, in the joint limitsϵ→0\\epsilon\\to 0andd→∞d\\to\\inftywe have the following convergence in probability

Z˙​\(0\)→ϵ2​L​\(L\+1\)​S⊗1nT\.\\dot\{Z\}\(0\)\\rightarrow\\epsilon^\{2L\}\(L\+1\)S\\otimes 1\_\{n\}^\{T\}\.

The proof appears in App\.[D\.6](https://arxiv.org/html/2605.23087#A4.SS6)\. The theorem shows that under random initialization and large network width, the initial velocity of the logits is biased toward alignment with the simplex ETF—the signature of DNC\. This is due to a concentration of measure among the initial eigenvalues of the logit derivative\. Although the result is stated for smallϵ\\epsilon, the result holds whenever the softmax outputs are approximately uniform at initialization, which does not require vanishingly small logit norms\.

By continuity of GF, the velocity remains aligned withSSfor a nontrivial interval after initialization\. SinceZ​\(0\)Z\(0\)has small norm and balancedness is preserved \(keeping layers approximately aligned\), the singular values develop nearly equally, up toϵ\\epsilon\- anddd\-suppressed corrections\. Figure[5](https://arxiv.org/html/2605.23087#S4.F5)confirms this: increasing width strengthens the alignment of the initial logit derivative with the simplex ETF, and after brief training the logits inherit this alignment\. Correspondingly, the effective rank at convergence increases, indicating that initialization meaningfully shapes the final outcome\.

Note that this initialization bias directly counteracts the rich\-get\-richer effect from Section[4\.3](https://arxiv.org/html/2605.23087#S4.SS3)\. While that effect amplifies disparities among singular values, its strength depends on how non\-uniform they are at initialization\. Because DNC remains a critical point of the linearized dynamics \(Theorem[4\.4](https://arxiv.org/html/2605.23087#S4.Thmtheorem4)\), nearly equal singular values drift only weakly\. Thus, if initialization places the network sufficiently close to DNC, the linearized dynamics will not move it far before norm growth pushes the system out of this regime\. The network then enters the large\-norm regime with high effective rank, where DNC is locally stable \(Theorem[4\.3](https://arxiv.org/html/2605.23087#S4.Thmtheorem3)\) and hence can converge to DNC\. This gives a dynamical explanation for why width affects the empirical persistence of DNC under standard training, and highlights the role of concentration\-of\-measure at initialization\.

## 5Empirical Verification of Implicit Bias

We evaluate our theory on MNIST\(Lecunet al\.,[1998](https://arxiv.org/html/2605.23087#bib.bib18)\)and CIFAR\-10\(Krizhevsky,[2009](https://arxiv.org/html/2605.23087#bib.bib17)\)\. In each experiment, we train a ResNet\-20 with a fully connected ReLU head of varying depth and width \(see App\.[E](https://arxiv.org/html/2605.23087#A5)for full details\)\. The first panel of Figure[6](https://arxiv.org/html/2605.23087#S4.F6)shows that the effective rank of the mean logit matrix at convergence decreases with head depth, matching the deep UFM: depth induces low\-rank structure\. We also include theL=1L=1case, where all effective ranks are approximatelyK−1K\-1, which corresponds to NC\. The low\-rank bias effect is stronger on MNIST than on CIFAR\-10, suggesting dataset complexity influences convergence behavior\. The second panel shows that increasing width generally increases the effective rank on both datasets\.

The final panel connects generalization with effective rank at convergence\. In general, lower effective rank is associated with poorer generalization, consistent with the idea that lower\-rank solutions achieve smaller margins and therefore generalize less well\. That said, the relationship is not absolute: randomly initialized networks can exhibit high effective rank while still generalizing poorly, whereas NC solutions have low rank yet generalize best among the solutions we observe\. The key conclusion is that generalization deteriorates when optimization favors low\-rank structure at the expense of the margins the network can achieve\.

Overall, these results suggest our theory extends beyond the modeling setting to more general deep classification networks\.

## 6Conclusion

We have shown that depth alone induces an implicit low\-rank bias in deep UFMs trained with CE loss, producing alternative low\-rank structures, which we partially characterize\. Our analysis provides the first rigorous account of implicit bias in these models\. This analysis opens several directions\. Our infinite\-depth theorem and empirics establish correspondence between low\-rank optima and softmax codes; extending the theory to finite depth remains open\. The low\-rank alternatives also provide a natural control group for studying NC’s role in generalization—unlike prior comparisons against less\-trained models\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129)\), they offer a cleaner baseline \(Figure[6](https://arxiv.org/html/2605.23087#S4.F6)provides preliminary evidence\)\. Finally, the absence of these alternatives inPapyanet al\.\([2020](https://arxiv.org/html/2605.23087#bib.bib129)\)’s experiments—most of which use residual connections, with VGG \(closest to our setup\) showing the weakest NC convergence—suggests skip connections may play a key role in NC’s prevalence\. Clarifying this mechanism is a promising direction for future work\.

#### Limitations:

Our work studies deep neural networks through a principled abstraction of the overparameterized limit\. Although this model is empirically well supported in that regime \(see App\. A ofGarrod and Keating \([2024b](https://arxiv.org/html/2605.23087#bib.bib141)\)\), its applicability may weaken in underparameterized or highly constrained settings\. Moreover, the UFM is primarily a model of optimization and therefore does not directly address generalization\. We formulate and empirically validate hypotheses about generalization via the correlated notion of dataset margin, but a formal link would require additional modeling\. Our real\-network experiments further indicate that dataset complexity affects the observed phenomena, and other practical factors—such as batch size, learning rate, and initialization scale—may also influence how the theory manifests in practice\. Finally, Theorems[3\.2](https://arxiv.org/html/2605.23087#S3.Thmtheorem2)and[4\.5](https://arxiv.org/html/2605.23087#S4.Thmtheorem5)are asymptotic; obtaining rates or finite analogues is an important direction for future work\.

Our finite\-time training dynamics require Hadamard initialization, which is substantially more structured than random initialization and requiresK=2mK=2^\{m\}for somem∈ℕm\\in\\mathbb\{N\}\. We provide evidence in App\.[E\.4](https://arxiv.org/html/2605.23087#A5.SS4)that the same phenomenology persists under small random initialization, even for generalKK\. Since Hadamard\-initialized paths are genuine trajectories of the loss surface, they remain informative about stability\. Nevertheless, formally extending these arguments to random initialization is an important open direction\.

## Acknowledgments

CG is supported by the Charles Coulson Scholarship\. The authors also acknowledge support from His Majesty’s Government in the development of this research\. CT acknowledges support by the NSERC Discovery Grant No\. 2021\-03677, the Alliance Grant ALLRP 581098\-22 and a gift from Google\. For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript \(AAM\) version arising from this submission\.

## Impact Statement

This paper presents work whose goal is to advance the field of machine learning, and more specifically, the impact of implicit biases on the representations of deep neural networks\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- G\. Andriopoulos, Z\. Dong, L\. Guo, Z\. Zhao, and K\. Ross \(2024\)The prevalence of neural collapse in neural multivariate regression\.arXiv preprint arXiv:2409\.04180\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1)\.
- S\. Arora, N\. Cohen, N\. Golowich, and W\. Hu \(2018a\)A convergence analysis of gradient descent for deep linear neural networks\.arXiv preprint arXiv:1810\.02281\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- S\. Arora, N\. Cohen, and E\. Hazan \(2018b\)On the optimization of deep networks: implicit acceleration by overparameterization\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 244–253\.External Links:[Link](https://proceedings.mlr.press/v80/arora18a.html)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- S\. Arora, N\. Cohen, W\. Hu, and Y\. Luo \(2019\)Implicit regularization in deep matrix factorization\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\.\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- N\. Azizan and B\. Hassibi \(2018\)Stochastic gradient/mirror descent: minimax optimality and implicit regularization\.arXiv preprint arXiv:1806\.00952\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- B\. Bah, H\. Rauhut, U\. Terstiege, and M\. Westdickenberg \(2022\)Learning deep linear neural networks: riemannian gradient flows and convergence to global minimizers\.Information and Inference: A Journal of the IMA11\(1\),pp\. 307–353\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- D\. Beaglehole, P\. Súkeník, M\. Mondelli, and M\. Belkin \(2024\)Average gradient outer product as a mechanism for deep neural collapse\.arXiv preprint arXiv:2402\.13728\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1)\.
- T\. Behnia, G\. R\. Kini, V\. Vakilian, and C\. Thrampoulidis \(2023\)On the implicit geometry of cross\-entropy parameterizations for label\-imbalanced data\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 10815–10838\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1)\.
- L\. Braun, C\. Dominé, J\. Fitzgerald, and A\. Saxe \(2022\)Exact learning dynamics of deep linear networks with prior knowledge\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 6615–6629\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/2b3bb2c95195130977a51b3bb251c40a-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1)\.
- Y\. Cai, K\. Zhou, J\. Wu, S\. Mei, M\. Lindsey, and P\. L\. Bartlett \(2025\)Implicit bias of gradient descent for non\-homogeneous deep networks\.arXiv preprint arXiv:2502\.16075\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- H\. Chou, C\. Gieshoff, J\. Maly, and H\. Rauhut \(2020\)Gradient descent for deep matrix factorization: dynamics and implicit bias towards low rank\.ArXivabs/2011\.13772\.External Links:[Link](https://api.semanticscholar.org/CorpusID:227209459)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- H\. Dang, T\. M\. Nguyen, T\. Tran, H\. T\. Tran, H\. Tran, and N\. Ho \(2023\)Neural collapse in deep linear networks: from balanced to imbalanced data\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:255372822)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.
- C\. C\. J\. Domin’e, N\. Anguita, A\. M\. Proca, L\. Braun, D\. Kunin, P\. A\. M\. Mediano, and A\. Saxe \(2024\)From lazy to rich: exact learning dynamics in deep linear networks\.ArXivabs/2409\.14623\.External Links:[Link](https://api.semanticscholar.org/CorpusID:272826705)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1)\.
- M\. Even, S\. Pesme, S\. Gunasekar, and N\. Flammarion \(2023\)\(S\) gd over diagonal linear networks: implicit bias, large stepsizes and edge of stability\.Advances in Neural Information Processing Systems36,pp\. 29406–29448\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- C\. Fan, M\. Schmidt, and C\. Thrampoulidis \(2025\)Implicit bias of spectral descent and muon on multiclass separable data\.arXiv preprint arXiv:2502\.04664\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- C\. Fang, H\. He, Q\. Long, and W\. J\. Su \(2021\)Exploring deep neural networks via layer\-peeled model: minority collapse in imbalanced training\.Proceedings of the National Academy of Sciences118\(43\),pp\. e2103091118\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3),[§2](https://arxiv.org/html/2605.23087#S2.p2.2)\.
- T\. Galanti, A\. György, and M\. Hutter \(2021\)On the role of neural collapse in transfer learning\.arXiv preprint arXiv:2112\.15121\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- P\. Gao, Q\. Xu, Y\. Yang, P\. Wen, H\. Shao, Z\. Yang, B\. Ghanem, and Q\. Huang \(2023\)Towards demystifying the generalization behaviors when neural collapse emerges\.ArXivabs/2310\.08358\.External Links:[Link](https://api.semanticscholar.org/CorpusID:263909223)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- C\. Garrod, J\. P\. Keating, and C\. Thrampoulidis \(2025\)Diagonalizing the softmax: hadamard initialization for tractable cross\-entropy dynamics\.External Links:[Link](https://api.semanticscholar.org/CorpusID:283466995)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p5.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1),[Lemma D\.4](https://arxiv.org/html/2605.23087#A4.Thmtheorem4.3.3),[§E\.3](https://arxiv.org/html/2605.23087#A5.SS3.p4.2),[§E\.4](https://arxiv.org/html/2605.23087#A5.SS4.p2.1),[§1\.1](https://arxiv.org/html/2605.23087#S1.SS1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p3.1),[§1](https://arxiv.org/html/2605.23087#S1.p4.1),[§2](https://arxiv.org/html/2605.23087#S2.p5.2),[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p3.1),[§4\.1](https://arxiv.org/html/2605.23087#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.23087#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.23087#S4.SS1.p3.3),[§4\.2](https://arxiv.org/html/2605.23087#S4.SS2.p1.3),[§4\.3](https://arxiv.org/html/2605.23087#S4.SS3.p2.1),[§4](https://arxiv.org/html/2605.23087#S4.p2.2)\.
- C\. Garrod and J\. P\. Keating \(2024a\)The persistence of neural collapse despite low\-rank bias: an analytic perspective through unconstrained features\.ArXivabs/2410\.23169\.External Links:[Link](https://api.semanticscholar.org/CorpusID:273695529)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[Appendix B](https://arxiv.org/html/2605.23087#A2.p5.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3),[§1](https://arxiv.org/html/2605.23087#S1.p3.2),[§2](https://arxiv.org/html/2605.23087#S2.p3.2),[§3\.1](https://arxiv.org/html/2605.23087#S3.SS1.p3.1),[§4\.4](https://arxiv.org/html/2605.23087#S4.SS4.p2.1)\.
- C\. Garrod and J\. P\. Keating \(2024b\)Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model\.arXiv preprint arXiv:2404\.06106\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§2](https://arxiv.org/html/2605.23087#S2.p2.2),[§6](https://arxiv.org/html/2605.23087#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Gidel, F\. Bach, and S\. Lacoste\-Julien \(2019\)Implicit regularization of discrete gradient dynamics in linear neural networks\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/f39ae9ff3a81f499230c4126e01f421b-Paper.pdf)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1),[§E\.4](https://arxiv.org/html/2605.23087#A5.SS4.p2.1),[§4\.1](https://arxiv.org/html/2605.23087#S4.SS1.p2.1)\.
- S\. Gunasekar, J\. D\. Lee, D\. Soudry, and N\. Srebro \(2018a\)Implicit bias of gradient descent on linear convolutional networks\.InNeural Information Processing Systems,External Links:[Link](https://api.semanticscholar.org/CorpusID:44099388)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- S\. Gunasekar, J\. Lee, D\. Soudry, and N\. Srebro \(2018b\)Characterizing implicit bias in terms of optimization geometry\.InInternational Conference on Machine Learning,pp\. 1832–1841\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- S\. Gunasekar, B\. E\. Woodworth, S\. Bhojanapalli, B\. Neyshabur, and N\. Srebro \(2017\)Implicit regularization in matrix factorization\.2018 Information Theory and Applications Workshop \(ITA\),pp\. 1–10\.External Links:[Link](https://api.semanticscholar.org/CorpusID:3909231)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- X\.Y\. Han, V\. Papyan, and D\. L\. Donoho \(2022\)Neural collapse under MSE loss: proximity to and dynamics on the central path\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=w1UbdvWH_R3)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3),[§2](https://arxiv.org/html/2605.23087#S2.p2.2)\.
- H\. He and W\. Su \(2022\)A law of data separation in deep learning\.Proceedings of the National Academy of Sciences of the United States of America120\.External Links:[Link](https://api.semanticscholar.org/CorpusID:253237619)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- W\. Hong and S\. Ling \(2023\)Neural collapse for unconstrained feature model under cross\-entropy loss with imbalanced data\.arXiv preprint arXiv:2309\.09725\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1)\.
- W\. Hong and S\. Ling \(2024\)Beyond unconstrained features: neural collapse for shallow neural networks with general data\.arXiv preprint arXiv:2409\.01832\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- K\. J\. Horadam \(2007\)Hadamard matrices and their applications\.Princeton University Press\.External Links:ISBN 9780691119212,[Link](http://www.jstor.org/stable/j.ctt7t6pw)Cited by:[Lemma D\.5](https://arxiv.org/html/2605.23087#A4.Thmtheorem5.3.3)\.
- M\. Huh, H\. Mobahi, R\. Zhang, B\. Cheung, P\. Agrawal, and P\. Isola \(2021\)The low\-rank simplicity bias in deep networks\.Trans\. Mach\. Learn\. Res\.2023\.External Links:[Link](https://api.semanticscholar.org/CorpusID:232269962)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- L\. Hui, M\. Belkin, and P\. Nakkiran \(2022\)Limitations of neural collapse for understanding generalization in deep learning\.arXiv preprint arXiv:2202\.08384\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- A\. Jacot, P\. Súkeník, Z\. Wang, and M\. Mondelli \(2025\)Wide neural networks trained with weight decay provably exhibit neural collapse\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 84780–84806\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/d33d31a87eb276ffe47fa7324648098b-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- A\. Jacot \(2023\)Implicit bias of large depth networks: a notion of rank for nonlinear functions\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6iDHce-0B-a)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- W\. Ji, Y\. Lu, Y\. Zhang, Z\. Deng, and W\. J\. Su \(2022\)An unconstrained layer\-peeled perspective on neural collapse\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WZ3yjh8coDg)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p5.1),[§D\.7](https://arxiv.org/html/2605.23087#A4.SS7.1.p1.1),[§2](https://arxiv.org/html/2605.23087#S2.p5.2),[§3\.1](https://arxiv.org/html/2605.23087#S3.SS1.p1.5)\.
- Z\. Ji, M\. Dudík, R\. E\. Schapire, and M\. Telgarsky \(2020\)Gradient descent follows the regularization path for general losses\.InProceedings of Thirty Third Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.125,pp\. 2109–2136\.External Links:[Link](https://proceedings.mlr.press/v125/ji20a.html)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- Z\. Ji and M\. Telgarsky \(2019\)Gradient descent aligns the layers of deep linear networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HJflg30qKX)Cited by:[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p3.1),[§2](https://arxiv.org/html/2605.23087#S2.p4.1)\.
- Z\. Ji and M\. Telgarsky \(2020\)Directional convergence and alignment in deep learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 17176–17186\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/c76e4b2fa54f8506719a5c0dc14c2eb9-Paper.pdf)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- J\. Jiang, J\. Zhou, P\. Wang, Q\. Qu, D\. Mixon, C\. You, and Z\. Zhu \(2023\)Generalized neural collapse for a large number of classes\.arXiv preprint arXiv:2310\.05351\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1\.1](https://arxiv.org/html/2605.23087#S1.SS1.p2.1),[Figure 3](https://arxiv.org/html/2605.23087#S3.F3),[Figure 3](https://arxiv.org/html/2605.23087#S3.F3.29.13),[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p3.1),[§3\.3](https://arxiv.org/html/2605.23087#S3.SS3.p3.8)\.
- A\. K\. Julistiono, D\. A\. Tarzanagh, and N\. Azizan \(2024\)Optimizing attention with mirror descent: generalized max\-margin token selection\.arXiv preprint arXiv:2410\.14581\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- V\. Kothapalli, E\. Rasromani, and V\. Awatramani \(2022\)Neural collapse: a review on modelling principles and generalization\.Trans\. Mach\. Learn\. Res\.2023\.External Links:[Link](https://api.semanticscholar.org/CorpusID:249461767)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- V\. Kothapalli and T\. Tirer \(2024\)Can kernel methods explain how the data affects neural collapse?\.Trans\. Mach\. Learn\. Res\.2025\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270226071)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- A\. Krizhevsky \(2009\)Learning multiple layers of features from tiny images\.Technical report, Toronto University,pp\.\.Cited by:[§5](https://arxiv.org/html/2605.23087#S5.p1.2)\.
- D\. Kunin, A\. Ravent’os, C\. C\. J\. Domin’e, F\. Chen, D\. Klindt, A\. Saxe, and S\. Ganguli \(2024\)Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning\.ArXivabs/2406\.06158\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270371640)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1)\.
- H\. Lawrence, K\. Georgiev, A\. Dienes, and B\. T\. Kiani \(2021\)Implicit bias of linear equivariant networks\.arXiv preprint arXiv:2110\.06084\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- Y\. Lecun, L\. Bottou, Y\. Bengio, and P\. Haffner \(1998\)Gradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.External Links:[Document](https://dx.doi.org/10.1109/5.726791)Cited by:[§5](https://arxiv.org/html/2605.23087#S5.p1.2)\.
- J\. Li and V\. Papyan \(2024\)Residual alignment: uncovering the mechanisms of residual networks\.Advances in Neural Information Processing Systems36\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- P\. Li, X\. Li, Y\. Wang, and Q\. Qu \(2023\)Neural collapse in multi\-label learning with pick\-all\-label loss\.arXiv preprint arXiv:2310\.15903\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.
- X\. Li, S\. Liu, J\. Zhou, X\. Lu, C\. Fernandez\-Granda, Z\. Zhu, and Q\. Qu \(2022\)Principled and efficient transfer learning of deep models via neural collapse\.arXiv preprint arXiv:2212\.12206\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- Z\. Li, Y\. Luo, and K\. Lyu \(2020\)Towards resolving the implicit bias of gradient descent for matrix factorization: greedy low\-rank learning\.arXiv preprint arXiv:2012\.09839\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- K\. Lyu and J\. Li \(2019\)Gradient descent maximizes the margin of homogeneous neural networks\.arXiv preprint arXiv:1906\.05890\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§D\.7](https://arxiv.org/html/2605.23087#A4.SS7.1.p1.1),[§1\.1](https://arxiv.org/html/2605.23087#S1.SS1.p2.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p3.1),[§3\.1](https://arxiv.org/html/2605.23087#S3.SS1.p1.3),[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p1.8),[§4](https://arxiv.org/html/2605.23087#S4.p1.2)\.
- C\. Ma, T\. Obuchi, and T\. Tanaka \(2025\)Neural collapse in cumulative link models for ordinal regression: an analysis with unconstrained feature model\.arXiv preprint arXiv:2506\.05801\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1)\.
- N\. Mainali and L\. Teixeira \(2025\)Exact learning dynamics of in\-context learning in linear transformers and its application to non\-linear transformers\.ArXivabs/2504\.12916\.External Links:[Link](https://api.semanticscholar.org/CorpusID:277857272)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1)\.
- D\. G\. Mixon, H\. Parshall, and J\. Pi \(2020\)Neural collapse with unconstrained features\.Sampling Theory, Signal Processing, and Data Analysis20\.External Links:[Link](https://api.semanticscholar.org/CorpusID:227127383)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3),[§2](https://arxiv.org/html/2605.23087#S2.p2.2)\.
- R\. O’Donnell \(2014\)Analysis of boolean functions\.Cambridge University Press\.Cited by:[§D\.7](https://arxiv.org/html/2605.23087#A4.SS7.7.p1.1),[Lemma D\.6](https://arxiv.org/html/2605.23087#A4.Thmtheorem6.3.3)\.
- V\. Papyan, X\. Han, and D\. L\. Donoho \(2020\)Prevalence of neural collapse during the terminal phase of deep learning training\.Proceedings of the National Academy of Sciences of the United States of America117,pp\. 24652 – 24663\.External Links:[Link](https://api.semanticscholar.org/CorpusID:221172897)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[Appendix B](https://arxiv.org/html/2605.23087#A2.p4.6),[§E\.1](https://arxiv.org/html/2605.23087#A5.SS1.p11.3),[§1](https://arxiv.org/html/2605.23087#S1.p1.1),[§2](https://arxiv.org/html/2605.23087#S2.p3.2),[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p3.1),[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p4.3),[§4\.4](https://arxiv.org/html/2605.23087#S4.SS4.p1.1),[§6](https://arxiv.org/html/2605.23087#S6.p1.1)\.
- L\. Parker, E\. Onal, A\. Stengel, and J\. Intrater \(2023\)Neural collapse in the intermediate hidden layers of classification neural networks\.ArXivabs/2308\.02760\.External Links:[Link](https://api.semanticscholar.org/CorpusID:260682642)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[Appendix B](https://arxiv.org/html/2605.23087#A2.p4.6),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§2](https://arxiv.org/html/2605.23087#S2.p3.2),[§4\.4](https://arxiv.org/html/2605.23087#S4.SS4.p1.1)\.
- S\. Pesme, R\. Dragomir, and N\. Flammarion \(2024\)Implicit bias of mirror flow on separable data\.Advances in Neural Information Processing Systems37,pp\. 114094–114121\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- A\. Rangamani, M\. Lindegaard, T\. Galanti, and T\. A\. Poggio \(2023\)Feature learning in deep classifiers through intermediate neural collapse\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 28729–28745\.External Links:[Link](https://proceedings.mlr.press/v202/rangamani23a.html)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- N\. Razin and N\. Cohen \(2020\)Implicit regularization in deep learning may not be explainable by norms\.Advances in neural information processing systems33,pp\. 21174–21187\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli \(2013\)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.CoRRabs/1312\.6120\.External Links:[Link](https://api.semanticscholar.org/CorpusID:17272965)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1),[§E\.4](https://arxiv.org/html/2605.23087#A5.SS4.p2.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p3.1),[§4\.1](https://arxiv.org/html/2605.23087#S4.SS1.p2.1)\.
- A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli \(2019\)A mathematical theory of semantic development in deep neural networks\.Proceedings of the National Academy of Sciences116\(23\),pp\. 11537–11546\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1820226116),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.1820226116),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.1820226116Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1),[§E\.4](https://arxiv.org/html/2605.23087#A5.SS4.p2.1),[§4\.1](https://arxiv.org/html/2605.23087#S4.SS1.p2.1)\.
- M\. Seleznova, D\. Weitzner, R\. Giryes, G\. Kutyniok, and H\. Chou \(2023\)Neural \(tangent kernel\) collapse\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 16240–16270\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- F\. Shang, Y\. Liu, F\. Shang, H\. Liu, L\. Kong, and L\. Jiao \(2020\)A unified scalable equivalent formulation for schatten quasi\-norms\.Mathematics8\(8\),pp\. 1325\.Cited by:[Lemma D\.7](https://arxiv.org/html/2605.23087#A4.Thmtheorem7.3.3)\.
- D\. Soudry, E\. Hoffer, M\. S\. Nacson, S\. Gunasekar, and N\. Srebro \(2018\)The implicit bias of gradient descent on separable data\.Journal of Machine Learning Research19\(70\),pp\. 1–57\.External Links:[Link](http://jmlr.org/papers/v19/18-188.html)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- J\. Su, Y\. S\. Zhang, N\. Tsilivis, and J\. Kempe \(2023\)On the robustness of neural collapse and the neural collapse of robustness\.arXiv preprint arXiv:2311\.07444\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- P\. Súkeník, C\. H\. Lampert, and M\. Mondelli \(2025\)Neural collapse is globally optimal in deep regularized resnets and transformers\.ArXivabs/2505\.15239\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278783174)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- P\. Súkeník, M\. Mondelli, and C\. H\. Lampert \(2023\)Deep neural collapse is provably optimal for the deep unconstrained features model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 52991–53024\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a60c43ba078b723d3d517d28c50ded4c-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- P\. Súkeník, M\. Mondelli, and C\. Lampert \(2024\)Neural collapse versus low\-rank bias: is deep neural collapse really optimal?\.arXiv preprint arXiv:2405\.14468\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3),[§1](https://arxiv.org/html/2605.23087#S1.p3.2),[§3\.1](https://arxiv.org/html/2605.23087#S3.SS1.p3.1)\.
- H\. Sun, K\. Ahn, C\. Thrampoulidis, and N\. Azizan \(2022\)Mirror descent maximizes generalized margin and can be implemented efficiently\.Advances in Neural Information Processing Systems35,pp\. 31089–31101\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- S\. Tarmoun, G\. Franca, B\. D\. Haeffele, and R\. Vidal \(2021\)Understanding the dynamics of gradient flow in overparameterized linear models\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 10153–10161\.External Links:[Link](https://proceedings.mlr.press/v139/tarmoun21a.html)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1)\.
- D\. A\. Tarzanagh, Y\. Li, C\. Thrampoulidis, and S\. Oymak \(2023\)Transformers as support vector machines\.arXiv preprint arXiv:2308\.16898\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- C\. Thrampoulidis, G\. R\. Kini, V\. Vakilian, and T\. Behnia \(2022\)Imbalance trouble: revisiting neural\-collapse geometry\.Advances in Neural Information Processing Systems35,pp\. 27225–27238\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p5.1),[§1](https://arxiv.org/html/2605.23087#S1.p3.2),[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p3.1)\.
- N\. Timor, G\. Vardi, and O\. Shamir \(2023\)Implicit regularization towards rank minimization in relu networks\.InInternational Conference on Algorithmic Learning Theory,pp\. 1429–1459\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[Appendix A](https://arxiv.org/html/2605.23087#A1.p6.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p4.1)\.
- T\. Tirer and J\. Bruna \(2022\)Extended unconstrained features model for exploring deep neural collapse\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:246867384)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p3.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.
- T\. Tirer, H\. Huang, and J\. Niles\-Weed \(2023\)Perturbation analysis of neural collapse\.InInternational Conference on Machine Learning,pp\. 34301–34329\.Cited by:[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.
- Z\. Tu, S\. Aranguri, and A\. Jacot \(2024\)Mixed dynamics in linear networks: unifying the lazy and active regimes\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 106059–106104\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- G\. Vardi, O\. Shamir, and N\. Srebro \(2022\)On margin maximization in linear and relu networks\.Advances in Neural Information Processing Systems35,pp\. 37024–37036\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- B\. Vasudeva, P\. Deora, and C\. Thrampoulidis \(2024\)Implicit bias and fast convergence rates for self\-attention\.Transactions on Machine Learning Research\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- S\. Wang, K\. Gai, and S\. Zhang \(2024\)Progressive feedforward collapse of resnet training\.arXiv preprint arXiv:2405\.00985\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- D\. Wu and M\. Mondelli \(2025\)Neural collapse beyond the unconstrained features model: landscape, dynamics, and generalization in the mean\-field regime\.arXiv preprint arXiv:2501\.19104\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- J\. Wu, V\. Braverman, and J\. D\. Lee \(2023\)Implicit bias of gradient descent for logistic regression at the edge of stability\.Advances in Neural Information Processing Systems36,pp\. 74229–74256\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- R\. Wu and V\. Papyan \(2024\)Linguistic collapse: neural collapse in \(large\) language models\.arXiv preprint arXiv:2405\.17767\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- Y\. Wu, R\. Yu, X\. Cheng, Z\. He, and X\. Huang \(2024\)Pursuing feature separation based on neural collapse for out\-of\-distribution detection\.arXiv preprint arXiv:2405\.17816\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- C\. Yaras, P\. Wang, W\. Hu, Z\. Zhu, L\. Balzano, and Q\. Qu \(2023\)The law of parsimony in gradient descent for learning deep linear networks\.ArXivabs/2306\.01154\.External Links:[Link](https://api.semanticscholar.org/CorpusID:262766856)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1),[§1\.2](https://arxiv.org/html/2605.23087#S1.SS2.p2.1)\.
- E\. Zangrando, P\. Deidda, S\. Brugiapaglia, N\. Guglielmi, and F\. Tudisco \(2024\)Provable emergence of deep neural collapse and low\-rank bias in l2\-regularized nonlinear networks\.arXiv preprint arXiv:2402\.03991\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- C\. Zhang, D\. Zou, and Y\. Cao \(2024a\)The implicit bias of adam on separable data\.Advances in Neural Information Processing Systems37,pp\. 23988–24021\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p4.1)\.
- C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals \(2017\)Understanding deep learning requires rethinking generalization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Sy8gdB9xx)Cited by:[§3\.2](https://arxiv.org/html/2605.23087#S3.SS2.p4.3)\.
- J\. Zhang, Y\. Chen, C\. Jin, L\. Zhu, and Y\. Gu \(2024b\)EPA: neural collapse inspired robust out\-of\-distribution detector\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 6515–6519\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- Y\. Zhao, T\. Behnia, V\. Vakilian, and C\. Thrampoulidis \(2024\)Implicit geometry of next\-token prediction: from language sparsity patterns to model representations\.ArXivabs/2408\.15417\.External Links:[Link](https://api.semanticscholar.org/CorpusID:271974203)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.
- Y\. Zhao and C\. Thrampoulidis \(2025a\)Geometry of semantics in next\-token prediction: how optimization implicitly organizes linguistic representations\.arXiv preprint arXiv:2505\.08348\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p7.1)\.
- Y\. Zhao and C\. Thrampoulidis \(2025b\)On the geometry of semantics in next\-token prediction\.ArXivabs/2505\.08348\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278534662)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p2.1)\.
- J\. Zhou, X\. Li, T\. Ding, C\. You, Q\. Qu, and Z\. Zhu \(2022a\)On the optimization landscape of neural collapse under MSE loss: global optimality with unconstrained features\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 27179–27202\.External Links:[Link](https://proceedings.mlr.press/v162/zhou22c.html)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.
- J\. Zhou, C\. You, X\. Li, K\. Liu, S\. Liu, Q\. Qu, and Z\. Zhu \(2022b\)Are all losses created equal: a neural collapse perspective\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 31697–31710\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/cdce17de141c9fba3bdf175a0b721941-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1)\.
- Z\. Zhu, T\. Ding, J\. Zhou, X\. Li, C\. You, J\. Sulam, and Q\. Qu \(2021\)A geometric analysis of neural collapse with unconstrained features\.Advances in Neural Information Processing Systems34,pp\. 29820–29834\.Cited by:[Appendix A](https://arxiv.org/html/2605.23087#A1.p1.1),[§1](https://arxiv.org/html/2605.23087#S1.p2.3)\.

## Appendix ADetailed Discussion on Related Work

Neural collapse:First introduced byPapyanet al\.\([2020](https://arxiv.org/html/2605.23087#bib.bib129)\), NC has evolved into a central object of study in the theory of deep classification models\. Much of the subsequent analysis has been conducted within the unconstrained feature model \(UFM\)\(Mixonet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib143); Fanget al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib125)\)with explicitL2L\_\{2\}regularization, where a broad range of loss functions has been examined\(Zhouet al\.,[2022b](https://arxiv.org/html/2605.23087#bib.bib147); Mixonet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib143); Jiet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib145); Hanet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib148); Maet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib5)\)\. Across these settings, NC consistently emerges as the globally optimal geometric configuration, with all other critical points being non\-degenerate saddles\(Zhouet al\.,[2022a](https://arxiv.org/html/2605.23087#bib.bib144); Zhuet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib42); Jiet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib145)\)\. Follow up research has since extended the original NC observation to imbalanced data\(Thrampoulidiset al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib122); Behniaet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib86); Fanget al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib125); Danget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib29); Hong and Ling,[2023](https://arxiv.org/html/2605.23087#bib.bib87)\)and to regimes with many classes\(Jianget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib121)\)\. Broader generalizations of the phenomenon have also been established in multilabel settings\(Liet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib85)\), soft\-label language models\(Zhaoet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib113)\), and multi\-class regression\(Andriopouloset al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib107)\)\. The relevance of the UFM for real neural networks has further been supported bySúkeníket al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib30)\), who show its equivalence for deep architectures with skip connections\.

Complementary perspectives on NC have emerged outside the UFM as well, including analyses in alternative theoretical frameworks\(Jacotet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib40); Zangrandoet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib64)\), as well as treatments via mean\-field methods\(Wu and Mondelli,[2025](https://arxiv.org/html/2605.23087#bib.bib33)\)and NTK\-based approaches\(Kothapalli and Tirer,[2024](https://arxiv.org/html/2605.23087#bib.bib114); Seleznovaet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib6)\)\. Recent works additionally highlight the role of dataset structure in shaping NC behavior\(Hong and Ling,[2024](https://arxiv.org/html/2605.23087#bib.bib106); Kothapalli and Tirer,[2024](https://arxiv.org/html/2605.23087#bib.bib114)\)\. Beyond theoretical significance, NC has been linked to a number of practical properties in deep learning, such as adversarial robustness\(Suet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib126)\), transfer learning\(Galantiet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib115); Liet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib44)\), generalization\(Huiet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib123); Gaoet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib90)\), and out\-of\-distribution detection\(Wuet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib116); Zhanget al\.,[2024b](https://arxiv.org/html/2605.23087#bib.bib45)\)\. Its dependence on architecture has been examined in the context of ResNets\(Li and Papyan,[2024](https://arxiv.org/html/2605.23087#bib.bib119); Súkeníket al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib30); Wanget al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib4)\)and large language models\(Wu and Papyan,[2024](https://arxiv.org/html/2605.23087#bib.bib120); Zhao and Thrampoulidis,[2025b](https://arxiv.org/html/2605.23087#bib.bib104)\)\. An overview of developments surrounding NC can be found in the survey byKothapalliet al\.\([2022](https://arxiv.org/html/2605.23087#bib.bib142)\)\.

DNC has also been explored empirically\(Parkeret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib140); Rangamaniet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib138); He and Su,[2022](https://arxiv.org/html/2605.23087#bib.bib136)\)\. Efforts to theoretically characterize DNC have relied predominantly on deep UFMs trained with MSE loss, including two\-layer constructions\(Tirer and Bruna,[2022](https://arxiv.org/html/2605.23087#bib.bib130)\), linear layers\(Danget al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib29); Garrod and Keating,[2024b](https://arxiv.org/html/2605.23087#bib.bib141)\), binary classification\(Súkeníket al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib131)\)and more general settings\(Súkeníket al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib28)\)\. In the CE setting,Garrod and Keating \([2024a](https://arxiv.org/html/2605.23087#bib.bib39)\)show thatL2L\_\{2\}regularization induces an explicit low\-rank bias that influences the emergence of DNC\. Additional perspectives outside the UFM—such as layer\-wise training schemes—have also been explored\(Beagleholeet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib124)\)\.

Implicit bias:A substantial literature investigates how gradient\-based optimization shapes model structure in the absence of explicit regularization\. Soudry et al\.\(Soudryet al\.,[2018](https://arxiv.org/html/2605.23087#bib.bib82)\)established that gradient descent converges in direction to the max\-margin classifier for logistic regression\. Researchers have also related the implicitly regularized path to explicitly regularized solutions in binary classification\(Jiet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib77)\)\. Subsequent studies have broadened our understanding of implicit bias to homogeneous\(Lyu and Li,[2019](https://arxiv.org/html/2605.23087#bib.bib108); Ji and Telgarsky,[2020](https://arxiv.org/html/2605.23087#bib.bib79); Vardiet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib53); Timoret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib54)\)and non\-homogeneous networks\(Caiet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib23)\), to alternative optimization algorithms\(Gunasekaret al\.,[2018b](https://arxiv.org/html/2605.23087#bib.bib75); Azizan and Hassibi,[2018](https://arxiv.org/html/2605.23087#bib.bib10); Sunet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib2); Pesmeet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib16); Zhanget al\.,[2024a](https://arxiv.org/html/2605.23087#bib.bib11); Fanet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib22)\), to large step\-size regimes\(Evenet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib15); Wuet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib13)\), and to specific architectural families such as convolutional networks\(Gunasekaret al\.,[2018a](https://arxiv.org/html/2605.23087#bib.bib74); Lawrenceet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib20)\)and transformers\(Tarzanaghet al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib12); Vasudevaet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib19); Julistionoet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib14)\)\. A parallel direction examines implicit regularization in linear networks trained with MSE loss, studying the interplay between gradient descent and depth\(Aroraet al\.,[2018b](https://arxiv.org/html/2605.23087#bib.bib38); Yaraset al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib100); Tuet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib83); Bahet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib98); Aroraet al\.,[2018a](https://arxiv.org/html/2605.23087#bib.bib61)\), including classical matrix factorization dynamics\(Liet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib112); Aroraet al\.,[2019](https://arxiv.org/html/2605.23087#bib.bib34); Razin and Cohen,[2020](https://arxiv.org/html/2605.23087#bib.bib99); Gunasekaret al\.,[2017](https://arxiv.org/html/2605.23087#bib.bib78)\)\.

Implicit bias has additionally served as a direct theoretical tool for studying NC within the unregularized CE UFM\. Ji et al\.\(Jiet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib145)\)established convergence to KKT points of a constrained norm\-minimization formulation\.Thrampoulidiset al\.\([2022](https://arxiv.org/html/2605.23087#bib.bib122)\)provided a characterization of the implicit solution path under class imbalance\.Garrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)proved convergence to NC under Hadamard\-style initializations\. Our work extends this line of inquiry by showing how depth\-induced implicit bias qualitatively alters the nature of convergence\.

Depth & low\-rank bias:Several lines of work have studied how network depth affects the rank of the solutions reached by optimization\. Much of this literature originates in matrix factorization\(Gunasekaret al\.,[2017](https://arxiv.org/html/2605.23087#bib.bib78); Aroraet al\.,[2019](https://arxiv.org/html/2605.23087#bib.bib34); Liet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib112); Chouet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib80)\), where low\-rank bias is induced in the absence of nonlinearities and is therefore conceptually distinct from the cross\-entropy classification setting studied here\. More recent work has explored the relationship between depth and rank in nonlinear architectures, both empirically\(Huhet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib71)\)and in homogeneous networks\(Jacot,[2023](https://arxiv.org/html/2605.23087#bib.bib81); Timoret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib54)\)\. In contrast, we analyze the representations that emerge in highly overparameterized classifiers, showing that depth induces a low\-rank bias that is reflected in smaller normalized margins and in the transition from neural collapse to softmax codes\.

Spectral initialization:Spectral initialization analyses aim to understand learning behavior by reducing gradient\-flow evolution to closed\-form differential equations governing the singular values of the weight matrices, usually for linear networks with MSE loss\. This began with work bySaxeet al\.\([2013](https://arxiv.org/html/2605.23087#bib.bib93),[2019](https://arxiv.org/html/2605.23087#bib.bib94)\), who established the foundational framework\. Subsequent research extended these tools to explore discrete\-time gradient descent\(Gidelet al\.,[2019](https://arxiv.org/html/2605.23087#bib.bib102)\), alternative initializations\(Kuninet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib101); Braunet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib95); Tarmounet al\.,[2021](https://arxiv.org/html/2605.23087#bib.bib76)\)and to consider lazy vs rich learning regimes\(Braunet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib95); Domin’eet al\.,[2024](https://arxiv.org/html/2605.23087#bib.bib96)\)\. More recent work has attempted to push these techniques beyond the purely linear setting\.Mainali and Teixeira \([2025](https://arxiv.org/html/2605.23087#bib.bib84)\)applied similar analyses to transformer architectures with linearized attention\.Zhao and Thrampoulidis \([2025a](https://arxiv.org/html/2605.23087#bib.bib27)\)used these tools in the UFM to study semantic structure in language data within the context of MSE loss\.Garrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)advanced this line by establishing spectral initialization for the UFM with CE loss for a single hidden layer via Hadamard initialization\. To date, however, no prior work has applied these techniques to deep networks trained with CE loss\.

## Appendix BFurther background on deep UFM and NC

Here we provide more details about NC and the UFM construction\. We study classification withKKclasses andnnsamples per class\. We denote data pointiiof classccbyxi​c∈ℝd0x\_\{ic\}\\in\\mathbb\{R\}^\{d\_\{0\}\}, with corresponding one\-hot encoded labelyc∈ℝKy\_\{c\}\\in\\mathbb\{R\}^\{K\}\. A deep neural networkf​\(x\):ℝd0→ℝKf\(x\):\\mathbb\{R\}^\{d\_\{0\}\}\\rightarrow\\mathbb\{R\}^\{K\}models the relationship between inputs and labels\. We decompose the network asfθ​\(x\)=WL​σ​\(WL−1​σ​\(…​σ​\(W1​hθ¯​\(x\)\)​…\)\),f\_\{\\theta\}\(x\)=W\_\{L\}\\sigma\(W\_\{L\-1\}\\sigma\(\.\.\.\\sigma\(W\_\{1\}h\_\{\\bar\{\\theta\}\}\(x\)\)\.\.\.\)\),whereWL∈ℝK×dW\_\{L\}\\in\\mathbb\{R\}^\{K\\times d\}andWL−1,…,W1∈ℝd×dW\_\{L\-1\},\\dots,W\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\}are weight matrices, andhθ¯:ℝd0→ℝdh\_\{\\bar\{\\theta\}\}:\\mathbb\{R\}^\{d\_\{0\}\}\\rightarrow\\mathbb\{R\}^\{d\}is a highly expressive feature map representing the remainder of the network\. The activation functionσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\rightarrow\\mathbb\{R\}is applied elementwise\. We denote the image of the data underhθ¯h\_\{\\bar\{\\theta\}\}byhθ¯​\(xi​c\)=hi​ch\_\{\\bar\{\\theta\}\}\(x\_\{ic\}\)=h\_\{ic\}, and define the feature matrixH1=\[h1,1,…,hn,1,h1,2,…,hn,K\]∈ℝd×K​nH\_\{1\}=\[h\_\{1,1\},\.\.\.,h\_\{n,1\},h\_\{1,2\},\.\.\.,h\_\{n,K\}\]\\in\\mathbb\{R\}^\{d\\times Kn\}as the collection of all features in class order\.

The parameters\{WL,…,W1,θ¯\}\\\{W\_\{L\},\.\.\.,W\_\{1\},\\bar\{\\theta\}\\\}are trained using a variant of gradient descent on CE loss without explicit regularization

ℒ​\(θ\)=∑i,cg​\(fθ​\(xi​c\),yc\),\\mathcal\{L\}\(\\theta\)=\\sum\_\{i,c\}g\(f\_\{\\theta\}\(x\_\{ic\}\),y\_\{c\}\),
whereggimplements the CE loss function\. The matrixH1H\_\{1\}denotes the features entering the first separated layer of the fully connected head\. Similarly, defineHlH\_\{l\}forl=2,…,Ll=2,\.\.\.,Lto be the features entering thelthl^\{\\textrm\{th\}\}layer of the fully connected head\. DNC then refers to the following observations, which are found to approximately hold in overparameterized neural networks as training continues\(Parkeret al\.,[2023](https://arxiv.org/html/2605.23087#bib.bib140)\)\. ForL=1L=1, this coincides with NC\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129)\)\.

###### Definition B\.1\(Deep Neural Collapse\)\.

The lastLLlayers obey DNC if, forl=1,…,Ll=1,\.\.\.,L, they satisfy:

- •DNC1: Features collapse to their means, meaning there is a matrixMl∈ℝK×KM\_\{l\}\\in\\mathbb\{R\}^\{K\\times K\}such thatHl=Ml⊗1nTH\_\{l\}=M\_\{l\}\\otimes 1\_\{n\}^\{T\}\.
- •DNC2: The feature mean matricesMlM\_\{l\}, after global centering, align with a simplex ETF, meaning\(Ml−μG\(l\)​1KT\)T​\(Ml−μG\(l\)​1KT\)∝S\(M\_\{l\}\-\\mu\_\{G\}^\{\(l\)\}1\_\{K\}^\{T\}\)^\{T\}\(M\_\{l\}\-\\mu\_\{G\}^\{\(l\)\}1\_\{K\}^\{T\}\)\\propto S, whereμG=1K​∑jMi​j\\mu\_\{G\}=\\frac\{1\}\{K\}\\sum\_\{j\}M\_\{ij\}\.
- •DNC3: The rows of the weight matricesWlW\_\{l\}are linear combinations of the columns of the matrixMl−μG\(l\)​1KTM\_\{l\}\-\\mu\_\{G\}^\{\(l\)\}1\_\{K\}^\{T\}\.

It is shown byGarrod and Keating \([2024a](https://arxiv.org/html/2605.23087#bib.bib39)\)that Definition[2\.1](https://arxiv.org/html/2605.23087#S2.Thmtheorem1)naturally implies all of these properties\.

### B\.1The Deep UFM Construction

To formally define the deep UFM, we approximate the feature maphθ¯​\(x\)h\_\{\\bar\{\\theta\}\}\(x\)as being capable of mapping the training data to arbitrary points in feature space\. Accordingly, we treat the feature vectorshi​ch\_\{ic\}as freely optimized variables and abstract away the parametersθ¯\\bar\{\\theta\}\. This is motivated by the fact thatθ¯\\bar\{\\theta\}interact with the loss only through the featureshi​ch\_\{ic\}; therefore, high performingθ¯\\bar\{\\theta\}should correspond to high performing feature vectors\. In particular, when the feature map is sufficiently expressive, it should be able to map the data close to the optimal features recovered by the deep UFM\. Consequently, the deep UFM is expected to describe deep neural networks in the overparametrization limit\.

The loss function is then parameterized byWL,…,W1W\_\{L\},\.\.\.,W\_\{1\}andH1H\_\{1\}, and becomes

ℒ=g​\(Z\)=−∑c=1K∑i=1nlog⁡\(exp⁡\(\(zi​c\)c\)∑c′=1Kexp⁡\(\(zi​c\)c′\)\),\\mathcal\{L\}=g\(Z\)=\-\\sum\_\{c=1\}^\{K\}\\sum\_\{i=1\}^\{n\}\\log\\left\(\\frac\{\\exp\(\(z\_\{ic\}\)\_\{c\}\)\}\{\\sum\_\{c^\{\\prime\}=1\}^\{K\}\\exp\(\(z\_\{ic\}\)\_\{c^\{\\prime\}\}\)\}\\right\),\(7\)wherezi​c=WL​σ​\(WL−1​σ​\(…​W2​σ​\(W1​hi​c\(1\)\)​…\)\)z\_\{ic\}=W\_\{L\}\\sigma\(W\_\{L\-1\}\\sigma\(\.\.\.W\_\{2\}\\sigma\(W\_\{1\}h\_\{ic\}^\{\(1\)\}\)\.\.\.\)\)are the logit vectors, which make up the columns of the logit matrixZ∈ℝK×K​nZ\\in\\mathbb\{R\}^\{K\\times Kn\}, using the same ordering as in the feature matrices\. In the main text we setσ=id\\sigma=\\textrm\{id\}throughout, and leave consideration of homogeneous networks to further work\.

## Appendix CFurther Results for Hadamard Initialization

Here we provide an additional result under the Hadamard initialization framework discussed in Section[4\.1](https://arxiv.org/html/2605.23087#S4.SS1)\.

### C\.1Minimal Rank Logit Matrices for Hadamard Initialization

We derive a rank lower bound for logit matrices in this Hadamard setting that are compatible with a continuously decreasing loss along gradient flow\. This hence informs us how low rank we can expect the logit matrix to become whilst still fitting the data in the Hadamard context\.

###### Proposition C\.1\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}, and letΦ\\Phibe the Sylvester\-Hadamard matrix of sizeKK\. The lowest rank achievable for a matrix of the formZ=Φ​diag​\(0,a\)​ΦT∈ℝK×KZ=\\Phi\\textrm\{diag\}\(0,a\)\\Phi^\{T\}\\in\\mathbb\{R\}^\{K\\times K\}satisfyingZc​c−Zc′​c\>0Z\_\{cc\}\-Z\_\{c^\{\\prime\}c\}\>0for allc′≠cc^\{\\prime\}\\neq c, wherea∈ℝ≥0K−1a\\in\\mathbb\{R\}^\{K\-1\}\_\{\\geq 0\}, ism=log2⁡\(K\)m=\\log\_\{2\}\(K\)\.

###### Proof\.

Denote theithi^\{\\textrm\{th\}\}column ofΦ\\Phibyϕi\\phi\_\{i\}, using zero indexingi=0,…,K−1i=0,\.\.\.,K\-1\. Let

S=\{u∈\{1,…,K−1\}:au≠0\},S=\\\{u\\in\\\{1,\.\.\.,K\-1\\\}:a\_\{u\}\\neq 0\\\},\(8\)so thatSSis the support ofaa, and note thatrank​\(Z\)=\|S\|\\textrm\{rank\}\(Z\)=\|S\|\. We can rewrite the entries ofZ=Φ​diag​\(0,a\)​ΦTZ=\\Phi\\textrm\{diag\}\(0,a\)\\Phi^\{T\}as

Zi​j=∑u,v=0K−1\(ϕu\)i​diag​\(0,a\)u​v​\(ϕv\)jZ\_\{ij\}=\\sum\_\{u,v=0\}^\{K\-1\}\(\\phi\_\{u\}\)\_\{i\}\\textrm\{diag\}\(0,a\)\_\{uv\}\(\\phi\_\{v\}\)\_\{j\}=∑u,v=1K−1\(ϕu\)i​au​δu​v​\(ϕv\)j=∑u=1K−1au​\(ϕu\)i​\(ϕu\)j\.=\\sum\_\{u,v=1\}^\{K\-1\}\(\\phi\_\{u\}\)\_\{i\}a\_\{u\}\\delta\_\{uv\}\(\\phi\_\{v\}\)\_\{j\}=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{u\}\)\_\{i\}\(\\phi\_\{u\}\)\_\{j\}\.Using thatΦ\\Phiis symmetric, this becomes

Zi​j=∑u=1K−1au​\(ϕi\)u​\(ϕj\)u=∑u=1K−1au​\(ϕi∘ϕj\)u,Z\_\{ij\}=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{i\}\)\_\{u\}\(\\phi\_\{j\}\)\_\{u\}=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{i\}\\circ\\phi\_\{j\}\)\_\{u\},where∘\\circdenotes the Hadamard \(entrywise\) product\. Using Lemma[D\.5](https://arxiv.org/html/2605.23087#A4.Thmtheorem5), the columns satisfyϕi∘ϕj=ϕi⊕j\\phi\_\{i\}\\circ\\phi\_\{j\}=\\phi\_\{i\\oplus j\}, where⊕\\oplusis the bitwise XOR\. Hence

Zi​j=∑u=1K−1au​\(ϕi⊕j\)u\.Z\_\{ij\}=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{i\\oplus j\}\)\_\{u\}\.In particular, wheni=ji=jwe haveZi​i=∑uauZ\_\{ii\}=\\sum\_\{u\}a\_\{u\}, while asjjranges overj≠ij\\neq i, the valuei⊕ji\\oplus jranges over\{1,…,K−1\}\\\{1,\.\.\.,K\-1\\\}, so the off\-diagonal entries in columniiare exactly

\{∑u=1K−1au​\(ϕk\)u:k=1,…,K−1\}\.\\left\\\{\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{k\}\)\_\{u\}:k=1,\.\.\.,K\-1\\right\\\}\.Therefore, the condition thatZc​c\>Zc′​cZ\_\{cc\}\>Z\_\{c^\{\\prime\}c\}for allc′≠cc^\{\\prime\}\\neq cis equivalent to

∑u=1Kau\>∑u=1K−1au​\(ϕk\)u,for all​k∈\{1,…,K−1\}\.\\sum\_\{u=1\}^\{K\}a\_\{u\}\>\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{k\}\)\_\{u\},\\quad\\textrm\{for all \}k\\in\\\{1,\.\.\.,K\-1\\\}\.Now identify the indices\{1,…,K−1\}\\\{1,\.\.\.,K\-1\\\}with elements of𝔽2m\\mathbb\{F\}\_\{2\}^\{m\}via binary strings, recalling thatK=2mK=2^\{m\}\. By Lemma[D\.6](https://arxiv.org/html/2605.23087#A4.Thmtheorem6), we haveΦi​j=\(−1\)i⋅j\\Phi\_\{ij\}=\(\-1\)^\{i\\cdot j\}, wherei⋅ji\\cdot jis the mod\-2 dot product\. Thus

∑u=1K−1au​\(ϕk\)u=∑u=1K−1au​\(−1\)k⋅u\.\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\\phi\_\{k\}\)\_\{u\}=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\-1\)^\{k\\cdot u\}\.We begin by proving the lower bound\|S\|≥m\|S\|\\geq mby contradiction\. Suppose\|S\|<m\|S\|<m\. ViewS⊂𝔽2mS\\subset\\mathbb\{F\}\_\{2\}^\{m\}and let

V=span𝔽2​\(S\)=\{∑j=1\|S\|λj​sj:sj∈S,λj∈𝔽2\}\.V=\\textrm\{span\}\_\{\\mathbb\{F\}\_\{2\}\}\(S\)=\\left\\\{\\sum\_\{j=1\}^\{\|S\|\}\\lambda\_\{j\}s\_\{j\}:s\_\{j\}\\in S,\\lambda\_\{j\}\\in\\mathbb\{F\}\_\{2\}\\right\\\}\.\(9\)SinceSShas fewer thanmmelements,\|V\|≤2\|S\|<2m\|V\|\\leq 2^\{\|S\|\}<2^\{m\}, and soV≠𝔽2mV\\neq\\mathbb\{F\}\_\{2\}^\{m\}\. consequentlydim​\(V\)≤m−1\\textrm\{dim\}\(V\)\\leq m\-1, and the orthogonal complementV⟂V^\{\\perp\}is non\-empty:

∃v∗≠0:v∗∈V⟂=\{x∈𝔽2m:x⋅v=0,∀v∈V\}\.\\exists v^\{\*\}\\neq 0:v^\{\*\}\\in V^\{\\perp\}=\\\{x\\in\\mathbb\{F\}\_\{2\}^\{m\}:x\\cdot v=0,\\forall v\\in V\\\}\.now note that the indexv∗∈V⟂v^\{\*\}\\in V^\{\\perp\}satisfies for allu∈S⊂Vu\\in S\\subset Vthatv∗⋅u=0v^\{\*\}\\cdot u=0, and hence\(−1\)v∗⋅u=1\(\-1\)^\{v^\{\*\}\\cdot u\}=1\. Consequently

a⋅ϕv∗=∑u∈Sau​\(−1\)v∗⋅u=∑u∈Sau=∑u=1K−1au\.a\\cdot\\phi\_\{v^\{\*\}\}=\\sum\_\{u\\in S\}a\_\{u\}\(\-1\)^\{v^\{\*\}\\cdot u\}=\\sum\_\{u\\in S\}a\_\{u\}=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\.This shows that fork=v∗≠0k=v^\{\*\}\\neq 0, the corresponding off\-diagonal value equals the diagonal value, i\.e\.Zc​c=Zc′​cZ\_\{cc\}=Z\_\{c^\{\\prime\}c\}for somec′≠cc^\{\\prime\}\\neq c, contradicting the strict inequality\. Hence we must have\|S\|≥m\|S\|\\geq m\.

To prove that the minimal size ofSSis exactlymm, chooseSSto be the standard basis of𝔽2m\\mathbb\{F\}\_\{2\}^\{m\}, and setai=1a\_\{i\}=1for alli∈Si\\in S\. then for any indexv≠0v\\neq 0we have:

∑u=1K−1au​\(−1\)v⋅u=∑r=1m\(−1\)vr=m−2​\|\{r:vr=1\}\|<m=∑u=1K−1au,\\sum\_\{u=1\}^\{K\-1\}a\_\{u\}\(\-1\)^\{v\\cdot u\}=\\sum\_\{r=1\}^\{m\}\(\-1\)^\{v\_\{r\}\}=m\-2\|\\\{r:v\_\{r\}=1\\\}\|<m=\\sum\_\{u=1\}^\{K\-1\}a\_\{u\},
wherevrv\_\{r\}is therthr^\{\\textrm\{th\}\}element of the binary representation ofvv\. sincev≠0v\\neq 0, the set\{r:vr=1\}\\\{r:v\_\{r\}=1\\\}must be non\-empty, and so we must haveZc​c−Zc′​c\>0Z\_\{cc\}\-Z\_\{c^\{\\prime\}c\}\>0for allc′≠cc^\{\\prime\}\\neq c\. This completes the proof\. ∎

## Appendix DProofs

### D\.1Proof of Theorem[3\.1](https://arxiv.org/html/2605.23087#S3.Thmtheorem1)

We consider the objective described in Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\)\. We initially focus on the DNC solution, as described in Definition[2\.1](https://arxiv.org/html/2605.23087#S2.Thmtheorem1)\. In particular, since this definition specifies that the normalized matrices are balanced, and the asymptotic problem characterizes the normalized matrices, we consider balanced matrices throughout the proof\. Hence DNC is given by

WL=Q​Σ​ULT,H1=U1​Σ′​VT,Wl=Ul\+1​Σ′′​UlT,l=1,…,L−1,W\_\{L\}=Q\\Sigma U\_\{L\}^\{T\},\\quad H\_\{1\}=U\_\{1\}\\Sigma^\{\\prime\}V^\{T\},\\quad W\_\{l\}=U\_\{l\+1\}\\Sigma^\{\\prime\\prime\}U\_\{l\}^\{T\},\\quad l=1,\.\.\.,L\-1,\(10\)whereUL,…,U1∈ℝd×dU\_\{L\},\.\.\.,U\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\}are any orthogonal matrices,Σ∈ℝK×d\\Sigma\\in\\mathbb\{R\}^\{K\\times d\},Σ′∈ℝd×K​n\\Sigma^\{\\prime\}\\in\\mathbb\{R\}^\{d\\times Kn\},Σ′′∈ℝd×d\\Sigma^\{\\prime\\prime\}\\in\\mathbb\{R\}^\{d\\times d\}all have their topK×KK\\times Kblock given byα​diag​\(1,1,…,1,0\)\\alpha\\textrm\{diag\}\(1,1,\.\.\.,1,0\), for someα∈ℝ\\alpha\\in\\mathbb\{R\}\. The matricesQ∈ℝK×KQ\\in\\mathbb\{R\}^\{K\\times K\}andV∈ℝK​n×K​nV\\in\\mathbb\{R\}^\{Kn\\times Kn\}are orthogonal such that

Q​diag​\(1,1,…,1,0\)​VT=1n​S⊗1nT,Q\\textrm\{diag\}\(1,1,\.\.\.,1,0\)V^\{T\}=\\frac\{1\}\{\\sqrt\{n\}\}S\\otimes 1\_\{n\}^\{T\},and henceZ=1n​αL\+1​S⊗1nTZ=\\frac\{1\}\{\\sqrt\{n\}\}\\alpha^\{L\+1\}S\\otimes 1\_\{n\}^\{T\}\.

Note as a consequence we have for alli=1,…,ni=1,\.\.\.,nandc≠c′=1,\.\.,Kc\\neq c^\{\\prime\}=1,\.\.,K

\(zi​c\)c−\(zi​c\)c′=1n​αL\+1​\[K−1K\+1K\]=1n​αL\+1,\(z\_\{ic\}\)\_\{c\}\-\(z\_\{ic\}\)\_\{c^\{\\prime\}\}=\\frac\{1\}\{\\sqrt\{n\}\}\\alpha^\{L\+1\}\\left\[\\frac\{K\-1\}\{K\}\+\\frac\{1\}\{K\}\\right\]=\\frac\{1\}\{\\sqrt\{n\}\}\\alpha^\{L\+1\},so the constraint of Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\) is simplyα≥\(n\)1L\+1\\alpha\\geq\(\\sqrt\{n\}\)^\{\\frac\{1\}\{L\+1\}\}\.

The objective value is then

‖H1‖F2\+∑l=1L‖Wl‖F2=\(L\+1\)​\(K−1\)​α2≥\(L\+1\)​\(K−1\)​n1L\+1,\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}=\(L\+1\)\(K\-1\)\\alpha^\{2\}\\geq\(L\+1\)\(K\-1\)n^\{\\frac\{1\}\{L\+1\}\},\(11\)and this gives the objective value achieved by the DNC solution\.

We shall now show that the cross\-polytope solution outperforms DNC for the given range ofKKandLL\. First we shall assumeKKis even, we will cover the odd case after\. The cross\-polytope uses a similarly balanced construction to Eq\. \([10](https://arxiv.org/html/2605.23087#A4.E10)\), and is given by

WL=Q~​Σ~​U~LT,H1=U~1​Σ~′​V~T,Wl=U~l\+1​Σ~′′​U~lT,l=1,…,L−1,W\_\{L\}=\\tilde\{Q\}\\tilde\{\\Sigma\}\\tilde\{U\}\_\{L\}^\{T\},\\quad H\_\{1\}=\\tilde\{U\}\_\{1\}\\tilde\{\\Sigma\}^\{\\prime\}\\tilde\{V\}^\{T\},\\quad W\_\{l\}=\\tilde\{U\}\_\{l\+1\}\\tilde\{\\Sigma\}^\{\\prime\\prime\}\\tilde\{U\}\_\{l\}^\{T\},\\quad l=1,\.\.\.,L\-1,\(12\)whereU~L,…,U~1∈ℝd×d\\tilde\{U\}\_\{L\},\.\.\.,\\tilde\{U\}\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\}are any orthogonal matrices,Σ~∈ℝK×d\\tilde\{\\Sigma\}\\in\\mathbb\{R\}^\{K\\times d\},Σ~′∈ℝd×K​n\\tilde\{\\Sigma\}^\{\\prime\}\\in\\mathbb\{R\}^\{d\\times Kn\},Σ~′′∈ℝd×d\\tilde\{\\Sigma\}^\{\\prime\\prime\}\\in\\mathbb\{R\}^\{d\\times d\}all have their topK×KK\\times Kblock given byβ​diag​\(1K2,0K2\)\\beta\\textrm\{diag\}\(1\_\{\\frac\{K\}\{2\}\},0\_\{\\frac\{K\}\{2\}\}\), for someβ∈ℝ\\beta\\in\\mathbb\{R\}\. The matricesQ~∈ℝK×K\\tilde\{Q\}\\in\\mathbb\{R\}^\{K\\times K\}andV~∈ℝK​n×K​n\\tilde\{V\}\\in\\mathbb\{R\}^\{Kn\\times Kn\}are orthogonal such that

Q~​diag​\(1K2,0K2\)​V~T=12​n​C⊗1nT,\\tilde\{Q\}\\textrm\{diag\}\(1\_\{\\frac\{K\}\{2\}\},0\_\{\\frac\{K\}\{2\}\}\)\\tilde\{V\}^\{T\}=\\frac\{1\}\{2\\sqrt\{n\}\}C\\otimes 1\_\{n\}^\{T\},\(13\)whereC∈ℝK×KC\\in\\mathbb\{R\}^\{K\\times K\}is given in block form by

C=\[X0…00X…0…………00…X\],where​X=\[1−1−11\]\.C=\\begin\{bmatrix\}X&0&\.\.\.&0\\\\ 0&X&\.\.\.&0\\\\ \.\.\.&\.\.\.&\.\.\.&\.\.\.\\\\ 0&0&\.\.\.&X\\end\{bmatrix\},\\quad\\textrm\{where \}X=\\begin\{bmatrix\}1&\-1\\\\ \-1&1\\end\{bmatrix\}\.\(14\)HenceZ=12​n​βL\+1​C⊗1nTZ=\\frac\{1\}\{2\\sqrt\{n\}\}\\beta^\{L\+1\}C\\otimes 1\_\{n\}^\{T\}\.

This gives that the constraint condition of the objective in Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\) is

\(zi​c\)c−\(zi​c\)c′=\{1n​βL\+1ifcandc′are paired in the cross\-polytope,12​n​βL\+1otherwise\.\(z\_\{ic\}\)\_\{c\}\-\(z\_\{ic\}\)\_\{c^\{\\prime\}\}=\\begin\{cases\}\\frac\{1\}\{\\sqrt\{n\}\}\\beta^\{L\+1\}&\\textrm\{if $c$ and $c^\{\\prime\}$ are paired in the cross\-polytope, \}\\\\ \\frac\{1\}\{2\\sqrt\{n\}\}\\beta^\{L\+1\}&\\textrm\{otherwise\.\}\\end\{cases\}Hence the constraint is satisfied whenβ≥\(2​n\)1L\+1\\beta\\geq\(2\\sqrt\{n\}\)^\{\\frac\{1\}\{L\+1\}\}\.

The objective value is then

‖H1‖F2\+∑l=1L‖Wl‖F2=\(L\+1\)​K2​β2≥\(L\+1\)​K2​\(2​n\)2L\+1\.\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}=\(L\+1\)\\frac\{K\}\{2\}\\beta^\{2\}\\geq\(L\+1\)\\frac\{K\}\{2\}\(2\\sqrt\{n\}\)^\{\\frac\{2\}\{L\+1\}\}\.\(15\)This outperforms the NC objective value stated in Eq\. \([11](https://arxiv.org/html/2605.23087#A4.E11)\) when

\(L\+1\)​K2​\(2​n\)2L\+1<\(L\+1\)​\(K−1\)​n1L\+1,\(L\+1\)\\frac\{K\}\{2\}\(2\\sqrt\{n\}\)^\{\\frac\{2\}\{L\+1\}\}<\(L\+1\)\(K\-1\)n^\{\\frac\{1\}\{L\+1\}\},which is equivalent to

22L\+1<2​\(1−1K\),2^\{\\frac\{2\}\{L\+1\}\}<2\\left\(1\-\\frac\{1\}\{K\}\\right\),and this inequality holds whenL=2,K≥6L=2,K\\geq 6, orL≥3,K≥4L\\geq 3,K\\geq 4\. This covers all evenKKin this range\.

It remains to cover the oddKK\. Here take the same solution as the even case specified in Eq\. \([12](https://arxiv.org/html/2605.23087#A4.E12)\) and Eq\. \([13](https://arxiv.org/html/2605.23087#A4.E13)\), but update the singular values so thatΣ~,Σ~′,Σ~′′\\tilde\{\\Sigma\},\\tilde\{\\Sigma\}^\{\\prime\},\\tilde\{\\Sigma\}^\{\\prime\\prime\}have their topK×KK\\times Kblock given byβ​diag​\(1K−12,2−1L\+1,0K−12\)\\beta\\textrm\{diag\}\(1\_\{\\frac\{K\-1\}\{2\}\},2^\{\-\\frac\{1\}\{L\+1\}\},0\_\{\\frac\{K\-1\}\{2\}\}\), and the orthogonal matricesQ~,V~\\tilde\{Q\},\\tilde\{V\}so that

Q~​diag​\(1K−12,12,0K−12\)​V~T=12​n​C¯⊗1nT,\\tilde\{Q\}\\textrm\{diag\}\(1\_\{\\frac\{K\-1\}\{2\}\},\\frac\{1\}\{2\},0\_\{\\frac\{K\-1\}\{2\}\}\)\\tilde\{V\}^\{T\}=\\frac\{1\}\{2\\sqrt\{n\}\}\\bar\{C\}\\otimes 1\_\{n\}^\{T\},where

C¯=\[C001\],whereC∈ℝK−1×K−1has entries as before\.\\bar\{C\}=\\begin\{bmatrix\}C&0\\\\ 0&1\\end\{bmatrix\},\\quad\\textrm\{where $C\\in\\mathbb\{R\}^\{K\-1\\times K\-1\}$ has entries as before\. \}ConsequentlyZ=12​n​βL\+1​C¯⊗1nTZ=\\frac\{1\}\{2\\sqrt\{n\}\}\\beta^\{L\+1\}\\bar\{C\}\\otimes 1\_\{n\}^\{T\}\.

The condition\(zi​c\)c−\(zi​c\)c′≥1\(z\_\{ic\}\)\_\{c\}\-\(z\_\{ic\}\)\_\{c^\{\\prime\}\}\\geq 1again reduces toβ≥\(2​n\)1L\+1\\beta\\geq\(2\\sqrt\{n\}\)^\{\\frac\{1\}\{L\+1\}\}, and now the objective takes value

‖H1‖F2\+∑l=1L‖Wl‖F2=\(L\+1\)​\[K−12​β2\+2−2L\+1​β2\]≥\(L\+1\)​\(2​n\)2L\+1​\[K−12\+2−2L\+1\],\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}=\(L\+1\)\\left\[\\frac\{K\-1\}\{2\}\\beta^\{2\}\+2^\{\-\\frac\{2\}\{L\+1\}\}\\beta^\{2\}\\right\]\\geq\(L\+1\)\(2\\sqrt\{n\}\)^\{\\frac\{2\}\{L\+1\}\}\\left\[\\frac\{K\-1\}\{2\}\+2^\{\-\\frac\{2\}\{L\+1\}\}\\right\],\(16\)and this outperforms the DNC objective value of Eq\. \([11](https://arxiv.org/html/2605.23087#A4.E11)\) when

\(L\+1\)​n1L\+1​\[K−12​22L\+1\+1\]≤\(L\+1\)​\(K−1\)​n1L\+1,\(L\+1\)n^\{\\frac\{1\}\{L\+1\}\}\\left\[\\frac\{K\-1\}\{2\}2^\{\\frac\{2\}\{L\+1\}\}\+1\\right\]\\leq\(L\+1\)\(K\-1\)n^\{\\frac\{1\}\{L\+1\}\},which is equivalent to

22L\+1≤2​\(1−1K−1\)\.2^\{\\frac\{2\}\{L\+1\}\}\\leq 2\\left\(1\-\\frac\{1\}\{K\-1\}\\right\)\.This holds whenL=2,K≥6L=2,K\\geq 6orL≥3,K≥5L\\geq 3,K\\geq 5\.

Consequently we have that the final values ofLLandKKfor which DNC is suboptimal are given byL=2,K≥6L=2,K\\geq 6andL≥3,K≥4L\\geq 3,K\\geq 4\.

It remains to prove local optimality of the DNC solution\.

Proof of Local Optimality:Denote the set of parameters byΘ=\(WL,…,W1,H1\)\\Theta=\(W\_\{L\},\\dots,W\_\{1\},H\_\{1\}\), and the max\-margin objective byE​\(Θ\)=‖H1‖F2\+∑l=1L‖Wl‖F2E\(\\Theta\)=\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}\. The logit matrix is a continuous mappingZ​\(Θ\)=WL​…​W1​H1Z\(\\Theta\)=W\_\{L\}\\dots W\_\{1\}H\_\{1\}\. By Lemma[D\.7](https://arxiv.org/html/2605.23087#A4.Thmtheorem7), for anyΘ\\Theta, the objective is bounded below as follows:

E\(Θ\)≥\(L\+1\)∥Z\(Θ\)∥Spp=:f\(Z\),p=2L\+1\.E\(\\Theta\)\\geq\(L\+1\)\\\|Z\(\\Theta\)\\\|\_\{S\_\{p\}\}^\{p\}=:f\(Z\)\\,,\\qquad p=\\frac\{2\}\{L\+1\}\.Moreover, forΘ∗\\Theta^\{\*\}the parameters of the DNC solution, for whichZ​\(Θ∗\)=Z∗=S⊗1n⊤Z\(\\Theta^\{\*\}\)=Z\_\{\*\}=S\\otimes 1\_\{n\}^\{\\top\}, we haveE​\(Θ∗\)=\(L\+1\)​‖Z∗‖Spp=f​\(Z∗\)E\(\\Theta^\{\*\}\)=\(L\+1\)\\\|Z\_\{\*\}\\\|\_\{S\_\{p\}\}^\{p\}=f\(Z\_\{\*\}\)\.

To show thatΘ∗\\Theta^\{\*\}is a local minimum ofE​\(Θ\)E\(\\Theta\), it suffices to prove thatZ∗Z\_\{\*\}is a strict local minimum off​\(Z\)f\(Z\)over the feasible set𝒵feas=\{Z∣gi​c​c′​\(Z\)≥0\}\\mathcal\{Z\}\_\{\\mathrm\{feas\}\}=\\\{Z\\mid g\_\{icc^\{\\prime\}\}\(Z\)\\geq 0\\\}, wheregi​c​c′​\(Z\)=\(Zi​c\)c−\(Zi​c\)c′−1g\_\{icc^\{\\prime\}\}\(Z\)=\(Z\_\{ic\}\)\_\{c\}\-\(Z\_\{ic\}\)\_\{c^\{\\prime\}\}\-1\. If this were true, then for anyΘ\\Thetasufficiently close toΘ∗\\Theta^\{\*\}such thatZ​\(Θ\)≠Z∗Z\(\\Theta\)\\neq Z\_\{\*\}andZ​\(Θ\)∈𝒵feasZ\(\\Theta\)\\in\\mathcal\{Z\}\_\{\\mathrm\{feas\}\}, we haveE​\(Θ\)≥f​\(Z​\(Θ\)\)\>f​\(Z∗\)=O​\(Θ∗\)E\(\\Theta\)\\geq f\(Z\(\\Theta\)\)\>f\(Z\_\{\*\}\)=O\(\\Theta^\{\*\}\)\.

Letr=K−1r=K\-1, and recall this is the rank ofZ∗Z\_\{\*\}which hasrridentical singular values equal ton\\sqrt\{n\}\. Define

fsmooth​\(Z\)=\(L\+1\)​∑i=1rσi​\(Z\)p,f\_\{\\mathrm\{smooth\}\}\(Z\)=\(L\+1\)\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}\(Z\)^\{p\}\\,,\(17\)over the top\-rrsingular values ofZZ\. Because these toprrsingular values ofZ∗Z\_\{\*\}are separated from the zero singular value by a strictly positive spectral gap, this function is twice continuously\-differentiable in a neighborhood ofZ∗Z\_\{\*\}\. Specifically, its gradient

∇fsmooth​\(Z\)=2​U​diag​\(σ1p−1,…,σrp−1\)​VT,where​Z=U​diag​\(σ1,…,σr\)​VT,\\nabla f\_\{\\mathrm\{smooth\}\}\(Z\)=2\\,U\\textrm\{diag\}\(\\sigma\_\{1\}^\{p\-1\},\\ldots,\\sigma\_\{r\}^\{p\-1\}\)\\,V^\{T\},\\quad\\textrm\{where \}Z=U\\textrm\{diag\}\(\\sigma\_\{1\},\\ldots,\\sigma\_\{r\}\)V^\{T\},atZ∗Z\_\{\*\}, evaluates to

G:=∇Zfsmooth​\(Z∗\)=2​n−L/\(L\+1\)​Z∗\.G:=\\nabla\_\{Z\}f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)=2n^\{\-L/\(L\+1\)\}Z\_\{\*\}\.
The constraintsgi​c​c′​\(Z\)g\_\{icc^\{\\prime\}\}\(Z\)defining𝒵feas\\mathcal\{Z\}\_\{\\mathrm\{feas\}\}are linear inZZ\. Also, atZ∗Z\_\{\*\}, all constraints are tight \(gi​c​c′​\(Z∗\)=0g\_\{icc^\{\\prime\}\}\(Z\_\{\*\}\)=0\) and, for any feasibleZk∈𝒵feasZ\_\{k\}\\in\\mathcal\{Z\}\_\{\\mathrm\{feas\}\}, we havegi​c​c′​\(Zk\)≥0g\_\{icc^\{\\prime\}\}\(Z\_\{k\}\)\\geq 0\. By linearity, this implies:

⟨∇gi​c​c′​\(Z\),Zk−Z∗⟩=gi​c​c′​\(Zk\)−gi​c​c′​\(Z∗\)≥0\.\\displaystyle\\langle\\nabla g\_\{icc^\{\\prime\}\}\(Z\),Z\_\{k\}\-Z\_\{\*\}\\rangle=g\_\{icc^\{\\prime\}\}\(Z\_\{k\}\)\-g\_\{icc^\{\\prime\}\}\(Z\_\{\*\}\)\\geq 0\\,\.\(18\)Here, since

gi​c​c′\(Z\)=\(ec−ec′\)⊤Zec​i−1⟹∇Zgi​c​c′\(Z\)=\(ec−ec′\)ec​i⊤=:∇Zgi​c​c′\.g\_\{icc^\{\\prime\}\}\(Z\)=\(e\_\{c\}\-e\_\{c^\{\\prime\}\}\)^\{\\top\}Ze\_\{ci\}\-1\\implies\\nabla\_\{Z\}g\_\{icc^\{\\prime\}\}\(Z\)=\(e\_\{c\}\-e\_\{c^\{\\prime\}\}\)e\_\{ci\}^\{\\top\}=:\\nabla\_\{Z\}g\_\{icc^\{\\prime\}\}\.Thus, it is easy to check that

G=2​n−L/\(L\+1\)​Z∗=λ​∑i,c,c′≠c∇gi​c​c′,for​λ:=2K​n−L/\(L\+1\)\>0\.\\displaystyle G=2n^\{\-L/\(L\+1\)\}Z\_\{\*\}=\\lambda\\sum\_\{i,c,c^\{\\prime\}\\neq c\}\\nabla g\_\{icc^\{\\prime\}\}\\,,\\qquad\\text\{for\}\\,\\,\\lambda:=\\frac\{2\}\{K\}n^\{\-L/\(L\+1\)\}\>0\\,\.\(19\)Combining the above displays, it follows that foranyfeasibleZkZ\_\{k\}:

⟨G,Zk−Z∗⟩=λ​∑i,c,c′≠c⟨∇gi​c​c′,Zk−Z∗⟩≥0\.\\displaystyle\\langle G,Z\_\{k\}\-Z\_\{\*\}\\rangle=\\lambda\\sum\_\{i,c,c^\{\\prime\}\\neq c\}\\langle\\nabla g\_\{icc^\{\\prime\}\},Z\_\{k\}\-Z\_\{\*\}\\rangle\\geq 0\\,\.\(20\)
Now, assume for contradiction thatZ∗Z\_\{\*\}is*not*a*strict*local minimum off​\(Z\)f\(Z\)over𝒵feas\\mathcal\{Z\}\_\{\\mathrm\{feas\}\}\. Then, for everyϵ\>0\\epsilon\>0, there exists at least one feasible pointZ≠Z∗Z\\neq Z\_\{\*\}within anϵ\\epsilon\-neighborhood ofZ∗Z\_\{\*\}that does not strictly increase the objective\. By takingϵk=1/k\\epsilon\_\{k\}=1/k, we can explicitly construct an infinite sequence\(Zk\)k=1∞\(Z\_\{k\}\)\_\{k=1\}^\{\\infty\}converging toZ∗Z\_\{\*\}such that for everyk≥1k\\geq 1: \(1\)Zk∈𝒵feasZ\_\{k\}\\in\\mathcal\{Z\}\_\{\\mathrm\{feas\}\}, \(2\)Zk≠Z∗Z\_\{k\}\\neq Z\_\{\*\}, and, \(3\)f​\(Zk\)≤f​\(Z∗\)f\(Z\_\{k\}\)\\leq f\(Z\_\{\*\}\)\.

Lettk=‖Zk−Z∗‖Ft\_\{k\}=\\\|Z\_\{k\}\-Z\_\{\*\}\\\|\_\{F\}, sotk→0t\_\{k\}\\rightarrow 0, andtk\>0t\_\{k\}\>0for allkk\. Define normalized directionsΔk=\(Zk−Z∗\)/tk\\Delta\_\{k\}=\(Z\_\{k\}\-Z\_\{\*\}\)/t\_\{k\}\. Since‖Δk‖F=1\\\|\\Delta\_\{k\}\\\|\_\{F\}=1for allkk, we can extract a convergent subsequenceΔk→Δ¯\\Delta\_\{k\}\\to\\bar\{\\Delta\}with‖Δ¯‖F=1\\\|\\bar\{\\Delta\}\\\|\_\{F\}=1\. Without loss of generality, relabel the subsequence indices askkso that the propertiesf​\(Zk\)≤f​\(Z∗\)f\(Z\_\{k\}\)\\leq f\(Z\_\{\*\}\)andtk\>0t\_\{k\}\>0continue to hold for allkkin the subsequence\.

Becausefsmoothf\_\{\\mathrm\{smooth\}\}is twice continuously differentiable in a neighborhood ofZ∗Z\_\{\*\}, its Hessian∇2fsmooth\\nabla^\{2\}f\_\{\\mathrm\{smooth\}\}is continuous\. On any sufficiently small compact ball centered atZ∗Z\_\{\*\}, the operator norm of the Hessian is bounded by a finite maximumMM\. For sufficiently largekk, all matricesZkZ\_\{k\}in the sequence lie within this ball\. In this case, by Taylor’s theorem, there exists a uniform constantC=M/2\>0C=M/2\>0independent ofkksuch that for all for all largekk:

fsmooth​\(Zk\)≥fsmooth​\(Z∗\)\+tk​⟨G,Δk⟩−C​tk2\\displaystyle f\_\{\\mathrm\{smooth\}\}\(Z\_\{k\}\)\\geq f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)\+t\_\{k\}\\langle G,\\Delta\_\{k\}\\rangle\-Ct\_\{k\}^\{2\}\(21\)
Because of Eq\. \([20](https://arxiv.org/html/2605.23087#A4.E20)\) andtk\>0t\_\{k\}\>0, it holds that⟨G,Δk⟩≥0\\langle G,\\Delta\_\{k\}\\rangle\\geq 0for allkk, which implies

⟨G,Δ¯⟩≥0\.\\displaystyle\\langle G,\\bar\{\\Delta\}\\rangle\\geq 0\\,\.\(22\)We split the analysis into two cases based onΔ¯\\bar\{\\Delta\}:

Case 1:⟨G,Δ¯⟩\>0\\langle G,\\bar\{\\Delta\}\\rangle\>0\.

Letη=⟨G,Δ¯⟩\>0\\eta=\\langle G,\\bar\{\\Delta\}\\rangle\>0\. Sincef​\(Z\)≥fsmooth​\(Z\)f\(Z\)\\geq f\_\{\\mathrm\{smooth\}\}\(Z\)andf​\(Z∗\)=fsmooth​\(Z∗\)f\(Z\_\{\*\}\)=f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\), by applying Eq\. \([21](https://arxiv.org/html/2605.23087#A4.E21)\), for all sufficiently largekk:

f​\(Zk\)−f​\(Z∗\)≥fsmooth​\(Zk\)−fsmooth​\(Z∗\)≥tk​⟨G,Δk⟩−C​tk2=tk​\(⟨G,Δk⟩−C​tk\)f\(Z\_\{k\}\)\-f\(Z\_\{\*\}\)\\geq f\_\{\\mathrm\{smooth\}\}\(Z\_\{k\}\)\-f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)\\geq t\_\{k\}\\langle G,\\Delta\_\{k\}\\rangle\-Ct\_\{k\}^\{2\}=t\_\{k\}\\left\(\\langle G,\\Delta\_\{k\}\\rangle\-Ct\_\{k\}\\right\)Sincetk→0t\_\{k\}\\to 0andCCis a constant independent ofkk, the termC​tk→0Ct\_\{k\}\\to 0\. Meanwhile, because the inner product is a continuous linear functional andΔk→Δ¯\\Delta\_\{k\}\\to\\bar\{\\Delta\}, we havelimk→∞⟨G,Δk⟩=η\>0\\lim\_\{k\\to\\infty\}\\langle G,\\Delta\_\{k\}\\rangle=\\eta\>0\. Therefore, for sufficiently largekk, we simultaneously have⟨G,Δk⟩\>η/2\\langle G,\\Delta\_\{k\}\\rangle\>\\eta/2andC​tk<η/2Ct\_\{k\}<\\eta/2\. For all suchkk, the term in the parentheses is strictly positive, yieldingf​\(Zk\)−f​\(Z∗\)\>0f\(Z\_\{k\}\)\-f\(Z\_\{\*\}\)\>0, contradicting our assumption thatf​\(Zk\)≤f​\(Z∗\)f\(Z\_\{k\}\)\\leq f\(Z\_\{\*\}\)\.

Case 2:⟨G,Δ¯⟩=0\\langle G,\\bar\{\\Delta\}\\rangle=0\.

In this case, by Eq\. \([19](https://arxiv.org/html/2605.23087#A4.E19)\),

0=⟨G,Δ¯⟩=λ​∑i,c,c′≠c⟨∇gi​c​c′,Δ¯⟩⟹⟨∇gi​c​c′,Δ¯⟩=0,∀i,c,c′≠c,0=\\langle G,\\bar\{\\Delta\}\\rangle=\\lambda\\sum\_\{i,c,c^\{\\prime\}\\neq c\}\\langle\\nabla g\_\{icc^\{\\prime\}\},\\bar\{\\Delta\}\\rangle\\implies\\langle\\nabla g\_\{icc^\{\\prime\}\},\\bar\{\\Delta\}\\rangle=0,\\forall i,c,c^\{\\prime\}\\neq c\\,,where for the implication we used thatλ\>0\\lambda\>0and from Eq\. \([18](https://arxiv.org/html/2605.23087#A4.E18)\) that⟨∇gi​c​c′,Δ¯⟩≥0\\langle\\nabla g\_\{icc^\{\\prime\}\},\\bar\{\\Delta\}\\rangle\\geq 0for alli,c,c′≠ci,c,c^\{\\prime\}\\neq c\.

In fact, this gives

⟨∇gi​c​c′,Δ¯⟩=Δ¯c,i​c−Δ¯c′,i​c=0⟹Δ¯c,i​c=Δ¯c′,i​c,∀i,c,c′≠c⟹Δ¯=1K​v⊤,\\langle\\nabla g\_\{icc^\{\\prime\}\},\\bar\{\\Delta\}\\rangle=\\bar\{\\Delta\}\_\{c,ic\}\-\\bar\{\\Delta\}\_\{c^\{\\prime\},ic\}=0\\implies\\bar\{\\Delta\}\_\{c,ic\}=\\bar\{\\Delta\}\_\{c^\{\\prime\},ic\},\\quad\\forall i,c,c^\{\\prime\}\\neq c\\,\\,\\implies\\,\\,\\bar\{\\Delta\}=1\_\{K\}v^\{\\top\}\\,,for somevv\. Since‖Δ¯‖F=1\\\|\\bar\{\\Delta\}\\\|\_\{F\}=1, we knowv≠0v\\neq 0\.

We will compute the singular values of

Mt:=Z∗\+t​Δ¯=Z∗\+t​1K​v⊤\.M\_\{t\}:=Z\_\{\*\}\+t\\bar\{\\Delta\}=Z\_\{\*\}\+t\\,1\_\{K\}v^\{\\top\}\.It suffices to work with the eigenvalues of

Mt⊤​Mt=Z∗⊤​Z∗\+K​t2​v​v⊤,M\_\{t\}^\{\\top\}M\_\{t\}=Z\_\{\*\}^\{\\top\}Z\_\{\*\}\+Kt^\{2\}vv^\{\\top\}\\,,where we used that1K⊤​Z∗=01\_\{K\}^\{\\top\}Z\_\{\*\}=0\.

Now decompose

v=v∥\+v0,v∥∈row⁡\(Z∗\),v0∈ker⁡\(Z∗\)\.v=v\_\{\\parallel\}\+v\_\{0\},\\qquad v\_\{\\parallel\}\\in\\operatorname\{row\}\(Z\_\{\*\}\),\\quad v\_\{0\}\\in\\ker\(Z\_\{\*\}\)\.Because allK−1K\-1nonzero singular values ofZ∗Z\_\{\*\}are equal ton\\sqrt\{n\},Z∗⊤​Z∗Z\_\{\*\}^\{\\top\}Z\_\{\*\}acts asn​InIonrow⁡\(Z∗\)\\operatorname\{row\}\(Z\_\{\*\}\)and as0onker⁡\(Z∗\)\\ker\(Z\_\{\*\}\)\. Hence

Z∗⊤​Z∗​v∥=n​v∥,Z∗⊤​Z∗​v0=0\.Z\_\{\*\}^\{\\top\}Z\_\{\*\}\\,v\_\{\\parallel\}=nv\_\{\\parallel\},\\qquad Z\_\{\*\}^\{\\top\}Z\_\{\*\}\\,v\_\{0\}=0\.We distinguish two subcases\.

*Subcase 2a:v0≠0v\_\{0\}\\neq 0\.*

We will show that there exists constantc1\>0c\_\{1\}\>0such that for all small enoughtt

σK​\(Mt\)≥c1​t\.\\displaystyle\\sigma\_\{K\}\(M\_\{t\}\)\\geq c\_\{1\}t\\,\.\(23\)If this were true then we argue as follows: Write

Zk=Mtk\+tk​Ek,Ek:=Δk−Δ¯,‖Ek‖F→0\.Z\_\{k\}=M\_\{t\_\{k\}\}\+t\_\{k\}E\_\{k\},\\qquad E\_\{k\}:=\\Delta\_\{k\}\-\\bar\{\\Delta\},\\qquad\\\|E\_\{k\}\\\|\_\{F\}\\to 0\.By Weyl’s inequality,

σK​\(Zk\)≥σK​\(Mtk\)−tk​‖Ek‖op\.\\sigma\_\{K\}\(Z\_\{k\}\)\\geq\\sigma\_\{K\}\(M\_\{t\_\{k\}\}\)\-t\_\{k\}\\\|E\_\{k\}\\\|\_\{\\mathrm\{op\}\}\.Since‖Ek‖op≤‖Ek‖F→0\\\|E\_\{k\}\\\|\_\{\\mathrm\{op\}\}\\leq\\\|E\_\{k\}\\\|\_\{F\}\\to 0, for all sufficiently largekk, by Eq\. \([23](https://arxiv.org/html/2605.23087#A4.E23)\):

σK​\(Zk\)≥c12​tk\.\\sigma\_\{K\}\(Z\_\{k\}\)\\geq\\frac\{c\_\{1\}\}\{2\}t\_\{k\}\.Using Taylor of the smooth part \([21](https://arxiv.org/html/2605.23087#A4.E21)\),

fsmooth​\(Zk\)−fsmooth​\(Z∗\)≥−C​tk2,f\_\{\\mathrm\{smooth\}\}\(Z\_\{k\}\)\-f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)\\geq\-Ct\_\{k\}^\{2\},from which we conclude that

f​\(Zk\)−f​\(Z∗\)≥\(L\+1\)​σK​\(Zk\)p−C​tk2≥\(L\+1\)​\(c12\)p​tkp−C​tk2\.f\(Z\_\{k\}\)\-f\(Z\_\{\*\}\)\\geq\(L\+1\)\\sigma\_\{K\}\(Z\_\{k\}\)^\{p\}\-Ct\_\{k\}^\{2\}\\geq\(L\+1\)\\Bigl\(\\frac\{c\_\{1\}\}\{2\}\\Bigr\)^\{p\}t\_\{k\}^\{p\}\-Ct\_\{k\}^\{2\}\.Becausep=2L\+1<1p=\\frac\{2\}\{L\+1\}<1whenL≥2L\\geq 2, the termtkpt\_\{k\}^\{p\}dominatestk2t\_\{k\}^\{2\}ask→∞k\\to\\infty\. Hencef​\(Zk\)\>f​\(Z∗\)f\(Z\_\{k\}\)\>f\(Z\_\{\*\}\)for all sufficiently largekk, contradicting the assumptionf​\(Zk\)≤f​\(Z∗\)f\(Z\_\{k\}\)\\leq f\(Z\_\{\*\}\)\.

It remains proving Eq\. \([23](https://arxiv.org/html/2605.23087#A4.E23)\)\. We first handle separately the degenerate casev∥=0v\_\{\\parallel\}=0\. Sincev​v⊤vv^\{\\top\}acts onker⁡\(Z∗\)\\ker\(Z\_\{\*\}\)whileZ∗⊤​Z∗Z\_\{\*\}^\{\\top\}Z\_\{\*\}acts asn​InIonrow⁡\(Z∗\)\\operatorname\{row\}\(Z\_\{\*\}\)and vanishes onker⁡\(Z∗\)\\ker\(Z\_\{\*\}\), the non\-zero eigenvalues ofMt⊤​MtM\_\{t\}^\{\\top\}M\_\{t\}are

nwith multiplicity​K−1,K​t2​‖v‖22with multiplicity​1\.n\\quad\\text\{with multiplicity \}K\-1,\\qquad Kt^\{2\}\\\|v\\\|\_\{2\}^\{2\}\\quad\\text\{with multiplicity \}1\\,\.Therefore, for small enought<n/\(K​‖v‖2\)t<\\sqrt\{n/\(K\\\|v\\\|^\{2\}\)\}

σK​\(Mt\)≥c1​t\\sigma\_\{K\}\(M\_\{t\}\)\\geq c\_\{1\}tforc1=K​‖v‖2c\_\{1\}=\\sqrt\{K\}\\,\\\|v\\\|\_\{2\}\.

Assume now that bothv∥≠0v\_\{\\parallel\}\\neq 0andv0≠0v\_\{0\}\\neq 0\. Define orthonormal basis ofspan⁡\{v∥,v0\}\\operatorname\{span\}\\\{v\_\{\\parallel\},v\_\{0\}\\\}:

e1:=v∥‖v∥‖2,e0:=v0‖v0‖2\.e\_\{1\}:=\\frac\{v\_\{\\parallel\}\}\{\\\|v\_\{\\parallel\}\\\|\_\{2\}\},\\qquad e\_\{0\}:=\\frac\{v\_\{0\}\}\{\\\|v\_\{0\}\\\|\_\{2\}\}\\,\.In this basis, the restriction ofMt⊤​MtM\_\{t\}^\{\\top\}M\_\{t\}tospan⁡\{v∥,v0\}\\operatorname\{span\}\\\{v\_\{\\parallel\},v\_\{0\}\\\}is

Ct=\[n\+K​t2​‖v∥‖22K​t2​‖v∥‖2​‖v0‖2K​t2​‖v∥‖2​‖v0‖2K​t2​‖v0‖22\]\.C\_\{t\}=\\begin\{bmatrix\}n\+Kt^\{2\}\\\|v\_\{\\parallel\}\\\|\_\{2\}^\{2\}&Kt^\{2\}\\\|v\_\{\\parallel\}\\\|\_\{2\}\\,\\\|v\_\{0\}\\\|\_\{2\}\\\\ Kt^\{2\}\\\|v\_\{\\parallel\}\\\|\_\{2\}\\,\\\|v\_\{0\}\\\|\_\{2\}&Kt^\{2\}\\\|v\_\{0\}\\\|\_\{2\}^\{2\}\\end\{bmatrix\}\.Indeed,Z∗⊤​Z∗Z\_\{\*\}^\{\\top\}Z\_\{\*\}contributesnnon thee1e\_\{1\}direction and0on thee0e\_\{0\}direction, whilev​v⊤=\(v∥\+v0\)​\(v∥\+v0\)⊤,vv^\{\\top\}=\(v\_\{\\parallel\}\+v\_\{0\}\)\(v\_\{\\parallel\}\+v\_\{0\}\)^\{\\top\}\\,,has the above2×22\\times 2coordinate matrix in the basis\(e1,e0\)\(e\_\{1\},e\_\{0\}\)\.

All remaining eigenvalues ofMt⊤​MtM\_\{t\}^\{\\top\}M\_\{t\}are unchanged: they are equal tonnon the orthogonal complement ofe1e\_\{1\}insiderow⁡\(Z∗\)\\operatorname\{row\}\(Z\_\{\*\}\), and equal to0on the orthogonal complement ofe0e\_\{0\}insideker⁡\(Z∗\)\\ker\(Z\_\{\*\}\)\. Thus the only nontrivial eigenvalues are those ofCtC\_\{t\}\.

Let0≤μ−​\(t\)≤μ\+​\(t\)0\\leq\\mu\_\{\-\}\(t\)\\leq\\mu\_\{\+\}\(t\)denote the eigenvalues of the positive semidefinite matrixCtC\_\{t\}\. Then,

μ−​\(t\)​μ\+​\(t\)=det\(Ct\)=\(n\+K​t2​‖v∥‖22\)​K​t2​‖v0‖22−K2​t4​‖v∥‖22​‖v0‖22=K​n​t2​‖v0‖22,\\mu\_\{\-\}\(t\)\\mu\_\{\+\}\(t\)=\\det\(C\_\{t\}\)=\\bigl\(n\+Kt^\{2\}\\\|v\_\{\\parallel\}\\\|\_\{2\}^\{2\}\\bigr\)Kt^\{2\}\\\|v\_\{0\}\\\|\_\{2\}^\{2\}\-K^\{2\}t^\{4\}\\\|v\_\{\\parallel\}\\\|\_\{2\}^\{2\}\\\|v\_\{0\}\\\|\_\{2\}^\{2\}=Knt^\{2\}\\\|v\_\{0\}\\\|\_\{2\}^\{2\}\\,,and

μ\+​\(t\)≤tr⁡\(Ct\)=n\+K​t2​‖v‖22\.\\mu\_\{\+\}\(t\)\\leq\\operatorname\{tr\}\(C\_\{t\}\)=n\+Kt^\{2\}\\\|v\\\|\_\{2\}^\{2\}\.Combining,

μ−​\(t\)≥K​n​t2​‖v0‖22n\+K​t2​‖v‖22\.\\mu\_\{\-\}\(t\)\\geq\\frac\{Knt^\{2\}\\\|v\_\{0\}\\\|\_\{2\}^\{2\}\}\{\\,n\+Kt^\{2\}\\\|v\\\|\_\{2\}^\{2\}\\,\}\.Moreover,

μ−​\(t\)≤tr⁡\(Ct\)/2=n/2\+K​t2​‖v‖2/2\.\\mu\_\{\-\}\(t\)\\leq\\operatorname\{tr\}\(C\_\{t\}\)/2=n/2\+Kt^\{2\}\\\|v\\\|^\{2\}/2\\,\.For all sufficiently smallt\>0t\>0, we haven\+K​t2​‖v‖22≤3​n/2,n\+Kt^\{2\}\\\|v\\\|\_\{2\}^\{2\}\\leq 3n/2,and hence

n\>3​n4≥μ−​\(t\)≥2​K3​‖v0‖22​t2\.n\>\\frac\{3n\}\{4\}\\geq\\mu\_\{\-\}\(t\)\\geq\\frac\{2K\}\{3\}\\\|v\_\{0\}\\\|\_\{2\}^\{2\}\\,t^\{2\}\.Thus, again, for all sufficiently smalltt,σK​\(Zk\)=μ−​\(tk\)\\sigma\_\{K\}\(Z\_\{k\}\)=\\sqrt\{\\mu\_\{\-\}\(t\_\{k\}\)\}and Eq\. \([23](https://arxiv.org/html/2605.23087#A4.E23)\) holds\.

*Subcase 2b:v0=0v\_\{0\}=0\.*

In this case,v=v∥∈row⁡\(Z∗\)v=v\_\{\\parallel\}\\in\\operatorname\{row\}\(Z\_\{\*\}\)\. Then

Mt⊤​Mt=Z∗⊤​Z∗\+K​t2​v​v⊤\.M\_\{t\}^\{\\top\}M\_\{t\}=Z\_\{\*\}^\{\\top\}Z\_\{\*\}\+Kt^\{2\}vv^\{\\top\}\.Sincevvlies in the row space ofZ∗Z\_\{\*\}, one eigenvalue ofZ∗⊤​Z∗Z\_\{\*\}^\{\\top\}Z\_\{\*\}equal tonnis perturbed to

n\+K​t2​‖v‖22,n\+Kt^\{2\}\\\|v\\\|\_\{2\}^\{2\},while all other nonzero eigenvalues remain equal tonn, and the zero eigenvalues remain unchanged\. In particular,σK​\(Mt\)=0\\sigma\_\{K\}\(M\_\{t\}\)=0, and therefore

f​\(Mt\)=fsmooth​\(Mt\)\.f\(M\_\{t\}\)=f\_\{\\mathrm\{smooth\}\}\(M\_\{t\}\)\.
Define

g¯​\(t\):=fsmooth​\(Z∗\+t​Δ¯\)=f​\(Mt\)\.\\bar\{g\}\(t\):=f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\+t\\bar\{\\Delta\}\)=f\(M\_\{t\}\)\.Then

g¯​\(t\)−g¯​\(0\)=\(L\+1\)​\[\(n\+K​t2​‖v‖22\)p/2−np/2\]\.\\bar\{g\}\(t\)\-\\bar\{g\}\(0\)=\(L\+1\)\\Bigl\[\\bigl\(n\+Kt^\{2\}\\\|v\\\|\_\{2\}^\{2\}\\bigr\)^\{p/2\}\-n^\{p/2\}\\Bigr\]\.A Taylor expansion att=0t=0yields

g¯​\(t\)−g¯​\(0\)=η​t2\+O​\(t4\),η:=np/2−1​K​‖v‖22\>0\.\\bar\{g\}\(t\)\-\\bar\{g\}\(0\)=\\eta t^\{2\}\+O\(t^\{4\}\),\\quad\\eta:=\\,n^\{p/2\-1\}K\\\|v\\\|\_\{2\}^\{2\}\>0\.Since also

g¯′​\(0\)=⟨G,Δ¯⟩=0,\\bar\{g\}^\{\\prime\}\(0\)=\\langle G,\\bar\{\\Delta\}\\rangle=0,it follows that

g¯′′​\(0\)=2​η\>0\.\\bar\{g\}^\{\\prime\\prime\}\(0\)=2\\eta\>0\.
Now define, for eachkk,

gk​\(t\):=fsmooth​\(Z∗\+t​Δk\)\.g\_\{k\}\(t\):=f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\+t\\Delta\_\{k\}\)\.By Taylor, there existssk∈\(0,tk\)s\_\{k\}\\in\(0,t\_\{k\}\)such that

fsmooth​\(Zk\)−fsmooth​\(Z∗\)=gk​\(tk\)−gk​\(0\)=tk​gk′​\(0\)\+tk22​gk′′​\(sk\)=tk​⟨G,Δk⟩\+tk22​gk′′​\(sk\)\.f\_\{\\mathrm\{smooth\}\}\(Z\_\{k\}\)\-f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)=g\_\{k\}\(t\_\{k\}\)\-g\_\{k\}\(0\)=t\_\{k\}g\_\{k\}^\{\\prime\}\(0\)\+\\frac\{t\_\{k\}^\{2\}\}\{2\}g\_\{k\}^\{\\prime\\prime\}\(s\_\{k\}\)=t\_\{k\}\\langle G,\\Delta\_\{k\}\\rangle\+\\frac\{t\_\{k\}^\{2\}\}\{2\}g\_\{k\}^\{\\prime\\prime\}\(s\_\{k\}\)\\,\.Here,

gk′′​\(t\)=∇2fsmooth​\(Z∗\+t​Δk\)​\[Δk,Δk\]\.g\_\{k\}^\{\\prime\\prime\}\(t\)=\\nabla^\{2\}f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\+t\\Delta\_\{k\}\)\[\\Delta\_\{k\},\\Delta\_\{k\}\]\.Becausesk→0s\_\{k\}\\to 0andΔk→Δ¯\\Delta\_\{k\}\\to\\bar\{\\Delta\}, and because∇2fsmooth\\nabla^\{2\}f\_\{\\mathrm\{smooth\}\}is continuous, we obtain

gk′′​\(sk\)→∇2fsmooth​\(Z∗\)​\[Δ¯,Δ¯\]=g¯′′​\(0\)=2​η\.g\_\{k\}^\{\\prime\\prime\}\(s\_\{k\}\)\\to\\nabla^\{2\}f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)\[\\bar\{\\Delta\},\\bar\{\\Delta\}\]=\\bar\{g\}^\{\\prime\\prime\}\(0\)=2\\eta\.Hence, for all sufficiently largekk,

gk′′​\(sk\)≥η\.g\_\{k\}^\{\\prime\\prime\}\(s\_\{k\}\)\\geq\\eta\.Using also⟨G,Δk⟩≥0\\langle G,\\Delta\_\{k\}\\rangle\\geq 0, we conclude that

fsmooth​\(Zk\)−fsmooth​\(Z∗\)≥η2​tk2\>0f\_\{\\mathrm\{smooth\}\}\(Z\_\{k\}\)\-f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\)\\geq\\frac\{\\eta\}\{2\}t\_\{k\}^\{2\}\>0for all sufficiently largekk\.

Sincef​\(Zk\)≥fsmooth​\(Zk\)f\(Z\_\{k\}\)\\geq f\_\{\\mathrm\{smooth\}\}\(Z\_\{k\}\)andf​\(Z∗\)=fsmooth​\(Z∗\),f\(Z\_\{\*\}\)=f\_\{\\mathrm\{smooth\}\}\(Z\_\{\*\}\),it follows that

f​\(Zk\)−f​\(Z∗\)\>0,f\(Z\_\{k\}\)\-f\(Z\_\{\*\}\)\>0,contradicting the assumptionf​\(Zk\)≤f​\(Z∗\)f\(Z\_\{k\}\)\\leq f\(Z\_\{\*\}\)\.

Thus both subcases lead to a contradiction, and Case 2 cannot occur\.

### D\.2Proof of Theorem[3\.2](https://arxiv.org/html/2605.23087#S3.Thmtheorem2)

Recall the constrained optimization problem stated in Eq\. \([3](https://arxiv.org/html/2605.23087#S3.E3)\)\. Since here we are only interested in the global minimum of the constrained optimization problem, we can apply Lemma[D\.7](https://arxiv.org/html/2605.23087#A4.Thmtheorem7)to reduce to the following alternative problem that shares its global minimizer

minZ⁡\{‖Z‖S2L\+12L\+1\}s\.t\.​\(zi​c\)c−\(zi​c\)c′≥1,c′≠c=1,…,K,i=1,…,n\.\\min\_\{Z\}\\left\\\{\\\|Z\\\|\_\{S\_\{\\frac\{2\}\{L\+1\}\}\}^\{\\frac\{2\}\{L\+1\}\}\\right\\\}\\quad\\textrm\{s\.t\. \}\(z\_\{ic\}\)\_\{c\}\-\(z\_\{ic\}\)\_\{c^\{\\prime\}\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K,\\quad i=1,\.\.\.,n\.\(24\)Where the Schatten quasi norm is defined in Eq\. \([59](https://arxiv.org/html/2605.23087#A4.E59)\)\. We now note that we assume the NC1 property, and soZ=Z′⊗1nTZ=Z^\{\\prime\}\\otimes 1\_\{n\}^\{T\}for someZ′∈ℝK×KZ^\{\\prime\}\\in\\mathbb\{R\}^\{K\\times K\}\. SinceZZandZ′Z^\{\\prime\}have the same singular values up to a scale ofn\\sqrt\{n\}, we can further reduce the problem in this case to

minZ′⁡\{‖Z′‖S2L\+12L\+1\}s\.t\.​Zc​c′−Zc′​c′≥1,c′≠c=1,…,K\.\\min\_\{Z^\{\\prime\}\}\\left\\\{\\\|Z^\{\\prime\}\\\|\_\{S\_\{\\frac\{2\}\{L\+1\}\}\}^\{\\frac\{2\}\{L\+1\}\}\\right\\\}\\quad\\textrm\{s\.t\. \}Z\_\{cc\}^\{\\prime\}\-Z\_\{c^\{\\prime\}c\}^\{\\prime\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K\.\(25\)Now expand the Schatten\-quasi norm term in theL→∞L\\to\\inftylimit

‖Z′‖S2L\+12L\+1=∑i=1rank​\(Z′\)σi2L\+1=∑i=1rank​\(Z′\)e2L\+1​log⁡\(σi\)\\\|Z^\{\\prime\}\\\|\_\{S\_\{\\frac\{2\}\{L\+1\}\}\}^\{\\frac\{2\}\{L\+1\}\}=\\sum\_\{i=1\}^\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}\\sigma\_\{i\}^\{\\frac\{2\}\{L\+1\}\}=\\sum\_\{i=1\}^\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}e^\{\\frac\{2\}\{L\+1\}\\log\(\\sigma\_\{i\}\)\}=rank​\(Z′\)\+2L\+1​∑i=1rank​\(Z′\)log⁡\(σi\)\+O​\(1L2\),=\\textrm\{rank\}\(Z^\{\\prime\}\)\+\\frac\{2\}\{L\+1\}\\sum\_\{i=1\}^\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}\\log\(\\sigma\_\{i\}\)\+O\\left\(\\frac\{1\}\{L^\{2\}\}\\right\),whereσ1,…,σrank​\(Z′\)\\sigma\_\{1\},\.\.\.,\\sigma\_\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}are the non\-zero singular values ofZ′Z^\{\\prime\}\. Hence we have the objective of Eq\. \([25](https://arxiv.org/html/2605.23087#A4.E25)\) becomes

minZ′⁡\{rank​\(Z′\)\+2L\+1​∑i=1rank​\(Z′\)log⁡\(σi\)\+O​\(1L2\)\}s\.t\.​Zc​c′−Zc′​c′≥1,c′≠c=1,…,K\.\\min\_\{Z^\{\\prime\}\}\\left\\\{\\textrm\{rank\}\(Z^\{\\prime\}\)\+\\frac\{2\}\{L\+1\}\\sum\_\{i=1\}^\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}\\log\(\\sigma\_\{i\}\)\+O\\left\(\\frac\{1\}\{L^\{2\}\}\\right\)\\right\\\}\\quad\\textrm\{s\.t\. \}Z\_\{cc\}^\{\\prime\}\-Z\_\{c^\{\\prime\}c\}^\{\\prime\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K\.\(26\)The global minimum in theL→∞L\\to\\inftylimit must be minimal rank such that the constraint is satisfied\. We first remark that the constraint can be satisfied by matrices of rank two\. The construction in the theorem statement suffices, since ifZ′=β​XT​XZ^\{\\prime\}=\\beta X^\{T\}X, whereX∈ℝ2×KX\\in\\mathbb\{R\}^\{2\\times K\}has each column vector of unit norm, and no vector repeated thenZc​c′=β​xcT​xc=βZ\_\{cc\}^\{\\prime\}=\\beta x\_\{c\}^\{T\}x\_\{c\}=\\beta,Zc′​c′=β​xc′T​xc<βZ\_\{c^\{\\prime\}c\}^\{\\prime\}=\\beta x\_\{c^\{\\prime\}\}^\{T\}x\_\{c\}<\\beta, and then settingβ\>maxc,c′≠c\(1−xc′Txc\)−1\\beta\>\\max\_\{c,c^\{\\prime\}\\neq c\}\(1\-x\_\{c^\{\\prime\}\}^\{T\}x\_\{c\}\)^\{\-1\}produces a rank two matrix that satisfies the constraint\.

We also quote Lemma[D\.2](https://arxiv.org/html/2605.23087#A4.Thmtheorem2), which states that, forK\>2K\>2,Z′Z^\{\\prime\}can only satisfy the condition ifrank​\(Z′\)≥2\\textrm\{rank\}\(Z^\{\\prime\}\)\\geq 2, hence the minimal rank in our setting must be exactly two, and we reduce our objective value from Eq\. \([26](https://arxiv.org/html/2605.23087#A4.E26)\) to

minZ′⁡\{2L\+1​∑i=1rank​\(Z′\)log⁡\(σi\)\+O​\(1L2\)\}s\.t\.​Zc​c′−Zc′​c′≥1,c′≠c=1,…,K,andrank​\(Z′\)=2\.\\min\_\{Z^\{\\prime\}\}\\left\\\{\\frac\{2\}\{L\+1\}\\sum\_\{i=1\}^\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}\\log\(\\sigma\_\{i\}\)\+O\\left\(\\frac\{1\}\{L^\{2\}\}\\right\)\\right\\\}\\quad\\textrm\{s\.t\. \}Z\_\{cc\}^\{\\prime\}\-Z\_\{c^\{\\prime\}c\}^\{\\prime\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K,\\textrm\{ and \}\\textrm\{rank\}\(Z^\{\\prime\}\)=2\.We now use the technical assumption thatZ′Z^\{\\prime\}is positive semi\-definite, so that we can writeZ′=XT​XZ^\{\\prime\}=X^\{T\}XforX∈ℝ2×KX\\in\\mathbb\{R\}^\{2\\times K\}\. Noting that minimizing∑i=1rank​\(Z′\)log⁡\(σi\)\\sum\_\{i=1\}^\{\\textrm\{rank\}\(Z^\{\\prime\}\)\}\\log\(\\sigma\_\{i\}\)is the same as minimizingσ1​σ2=det​\(X​XT\)\\sigma\_\{1\}\\sigma\_\{2\}=\\textrm\{det\}\(XX^\{T\}\), and thatZc′​c′=xc′T​xcZ^\{\\prime\}\_\{c^\{\\prime\}c\}=x\_\{c^\{\\prime\}\}^\{T\}x\_\{c\},this reduces for largeLLto

minX∈ℝ2×K⁡\{det​\(X​XT\)\}s\.t\.​xcT​xc−xc′T​xc≥1,c′≠c=1,…,K,\\min\_\{X\\in\\mathbb\{R\}^\{2\\times K\}\}\\left\\\{\\textrm\{det\}\(XX^\{T\}\)\\right\\\}\\quad\\textrm\{s\.t\. \}x\_\{c\}^\{T\}x\_\{c\}\-x\_\{c^\{\\prime\}\}^\{T\}x\_\{c\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K,or stated entirely in terms of the column vectors

minxi∈ℝ2,i=1,…,K⁡\{det​\(∑ixi​xiT\)\}s\.t\.​xcT​xc−xc′T​xc≥1,c′≠c=1,…,K\.\\min\_\{x\_\{i\}\\in\\mathbb\{R\}^\{2\},i=1,\.\.\.,K\}\\left\\\{\\textrm\{det\}\\left\(\\sum\_\{i\}x\_\{i\}x\_\{i\}^\{T\}\\right\)\\right\\\}\\quad\\textrm\{s\.t\. \}x\_\{c\}^\{T\}x\_\{c\}\-x\_\{c^\{\\prime\}\}^\{T\}x\_\{c\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K\.\(27\)We now use that the features have equal norm, meaning‖xi‖=r\\\|x\_\{i\}\\\|=rfor alli=1,…,Ki=1,\.\.\.,K\. Writingxi=\(r​cos⁡\(θi\),r​sin⁡\(θi\)\)x\_\{i\}=\(r\\cos\(\\theta\_\{i\}\),r\\sin\(\\theta\_\{i\}\)\), and using the shorthandci=cos⁡\(θi\),si=sin⁡\(θi\)c\_\{i\}=\\cos\(\\theta\_\{i\}\),s\_\{i\}=\\sin\(\\theta\_\{i\}\)we have:

∑ixi​xiT=∑i\[r​cir​si\]​\[r​cir​si\]=∑i\[r2​ci2r2​ci​sir2​ci​sir2​si2\],\\sum\_\{i\}x\_\{i\}x\_\{i\}^\{T\}=\\sum\_\{i\}\\begin\{bmatrix\}rc\_\{i\}&rs\_\{i\}\\end\{bmatrix\}\\begin\{bmatrix\}rc\_\{i\}\\\\ rs\_\{i\}\\end\{bmatrix\}=\\sum\_\{i\}\\begin\{bmatrix\}r^\{2\}c\_\{i\}^\{2\}&r^\{2\}c\_\{i\}s\_\{i\}\\\\ r^\{2\}c\_\{i\}s\_\{i\}&r^\{2\}s\_\{i\}^\{2\}\\end\{bmatrix\},and hence

det​\(∑ixi​xiT\)=r4​∑i,j=1K\[ci2​sj2−ci​cj​si​sj\]\.\\textrm\{det\}\\left\(\\sum\_\{i\}x\_\{i\}x\_\{i\}^\{T\}\\right\)=r^\{4\}\\sum\_\{i,j=1\}^\{K\}\[c\_\{i\}^\{2\}s\_\{j\}^\{2\}\-c\_\{i\}c\_\{j\}s\_\{i\}s\_\{j\}\]\.Note that the two terms cancel wheni=ji=j\. Combining the terms\(i,j\)\(i,j\)and\(j,i\)\(j,i\), this then becomes:

r4​∑i<j\[ci2​sj2\+cj2​si2−2​ci​cj​si​sj\]=r4​∑i<j\(ci​sj−cj​si\)2,r^\{4\}\\sum\_\{i<j\}\[c\_\{i\}^\{2\}s\_\{j\}^\{2\}\+c\_\{j\}^\{2\}s\_\{i\}^\{2\}\-2c\_\{i\}c\_\{j\}s\_\{i\}s\_\{j\}\]=r^\{4\}\\sum\_\{i<j\}\(c\_\{i\}s\_\{j\}\-c\_\{j\}s\_\{i\}\)^\{2\},and hence

det​\(∑ixi​xiT\)=r4​∑i<jsin2⁡\(θi−θj\)\.\\textrm\{det\}\\left\(\\sum\_\{i\}x\_\{i\}x\_\{i\}^\{T\}\\right\)=r^\{4\}\\sum\_\{i<j\}\\sin^\{2\}\(\\theta\_\{i\}\-\\theta\_\{j\}\)\.The constraint of Eq\. \([27](https://arxiv.org/html/2605.23087#A4.E27)\) becomes

xiT​xi−xiT​xj=r2−r2​\[ci​cj\+si​sj\]=r2​\(1−cos⁡\(θi−θj\)\)≥1,x\_\{i\}^\{T\}x\_\{i\}\-x\_\{i\}^\{T\}x\_\{j\}=r^\{2\}\-r^\{2\}\[c\_\{i\}c\_\{j\}\+s\_\{i\}s\_\{j\}\]=r^\{2\}\(1\-\\cos\(\\theta\_\{i\}\-\\theta\_\{j\}\)\)\\geq 1,so the optimization problem of Eq\. \([27](https://arxiv.org/html/2605.23087#A4.E27)\) has reduced to

minr,θ1,…,θK⁡\{r4​∑i<jsin2⁡\(θi−θj\)\}s\.t\.​r2​\(1−cos⁡\(θi−θj\)\)≥1,j≠i=1,…,K\.\\min\_\{r,\\theta\_\{1\},\.\.\.,\\theta\_\{K\}\}\\left\\\{r^\{4\}\\sum\_\{i<j\}\\sin^\{2\}\(\\theta\_\{i\}\-\\theta\_\{j\}\)\\right\\\}\\quad\\textrm\{s\.t\. \}r^\{2\}\(1\-\\cos\(\\theta\_\{i\}\-\\theta\_\{j\}\)\)\\geq 1,\\quad j\\neq i=1,\.\.\.,K\.\(28\)Note that our set of constraints can only be satisfied when

r2≥maxi,j≠i\(1−cos\(θi−θj\)\)−1\.r^\{2\}\\geq\\max\_\{i,j\\neq i\}\(1\-\\cos\(\\theta\_\{i\}\-\\theta\_\{j\}\)\)^\{\-1\}\.Letδ=mini,j≠i⁡\{\|θi−θj\|\}\\delta=\\min\_\{i,j\\neq i\}\\left\\\{\|\\theta\_\{i\}\-\\theta\_\{j\}\|\\right\\\}, where the angular difference is measured modulo2​π2\\pi, so as to capture the wrap around value as well\. By the pigeonhole principleδ≤2​πK\\delta\\leq\\frac\{2\\pi\}\{K\}, andδ\>0\\delta\>0else the constraint is not satisfied\. Given thatK\>2K\>2, we have our constraints satisfied when

r2≥\(1−cos⁡\(δ\)\)−1\.r^\{2\}\\geq\(1\-\\cos\(\\delta\)\)^\{\-1\}\.Since the objective is monotonic increasing inrr, when considering global minima we can setrrto this minimal value, and our objective of Eq\. \([28](https://arxiv.org/html/2605.23087#A4.E28)\) is now

minθ1,…,θK⁡\{∑i<jsin2⁡\(θi−θj\)\(1−cos⁡\(δ\)\)2\}s\.t\.​δ=mini,j≠i⁡\{\|θi−θj\|\}\.\\min\_\{\\theta\_\{1\},\.\.\.,\\theta\_\{K\}\}\\left\\\{\\frac\{\\sum\_\{i<j\}\\sin^\{2\}\(\\theta\_\{i\}\-\\theta\_\{j\}\)\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\}\\quad\\textrm\{s\.t\. \}\\delta=\\min\_\{i,j\\neq i\}\\\{\|\\theta\_\{i\}\-\\theta\_\{j\}\|\\\}\.\(29\)Note that this has a rotational symmetry,θi→θi\+α\\theta\_\{i\}\\rightarrow\\theta\_\{i\}\+\\alphaleaves the optimal value unchanged\.

We now aim to further simplify the expression in the numerator\. using that for any angleϕ\\phiwe have2​sin2⁡\(ϕ\)=1−cos⁡\(2​ϕ\)2\\sin^\{2\}\(\\phi\)=1\-\\cos\(2\\phi\), we can rewrite the objective as

14​K​\(K−1\)−12​∑i<j\(cos⁡2​\(θi−θj\)\)\(1−cos⁡\(δ\)\)2\.\\frac\{\\frac\{1\}\{4\}K\(K\-1\)\-\\frac\{1\}\{2\}\\sum\_\{i<j\}\(\\cos 2\(\\theta\_\{i\}\-\\theta\_\{j\}\)\)\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\}\.Next use that

cos2\(θu−θv\)\)=Re\(e2i\(θu−θv\)=Re\(zuz¯v\),\\cos 2\(\\theta\_\{u\}\-\\theta\_\{v\}\)\)=\\textrm\{Re\}\(e^\{2i\(\\theta\_\{u\}\-\\theta\_\{v\}\}\)=\\textrm\{Re\}\(z\_\{u\}\\bar\{z\}\_\{v\}\),wherezu=e2​i​θuz\_\{u\}=e^\{2i\\theta\_\{u\}\},Redenotes the real part, and a bar denotes the complex conjugate\. Also note that

\|∑uzu\|2=∑u,vzu​z¯v=K\+∑u<v\(zu​z¯v\+zv​z¯u\)=K\+2​∑u<vRe​\(zu​z¯v\),\\left\|\\sum\_\{u\}z\_\{u\}\\right\|^\{2\}=\\sum\_\{u,v\}z\_\{u\}\\bar\{z\}\_\{v\}=K\+\\sum\_\{u<v\}\(z\_\{u\}\\bar\{z\}\_\{v\}\+z\_\{v\}\\bar\{z\}\_\{u\}\)=K\+2\\sum\_\{u<v\}\\textrm\{Re\}\(z\_\{u\}\\bar\{z\}\_\{v\}\),and hence we can write the objective as

14​K2−14​\|∑ue2​i​θu\|2\(1−cos⁡\(δ\)\)2\.\\frac\{\\frac\{1\}\{4\}K^\{2\}\-\\frac\{1\}\{4\}\|\\sum\_\{u\}e^\{2i\\theta\_\{u\}\}\|^\{2\}\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\}\.We now use the rotational symmetry, choosing to transformθi→θi\+α\\theta\_\{i\}\\rightarrow\\theta\_\{i\}\+\\alpha, whereα\\alphais such that

∑usin⁡\(2​θu\+2​α\)=0\.\\sum\_\{u\}\\sin\(2\\theta\_\{u\}\+2\\alpha\)=0\.Doing this, and redefiningθu\\theta\_\{u\}, we have the reduction:

\|∑ue2​i​θu\|2=\[∑ucos⁡\(2​θu\)\]2,\\left\|\\sum\_\{u\}e^\{2i\\theta\_\{u\}\}\\right\|^\{2\}=\\left\[\\sum\_\{u\}\\cos\(2\\theta\_\{u\}\)\\right\]^\{2\},and so our objective of Eq\. \([29](https://arxiv.org/html/2605.23087#A4.E29)\) is now:

minθ1,…,θK⁡\{14​K2−14​\[∑ucos⁡\(2​θu\)\]2\(1−cos⁡\(δ\)\)2\},\\min\_\{\\theta\_\{1\},\.\.\.,\\theta\_\{K\}\}\\left\\\{\\frac\{\\frac\{1\}\{4\}K^\{2\}\-\\frac\{1\}\{4\}\\left\[\\sum\_\{u\}\\cos\(2\\theta\_\{u\}\)\\right\]^\{2\}\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\},\(30\)with the constraint that the gaps are at leastδ\\delta\.

We reduce this to a problem solely inδ\\deltaby minimizing the numerator subject to the condition\. This is equivalent to:

maxθ1,…,θk⁡\{∑ucos⁡\(2​θu\)\},\\max\_\{\\theta\_\{1\},\.\.\.,\\theta\_\{k\}\}\\left\\\{\\sum\_\{u\}\\cos\(2\\theta\_\{u\}\)\\right\\\},\(31\)subject to the smallest gap beingδ\\delta\. Here we use Lemma[D\.3](https://arxiv.org/html/2605.23087#A4.Thmtheorem3), which states that forKKeven, orKKodd andδ∈\(0,2​πK\+1\]\\delta\\in\(0,\\frac\{2\\pi\}\{K\+1\}\], the maximum value of this sum, subject to the constraint, is given by

sin⁡\(⌈K2⌉​δ\)\+sin⁡\(⌊K2⌋​δ\)sin⁡\(δ\)\.\\frac\{\\sin\\left\(\\lceil\\frac\{K\}\{2\}\\rceil\\delta\\right\)\+\\sin\\left\(\\lfloor\\frac\{K\}\{2\}\\rfloor\\delta\\right\)\}\{\\sin\(\\delta\)\}\.We will return to the oddKK,δ∈\(2​πK\+1,2​πK\]\\delta\\in\(\\frac\{2\\pi\}\{K\+1\},\\frac\{2\\pi\}\{K\}\]case later\.

Our full optimization problem of Eq\. \([30](https://arxiv.org/html/2605.23087#A4.E30)\) becomes:

minδ∈\(0,2​πK\]⁡\{K2​sin2⁡\(δ\)−\(sin⁡\(⌈K2⌉​δ\)\+sin⁡\(⌊K2⌋​δ\)\)24​\(sin⁡\(δ\)\)2​\(1−cos⁡\(δ\)\)2\}\.\\min\_\{\\delta\\in\(0,\\frac\{2\\pi\}\{K\}\]\}\\left\\\{\\frac\{K^\{2\}\\sin^\{2\}\(\\delta\)\-\(\\sin\\left\(\\lceil\\frac\{K\}\{2\}\\rceil\\delta\\right\)\+\\sin\\left\(\\lfloor\\frac\{K\}\{2\}\\rfloor\\delta\\right\)\)^\{2\}\}\{4\(\\sin\(\\delta\)\)^\{2\}\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\}\.\(32\)From here we focus on the case whereK=2​mK=2mfor some integermm, the proof for the odd case for this objective is similar\. The goal is to show this is minimized atδ=2​πK\\delta=\\frac\{2\\pi\}\{K\}, and hence the vectors are uniformly spaced on a circle\. WritingK=2​mK=2m, we have

minδ∈\(0,πm\]⁡\{m2​sin2⁡\(δ\)−sin2⁡\(m​δ\)sin2⁡\(δ\)​\(1−cos⁡\(δ\)\)2\}\.\\min\_\{\\delta\\in\(0,\\frac\{\\pi\}\{m\}\]\}\\left\\\{\\frac\{m^\{2\}\\sin^\{2\}\(\\delta\)\-\\sin^\{2\}\\left\(m\\delta\\right\)\}\{\\sin^\{2\}\(\\delta\)\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\}\.\(33\)Define the function

F​\(δ\)=sin⁡\(m​δ\)sin⁡\(δ\)=ei​m​δ−e−i​m​δei​δ−ei​δ=∑u=0m−1e\(2​u−\(m−1\)\)​i​δ\.F\(\\delta\)=\\frac\{\\sin\(m\\delta\)\}\{\\sin\(\\delta\)\}=\\frac\{e^\{im\\delta\}\-e^\{\-im\\delta\}\}\{e^\{i\\delta\}\-e^\{i\\delta\}\}=\\sum\_\{u=0\}^\{m\-1\}e^\{\(2u\-\(m\-1\)\)i\\delta\}\.Taking a square

F2​\(δ\)=∑u,v=0m−1e−2​\(m−1\)​i​δ​e\(2​u\+2​v\)​i​δ=m\+2​∑u=1m−1\(m−u\)​cos⁡\(2​u​δ\)\.F^\{2\}\(\\delta\)=\\sum\_\{u,v=0\}^\{m\-1\}e^\{\-2\(m\-1\)i\\delta\}e^\{\(2u\+2v\)i\\delta\}=m\+2\\sum\_\{u=1\}^\{m\-1\}\(m\-u\)\\cos\(2u\\delta\)\.So we can rewrite our objective as

minδ∈\(0,πm\]⁡\{m2−m−2​∑u=1m−1\(m−u\)​cos⁡\(2​u​δ\)\(1−cos⁡\(δ\)\)2\}\.\\min\_\{\\delta\\in\(0,\\frac\{\\pi\}\{m\}\]\}\\left\\\{\\frac\{m^\{2\}\-m\-2\\sum\_\{u=1\}^\{m\-1\}\(m\-u\)\\cos\(2u\\delta\)\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\}\.\(34\)Now use that

2​∑j=1m−1\(m−j\)=m2−m,2\\sum\_\{j=1\}^\{m\-1\}\(m\-j\)=m^\{2\}\-m,to further reduce to

minδ∈\(0,πm\]⁡\{2​∑u=1m−1\(m−u\)​\(1−cos⁡\(2​u​δ\)\)\(1−cos⁡\(δ\)\)2\}\.\\min\_\{\\delta\\in\(0,\\frac\{\\pi\}\{m\}\]\}\\left\\\{2\\sum\_\{u=1\}^\{m\-1\}\(m\-u\)\\frac\{\(1\-\\cos\(2u\\delta\)\)\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\}\.It is clear that it is sufficient for the objective to be monotonic decreasing on its domain, and a sufficient condition for this to occur is that each of the functions:

fu\(δ\)=1−cos⁡\(2​u​δ\)\(1−cos⁡\(δ\)\)2,u=1,\.\.,m−1,f\_\{u\}\(\\delta\)=\\frac\{1\-\\cos\(2u\\delta\)\}\{\(1\-\\cos\(\\delta\)\)^\{2\}\},\\quad u=1,\.\.,m\-1,\(35\)is monotonic decreasing on the range\(0,πm\]\(0,\\frac\{\\pi\}\{m\}\]\.

Using standard trigonometric identities, and definingδ=2​t\\delta=2t, this reduces to

fu​\(t\)=2​sin2⁡\(2​u​t\)4​sin4⁡\(t\),u=1,…,m−1,t∈\(0,π2​m\]\.f\_\{u\}\(t\)=\\frac\{2\\sin^\{2\}\(2ut\)\}\{4\\sin^\{4\}\(t\)\},\\quad u=1,\.\.\.,m\-1,\\quad t\\in\\left\(0,\\frac\{\\pi\}\{2m\}\\right\]\.Dropping a multiplicative factor, and using a square root, which doesn’t change monotonicity, we can redefine our functions to

fu​\(t\)=sin⁡\(2​u​t\)sin2⁡\(t\),u=1,…,m−1,t∈\(0,π2​m\]\.f\_\{u\}\(t\)=\\frac\{\\sin\(2ut\)\}\{\\sin^\{2\}\(t\)\},\\quad u=1,\.\.\.,m\-1,\\quad t\\in\\left\(0,\\frac\{\\pi\}\{2m\}\\right\]\.Taking a derivative, the condition for monotonic decreasing is equivalent to

u​cot⁡\(2​u​t\)<cot⁡\(t\),u=1,…,m−1,t∈\(0,π2​m\]\.u\\cot\(2ut\)<\\cot\(t\),\\quad u=1,\.\.\.,m\-1,\\quad t\\in\\left\(0,\\frac\{\\pi\}\{2m\}\\right\]\.Note the right hand side is always positive on the range, so this inequality holds trivially if the left hand side is negative\. This occurs if2​u​t\>π/22ut\>\\pi/2\. Otherwise, we use the fact that the functionx​cot⁡\(x\)x\\cot\(x\)is strictly decreasing onx∈\(0,π/2\]x\\in\(0,\\pi/2\], and hence, since2​u​t\>t2ut\>t, we have

2​u​t​cot⁡\(2​u​t\)<t​cot⁡\(t\),2ut\\cot\(2ut\)<t\\cot\(t\),so that

u​cot⁡\(2​u​t\)≤2​u​cot⁡\(2​u​t\)<t​cot⁡\(t\),u\\cot\(2ut\)\\leq 2u\\cot\(2ut\)<t\\cot\(t\),So each of these functions is monotonic decreasing on the given range\.

Hence our objective is monotonic decreasing in the minimum gapδ\\delta, and so the best objective occurs whenδ\\deltais maximized, which corresponds with the Gram vectorsx1,…,xKx\_\{1\},\.\.\.,x\_\{K\}being uniformly spaced on a circle in some order\.

The remaining sub\-case isKKodd, andδ∈\(2​πK\+1,2​πK\]\\delta\\in\(\\frac\{2\\pi\}\{K\+1\},\\frac\{2\\pi\}\{K\}\]\. Lemma[D\.3](https://arxiv.org/html/2605.23087#A4.Thmtheorem3)also provides the maximum of the cosine sum in Eq\. \([31](https://arxiv.org/html/2605.23087#A4.E31)\) in this case, giving

maxθ1,…,θk​s\.t\. gaps≥δ⁡\{∑ucos⁡\(2​θu\)\}=\|sin⁡\(K​δ\)\|sin⁡\(δ\)\.\\max\_\{\\theta\_\{1\},\.\.\.,\\theta\_\{k\}\\textrm\{ s\.t\. gaps \}\\geq\\delta\}\\left\\\{\\sum\_\{u\}\\cos\(2\\theta\_\{u\}\)\\right\\\}=\\frac\{\|\\sin\(K\\delta\)\|\}\{\\sin\(\\delta\)\}\.Following the same steps above, it only remains to prove that the following objective

minδ∈\(2​πK\+1,2​πK\]⁡\{K2−sin2⁡\(K​δ\)sin⁡\(δ\)4​\(1−cos⁡\(δ\)\)2\},\\min\_\{\\delta\\in\(\\frac\{2\\pi\}\{K\+1\},\\frac\{2\\pi\}\{K\}\]\}\\left\\\{\\frac\{K^\{2\}\-\\frac\{\\sin^\{2\}\(K\\delta\)\}\{\\sin\(\\delta\)\}\}\{4\(1\-\\cos\(\\delta\)\)^\{2\}\}\\right\\\},\(36\)takes its minimal value atδ=2​πK\\delta=\\frac\{2\\pi\}\{K\}\.

To show this, writeu​\(δ\)=sin⁡\(K​δ\)/sin⁡\(δ\)u\(\\delta\)=\\sin\(K\\delta\)/\\sin\(\\delta\)and denote the objective function in Eq\. \([36](https://arxiv.org/html/2605.23087#A4.E36)\) byf​\(δ\)f\(\\delta\)\. Also, letγ=2​π−K​δ\\gamma=2\\pi\-K\\delta, so thatγ∈\[0,2​πK\+1\)\\gamma\\in\[0,\\frac\{2\\pi\}\{K\+1\}\)\. We note thatγ≤δ\\gamma\\leq\\delta, andδ\+γ≤4​πK\+1≤π\\delta\+\\gamma\\leq\\frac\{4\\pi\}\{K\+1\}\\leq\\pi\. Hence

0≤γ≤δ≤π−γ⟹sin⁡\(γ\)≤sin⁡\(δ\)\.0\\leq\\gamma\\leq\\delta\\leq\\pi\-\\gamma\\implies\\sin\(\\gamma\)\\leq\\sin\(\\delta\)\.and so

u​\(δ\)=−sin⁡\(γ\)sin⁡\(δ\)∈\[−1,0\],u\(\\delta\)=\\frac\{\-\\sin\(\\gamma\)\}\{\\sin\(\\delta\)\}\\in\[\-1,0\],and

f′​\(δ\)f​\(δ\)=−2​u​u′K2−u2−2​sin⁡\(δ\)1−cos⁡\(δ\)\.\\frac\{f^\{\\prime\}\(\\delta\)\}\{f\(\\delta\)\}=\\frac\{\-2uu^\{\\prime\}\}\{K^\{2\}\-u^\{2\}\}\-\\frac\{2\\sin\(\\delta\)\}\{1\-\\cos\(\\delta\)\}\.Note since−u∈\[0,1\]\-u\\in\[0,1\], we haveK2−u2≥K2−1K^\{2\}\-u^\{2\}\\geq K^\{2\}\-1\. Also

u′​\(δ\)=1sin2⁡\(δ\)​\[K​cos⁡\(K​δ\)​sin⁡\(δ\)−sin⁡\(K​δ\)​cos⁡\(δ\)\]\.u^\{\\prime\}\(\\delta\)=\\frac\{1\}\{\\sin^\{2\}\(\\delta\)\}\[K\\cos\(K\\delta\)\\sin\(\\delta\)\-\\sin\(K\\delta\)\\cos\(\\delta\)\]\.We then use thatcos⁡\(K​δ\)≤1\\cos\(K\\delta\)\\leq 1,−sin⁡\(K​δ\)≤sin⁡\(δ\)\-\\sin\(K\\delta\)\\leq\\sin\(\\delta\)andcos⁡\(δ\)≤1\\cos\(\\delta\)\\leq 1, which gives

u′​\(δ\)≤K​sin⁡\(δ\)\+sin⁡\(δ\)sin2⁡\(δ\)=K\+1sin⁡\(δ\)\.u^\{\\prime\}\(\\delta\)\\leq\\frac\{K\\sin\(\\delta\)\+\\sin\(\\delta\)\}\{\\sin^\{2\}\(\\delta\)\}=\\frac\{K\+1\}\{\\sin\(\\delta\)\}\.Also note that sinceδ≤2​πK≤2​π3\\delta\\leq\\frac\{2\\pi\}\{K\}\\leq\\frac\{2\\pi\}\{3\}, we have

sin⁡\(δ\)1−cos⁡\(δ\)=1\+cos⁡\(δ\)sin⁡\(δ\)≥12​sin⁡\(δ\),\\frac\{\\sin\(\\delta\)\}\{1\-\\cos\(\\delta\)\}=\\frac\{1\+\\cos\(\\delta\)\}\{\\sin\(\\delta\)\}\\geq\\frac\{1\}\{2\\sin\(\\delta\)\},since the inequality reduces to1\+2​cos⁡\(δ\)≥01\+2\\cos\(\\delta\)\\geq 0, which is true on our range ofδ\\delta\.

Also note it is the case thatK2=1≥2​K\+2K^\{2\}=1\\geq 2K\+2forK≥3K\\geq 3\. Combining all of this gives

−2​u​u′K2−u2≤2​u′K2−1≤2​\(K\+1\)\(K2−1\)​sin⁡\(δ\)≤1sin⁡\(δ\)≤2​sin⁡\(δ\)1−cos⁡\(δ\),\\frac\{\-2uu^\{\\prime\}\}\{K^\{2\}\-u^\{2\}\}\\leq\\frac\{2u^\{\\prime\}\}\{K^\{2\}\-1\}\\leq\\frac\{2\(K\+1\)\}\{\(K^\{2\}\-1\)\\sin\(\\delta\)\}\\leq\\frac\{1\}\{\\sin\(\\delta\)\}\\leq\\frac\{2\\sin\(\\delta\)\}\{1\-\\cos\(\\delta\)\},and hence

f′​\(δ\)f​\(δ\)≤0\.\\frac\{f^\{\\prime\}\(\\delta\)\}\{f\(\\delta\)\}\\leq 0\.Finally, noting thatf​\(δ\)f\(\\delta\)is always greater than zero gives the result thatffis decreasing on this range, and hence the minimal value occurs atδ=2​πK\\delta=\\frac\{2\\pi\}\{K\}\.

### D\.3Proof of Theorem[4\.2](https://arxiv.org/html/2605.23087#S4.Thmtheorem2)

We begin by considering the deep UFM with linear activations as defined in Eq\. \([1](https://arxiv.org/html/2605.23087#S2.E1)\), denoting the logit matrix asZ=WL​…​W1​H1Z=W\_\{L\}\.\.\.W\_\{1\}H\_\{1\}\. We will denoteW0:=H1W\_\{0\}:=H\_\{1\}to make equations more compact\.

We have the following elementary calculations:

∂Zx​y∂\(Wl\)a​b=\(Al\+1\)x​a​\(Hl\)b​y,\\frac\{\\partial Z\_\{xy\}\}\{\\partial\(W\_\{l\}\)\_\{ab\}\}=\(A\_\{l\+1\}\)\_\{xa\}\(H\_\{l\}\)\_\{by\},∂g​\(Z\)∂Z=P−Y,\\frac\{\\partial g\(Z\)\}\{\\partial Z\}=P\-Y,where we have defined the following quantities:

Al\+1=WL​…​Wl\+1,AL\+1=IK,Hl=Wl−1​…​W1​H1,H0=IK​n,A\_\{l\+1\}=W\_\{L\}\.\.\.W\_\{l\+1\},\\quad A\_\{L\+1\}=I\_\{K\},\\quad H\_\{l\}=W\_\{l\-1\}\.\.\.W\_\{1\}H\_\{1\},\\quad H\_\{0\}=I\_\{Kn\},\(37\)Y=IK⊗1nT,Y=I\_\{K\}\\otimes 1\_\{n\}^\{T\},Pi​j=exp⁡\(Zi​j\)∑kexp\(Zkj\),P\_\{ij\}=\\frac\{\\exp\(Z\_\{ij\}\)\}\{\\sum\_\{k\}\\exp\(Z\_\{kj\)\}\},from which it is clear that the equations of gradient flow are

d​Wld​t=Al\+1T​\(Y−P\)​HlT\.\\frac\{dW\_\{l\}\}\{dt\}=A\_\{l\+1\}^\{T\}\(Y\-P\)H\_\{l\}^\{T\}\.\(38\)We shall now specialize to Hadamard initialization\. Recall we initialize withWL=U​DL​RLTW\_\{L\}=UD\_\{L\}R\_\{L\}^\{T\},Wl=Rl\+1​Dl​RlTW\_\{l\}=R\_\{l\+1\}D\_\{l\}R\_\{l\}^\{T\}forl=1,…,L−1l=1,\.\.\.,L\-1\.H1=R1​D0​VTH\_\{1\}=R\_\{1\}D\_\{0\}V^\{T\}, whereU=1K​ΦU=\\frac\{1\}\{\\sqrt\{K\}\}\\Phi,V=U⊗QV=U\\otimes Q, whereΦ\\Phiis theK×KK\\times KSylvester Hadamard matrix, andQQis a right singular matrix of1nT1\_\{n\}^\{T\}, meaning1nT=n​e1\(n\)​T​QT1\_\{n\}^\{T\}=\\sqrt\{n\}e\_\{1\}^\{\(n\)T\}Q^\{T\}, wheree1\(n\)e\_\{1\}^\{\(n\)\}is the first standard basis vector inℝn\\mathbb\{R\}^\{n\}\. Also eachDlD\_\{l\}has its firstKKdiagonal entries given byα0\(l\),…,αK−1\(l\)\\alpha\_\{0\}^\{\(l\)\},\.\.\.,\\alpha\_\{K\-1\}^\{\(l\)\}, with all other entries being zero, so that the initializations are in the form of a singular value decomposition\.

As a consequence, we have at initialization thatZ=U​DZ​VTZ=UD\_\{Z\}V^\{T\}, where the firstKKdiagonal entries ofDZD\_\{Z\}are given byai=∏l=0Lαi\(l\)a\_\{i\}=\\prod\_\{l=0\}^\{L\}\\alpha\_\{i\}^\{\(l\)\}fori=0,…,K−1i=0,\.\.\.,K\-1\.

We can write this as

DZ=\[Da,0K×K​\(n−1\)\]=Da⊗e1\(n\)​T,D\_\{Z\}=\[D\_\{a\},0\_\{K\\times K\(n\-1\)\}\]=D\_\{a\}\\otimes e\_\{1\}^\{\(n\)T\},whereDa=diag​\(a0,…,aK−1\)∈ℝKD\_\{a\}=\\textrm\{diag\}\(a\_\{0\},\.\.\.,a\_\{K\-1\}\)\\in\\mathbb\{R\}^\{K\}\. Hence

Z=U​\(Da⊗e1\(n\)​T\)​\(U⊗Q\)T=U​\[\(Da​UT\)⊗\(e1\(n\)​T​QT\)\],Z=U\(D\_\{a\}\\otimes e\_\{1\}^\{\(n\)T\}\)\(U\\otimes Q\)^\{T\}=U\[\(D\_\{a\}U^\{T\}\)\\otimes\(e\_\{1\}^\{\(n\)T\}Q^\{T\}\)\],then using the SVD of1nT1\_\{n\}^\{T\}

Z=1n​U​\[Da​UT⊗1nT\],Z=\\frac\{1\}\{\\sqrt\{n\}\}U\[D\_\{a\}U^\{T\}\\otimes 1\_\{n\}^\{T\}\],and using that this has repeated columns

Z=\(U​Da~​UT\)⊗1nT,Z=\(UD\_\{\\tilde\{a\}\}U^\{T\}\)\\otimes 1\_\{n\}^\{T\},where we have defineda~=1n​a\\tilde\{a\}=\\frac\{1\}\{\\sqrt\{n\}\}aandDa~=diag​\(a~0,…,a~K−1\)D\_\{\\tilde\{a\}\}=\\textrm\{diag\}\(\\tilde\{a\}\_\{0\},\.\.\.,\\tilde\{a\}\_\{K\-1\}\)\. We now apply Lemma[D\.4](https://arxiv.org/html/2605.23087#A4.Thmtheorem4), and use that there are repeated columns, to get the form of the softmax matrix

P​\(Z\)=\(U​Dν~​UT\)⊗1nT,P\(Z\)=\(UD\_\{\\tilde\{\\nu\}\}U^\{T\}\)\\otimes 1\_\{n\}^\{T\},where

ν~i=∑jΦi​j​exp⁡\(1K​\(Φ​a~\)j\)∑jexp⁡\(1K​\(Φ​a~\)j\)\.\\tilde\{\\nu\}\_\{i\}=\\frac\{\\sum\_\{j\}\\Phi\_\{ij\}\\exp\(\\frac\{1\}\{K\}\(\\Phi\\tilde\{a\}\)\_\{j\}\)\}\{\\sum\_\{j\}\\exp\(\\frac\{1\}\{K\}\(\\Phi\\tilde\{a\}\)\_\{j\}\)\}\.\(39\)Noting thatY=n​U​\[IK,0K×K​\(n−1\)\]​VTY=\\sqrt\{n\}U\[I\_\{K\},0\_\{K\\times K\(n\-1\)\}\]V^\{T\}, and that under our initialization

Al\+1=U​DL​…​Dl\+1​Rl\+1T,Hl=Rl​Dl−1​…​D0​VT,A\_\{l\+1\}=UD\_\{L\}\.\.\.D\_\{l\+1\}R\_\{l\+1\}^\{T\},\\quad H\_\{l\}=R\_\{l\}D\_\{l\-1\}\.\.\.D\_\{0\}V^\{T\},we have our gradient flow equations specified in Eq\. \([38](https://arxiv.org/html/2605.23087#A4.E38)\) become

d​Wld​t=n​Rl\+1​\[∏r=l\+1LDrT\]​diag​\(1K−ν~,0K×K​\(n−1\)\)​\[∏r=0l−1DrT\]​RlT\.\\frac\{dW\_\{l\}\}\{dt\}=\\sqrt\{n\}R\_\{l\+1\}\\left\[\\prod\_\{r=l\+1\}^\{L\}D\_\{r\}^\{T\}\\right\]\\textrm\{diag\}\(1\_\{K\}\-\\tilde\{\\nu\},0\_\{K\\times K\(n\-1\)\}\)\\left\[\\prod\_\{r=0\}^\{l\-1\}D\_\{r\}^\{T\}\\right\]R\_\{l\}^\{T\}\.We see that ifWlW\_\{l\}has singular vectors given byRl\+1R\_\{l\+1\}andRlR\_\{l\}at initialization, then the derivative has the same singular vectors\. This implies that this property is preserved for all future times if it holds at initialization\.

Consequently, only the singular values ofWlW\_\{l\}evolve, and the singular vectors remain fixed\. These singular values evolve under

d​Dld​t=n​\[∏r=l\+1LDrT\]​diag​\(1K−ν~,0K×K​\(n−1\)\)​\[∏r=0l−1DrT\]\.\\frac\{dD\_\{l\}\}\{dt\}=\\sqrt\{n\}\\left\[\\prod\_\{r=l\+1\}^\{L\}D\_\{r\}^\{T\}\\right\]\\textrm\{diag\}\(1\_\{K\}\-\\tilde\{\\nu\},0\_\{K\\times K\(n\-1\)\}\)\\left\[\\prod\_\{r=0\}^\{l\-1\}D\_\{r\}^\{T\}\\right\]\.Stated more succinctly in terms of the individual values:

d​αi\(l\)d​t=n​\(1−ν~i\)​∏r≠lαi\(r\)\.\\frac\{d\\alpha\_\{i\}^\{\(l\)\}\}\{dt\}=\\sqrt\{n\}\(1\-\\tilde\{\\nu\}\_\{i\}\)\\prod\_\{r\\neq l\}\\alpha\_\{i\}^\{\(r\)\}\.\(40\)We now note that

1−ν~i=∑j\(1−Φi​j\)​exp⁡\(1K​\(Φ​a~\)j\)∑jexp⁡\(1K​\(Φ​a~\)j\)\.1\-\\tilde\{\\nu\}\_\{i\}=\\frac\{\\sum\_\{j\}\(1\-\\Phi\_\{ij\}\)\\exp\(\\frac\{1\}\{K\}\(\\Phi\\tilde\{a\}\)\_\{j\}\)\}\{\\sum\_\{j\}\\exp\(\\frac\{1\}\{K\}\(\\Phi\\tilde\{a\}\)\_\{j\}\)\}\.
Fori=0i=0, we clearly have1−ν~i=01\-\\tilde\{\\nu\}\_\{i\}=0since the first column and row ofΦ\\Phiare given by1K1\_\{K\}\. Hencedd​t​\(α0\(l\)\)=0\\frac\{d\}\{dt\}\(\\alpha\_\{0\}^\{\(l\)\}\)=0forl=0,…,Ll=0,\.\.\.,L\. For the other cases, dividing the numerator and denominator byexp⁡\(1K​∑ja~j\)\\exp\(\\frac\{1\}\{K\}\\sum\_\{j\}\\tilde\{a\}\_\{j\}\)gives

1−ν~i=∑j=1K−1Ψi​j​exp⁡\(−1K​n​\(Ψ​a\)j\)1\+∑j=1K−1exp⁡\(−1K​n​\(Ψ​a\)j\),1\-\\tilde\{\\nu\}\_\{i\}=\\frac\{\\sum\_\{j=1\}^\{K\-1\}\\Psi\_\{ij\}\\exp\(\-\\frac\{1\}\{K\\sqrt\{n\}\}\(\\Psi a\)\_\{j\}\)\}\{1\+\\sum\_\{j=1\}^\{K\-1\}\\exp\(\-\\frac\{1\}\{K\\sqrt\{n\}\}\(\\Psi a\)\_\{j\}\)\},whereΨ∈ℝK−1×K−1\\Psi\\in\\mathbb\{R\}^\{K\-1\\times K\-1\}is the matrix formed by deleting the first column and row of1K​1K−1T−Φ1\_\{K\}1\_\{K\-1\}^\{T\}\-\\Phi, and we used thata~=1n​a\\tilde\{a\}=\\frac\{1\}\{\\sqrt\{n\}\}a\. Hence we can write our gradient flow as

d​αi\(l\)d​t=nD​bi​∏r≠lαi\(r\),i=1,…,K−1,d​α0\(l\)d​t=0,\\frac\{d\\alpha\_\{i\}^\{\(l\)\}\}\{dt\}=\\frac\{\\sqrt\{n\}\}\{D\}b\_\{i\}\\prod\_\{r\\neq l\}\\alpha\_\{i\}^\{\(r\)\},\\quad i=1,\.\.\.,K\-1,\\quad\\frac\{d\\alpha\_\{0\}^\{\(l\)\}\}\{dt\}=0,\(41\)where

bi=∑j=1K−1Ψi​j​e−1K​n​\(Ψ​a\)j,D=1\+∑j=1K−1e−1K​n​\(Ψ​a\)j,b\_\{i\}=\\sum\_\{j=1\}^\{K\-1\}\\Psi\_\{ij\}e^\{\-\\frac\{1\}\{K\\sqrt\{n\}\}\(\\Psi a\)\_\{j\}\},\\quad D=1\+\\sum\_\{j=1\}^\{K\-1\}e^\{\-\\frac\{1\}\{K\\sqrt\{n\}\}\(\\Psi a\)\_\{j\}\},withai=∏l=0Lαi\(l\)a\_\{i\}=\\prod\_\{l=0\}^\{L\}\\alpha\_\{i\}^\{\(l\)\}\.

It is straightforward to verify that for alll,r=0,…,L\+1l,r=0,\.\.\.,L\+1, the quantities\(αi\(l\)\)2−\(αi\(r\)\)2\(\\alpha\_\{i\}^\{\(l\)\}\)^\{2\}\-\(\\alpha\_\{i\}^\{\(r\)\}\)^\{2\}are conserved along the flow given in Eq\. \([41](https://arxiv.org/html/2605.23087#A4.E41)\)\. From small initialization such conserved quantities would be approximately zero\. Here we set them exactly to zero, in which case we have

αi\(l\)=αi\(r\)∀l,r=0,…,L,\\alpha\_\{i\}^\{\(l\)\}=\\alpha\_\{i\}^\{\(r\)\}\\quad\\forall l,r=0,\.\.\.,L,\(42\)where we have used that singular values are non\-negative\. This is known as the balancedness condition\. Hence we can drop thellindex from these singular values, writing them asαi\\alpha\_\{i\}\. This reduces us to onlyK−1K\-1evolving variables, one for eachi=1,…,K−1i=1,\.\.\.,K\-1\. We define the logit singular values to be

ai=∏l=0Lαi\(l\)=αiL\+1\.a\_\{i\}=\\prod\_\{l=0\}^\{L\}\\alpha\_\{i\}^\{\(l\)\}=\\alpha\_\{i\}^\{L\+1\}\.Transforming to such variables captures the full equations\. Using Eq\. \([41](https://arxiv.org/html/2605.23087#A4.E41)\), these quantities evolve as:

d​aid​t=\(L\+1\)​αiL​d​αid​t=n​\(L\+1\)​biD​αi2​L=n​\(L\+1\)​biD​ai2​LL\+1,\\frac\{da\_\{i\}\}\{dt\}=\(L\+1\)\\alpha\_\{i\}^\{L\}\\frac\{d\\alpha\_\{i\}\}\{dt\}=\\sqrt\{n\}\(L\+1\)\\frac\{b\_\{i\}\}\{D\}\\alpha\_\{i\}^\{2L\}=\\sqrt\{n\}\(L\+1\)\\frac\{b\_\{i\}\}\{D\}a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\},where here

bi=∑j=1K−1Ψi​j​e−1K​n​\(Ψ​a\)j,D=1\+∑j=1K−1e−1K​n​\(Ψ​a\)j,b\_\{i\}=\\sum\_\{j=1\}^\{K\-1\}\\Psi\_\{ij\}e^\{\-\\frac\{1\}\{K\\sqrt\{n\}\}\(\\Psi a\)\_\{j\}\},\\quad D=1\+\\sum\_\{j=1\}^\{K\-1\}e^\{\-\\frac\{1\}\{K\\sqrt\{n\}\}\(\\Psi a\)\_\{j\}\},Absorbing constants into theaaandttvariables via the transform

a′=1K​n​a,t′=1K​\(L\+1\)​\(K​n\)2​LL\+1​t,a^\{\\prime\}=\\frac\{1\}\{K\\sqrt\{n\}\}a,\\quad t^\{\\prime\}=\\frac\{1\}\{K\}\(L\+1\)\(K\\sqrt\{n\}\)^\{\\frac\{2L\}\{L\+1\}\}t,and dropping the primes then gives the final reduction stated in Eq\. \([6](https://arxiv.org/html/2605.23087#S4.E6)\)\.

d​aid​t=1D​bi​ai2​LL\+1,\\frac\{da\_\{i\}\}\{dt\}=\\frac\{1\}\{D\}b\_\{i\}a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\},withbi=∑j=1K−1Ψi​j​e−\(Ψ​a\)jb\_\{i\}=\\sum\_\{j=1\}^\{K\-1\}\\Psi\_\{ij\}e^\{\-\(\\Psi a\)\_\{j\}\}andD=1\+∑j=1K−1e−\(Ψ​a\)j\.D=1\+\\sum\_\{j=1\}^\{K\-1\}e^\{\-\(\\Psi a\)\_\{j\}\}\.

### D\.4Proof of Theorem[4\.3](https://arxiv.org/html/2605.23087#S4.Thmtheorem3)

We write our singular values in the following form, so as to linearize around the DNC solution

ai=μ\+μ​ϵi,∑iϵi=0,μ=1K−1​∑iai=1K−1​‖a‖1,a\_\{i\}=\\mu\+\\mu\\epsilon\_\{i\},\\quad\\sum\_\{i\}\\epsilon\_\{i\}=0,\\quad\\mu=\\frac\{1\}\{K\-1\}\\sum\_\{i\}a\_\{i\}=\\frac\{1\}\{K\-1\}\\\|a\\\|\_\{1\},\(43\)and let∑i\|ϵi\|=‖ϵ‖1≪1\\sum\_\{i\}\|\\epsilon\_\{i\}\|=\\\|\\epsilon\\\|\_\{1\}\\ll 1\. This gives that

\(Ψ​a\)i=K​μ\+μ​\(Ψ​ϵ\)i\.\(\\Psi a\)\_\{i\}=K\\mu\+\\mu\(\\Psi\\epsilon\)\_\{i\}\.Consequently

D=1\+e−K​μ​∑j\(1−μ​\(Ψ​ϵ\)j\)\+O​\(‖ϵ‖12\)\.D=1\+e^\{\-K\\mu\}\\sum\_\{j\}\(1\-\\mu\(\\Psi\\epsilon\)\_\{j\}\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.Then using that∑iΨi​j=K\\sum\_\{i\}\\Psi\_\{ij\}=K, this reduces to

D=1\+\(K−1\)​e−K​μ\+O​\(‖ϵ‖12\)\.D=1\+\(K\-1\)e^\{\-K\\mu\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.\(44\)Similarly

bi=e−K​μ​∑jΨi​j​\[1−μ​\(Ψ​ϵ\)j\]\+O​\(‖ϵ‖12\)b\_\{i\}=e^\{\-K\\mu\}\\sum\_\{j\}\\Psi\_\{ij\}\[1\-\\mu\(\\Psi\\epsilon\)\_\{j\}\]\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)=e−K​μ​\[K−μ​\(Ψ2​ϵ\)i\]\+O​\(‖ϵ‖12\)\.=e^\{\-K\\mu\}\[K\-\\mu\(\\Psi^\{2\}\\epsilon\)\_\{i\}\]\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.Then using thatΨ2=K​\(IK−1\+1K−1​1K−1T\)\\Psi^\{2\}=K\(I\_\{K\-1\}\+1\_\{K\-1\}1\_\{K\-1\}^\{T\}\)gives

bi=K​e−K​μ​\(1−μ​ϵi\)\+O​\(‖ϵ‖12\)\.b\_\{i\}=Ke^\{\-K\\mu\}\(1\-\\mu\\epsilon\_\{i\}\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.\(45\)Hence, inputting Eq\. \([44](https://arxiv.org/html/2605.23087#A4.E44)\) and \([45](https://arxiv.org/html/2605.23087#A4.E45)\) into the gradient flow equation \([6](https://arxiv.org/html/2605.23087#S4.E6)\) the derivative is given by

d​aid​t=K​e−K​μ1\+\(K−1\)​e−K​μ​\(1−μ​ϵi\)​\(μ\+μ​ϵi\)2​LL\+1\+O​\(‖ϵ‖12\)\\frac\{da\_\{i\}\}\{dt\}=\\frac\{Ke^\{\-K\\mu\}\}\{1\+\(K\-1\)e^\{\-K\\mu\}\}\(1\-\\mu\\epsilon\_\{i\}\)\(\\mu\+\\mu\\epsilon\_\{i\}\)^\{\\frac\{2L\}\{L\+1\}\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)=K​e−K​μ​μ2​LL\+11\+\(K−1\)​e−K​μ​\(1−μ​ϵi\)​\(1\+2​LL\+1​ϵi\)\+O​\(‖ϵ‖12\)=\\frac\{Ke^\{\-K\\mu\}\\mu^\{\\frac\{2L\}\{L\+1\}\}\}\{1\+\(K\-1\)e^\{\-K\\mu\}\}\(1\-\\mu\\epsilon\_\{i\}\)\\left\(1\+\\frac\{2L\}\{L\+1\}\\epsilon\_\{i\}\\right\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)=K​e−K​μ​μ2​LL\+11\+\(K−1\)​e−K​μ⏟C​\(μ\)​\[1\+\(2​LL\+1−μ\)⏟B​\(μ\)​ϵi\]\+O​\(‖ϵ‖12\),=\\underbrace\{\\frac\{Ke^\{\-K\\mu\}\\mu^\{\\frac\{2L\}\{L\+1\}\}\}\{1\+\(K\-1\)e^\{\-K\\mu\}\}\}\_\{C\(\\mu\)\}\\left\[1\+\\underbrace\{\\left\(\\frac\{2L\}\{L\+1\}\-\\mu\\right\)\}\_\{B\(\\mu\)\}\\epsilon\_\{i\}\\right\]\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\),where we have defined two constantsC​\(μ\)C\(\\mu\)andB​\(μ\)B\(\\mu\)\. Now consider how the ratio of normalized logit values changes in this regime

dd​t​\(a^ia^j\)=1aj2​\(aj​d​aid​t−ai​d​ajd​t\)\\frac\{d\}\{dt\}\\left\(\\frac\{\\hat\{a\}\_\{i\}\}\{\\hat\{a\}\_\{j\}\}\\right\)=\\frac\{1\}\{a\_\{j\}^\{2\}\}\\left\(a\_\{j\}\\frac\{da\_\{i\}\}\{dt\}\-a\_\{i\}\\frac\{da\_\{j\}\}\{dt\}\\right\)=C​\(μ\)aj2​\[\(μ\+μ​ϵj\)​\(1\+B​\(μ\)​ϵi\)−\(μ\+μ​ϵi\)​\(1\+B​\(μ\)​ϵj\)\]\+O​\(‖ϵ‖12\)=\\frac\{C\(\\mu\)\}\{a\_\{j\}^\{2\}\}\\left\[\(\\mu\+\\mu\\epsilon\_\{j\}\)\(1\+B\(\\mu\)\\epsilon\_\{i\}\)\-\(\\mu\+\\mu\\epsilon\_\{i\}\)\(1\+B\(\\mu\)\\epsilon\_\{j\}\)\\right\]\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)=μ​C​\(μ\)aj2​\(B​\(μ\)−1\)​\(ϵi−ϵj\)\+O​\(‖ϵ‖12\)\.=\\frac\{\\mu C\(\\mu\)\}\{a\_\{j\}^\{2\}\}\(B\(\\mu\)\-1\)\(\\epsilon\_\{i\}\-\\epsilon\_\{j\}\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.Then using the expression forB​\(μ\)B\(\\mu\)this becomes

dd​t​\(a^ia^j\)=μ​C​\(μ\)aj2​\[L−1L\+1−μ\]​\(ϵi−ϵj\)\+O​\(‖ϵ‖12\)\.\\frac\{d\}\{dt\}\\left\(\\frac\{\\hat\{a\}\_\{i\}\}\{\\hat\{a\}\_\{j\}\}\\right\)=\\frac\{\\mu C\(\\mu\)\}\{a\_\{j\}^\{2\}\}\\left\[\\frac\{L\-1\}\{L\+1\}\-\\mu\\right\]\(\\epsilon\_\{i\}\-\\epsilon\_\{j\}\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.Noting that the sign ofa^i−a^j\\hat\{a\}\_\{i\}\-\\hat\{a\}\_\{j\}is the same as the sign ofϵi−ϵj\\epsilon\_\{i\}\-\\epsilon\_\{j\}, we see that in the regime linearized around DNC whether two distinct modes grow towards each other depends on the sign ofL−1L\+1−μ\\frac\{L\-1\}\{L\+1\}\-\\mu\. In particular forL−1L\+1\>μ\\frac\{L\-1\}\{L\+1\}\>\\mu, DNC is unstable at all scales, whilst forL−1L\+1<μ\\frac\{L\-1\}\{L\+1\}<\\muit is stable at all scales\. This recovers the bound in the theorem statement when we recall that‖a‖1=\(K−1\)​μ\\\|a\\\|\_\{1\}=\(K\-1\)\\mu\. We also note this bound is trivial forL=1L=1, and so DNC is stable throughout the full loss surface in this case\.

We now show an alternative low\-rank structure that is stable above a threshold in the norm of the logit singular values\. Define the vectorc∈ℝK−1c\\in\\mathbb\{R\}^\{K\-1\}that has value one in odd indices and value 0 in even indices\. Write the singular values in the following form:

a=μ​c\+μ​ϵ,μ=2K​‖a‖1,∑iϵi=0,a=\\mu c\+\\mu\\epsilon,\\quad\\mu=\\frac\{2\}\{K\}\\\|a\\\|\_\{1\},\\quad\\sum\_\{i\}\\epsilon\_\{i\}=0,\(46\)and consider the regime where‖ϵ‖1=∑i\|ϵi\|≪1\\\|\\epsilon\\\|\_\{1\}=\\sum\_\{i\}\|\\epsilon\_\{i\}\|\\ll 1\. Using Lemma[D\.9](https://arxiv.org/html/2605.23087#A4.Thmtheorem9), we have that the correspondingbbvector is given by:

bi=\{K​e−K2​μ\+2​\(e−K​μ−eK2​μ\)​\(1−μ​\(Ψ​ϵ\)1\)−K​e−K2​μ​μ​ϵi\+O​\(‖ϵ‖12\),if​i​is odd,K​e−K2​μ−K​e−K2​μ​μ​ϵi\+O​\(‖ϵ‖12\),if​i​is even\.b\_\{i\}=\\begin\{cases\}Ke^\{\-\\frac\{K\}\{2\}\\mu\}\+2\(e^\{\-K\\mu\}\-e^\{\\frac\{K\}\{2\}\\mu\}\)\(1\-\\mu\(\\Psi\\epsilon\)\_\{1\}\)\-Ke^\{\-\\frac\{K\}\{2\}\\mu\}\\mu\\epsilon\_\{i\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\),&\\textrm\{if \}i\\textrm\{ is odd\},\\\\ Ke^\{\-\\frac\{K\}\{2\}\\mu\}\-Ke^\{\-\\frac\{K\}\{2\}\\mu\}\\mu\\epsilon\_\{i\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\),&\\textrm\{if \}i\\textrm\{ is even\}\.\\end\{cases\}We also have

ai2​LL\+1=\{μ2​LL\+1​\(1\+2​LL\+1​ϵi\)\+O​\(‖ϵ‖12\),if​i​is odd,μ2​LL\+1​ϵi2​LL\+1,if​i​is even\.a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}=\\begin\{cases\}\\mu^\{\\frac\{2L\}\{L\+1\}\}\(1\+\\frac\{2L\}\{L\+1\}\\epsilon\_\{i\}\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\),&\\textrm\{if \}i\\textrm\{ is odd\},\\\\ \\mu^\{\\frac\{2L\}\{L\+1\}\}\\epsilon\_\{i\}^\{\\frac\{2L\}\{L\+1\}\},&\\textrm\{if \}i\\textrm\{ is even\}\.\\end\{cases\}Hence the derivatives of the logit singular values are given by

d​aid​t=\{1D​μ2​LL\+1​e−K2​μ​\[K​\(1\+2​LL\+1​ϵi−μ​ϵi\)\+2​\(e−K2​μ−1\)​\(1\+2​LL\+1​ϵi−μ​\(Ψ​ϵ\)1\)\]\+O​\(‖ϵ‖12\),if​i​is odd,1D​K​e−K2​μ​μ2​LL\+1​ϵi2​LL\+1\+O​\(‖ϵ‖12\),if​i​is even\.\\frac\{da\_\{i\}\}\{dt\}=\\begin\{cases\}\\frac\{1\}\{D\}\\mu^\{\\frac\{2L\}\{L\+1\}\}e^\{\-\\frac\{K\}\{2\}\\mu\}\[K\(1\+\\frac\{2L\}\{L\+1\}\\epsilon\_\{i\}\-\\mu\\epsilon\_\{i\}\)\+2\(e^\{\-\\frac\{K\}\{2\}\\mu\}\-1\)\(1\+\\frac\{2L\}\{L\+1\}\\epsilon\_\{i\}\-\\mu\(\\Psi\\epsilon\)\_\{1\}\)\]\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\),&\\textrm\{if \}i\\textrm\{ is odd\},\\\\ \\frac\{1\}\{D\}Ke^\{\-\\frac\{K\}\{2\}\\mu\}\\mu^\{\\frac\{2L\}\{L\+1\}\}\\epsilon\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\),&\\textrm\{if \}i\\textrm\{ is even\}\.\\end\{cases\}\(47\)We now can compare how the relative size of the modes grows\. First letiibe odd, andjjbe even, then we have

d​a^id​a^j=1aj2​\[aj​d​aid​t−ai​d​ajd​t\]\\frac\{d\\hat\{a\}\_\{i\}\}\{d\\hat\{a\}\_\{j\}\}=\\frac\{1\}\{a\_\{j\}^\{2\}\}\\left\[a\_\{j\}\\frac\{da\_\{i\}\}\{dt\}\-a\_\{i\}\\frac\{da\_\{j\}\}\{dt\}\\right\]=e−K2​μ​μ1\+2​LL\+1D​aj2​\[\(K−2\+2​e−K2​μ\)​ϵj−K​e−K2​μ​ϵj2​LL\+1\+O​\(‖ϵ‖12\)\]\.=\\frac\{e^\{\-\\frac\{K\}\{2\}\\mu\}\\mu^\{1\+\\frac\{2L\}\{L\+1\}\}\}\{Da\_\{j\}^\{2\}\}\\left\[\(K\-2\+2e^\{\-\\frac\{K\}\{2\}\\mu\}\)\\epsilon\_\{j\}\-Ke^\{\-\\frac\{K\}\{2\}\\mu\}\\epsilon\_\{j\}^\{\\frac\{2L\}\{L\+1\}\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\\right\]\.\(48\)WhenL\>1L\>1the second term in the brackets is higher order, and so to leading order this is given by

=e−K2​μ​μ1\+2​LL\+1D​aj2​\[\(K−2\+2​e−K2​μ\)​ϵj\+O​\(‖ϵ‖12​LL\+1\)\]\.=\\frac\{e^\{\-\\frac\{K\}\{2\}\\mu\}\\mu^\{1\+\\frac\{2L\}\{L\+1\}\}\}\{Da\_\{j\}^\{2\}\}\\left\[\(K\-2\+2e^\{\-\\frac\{K\}\{2\}\\mu\}\)\\epsilon\_\{j\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{\\frac\{2L\}\{L\+1\}\}\)\\right\]\.We see that when‖ϵ‖1\\\|\\epsilon\\\|\_\{1\}is small enough, the dominant term is given byϵj\\epsilon\_\{j\}up to a positive factor\. Note, thatϵj\>0\\epsilon\_\{j\}\>0\. This is sinceaj=μ​ϵja\_\{j\}=\\mu\\epsilon\_\{j\}, and we initialize the singular values as positive, from which they must remain so\. Hence the derivativedd​t​\(a^i/a^j\)\\frac\{d\}\{dt\}\(\\hat\{a\}\_\{i\}/\\hat\{a\}\_\{j\}\)is positive, and the odd modes grow relative to the even modes, whenL\>1L\>1\. It is also simple to see from Eq\. \([48](https://arxiv.org/html/2605.23087#A4.E48)\) that the opposite is the case forL=1L=1\.

Next letiiandjjboth be odd, after simplifying the expression we find in this case the relative size grows as

d​a^id​a^j=μ1\+2​LL\+1​e−K2​μD​aj2​\[\(K​\(μ−L−1L\+1\)\+2​L−1L\+1​\(1−e−K2​μ\)\)​\(ϵj−ϵi\)\+O​\(‖ϵ‖12\)\]\.\\frac\{d\\hat\{a\}\_\{i\}\}\{d\\hat\{a\}\_\{j\}\}=\\frac\{\\mu^\{1\+\\frac\{2L\}\{L\+1\}\}e^\{\-\\frac\{K\}\{2\}\\mu\}\}\{Da\_\{j\}^\{2\}\}\\left\[\\left\(K\(\\mu\-\\frac\{L\-1\}\{L\+1\}\)\+2\\frac\{L\-1\}\{L\+1\}\(1\-e^\{\-\\frac\{K\}\{2\}\\mu\}\)\\right\)\(\\epsilon\_\{j\}\-\\epsilon\_\{i\}\)\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\\right\]\.When‖ϵ‖1\\\|\\epsilon\\\|\_\{1\}is small enough, a sufficient condition for the leading order term to have the same sign asϵj−ϵi\\epsilon\_\{j\}\-\\epsilon\_\{i\}isμ\>L−1L\+1\\mu\>\\frac\{L\-1\}\{L\+1\}\.

Hence we see that the modes close to zero in the structure continue to become small relative to the large modes, and the large modes are drawn to equality, implyinga^\\hat\{a\}is moving towards2K​c\\frac\{2\}\{K\}cwhenL\>1L\>1andμ\>L−1L\+1\\mu\>\\frac\{L\-1\}\{L\+1\}, implying this structure is stable throughout this region\.

Lastly, we demonstrate the KL divergence can monotonically increase to infinity\. To demonstrate this, consider an initialization of the logit singular values of the form

a=\[γ,δ,γ,…,δ,γ\],γ,δ\>0,a=\[\\gamma,\\delta,\\gamma,\.\.\.,\\delta,\\gamma\],\\quad\\gamma,\\delta\>0,\(49\)so that there areK2\\frac\{K\}\{2\}appearances ofγ\\gammaandK2−1\\frac\{K\}\{2\}\-1appearances ofδ\\delta\. Using Lemma[D\.8](https://arxiv.org/html/2605.23087#A4.Thmtheorem8), we have that the correspondingbib\_\{i\}is given by

bi=\{2​e−K​γ\+\(K−2\)​e−K2​\(γ\+δ\),if​i​is odd,K​e−K2​\(γ\+δ\),if​i​is even\.b\_\{i\}=\\begin\{cases\}2e^\{\-K\\gamma\}\+\(K\-2\)e^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\},&\\textrm\{if \}i\\textrm\{ is odd\},\\\\ Ke^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\},&\\textrm\{if \}i\\textrm\{ is even\}\.\\end\{cases\}Consequently, the logit singular value evolution reduces to

d​aid​t=\{1D​γ2​LL\+1​\(2​e−K​γ\+\(K−2\)​e−K2​\(γ\+δ\)\),if​i​is odd,1D​δ2​LL\+1​\(K​e−K2​\(γ\+δ\)\),if​i​is even\.\\frac\{da\_\{i\}\}\{dt\}=\\begin\{cases\}\\frac\{1\}\{D\}\\gamma^\{\\frac\{2L\}\{L\+1\}\}\\left\(2e^\{\-K\\gamma\}\+\(K\-2\)e^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\}\\right\),&\\textrm\{if \}i\\textrm\{ is odd\},\\\\ \\frac\{1\}\{D\}\\delta^\{\\frac\{2L\}\{L\+1\}\}\\left\(Ke^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\}\\right\),&\\textrm\{if \}i\\textrm\{ is even\}\.\\end\{cases\}\(50\)We see that ifaais initialized in the form of Eq\. \([49](https://arxiv.org/html/2605.23087#A4.E49)\), it remains in this form under gradient flow\. Now we consider how the relative size of the two distinct values changes

dd​t​\(γδ\)=1δ2​\(δ​d​γd​t−γ​d​δd​t\)\\frac\{d\}\{dt\}\\left\(\\frac\{\\gamma\}\{\\delta\}\\right\)=\\frac\{1\}\{\\delta^\{2\}\}\\left\(\\delta\\frac\{d\\gamma\}\{dt\}\-\\gamma\\frac\{d\\delta\}\{dt\}\\right\)=1D​δ2​\[δ​γ2​LL\+1​\(2​e−K​γ\+\(K−2\)​e−K2​\(γ\+δ\)\)−γ​δ2​LL\+1​K​e−K2​\(γ\+δ\)\]=\\frac\{1\}\{D\\delta^\{2\}\}\\left\[\\delta\\gamma^\{\\frac\{2L\}\{L\+1\}\}\\left\(2e^\{\-K\\gamma\}\+\(K\-2\)e^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\}\\right\)\-\\gamma\\delta^\{\\frac\{2L\}\{L\+1\}\}Ke^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\}\\right\]=γ​e−K​γD​δ​\[γL−1L\+1​\(2\+\(K−2\)​eK2​\(γ−δ\)\)−K​δL−1L\+1​eK2​\(γ−δ\)\]\.=\\frac\{\\gamma e^\{\-K\\gamma\}\}\{D\\delta\}\\left\[\\gamma^\{\\frac\{L\-1\}\{L\+1\}\}\\left\(2\+\(K\-2\)e^\{\\frac\{K\}\{2\}\(\\gamma\-\\delta\)\}\\right\)\-K\\delta^\{\\frac\{L\-1\}\{L\+1\}\}e^\{\\frac\{K\}\{2\}\(\\gamma\-\\delta\)\}\\right\]\.This derivative is positive when

γL−1L\+1​\(2\+\(K−2\)​eK2​\(γ−δ\)\)\>K​δL−1L\+1​eK2​\(γ−δ\),\\gamma^\{\\frac\{L\-1\}\{L\+1\}\}\\left\(2\+\(K\-2\)e^\{\\frac\{K\}\{2\}\(\\gamma\-\\delta\)\}\\right\)\>K\\delta^\{\\frac\{L\-1\}\{L\+1\}\}e^\{\\frac\{K\}\{2\}\(\\gamma\-\\delta\)\},for which a sufficient condition is that

\(K−2\)​γL−1L\+1​eK2​\(γ−δ\)\>K​δL−1L\+1​eK2​\(γ−δ\),\(K\-2\)\\gamma^\{\\frac\{L\-1\}\{L\+1\}\}e^\{\\frac\{K\}\{2\}\(\\gamma\-\\delta\)\}\>K\\delta^\{\\frac\{L\-1\}\{L\+1\}\}e^\{\\frac\{K\}\{2\}\(\\gamma\-\\delta\)\},which can be rewritten as

γδ\>\(KK−2\)L\+1L−1\.\\frac\{\\gamma\}\{\\delta\}\>\\left\(\\frac\{K\}\{K\-2\}\\right\)^\{\\frac\{L\+1\}\{L\-1\}\}\.\(51\)We see that ifγ/δ\\gamma/\\deltais above the threshold given in Eq\. \([51](https://arxiv.org/html/2605.23087#A4.E51)\), thendd​t​\(γ/δ\)\\frac\{d\}\{dt\}\(\\gamma/\\delta\)is positive, and soγ/δ\\gamma/\\deltawill monotonically increase under gradient flow, and so remains above the threshold\. Consequently if we initialize in the region specified by this threshold, then the normalized logit singular valuesa^\\hat\{a\}must tend monotonically towards the vector2K​\(1,0,1,…,0,1\)\\frac\{2\}\{K\}\(1,0,1,\.\.\.,0,1\)\. This is since the normalized singular values, for even and odd index are given by

aodd=1K2\+\(K2−1\)​δγ,aeven=1K2​γδ\+\(K2−1\),a\_\{\\textrm\{odd\}\}=\\frac\{1\}\{\\frac\{K\}\{2\}\+\\left\(\\frac\{K\}\{2\}\-1\\right\)\\frac\{\\delta\}\{\\gamma\}\},\\quad a\_\{\\textrm\{even\}\}=\\frac\{1\}\{\\frac\{K\}\{2\}\\frac\{\\gamma\}\{\\delta\}\+\\left\(\\frac\{K\}\{2\}\-1\\right\)\},which clearly tend to2K\\frac\{2\}\{K\}and0respectively\. To note that this monotonic approach must guarantee we tend to the limit point, rather than stalling at some other structure, recall in Lemma[D\.1](https://arxiv.org/html/2605.23087#A4.Thmtheorem1)it was shown that the direction must tend to a KKT point, meaning a critical direction under the dynamics at infinity\. It is simple to observe that the same condition in Eq\. \([51](https://arxiv.org/html/2605.23087#A4.E51)\) also guarantees that the only direction for whichdd​t​\(a^\)=0\\frac\{d\}\{dt\}\(\\hat\{a\}\)=0is the limit point, and so it is the only KKT point in this portion of the space and so must be to where the path converges\.

This guarantees that the KL divergence tends to infinity since

DKL​\(1K−1​1K−1∥a^\)=−log⁡\(K−1\)−1K−1​∑ilog⁡\(a^i\),D\_\{\\mathrm\{KL\}\}\(\\frac\{1\}\{K\-1\}1\_\{K\-1\}\\,\\\|\\,\\hat\{a\}\)=\-\\log\(K\-1\)\-\\frac\{1\}\{K\-1\}\\sum\_\{i\}\\log\(\\hat\{a\}\_\{i\}\),\(52\)and clearly if anya^i\\hat\{a\}\_\{i\}tends to zero this diverges to infinity, which we have shown occurs for the described initialization\.

### D\.5Proof of Theorem[4\.4](https://arxiv.org/html/2605.23087#S4.Thmtheorem4)

Recall our dynamics are given by

d​aid​t=1D​bi​ai2​LL\+1,bi=∑j=1K−1Ψi​j​e−\(Ψ​a\)j,D=1\+∑j=1K−1e−\(Ψ​a\)j\.\\frac\{da\_\{i\}\}\{dt\}=\\frac\{1\}\{D\}b\_\{i\}a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\},\\quad b\_\{i\}=\\sum\_\{j=1\}^\{K\-1\}\\Psi\_\{ij\}e^\{\-\(\\Psi a\)\_\{j\}\},\\quad D=1\+\\sum\_\{j=1\}^\{K\-1\}e^\{\-\(\\Psi a\)\_\{j\}\}\.Suppose that‖a‖1=∑iai=ϵ≪1\\\|a\\\|\_\{1\}=\\sum\_\{i\}a\_\{i\}=\\epsilon\\ll 1\. Expanding the exponential gives

D=1\+∑j\[1−\(Ψ​a\)j\]\+O​\(ϵ2\)=K−K​ϵ\+O​\(ϵ2\),D=1\+\\sum\_\{j\}\[1\-\(\\Psi a\)\_\{j\}\]\+O\(\\epsilon^\{2\}\)=K\-K\\epsilon\+O\(\\epsilon^\{2\}\),\(53\)where we used that∑jΨi​j=K\\sum\_\{j\}\\Psi\_\{ij\}=K, which is inherited from the properties of the Sylvester Hadamard matrix\. Similarly

bi=∑jΨi​j​\(1−\(Ψ​a\)j\)\+O​\(ϵ2\)=K−\(Ψ2​a\)i\+O​\(ϵ2\)\.b\_\{i\}=\\sum\_\{j\}\\Psi\_\{ij\}\(1\-\(\\Psi a\)\_\{j\}\)\+O\(\\epsilon^\{2\}\)=K\-\(\\Psi^\{2\}a\)\_\{i\}\+O\(\\epsilon^\{2\}\)\.We then use thatΨ2=K​\(IK−1\+1K−1​1K−1T\)\\Psi^\{2\}=K\(I\_\{K\-1\}\+1\_\{K\-1\}1\_\{K\-1\}^\{T\}\), so that

bi=K−K​ai−K​ϵ\+O​\(ϵ2\)\.b\_\{i\}=K\-Ka\_\{i\}\-K\\epsilon\+O\(\\epsilon^\{2\}\)\.\(54\)Inputting Eq\. \([53](https://arxiv.org/html/2605.23087#A4.E53)\) and \([54](https://arxiv.org/html/2605.23087#A4.E54)\) into the expression of the derivative gives:

d​aid​t=ai2​LL\+1​\[K−K​ai−K​ϵ\+O​\(ϵ2\)\]​\[K​\(1−ϵ\)\+O​\(ϵ2\)\]−1\\frac\{da\_\{i\}\}\{dt\}=a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}\[K\-Ka\_\{i\}\-K\\epsilon\+O\(\\epsilon^\{2\}\)\]\[K\(1\-\\epsilon\)\+O\(\\epsilon^\{2\}\)\]^\{\-1\}=ai2​LL\+1​\[1−ai−ϵ\+O​\(ϵ2\)\]​\[1\+ϵ\+O​\(ϵ2\)\]=a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}\[1\-a\_\{i\}\-\\epsilon\+O\(\\epsilon^\{2\}\)\]\[1\+\\epsilon\+O\(\\epsilon^\{2\}\)\]=ai2​LL\+1​\(1−ai\)\+O​\(ϵ2\+2​LL\+1\)\.=a\_\{i\}^\{\\frac\{2L\}\{L\+1\}\}\(1\-a\_\{i\}\)\+O\\left\(\\epsilon^\{2\+\\frac\{2L\}\{L\+1\}\}\\right\)\.We now consider how the ratio of normalized singular values changes in this linearized regime\. This is given by

dd​t​\(a^ia^j\)=dd​t​\(aiaj\)=aiaj​\[aiL−1L\+1​\(1−ai\)−ajL−1L\+1​\(1−aj\)\+O​\(ϵ2\+L−1L\+1\)\]\.\\frac\{d\}\{dt\}\\left\(\\frac\{\\hat\{a\}\_\{i\}\}\{\\hat\{a\}\_\{j\}\}\\right\)=\\frac\{d\}\{dt\}\\left\(\\frac\{a\_\{i\}\}\{a\_\{j\}\}\\right\)=\\frac\{a\_\{i\}\}\{a\_\{j\}\}\\left\[a\_\{i\}^\{\\frac\{L\-1\}\{L\+1\}\}\(1\-a\_\{i\}\)\-a\_\{j\}^\{\\frac\{L\-1\}\{L\+1\}\}\(1\-a\_\{j\}\)\+O\\left\(\\epsilon^\{2\+\\frac\{L\-1\}\{L\+1\}\}\\right\)\\right\]\.\(55\)WhenL=1L=1, this reduces to

dd​t​\(a^ia^j\)=−aiaj​‖a‖1​\[a^i−a^j\+O​\(ϵ\)\]\.\\frac\{d\}\{dt\}\\left\(\\frac\{\\hat\{a\}\_\{i\}\}\{\\hat\{a\}\_\{j\}\}\\right\)=\-\\frac\{a\_\{i\}\}\{a\_\{j\}\\\|a\\\|\_\{1\}\}\\left\[\\hat\{a\}\_\{i\}\-\\hat\{a\}\_\{j\}\+O\\left\(\\epsilon\\right\)\\right\]\.Whenϵ\\epsilonsmall enough, this clearly has opposite sign toa^i−a^j\\hat\{a\}\_\{i\}\-\\hat\{a\}\_\{j\}\. Consequently the ratioa^i/a^j\\hat\{a\}\_\{i\}/\\hat\{a\}\_\{j\}must monotonically approach 1 in the linearized regime, meaning any two modes that are different must approach each other\. Clearly under such dynamics, the only stable structure is whena^\\hat\{a\}is the uniform vector, meaning DNC is the only stable structure under the linearized dynamics forL=1L=1\.

WhenL≥2L\\geq 2and‖a‖1\\\|a\\\|\_\{1\}is small as in the linearized regime, we have

a^i\>aj^⟹aiL−1L\+1​\(1−ai\)\>ajL−1L\+1​\(1−aj\)\.\\hat\{a\}\_\{i\}\>\\hat\{a\_\{j\}\}\\implies a\_\{i\}^\{\\frac\{L\-1\}\{L\+1\}\}\(1\-a\_\{i\}\)\>a\_\{j\}^\{\\frac\{L\-1\}\{L\+1\}\}\(1\-a\_\{j\}\)\.Consequently the sign of the derivative in Eq\. \([55](https://arxiv.org/html/2605.23087#A4.E55)\) is the same as that ofa^i−a^j\\hat\{a\}\_\{i\}\-\\hat\{a\}\_\{j\}whenϵ\\epsilonis small enough\. This implies that any two non\-equal singular values must grow further apart\. Thus, DNC is critical, but not stable, in the linearized regime forL≥2L\\geq 2\.

### D\.6Proof of Theorem[4\.5](https://arxiv.org/html/2605.23087#S4.Thmtheorem5)

Recall our optimization problem, given in Eq\. \([1](https://arxiv.org/html/2605.23087#S2.E1)\)\. It was shown in Eq\. \([38](https://arxiv.org/html/2605.23087#A4.E38)\) that the gradient flow equations for the parameter matrices are given by \(using the shorthandW0:=H1W\_\{0\}:=H\_\{1\}\)

d​Wld​t=Al\+1T​\(Y−P\)​HlT,for​l=0,…,L,\\frac\{dW\_\{l\}\}\{dt\}=A\_\{l\+1\}^\{T\}\(Y\-P\)H\_\{l\}^\{T\},\\quad\\textrm\{for \}l=0,\.\.\.,L,where

Al\+1=WL​…​Wl\+1,for​l=0,…,L−1,AL\+1=IK,A\_\{l\+1\}=W\_\{L\}\.\.\.W\_\{l\+1\},\\quad\\textrm\{for \}l=0,\.\.\.,L\-1,\\quad A\_\{L\+1\}=I\_\{K\},Hl=Wl−1​…​W1​H1,for​l=1,…,L,H0=IK​n,H\_\{l\}=W\_\{l\-1\}\.\.\.W\_\{1\}H\_\{1\},\\quad\\textrm\{for \}l=1,\.\.\.,L,\\quad H\_\{0\}=I\_\{Kn\},andPPis the softmax output of the network

Pi​j=exp⁡\(Zi​j\)∑uexp⁡\(Zu​j\)\.P\_\{ij\}=\\frac\{\\exp\(Z\_\{ij\}\)\}\{\\sum\_\{u\}\\exp\(Z\_\{uj\}\)\}\.We can consider how the logit matrixZ=WL​…​W1​H1Z=W\_\{L\}\.\.\.W\_\{1\}H\_\{1\}is updated directly

d​Zd​t=∑l=0LWL​…​Wl\+1​d​Wld​t​Wl−1​…​H1\\frac\{dZ\}\{dt\}=\\sum\_\{l=0\}^\{L\}W\_\{L\}\.\.\.W\_\{l\+1\}\\frac\{dW\_\{l\}\}\{dt\}W\_\{l\-1\}\.\.\.H\_\{1\}=∑l=0LAl\+1​Al\+1T​\(Y−P\)​HlT​Hl\.=\\sum\_\{l=0\}^\{L\}A\_\{l\+1\}A\_\{l\+1\}^\{T\}\(Y\-P\)H\_\{l\}^\{T\}H\_\{l\}\.\(56\)Now recall that we initialize our parameter matrices asWl​\(0\)=ϵ​BlW\_\{l\}\(0\)=\\epsilon B\_\{l\},l=0,…,Ll=0,\.\.\.,L, where eachBlB\_\{l\}has entries samplied i\.i\.d\. from a Gaussian distribution with mean 0 and variance1/d1/d\. We now quote Lemma[D\.11](https://arxiv.org/html/2605.23087#A4.Thmtheorem11), which states that we have the following convergence in probability asd→∞d\\rightarrow\\infty:

Al\+1​Al\+1T→ϵ2​\(L−l\)​IK,HlT​Hl→ϵ2​l​IK​n\.A\_\{l\+1\}A\_\{l\+1\}^\{T\}\\rightarrow\\epsilon^\{2\(L\-l\)\}I\_\{K\},\\quad H\_\{l\}^\{T\}H\_\{l\}\\rightarrow\\epsilon^\{2l\}I\_\{Kn\}\.Consequently we have

d​Zd​t​\(0\)→\(L\+1\)​ϵ2​L​\(Y−P​\(0\)\)\.\\frac\{dZ\}\{dt\}\(0\)\\rightarrow\(L\+1\)\\epsilon^\{2L\}\(Y\-P\(0\)\)\.Now we note that asϵ→0\\epsilon\\rightarrow 0, we have

Pi​j=exp⁡\(Zi​j\)∑uexp⁡\(Zu​j\)→1K,P\_\{ij\}=\\frac\{\\exp\(Z\_\{ij\}\)\}\{\\sum\_\{u\}\\exp\(Z\_\{uj\}\)\}\\rightarrow\\frac\{1\}\{K\},and hence

P→1K​1K​1K​nT,P\\rightarrow\\frac\{1\}\{K\}1\_\{K\}1\_\{Kn\}^\{T\},so that

Y−P→IK⊗1nT−1K​1K​1K​nT=S⊗1nT\.Y\-P\\rightarrow I\_\{K\}\\otimes 1\_\{n\}^\{T\}\-\\frac\{1\}\{K\}1\_\{K\}1\_\{Kn\}^\{T\}=S\\otimes 1\_\{n\}^\{T\}\.Hence in these joint limits we have the following convergence in probability

d​Zd​t​\(0\)→\(L\+1\)​ϵ2​L​S⊗1nT\.\\frac\{dZ\}\{dt\}\(0\)\\rightarrow\(L\+1\)\\epsilon^\{2L\}S\\otimes 1\_\{n\}^\{T\}\.

### D\.7Supporting Lemmas

###### Lemma D\.1\.

Consider GF on the deep UFM \(Eq\.[1](https://arxiv.org/html/2605.23087#S2.E1)\) with network widthd≥Kd\\geq K\. If there existst0t\_\{0\}such thatℒ​\(t0\)<log⁡\(2\)\\mathcal\{L\}\(t\_\{0\}\)<\\log\(2\), then any limit point of the normalized parameters\(WL^,…,W^1,H1^\)\(\\hat\{W\_\{L\}\},\.\.\.,\\hat\{W\}\_\{1\},\\hat\{H\_\{1\}\}\)lies in the direction of a Karush\-Kuhn\-Tucker \(KKT\) point of the constrained optimization problem

minWL,…,W1,H1⁡‖H1‖F2\+∑l=1L‖Wl‖F2\\qquad\\qquad\\min\_\{W\_\{L\},\.\.\.,W\_\{1\},H\_\{1\}\}\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}s\.t\.​\(zi​c\)c−\(zi​c\)c′≥1,c′≠c=1,…,K,i=1,…,n,\\textrm\{s\.t\. \}\(z\_\{ic\}\)\_\{c\}\-\(z\_\{ic\}\)\_\{c^\{\\prime\}\}\\geq 1,\\quad c^\{\\prime\}\\neq c=1,\.\.\.,K,\\quad i=1,\.\.\.,n,wherezi​cz\_\{ic\}are the columns in class order of the logit matrixZ=WL​…​W1​H1Z=W\_\{L\}\.\.\.W\_\{1\}H\_\{1\}\.

###### Proof\.

Note the arguments given in the proof of Theorem 3\.2 of\(Jiet al\.,[2022](https://arxiv.org/html/2605.23087#bib.bib145)\), though for theL=1L=1UFM, can be applied directly to produce the constrained optimization problem above from the gradient flow dynamics on the deep UFM\. Alternatively, the arguments in the proof of Theorem 4\.4 of\(Lyu and Li,[2019](https://arxiv.org/html/2605.23087#bib.bib108)\), as described in their main text and their App\. G, can be applied to this model since the deep UFM is an example of a homogeneous network, from which the result follows as a corollary of their main theorem\. ∎

###### Lemma D\.2\.

LetK\>2K\>2, andZ′∈ℝK×KZ^\{\\prime\}\\in\\mathbb\{R\}^\{K\\times K\}satisfyZc​c′−Zc′​c′≥1Z^\{\\prime\}\_\{cc\}\-Z^\{\\prime\}\_\{c^\{\\prime\}c\}\\geq 1forc,c′≠c=1,…,Kc,c^\{\\prime\}\\neq c=1,\.\.\.,K\. Thenrank​\(Z′\)≥2\\textrm\{rank\}\(Z^\{\\prime\}\)\\geq 2\.

###### Proof\.

Clearly the condition does not hold ifrank​\(Z′\)=0\\textrm\{rank\}\(Z^\{\\prime\}\)=0\. Ifrank​\(Z′\)=1\\textrm\{rank\}\(Z^\{\\prime\}\)=1, then

Z′=u​vT,for​u,v∈ℝK\.Z^\{\\prime\}=uv^\{T\},\\quad\\textrm\{for \}u,v\\in\\mathbb\{R\}^\{K\}\.Hence our condition becomes

uc​vc−uc′​vc≥1\.u\_\{c\}v\_\{c\}\-u\_\{c^\{\\prime\}\}v\_\{c\}\\geq 1\.Clearlyvcv\_\{c\}is always non\-zero, else the condition is not satisfied\. ifvcv\_\{c\}is positive then

uc≥uc′\+1vc\>uc′,u\_\{c\}\\geq u\_\{c^\{\\prime\}\}\+\\frac\{1\}\{v\_\{c\}\}\>u\_\{c^\{\\prime\}\},and soucu\_\{c\}must be the unique largest element of the vectoruu\. Ifvcv\_\{c\}is negative then

uc<uc−1vc′≤uc′,u\_\{c\}<u\_\{c\}\-\\frac\{1\}\{v\_\{c^\{\\prime\}\}\}\\leq u\_\{c^\{\\prime\}\},and soucu\_\{c\}must be the unique smallest element of the vectoruu\.

Hence each entry ofuumust be either the largest or the smallest element of the vector in order forZ′Z^\{\\prime\}to satisfy the condition\. However at most two entries can be either the unique largest or the unique smallest entry\. Given thatK\>2K\>2this condition can hence never be satisfied, and so we must haverank​\(Z′\)≥2\\textrm\{rank\}\(Z^\{\\prime\}\)\\geq 2\. ∎

###### Lemma D\.3\.

Consider the following optimization problem forK≥3K\\geq 3

maxθ1,…,θK∈\(0,2​π\]⁡\{∑u=1Kcos⁡\(2​θu\)\},\\max\_\{\\theta\_\{1\},\.\.\.,\\theta\_\{K\}\\in\(0,2\\pi\]\}\\left\\\{\\sum\_\{u=1\}^\{K\}\\cos\(2\\theta\_\{u\}\)\\right\\\},subject to the condition that all angles are separated by at leastδ∈\(0,2​πK\]\\delta\\in\(0,\\frac\{2\\pi\}\{K\}\], including the wrap around value\. The attained value of this objective whenKKis even, orKKis odd andδ∈\(0,2​πK\+1\]\\delta\\in\(0,\\frac\{2\\pi\}\{K\+1\}\], is given by

sin⁡\(⌊K2⌋​δ\)\+sin⁡\(⌈K2⌉​δ\)sin⁡\(δ\),\\frac\{\\sin\\left\(\\lfloor\\frac\{K\}\{2\}\\rfloor\\delta\\right\)\+\\sin\\left\(\\lceil\\frac\{K\}\{2\}\\rceil\\delta\\right\)\}\{\\sin\(\\delta\)\},whilst for oddKKandδ∈\(2​πK\+1,2​πK\)\\delta\\in\(\\frac\{2\\pi\}\{K\+1\},\\frac\{2\\pi\}\{K\}\), it is given by

\|sin⁡\(K​δ\)\|sin⁡\(δ\)\.\\frac\{\|\\sin\(K\\delta\)\|\}\{\\sin\(\\delta\)\}\.

###### Proof\.

Note we can redefine the domain to be\(−π2,3​π2\]\(\-\\frac\{\\pi\}\{2\},\\frac\{3\\pi\}\{2\}\]without changing the value of the maximum\. Define the regionsC0=\(−π2,π2\]C\_\{0\}=\(\-\\frac\{\\pi\}\{2\},\\frac\{\\pi\}\{2\}\]andCπ=\(π2,3​π2\]C\_\{\\pi\}=\(\\frac\{\\pi\}\{2\},\\frac\{3\\pi\}\{2\}\]\. We begin by considering a configuration ofθ1,…,θK\\theta\_\{1\},\.\.\.,\\theta\_\{K\}withrrpoints inC0C\_\{0\}, denotedx1<…<xrx\_\{1\}<\.\.\.<x\_\{r\}, ands=K−rs=K\-rpoints inCπC\_\{\\pi\}, denotedy1<…<ysy\_\{1\}<\.\.\.<y\_\{s\}\.

Suppose that the configuration has a choice ofjjsuch thatxj\+1−xj\>δx\_\{j\+1\}\-x\_\{j\}\>\\delta\. Then clearly there exists some valueϵ\\epsilonsuch that we can move the furthest ofxj,xj\+1x\_\{j\},x\_\{j\+1\}byϵ\\epsilonin the direction of zero without breaking feasibility\. This must improve the objective, sincecos⁡\(2​θ\)\\cos\(2\\theta\)increases as you approach0within the setC0C\_\{0\}\. Hence in order for such a configuration to be optimal, we must havexj\+1−xj=δx\_\{j\+1\}\-x\_\{j\}=\\deltaforj=1,…,r−1j=1,\.\.\.,r\-1\. The exact same reasoning holds for theyy’s within the setCπC\_\{\\pi\}\. Hence any optimal configuration must have

xj=α\+\(j−r\+12\)​δ,yj=π\+β\+\(j−s\+12\)​δ,x\_\{j\}=\\alpha\+\\left\(j\-\\frac\{r\+1\}\{2\}\\right\)\\delta,\\qquad y\_\{j\}=\\pi\+\\beta\+\\left\(j\-\\frac\{s\+1\}\{2\}\\right\)\\delta,\(57\)where we have written this so that the two clusters are centered atα,π\+β\\alpha,\\pi\+\\betarespectively\. We have ensured such a configuration has feasibility withinC0C\_\{0\}andCπC\_\{\\pi\}\. To ensure feasibility between the two subsets, we consider the edge points ofC0,CπC\_\{0\},C\_\{\\pi\}\. The leftmost point inC0C\_\{0\}isα−r−12​δ\\alpha\-\\frac\{r\-1\}\{2\}\\deltaand the rightmost point inCπC\_\{\\pi\}isβ\+s−12​δ\\beta\+\\frac\{s\-1\}\{2\}\\delta\. Demanding the difference, after adding2​π2\\pito get the wrap around value, is at leastδ\\deltagives

α−β≥−\(π−K2​δ\)\.\\alpha\-\\beta\\geq\-\\left\(\\pi\-\\frac\{K\}\{2\}\\delta\\right\)\.Doing the same on the other side gives

α−β≤π−K2​δ,\\alpha\-\\beta\\leq\\pi\-\\frac\{K\}\{2\}\\delta,and so the condition for feasibility at the edge points can be stated as

\|β−α\|≤π−K2​δ\.\|\\beta\-\\alpha\|\\leq\\pi\-\\frac\{K\}\{2\}\\delta\.Returning to the objective value, the contribution from the points inC0C\_\{0\}is given by

∑j=1rcos⁡\(2​α\+2​j−\(r\+1\)​δ\)=Re​\(e2​i​α−\(r\+1\)​i​δ​∑j=1re2​i​j​δ\)\\sum\_\{j=1\}^\{r\}\\cos\(2\\alpha\+2j\-\(r\+1\)\\delta\)=\\textrm\{Re\}\\left\(e^\{2i\\alpha\-\(r\+1\)i\\delta\}\\sum\_\{j=1\}^\{r\}e^\{2ij\\delta\}\\right\)=Re​\(e2​i​α​e−\(r\+1\)​i​δ​e2​i​δ​\(e2​r​i​δ−1\)e2​i​δ−1\)=Re​\(e2​i​α​sin⁡\(r​δ\)sin⁡\(δ\)\)\.=\\textrm\{Re\}\\left\(e^\{2i\\alpha\}e^\{\-\(r\+1\)i\\delta\}\\frac\{e^\{2i\\delta\}\\left\(e^\{2ri\\delta\}\-1\\right\)\}\{e^\{2i\\delta\}\-1\}\\right\)=\\textrm\{Re\}\\left\(e^\{2i\\alpha\}\\frac\{\\sin\(r\\delta\)\}\{\\sin\(\\delta\)\}\\right\)\.The contribution from the points inCπC\_\{\\pi\}can be calculated similarly, giving a total objective value of

Re​\(e2​i​α​sin⁡\(r​δ\)sin⁡\(δ\)\+e2​i​β​sin⁡\(s​δ\)sin⁡\(δ\)\)\.\\textrm\{Re\}\\left\(e^\{2i\\alpha\}\\frac\{\\sin\(r\\delta\)\}\{\\sin\(\\delta\)\}\+e^\{2i\\beta\}\\frac\{\\sin\(s\\delta\)\}\{\\sin\(\\delta\)\}\\right\)\.\(58\)Note this is clearly upper bounded by the modulus, where a global rotation allows for attainment of this bound, so we can rewrite this as

=\|sin⁡\(r​δ\)sin⁡\(δ\)\+e2​i​\(β−α\)​sin⁡\(s​δ\)sin⁡\(δ\)\|\.=\\left\|\\frac\{\\sin\(r\\delta\)\}\{\\sin\(\\delta\)\}\+e^\{2i\(\\beta\-\\alpha\)\}\\frac\{\\sin\(s\\delta\)\}\{\\sin\(\\delta\)\}\\right\|\.Define the quantityν=2​\(β−α\)\\nu=2\(\\beta\-\\alpha\)\. Clearly if bothsin⁡\(r​δ\)\\sin\(r\\delta\)andsin⁡\(s​δ\)\\sin\(s\\delta\)are non\-negative, then the optimal value isν=0\\nu=0\. We call this Case 1, and it has objective value

sin⁡\(r​δ\)\+sin⁡\(s​δ\)sin⁡\(δ\)\.\\frac\{\\sin\(r\\delta\)\+\\sin\(s\\delta\)\}\{\\sin\(\\delta\)\}\.Note that at most one ofsin⁡\(r​δ\)\\sin\(r\\delta\)andsin⁡\(s​δ\)\\sin\(s\\delta\)is negative, since we requirer​δ≥πr\\delta\\geq\\piors​δ≥πs\\delta\\geq\\pifor this to happen, which sinceδ≥2​πK\\delta\\geq\\frac\{2\\pi\}\{K\}, requires as a necessary conditionrrorssis greater thanK2\\frac\{K\}\{2\}, which can only happen for at most one\. We call the case where one of them is negative Case 2\. In this case our objective value is

1sin⁡\(δ\)​\(sin2⁡\(r​δ\)\+sin2⁡\(s​δ\)\+2​sin⁡\(r​δ\)​sin⁡\(s​δ\)​cos⁡\(ν\)\)12\.\\frac\{1\}\{\\sin\(\\delta\)\}\\left\(\\sin^\{2\}\(r\\delta\)\+\\sin^\{2\}\(s\\delta\)\+2\\sin\(r\\delta\)\\sin\(s\\delta\)\\cos\(\\nu\)\\right\)^\{\\frac\{1\}\{2\}\}\.We must haveδ\>πK\\delta\>\\frac\{\\pi\}\{K\}for this to happen, sincer,s≤Kr,s\\leq K, then from the feasibility condition we have\|ν\|≤π\|\\nu\|\\leq\\pi\. Hence, the objective is maximized when\|ν\|\|\\nu\|is at its upper bound, and so\|ν\|=2​π−K​δ\|\\nu\|=2\\pi\-K\\delta\. Then using standard trigonometric identities, this case reduces to an objective value of

\|sin⁡\(K​δ\)\|sin⁡\(δ\)\.\\frac\{\|\\sin\(K\\delta\)\|\}\{\\sin\(\\delta\)\}\.We have now calculated the best objective value for fixedrrconditional on the sign of the sine terms, it remains to optimizerr\. First we optimize among Case 1:

sin⁡\(r​δ\)\+sin⁡\(s​δ\)=2​sin⁡\(K2​δ\)​cos⁡\(2​r−K2​δ\)\.\\sin\(r\\delta\)\+\\sin\(s\\delta\)=2\\sin\\left\(\\frac\{K\}\{2\}\\delta\\right\)\\cos\\left\(\\frac\{2r\-K\}\{2\}\\delta\\right\)\.Since we have0<K2​δ≤π0<\\frac\{K\}\{2\}\\delta\\leq\\pi, thesin\\sinterm is non\-negative\. We also have−π≤2​r−K2​δ≤π\-\\pi\\leq\\frac\{2r\-K\}\{2\}\\delta\\leq\\pi, and so the cosine term is maximized when2​r−K2r\-Kis closest to zero, meaningr=⌊K2⌋r=\\lfloor\\frac\{K\}\{2\}\\rfloor\. This solution:r=⌊K2⌋r=\\lfloor\\frac\{K\}\{2\}\\rfloor,s=⌈K2⌉s=\\lceil\\frac\{K\}\{2\}\\rceilmaintains non\-negativity of bothsin⁡\(r​δ\)\\sin\(r\\delta\)andsin⁡\(s​δ\)\\sin\(s\\delta\)provided⌈K2⌉​δ≤π\\lceil\\frac\{K\}\{2\}\\rceil\\delta\\leq\\pi\. Assuming this is the case, we show Case 2 cannot beat Case 1\. The Case 2 objective value is

\|sin⁡\(K​δ\)\|=2​sin⁡\(K2​δ\)​\|cos⁡\(K​δ2\)\|\.\|\\sin\(K\\delta\)\|=2\\sin\\left\(\\frac\{K\}\{2\}\\delta\\right\)\\left\|\\cos\\left\(\\frac\{K\\delta\}\{2\}\\right\)\\right\|\.For evenKKwe have

2​sin⁡\(K2​δ\)​\|cos⁡\(K​δ2\)\|≤2​sin⁡\(K2​δ\),2\\sin\\left\(\\frac\{K\}\{2\}\\delta\\right\)\\left\|\\cos\\left\(\\frac\{K\\delta\}\{2\}\\right\)\\right\|\\leq 2\\sin\\left\(\\frac\{K\}\{2\}\\delta\\right\),and so Case 1 beats Case 2 for evenKK\. For oddKKwhen⌈K2⌉​δ≤π\\lceil\\frac\{K\}\{2\}\\rceil\\delta\\leq\\piwe have

2​sin⁡\(K2​δ\)​\|cos⁡\(K​δ2\)\|≤2​sin⁡\(K2​δ\)​cos⁡\(12​δ\)=sin⁡\(⌊K2⌋​δ\)\+sin⁡\(⌈K2⌉​δ\),2\\sin\\left\(\\frac\{K\}\{2\}\\delta\\right\)\\left\|\\cos\\left\(\\frac\{K\\delta\}\{2\}\\right\)\\right\|\\leq 2\\sin\\left\(\\frac\{K\}\{2\}\\delta\\right\)\\cos\\left\(\\frac\{1\}\{2\}\\delta\\right\)=\\sin\\left\(\\left\\lfloor\\frac\{K\}\{2\}\\right\\rfloor\\delta\\right\)\+\\sin\\left\(\\left\\lceil\\frac\{K\}\{2\}\\right\\rceil\\delta\\right\),where we used that0≤12​δ<K2​δ≤⌈K2⌉​δ≤π0\\leq\\frac\{1\}\{2\}\\delta<\\frac\{K\}\{2\}\\delta\\leq\\lceil\\frac\{K\}\{2\}\\rceil\\delta\\leq\\pi\. Hence for all cases wherer=⌊K2⌋​δr=\\lfloor\\frac\{K\}\{2\}\\rfloor\\deltaleads to positivesin⁡\(r​δ\)\\sin\(r\\delta\)andsin⁡\(s​δ\)\\sin\(s\\delta\), we have that the best objective value is

1sin⁡\(δ\)​\(sin⁡\(⌊K2⌋​δ\)\+sin⁡\(⌈K2⌉​δ\)\)\.\\frac\{1\}\{\\sin\(\\delta\)\}\\left\(\\sin\\left\(\\left\\lfloor\\frac\{K\}\{2\}\\right\\rfloor\\delta\\right\)\+\\sin\\left\(\\left\\lceil\\frac\{K\}\{2\}\\right\\rceil\\delta\\right\)\\right\)\.The only case where we do not get both sine terms positive is when⌈K2⌉​δ\>π\\lceil\\frac\{K\}\{2\}\\rceil\\delta\>\\pi, which occurs whenKKis odd andδ∈\(2​πK\+1,2​πK\]\\delta\\in\\left\(\\frac\{2\\pi\}\{K\+1\},\\frac\{2\\pi\}\{K\}\\right\]\. In this case any split produces a negative sine term, and so the maximum must instead occur for Case 2, meaning we get the alternative optimal value of

1sin⁡\(δ\)​\|sin⁡\(K​δ\)\|\.\\frac\{1\}\{\\sin\(\\delta\)\}\\left\|\\sin\(K\\delta\)\\right\|\.
∎

###### Lemma D\.4\(From Theorem 1 ofGarrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)\)\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}\. SupposeU=1K​ΦU=\\frac\{1\}\{\\sqrt\{K\}\}\\Phi, whereΦ\\Phiis theK×KK\\times KSylvester Hadamard matrix\. Denote the columns ofUUbyuiu\_\{i\},i=1,…,Ki=1,\\dots,K\. For a matrix of the formZ=∑iai​ui​uiTZ=\\sum\_\{i\}a\_\{i\}u\_\{i\}u\_\{i\}^\{T\}, the corresponding softmax matrix is

P​\(Z\)=∑iνi​ui​uiT,P\(Z\)=\\sum\_\{i\}\\nu\_\{i\}u\_\{i\}u\_\{i\}^\{T\},where

νi=∑jΦi​j​e1K​\(Φ​a\)j∑je1K​\(Φ​a\)j\.\\nu\_\{i\}=\\frac\{\\sum\_\{j\}\\Phi\_\{ij\}e^\{\\frac\{1\}\{K\}\(\\Phi a\)\_\{j\}\}\}\{\\sum\_\{j\}e^\{\\frac\{1\}\{K\}\(\\Phi a\)\_\{j\}\}\}\.

###### Lemma D\.5\(From Lemma 3\.6Horadam \([2007](https://arxiv.org/html/2605.23087#bib.bib110)\)\)\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}\. The columns of theK×KK\\times KSylvester Hadamard matrixΦ\\Phi, denoted byϕi\\phi\_\{i\}fori=0,…,K−1i=0,\.\.\.,K\-1, satisfy

ϕi∘ϕj=ϕi⊕j\\phi\_\{i\}\\circ\\phi\_\{j\}=\\phi\_\{i\\oplus j\}where∘\\circdenotes the Hadamard product, and⊕\\oplusdenotes the bitwise XOR\.

###### Lemma D\.6\(From Exercise 1\.12 ofO’Donnell \([2014](https://arxiv.org/html/2605.23087#bib.bib109)\)\)\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}\. The entries of theK×KK\\times KSylvester Hadamard matrixΦ\\Phi, are given by

Φi​j=\(−1\)i⋅j\\Phi\_\{ij\}=\(\-1\)^\{i\\cdot j\}where⋅\\cdotdenotes the mod\-2 dot product, acting on the binary representations ofiiandjj\.

###### Proof\.

SinceO’Donnell \([2014](https://arxiv.org/html/2605.23087#bib.bib109)\)does not feature a proof we include a proof by induction onm∈ℕm\\in\\mathbb\{N\}here for completeness\.

First form=1m=1, and denoting the Sylvester Hadamard matrix byΦK\\Phi\_\{K\}to make clear the dimension, we have

\(Φ2\)=\{−1if​i=j=11otherwise=\(−1\)i⋅j\.\(\\Phi\_\{2\}\)=\\begin\{cases\}\-1&\\textrm\{if \}i=j=1\\\\ 1&\\textrm\{otherwise\}\\end\{cases\}=\(\-1\)^\{i\\cdot j\}\.Hence, the result holds form=1m=1\. Now take it true form∈ℕm\\in\\mathbb\{N\}, and consider them\+1m\+1case\. We can writei,j∈𝔽2m\+1i,j\\in\\mathbb\{F\}^\{m\+1\}\_\{2\}asi=\[i′;x\]i=\[i^\{\\prime\};x\],j=\[j′;y\]j=\[j^\{\\prime\};y\], where we have concatenated the extra index so thati′,j′∈𝔽2mi^\{\\prime\},j^\{\\prime\}\\in\\mathbb\{F\}^\{m\}\_\{2\}andx,y∈𝔽2x,y\\in\\mathbb\{F\}\_\{2\}\. Now using the properties of the Hadamard matrix construction, we have:

\(Φ2m\+1\)i​j=\{−\(Φ2m\)i′​j′if​x=y=1\(Φ2m\)i′​j′otherwise=\(Φ2m\)i′​j′​\(−1\)x⋅y=\(−1\)i′⋅j′​\(−1\)x⋅y=\(−1\)i⋅j,\(\\Phi\_\{2^\{m\+1\}\}\)\_\{ij\}=\\begin\{cases\}\-\(\\Phi\_\{2^\{m\}\}\)\_\{i^\{\\prime\}j^\{\\prime\}\}&\\textrm\{if \}x=y=1\\\\ \(\\Phi\_\{2^\{m\}\}\)\_\{i^\{\\prime\}j^\{\\prime\}\}&\\textrm\{otherwise\}\\end\{cases\}=\(\\Phi\_\{2^\{m\}\}\)\_\{i^\{\\prime\}j^\{\\prime\}\}\(\-1\)^\{x\\cdot y\}=\(\-1\)^\{i^\{\\prime\}\\cdot j^\{\\prime\}\}\(\-1\)^\{x\\cdot y\}=\(\-1\)^\{i\\cdot j\},Which gives the result\.

∎

###### Lemma D\.7\(From Corollary 3 ofShanget al\.\([2020](https://arxiv.org/html/2605.23087#bib.bib117)\)\)\.

LetWL∈ℝK×dW\_\{L\}\\in\\mathbb\{R\}^\{K\\times d\},WL−1,…,W1∈ℝd×dW\_\{L\-1\},\.\.\.,W\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\},H1∈ℝd×K​nH\_\{1\}\\in\\mathbb\{R\}^\{d\\times Kn\}, whered≥Kd\\geq K\. The minimization of the sum of their Frobenius norms squared, subject to a known product, is given by

minWL,…,W1,H1​such that​Z=WL​…​W1​H1⁡\{‖H1‖F2\+∑l=1L‖Wl‖F2\}=\(L\+1\)​‖Z‖S2L\+12L\+1,\\min\_\{W\_\{L\},\.\.\.,W\_\{1\},H\_\{1\}\\textrm\{ such that \}Z=W\_\{L\}\.\.\.W\_\{1\}H\_\{1\}\}\\left\\\{\\\|H\_\{1\}\\\|\_\{F\}^\{2\}\+\\sum\_\{l=1\}^\{L\}\\\|W\_\{l\}\\\|\_\{F\}^\{2\}\\right\\\}=\(L\+1\)\\,\\\|Z\\\|\_\{S\_\{\\frac\{2\}\{L\+1\}\}\}^\{\\frac\{2\}\{L\+1\}\},where∥⋅∥Spp\\\|\\cdot\\\|\_\{S\_\{p\}\}^\{p\}denotes the Schatten\-quasi norm with parameterpp, given by

‖M‖Spp=∑i=1rank​\(M\)σip,\\\|M\\\|\_\{S\_\{p\}\}^\{p\}=\\sum\_\{i=1\}^\{\\textrm\{rank\}\(M\)\}\\sigma\_\{i\}^\{p\},\(59\)whereσ1,…,σrank​\(M\)\\sigma\_\{1\},\.\.\.,\\sigma\_\{\\textrm\{rank\}\(M\)\}are the non\-zero singular values of the matrixMM\.

###### Lemma D\.8\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}, and leta∈ℝK−1a\\in\\mathbb\{R\}^\{K\-1\}be of the forma=\[γ,δ,…,γ,δ\]a=\[\\gamma,\\delta,\.\.\.,\\gamma,\\delta\], so thatγ\\gammaappears inK2\\frac\{K\}\{2\}entries andδ\\deltaappears inK2−1\\frac\{K\}\{2\}\-1entries, with the two alternating, and we haveγ,δ\>0\\gamma,\\delta\>0\. Then

\(Ψ​e−Ψ​a\)i=\{2​e−K​γ\+\(K−2\)​e−K2​\(γ\+δ\),if​i​is odd,K​e−K2​\(γ\+δ\),if​i​is even\.\(\\Psi e^\{\-\\Psi a\}\)\_\{i\}=\\begin\{cases\}2e^\{\-K\\gamma\}\+\(K\-2\)e^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\},&\\textrm\{if \}i\\textrm\{ is odd\},\\\\ Ke^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\},&\\textrm\{if \}i\\textrm\{ is even\}\.\\end\{cases\}

###### Proof\.

begin by writing the vectoraaasa=δ​1K−1\+\(γ−δ\)​ca=\\delta 1\_\{K\-1\}\+\(\\gamma\-\\delta\)c, where the vectorcchas 1 in entries with odd indices, and 0 for even indices\. Sincecccorresponds with the first column ofΨ\\Psi, up to a factor of two, the properties of the underlying Hadamard matrix give that

Ψ​c=K2​e1\+K2​1K−1,\\Psi c=\\frac\{K\}\{2\}e\_\{1\}\+\\frac\{K\}\{2\}1\_\{K\-1\},wheree1e\_\{1\}is the first basis vector inℝK−1\\mathbb\{R\}^\{K\-1\}\. Consequently

Ψ​a=K2​\(γ\+δ\)​1\+K2​\(γ−δ\)​e1,\\Psi a=\\frac\{K\}\{2\}\(\\gamma\+\\delta\)1\+\\frac\{K\}\{2\}\(\\gamma\-\\delta\)e\_\{1\},and hence

exp⁡\(−\(Ψ​a\)1\)=e−K​γ⏟α,exp⁡\(−\(Ψ​a\)i\)=e−K2​\(γ\+δ\)⏟β,i=2,…,K−1,\\exp\(\-\(\\Psi a\)\_\{1\}\)=\\underbrace\{e^\{\-K\\gamma\}\}\_\{\\alpha\},\\quad\\exp\(\-\(\\Psi a\)\_\{i\}\)=\\underbrace\{e^\{\-\\frac\{K\}\{2\}\(\\gamma\+\\delta\)\}\}\_\{\\beta\},\\quad i=2,\.\.\.,K\-1,where we have denote these quantities byα\\alphaandβ\\beta\. This can then be written as

exp⁡\(−Ψ​a\)=\(α−β\)​e1\+β​1K−1\.\\exp\(\-\\Psi a\)=\(\\alpha\-\\beta\)e\_\{1\}\+\\beta 1\_\{K\-1\}\.Again noting that the first column ofΨ\\Psiis2​c2c, we have thatΨ​e1=2​c\\Psi e\_\{1\}=2c, and so

Ψ​exp⁡\(−Ψ​a\)=2​\(α−β\)​c\+K​β​1K−1\.\\Psi\\exp\(\-\\Psi a\)=2\(\\alpha\-\\beta\)c\+K\\beta 1\_\{K\-1\}\.Using the explicit forms ofα\\alphaandβ\\betathen gives the result\. ∎

###### Lemma D\.9\.

LetK=2mK=2^\{m\}form∈ℕm\\in\\mathbb\{N\}\. Denote by the vectorc∈ℝK−1c\\in\\mathbb\{R\}^\{K\-1\}the vector that takes value one in its odd indices, and zero in its even indices\. ifa∈ℝK−1a\\in\\mathbb\{R\}^\{K\-1\}can be written in the form

a=μ​c\+μ​ϵ,μ=2K​‖a‖1∈ℝ,ϵ∈ℝK−1,∑iϵi=0,a=\\mu c\+\\mu\\epsilon,\\quad\\mu=\\frac\{2\}\{K\}\\\|a\\\|\_\{1\}\\in\\mathbb\{R\},\\quad\\epsilon\\in\\mathbb\{R\}^\{K\-1\},\\quad\\sum\_\{i\}\\epsilon\_\{i\}=0,then to linear order in‖ϵ‖1\\\|\\epsilon\\\|\_\{1\}, we have

Ψ​exp⁡\(−Ψ​a\)=K​e−K​μ2​1K−1−K​e−K​μ2​μ​ϵ\+2​\(e−K​μ−e−K​μ2\)​\(1−μ​\(Ψ​ϵ\)1\)​c\+O​\(‖ϵ‖12\)\.\\Psi\\exp\(\-\\Psi a\)=Ke^\{\\frac\{\-K\\mu\}\{2\}\}1\_\{K\-1\}\-Ke^\{\-\\frac\{K\\mu\}\{2\}\}\\mu\\epsilon\+2\\left\(e^\{\-K\\mu\}\-e^\{\-\\frac\{K\\mu\}\{2\}\}\\right\)\(1\-\\mu\(\\Psi\\epsilon\)\_\{1\}\)c\+O\\left\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\\right\)\.

###### Proof\.

Using thatΨ​c=K2​\(e1\+1K−1\)\\Psi c=\\frac\{K\}\{2\}\(e\_\{1\}\+1\_\{K\-1\}\), wheree1e\_\{1\}is the first basis vector inℝK−1\\mathbb\{R\}^\{K\-1\}, we have

Ψ​a=K​μ2​\(e1\+1K−1\)\+μ​\(Ψ​ϵ\)\.\\Psi a=\\frac\{K\\mu\}\{2\}\(e\_\{1\}\+1\_\{K\-1\}\)\+\\mu\(\\Psi\\epsilon\)\.Hence

\(exp⁡\(−Ψ​a\)\)i=\{e−K​μ​e−μ​\(Ψ​ϵ\)1,if​i=1,e−K2​μ​e−μ​\(Ψ​ϵ\)i,otherwise\.,\(\\exp\(\-\\Psi a\)\)\_\{i\}=\\begin\{cases\}e^\{\-K\\mu\}e^\{\-\\mu\(\\Psi\\epsilon\)\_\{1\}\},&\\textrm\{if \}i=1,\\\\ e^\{\-\\frac\{K\}\{2\}\\mu\}e^\{\-\\mu\(\\Psi\\epsilon\)\_\{i\}\},&\\textrm\{otherwise\}\.\\end\{cases\},which we can write as

\(exp⁡\(−Ψ​a\)\)i=\[e−K2​μ​1K−1\+\(e−K​μ−e−K2​μ\)​e1\]i​e−μ​\(Ψ​ϵ\)i\.\(\\exp\(\-\\Psi a\)\)\_\{i\}=\\left\[e^\{\-\\frac\{K\}\{2\}\\mu\}1\_\{K\-1\}\+\(e^\{\-K\\mu\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\)e\_\{1\}\\right\]\_\{i\}e^\{\-\\mu\(\\Psi\\epsilon\)\_\{i\}\}\.Now expand the exponential to linear order in‖ϵ‖1\\\|\\epsilon\\\|\_\{1\}

=\[e−K2​μ​1K−1\+\(e−K​μ−e−K2​μ\)​e1\]i−e−K2​μ​μ​\(Ψ​ϵ\)i−\(e−K​μ−e−K2​μ\)​μ​\(e1\)i​\(Ψ​ϵ\)i\+O​\(‖ϵ‖12\)\.=\\left\[e^\{\-\\frac\{K\}\{2\}\\mu\}1\_\{K\-1\}\+\(e^\{\-K\\mu\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\)e\_\{1\}\\right\]\_\{i\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\\mu\(\\Psi\\epsilon\)\_\{i\}\-\(e^\{\-K\\mu\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\)\\mu\(e\_\{1\}\)\_\{i\}\(\\Psi\\epsilon\)\_\{i\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.Next use that\(e1\)i​\(Ψ​ϵ\)i=\(Ψ​ϵ\)1​\(e1\)i\(e\_\{1\}\)\_\{i\}\(\\Psi\\epsilon\)\_\{i\}=\(\\Psi\\epsilon\)\_\{1\}\(e\_\{1\}\)\_\{i\}, and group thee1e\_\{1\}terms, giving

exp⁡\(−Ψ​a\)=e−K2​μ​1K−1−e−K2​μ​μ​\(Ψ​ϵ\)\+\(e−K​μ−e−K2​μ\)​\(1−μ​\(Ψ​ϵ\)1\)​e1\+O​\(‖ϵ‖12\)\.\\exp\(\-\\Psi a\)=e^\{\-\\frac\{K\}\{2\}\\mu\}1\_\{K\-1\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\\mu\(\\Psi\\epsilon\)\+\(e^\{\-K\\mu\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\)\(1\-\\mu\(\\Psi\\epsilon\)\_\{1\}\)e\_\{1\}\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.Then using thatΨ​1K−1=K​1K−1\\Psi 1\_\{K\-1\}=K1\_\{K\-1\},Ψ​e1=2​c\\Psi e\_\{1\}=2candΨ2​ϵ=K​ϵ\\Psi^\{2\}\\epsilon=K\\epsilon, gives that

Ψ​exp⁡\(−Ψ​a\)=K​e−K2​μ​1K−1−K​e−K2​μ​μ​ϵ\+2​\(e−K​μ−e−K2​μ\)​\(1−μ​\(Ψ​ϵ\)1\)​c\+O​\(‖ϵ‖12\)\.\\Psi\\exp\(\-\\Psi a\)=Ke^\{\-\\frac\{K\}\{2\}\\mu\}1\_\{K\-1\}\-Ke^\{\-\\frac\{K\}\{2\}\\mu\}\\mu\\epsilon\+2\(e^\{\-K\\mu\}\-e^\{\-\\frac\{K\}\{2\}\\mu\}\)\(1\-\\mu\(\\Psi\\epsilon\)\_\{1\}\)c\+O\(\\\|\\epsilon\\\|\_\{1\}^\{2\}\)\.∎

###### Lemma D\.10\.

LetX∈ℝd×vX\\in\\mathbb\{R\}^\{d\\times v\}be a random matrix independent ofW∈ℝd×dW\\in\\mathbb\{R\}^\{d\\times d\}, where the entries ofWWare i\.i\.d\.N​\(0,1/d\)N\(0,1/d\)\. If we have the following convergence in probability in thed→∞d\\to\\inftylimit, wherevvis fixed

then we also have

XT​WT​W​X→Iv\.X^\{T\}W^\{T\}WX\\to I\_\{v\}\.

###### Proof\.

Note it is simple to show that𝔼​\[WT​W\]=Id\\mathbb\{E\}\[W^\{T\}W\]=I\_\{d\}, and from this we clearly have

𝔼​\[XT​WT​W​X\|X\]=XT​X\.\\mathbb\{E\}\[X^\{T\}W^\{T\}WX\|X\]=X^\{T\}X\.We can now calculate the conditional variances\. Write the columns ofXXasx1,…,xmx\_\{1\},\.\.\.,x\_\{m\}, then

\(XT​WT​W​X\)i​j=\(W​xi\)T​\(W​xj\)=∑u=1d\(wuT​xi\)​\(wuT​xj\),\(X^\{T\}W^\{T\}WX\)\_\{ij\}=\(Wx\_\{i\}\)^\{T\}\(Wx\_\{j\}\)=\\sum\_\{u=1\}^\{d\}\(w\_\{u\}^\{T\}x\_\{i\}\)\(w\_\{u\}^\{T\}x\_\{j\}\),where we denote the rows ofWWbywuTw\_\{u\}^\{T\}\.

Conditional onXX, the rowswrw\_\{r\}are independent Gaussian vectors, so the summands are independent\. Hence:

Var​\[\(XT​WT​W​X\)i​j\|X\]=d​Var​\[\(w1T​xi\)​\(w1T​xj\)\|X\]\.\\textrm\{Var\}\[\(X^\{T\}W^\{T\}WX\)\_\{ij\}\|X\]=d\\ \\textrm\{Var\}\[\(w\_\{1\}^\{T\}x\_\{i\}\)\(w\_\{1\}^\{T\}x\_\{j\}\)\|X\]\.Denote the variablesA=w1T​xi,B=w1T​xjA=w\_\{1\}^\{T\}x\_\{i\},B=w\_\{1\}^\{T\}x\_\{j\}\. Now using standard properties of centered Gaussians we have

Var​\(A\)=1d​xiT​xi,Var​\(B\)=1d​xjT​xj,Cov​\(A,B\)=1d​xiT​xj,\\textrm\{Var\}\(A\)=\\frac\{1\}\{d\}x\_\{i\}^\{T\}x\_\{i\},\\quad\\textrm\{Var\}\(B\)=\\frac\{1\}\{d\}x\_\{j\}^\{T\}x\_\{j\},\\quad\\textrm\{Cov\}\(A,B\)=\\frac\{1\}\{d\}x\_\{i\}^\{T\}x\_\{j\},along with the fact that for centered jointly Gaussian variables we have

Var​\(U​V\)=Var​\(U\)​Var​\(V\)\+Cov​\(U,V\)2,\\textrm\{Var\}\(UV\)=\\textrm\{Var\}\(U\)\\textrm\{Var\}\(V\)\+\\textrm\{Cov\}\(U,V\)^\{2\},gives

Var​\(\(XT​WT​W​X\)i​j\|X\)=1d​\[\(XT​X\)i​i​\(XT​X\)j​j\+\(XT​X\)i​j2\]\.\\textrm\{Var\}\(\(X^\{T\}W^\{T\}WX\)\_\{ij\}\|X\)=\\frac\{1\}\{d\}\[\(X^\{T\}X\)\_\{ii\}\(X^\{T\}X\)\_\{jj\}\+\(X^\{T\}X\)\_\{ij\}^\{2\}\]\.Then sincevvis fixed, andXT​X→IX^\{T\}X\\to Iin probability, we have that the conditional variance converges to zero in probability asd→∞d\\to\\infty\. Chebyshev’s inequality applied entrywise then gives

XT​WT​W​X−XT​X→0,X^\{T\}W^\{T\}WX\-X^\{T\}X\\to 0,and therefore

XT​WT​W​X→I\.X^\{T\}W^\{T\}WX\\to I\.
∎

###### Lemma D\.11\.

LetK,n,L∈ℕK,n,L\\in\\mathbb\{N\}and let the matricesWL∈ℝK×dW\_\{L\}\\in\\mathbb\{R\}^\{K\\times d\},Wl∈ℝd×dW\_\{l\}\\in\\mathbb\{R\}^\{d\\times d\},W0∈ℝd×K​nW\_\{0\}\\in\\mathbb\{R\}^\{d\\times Kn\}be initialized with entries sampled i\.i\.d\. from a Gaussian with mean 0 and variance1/d1/d\. Define the matrices

Al\+1=WL​…​Wl\+1,l=0,…,L−1,Hl=Wl−1​…​W1​H1,l=1,…,L\.A\_\{l\+1\}=W\_\{L\}\.\.\.W\_\{l\+1\},\\quad l=0,\.\.\.,L\-1,\\quad H\_\{l\}=W\_\{l\-1\}\.\.\.W\_\{1\}H\_\{1\},\\quad l=1,\.\.\.,L\.Then we have the following convergence in probability in thed→∞d\\to\\inftylimit, withK,n,LK,n,Lfixed

Al\+1​Al\+1T→IK,HlT​Hl→IK​n\.A\_\{l\+1\}A\_\{l\+1\}^\{T\}\\rightarrow I\_\{K\},\\quad H\_\{l\}^\{T\}H\_\{l\}\\rightarrow I\_\{Kn\}\.

###### Proof\.

We shall show forHlH\_\{l\}, the proof forAl\+1A\_\{l\+1\}is almost identical\. We apply Lemma[D\.10](https://arxiv.org/html/2605.23087#A4.Thmtheorem10)iteratively\. It is simple to showH1∈ℝd×K​nH\_\{1\}\\in\\mathbb\{R\}^\{d\\times Kn\}satisfiesH1T​H1→IH\_\{1\}^\{T\}H\_\{1\}\\to Iin probability\. Then applying Lemma[D\.10](https://arxiv.org/html/2605.23087#A4.Thmtheorem10)withv=K​nv=Knand first withX=H1,W=W1X=H\_\{1\},W=W\_\{1\}, thenX=W1​H1,W=W2X=W\_\{1\}H\_\{1\},W=W\_\{2\}, so on, gives

HlT​Hl=\(Wl−1​…​W1​H1\)T​\(Wl−1​…​W1​H1\)→IK​n,H\_\{l\}^\{T\}H\_\{l\}=\(W\_\{l\-1\}\.\.\.W\_\{1\}H\_\{1\}\)^\{T\}\(W\_\{l\-1\}\.\.\.W\_\{1\}H\_\{1\}\)\\to I\_\{Kn\},forl=1,…,Ll=1,\.\.\.,L\. ∎

## Appendix EFurther Numerical Experiments

### E\.1Main Text Experimental Details

![Refer to caption](https://arxiv.org/html/2605.23087v1/x9.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x10.png)

Figure 7:Demonstrations of different normalized mean logit matrices after training forL=3,K=10L=3,K=10\.Left:logits that are approximately DNC, with effective rank8\.9≈K−18\.9\\approx K\-1\.Right:logits of a low rank alternative, with effective rank4\.34\.3\.In this section we provide further details for the plots of the main text\.

Figure[1](https://arxiv.org/html/2605.23087#S3.F1)is produced by computing the matrixZ~\\tilde\{Z\}defined in Eq\. \([4](https://arxiv.org/html/2605.23087#S3.E4)\) for hyperparametersK=4K=4,L=3L=3, andn=1n=1, for both the DNC solution \(Definition[2\.1](https://arxiv.org/html/2605.23087#S2.Thmtheorem1)\) and a canonical low\-rank solution \(Eq\. \([14](https://arxiv.org/html/2605.23087#A4.E14)\)\)\. The angles are computed by taking dot products between columns, and the logit distances by taking the difference between diagonal and off\-diagonal elements\. In the low\-rank case there are multiple values for the angular and logit distances; we consider only the smallest, since this is the one that determines the loss at large norm\. These quantities are then plotted in the same hyperplane to enable a direct comparison of distances\.

Figure[2](https://arxiv.org/html/2605.23087#S3.F2)is produced by running deep UFM simulations from random initialization withK=10K=10,n=5n=5,d=100d=100, and a range of network depthsLL\. We report the effective rank, a quantity that aims to approximate the hard rank in settings where singular values are not identically zero\. The effective rank is defined as

Reff​\(M\)=exp⁡\(−∑i=1rank​\(M\)pi​log⁡\(pi\)\),where​pi=σi∑j=1rank​\(M\)σj,R\_\{\\textrm\{eff\}\}\(M\)=\\exp\\left\(\-\\sum\_\{i=1\}^\{\\textrm\{rank\}\(M\)\}p\_\{i\}\\log\(p\_\{i\}\)\\right\),\\quad\\textrm\{where \}p\_\{i\}=\\frac\{\\sigma\_\{i\}\}\{\\sum\_\{j=1\}^\{\\textrm\{rank\}\(M\)\}\\sigma\_\{j\}\},\(60\)andσ1,…,σrank​\(M\)\\sigma\_\{1\},\.\.\.,\\sigma\_\{\\textrm\{rank\}\(M\)\}are the non\-zero singular values of the matrixMM\.

For each value ofLLwe conduct five runs, and we plot the mean effective rank with one\-standard\-deviation error bars\.

The first two plots of Figure[3](https://arxiv.org/html/2605.23087#S3.F3)show experiments conducted for a range of class numbersKK, with network depthL=10L=10, network widthd=Kd=K,n=5n=5, and learning rate0\.080\.08\. To aid convergence, the networks were initially trained with a small amount of regularization,λ=10−3\\lambda=10^\{\-3\}, for2×1052\\times 10^\{5\}epochs, followed by a further1×1051\\times 10^\{5\}epochs with no regularization\. The Gram factorXXis obtained using the SVD of the mean logit matrix\. For the robustness experiment, we trained five models for which rank22was observed\. SinceZZwas symmetric positive semidefinite of rank22,XXwas computed via its singular value decomposition \(note thatXXis unique up to rotation\)\. We then computed the angles between adjacent feature means and, across the five runs, reported the maximum and minimum angles observed in each case\.

The final plot in Figure[3](https://arxiv.org/html/2605.23087#S3.F3)shows the normalized margins for a range of class numbers and ranks, forL=4L=4andn=5n=5, overlaid with the same quantity forL=1L=1bottlenecked UFMs\. To obtain a range of ranks, we used a variety of network widths and a new random seed for each run\. Due to the non\-benign loss landscape, different ranks can arise even for the same network width\. Results were logged throughout training, and when the same rank appeared across multiple runs we report the average\. To produce the bottlenecked cases, we trained anL=1L=1UFM with network widthr=2,…,K−1r=2,\\ldots,K\-1for eachKK\. In both sets of experiments we used a small amount of regularization,λ=10−3\\lambda=10^\{\-3\}, for the first2×1052\\times 10^\{5\}epochs, followed by unregularized training for1×1051\\times 10^\{5\}epochs\. The normalized margins are the smallest gaps between the correct normalized logit value and any other logit value for the same datapoint, taken over the entire dataset\.

Figure[4](https://arxiv.org/html/2605.23087#S4.F4)is produced by evolving a vectoraaunder the Hadamard dynamics of Eq\. \([6](https://arxiv.org/html/2605.23087#S4.E6)\) forK=16K=16andL=2L=2\. The vector is initialized with entries sampled uniformly from\[0,1\]\[0,1\], and then renormalized to haveL1L\_\{1\}norm10−310^\{\-3\}\. We show a single training run for simplicity, but the dynamics exhibit the same phenomenology under this setup across a large number of random initializations\.

Figure[5](https://arxiv.org/html/2605.23087#S4.F5)shows the impact of width on the early dynamics ofZZ\. For the first plot, the network parameters are as described in Theorem[4\.5](https://arxiv.org/html/2605.23087#S4.Thmtheorem5), withϵ=0\.01\\epsilon=0\.01,K=10K=10,L=3L=3,n=5n=5, and a range of widthsdd\. The logit derivative is computed using Eq\. \([56](https://arxiv.org/html/2605.23087#A4.E56)\)\. The reported metric is the normalized squared Frobenius distance,

M=‖Z^−S^‖F2\.M=\\\|\\hat\{Z\}\-\\hat\{S\}\\\|\_\{F\}^\{2\}\.\(61\)
For the second plot, we use the same parameters and metric, but evaluate the logit matrix after the logit norm has increased by a factor of ten\. For the final plot, we report the effective rank of the mean logit matrix for a range of network widths withL=3L=3andK=10K=10, after training for2×1052\\times 10^\{5\}epochs\. In all experiments, we conduct five runs and report the mean with one\-standard\-deviation error bars\.

Figure[6](https://arxiv.org/html/2605.23087#S4.F6)shows the outcome of training a ResNet\-20 with a ReLU head on the MNIST and CIFAR\-10 datasets from random initialization\. For the MNIST experiments, we subsample 5,000 examples per class to match the class balance of CIFAR\-10\. MNIST is trained for 4,000 epochs and CIFAR\-10 for 5,000 epochs\. In all cases, the network attains≈100%\\approx 100\\%training accuracy early in training, so learning primarily occurs in the terminal phase\(Papyanet al\.,[2020](https://arxiv.org/html/2605.23087#bib.bib129)\)\. The input data are preprocessed by subtracting the mean and dividing by the standard deviation\. We use batch gradient descent with batch size 10,000 to approximate full\-batch gradient descent, as assumed by the model\. No explicit regularization is used in any experiment\. In the first plot, we setd=50d=50and vary the number of layers in the ReLU head between 1 and 8\. In the second plot, we fix the number of layers in the ReLU head toL=3L=3and vary the head width between 50 and 200\. For both plots, we train five models at each parameter setting and report the mean with one\-standard\-deviation error bars\. The final plot shows generalization performance versus the effective rank of the network across the forty five MNIST models from the previous two plots\. We also include the line of best fit to illustrate the overall trend\.

### E\.2Further Model Experiments

In Figure[7](https://arxiv.org/html/2605.23087#A5.F7)we show examples of normalized mean logit matrices recovered during training of a deep UFM withK=10K=10,L=3L=3,n=5n=5, and network widthd=100d=100\. Even though the DNC solution clearly has a larger normalized margin over the data than the low\-rank alternative, the network often converges to low\-rank solutions such as the one shown\.

In Figure[8](https://arxiv.org/html/2605.23087#A5.F8)we show the result of training a deep UFM withK=8K=8,L=10L=10,n=5n=5, andd=8d=8\. The classes were permuted to produce the symmetric structure shown\. This corresponds to the uniformly separated circle solution described in Figure[3](https://arxiv.org/html/2605.23087#S3.F3)of the main text\.

![Refer to caption](https://arxiv.org/html/2605.23087v1/x11.png)Figure 8:Plot of the normalized mean logit matrix at convergence of a rank two solution forK=8,L=10K=8,L=10\.
### E\.3Further Hadamard Experiments

The left panel of Figure[9](https://arxiv.org/html/2605.23087#A5.F9)demonstrates how the effective rank changes with depth in the Hadamard setting\. In Appendix[C\.1](https://arxiv.org/html/2605.23087#A3.SS1)we show that, in this Hadamard setting, the minimal rank the logit matrix can achieve while remaining compatible with a continuously decreasing loss along gradient flow ism=log2⁡\(K\)m=\\log\_\{2\}\(K\)\. The left panel shows that the low\-rank bias becomes stronger asLLincreases\. Specifically, after long training times the effective rank decreases monotonically with depth: it starts atK−1=15K\-1=15, corresponding to NC whenL=1L=1, and eventually concentrates at the lower boundlog2⁡\(K\)=4\\log\_\{2\}\(K\)=4for largeLL\. This provides further evidence that increasing depth induces a low\-rank bias in our model\.

The description of the “rich\-get\-richer” effect in the linearized regime naturally raises the question of how the initialization scale might affect the rank of the converged solution\. Since a larger initialization scale implies less time spent in the linearized regime, it may lead to a higher rank at convergence\. This is explored in the right panel of Figure[9](https://arxiv.org/html/2605.23087#A5.F9), where, as the theory predicts, the effective rank at convergence increases as the initialization scale is increased in the Hadamard regime\.

Here we also provide an example of a gradient\-flow trajectory along which the KL divergence provably diverges\. This corresponds to the initialization considered in the proof of Theorem[4\.3](https://arxiv.org/html/2605.23087#S4.Thmtheorem3)in App\.[D\.4](https://arxiv.org/html/2605.23087#A4.SS4)\. In Figure[10](https://arxiv.org/html/2605.23087#A5.F10), we show the evolution of the normalized logit singular values from the initializationa=\(γ,δ,…,δ,γ\)a=\(\\gamma,\\delta,\\ldots,\\delta,\\gamma\), withγ=0\.2\\gamma=0\.2andδ=0\.1\\delta=0\.1\. Note that DNC corresponds to the caseγ/δ→1\\gamma/\\delta\\rightarrow 1, whereas the cross\-polytope geometry—an alternative stable point of the loss landscape—corresponds toγ/δ→∞\\gamma/\\delta\\rightarrow\\infty\.

In App\.[D\.4](https://arxiv.org/html/2605.23087#A4.SS4), we show that ifγ/δ\\gamma/\\deltasatisfies

γδ\>\(KK−2\)L\+1L−1,\\frac\{\\gamma\}\{\\delta\}\>\\left\(\\frac\{K\}\{K\-2\}\\right\)^\{\\frac\{L\+1\}\{L\-1\}\},then the ratio must continue to increase for all future times\. The specific choice in the figure satisfies this inequality at initialization, and it is clear that the evolution tends toward the cross\-polytope geometry rather than DNC\. Note that the KL divergence increases monotonically throughout this trajectory\. In fact, it must diverge in the limitt→∞t\\to\\infty, but it does so slowly due to the lazy dynamics that emerge at large norm\(Garrodet al\.,[2025](https://arxiv.org/html/2605.23087#bib.bib9)\), as well as the fact that the KL divergence involves the logarithm of the singular values\.

![Refer to caption](https://arxiv.org/html/2605.23087v1/x12.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x13.png)

Figure 9:Left:Empirical mean of the effective rank of the mean logit matrix att=103t=10^\{3\}under the dynamics of Equation \([6](https://arxiv.org/html/2605.23087#S4.E6)\) with parameterK=16K=16for various network depthsLL\. The vectora∈ℝK−1a\\in\\mathbb\{R\}^\{K\-1\}is initialized with entries uniformly sampled from\[0\.5,1\]\[0\.5,1\], then normalized to haveL1L\_\{1\}norm of10−310^\{\-3\}\. For eachLLthe average empirical effective rank over 10 runs is reported, along with one\-standard\-deviation error bars\.Right:consideration of various initialization scales atL=3L=3\. Same initialization set up, similarly average and one\-standard\-deviation error bars over 10 runs is reported\.![Refer to caption](https://arxiv.org/html/2605.23087v1/x14.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x15.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x16.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x17.png)

Figure 10:Evolution dynamics under Eq\. \([6](https://arxiv.org/html/2605.23087#S4.E6)\) from the initialization described in Eq\. \([49](https://arxiv.org/html/2605.23087#A4.E49)\) withγ=0\.2\\gamma=0\.2,δ=0\.1\\delta=0\.1for parametersL=2,K=16L=2,K=16\.Top Left: Evolution of the logit norm‖a‖1=∑iai\\\|a\\\|\_\{1\}=\\sum\_\{i\}a\_\{i\}\.Top Right: Evolution of the normalized singular valuesa^i\\hat\{a\}\_\{i\}\.Bottom Left: Evolution of the KL\-divergenceDKL​\(1K−1​1K−1∥a^\)D\_\{\\mathrm\{KL\}\}\(\\frac\{1\}\{K\-1\}1\_\{K\-1\}\\,\\\|\\,\\hat\{a\}\), defined in Eq\. \([52](https://arxiv.org/html/2605.23087#A4.E52)\)\.Bottom Right: mean logit matrix at the end of training, normalized to have unit Frobenius norm\.
### E\.4Hadamard Initialization is Representative of Random Initialization

In Section[4](https://arxiv.org/html/2605.23087#S4), we used Hadamard initialization to study the finite\-time dynamics of the deep UFM\. Although this structured initialization yields analytically tractable equations, the resulting phenomenology is also representative of small random initialization\. The evolution of the singular values under small random initialization is shown in Figure[11](https://arxiv.org/html/2605.23087#A5.F11), which may be contrasted with the Hadamard case in Figure[4](https://arxiv.org/html/2605.23087#S4.F4)of the main text\. We present one illustrative example, although all experiments of this kind exhibited the same qualitative behavior\. The two cases display remarkably similar phenomenology; most notably, both exhibit the “rich\-get\-richer” effect, which explains how low\-rank bias emerges at the level of individual training trajectories\.

This is a standard observation in the spectral initialization literature, where structured initializations often trace representative trajectories for a much broader class of initial conditions\. This phenomenon was first studied bySaxeet al\.\([2013](https://arxiv.org/html/2605.23087#bib.bib93),[2019](https://arxiv.org/html/2605.23087#bib.bib94)\)andGidelet al\.\([2019](https://arxiv.org/html/2605.23087#bib.bib102)\)in the MSE\-loss setting, and later extended to the CE\-loss setting byGarrodet al\.\([2025](https://arxiv.org/html/2605.23087#bib.bib9)\)\. Across these works, the dynamics consistently exhibit a two\-stage evolution: in the initial phase, the singular vectors rapidly align with the spectral basis; after this transient phase, the singular value trajectories closely follow those obtained under spectral initialization\. Our results provide evidence of the same observation\. Early in training, while the singular values remain approximately stable, the singular vectors continue to evolve; once this stage has passed, the dynamics closely resemble those under Hadamard initialization, exhibiting the key phenomenology responsible for low\-rank bias in the model\.

![Refer to caption](https://arxiv.org/html/2605.23087v1/x18.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x19.png)
![Refer to caption](https://arxiv.org/html/2605.23087v1/x20.png)

Figure 11:Experiments under Random initialization for the deep UFM\.Left:Evolution of the normalized mean logit singular valuesa^i\\hat\{a\}\_\{i\}under gradient flow on the loss function of Eq\. \([1](https://arxiv.org/html/2605.23087#S2.E1)\) for depthL=3L=3, widthd=100d=100,K=10K=10classes andn=5n=5data points per class\. Network was initialized with Gaussian normal entries of standard deviation0\.030\.03\. Note convergence to low\-rank solution where many modes approach zero as also shown in the Hadamard case in Figure[4](https://arxiv.org/html/2605.23087#S4.F4)\.Middle:Corresponding evolution of the mean logit nuclear norm\. By the time norm increases and system exits the linearized regime, the low\-rank structure has already taken hold, just as in the Hadamard case\.Right: The corresponding KL divergence\(definition in Eq\. \([52](https://arxiv.org/html/2605.23087#A4.E52)\)\.\)\.