正交化如何适应神经网络的海森矩阵结构？初始化时的梯度自外积分析

# How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization
Source: [https://arxiv.org/html/2610.28975](https://arxiv.org/html/2610.28975)
###### Abstract

Muon orthogonalizes a weight matrix’s momentum before each step, and on neural networks this simple preconditioner beats AdamW in a wide range of tasks\. Existing analyses explain Muon within general optimization frameworks or on a few specific models, and many of them view orthogonalization as a preconditioner\. Since the Hessian is the optimal preconditioner under a quadratic approximation of the loss, these works leave a basic question open: what makes orthogonalization suited to the Hessian of a neural network? Prior work shows that the layer\-wise Hessian of a neural network is row\-block diagonally dominant, and Muon’s implicit preconditioner, the Kronecker product\(𝑽​𝑽⊤\)1/2⊗𝑰\(\\bm\{V\}\\bm\{V\}^\{\\top\}\)^\{1/2\}\\otimes\\bm\{I\}with𝑽\\bm\{V\}the momentum, has the same row\-block structure exactly when𝑽​𝑽⊤\\bm\{V\}\\bm\{V\}^\{\\top\}is diagonally dominant\. Diagonal dominance of𝑽​𝑽⊤\\bm\{V\}\\bm\{V\}^\{\\top\}has been observed in language models, but why a neural network produces it has not been explained\. This work provides a quantitative asymptotic theory for neural networks at initialization, and points out the intuition that the overparameterization of the weight matrices drives the diagonal dominance\. We analyze𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\], which equals𝔼⁡\[𝑽​𝑽⊤\]\\mathbb\{E\}\[\\bm\{V\}\\bm\{V\}^\{\\top\}\]at initialization, in four canonical models: symmetric matrix factorization, deep linear networks, two\-layer ReLU networks, and single\-layer linear attention\. In each case the diagonal entries outgrow the off\-diagonal ones at explicit rates as the hidden dimensions grow; for the query, key and value weights of attention the off\-diagonal vanishes altogether\. Simulations of the four models and experiments on language models at initialization confirm the predicted scaling\. The code for all our simulations and language\-model experiments is available at this[link](https://anonymous.4open.science/r/orthogonalization-hessian-CB2C/)\.

## 1Introduction

Muon\([Jordan et al\., 2024](https://arxiv.org/html/2610.28975#bib.bib1)\)optimizes a matrix\-shaped weight𝑾t∈ℝm×n\\bm\{W\}\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}by orthogonalizing the momentum buffer instead of using it directly\. From a stochastic gradient𝑮t\\bm\{G\}\_\{t\},

𝑽t=β𝑽t−1\+\(1−β\)𝑮t,𝑫t=NS5\(𝑽t\)≈\(𝑽t𝑽t⊤\)−1/2𝑽t,𝑾t\+1=𝑾t−ηt𝑫t,\\bm\{V\}\_\{t\}=\\beta\\bm\{V\}\_\{t\-1\}\+\(1\-\\beta\)\\bm\{G\}\_\{t\},\\qquad\\bm\{D\}\_\{t\}=\\mathrm\{NS\}\_\{5\}\(\\bm\{V\}\_\{t\}\)\\approx\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}\\bm\{V\}\_\{t\},\\qquad\\bm\{W\}\_\{t\+1\}=\\bm\{W\}\_\{t\}\-\\eta\_\{t\}\\bm\{D\}\_\{t\},\(1\)with𝑽0=𝑮0\\bm\{V\}\_\{0\}=\\bm\{G\}\_\{0\}andNS5\\mathrm\{NS\}\_\{5\}a fixed\-point iteration for the orthogonal polar factor: if𝑽t=𝑷​𝚺​𝑸⊤\\bm\{V\}\_\{t\}=\\bm\{P\}\\bm\{\\Sigma\}\\bm\{Q\}^\{\\top\}is a thin SVD then𝑫t=𝑷𝑸⊤=\(𝑽t𝑽t⊤\)−1/2𝑽t\\bm\{D\}\_\{t\}=\\bm\{P\}\\bm\{Q\}^\{\\top\}=\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}\\bm\{V\}\_\{t\}\. A growing body of work studies why Muon works within general optimization frameworks: as momentum steepest descent under a chosen operator norm\([Bernstein and Newhouse, 2024b](https://arxiv.org/html/2610.28975#bib.bib9);[Bernstein and Newhouse, 2024a](https://arxiv.org/html/2610.28975#bib.bib10);[Pethick et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib13);[Lau et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib11)\), as a non\-Euclidean trust\-region step\([Kovalev, 2025](https://arxiv.org/html/2610.28975#bib.bib6)\), under an isotropic curvature model\([Su, 2025](https://arxiv.org/html/2610.28975#bib.bib19)\), with an implicit spectral\-norm constraint on the iterates\([Chen et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib7)\), and with convergence guarantees under smoothness or Hölder smoothness, heavy\-tailed noise, inexact orthogonalization, variance reduction, and nonsmooth or convex Lipschitz objectives\([Li and Hong, 2025](https://arxiv.org/html/2610.28975#bib.bib4);[Shen et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib5);[An et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib12);[Kovalev and Borodich, 2026](https://arxiv.org/html/2610.28975#bib.bib54);[Nagashima and Iiduka, 2026](https://arxiv.org/html/2610.28975#bib.bib70);[Sato et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib71);[Gratton and Toint, 2026](https://arxiv.org/html/2610.28975#bib.bib77);[Hübler et al\., 2026b](https://arxiv.org/html/2610.28975#bib.bib50);[Iiduka, 2026](https://arxiv.org/html/2610.28975#bib.bib51);[Choudhury et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib76);[Shulgin et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib68);[Kim and Oh, 2026](https://arxiv.org/html/2610.28975#bib.bib69);[Qian and Richtárik, 2026](https://arxiv.org/html/2610.28975#bib.bib75);[Do et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib74);[Chang et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib66);[Qian et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib67);[Li and Tsuchiya, 2026](https://arxiv.org/html/2610.28975#bib.bib52);[Yang et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib72);[Lai et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib73);[Parshakova et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib49);[Liu et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib53);[Long et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib48)\)\.

However, those works derive their analyses from abstract assumptions on the loss, such as smoothness or convexity, imposed uniformly over a problem class rather than from any specific model\. The limitation of those works is that they cannot explain why Muon, a simple matrix preconditioner, works so well*for neural networks specifically*\. The few theoretical results for specific models, such as matrix factorization and in\-context learning with linear transformers\([Ma et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib14)\), linear or bilinear classifiers on separable or imbalanced data\([Fan et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib15);[Vasudeva et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib16)\), and random\-feature and transformer blocks at initialization\([Davis and Drusvyatskiy, 2025](https://arxiv.org/html/2610.28975#bib.bib8)\), sharpen the picture in particular regimes\. Nevertheless, since the Hessian is the optimal preconditioner under a quadratic approximation of the loss, those previous works leave a basic structural question for preconditioning methods open:What makes the orthogonalization preconditioner suited to the Hessian of a neural network specifically?This paper gives a theoretical answer that Muon’s implicit preconditioner shares a similar row\-block diagonally dominant heterogeneous structure to the neural\-network Hessian described by[Zhang et al\. \(2024a\)](https://arxiv.org/html/2610.28975#bib.bib2);[Zhang et al\. \(2024b\)](https://arxiv.org/html/2610.28975#bib.bib20)\.

#### Preconditioner structure versus Hessian structure\.

Let us first take a view of the equivalent preconditioner form of orthogonalization\. The orthogonalized update is𝑫t=\(𝑽t𝑽t⊤\)−1/2𝑽t\\bm\{D\}\_\{t\}=\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}\\bm\{V\}\_\{t\}, a left multiplication that mixes the rows of𝑽t\\bm\{V\}\_\{t\}\. Stacking the rows of𝑫t\\bm\{D\}\_\{t\}and of𝑽t\\bm\{V\}\_\{t\}into vectors turns it intovec\(𝑫t\)=\(\(𝑽t𝑽t⊤\)−1/2⊗𝑰n\)vec\(𝑽t\)\\mathrm\{vec\}\(\\bm\{D\}\_\{t\}\)=\\bigl\(\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}\\otimes\\bm\{I\}\_\{n\}\\bigr\)\\mathrm\{vec\}\(\\bm\{V\}\_\{t\}\), so Muon descends in the metric

HMuon=𝑷t1/2⊗𝑰n,𝑷t:=𝑽t​𝑽t⊤∈ℝm×m,H\_\{\\mathrm\{Muon\}\}=\\bm\{P\}\_\{t\}^\{1/2\}\\otimes\\bm\{I\}\_\{n\},\\qquad\\bm\{P\}\_\{t\}:=\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\\in\\mathbb\{R\}^\{m\\times m\},\(2\)a Kronecker structure determined entirely by𝑷t\\bm\{P\}\_\{t\}\. At initialization,𝑽0=𝑮0\\bm\{V\}\_\{0\}=\\bm\{G\}\_\{0\}and hence𝑷0=𝑮0​𝑮0⊤\\bm\{P\}\_\{0\}=\\bm\{G\}\_\{0\}\\bm\{G\}\_\{0\}^\{\\top\}\. Previous work has shown that the neural\-network Hessian has a special heterogeneous structure: for a weight𝑾∈ℝm×n\\bm\{W\}\\in\\mathbb\{R\}^\{m\\times n\}, them×mm\\times mblocks indexed by its rows \(output neurons\) are dominated by the diagonal ones, that is, the layer\-wise Hessian is*row\-block diagonally dominant*\([Zhang et al\., 2024a](https://arxiv.org/html/2610.28975#bib.bib2);[Zhang et al\., 2024b](https://arxiv.org/html/2610.28975#bib.bib20);[Dong et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib3)\)\. Meanwhile,[Deng et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib27)point out empirically that𝑮t​𝑮t⊤\\bm\{G\}\_\{t\}\\bm\{G\}\_\{t\}^\{\\top\}is diagonally dominant throughout training, and the larger the model, the stronger the dominance\. When𝑷t\\bm\{P\}\_\{t\}is close to diagonal,HMuonH\_\{\\mathrm\{Muon\}\}in \([2](https://arxiv.org/html/2610.28975#S1.E2)\) shares the row\-block structure of the Hessian \([Figure1](https://arxiv.org/html/2610.28975#S1.F1)\), so the question reduces to a clean property of the gradient\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/fig1_9263.png)Figure 1:Left: Hessian of a Transformer weight matrix at initialization, whose mass concentrates on the diagonal blocks\. Right: the momentum self outer\-product𝑽t​𝑽t⊤\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}of a Transformer weight matrix at initialization \(log scale\), whose mass concentrates on the diagonal\.Question\.How does Neural Networks produce such a diagonally dominant𝑽t​𝑽t⊤\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\(equivalently,𝑮t​𝑮t⊤\\bm\{G\}\_\{t\}\\bm\{G\}\_\{t\}^\{\\top\}at initialization\)?

This paper answers this question for neural networks in an asymptotic regime, going a step further than the empirical observation of[Deng et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib27)by turning it into a quantitative asymptotic theoretical result at initialization\. We prove that in four canonical neural\-network models the diagonal of𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]outgrows the off\-diagonal as the hidden dimensions of the weight matrices grow, at rates summarized in[Table1](https://arxiv.org/html/2610.28975#S4.T1)\. The overparameterization of the weight matrices is thus a key indicator of this diagonalization\. Our contributions can be summarized as follows\.

1. 1\.Hidden\-dimension asymptotics theory for Muon’s preconditioner\.For symmetric matrix factorization, deep linear networks, two\-layer ReLU networks and single\-layer linear attention, we prove that Muon’s implicit preconditioner becomes diagonally dominant as the hidden dimensions grow, with explicit rates \([Theorems1](https://arxiv.org/html/2610.28975#Thmtheorem1),[2](https://arxiv.org/html/2610.28975#Thmtheorem2),[3](https://arxiv.org/html/2610.28975#Thmtheorem3)and[4](https://arxiv.org/html/2610.28975#Thmtheorem4)and[Table1](https://arxiv.org/html/2610.28975#S4.T1)\)\.
2. 2\.Empirical evaluation of the predicted scaling, from simulations to language models\.Simulations of the four models reproduce the predicted rates \([Section5](https://arxiv.org/html/2610.28975#S5)\), and Qwen3\.5, Llama\-3\.2 and Granite\-4\.0 at initialization show the dominance strengthening with model width \([Section6](https://arxiv.org/html/2610.28975#S6)\)\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/main_fig.png)Figure 2:Empirical𝔼^​\[𝑮​𝑮⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\(log scale\) of the query, key, value and output projections at initialization, for single\-layer linear attention \(left, details in[SectionG\.4](https://arxiv.org/html/2610.28975#A7.SS4)\) and the first attention block of Qwen3\.5 \(right, details in[SectionH\.1](https://arxiv.org/html/2610.28975#A8.SS1)\)\. In both cases the mass concentrates on the diagonal\. Each heatmap is normalized by its empirical mean diagonal magnitude\.

## 2Related Work

#### Mechanistic analyses of Muon\.

Muon\([Jordan et al\., 2024](https://arxiv.org/html/2610.28975#bib.bib1)\)replaces the momentum of each weight matrix by its polar factor\. This is the momentum form of steepest descent in the spectral norm, a view developed by[Bernstein and Newhouse \(2024b\)](https://arxiv.org/html/2610.28975#bib.bib9);[Bernstein and Newhouse \(2024a\)](https://arxiv.org/html/2610.28975#bib.bib10)\. Work on why this update suits neural networks follows two routes\. The first route models the curvature of the loss\.[Su \(2025\)](https://arxiv.org/html/2610.28975#bib.bib19)introduces an isotropic curvature model and shows that, under a growth condition on the curvature, the optimal update flattens the spectrum of the gradient, as orthogonalization does\.[Davis and Drusvyatskiy \(2025\)](https://arxiv.org/html/2610.28975#bib.bib8)give a layerwise condition, comparing the nuclear\-to\-Frobenius ratio of the gradient with the stable rank of the incoming activations, under which one spectral step decreases the loss more than one Euclidean step\.[Ma et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib14)prove that a simplified Muon converges linearly on matrix factorization and on in\-context learning with linear transformers, at a rate independent of the condition number and ahead of gradient descent and Adam\.[Nguyen et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib35)recast Muon as a preconditioned gradient method and show, under a Kronecker\-factored curvature model, that spectral flattening improves the convergence factor\.[Wang et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib40)compare the one\-step loss decrease of Muon and Adam under a second\-order Taylor approximation and attribute Muon’s advantage to a smaller curvature penalty, and[Wu et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib41)probe the loss landscape along real training runs, estimating the loss\-optimal step size along each singular direction of the momentum\. Since the isotropic geometry behind Muon only approximates the true curvature, several curvature\-aware variants have followed\([Zhang et al\., 2026c](https://arxiv.org/html/2610.28975#bib.bib36);[Du and Su, 2026](https://arxiv.org/html/2610.28975#bib.bib37);[Che, 2026](https://arxiv.org/html/2610.28975#bib.bib43);[Zhang et al\., 2026b](https://arxiv.org/html/2610.28975#bib.bib42)\)\. The second route studies the update itself\.[Beneventano et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib38)characterize the polar factor as the entropy\-maximizing bounded update that keeps the singular directions of the gradient and derive its spectral dynamics\.[Shumaylov et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib39)find that random or inverted spectra train equally well, and[Wang et al\. \(2025b\)](https://arxiv.org/html/2610.28975#bib.bib17)attribute Muon’s advantage on heavy\-tailed data to its more isotropic update spectrum\.[Zhang \(2026\)](https://arxiv.org/html/2610.28975#bib.bib45)separate spectral control from gauge equivariance,[Eschenhagen et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib46)relate Muon to Shampoo, and[Zhang et al\. \(2026a\)](https://arxiv.org/html/2610.28975#bib.bib44)ask at which granularity attention weights should be orthogonalized\. Both routes take a curvature model or a landscape as given\. None of them compares the structure of Muon’s implicit preconditioner with that of the network’s Hessian, which is the question we study\.

#### Block structure of the neural\-network Hessian\.

The Hessian of a neural network is near block diagonal, with most of its mass in dense principal blocks\. This observation goes back to[Collobert \(2004\)](https://arxiv.org/html/2610.28975#bib.bib64), and the same structure is assumed by the block\-diagonal curvature approximations of[Le Roux et al\. \(2008\)](https://arxiv.org/html/2610.28975#bib.bib65)and[Martens and Grosse \(2015\)](https://arxiv.org/html/2610.28975#bib.bib57)\.[Dong et al\. \(2025\)](https://arxiv.org/html/2610.28975#bib.bib3)quantify the effect at random initialization: for linear and one\-hidden\-layer classifiers, the off\-diagonal blocks vanish relative to the diagonal ones as the number of classes grows\. The dense blocks within a layer are indexed by output neurons\. This is the row\-block structure we build on, and it is the partition by which Adam\-mini assigns one learning rate per block to the value, projection and MLP weights\([Zhang et al\., 2024b](https://arxiv.org/html/2610.28975#bib.bib20)\)\. Beyond being block diagonal, the Hessian is heterogeneous across parameter blocks\.[Zhang et al\. \(2024a\)](https://arxiv.org/html/2610.28975#bib.bib2)show that in Transformers, unlike CNNs, the Hessian spectrum varies strongly from one weight matrix to another, so that no single learning rate serves all of them\.[Ormaniec et al\. \(2024\)](https://arxiv.org/html/2610.28975#bib.bib61)derive the Hessian of a self\-attention layer and trace this heterogeneity to the softmax and the query\-key parameterization,[Tomihari and Sato \(2025\)](https://arxiv.org/html/2610.28975#bib.bib62)show that gradient heterogeneity across blocks complements it, and[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.28975#bib.bib63)exploit the sharpness disparity between blocks to speed up pre\-training\. Within a block,[Zhang et al\. \(2025\)](https://arxiv.org/html/2610.28975#bib.bib18)prove that under anisotropic inputs a layer\-wise Kronecker\-factored preconditioner recovers the feature\-learning rate that SGD and Adam miss\. Our work builds on the block\-diagonal part of this picture: a preconditioner should be block diagonal in the row\-block basis, and Muon’s\(𝑽​𝑽⊤\)1/2⊗𝑰\(\\bm\{V\}\\bm\{V\}^\{\\top\}\)^\{1/2\}\\otimes\\bm\{I\}has this form precisely when𝑽​𝑽⊤\\bm\{V\}\\bm\{V\}^\{\\top\}is diagonal\. Further preconditioning methods are discussed in[AppendixA](https://arxiv.org/html/2610.28975#A1)\.

## 3Preliminaries

Throughout,𝑾∈ℝm×n\\bm\{W\}\\in\\mathbb\{R\}^\{m\\times n\}denotes a weight matrix,ℒ\\mathcal\{L\}the population loss and𝑮=∂ℒ/∂𝑾\\bm\{G\}=\\partial\\mathcal\{L\}/\\partial\\bm\{W\}the gradient of the loss with respect to that matrix\. Our object of study is them×mm\\times mmatrix𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]at a Gaussian initialization\. This matrix is the expectation of𝑷0=𝑽0​𝑽0⊤\\bm\{P\}\_\{0\}=\\bm\{V\}\_\{0\}\\bm\{V\}\_\{0\}^\{\\top\}in \([2](https://arxiv.org/html/2610.28975#S1.E2)\)\. We analyze it in four settings\.[Section3\.1](https://arxiv.org/html/2610.28975#S3.SS1)introduces the four models, and[Section3\.2](https://arxiv.org/html/2610.28975#S3.SS2)states the initialization assumed for each\. We write𝑰d\\bm\{I\}\_\{d\}for thed×dd\\times didentity,tr\\trfor the trace and‖⋅‖F\\left\\\|\\cdot\\right\\\|\_\{\\mathrm\{F\}\}for the Frobenius norm\. The ReLU isσ⁡\(z\)=max⁡\(z,0\)\\sigma\(z\)=\\max\(z,0\), with derivativeσ′\(z\)=𝟏\[z\>0\]\\sigma^\{\\prime\}\(z\)=\\mathbf\{1\}\[z\>0\], and⊗\\otimesis the Kronecker product\. Finally,a=Θ⁡\(b\)a=\\Theta\(b\)means thatc​b≤a≤C​bc\\,b\\leq a\\leq C\\,bfor some constants0<c≤C0<c\\leq C\.

### 3\.1Setup

#### Symmetric matrix factorization\.

For positive integersd,k,rd,k,rwithk≥rk\\geq r, a weight𝑾∈ℝd×k\\bm\{W\}\\in\\mathbb\{R\}^\{d\\times k\}fits a fixed target𝑴⋆∈ℝd×d\\bm\{M\}^\{\\star\}\\in\\mathbb\{R\}^\{d\\times d\}, symmetric positive semidefinite of rankrr, under the over\-parameterized lossℒ⁡\(𝑾\):=14​‖𝑾​𝑾⊤−𝑴⋆‖F2,𝑮:=∂ℒ∂𝑾=\(𝑾​𝑾⊤−𝑴⋆\)​𝑾∈ℝd×k\.\\mathcal\{L\}\(\\bm\{W\}\):=\\tfrac\{1\}\{4\}\\left\\\|\\bm\{W\}\\bm\{W\}^\{\\top\}\-\\bm\{M\}^\{\\star\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},\\qquad\\bm\{G\}:=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\bm\{W\}\}=\(\\bm\{W\}\\bm\{W\}^\{\\top\}\-\\bm\{M\}^\{\\star\}\)\\,\\bm\{W\}\\in\\mathbb\{R\}^\{d\\times k\}\.\(3\)The hidden dimension iskk\.
#### Deep linear network\.

ForL≥1L\\geq 1and layer dimensionsd0,…,dLd\_\{0\},\\dots,d\_\{L\}, weights𝑾i∈ℝdi×di−1\\bm\{W\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{i\-1\}\}fit a fixed linear target𝚽∈ℝdL×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{L\}\\times d\_\{0\}\}on inputs𝒙∼𝒩⁡\(𝟎,𝑰d0\)\\bm\{x\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\),f\(𝒙\)=𝑾L⋯𝑾1𝒙,ℒ\(𝑾1,…,𝑾L\):=12𝔼𝒙‖f\(𝒙\)−𝚽𝒙‖2\.f\(\\bm\{x\}\)=\\bm\{W\}\_\{L\}\\cdots\\bm\{W\}\_\{1\}\\bm\{x\},\\qquad\\mathcal\{L\}\(\\bm\{W\}\_\{1\},\\dots,\\bm\{W\}\_\{L\}\):=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\\left\\\|f\(\\bm\{x\}\)\-\\bm\{\\Phi\}\\bm\{x\}\\right\\\|^\{2\}\.\(4\)With𝑩i:=𝑾i−1⋯𝑾1\\bm\{B\}\_\{i\}:=\\bm\{W\}\_\{i\-1\}\\cdots\\bm\{W\}\_\{1\}\(𝑩1:=𝑰\\bm\{B\}\_\{1\}:=\\bm\{I\}\),𝑨i:=𝑾L⋯𝑾i\+1\\bm\{A\}\_\{i\}:=\\bm\{W\}\_\{L\}\\cdots\\bm\{W\}\_\{i\+1\}\(𝑨L:=𝑰\\bm\{A\}\_\{L\}:=\\bm\{I\}\) and𝑹:=𝑾L⋯𝑾1−𝚽\\bm\{R\}:=\\bm\{W\}\_\{L\}\\cdots\\bm\{W\}\_\{1\}\-\\bm\{\\Phi\}, the layer\-iigradient is𝑮i:=∂ℒ∂𝑾i=𝑨i⊤​𝑹​𝑩i⊤∈ℝdi×di−1\.\\bm\{G\}\_\{i\}:=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\bm\{W\}\_\{i\}\}=\\bm\{A\}\_\{i\}^\{\\top\}\\,\\bm\{R\}\\,\\bm\{B\}\_\{i\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{i\-1\}\}\.\(5\)The hidden dimensions ared1,…,dL−1d\_\{1\},\\dots,d\_\{L\-1\}\.
#### Two\-layer ReLU network\.

For positived0,d1,d2d\_\{0\},d\_\{1\},d\_\{2\}, weights𝑾1∈ℝd1×d0\\bm\{W\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{0\}\}and𝑾2∈ℝd2×d1\\bm\{W\}\_\{2\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{1\}\}fit a fixed linear target𝚽∈ℝd2×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{0\}\}on inputs𝒙∼𝒩⁡\(𝟎,𝑰d0\)\\bm\{x\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\),f⁡\(𝒙\)=𝑾2​σ​\(𝑾1​𝒙\),ℒ⁡\(𝑾1,𝑾2\):=12​𝔼𝒙​‖f⁡\(𝒙\)−𝚽​𝒙‖2\.f\(\\bm\{x\}\)=\\bm\{W\}\_\{2\}\\,\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\),\\qquad\\mathcal\{L\}\(\\bm\{W\}\_\{1\},\\bm\{W\}\_\{2\}\):=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\\left\\\|f\(\\bm\{x\}\)\-\\bm\{\\Phi\}\\bm\{x\}\\right\\\|^\{2\}\.\(6\)With𝒉⁡\(𝒙\):=σ⁡\(𝑾1​𝒙\)\\bm\{h\}\(\\bm\{x\}\):=\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\)the hidden features,𝒓⁡\(𝒙\):=f⁡\(𝒙\)−𝚽​𝒙\\bm\{r\}\(\\bm\{x\}\):=f\(\\bm\{x\}\)\-\\bm\{\\Phi\}\\bm\{x\}the residual and𝑫⁡\(𝒙\):=diag⁡\(σ′​\(𝑾1​𝒙\)\)\\bm\{D\}\(\\bm\{x\}\):=\\diagop\(\\sigma^\{\\prime\}\(\\bm\{W\}\_\{1\}\\bm\{x\}\)\), the gradients𝑮i:=∂ℒ/∂𝑾i\\bm\{G\}\_\{i\}:=\\partial\\mathcal\{L\}/\\partial\\bm\{W\}\_\{i\}are𝑮1=𝔼𝒙​\[𝑫⁡\(𝒙\)​𝑾2⊤​𝒓​\(𝒙\)​𝒙⊤\]∈ℝd1×d0,𝑮2=𝔼𝒙​\[𝒓⁡\(𝒙\)​𝒉​\(𝒙\)⊤\]∈ℝd2×d1\.\\bm\{G\}\_\{1\}=\\mathbb\{E\}\_\{\\bm\{x\}\}\\bigl\[\\bm\{D\}\(\\bm\{x\}\)\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{r\}\(\\bm\{x\}\)\\bm\{x\}^\{\\top\}\\bigr\]\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{0\}\},\\qquad\\bm\{G\}\_\{2\}=\\mathbb\{E\}\_\{\\bm\{x\}\}\\bigl\[\\bm\{r\}\(\\bm\{x\}\)\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\\bigr\]\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{1\}\}\.\(7\)Let𝒘a\\bm\{w\}\_\{a\}denote theaa\-th row of𝑾1\\bm\{W\}\_\{1\}\. The hidden dimension isd1d\_\{1\}\.
#### Single\-layer linear attention\.

For positiveT,d0,dk,dv,d2T,d\_\{0\},d\_\{k\},d\_\{v\},d\_\{2\}, tokens𝑿∈ℝT×d0\\bm\{X\}\\in\\mathbb\{R\}^\{T\\times d\_\{0\}\}\(one per row\) pass through weights𝑾Q,𝑾K∈ℝdk×d0\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{0\}\},𝑾V∈ℝdv×d0\\bm\{W\}\_\{V\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{0\}\},𝑾O∈ℝd2×dv\\bm\{W\}\_\{O\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{v\}\}, which fit a fixed linear target𝚽∈ℝd2×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{0\}\},𝑨:=1dk𝑿𝑾Q⊤𝑾K𝑿⊤,f\(𝑿\)=𝑨𝑿𝑾V⊤𝑾O⊤,ℒ⁡\(𝑾Q,𝑾K,𝑾V,𝑾O\):=12​‖f⁡\(𝑿\)−𝑿​𝚽⊤‖F2\.\\begin\{gathered\}\\bm\{A\}:=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\,\\bm\{X\}\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{K\}\\bm\{X\}^\{\\top\},\\qquad f\(\\bm\{X\}\)=\\bm\{A\}\\,\\bm\{X\}\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{W\}\_\{O\}^\{\\top\},\\\\ \\mathcal\{L\}\(\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\):=\\tfrac\{1\}\{2\}\\left\\\|f\(\\bm\{X\}\)\-\\bm\{X\}\\bm\{\\Phi\}^\{\\top\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\\end\{gathered\}\(8\)Let𝑪:=𝑿⊤​𝑿\\bm\{C\}:=\\bm\{X\}^\{\\top\}\\bm\{X\},cp:=tr⁡\(𝑪p\)c\_\{p\}:=\\tr\(\\bm\{C\}^\{p\}\),𝒁:=𝑿​𝑾V⊤\\bm\{Z\}:=\\bm\{X\}\\bm\{W\}\_\{V\}^\{\\top\}and𝑹:=f⁡\(𝑿\)−𝑿​𝚽⊤\\bm\{R\}:=f\(\\bm\{X\}\)\-\\bm\{X\}\\bm\{\\Phi\}^\{\\top\}\. The gradients𝑮∙:=∂ℒ/∂𝑾∙\\bm\{G\}\_\{\\bullet\}:=\\partial\\mathcal\{L\}/\\partial\\bm\{W\}\_\{\\bullet\},∙∈\{Q,K,V,O\}\\bullet\\in\\\{Q,K,V,O\\\}, are𝑮Q\\displaystyle\\bm\{G\}\_\{Q\}=1dk​𝑾K​𝑿⊤​𝒁​𝑾O⊤​𝑹⊤​𝑿,\\displaystyle=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\,\\bm\{W\}\_\{K\}\\bm\{X\}^\{\\top\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{R\}^\{\\top\}\\bm\{X\},𝑮K\\displaystyle\\qquad\\bm\{G\}\_\{K\}=1dk​𝑾Q​𝑿⊤​𝑹​𝑾O​𝒁⊤​𝑿,\\displaystyle=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\,\\bm\{W\}\_\{Q\}\\bm\{X\}^\{\\top\}\\bm\{R\}\\bm\{W\}\_\{O\}\\bm\{Z\}^\{\\top\}\\bm\{X\},\(9\)𝑮V\\displaystyle\\bm\{G\}\_\{V\}=𝑾O⊤​𝑹⊤​𝑨​𝑿,\\displaystyle=\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{R\}^\{\\top\}\\bm\{A\}\\bm\{X\},𝑮O\\displaystyle\\qquad\\bm\{G\}\_\{O\}=𝑹⊤​𝑨​𝒁\.\\displaystyle=\\bm\{R\}^\{\\top\}\\bm\{A\}\\bm\{Z\}\.The hidden dimensions aredkd\_\{k\}anddvd\_\{v\}\. In most practical language models the input and output dimensions of an attention block both equal the model width,d0=d2d\_\{0\}=d\_\{2\}\([Vaswani et al\., 2017](https://arxiv.org/html/2610.28975#bib.bib84);[Radford et al\., 2019](https://arxiv.org/html/2610.28975#bib.bib86);[Brown and others, 2020](https://arxiv.org/html/2610.28975#bib.bib87);[Touvron and others, 2023](https://arxiv.org/html/2610.28975#bib.bib88);[Grattafiori and others, 2024](https://arxiv.org/html/2610.28975#bib.bib85);[Meta AI, 2024](https://arxiv.org/html/2610.28975#bib.bib82);[Qwen Team, 2026](https://arxiv.org/html/2610.28975#bib.bib80);[IBM Granite Team, 2025](https://arxiv.org/html/2610.28975#bib.bib81)\), and scaling a model up increases this width whiledk,dvd\_\{k\},d\_\{v\}stay fixed, so for attention we also trackd0d\_\{0\}andd2d\_\{2\}\.

### 3\.2Assumptions

Each setting is analyzed at a standard Gaussian initialization, stated separately below\.

###### Assumption 1\(Matrix factorization\)\.

The entries of𝐖\\bm\{W\}are i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\), and𝐌⋆\\bm\{M\}^\{\\star\}is deterministic and independent of𝐖\\bm\{W\}\. Hence𝐒:=𝐖​𝐖⊤∼𝒲d​\(k,𝐈d\)\\bm\{S\}:=\\bm\{W\}\\bm\{W\}^\{\\top\}\\sim\\mathcal\{W\}\_\{d\}\(k,\\bm\{I\}\_\{d\}\)is Wishart withkkdegrees of freedom\.

###### Assumption 2\(Deep linear network\)\.

The entries of𝐖1,…,𝐖L\\bm\{W\}\_\{1\},\\dots,\\bm\{W\}\_\{L\}are i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\); the input𝐱∼𝒩⁡\(𝟎,𝐈d0\)\\bm\{x\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\)is integrated out inℒ\\mathcal\{L\}; and𝚽\\bm\{\\Phi\}is deterministic and independent of the weights\. Expectations are over𝐖1,…,𝐖L\\bm\{W\}\_\{1\},\\dots,\\bm\{W\}\_\{L\}\.

###### Assumption 3\(Two\-layer ReLU network\)\.

The entries of𝐖1\\bm\{W\}\_\{1\}and𝐖2\\bm\{W\}\_\{2\}are i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\); the input𝐱∼𝒩⁡\(0,𝐈d0\)\\bm\{x\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{d\_\{0\}\}\)is integrated out inℒ\\mathcal\{L\}; and𝚽\\bm\{\\Phi\}is deterministic and independent of𝐖1,𝐖2\\bm\{W\}\_\{1\},\\bm\{W\}\_\{2\}\. Expectations are over𝐖1,𝐖2\\bm\{W\}\_\{1\},\\bm\{W\}\_\{2\}\.

###### Assumption 4\(Linear attention\)\.

𝑾Q,𝑾K,𝑾V,𝑾O\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}are mutually independent with i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries; the rows of𝐗\\bm\{X\}are i\.i\.d\.𝒩⁡\(𝟎,𝐈d0\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\)and independent of the weights, so that𝐂∼𝒲d0​\(T,𝐈d0\)\\bm\{C\}\\sim\\mathcal\{W\}\_\{d\_\{0\}\}\(T,\\bm\{I\}\_\{d\_\{0\}\}\)is Wishart; and𝚽\\bm\{\\Phi\}is deterministic\. Expectations are over the weights and the tokens jointly\.

## 4Main Results

We state the closed form of𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]in each setup, discuss its diagonal dominance, and defer the proofs to[AppendicesB](https://arxiv.org/html/2610.28975#A2),[C](https://arxiv.org/html/2610.28975#A3),[D](https://arxiv.org/html/2610.28975#A4)and[E](https://arxiv.org/html/2610.28975#A5)\.

### 4\.1Symmetric matrix factorization

###### Theorem 1\.

Under[Assumption1](https://arxiv.org/html/2610.28975#Thmassumption1), with𝐒:=𝐖​𝐖⊤∼𝒲d​\(k,𝐈d\)\\bm\{S\}:=\\bm\{W\}\\bm\{W\}^\{\\top\}\\sim\\mathcal\{W\}\_\{d\}\(k,\\bm\{I\}\_\{d\}\)Wishart,

𝔼⁡\[𝑮​𝑮⊤\]=α3​𝑰d−2​α2​𝑴⋆\+k​\(𝑴⋆\)2,\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]=\\alpha\_\{3\}\\,\\bm\{I\}\_\{d\}\-2\\alpha\_\{2\}\\,\\bm\{M\}^\{\\star\}\+k\\,\(\\bm\{M\}^\{\\star\}\)^\{2\},\(10\)whereα2=k⁡\(k\+d\+1\)\\alpha\_\{2\}=k\(k\+d\+1\)andα3=k3\+3​\(d\+1\)​k2\+\(d2\+3​d\+4\)​k\\alpha\_\{3\}=k^\{3\}\+3\(d\+1\)k^\{2\}\+\(d^\{2\}\+3d\+4\)kare the \(isotropic\) diagonal entries of𝔼⁡\[𝐒2\]\\mathbb\{E\}\[\\bm\{S\}^\{2\}\]and𝔼⁡\[𝐒3\]\\mathbb\{E\}\[\\bm\{S\}^\{3\}\]\. Consequently, every diagonal entryα3−2​α2​𝐌i​i⋆\+k​\(𝐌⋆\)i​i2\\alpha\_\{3\}\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\_\{ii\}\+k\(\\bm\{M\}^\{\\star\}\)\_\{ii\}^\{2\}isΘ⁡\(k3\)\\Theta\(k^\{3\}\), and every off\-diagonal entry−2​α2​𝐌i​j⋆\+k​\(𝐌⋆\)i​j2\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\_\{ij\}\+k\(\\bm\{M\}^\{\\star\}\)\_\{ij\}^\{2\}with𝐌i​j⋆≠0\\bm\{M\}^\{\\star\}\_\{ij\}\\neq 0isΘ⁡\(k2\)\\Theta\(k^\{2\}\)\.

As the hidden dimensionkkgrows, the ratio of diagonal to off\-diagonal entries is thereforeΘ⁡\(k\)\\Theta\(k\), and𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]approaches the isotropic matrixα3​𝑰d\\alpha\_\{3\}\\bm\{I\}\_\{d\}\. At initialization the preconditioner thus reduces, to leading order, to a uniform rescaling of the momentum, and the target enters only through the lower\-order terms−2​α2​𝑴⋆\+k​\(𝑴⋆\)2\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\+k\(\\bm\{M\}^\{\\star\}\)^\{2\}\.

### 4\.2Deep linear network

###### Theorem 2\.

Under[Assumption2](https://arxiv.org/html/2610.28975#Thmassumption2), for every layeri∈\{1,…,L\}i\\in\\\{1,\\dots,L\\\},

𝔼⁡\[𝑮i​𝑮i⊤\]=Vi​si​𝑰di\+𝑻i,𝑻i=\{\(∏j=1L−1dj\)​𝚽​𝚽⊤,i=L,\(∏j≠i,1≤j≤L−1dj\)​‖𝚽‖F2​𝑰di,1≤i<L,\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]=V\_\{i\}\\,s\_\{i\}\\,\\bm\{I\}\_\{d\_\{i\}\}\+\\bm\{T\}\_\{i\},\\qquad\\bm\{T\}\_\{i\}=\\begin\{cases\}\\bigl\(\\textstyle\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\bigr\)\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},&i=L,\\\\\[3\.0pt\] \\bigl\(\\textstyle\\prod\_\{j\\neq i,\\,1\\leq j\\leq L\-1\}d\_\{j\}\\bigr\)\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\\bm\{I\}\_\{d\_\{i\}\},&1\\leq i<L,\\end\{cases\}\(11\)whereVi,siV\_\{i\},s\_\{i\}are positive dimension\-only scalars given by two\-step recursions in\(d0,…,dL\)\(d\_\{0\},\\dots,d\_\{L\}\)\([AppendixC](https://arxiv.org/html/2610.28975#A3)\), with

Vi​si=\{Θ⁡\(di​∏j=1,j≠iL−1dj2\),1≤i<L,Θ⁡\(∏j=1L−1dj2\),i=L\.V\_\{i\}s\_\{i\}=\\begin\{cases\}\\Theta\\bigl\(d\_\{i\}\\prod\_\{j=1,\\,j\\neq i\}^\{L\-1\}d\_\{j\}^\{2\}\\bigr\),&1\\leq i<L,\\\\\[3\.0pt\] \\Theta\\bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}^\{2\}\\bigr\),&i=L\.\\end\{cases\}\(12\)

For a hidden layer \(i<Li<L\) the off\-diagonal block𝑻i\\bm\{T\}\_\{i\}is a multiple of𝑰di\\bm\{I\}\_\{d\_\{i\}\}, so𝔼⁡\[𝑮i​𝑮i⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]is exactly diagonal\. At the output layer the diagonal noiseVL​sL=Θ⁡\(∏j=1L−1dj2\)V\_\{L\}s\_\{L\}=\\Theta\\bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}^\{2\}\\bigr\)grows with the hidden dimensions while the off\-diagonal stays proportional to\(∏j=1L−1dj\)​\(𝚽​𝚽⊤\)a​b\\bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\bigr\)\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}, so the ratio, of order∏j=1L−1dj\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}, still diverges\. For equal hidden dimensionsd1=⋯=dL−1=dd\_\{1\}=\\dots=d\_\{L\-1\}=dthe output\-layer diagonal isΘ⁡\(d2​\(L−1\)\)\\Theta\(d^\{2\(L\-1\)\}\)against an off\-diagonal ofΘ⁡\(dL−1\)\\Theta\(d^\{L\-1\}\)\.

### 4\.3Two\-layer ReLU network

Letθ:=∠⁡\(𝒖,𝒗\)\\theta:=\\angle\(\\bm\{u\},\\bm\{v\}\)for independent𝒖,𝒗∼𝒩⁡\(0,𝑰d0\)\\bm\{u\},\\bm\{v\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{d\_\{0\}\}\), and set the constantsκ1:=𝔼⁡\[sin2⁡θ\+\(π−θ\)24​π2\+sin⁡θ​cos⁡θ​\(π−θ\)2​π2\]\\kappa\_\{1\}:=\\mathbb\{E\}\\bigl\[\\tfrac\{\\sin^\{2\}\\theta\+\(\\pi\-\\theta\)^\{2\}\}\{4\\pi^\{2\}\}\+\\tfrac\{\\sin\\theta\\cos\\theta\(\\pi\-\\theta\)\}\{2\\pi^\{2\}\}\\bigr\]andκ2:=𝔼⁡\[sin⁡θ⁡\(π−θ\)\]2​π2\\kappa\_\{2\}:=\\tfrac\{\\mathbb\{E\}\[\\sin\\theta\(\\pi\-\\theta\)\]\}\{2\\pi^\{2\}\}\.

###### Theorem 3\.

Under[Assumption3](https://arxiv.org/html/2610.28975#Thmassumption3), with𝐇:=𝔼𝐱​\[𝐡⁡\(𝐱\)​𝐡​\(𝐱\)⊤\]\\bm\{H\}:=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{h\}\(\\bm\{x\}\)\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\]andα:=𝔼𝐖1​\[tr⁡\(𝐇2\)\]\\alpha:=\\mathbb\{E\}\_\{\\bm\{W\}\_\{1\}\}\[\\tr\(\\bm\{H\}^\{2\}\)\],

𝔼⁡\[𝑮2​𝑮2⊤\]=α​𝑰d2\+d14​𝚽​𝚽⊤,α=Θ⁡\(d12\),\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]=\\alpha\\,\\bm\{I\}\_\{d\_\{2\}\}\+\\tfrac\{d\_\{1\}\}\{4\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},\\qquad\\alpha=\\Theta\(d\_\{1\}^\{2\}\),and𝔼⁡\[𝐆1​𝐆1⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]is permutation\-invariant in the hidden index with diagonal entries

𝔼⁡\[\(𝑮1​𝑮1⊤\)a​a\]=d0​d224\+d0​d2​\[\(d1−1\)​κ1\+12\]\+‖𝚽‖F24\\mathbb\{E\}\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{aa\}\]=\\tfrac\{d\_\{0\}d\_\{2\}^\{2\}\}\{4\}\+d\_\{0\}d\_\{2\}\\bigl\[\(d\_\{1\}\-1\)\\kappa\_\{1\}\+\\tfrac\{1\}\{2\}\\bigr\]\+\\tfrac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\}and off\-diagonal entries𝔼⁡\[\(𝐆1​𝐆1⊤\)a​c\]=d0​d2​κ2\+O⁡\(d2​d0\)\\mathbb\{E\}\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{ac\}\]=d\_\{0\}d\_\{2\}\\,\\kappa\_\{2\}\+O\(d\_\{2\}\\sqrt\{d\_\{0\}\}\)fora≠ca\\neq c\. Consequently, for𝐆1\\bm\{G\}\_\{1\}the diagonal entries areΘ⁡\(d1\)\\Theta\(d\_\{1\}\)and the off\-diagonal entriesΘ⁡\(1\)\\Theta\(1\), and for𝐆2\\bm\{G\}\_\{2\}they areΘ⁡\(d12\)\\Theta\(d\_\{1\}^\{2\}\)andΘ⁡\(d1\)\\Theta\(d\_\{1\}\)\.

As the hidden dimensiond1d\_\{1\}grows, the ratio of diagonal to off\-diagonal entries is thereforeΘ⁡\(d1\)\\Theta\(d\_\{1\}\)for both layers\. For𝑮2\\bm\{G\}\_\{2\}the off\-diagonald14​\(𝚽​𝚽⊤\)a​c\\tfrac\{d\_\{1\}\}\{4\}\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ac\}comes only from the target, as at the output layer of[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)\. For𝑮1\\bm\{G\}\_\{1\}the off\-diagonald0​d2​κ2d\_\{0\}d\_\{2\}\\kappa\_\{2\}is target\-free: it comes from a single term in the proof and traces back to ReLU’s lack of odd symmetry, which is why the dominance here is asymptotic rather than exact\. Simulations match every formula \([SectionG\.3](https://arxiv.org/html/2610.28975#A7.SS3)\)\.

### 4\.4Single\-layer linear attention

###### Theorem 4\.

Under[Assumption4](https://arxiv.org/html/2610.28975#Thmassumption4),

𝔼\[𝑮Q𝑮Q⊤\]=cQ𝑰dk,𝔼\[𝑮K𝑮K⊤\]=cK𝑰dk,𝔼\[𝑮V𝑮V⊤\]=cV𝑰dv,𝔼⁡\[𝑮O​𝑮O⊤\]=α¯O​𝑰d2\+dv​γ​𝚽​𝚽⊤,γ=1d0​𝔼​\[c22\]=T⁡\[\(T​d0\+8\)​\(T\+d0\)2\+2​\(T​d0\+10\)​\(T\+d0\)\+5​\(T​d0\+4\)\],\\begin\{gathered\}\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\]=c\_\{Q\}\\,\\bm\{I\}\_\{d\_\{k\}\},\\qquad\\mathbb\{E\}\[\\bm\{G\}\_\{K\}\\bm\{G\}\_\{K\}^\{\\top\}\]=c\_\{K\}\\,\\bm\{I\}\_\{d\_\{k\}\},\\qquad\\mathbb\{E\}\[\\bm\{G\}\_\{V\}\\bm\{G\}\_\{V\}^\{\\top\}\]=c\_\{V\}\\,\\bm\{I\}\_\{d\_\{v\}\},\\\\ \\mathbb\{E\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\]=\\bar\{\\alpha\}\_\{O\}\\,\\bm\{I\}\_\{d\_\{2\}\}\+d\_\{v\}\\,\\gamma\\;\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},\\\\ \\gamma=\\tfrac\{1\}\{d\_\{0\}\}\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=T\\bigl\[\(Td\_\{0\}\{\+\}8\)\(T\{\+\}d\_\{0\}\)^\{2\}\+2\(Td\_\{0\}\{\+\}10\)\(T\{\+\}d\_\{0\}\)\+5\(Td\_\{0\}\{\+\}4\)\\bigr\],\\end\{gathered\}\(13\)wherecQ,cK,cV,α¯Oc\_\{Q\},c\_\{K\},c\_\{V\},\\bar\{\\alpha\}\_\{O\}are explicit positive polynomials in the dimensions and‖𝚽‖F2\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\([AppendixE](https://arxiv.org/html/2610.28975#A5)\); for fixed𝚽\\bm\{\\Phi\},

cQ≍cK=Θ⁡\(d22​dv2​d06/dk\),cV=Θ⁡\(d2​\(d2\+dv\)​d06\),α¯O=Θ⁡\(dv2​d06\)\.c\_\{Q\}\\asymp c\_\{K\}=\\Theta\\bigl\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}/d\_\{k\}\\bigr\),\\qquad c\_\{V\}=\\Theta\\bigl\(d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\)\\,d\_\{0\}^\{6\}\\bigr\),\\qquad\\bar\{\\alpha\}\_\{O\}=\\Theta\\bigl\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\\bigr\)\.

For𝑾Q,𝑾K,𝑾V\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\}the off\-diagonal is exactly zero \([Lemma34](https://arxiv.org/html/2610.28975#Thmlemma34)\)\. For𝑾O\\bm\{W\}\_\{O\}, as at the output layer of[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2), the off\-diagonaldv​γ​\(𝚽​𝚽⊤\)a​b=Θ⁡\(dv​d03\)d\_\{v\}\\gamma\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)comes only from the target while the diagonalα¯O=Θ⁡\(dv2​d06\)\\bar\{\\alpha\}\_\{O\}=\\Theta\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)is target\-free noise, so the ratioΘ⁡\(dv​d03\)\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)diverges withdvd\_\{v\}and, faster, withd0d\_\{0\};dkd\_\{k\}does not affect it\. Withd0=d2d\_\{0\}=d\_\{2\}equal to the model width \([Section3\.1](https://arxiv.org/html/2610.28975#S3.SS1)\), the ratio grows like the cube of the model width \([Section6](https://arxiv.org/html/2610.28975#S6)\)\.

### 4\.5Summary and proof sketch

In all four settings the diagonal of𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]outgrows the off\-diagonal as the hidden dimension grows \([Table1](https://arxiv.org/html/2610.28975#S4.T1)\), so𝑷t=𝑽t​𝑽t⊤\\bm\{P\}\_\{t\}=\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}is asymptotically diagonal,HMuon=𝑷t1/2⊗𝑰nH\_\{\\mathrm\{Muon\}\}=\\bm\{P\}\_\{t\}^\{1/2\}\\otimes\\bm\{I\}\_\{n\}is asymptotically block\-diagonal in the Hessian’s row\-block structure, and \([AppendixF](https://arxiv.org/html/2610.28975#A6)\) orthogonalization is asymptotically a row\-wise normalization\.

Table 1:Summary of the orders of𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]at a Gaussian initialization in the hidden dimensions of each model and, for attention, the token dimensionsd0,d2d\_\{0\},d\_\{2\}\.The four proofs share a similar strategy\. Each row of𝑮\\bm\{G\}is a random vector, and the\(a,b\)\(a,b\)entry of𝑮​𝑮⊤\\bm\{G\}\\bm\{G\}^\{\\top\}is the inner product of rowsaaandbb, a measure of their correlation\. A diagonal entry is the correlation of a row with itself, its squared norm, which adds up over all hidden coordinates\. An off\-diagonal entry is the correlation between two different rows, which are built from independent Gaussian vectors at initialization, so most of its terms cancel in expectation and only the lower\-order terms carried by the fixed target remain\. The gap between the orders of the diagonal and off\-diagonal entries in[Table1](https://arxiv.org/html/2610.28975#S4.T1)comes from this difference\. In the proofs we compute these statistics of𝑮​𝑮⊤\\bm\{G\}\\bm\{G\}^\{\\top\}exactly, using moments of the Wishart distribution for matrix factorization, deep linear networks and linear attention, and Stein’s lemma together with the arc\-cosine kernel of[Cho and Saul \(2009\)](https://arxiv.org/html/2610.28975#bib.bib34)for the ReLU network, whose nonlinearity is not a polynomial\. Each closed form is reproduced by the simulations in[Section5](https://arxiv.org/html/2610.28975#S5)\.

## 5Simulation of Theoretical Results

#### Simulation setup\.

We verify the dimension scalings of[Theorems1](https://arxiv.org/html/2610.28975#Thmtheorem1),[2](https://arxiv.org/html/2610.28975#Thmtheorem2),[3](https://arxiv.org/html/2610.28975#Thmtheorem3)and[4](https://arxiv.org/html/2610.28975#Thmtheorem4)under[Assumptions1](https://arxiv.org/html/2610.28975#Thmassumption1),[2](https://arxiv.org/html/2610.28975#Thmassumption2),[3](https://arxiv.org/html/2610.28975#Thmassumption3)and[4](https://arxiv.org/html/2610.28975#Thmassumption4)by varying the model dimensions\. For each model, we vary the dimensions in the corresponding theoretical results and examine the resulting structure of the expected gradient self outer\-product𝑷:=𝔼⁡\[𝑮​𝑮⊤\]∈ℝm×m\\bm\{P\}:=\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\\in\\mathbb\{R\}^\{m\\times m\}, that is, the expectation of𝑷0\\bm\{P\}\_\{0\}in \([2](https://arxiv.org/html/2610.28975#S1.E2)\)\. For this matrix, we summarize the mean diagonal magnitude, the mean off\-diagonal magnitude, and their ratio,

D⁡\(𝑷\)=1m​∑i=1m\|Pi​i\|,OD⁡\(𝑷\)=1m⁡\(m−1\)​∑i≠j\|Pi​j\|,R⁡\(𝑷\)=D⁡\(𝑷\)OD⁡\(𝑷\),D\(\\bm\{P\}\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\|P\_\{ii\}\|,\\qquad\\mathrm\{OD\}\(\\bm\{P\}\)=\\frac\{1\}\{m\(m\-1\)\}\\sum\_\{i\\neq j\}\|P\_\{ij\}\|,\\qquad R\(\\bm\{P\}\)=\\frac\{D\(\\bm\{P\}\)\}\{\\mathrm\{OD\}\(\\bm\{P\}\)\},\(14\)We estimate𝑷\\bm\{P\}by𝔼^​\[𝑮​𝑮⊤\]=1S​∑s=1S𝑮\(s\)​𝑮\(s\)⊤\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]=\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}\\bm\{G\}^\{\(s\)\}\\bm\{G\}^\{\(s\)\\top\}overSSindependent initializations;D^,OD^,R^\\widehat\{D\},\\widehat\{\\mathrm\{OD\}\},\\widehat\{R\}denote these estimates\. Where the expected off\-diagonal is exactly zero, the nonzeroOD^\\widehat\{\\mathrm\{OD\}\}is Monte Carlo variance \(details in[AppendixG](https://arxiv.org/html/2610.28975#A7)\)\. We next apply this procedure to matrix factorization, deep linear networks, two\-layer ReLU networks, and single\-layer linear attention, varying in each case the dimensions governing the corresponding theoretical scaling\.

#### Symmetric matrix factorization\.

We begin with symmetric matrix factorization, varyingk∈\{5,10,20,40,80,160,320,640\}k\\in\\\{5,\\allowbreak 10,\\allowbreak 20,\\allowbreak 40,\\allowbreak 80,\\allowbreak 160,\\allowbreak 320,\\allowbreak 640\\\}withd=25d=25and a fixed rank\-5 target, usingS=10,000S=10\{,\}000initializations for eachkk\. In[Figure3](https://arxiv.org/html/2610.28975#S5.F3)\(left\),R^\\widehat\{R\}increases withkk, in line withR=Θ⁡\(k\)R=\\Theta\(k\)from[Theorem1](https://arxiv.org/html/2610.28975#Thmtheorem1);D^\\widehat\{D\}andOD^\\widehat\{\\mathrm\{OD\}\}followΘ⁡\(k3\)\\Theta\(k^\{3\}\)andΘ⁡\(k2\)\\Theta\(k^\{2\}\)\([SectionG\.1](https://arxiv.org/html/2610.28975#A7.SS1)\)\.

Figure 3:Diagonal\-dominance ratioR^\\widehat\{R\}for symmetric matrix factorization againstkk\(left;d=25d=25, rank\-5 target\) and for the two\-layer ReLU network againstd1d\_\{1\}\(right;d0=512d\_\{0\}=512,d2=12d\_\{2\}=12\)\.
#### Deep linear network\.

We next consider a depth\-44linear network with baseline dimensions\(d0,…,d4\)=\(512,80,80,80,512\)\(d\_\{0\},\\dots,d\_\{4\}\)=\(512,80,80,80,512\)\. We vary each hidden dimensiond1,d2,d3d\_\{1\},d\_\{2\},d\_\{3\}in turn over the same grid while holding the remaining dimensions fixed, usingS=10,000S=10\{,\}000initializations per configuration\. In[Figure4](https://arxiv.org/html/2610.28975#S5.F4),R^\\widehat\{R\}increases with every hidden dimension for all layers\. For the output layer,[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)givesD=Θ⁡\(\(d1​d2​d3\)2\)D=\\Theta\(\(d\_\{1\}d\_\{2\}d\_\{3\}\)^\{2\}\)andOD=Θ⁡\(d1​d2​d3\)\\mathrm\{OD\}=\\Theta\(d\_\{1\}d\_\{2\}d\_\{3\}\), and henceR=Θ⁡\(d1​d2​d3\)R=\\Theta\(d\_\{1\}d\_\{2\}d\_\{3\}\); accordingly,R^\\widehat\{R\}grows approximately linearly when any one hidden dimension is varied\. For the hidden layers, the expected off\-diagonal is zero \([SectionG\.2](https://arxiv.org/html/2610.28975#A7.SS2)\)\.

Figure 4:Diagonal\-dominance ratioR^\\widehat\{R\}of𝑮1,…,𝑮4\\bm\{G\}\_\{1\},\\dots,\\bm\{G\}\_\{4\}for the depth\-44linear network whend1d\_\{1\},d2d\_\{2\}ord3d\_\{3\}is varied \(left to right\)\. Each curve is normalized by its value at the baseline \(d1=d2=d3=80d\_\{1\}=d\_\{2\}=d\_\{3\}=80\) to facilitate comparison across layers with different absolute scales\.
#### Two\-layer ReLU network\.

We then study the two\-layer ReLU network, fixingd0=512d\_\{0\}=512andd2=12d\_\{2\}=12and varying the hidden widthd1d\_\{1\}over the same grid\. For each configuration, we useS=10,000S=10\{,\}000weight initializations and estimate the population gradient using128,000128\{,\}000Gaussian inputs\. In[Figure3](https://arxiv.org/html/2610.28975#S5.F3)\(right\),R^\\widehat\{R\}increases withd1d\_\{1\}for both layers, matchingR=Θ⁡\(d1\)R=\\Theta\(d\_\{1\}\)in[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)\([SectionG\.3](https://arxiv.org/html/2610.28975#A7.SS3)\)\.

#### Single\-layer linear attention\.

Finally, we consider single\-layer linear attention withT=32T=32,d0=512d\_\{0\}=512, andd2=12d\_\{2\}=12\. We examine four dimension sweeps:dk=dvd\_\{k\}=d\_\{v\}jointly,dkd\_\{k\}withdv=80d\_\{v\}=80,dvd\_\{v\}withdk=80d\_\{k\}=80, andd0=d2d\_\{0\}=d\_\{2\}jointly\. In[Figure5](https://arxiv.org/html/2610.28975#S5.F5),R^\\widehat\{R\}of𝑮O\\bm\{G\}\_\{O\}grows approximately linearly withdvd\_\{v\}, is nearly unchanged withdkd\_\{k\}, and grows superlinearly withd0d\_\{0\}, consistent withR=Θ⁡\(dv​d03\)R=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)from[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)\. For𝑮Q,𝑮K,𝑮V\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{V\}, the expected off\-diagonal entries are zero\. Their finiteR^\\widehat\{R\}values nevertheless follow the corresponding diagonal trends: they increase withdk=dvd\_\{k\}=d\_\{v\}, increase withdvd\_\{v\}, and grow strongly withd0=d2d\_\{0\}=d\_\{2\}, while𝑮Q\\bm\{G\}\_\{Q\}and𝑮K\\bm\{G\}\_\{K\}decrease withdkd\_\{k\}, consistent withD=Θ⁡\(d22​dv2​d06/dk\)D=\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}/d\_\{k\}\)\([SectionG\.4](https://arxiv.org/html/2610.28975#A7.SS4)\)\.

Figure 5:Diagonal\-dominance ratioR^\\widehat\{R\}of the four attention projections whendk=dvd\_\{k\}=d\_\{v\},dkd\_\{k\},dvd\_\{v\}, andd0=d2d\_\{0\}=d\_\{2\}are varied \(left to right\)\. Each curve is normalized by its value at the baseline \(first three:T=32T=32,d0=512d\_\{0\}=512,dk=dv=80d\_\{k\}=d\_\{v\}=80,d2=12d\_\{2\}=12; fourth:T=16,d0=d2=32,dk=dv=20T=16,d\_\{0\}=d\_\{2\}=32,d\_\{k\}=d\_\{v\}=20\) to facilitate comparison across projections with different absolute scales\.

## 6Empirical Evaluation on Large Language Models

We next test whether the diagonal dominance of[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)persists in Transformer\-based language models at random initialization: Qwen3\.5\-0\.8B\-Base\([Qwen Team, 2026](https://arxiv.org/html/2610.28975#bib.bib80)\), Granite\-4\.0\-1B\-Base\([IBM Granite Team, 2025](https://arxiv.org/html/2610.28975#bib.bib81)\)and Llama\-3\.2\-1B\([Meta AI, 2024](https://arxiv.org/html/2610.28975#bib.bib82)\)\.[Figure2](https://arxiv.org/html/2610.28975#S1.F2)\(right\) shows the empirical𝔼^​\[𝑮​𝑮⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]of the four projections in Qwen3\.5, whose mass concentrates on the diagonal\. For the first head of the first eligible attention block of each model we vary the model widthd0=d2d\_\{0\}=d\_\{2\}at fixed head dimensiondk=dvd\_\{k\}=d\_\{v\}, using2525random initializations and6464fixed FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.28975#bib.bib83)\)sequences per configuration \(details in[SectionH\.2](https://arxiv.org/html/2610.28975#A8.SS2)\)\.[Figure6](https://arxiv.org/html/2610.28975#S6.F6)showsR^\\widehat\{R\}increasing with model width for all four projections in all three models, as theΘ⁡\(dv​d03\)\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)ratio of[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)predicts\.

Figure 6:Model width scaling ofR^\\widehat\{R\}for the query, key, value, and output projections in the first head of the first eligible attention block of the three language models at random initialization\.
## 7Conclusion and Limitations

#### Conclusion\.

This work provides an asymptotic theory that shows Muon’s implicit preconditioner matches the row\-block diagonally dominant structure of the neural\-network Hessian, because𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]itself becomes diagonally dominant as the weight matrices are overparameterized, at the explicit rates of[Table1](https://arxiv.org/html/2610.28975#S4.T1)\. Specifically, in matrix factorization the isotropicΘ⁡\(k3\)\\Theta\(k^\{3\}\)diagonal outgrows the target\-drivenΘ⁡\(k2\)\\Theta\(k^\{2\}\)off\-diagonal\. In deep linear networks the hidden layers are exactly diagonal, and the ratio at the output layer grows with the product of all hidden dimensions\. In the two\-layer ReLU network the nonlinearity leaves an off\-diagonal of constant order, yet the ratio still grows linearly with the hidden dimension\. In linear attention the query, key and value weights are exactly diagonal\. For the output weight the ratioΘ⁡\(dv​d03\)\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)grows with the head dimension and grows as the cube of the model width, so the dominance strengthens as models get wider\. Our experiments on language models of increasing width match this intuition\.

#### Limitations\.

Our theory is stated at initialization and in expectation: it does not characterize the training dynamics of𝑽t​𝑽t⊤\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}, although the empirical evidence of[Deng et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib27)indicates that the dominance persists during training\. It is also worth studying whether this dominance reflects a degenerate behavior of the model that emerges at large scale\. We leave these questions to future work\.

### Ethics statement

This work is primarily theoretical: it characterizes the expected gradient self outer\-product of standard neural\-network models at initialization\. The experiments consist of synthetic simulations and training runs of publicly available language models on standard public corpora; they involve no human subjects, no personal or sensitive data, and no newly collected datasets\. We do not foresee ethical concerns or direct negative societal impacts arising from this work\.

### Reproducibility statement

All results are stated under explicit assumptions \([Section3\.2](https://arxiv.org/html/2610.28975#S3.SS2)\) and proved in full in the appendix: Appendix[B](https://arxiv.org/html/2610.28975#A2)\(symmetric matrix factorization\), Appendix[C](https://arxiv.org/html/2610.28975#A3)\(deep linear networks\), Appendix[D](https://arxiv.org/html/2610.28975#A4)\(two\-layer ReLU networks\) and Appendix[E](https://arxiv.org/html/2610.28975#A5)\(single\-layer linear attention\)\. Each appendix restates the corresponding theorem together with its standing assumption and all supporting lemmas, and the closed\-form Wishart trace moments used for the attention layer are listed explicitly in Lemma[33](https://arxiv.org/html/2610.28975#Thmlemma33)\. The numerical checks \(Appendix[G](https://arxiv.org/html/2610.28975#A7)\) specify every dimension, sample size and initialization used, and the language\-model experiments report the model sizes, datasets, optimizer settings and number of training steps; the code for both the simulations and the language\-model runs is provided at the anonymous repository[https://anonymous\.4open\.science/r/orthogonalization\-hessian\-CB2C/](https://anonymous.4open.science/r/orthogonalization-hessian-CB2C/)\.

## References

- Anet al\.\(2025\)K\. An, Y\. Liu, R\. Pan, Y\. Ren, S\. Ma, D\. Goldfarb, and T\. ZhangASGO: adaptive structured gradient optimization\.arXiv preprint arXiv:2503\.20762\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Anilet al\.\(2020\)R\. Anil, V\. Gupta, T\. Koren, K\. Regan, and Y\. SingerScalable second order optimization for deep learning\.arXiv preprint arXiv:2002\.09018\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Anilet al\.\(2019\)R\. Anil, V\. Gupta, T\. Koren, and Y\. SingerMemory\-efficient adaptive optimization\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Beneventanoet al\.\(2026\)P\. Beneventano, M\. Abdelmoneum, and T\. PoggioThe spectral dynamics and noise geometry of Muon\.arXiv preprint arXiv:2606\.08388\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Bernstein and Newhouse \(2024a\)J\. Bernstein and L\. NewhouseModular duality in deep learning\.arXiv preprint arXiv:2410\.21265\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Bernstein and Newhouse \(2024b\)J\. Bernstein and L\. NewhouseOld optimizer, new norm: an anthology\.arXiv preprint arXiv:2409\.20325\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brownet al\.Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4)\.
- Changet al\.\(2026\)D\. Chang, Y\. Liu, and G\. YuanOn the convergence of Muon and beyond\.arXiv preprint arXiv:2509\.15816\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Che \(2026\)T\. CheSecond\-order Muon done right: a principled marriage of spectral geometry and curvature\.arXiv preprint arXiv:2608\.09763\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2025\)L\. Chen, J\. Li, and Q\. LiuMuon optimizes under spectral norm constraints\.arXiv preprint arXiv:2506\.15054\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Cho and Saul \(2009\)Y\. Cho and L\. K\. SaulKernel methods for deep learning\.InAdvances in Neural Information Processing Systems,Vol\.22,pp\. 342–350\.Cited by:[§4\.5](https://arxiv.org/html/2610.28975#S4.SS5.p2.1),[Lemma 19](https://arxiv.org/html/2610.28975#Thmlemma19.p1.1.1)\.
- Choudhuryet al\.\(2026\)S\. Choudhury, X\. Cheng, M\. Takáč, S\. Na, and M\. KolarMuon with Nesterov momentum: heavy\-tailed noise and \(randomized\) inexact polar decomposition\.arXiv preprint arXiv:2605\.06884\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Collobert \(2004\)R\. CollobertLarge scale machine learning\.Ph\.D\. Thesis,Université de Paris VI\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Davis and Drusvyatskiy \(2025\)D\. Davis and D\. DrusvyatskiyWhen do spectral gradient updates help in deep learning?\.arXiv preprint arXiv:2512\.04299\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p2.1),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Denget al\.\(2026\)S\. Deng, Z\. Ouyang, T\. Pang, Z\. Liu, R\. Jin, S\. Yu, and Y\. YangRMNP: row\-momentum normalized preconditioning for scalable matrix\-based optimization\.arXiv preprint arXiv:2603\.20527\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.28975#A6.p1.2),[§1](https://arxiv.org/html/2610.28975#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2610.28975#S1.SS0.SSS0.Px1.p3.1),[§7](https://arxiv.org/html/2610.28975#S7.SS0.SSS0.Px2.p1.1),[Remark 6](https://arxiv.org/html/2610.28975#Thmremark6.p1.1.1)\.
- Doet al\.\(2026\)T\. Do, S\. Dereich, and A\. JentzenOn muon optimization: from non\-convergence to an error analysis with polar express and the Newton\-Schulz polynomial from implementations\.arXiv preprint arXiv:2608\.04607\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Donget al\.\(2025\)Z\. Dong, Y\. Zhang, Z\. Luo, J\. Yao, and R\. SunTowards quantifying the hessian structure of neural networks\.External Links:2505\.02809,[Link](https://arxiv.org/abs/2505.02809)Cited by:[§1](https://arxiv.org/html/2610.28975#S1.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Du and Su \(2026\)Z\. Du and W\. SuThe Newton\-Muon optimizer\.arXiv preprint arXiv:2604\.01472\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Eschenhagenet al\.\(2026\)R\. Eschenhagen, A\. Cai, T\. Lee, and H\. M\. ShiClarifying shampoo: adapting spectral descent to stochasticity and the parameter trajectory\.arXiv preprint arXiv:2602\.09314\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Fanet al\.\(2025\)C\. Fan, M\. Schmidt, and C\. ThrampoulidisImplicit bias of spectral descent and Muon on multiclass separable data\.arXiv preprint arXiv:2502\.04664\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p2.1)\.
- Glentiset al\.\(2025\)A\. Glentis, J\. Li, A\. Han, and M\. HongA minimalist optimizer design for LLM pretraining\.arXiv preprint arXiv:2506\.16659\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.28975#A6.p1.2)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4)\.
- Gratton and Toint \(2026\)S\. Gratton and Ph\. L\. TointA unified convergence theory for adaptive first\-order methods in the nonconvex case, including adanorm, full and diagonal adagrad and Muon\.arXiv preprint arXiv:2604\.17423\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Gu and Xie \(2026\)Y\. Gu and Z\. XieMano: restriking manifold optimization for LLM training\.arXiv preprint arXiv:2601\.23000\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Guptaet al\.\(2018\)V\. Gupta, T\. Koren, and Y\. SingerShampoo: preconditioned stochastic tensor optimization\.arXiv preprint arXiv:1802\.09568\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Hübleret al\.\(2026a\)F\. Hübler, K\. Lion, A\. Orvieto, and N\. HeMuown implicitly performs angular step\-size decay\.arXiv preprint arXiv:2606\.23637\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Hübleret al\.\(2026b\)F\. Hübler, T\. Pethick, and S\. SraFree heavy\-tailed lunch for Muon: a theoretical justification of empirical success\.arXiv preprint arXiv:2606\.14560\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- IBM Granite Team \(2025\)IBM Granite TeamGranite\-4\.0\-1b\-base\.Note:[https://huggingface\.co/ibm\-granite/granite\-4\.0\-1b\-base](https://huggingface.co/ibm-granite/granite-4.0-1b-base)Model cardCited by:[Appendix H](https://arxiv.org/html/2610.28975#A8.p1.1),[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4),[§6](https://arxiv.org/html/2610.28975#S6.p1.1)\.
- Iiduka \(2026\)H\. IidukaMuon converges under heavy\-tailed noise: nonconvex hölder\-smooth empirical risk minimization\.arXiv preprint arXiv:2603\.15059\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Isserlis \(1918\)L\. IsserlisOn a formula for the product\-moment coefficient of any order of a normal frequency distribution in any number of variables\.Biometrika12\(1–2\),pp\. 134–139\.Cited by:[Lemma 3](https://arxiv.org/html/2610.28975#Thmlemma3.p1.2.1)\.
- Janson \(1997\)S\. JansonGaussian Hilbert spaces\.Cambridge University Press\.Cited by:[§B\.4](https://arxiv.org/html/2610.28975#A2.SS4.p3.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.Note:[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.1),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Kim and Oh \(2026\)G\. Y\. Kim and M\. OhConvergence of Muon with Newton\-Schulz\.arXiv preprint arXiv:2601\.19156\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Kovalev and Borodich \(2026\)D\. Kovalev and E\. BorodichNon\-Euclidean SGD for structured optimization: unified analysis and improved rates\.arXiv preprint arXiv:2511\.11466\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Kovalev \(2025\)D\. KovalevUnderstanding gradient orthogonalization for deep learning via non\-Euclidean trust\-region optimization\.arXiv preprint arXiv:2503\.12645\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Laiet al\.\(2026\)L\. Lai, T\. Lin, and J\. ZhangNonsmooth optimization via orthogonalized momentum\.arXiv preprint arXiv:2609\.13677\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Lauet al\.\(2025\)T\. T\. Lau, Q\. Long, and W\. SuPolarGrad: a class of matrix\-gradient optimizers from a unifying preconditioning perspective\.arXiv preprint arXiv:2505\.21799\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Le Rouxet al\.\(2008\)N\. Le Roux, P\. Manzagol, and Y\. BengioTopmoumoute online natural gradient algorithm\.InAdvances in Neural Information Processing Systems 20,Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Li and Hong \(2025\)J\. Li and M\. HongA note on the convergence of Muon\.arXiv preprint arXiv:2502\.02900\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Li and Tsuchiya \(2026\)M\. Li and T\. TsuchiyaMuon with finite Newton\-Schulz: the smoothing benefit in nonsmooth nonconvex optimization\.arXiv preprint arXiv:2608\.26288\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Lionet al\.\(2026\)K\. Lion, F\. Hübler, B\. Li, A\. Orvieto, and N\. HeMuown: row\-norm control for Muon optimization\.arXiv preprint arXiv:2605\.10797\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, S\. Chen, J\. Liu, and C\. XuA continuous\-time analysis of smoothed matrix\-polar spectral gradient flows for Muon\-type optimization\.arXiv preprint arXiv:2608\.01911\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Longet al\.\(2026\)Y\. Long, Y\. Gu, and Z\. XieDimension\-free saddle\-point escape in Muon\.arXiv preprint arXiv:2605\.09331\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Maet al\.\(2025\)C\. Ma, W\. Gong, M\. Scetbon, and E\. MeedsSWAN: SGD with normalization and whitening enables stateless LLM training\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.28975#A6.p1.2)\.
- Maet al\.\(2026\)J\. Ma, Y\. Huang, Y\. Chi, and Y\. ChenPreconditioning benefits of spectral orthogonalization in muon\.External Links:2601\.13474,[Link](https://arxiv.org/abs/2601.13474)Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p2.1),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1),[Remark 1](https://arxiv.org/html/2610.28975#Thmremark1.p1.1.1)\.
- Martens and Grosse \(2015\)J\. Martens and R\. GrosseOptimizing neural networks with Kronecker\-factored approximate curvature\.arXiv preprint arXiv:1503\.05671\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Meta AI \(2024\)Meta AILlama 3\.2 1B\.Note:[https://huggingface\.co/meta\-llama/Llama\-3\.2\-1B](https://huggingface.co/meta-llama/Llama-3.2-1B)Model cardCited by:[Appendix H](https://arxiv.org/html/2610.28975#A8.p1.1),[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4),[§6](https://arxiv.org/html/2610.28975#S6.p1.1)\.
- Morwaniet al\.\(2024\)D\. Morwani, I\. Shapira, N\. Vyas, E\. Malach, S\. Kakade, and L\. JansonA new perspective on Shampoo’s preconditioner\.arXiv preprint arXiv:2406\.17748\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Nagashima and Iiduka \(2026\)S\. Nagashima and H\. IidukaImproved convergence rates of Muon optimizer for nonconvex optimization\.arXiv preprint arXiv:2601\.19400\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Nguyenet al\.\(2026\)T\. Nguyen, T\. Nguyen, M\. Truong, T\. Nguyen, J\. Bailey, and T\. LeSpectral flattening is all Muon needs: how orthogonalization controls learning rate and convergence\.arXiv preprint arXiv:2605\.13079\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Nourdin and Peccati \(2012\)I\. Nourdin and G\. PeccatiNormal approximations with Malliavin calculus: from Stein’s method to universality\.Cambridge University Press\.Cited by:[§D\.4](https://arxiv.org/html/2610.28975#A4.SS4.p1.1)\.
- Ormaniecet al\.\(2024\)W\. Ormaniec, F\. Dangel, and S\. P\. SinghWhat does it mean to be a Transformer? insights from a theoretical Hessian analysis\.arXiv preprint arXiv:2410\.10986\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Panget al\.\(2026\)T\. Pang, Y\. Fang, Z\. Liu, S\. Deng, L\. Hsiung, S\. Yu, and Y\. YangHTMuon: improving muon via heavy\-tailed spectral correction\.External Links:2603\.10067,[Link](https://arxiv.org/abs/2603.10067)Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Parshakovaet al\.\(2026\)T\. Parshakova, A\. Khaled, M\. Crawshaw, G\. Garrigos, and R\. M\. GowerMuon does not converge on convex Lipschitz functions\.arXiv preprint arXiv:2605\.08980\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,Cited by:[§H\.1](https://arxiv.org/html/2610.28975#A8.SS1.p2.2),[§6](https://arxiv.org/html/2610.28975#S6.p1.1)\.
- Pethicket al\.\(2025\)T\. Pethick, W\. Xie, K\. Antonakopoulos, Z\. Zhu, A\. Silveti\-Falls, and V\. CevherTraining deep learning models with norm\-constrained LMOs\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.28975#A6.p1.2),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Pethick \(2026\)T\. PethickWhen to use what Schatten\-ppnorm in deep learning?\.arXiv preprint arXiv:2606\.15268\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Price \(1958\)R\. PriceA useful theorem for nonlinear devices having Gaussian inputs\.IRE Transactions on Information TheoryIT\-4\(2\),pp\. 69–72\.Cited by:[§D\.4](https://arxiv.org/html/2610.28975#A4.SS4.p3.1)\.
- Qianet al\.\(2025\)X\. Qian, H\. Rammal, D\. Kovalev, and P\. RichtárikMuon is provably faster with momentum variance reduction\.arXiv preprint arXiv:2512\.16598\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Qian and Richtárik \(2026\)X\. Qian and P\. RichtárikConvergence analysis of Muon\-type methods with inexact lmo in the degenerate case\.arXiv preprint arXiv:2606\.21581\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.Note:[https://huggingface\.co/Qwen/Qwen3\.5\-0\.8B\-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)Qwen3\.5\-0\.8B\-Base model cardCited by:[Appendix H](https://arxiv.org/html/2610.28975#A8.p1.1),[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4),[§6](https://arxiv.org/html/2610.28975#S6.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Note:OpenAI technical reportCited by:[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4)\.
- Satoet al\.\(2026\)N\. Sato, H\. Naganuma, and H\. IidukaConvergence bound and critical batch size of Muon optimizer\.arXiv preprint arXiv:2507\.01598\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Scetbonet al\.\(2025\)M\. Scetbon, C\. Ma, W\. Gong, and E\. MeedsGradient multi\-normalization for stateless and scalable LLM training\.arXiv preprint arXiv:2502\.06742\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.28975#A6.p1.2)\.
- Shenet al\.\(2025\)W\. Shen, R\. Huang, M\. Huang, C\. Shen, and J\. ZhangOn the convergence analysis of Muon\.arXiv preprint arXiv:2505\.23737\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Shulginet al\.\(2025\)E\. Shulgin, S\. AlRashed, F\. Orabona, and P\. RichtárikBeyond the ideal: analyzing the inexact Muon update\.arXiv preprint arXiv:2510\.19933\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Shumaylovet al\.\(2026\)Z\. Shumaylov, N\. D\. Costa, P\. Zaika, B\. Mucsányi, A\. Massucco, Y\. Gelberg, C\. Schönlieb, Y\. Gal, and P\. HennigMuon is not that special: random or inverted spectra work just as well\.arXiv preprint arXiv:2605\.11181\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Stein \(1981\)C\. M\. SteinEstimation of the mean of a multivariate normal distribution\.The Annals of Statistics9\(6\),pp\. 1135–1151\.Cited by:[§D\.4](https://arxiv.org/html/2610.28975#A4.SS4.p1.1)\.
- Su \(2025\)W\. SuIsotropic curvature model for understanding deep learning optimization: is gradient orthogonalization optimal?\.arXiv preprint arXiv:2511\.00674\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Tomihari and Sato \(2025\)A\. Tomihari and I\. SatoGradient heterogeneity complements Hessian heterogeneity in Transformer optimization\.arXiv preprint arXiv:2502\.00213\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvronet al\.LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4)\.
- Vasudevaet al\.\(2025\)B\. Vasudeva, P\. Deora, Y\. Zhao, V\. Sharan, and C\. ThrampoulidisHow Muon’s spectral design benefits generalization: a study on imbalanced data\.arXiv preprint arXiv:2510\.22980\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p2.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.1](https://arxiv.org/html/2610.28975#S3.SS1.SSS0.Px4.p1.4)\.
- Vyaset al\.\(2024\)N\. Vyas, D\. Morwani, R\. Zhao, M\. Kwun, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. KakadeSOAP: improving and stabilizing Shampoo using Adam\.arXiv preprint arXiv:2409\.11321\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025a\)J\. Wang, M\. Wang, Z\. Zhou, J\. Yan, W\. E, and L\. WuThe sharpness disparity principle in Transformers for accelerating language model pre\-training\.arXiv preprint arXiv:2502\.19002\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)S\. Wang, F\. Zhang, J\. Li, D\. Bergemann, and Z\. YangWhy Muon outperforms Adam: a curvature perspective\.arXiv preprint arXiv:2606\.04662\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)S\. Wang, F\. Zhang, J\. Li, C\. Du, C\. Du, T\. Pang, Z\. Yang, M\. Hong, and V\. Y\. F\. TanMuon outperforms Adam in tail\-end associative memory learning\.arXiv preprint arXiv:2509\.26030\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Wenet al\.\(2025\)Z\. Wen, Y\. Shi, J\. Wang, P\. Luo, L\. Qiao, D\. Li, and T\. SunSRON: state\-free LLM training via row\-wise gradient normalization\.InOpenReview preprint,Note:[https://openreview\.net/forum?id=BtQLBWr6zI](https://openreview.net/forum?id=BtQLBWr6zI)Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.28975#A6.p1.2)\.
- Wick \(1950\)G\. WickThe evaluation of the collision matrix\.Physical Review80\(2\),pp\. 268–272\.Cited by:[Lemma 3](https://arxiv.org/html/2610.28975#Thmlemma3.p1.2.1)\.
- Wolfet al\.\(2020\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. RushTransformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 38–45\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[§H\.1](https://arxiv.org/html/2610.28975#A8.SS1.p1.1)\.
- Wuet al\.\(2026\)X\. Wu, W\. Yu, C\. Zhang, and P\. WoodlandSpectral allocation: why Muon outperforms Adam, and how to improve Muon\.arXiv preprint arXiv:2608\.25990\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2026\)R\. Xu, J\. Li, and Y\. LuOn the width scaling of neural optimizers under matrix operator norms I: row/column normalization and hyperparameter transfer\.arXiv preprint arXiv:2603\.09952\.Cited by:[Appendix A](https://arxiv.org/html/2610.28975#A1.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, Y\. He, and S\. LiConvergence of spectral descent for non\-smooth optimization\.arXiv preprint arXiv:2605\.26977\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.p1.2)\.
- Zhang \(2026\)H\. ZhangPolarAdamW: disentangling spectral control and schur gauge\-equivariance in matrix optimisation\.arXiv preprint arXiv:2605\.07067\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026a\)H\. Zhang, W\. Zhou, W\. Chen, and X\. ChengWhen and why grouping attention heads accelerates Muon optimization\.arXiv preprint arXiv:2605\.08933\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026b\)Q\. Zhang, J\. Sun, and Y\. LiuBeyond the matrix sign: quadratic spectral descent\.arXiv preprint arXiv:2609\.07597\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)T\. T\. Zhang, B\. Moniri, A\. Nagwekar, F\. Rahman, A\. Xue, H\. Hassani, and N\. MatniOn the concurrence of layer\-wise preconditioning methods and provable feature learning\.arXiv preprint arXiv:2502\.01763\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026c\)Y\. Zhang, S\. Xing, J\. Huang, K\. Lv, Y\. Zhou, X\. Qiu, Q\. Guo, and K\. ChenMousse: rectifying the geometry of Muon with curvature\-aware preconditioning\.arXiv preprint arXiv:2603\.09697\.Cited by:[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024a\)Y\. Zhang, C\. Chen, T\. Ding, Z\. Li, R\. Sun, and Z\. LuoWhy transformers need Adam: a Hessian perspective\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.28975#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2610.28975#S1.p2.1.3),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2024b\)Y\. Zhang, C\. Chen, Z\. Li, T\. Ding, C\. Wu, Y\. Ye, Z\. Luo, and R\. SunAdam\-mini: use fewer learning rates to gain more\.arXiv preprint arXiv:2406\.16793\.Cited by:[§1](https://arxiv.org/html/2610.28975#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2610.28975#S1.p2.1.3),[§2](https://arxiv.org/html/2610.28975#S2.SS0.SSS0.Px2.p1.1)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2610.28975#S1)
2. [2Related Work](https://arxiv.org/html/2610.28975#S2)
3. [3Preliminaries](https://arxiv.org/html/2610.28975#S3)1. [3\.1Setup](https://arxiv.org/html/2610.28975#S3.SS1) 2. [3\.2Assumptions](https://arxiv.org/html/2610.28975#S3.SS2)
4. [4Main Results](https://arxiv.org/html/2610.28975#S4)1. [4\.1Symmetric matrix factorization](https://arxiv.org/html/2610.28975#S4.SS1) 2. [4\.2Deep linear network](https://arxiv.org/html/2610.28975#S4.SS2) 3. [4\.3Two\-layer ReLU network](https://arxiv.org/html/2610.28975#S4.SS3) 4. [4\.4Single\-layer linear attention](https://arxiv.org/html/2610.28975#S4.SS4) 5. [4\.5Summary and proof sketch](https://arxiv.org/html/2610.28975#S4.SS5)
5. [5Simulation of Theoretical Results](https://arxiv.org/html/2610.28975#S5)
6. [6Empirical Evaluation on Large Language Models](https://arxiv.org/html/2610.28975#S6)
7. [7Conclusion and Limitations](https://arxiv.org/html/2610.28975#S7)
8. [References](https://arxiv.org/html/2610.28975#bib)
9. [AExtended Related Work](https://arxiv.org/html/2610.28975#A1)
10. [BSymmetric Matrix Factorization](https://arxiv.org/html/2610.28975#A2)1. [B\.1Setup](https://arxiv.org/html/2610.28975#A2.SS1) 2. [B\.2Standing assumption](https://arxiv.org/html/2610.28975#A2.SS2) 3. [B\.3Main results restated](https://arxiv.org/html/2610.28975#A2.SS3) 4. [B\.4Preliminary lemmas](https://arxiv.org/html/2610.28975#A2.SS4) 5. [B\.5Proofs](https://arxiv.org/html/2610.28975#A2.SS5)
11. [CDeep Linear Networks](https://arxiv.org/html/2610.28975#A3)1. [C\.1Setup](https://arxiv.org/html/2610.28975#A3.SS1) 2. [C\.2Standing assumption](https://arxiv.org/html/2610.28975#A3.SS2) 3. [C\.3Main results restated](https://arxiv.org/html/2610.28975#A3.SS3) 4. [C\.4Preliminary lemmas](https://arxiv.org/html/2610.28975#A3.SS4) 5. [C\.5Proofs](https://arxiv.org/html/2610.28975#A3.SS5)
12. [DTwo\-Layer ReLU Networks](https://arxiv.org/html/2610.28975#A4)1. [D\.1Setup](https://arxiv.org/html/2610.28975#A4.SS1) 2. [D\.2Standing assumption](https://arxiv.org/html/2610.28975#A4.SS2) 3. [D\.3Main results restated](https://arxiv.org/html/2610.28975#A4.SS3) 4. [D\.4Preliminary lemmas](https://arxiv.org/html/2610.28975#A4.SS4) 5. [D\.5Proof of the gradient self outer\-product of𝑾2\\bm\{W\}\_\{2\}](https://arxiv.org/html/2610.28975#A4.SS5) 6. [D\.6Proof of the gradient self outer\-product of𝑾1\\bm\{W\}\_\{1\}](https://arxiv.org/html/2610.28975#A4.SS6) 7. [D\.7Discussion](https://arxiv.org/html/2610.28975#A4.SS7)
13. [ESingle\-Layer Linear Attention](https://arxiv.org/html/2610.28975#A5)1. [E\.1Setup](https://arxiv.org/html/2610.28975#A5.SS1) 2. [E\.2Standing assumption](https://arxiv.org/html/2610.28975#A5.SS2) 3. [E\.3Main results restated](https://arxiv.org/html/2610.28975#A5.SS3) 4. [E\.4Preliminary lemmas](https://arxiv.org/html/2610.28975#A5.SS4) 5. [E\.5Proofs](https://arxiv.org/html/2610.28975#A5.SS5) 6. [E\.6Discussion](https://arxiv.org/html/2610.28975#A5.SS6)
14. [FAsymptotic Approximation between Orthogonalization and Row Normalization](https://arxiv.org/html/2610.28975#A6)
15. [GSimulation of Theoretical Results](https://arxiv.org/html/2610.28975#A7)1. [G\.1Symmetric Matrix Factorization](https://arxiv.org/html/2610.28975#A7.SS1) 2. [G\.2Deep Linear Network](https://arxiv.org/html/2610.28975#A7.SS2) 3. [G\.3Two\-Layer ReLU Network](https://arxiv.org/html/2610.28975#A7.SS3) 4. [G\.4Single\-Layer Attention](https://arxiv.org/html/2610.28975#A7.SS4)
16. [HEmpirical Evaluation on Large Language Models](https://arxiv.org/html/2610.28975#A8)1. [H\.1Gradient Gram Structure at Initialization](https://arxiv.org/html/2610.28975#A8.SS1) 2. [H\.2Model Width Scaling](https://arxiv.org/html/2610.28975#A8.SS2)

## Appendix AExtended Related Work

Mechanistic analyses of Muon and the heterogeneity of the neural\-network Hessian are discussed in[Section2](https://arxiv.org/html/2610.28975#S2); this appendix covers the remaining line of work\.

#### Preconditioning methods for neural\-network optimization\.

Muon belongs to the family of optimizers defined by a linear minimization oracle \(LMO\) over a norm ball\([Pethick et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib13);[Chen et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib7);[Kovalev, 2025](https://arxiv.org/html/2610.28975#bib.bib6)\), which also contains Scion and Gluon, unifies sign, spectral and Schatten\-ppupdates\([Lau et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib11);[An et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib12);[Pang et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib89);[Pethick, 2026](https://arxiv.org/html/2610.28975#bib.bib47)\), and comes with convergence theory for the idealized and the practical algorithm\([Li and Hong, 2025](https://arxiv.org/html/2610.28975#bib.bib4);[Shen et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib5);[Kovalev and Borodich, 2026](https://arxiv.org/html/2610.28975#bib.bib54)\)\. Two\-sided Kronecker preconditioners such as Shampoo and SOAP whiten both the row and the column side of a matrix gradient\([Gupta et al\., 2018](https://arxiv.org/html/2610.28975#bib.bib58);[Morwani et al\., 2024](https://arxiv.org/html/2610.28975#bib.bib59);[Vyas et al\., 2024](https://arxiv.org/html/2610.28975#bib.bib60)\);[Anil et al\. \(2020\)](https://arxiv.org/html/2610.28975#bib.bib79)make this second\-order preconditioning practical at the scale of modern deep learning, and[Anil et al\. \(2019\)](https://arxiv.org/html/2610.28975#bib.bib78)go in the opposite direction, compressing the state of adaptive optimizers to a memory\-efficient factored form\. A parallel line keeps only the row side and drops optimizer state by preconditioning the gradient row by row: SWAN, gradient multi\-normalization, a minimalist optimizer and SRON replace Adam’s state with a row\-wiseℓ2\\ell\_\{2\}normalization\([Ma et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib21);[Scetbon et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib22);[Glentis et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib23);[Wen et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib24)\);[Gu and Xie \(2026\)](https://arxiv.org/html/2610.28975#bib.bib25);[Xu et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib26)study its manifold\-optimization and width\-scaling sides;[Pethick et al\. \(2025\)](https://arxiv.org/html/2610.28975#bib.bib13)reach it from norm\-constrained steepest descent;[Lion et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib55)make the row magnitudes an explicit optimizer variable, which[Hübler et al\. \(2026a\)](https://arxiv.org/html/2610.28975#bib.bib56)show implicitly decays the angular step size; and[Deng et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib27)show that row\-momentum normalization matches Muon at scale\. Our analysis explains the common thread: once𝑽​𝑽⊤\\bm\{V\}\\bm\{V\}^\{\\top\}is diagonally dominant, row normalization and orthogonalization are the same operation \([AppendixF](https://arxiv.org/html/2610.28975#A6)\)\.

## Appendix BSymmetric Matrix Factorization

### B\.1Setup

Letd,k,rd,k,rbe positive integers withk≥rk\\geq r\. The over\-parameterized symmetric matrix\-factorization loss is

ℒ⁡\(𝑾\):=14​‖𝑾​𝑾⊤−𝑴⋆‖F2,𝑾∈ℝd×k,\\mathcal\{L\}\(\\bm\{W\}\)\\;:=\\;\\tfrac\{1\}\{4\}\\,\\left\\\|\\bm\{W\}\\bm\{W\}^\{\\top\}\-\\bm\{M\}^\{\\star\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},\\qquad\\bm\{W\}\\in\\mathbb\{R\}^\{d\\times k\},\(15\)with target𝑴⋆∈ℝd×d\\bm\{M\}^\{\\star\}\\in\\mathbb\{R\}^\{d\\times d\}symmetric positive semidefinite of rankrr\. A direct computation gives the gradient

𝑮:=∇ℒ​\(𝑾\)=\(𝑾​𝑾⊤−𝑴⋆\)​𝑾∈ℝd×k,\\bm\{G\}\\;:=\\;\\nabla\\mathcal\{L\}\(\\bm\{W\}\)\\;=\\;\(\\bm\{W\}\\bm\{W\}^\{\\top\}\-\\bm\{M\}^\{\\star\}\)\\,\\bm\{W\}\\;\\in\\;\\mathbb\{R\}^\{d\\times k\},\(16\)and the object of interest is thed×dd\\times dmatrix𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\.

### B\.2Standing assumption

###### Assumption 5\.

The entriesWi​jW\_\{ij\}are i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\), and𝐌⋆\\bm\{M\}^\{\\star\}is deterministic and independent of𝐖\\bm\{W\}\. Hence𝐒:=𝐖​𝐖⊤\\bm\{S\}:=\\bm\{W\}\\bm\{W\}^\{\\top\}is Wishart withkkdegrees of freedom and identity scale,𝐒∼𝒲d​\(k,𝐈d\)\\bm\{S\}\\sim\\mathcal\{W\}\_\{d\}\(k,\\bm\{I\}\_\{d\}\)\.

### B\.3Main results restated

Theorem \([Theorem1](https://arxiv.org/html/2610.28975#Thmtheorem1), restated\)\.Under[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5),

𝔼⁡\[𝑮​𝑮⊤\]=α3​\(d,k\)​𝑰d−2​α2​\(d,k\)​𝑴⋆\+k​\(𝑴⋆\)2,\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\\;=\\;\\alpha\_\{3\}\(d,k\)\\,\\bm\{I\}\_\{d\}\\;\-\\;2\\,\\alpha\_\{2\}\(d,k\)\\,\\bm\{M\}^\{\\star\}\\;\+\\;k\\,\(\\bm\{M\}^\{\\star\}\)^\{2\},\(17\)where the scalars are the diagonal entries of the first three Wishart moments,

α1​\(d,k\)\\displaystyle\\alpha\_\{1\}\(d,k\)=k,\\displaystyle=k,\(18\)α2​\(d,k\)\\displaystyle\\alpha\_\{2\}\(d,k\)=k⁡\(k\+d\+1\),\\displaystyle=k\(k\+d\+1\),\(19\)α3​\(d,k\)\\displaystyle\\alpha\_\{3\}\(d,k\)=k3\+3​\(d\+1\)​k2\+\(d2\+3​d\+4\)​k\.\\displaystyle=k^\{3\}\+3\(d\+1\)k^\{2\}\+\(d^\{2\}\+3d\+4\)k\.\(20\)
###### Corollary 1\.

Under[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5),𝔼⁡\[𝐆​𝐆⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]is invariant under any orthogonal change of basis fixing𝐌⋆\\bm\{M\}^\{\\star\}, with

𝔼⁡\[\(𝑮​𝑮⊤\)i​i\]\\displaystyle\\mathbb\{E\}\\bigl\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ii\}\\bigr\]=α3−2​α2​𝑴i​i⋆\+k​\(𝑴⋆\)i​i2,\\displaystyle=\\alpha\_\{3\}\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\_\{ii\}\+k\(\\bm\{M\}^\{\\star\}\)^\{2\}\_\{ii\},\(21\)𝔼⁡\[\(𝑮​𝑮⊤\)i​j\]\\displaystyle\\mathbb\{E\}\\bigl\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ij\}\\bigr\]=−2α2𝑴i​j⋆\+k\(𝑴⋆\)i​j2\(i≠j\)\.\\displaystyle=\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\_\{ij\}\+k\(\\bm\{M\}^\{\\star\}\)^\{2\}\_\{ij\}\\qquad\(i\\neq j\)\.\(22\)Ask→∞k\\to\\infty,𝔼⁡\[\(𝐆​𝐆⊤\)i​i\]=Θ⁡\(k3\)\\mathbb\{E\}\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ii\}\]=\\Theta\(k^\{3\}\)and𝔼⁡\[\(𝐆​𝐆⊤\)i​j\]=Θ⁡\(k2\)\\mathbb\{E\}\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ij\}\]=\\Theta\(k^\{2\}\)whenever𝐌i​j⋆≠0\\bm\{M\}^\{\\star\}\_\{ij\}\\neq 0, so the dominance ratio isΘ⁡\(k/𝐌i​j⋆\)\\Theta\(k/\\bm\{M\}^\{\\star\}\_\{ij\}\)\.

### B\.4Preliminary lemmas

###### Lemma 1\.

Let𝐗∈ℝm×n\\bm\{X\}\\in\\mathbb\{R\}^\{m\\times n\}have i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries\. Then for deterministic𝐌∈ℝn×n\\bm\{M\}\\in\\mathbb\{R\}^\{n\\times n\}and𝐍∈ℝm×m\\bm\{N\}\\in\\mathbb\{R\}^\{m\\times m\},

𝔼⁡\[𝑿​𝑴​𝑿⊤\]=tr⁡\(𝑴\)​𝑰m,𝔼⁡\[𝑿⊤​𝑵​𝑿\]=tr⁡\(𝑵\)​𝑰n\.\\mathbb\{E\}\[\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\]=\\tr\(\\bm\{M\}\)\\,\\bm\{I\}\_\{m\},\\qquad\\mathbb\{E\}\[\\bm\{X\}^\{\\top\}\\bm\{N\}\\bm\{X\}\]=\\tr\(\\bm\{N\}\)\\,\\bm\{I\}\_\{n\}\.\(23\)

Proof\.Using𝔼⁡\[𝑿a​p​𝑿b​q\]=δa​b​δp​q\\mathbb\{E\}\[\\bm\{X\}\_\{ap\}\\bm\{X\}\_\{bq\}\]=\\delta\_\{ab\}\\delta\_\{pq\},

𝔼⁡\[\(𝑿​𝑴​𝑿⊤\)a​b\]=∑p,q𝑴p​q​δa​b​δp​q=δa​b​tr⁡\(𝑴\),\\mathbb\{E\}\[\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)\_\{ab\}\]=\\sum\_\{p,q\}\\bm\{M\}\_\{pq\}\\,\\delta\_\{ab\}\\delta\_\{pq\}=\\delta\_\{ab\}\\tr\(\\bm\{M\}\),which is the first identity; the second follows by applying it to𝑿⊤\\bm\{X\}^\{\\top\}\(entries i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\)\.

###### Lemma 2\.

Let𝐒=𝐖​𝐖⊤\\bm\{S\}=\\bm\{W\}\\bm\{W\}^\{\\top\}with𝐖\\bm\{W\}as in[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5)\. For everym∈ℕm\\in\\mathbb\{N\}there is a scalarαm​\(d,k\)\\alpha\_\{m\}\(d,k\)with

𝔼⁡\[𝑺m\]=αm​\(d,k\)​𝑰d,αm​\(d,k\)=1d​𝔼​\[tr⁡\(𝑺m\)\]\.\\mathbb\{E\}\[\\bm\{S\}^\{m\}\]=\\alpha\_\{m\}\(d,k\)\\,\\bm\{I\}\_\{d\},\\qquad\\alpha\_\{m\}\(d,k\)=\\tfrac\{1\}\{d\}\\,\\mathbb\{E\}\[\\tr\(\\bm\{S\}^\{m\}\)\]\.\(24\)

Proof\.For𝑸∈O⁡\(d\)\\bm\{Q\}\\in O\(d\),𝑸​𝑾=d𝑾\\bm\{Q\}\\bm\{W\}\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\\bm\{W\}, hence𝑸​𝑺m​𝑸⊤=d𝑺m\\bm\{Q\}\\bm\{S\}^\{m\}\\bm\{Q\}^\{\\top\}\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\\bm\{S\}^\{m\}and

𝔼⁡\[𝑺m\]=𝑸​𝔼​\[𝑺m\]​𝑸⊤for all​𝑸∈O⁡\(d\)\.\\mathbb\{E\}\[\\bm\{S\}^\{m\}\]=\\bm\{Q\}\\,\\mathbb\{E\}\[\\bm\{S\}^\{m\}\]\\,\\bm\{Q\}^\{\\top\}\\qquad\\text\{for all \}\\bm\{Q\}\\in O\(d\)\.By Schur’s lemma𝔼⁡\[𝑺m\]=αm​𝑰d\\mathbb\{E\}\[\\bm\{S\}^\{m\}\]=\\alpha\_\{m\}\\bm\{I\}\_\{d\}; taking the trace givesd​αm=𝔼⁡\[tr⁡\(𝑺m\)\]d\\alpha\_\{m\}=\\mathbb\{E\}\[\\tr\(\\bm\{S\}^\{m\}\)\]\.

###### Lemma 3\(Wick/Isserlis\)\.

LetX1,…,X2​nX\_\{1\},\\dots,X\_\{2n\}be jointly centered Gaussian\. Then

𝔼\[X1⋯X2​n\]=∑P∏\{i,j\}∈P𝔼\[XiXj\],\\mathbb\{E\}\[X\_\{1\}\\cdots X\_\{2n\}\]=\\sum\_\{P\}\\prod\_\{\\\{i,j\\\}\\in P\}\\mathbb\{E\}\[X\_\{i\}X\_\{j\}\],\(25\)the sum over all pairingsPPof\{1,…,2​n\}\\\{1,\\dots,2n\\\}; odd moments vanish\([Isserlis, 1918](https://arxiv.org/html/2610.28975#bib.bib28);[Wick, 1950](https://arxiv.org/html/2610.28975#bib.bib29)\)\.

Proof\.Differentiate𝔼⁡\[exp⁡\(𝒕⊤​𝑿\)\]=exp⁡\(12​𝒕⊤​𝚺​𝒕\)\\mathbb\{E\}\[\\exp\(\\bm\{t\}^\{\\top\}\\bm\{X\}\)\]=\\exp\(\\tfrac\{1\}\{2\}\\bm\{t\}^\{\\top\}\\bm\{\\Sigma\}\\bm\{t\}\)int1,…,t2​nt\_\{1\},\\dots,t\_\{2n\}at𝒕=𝟎\\bm\{t\}=\\bm\{0\}; cf\.[Janson \(1997, §1\.4\)](https://arxiv.org/html/2610.28975#bib.bib30)\.

### B\.5Proofs

###### Lemma 4\.

With𝐒:=𝐖​𝐖⊤\\bm\{S\}:=\\bm\{W\}\\bm\{W\}^\{\\top\}and𝐁:=𝐒−𝐌⋆\\bm\{B\}:=\\bm\{S\}\-\\bm\{M\}^\{\\star\},

𝑮​𝑮⊤=𝑩​𝑺​𝑩=𝑺3−𝑺2​𝑴⋆−𝑴⋆​𝑺2\+𝑴⋆​𝑺​𝑴⋆\.\\bm\{G\}\\bm\{G\}^\{\\top\}=\\bm\{B\}\\bm\{S\}\\bm\{B\}=\\bm\{S\}^\{3\}\-\\bm\{S\}^\{2\}\\bm\{M\}^\{\\star\}\-\\bm\{M\}^\{\\star\}\\bm\{S\}^\{2\}\+\\bm\{M\}^\{\\star\}\\bm\{S\}\\bm\{M\}^\{\\star\}\.\(26\)

Proof\.Since𝑩=𝑩⊤\\bm\{B\}=\\bm\{B\}^\{\\top\},

𝑮​𝑮⊤=𝑩​𝑾​\(𝑩​𝑾\)⊤=𝑩​𝑾​𝑾⊤​𝑩⊤=𝑩​𝑺​𝑩,\\bm\{G\}\\bm\{G\}^\{\\top\}=\\bm\{B\}\\bm\{W\}\(\\bm\{B\}\\bm\{W\}\)^\{\\top\}=\\bm\{B\}\\bm\{W\}\\bm\{W\}^\{\\top\}\\bm\{B\}^\{\\top\}=\\bm\{B\}\\bm\{S\}\\bm\{B\},and substituting𝑩=𝑺−𝑴⋆\\bm\{B\}=\\bm\{S\}\-\\bm\{M\}^\{\\star\}and expanding gives \([26](https://arxiv.org/html/2610.28975#A2.E26)\)\.

###### Lemma 5\.

Under[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5),𝔼⁡\[𝐒\]=k​𝐈d\\mathbb\{E\}\[\\bm\{S\}\]=k\\,\\bm\{I\}\_\{d\}, henceα1​\(d,k\)=k\\alpha\_\{1\}\(d,k\)=k\.

Proof\.Let𝒘a\\bm\{w\}\_\{a\}denote theaa\-th column of𝑾\\bm\{W\}\. Then

𝔼⁡\[𝑺\]=𝔼⁡\[∑a=1k𝒘a​𝒘a⊤\]=∑a=1k𝔼⁡\[𝒘a​𝒘a⊤\]=k​𝑰d\.\\mathbb\{E\}\[\\bm\{S\}\]=\\mathbb\{E\}\\\!\\Bigl\[\\sum\_\{a=1\}^\{k\}\\bm\{w\}\_\{a\}\\bm\{w\}\_\{a\}^\{\\top\}\\Bigr\]=\\sum\_\{a=1\}^\{k\}\\mathbb\{E\}\[\\bm\{w\}\_\{a\}\\bm\{w\}\_\{a\}^\{\\top\}\]=k\\,\\bm\{I\}\_\{d\}\.
###### Lemma 6\.

Under[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5),𝔼⁡\[𝐒2\]=k⁡\(k\+d\+1\)​𝐈d\\mathbb\{E\}\[\\bm\{S\}^\{2\}\]=k\(k\+d\+1\)\\,\\bm\{I\}\_\{d\}, henceα2​\(d,k\)=k⁡\(k\+d\+1\)\\alpha\_\{2\}\(d,k\)=k\(k\+d\+1\)\.

Proof\.By[Lemma2](https://arxiv.org/html/2610.28975#Thmlemma2),𝔼⁡\[𝑺2\]=α2​𝑰d\\mathbb\{E\}\[\\bm\{S\}^\{2\}\]=\\alpha\_\{2\}\\bm\{I\}\_\{d\}\. With columns𝒘a∼iid𝒩⁡\(𝟎,𝑰d\)\\bm\{w\}\_\{a\}\\stackrel\{\{\\scriptstyle\\text\{iid\}\}\}\{\{\\sim\}\}\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\}\),

𝑺2=∑a,b=1k⟨𝒘a,𝒘b⟩​𝒘a​𝒘b⊤\.\\bm\{S\}^\{2\}=\\sum\_\{a,b=1\}^\{k\}\\langle\\bm\{w\}\_\{a\},\\bm\{w\}\_\{b\}\\rangle\\,\\bm\{w\}\_\{a\}\\bm\{w\}\_\{b\}^\{\\top\}\.Split into thekkterms witha=ba=band thek⁡\(k−1\)k\(k\-1\)witha≠ba\\neq b\. By[Lemma3](https://arxiv.org/html/2610.28975#Thmlemma3)and independence,

𝔼⁡\[‖𝒘a‖2​𝒘a​𝒘a⊤\]\\displaystyle\\mathbb\{E\}\[\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}\\bm\{w\}\_\{a\}\\bm\{w\}\_\{a\}^\{\\top\}\]=\(d\+2\)​𝑰d,\\displaystyle=\(d\+2\)\\,\\bm\{I\}\_\{d\},𝔼⁡\[⟨𝒘a,𝒘b⟩​𝒘a​𝒘b⊤\]\\displaystyle\\mathbb\{E\}\[\\langle\\bm\{w\}\_\{a\},\\bm\{w\}\_\{b\}\\rangle\\,\\bm\{w\}\_\{a\}\\bm\{w\}\_\{b\}^\{\\top\}\]=∑p𝔼⁡\[wa,p​𝒘a\]​𝔼​\[wb,p​𝒘b⊤\]=∑p𝒆p​𝒆p⊤=𝑰d\.\\displaystyle=\\sum\_\{p\}\\mathbb\{E\}\[w\_\{a,p\}\\bm\{w\}\_\{a\}\]\\,\\mathbb\{E\}\[w\_\{b,p\}\\bm\{w\}\_\{b\}^\{\\top\}\]=\\sum\_\{p\}\\bm\{e\}\_\{p\}\\bm\{e\}\_\{p\}^\{\\top\}=\\bm\{I\}\_\{d\}\.Summing,𝔼⁡\[𝑺2\]=k⁡\(d\+2\)​𝑰d\+k⁡\(k−1\)​𝑰d=k⁡\(k\+d\+1\)​𝑰d\\mathbb\{E\}\[\\bm\{S\}^\{2\}\]=k\(d\+2\)\\bm\{I\}\_\{d\}\+k\(k\-1\)\\bm\{I\}\_\{d\}=k\(k\+d\+1\)\\bm\{I\}\_\{d\}\.

###### Lemma 7\.

Under[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5),𝔼⁡\[𝐒3\]=\[k3\+3​\(d\+1\)​k2\+\(d2\+3​d\+4\)​k\]​𝐈d\\mathbb\{E\}\[\\bm\{S\}^\{3\}\]=\\bigl\[k^\{3\}\+3\(d\+1\)k^\{2\}\+\(d^\{2\}\+3d\+4\)k\\bigr\]\\bm\{I\}\_\{d\}, henceα3​\(d,k\)\\alpha\_\{3\}\(d,k\)as in \([20](https://arxiv.org/html/2610.28975#A2.E20)\)\.

Proof\.By[Lemma2](https://arxiv.org/html/2610.28975#Thmlemma2),α3=1d​𝔼​\[tr⁡\(𝑺3\)\]\\alpha\_\{3\}=\\tfrac\{1\}\{d\}\\mathbb\{E\}\[\\tr\(\\bm\{S\}^\{3\}\)\], and with𝑺=∑a𝒘a​𝒘a⊤\\bm\{S\}=\\sum\_\{a\}\\bm\{w\}\_\{a\}\\bm\{w\}\_\{a\}^\{\\top\}and cyclicity of the trace,

tr⁡\(𝑺3\)=∑a,b,c⟨𝒘a,𝒘b⟩​⟨𝒘b,𝒘c⟩​⟨𝒘c,𝒘a⟩\.\\tr\(\\bm\{S\}^\{3\}\)=\\sum\_\{a,b,c\}\\langle\\bm\{w\}\_\{a\},\\bm\{w\}\_\{b\}\\rangle\\,\\langle\\bm\{w\}\_\{b\},\\bm\{w\}\_\{c\}\\rangle\\,\\langle\\bm\{w\}\_\{c\},\\bm\{w\}\_\{a\}\\rangle\.Partition\(a,b,c\)∈\{1,…,k\}3\(a,b,c\)\\in\\\{1,\\dots,k\\\}^\{3\}by multiplicity pattern:

\#\{a=b=c\}=k:\\displaystyle\\\#\\\{a=b=c\\\}=k:\\qquad𝔼⁡\[‖𝒘a‖6\]=d⁡\(d\+2\)​\(d\+4\),\\displaystyle\\mathbb\{E\}\[\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{6\}\]=d\(d\+2\)\(d\+4\),\#\{\|\{a,b,c\}\|=2\}=3k\(k−1\):\\displaystyle\\\#\\\{\|\\\{a,b,c\\\}\|=2\\\}=3k\(k\-1\):\\qquad𝔼⁡\[‖𝒘a‖2​⟨𝒘a,𝒘c⟩2\]=𝔼⁡\[‖𝒘a‖4\]=d⁡\(d\+2\),\\displaystyle\\mathbb\{E\}\[\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}\\langle\\bm\{w\}\_\{a\},\\bm\{w\}\_\{c\}\\rangle^\{2\}\]=\\mathbb\{E\}\[\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{4\}\]=d\(d\+2\),\#⁡\{a,b,c​distinct\}=k⁡\(k−1\)​\(k−2\):\\displaystyle\\\#\\\{a,b,c\\text\{ distinct\}\\\}=k\(k\-1\)\(k\-2\):\\qquad∑p,q,rδp​r​δp​q​δq​r=d,\\displaystyle\\textstyle\\sum\_\{p,q,r\}\\delta\_\{pr\}\\delta\_\{pq\}\\delta\_\{qr\}=d,where for the middle pattern, e\.g\.a=b≠ca=b\\neq cgives‖𝒘a‖2​⟨𝒘a,𝒘c⟩2\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}\\langle\\bm\{w\}\_\{a\},\\bm\{w\}\_\{c\}\\rangle^\{2\}and, conditioning on𝒘a\\bm\{w\}\_\{a\},⟨𝒘a,𝒘c⟩\|𝒘a∼𝒩⁡\(0,‖𝒘a‖2\)\\langle\\bm\{w\}\_\{a\},\\bm\{w\}\_\{c\}\\rangle\\mid\\bm\{w\}\_\{a\}\\sim\\mathcal\{N\}\(0,\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}\); the other two cases are symmetric\. Hence

d​α3\\displaystyle d\\,\\alpha\_\{3\}=k​d​\(d\+2\)​\(d\+4\)\+3​k​\(k−1\)​d​\(d\+2\)\+k⁡\(k−1\)​\(k−2\)​d,\\displaystyle=k\\,d\(d\+2\)\(d\+4\)\+3k\(k\-1\)\\,d\(d\+2\)\+k\(k\-1\)\(k\-2\)\\,d,α3\\displaystyle\\alpha\_\{3\}=k⁡\(d\+2\)​\(d\+4\)\+3​k​\(k−1\)​\(d\+2\)\+k⁡\(k−1\)​\(k−2\)=k3\+3​\(d\+1\)​k2\+\(d2\+3​d\+4\)​k\.\\displaystyle=k\(d\+2\)\(d\+4\)\+3k\(k\-1\)\(d\+2\)\+k\(k\-1\)\(k\-2\)=k^\{3\}\+3\(d\+1\)k^\{2\}\+\(d^\{2\}\+3d\+4\)k\.
Proof of[Theorem1](https://arxiv.org/html/2610.28975#Thmtheorem1)\.Take expectations in \([26](https://arxiv.org/html/2610.28975#A2.E26)\)\. Since𝑴⋆\\bm\{M\}^\{\\star\}is deterministic,[Lemma2](https://arxiv.org/html/2610.28975#Thmlemma2)gives

𝔼⁡\[𝑺3\]\\displaystyle\\mathbb\{E\}\[\\bm\{S\}^\{3\}\]=α3​𝑰d,\\displaystyle=\\alpha\_\{3\}\\bm\{I\}\_\{d\},𝔼⁡\[𝑺2​𝑴⋆\]=𝔼⁡\[𝑴⋆​𝑺2\]\\displaystyle\\mathbb\{E\}\[\\bm\{S\}^\{2\}\\bm\{M\}^\{\\star\}\]=\\mathbb\{E\}\[\\bm\{M\}^\{\\star\}\\bm\{S\}^\{2\}\]=α2​𝑴⋆,\\displaystyle=\\alpha\_\{2\}\\bm\{M\}^\{\\star\},𝔼⁡\[𝑴⋆​𝑺​𝑴⋆\]=𝑴⋆​𝔼​\[𝑺\]​𝑴⋆\\displaystyle\\mathbb\{E\}\[\\bm\{M\}^\{\\star\}\\bm\{S\}\\bm\{M\}^\{\\star\}\]=\\bm\{M\}^\{\\star\}\\,\\mathbb\{E\}\[\\bm\{S\}\]\\,\\bm\{M\}^\{\\star\}=k​\(𝑴⋆\)2,\\displaystyle=k\\,\(\\bm\{M\}^\{\\star\}\)^\{2\},so summing gives \([17](https://arxiv.org/html/2610.28975#A2.E17)\); the values ofα1,α2,α3\\alpha\_\{1\},\\alpha\_\{2\},\\alpha\_\{3\}are[Lemmas5](https://arxiv.org/html/2610.28975#Thmlemma5),[6](https://arxiv.org/html/2610.28975#Thmlemma6)and[7](https://arxiv.org/html/2610.28975#Thmlemma7)\.[Corollary1](https://arxiv.org/html/2610.28975#Thmcorollary1)follows from the entries of \([17](https://arxiv.org/html/2610.28975#A2.E17)\): withd,𝑴⋆d,\\bm\{M\}^\{\\star\}fixed,α3=Θ⁡\(k3\)\\alpha\_\{3\}=\\Theta\(k^\{3\}\),α2=Θ⁡\(k2\)\\alpha\_\{2\}=\\Theta\(k^\{2\}\),k​\(𝑴⋆\)∙2=Θ⁡\(k\)k\(\\bm\{M\}^\{\\star\}\)^\{2\}\_\{\\bullet\}=\\Theta\(k\)\.

## Appendix CDeep Linear Networks

### C\.1Setup

LetL≥1L\\geq 1andd0,…,dLd\_\{0\},\\dots,d\_\{L\}be positive integers\. The depth\-LLlinear network

f\(𝒙\)=𝑾L𝑾L−1⋯𝑾1𝒙,𝑾i∈ℝdi×di−1,f\(\\bm\{x\}\)=\\bm\{W\}\_\{L\}\\bm\{W\}\_\{L\-1\}\\cdots\\bm\{W\}\_\{1\}\\bm\{x\},\\qquad\\bm\{W\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{i\-1\}\},\(27\)learns a deterministic linear target𝚽∈ℝdL×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{L\}\\times d\_\{0\}\}under the population squared loss

ℒ\(𝑾1,…,𝑾L\):=12𝔼𝒙‖𝑾L⋯𝑾1𝒙−𝚽𝒙‖2\.\\mathcal\{L\}\(\\bm\{W\}\_\{1\},\\dots,\\bm\{W\}\_\{L\}\)\\;:=\\;\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\\left\\\|\\bm\{W\}\_\{L\}\\cdots\\bm\{W\}\_\{1\}\\bm\{x\}\-\\bm\{\\Phi\}\\bm\{x\}\\right\\\|^\{2\}\.\(28\)Define the layer\-iiforward, backward and residual factors

𝑩i:=𝑾i−1⋯𝑾1∈ℝdi−1×d0,𝑨i:=𝑾L⋯𝑾i\+1∈ℝdL×di,𝑹:=𝑾L⋯𝑾1−𝚽,\\bm\{B\}\_\{i\}:=\\bm\{W\}\_\{i\-1\}\\cdots\\bm\{W\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{i\-1\}\\times d\_\{0\}\},\\qquad\\bm\{A\}\_\{i\}:=\\bm\{W\}\_\{L\}\\cdots\\bm\{W\}\_\{i\+1\}\\in\\mathbb\{R\}^\{d\_\{L\}\\times d\_\{i\}\},\\qquad\\bm\{R\}:=\\bm\{W\}\_\{L\}\\cdots\\bm\{W\}\_\{1\}\-\\bm\{\\Phi\},\(29\)with𝑩1:=𝑰d0\\bm\{B\}\_\{1\}:=\\bm\{I\}\_\{d\_\{0\}\}and𝑨L:=𝑰dL\\bm\{A\}\_\{L\}:=\\bm\{I\}\_\{d\_\{L\}\}\. The layer\-iigradient is

𝑮i:=∂ℒ∂𝑾i=𝑨i⊤​𝑹​𝑩i⊤∈ℝdi×di−1\.\\bm\{G\}\_\{i\}\\;:=\\;\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\bm\{W\}\_\{i\}\}\\;=\\;\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{R\}\\,\\bm\{B\}\_\{i\}^\{\\top\}\\;\\in\\;\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{i\-1\}\}\.\(30\)The closed form below uses two pairs of dimension\-only scalar sequences\. For the backward factors\{𝑩i\}\\\{\\bm\{B\}\_\{i\}\\\},

r1\\displaystyle r\_\{1\}=1,\\displaystyle=1,v1\\displaystyle v\_\{1\}=d02,\\displaystyle=d\_\{0\}^\{2\},\(31\)ri\\displaystyle r\_\{i\}=vi−1\+\(di−1\+1\)​di−2​ri−1,\\displaystyle=v\_\{i\-1\}\+\(d\_\{i\-1\}\+1\)d\_\{i\-2\}r\_\{i\-1\},vi\\displaystyle v\_\{i\}=di−12vi−1\+2di−1di−2ri−1\(i≥2\);\\displaystyle=d\_\{i\-1\}^\{2\}v\_\{i\-1\}\+2d\_\{i\-1\}d\_\{i\-2\}r\_\{i\-1\}\\quad\(i\\geq 2\);\(32\)for the forward factors\{𝑨i\}\\\{\\bm\{A\}\_\{i\}\\\},

sL\\displaystyle s\_\{L\}=1,\\displaystyle=1,uL\\displaystyle u\_\{L\}=dL2,\\displaystyle=d\_\{L\}^\{2\},\(33\)si−1\\displaystyle s\_\{i\-1\}=ui\+\(di−1\+1\)​di​si,\\displaystyle=u\_\{i\}\+\(d\_\{i\-1\}\+1\)d\_\{i\}s\_\{i\},ui−1\\displaystyle u\_\{i\-1\}=di−12ui\+2di−1disi\(i≤L\);\\displaystyle=d\_\{i\-1\}^\{2\}u\_\{i\}\+2d\_\{i\-1\}d\_\{i\}s\_\{i\}\\quad\(i\\leq L\);\(34\)and the composite Frobenius coefficient is

Vi:=di−1​ri=𝔼⁡\[‖𝑩i​𝑩i⊤‖F2\]\.V\_\{i\}\\;:=\\;d\_\{i\-1\}r\_\{i\}\\;=\\;\\mathbb\{E\}\\bigl\[\\left\\\|\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bigr\]\.\(35\)

### C\.2Standing assumption

###### Assumption 6\.

The entries\(𝐖i\)a​b\(\\bm\{W\}\_\{i\}\)\_\{ab\}are i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)across alli,a,bi,a,b;𝐱∼𝒩⁡\(𝟎,𝐈d0\)\\bm\{x\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\); and𝚽\\bm\{\\Phi\}is deterministic and independent of𝐖1,…,𝐖L\\bm\{W\}\_\{1\},\\dots,\\bm\{W\}\_\{L\}\. Expectations𝔼⁡\[⋅\]\\mathbb\{E\}\[\\cdot\]are over𝐖1,…,𝐖L\\bm\{W\}\_\{1\},\\dots,\\bm\{W\}\_\{L\}\(𝐱\\bm\{x\}has been integrated out inℒ\\mathcal\{L\}\)\.

### C\.3Main results restated

Theorem \([Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2), restated\)\.Under[Assumption6](https://arxiv.org/html/2610.28975#Thmassumption6), for every layeri∈\{1,…,L\}i\\in\\\{1,\\dots,L\\\},

𝔼⁡\[𝑮i​𝑮i⊤\]=Vi​si​𝑰di\+𝑻i,𝑻i=\{\(∏j=1L−1dj\)​𝚽​𝚽⊤,i=L,\(∏j=1,j≠iL−1dj\)​‖𝚽‖F2​𝑰di,1≤i<L,\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]=V\_\{i\}\\,s\_\{i\}\\,\\bm\{I\}\_\{d\_\{i\}\}\+\\bm\{T\}\_\{i\},\\qquad\\bm\{T\}\_\{i\}=\\begin\{cases\}\\Bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\Bigr\)\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},&i=L,\\\\\[6\.0pt\] \\Bigl\(\\prod\_\{j=1,\\,j\\neq i\}^\{L\-1\}d\_\{j\}\\Bigr\)\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\\bm\{I\}\_\{d\_\{i\}\},&1\\leq i<L,\\end\{cases\}\(36\)withVi,siV\_\{i\},s\_\{i\}from \([31](https://arxiv.org/html/2610.28975#A3.E31)\)–\([35](https://arxiv.org/html/2610.28975#A3.E35)\)\.

###### Corollary 2\.

Under[Assumption6](https://arxiv.org/html/2610.28975#Thmassumption6), the diagonal entries of𝔼⁡\[𝐆i​𝐆i⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]are

𝔼⁡\[\(𝑮i​𝑮i⊤\)a​a\]=\{Vi​si\+\(∏j≠i,1≤j≤L−1dj\)​‖𝚽‖F2,1≤i<L,VL​sL\+\(∏j=1L−1dj\)​\(𝚽​𝚽⊤\)a​a,i=L,\\mathbb\{E\}\\bigl\[\(\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\)\_\{aa\}\\bigr\]=\\begin\{cases\}V\_\{i\}s\_\{i\}\+\\bigl\(\\prod\_\{j\\neq i,\\,1\\leq j\\leq L\-1\}d\_\{j\}\\bigr\)\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},&1\\leq i<L,\\\\\[4\.0pt\] V\_\{L\}s\_\{L\}\+\\bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\bigr\)\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{aa\},&i=L,\\end\{cases\}\(37\)and, fora≠ba\\neq b, the off\-diagonal entries are𝔼⁡\[\(𝐆i​𝐆i⊤\)a​b\]=0\\mathbb\{E\}\[\(\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\)\_\{ab\}\]=0if1≤i<L1\\leq i<Land\(∏j=1L−1dj\)​\(𝚽​𝚽⊤\)a​b\\bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\bigr\)\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}ifi=Li=L\. In particular𝔼⁡\[𝐆i​𝐆i⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]is a scalar multiple of𝐈di\\bm\{I\}\_\{d\_\{i\}\}for everyi<Li<L\. Moreover,

Vi​si=Θ⁡\(di​∏j=1,j≠iL−1dj2\)\(1≤i<L\),VL​sL=Θ⁡\(∏j=1L−1dj2\)\.V\_\{i\}s\_\{i\}=\\Theta\\Bigl\(d\_\{i\}\\prod\_\{j=1,\\,j\\neq i\}^\{L\-1\}d\_\{j\}^\{2\}\\Bigr\)\\quad\(1\\leq i<L\),\\qquad V\_\{L\}s\_\{L\}=\\Theta\\Bigl\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}^\{2\}\\Bigr\)\.\(38\)

### C\.4Preliminary lemmas

###### Lemma 8\.

Same as[Lemma1](https://arxiv.org/html/2610.28975#Thmlemma1)\.

###### Lemma 9\(Wick/Isserlis\)\.

Same as[Lemma3](https://arxiv.org/html/2610.28975#Thmlemma3)\.

###### Lemma 10\.

Let𝐗∈ℝm×n\\bm\{X\}\\in\\mathbb\{R\}^\{m\\times n\}have i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries and let𝐌∈ℝn×n\\bm\{M\}\\in\\mathbb\{R\}^\{n\\times n\},𝐍∈ℝm×m\\bm\{N\}\\in\\mathbb\{R\}^\{m\\times m\}be deterministic symmetric\. Then

𝔼⁡\[𝑿​𝑴​𝑿⊤​𝑿​𝑴​𝑿⊤\]\\displaystyle\\mathbb\{E\}\\bigl\[\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\\bigr\]=\(tr⁡\(𝑴\)2\+\(m\+1\)​‖𝑴‖F2\)​𝑰m,\\displaystyle=\\bigl\(\\tr\(\\bm\{M\}\)^\{2\}\+\(m\+1\)\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bigr\)\\bm\{I\}\_\{m\},\(39\)𝔼⁡\[𝑿⊤​𝑵​𝑿​𝑿⊤​𝑵​𝑿\]\\displaystyle\\mathbb\{E\}\\bigl\[\\bm\{X\}^\{\\top\}\\bm\{N\}\\bm\{X\}\\bm\{X\}^\{\\top\}\\bm\{N\}\\bm\{X\}\\bigr\]=\(tr⁡\(𝑵\)2\+\(n\+1\)​‖𝑵‖F2\)​𝑰n\.\\displaystyle=\\bigl\(\\tr\(\\bm\{N\}\)^\{2\}\+\(n\+1\)\\left\\\|\\bm\{N\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bigr\)\\bm\{I\}\_\{n\}\.\(40\)

Proof\.We prove \([39](https://arxiv.org/html/2610.28975#A3.E39)\); \([40](https://arxiv.org/html/2610.28975#A3.E40)\) follows by applying it to𝑿⊤\\bm\{X\}^\{\\top\}\. Fora,b,c,d∈\{1,…,m\}a,b,c,d\\in\\\{1,\\dots,m\\\},[Lemma9](https://arxiv.org/html/2610.28975#Thmlemma9)gives

𝔼⁡\[𝑿a​p​𝑿b​q​𝑿c​r​𝑿d​s\]=δa​b​δp​q​δc​d​δr​s\+δa​c​δp​r​δb​d​δq​s\+δa​d​δp​s​δb​c​δq​r,\\mathbb\{E\}\[\\bm\{X\}\_\{ap\}\\bm\{X\}\_\{bq\}\\bm\{X\}\_\{cr\}\\bm\{X\}\_\{ds\}\]=\\delta\_\{ab\}\\delta\_\{pq\}\\delta\_\{cd\}\\delta\_\{rs\}\+\\delta\_\{ac\}\\delta\_\{pr\}\\delta\_\{bd\}\\delta\_\{qs\}\+\\delta\_\{ad\}\\delta\_\{ps\}\\delta\_\{bc\}\\delta\_\{qr\},and contracting against𝑴p​q​𝑴r​s\\bm\{M\}\_\{pq\}\\bm\{M\}\_\{rs\}with𝑴=𝑴⊤\\bm\{M\}=\\bm\{M\}^\{\\top\},

𝔼⁡\[\(𝑿​𝑴​𝑿⊤\)a​b​\(𝑿​𝑴​𝑿⊤\)c​d\]=δa​b​δc​d​tr⁡\(𝑴\)2\+\(δa​c​δb​d\+δa​d​δb​c\)​‖𝑴‖F2\.\\mathbb\{E\}\[\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)\_\{ab\}\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)\_\{cd\}\]=\\delta\_\{ab\}\\delta\_\{cd\}\\tr\(\\bm\{M\}\)^\{2\}\+\(\\delta\_\{ac\}\\delta\_\{bd\}\+\\delta\_\{ad\}\\delta\_\{bc\}\)\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\.Summing overb=cb=c,

𝔼⁡\[\(𝑿​𝑴​𝑿⊤​𝑿​𝑴​𝑿⊤\)a​c\]=∑b𝔼⁡\[\(𝑿​𝑴​𝑿⊤\)a​b​\(𝑿​𝑴​𝑿⊤\)b​c\]=δa​c​\(tr⁡\(𝑴\)2\+\(m\+1\)​‖𝑴‖F2\)\.\\mathbb\{E\}\[\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)\_\{ac\}\]=\\sum\_\{b\}\\mathbb\{E\}\[\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)\_\{ab\}\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)\_\{bc\}\]=\\delta\_\{ac\}\\bigl\(\\tr\(\\bm\{M\}\)^\{2\}\+\(m\+1\)\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bigr\)\.
###### Lemma 11\.

Let𝐗∈ℝm×n\\bm\{X\}\\in\\mathbb\{R\}^\{m\\times n\}have i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries and𝐌∈ℝn×n\\bm\{M\}\\in\\mathbb\{R\}^\{n\\times n\}be deterministic symmetric\. Then

𝔼⁡\[tr⁡\(𝑿​𝑴​𝑿⊤\)2\]=m2​tr⁡\(𝑴\)2\+2​m​‖𝑴‖F2\.\\mathbb\{E\}\\bigl\[\\tr\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)^\{2\}\\bigr\]=m^\{2\}\\tr\(\\bm\{M\}\)^\{2\}\+2m\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(41\)

Proof\.Expanding,

tr⁡\(𝑿​𝑴​𝑿⊤\)2=∑a,a′,p,q,p′,q′𝑿a​p​𝑴p​q​𝑿a​q​𝑿a′​p′​𝑴p′​q′​𝑿a′​q′;\\tr\(\\bm\{X\}\\bm\{M\}\\bm\{X\}^\{\\top\}\)^\{2\}=\\sum\_\{a,a^\{\\prime\},p,q,p^\{\\prime\},q^\{\\prime\}\}\\bm\{X\}\_\{ap\}\\bm\{M\}\_\{pq\}\\bm\{X\}\_\{aq\}\\,\\bm\{X\}\_\{a^\{\\prime\}p^\{\\prime\}\}\\bm\{M\}\_\{p^\{\\prime\}q^\{\\prime\}\}\\bm\{X\}\_\{a^\{\\prime\}q^\{\\prime\}\};by[Lemma9](https://arxiv.org/html/2610.28975#Thmlemma9)the three pair contractions of𝔼⁡\[𝑿a​p​𝑿a​q​𝑿a′​p′​𝑿a′​q′\]\\mathbb\{E\}\[\\bm\{X\}\_\{ap\}\\bm\{X\}\_\{aq\}\\bm\{X\}\_\{a^\{\\prime\}p^\{\\prime\}\}\\bm\{X\}\_\{a^\{\\prime\}q^\{\\prime\}\}\]contribute \(after summing over𝑴​𝑴\\bm\{M\}\\bm\{M\}\)m2​tr⁡\(𝑴\)2m^\{2\}\\tr\(\\bm\{M\}\)^\{2\},m​‖𝑴‖F2m\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}andm​tr⁡\(𝑴2\)=m​‖𝑴‖F2m\\tr\(\\bm\{M\}^\{2\}\)=m\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}, which sum to \([41](https://arxiv.org/html/2610.28975#A3.E41)\)\.

### C\.5Proofs

The proof of[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)has three steps: a𝑾i\\bm\{W\}\_\{i\}\-expansion; recursive moments of𝑩i​𝑩i⊤\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}and𝑨i⊤​𝑨i\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}; and the target\-dependent term\.

###### Lemma 12\.

Under[Assumption6](https://arxiv.org/html/2610.28975#Thmassumption6), for everyii,

𝔼\[𝑮i𝑮i⊤\|𝑨i,𝑩i\]=tr\(\(𝑩i𝑩i⊤\)2\)\(𝑨i⊤𝑨i\)2\+𝑨i⊤𝚽𝑩i⊤𝑩i𝚽⊤𝑨i\.\\mathbb\{E\}\\bigl\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\\bigm\|\\bm\{A\}\_\{i\},\\bm\{B\}\_\{i\}\\bigr\]=\\tr\\bigl\(\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\\bigr\)\\,\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\+\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{\\Phi\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\}\.\(42\)

Proof\.Substituting𝑹=𝑨i​𝑾i​𝑩i−𝚽\\bm\{R\}=\\bm\{A\}\_\{i\}\\bm\{W\}\_\{i\}\\bm\{B\}\_\{i\}\-\\bm\{\\Phi\}into \([30](https://arxiv.org/html/2610.28975#A3.E30)\),

𝑮i​𝑮i⊤=\(𝑨i⊤​𝑨i\)​𝑾i​\(𝑩i​𝑩i⊤​𝑩i​𝑩i⊤\)​𝑾i⊤​\(𝑨i⊤​𝑨i\)\+\(cross\)\+𝑨i⊤​𝚽​𝑩i⊤​𝑩i​𝚽⊤​𝑨i,\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}=\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)\\bm\{W\}\_\{i\}\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)\\bm\{W\}\_\{i\}^\{\\top\}\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)\+\(\\text\{cross\}\)\+\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{\\Phi\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\},where\(cross\)\(\\text\{cross\}\)collects the two terms linear in𝑾i\\bm\{W\}\_\{i\}; since𝑾i=d−𝑾i\\bm\{W\}\_\{i\}\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\-\\bm\{W\}\_\{i\}is independent of\(𝑨i,𝑩i\)\(\\bm\{A\}\_\{i\},\\bm\{B\}\_\{i\}\),𝔼\[\(cross\)∣𝑨i,𝑩i\]=𝟎\\mathbb\{E\}\[\(\\text\{cross\}\)\\mid\\bm\{A\}\_\{i\},\\bm\{B\}\_\{i\}\]=\\bm\{0\}\. By[Lemma8](https://arxiv.org/html/2610.28975#Thmlemma8)with𝑲=𝑩i​𝑩i⊤​𝑩i​𝑩i⊤\\bm\{K\}=\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\},

𝔼𝑾i\[𝑾i𝑲𝑾i⊤\|𝑨i,𝑩i\]=tr\(𝑲\)𝑰di=tr\(\(𝑩i𝑩i⊤\)2\)𝑰di,\\mathbb\{E\}\_\{\\bm\{W\}\_\{i\}\}\\bigl\[\\bm\{W\}\_\{i\}\\bm\{K\}\\bm\{W\}\_\{i\}^\{\\top\}\\bigm\|\\bm\{A\}\_\{i\},\\bm\{B\}\_\{i\}\\bigr\]=\\tr\(\\bm\{K\}\)\\,\\bm\{I\}\_\{d\_\{i\}\}=\\tr\\bigl\(\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\\bigr\)\\bm\{I\}\_\{d\_\{i\}\},which yields \([42](https://arxiv.org/html/2610.28975#A3.E42)\)\.

###### Lemma 13\.

Under[Assumption6](https://arxiv.org/html/2610.28975#Thmassumption6), for everyi∈\{1,…,L\}i\\in\\\{1,\\dots,L\\\},

𝔼⁡\[\(𝑩i​𝑩i⊤\)2\]=ri​𝑰di−1,𝔼⁡\[tr⁡\(𝑩i​𝑩i⊤\)2\]=vi,\\mathbb\{E\}\\bigl\[\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\\bigr\]=r\_\{i\}\\,\\bm\{I\}\_\{d\_\{i\-1\}\},\\qquad\\mathbb\{E\}\\bigl\[\\tr\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\\bigr\]=v\_\{i\},\(43\)with\(ri,vi\)\(r\_\{i\},v\_\{i\}\)from \([31](https://arxiv.org/html/2610.28975#A3.E31)\)–\([32](https://arxiv.org/html/2610.28975#A3.E32)\)\.

Proof\.Ati=1i=1,𝑩1=𝑰d0\\bm\{B\}\_\{1\}=\\bm\{I\}\_\{d\_\{0\}\}, so\(𝑩1​𝑩1⊤\)2=𝑰d0\(\\bm\{B\}\_\{1\}\\bm\{B\}\_\{1\}^\{\\top\}\)^\{2\}=\\bm\{I\}\_\{d\_\{0\}\}andtr⁡\(𝑩1​𝑩1⊤\)2=d02\\tr\(\\bm\{B\}\_\{1\}\\bm\{B\}\_\{1\}^\{\\top\}\)^\{2\}=d\_\{0\}^\{2\}, matchingr1=1r\_\{1\}=1,v1=d02v\_\{1\}=d\_\{0\}^\{2\}\. Fori≥2i\\geq 2,𝑩i=𝑾i−1​𝑩i−1\\bm\{B\}\_\{i\}=\\bm\{W\}\_\{i\-1\}\\bm\{B\}\_\{i\-1\}with𝑾i−1∈ℝdi−1×di−2\\bm\{W\}\_\{i\-1\}\\in\\mathbb\{R\}^\{d\_\{i\-1\}\\times d\_\{i\-2\}\}independent of𝑩i−1\\bm\{B\}\_\{i\-1\}, so𝑩i​𝑩i⊤=𝑾i−1​𝑴​𝑾i−1⊤\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}=\\bm\{W\}\_\{i\-1\}\\bm\{M\}\\bm\{W\}\_\{i\-1\}^\{\\top\}with𝑴:=𝑩i−1​𝑩i−1⊤\\bm\{M\}:=\\bm\{B\}\_\{i\-1\}\\bm\{B\}\_\{i\-1\}^\{\\top\}\. By[Lemma10](https://arxiv.org/html/2610.28975#Thmlemma10),

𝔼⁡\[\(𝑩i​𝑩i⊤\)2\|𝑩i−1\]=\(tr⁡\(𝑴\)2\+\(di−1\+1\)​‖𝑴‖F2\)​𝑰di−1,\\mathbb\{E\}\\bigl\[\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\\bigm\|\\bm\{B\}\_\{i\-1\}\\bigr\]=\\bigl\(\\tr\(\\bm\{M\}\)^\{2\}\+\(d\_\{i\-1\}\+1\)\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bigr\)\\bm\{I\}\_\{d\_\{i\-1\}\},and taking𝔼\\mathbb\{E\}over𝑩i−1\\bm\{B\}\_\{i\-1\}with𝔼⁡\[tr⁡\(𝑴2\)\]=di−2​ri−1\\mathbb\{E\}\[\\tr\(\\bm\{M\}^\{2\}\)\]=d\_\{i\-2\}r\_\{i\-1\}givesri=vi−1\+\(di−1\+1\)​di−2​ri−1r\_\{i\}=v\_\{i\-1\}\+\(d\_\{i\-1\}\+1\)d\_\{i\-2\}r\_\{i\-1\}\. Likewise, by[Lemma11](https://arxiv.org/html/2610.28975#Thmlemma11),

𝔼⁡\[tr⁡\(𝑩i​𝑩i⊤\)2\|𝑩i−1\]=di−12​tr⁡\(𝑴\)2\+2​di−1​‖𝑴‖F2,\\mathbb\{E\}\\bigl\[\\tr\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\\bigm\|\\bm\{B\}\_\{i\-1\}\\bigr\]=d\_\{i\-1\}^\{2\}\\tr\(\\bm\{M\}\)^\{2\}\+2d\_\{i\-1\}\\left\\\|\\bm\{M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},sovi=di−12​vi−1\+2​di−1​di−2​ri−1v\_\{i\}=d\_\{i\-1\}^\{2\}v\_\{i\-1\}\+2d\_\{i\-1\}d\_\{i\-2\}r\_\{i\-1\}\.

###### Lemma 14\.

Under[Assumption6](https://arxiv.org/html/2610.28975#Thmassumption6), for everyi∈\{0,…,L\}i\\in\\\{0,\\dots,L\\\},

𝔼⁡\[\(𝑨i⊤​𝑨i\)2\]=si​𝑰di,𝔼⁡\[tr⁡\(𝑨i⊤​𝑨i\)2\]=ui,\\mathbb\{E\}\\bigl\[\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\\bigr\]=s\_\{i\}\\,\\bm\{I\}\_\{d\_\{i\}\},\\qquad\\mathbb\{E\}\\bigl\[\\tr\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\\bigr\]=u\_\{i\},\(44\)with\(si,ui\)\(s\_\{i\},u\_\{i\}\)from \([33](https://arxiv.org/html/2610.28975#A3.E33)\)–\([34](https://arxiv.org/html/2610.28975#A3.E34)\)\.

Proof\.Ati=Li=L,𝑨L=𝑰dL\\bm\{A\}\_\{L\}=\\bm\{I\}\_\{d\_\{L\}\}, matchingsL=1s\_\{L\}=1,uL=dL2u\_\{L\}=d\_\{L\}^\{2\}\. Fori<Li<L,𝑨i=𝑨i\+1​𝑾i\+1\\bm\{A\}\_\{i\}=\\bm\{A\}\_\{i\+1\}\\bm\{W\}\_\{i\+1\}with𝑾i\+1∈ℝdi\+1×di\\bm\{W\}\_\{i\+1\}\\in\\mathbb\{R\}^\{d\_\{i\+1\}\\times d\_\{i\}\}independent of𝑨i\+1\\bm\{A\}\_\{i\+1\}, so𝑨i⊤​𝑨i=𝑾i\+1⊤​𝑵​𝑾i\+1\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}=\\bm\{W\}\_\{i\+1\}^\{\\top\}\\bm\{N\}\\bm\{W\}\_\{i\+1\}with𝑵:=𝑨i\+1⊤​𝑨i\+1\\bm\{N\}:=\\bm\{A\}\_\{i\+1\}^\{\\top\}\\bm\{A\}\_\{i\+1\}\. By[Lemma10](https://arxiv.org/html/2610.28975#Thmlemma10)\(right identity\),

𝔼⁡\[\(𝑨i⊤​𝑨i\)2\|𝑨i\+1\]=\(tr⁡\(𝑵\)2\+\(di\+1\)​‖𝑵‖F2\)​𝑰di,\\mathbb\{E\}\\bigl\[\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\\bigm\|\\bm\{A\}\_\{i\+1\}\\bigr\]=\\bigl\(\\tr\(\\bm\{N\}\)^\{2\}\+\(d\_\{i\}\+1\)\\left\\\|\\bm\{N\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bigr\)\\bm\{I\}\_\{d\_\{i\}\},sosi=ui\+1\+\(di\+1\)​di\+1​si\+1s\_\{i\}=u\_\{i\+1\}\+\(d\_\{i\}\+1\)d\_\{i\+1\}s\_\{i\+1\}; the recursion foruiu\_\{i\}is that ofviv\_\{i\}withdi−1↔did\_\{i\-1\}\\leftrightarrow d\_\{i\}anddi−2↔di\+1d\_\{i\-2\}\\leftrightarrow d\_\{i\+1\}, givingui−1=di−12​ui\+2​di−1​di​siu\_\{i\-1\}=d\_\{i\-1\}^\{2\}u\_\{i\}\+2d\_\{i\-1\}d\_\{i\}s\_\{i\}\.

###### Lemma 15\.

Under[Assumption6](https://arxiv.org/html/2610.28975#Thmassumption6),𝔼⁡\[𝐀i⊤​𝚽​𝐁i⊤​𝐁i​𝚽⊤​𝐀i\]=𝐓i\\mathbb\{E\}\[\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{\\Phi\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\}\]=\\bm\{T\}\_\{i\}, with𝐓i\\bm\{T\}\_\{i\}as in \([36](https://arxiv.org/html/2610.28975#A3.E36)\)\.

Proof\.By independence of𝑨i,𝑩i\\bm\{A\}\_\{i\},\\bm\{B\}\_\{i\}and[Lemma8](https://arxiv.org/html/2610.28975#Thmlemma8)iterated through𝑩i=𝑾i−1⋯𝑾1\\bm\{B\}\_\{i\}=\\bm\{W\}\_\{i\-1\}\\cdots\\bm\{W\}\_\{1\},

𝔼⁡\[𝑩i⊤​𝑩i\]=\(∏j=1i−1dj\)​𝑰d0⟹𝔼⁡\[𝑨i⊤​𝚽​𝑩i⊤​𝑩i​𝚽⊤​𝑨i\]=\(∏j=1i−1dj\)​𝔼​\[𝑨i⊤​𝚽​𝚽⊤​𝑨i\]\\mathbb\{E\}\[\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\]=\\Bigl\(\\prod\_\{j=1\}^\{i\-1\}d\_\{j\}\\Bigr\)\\bm\{I\}\_\{d\_\{0\}\}\\qquad\\Longrightarrow\\qquad\\mathbb\{E\}\[\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{\\Phi\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\}\]=\\Bigl\(\\prod\_\{j=1\}^\{i\-1\}d\_\{j\}\\Bigr\)\\,\\mathbb\{E\}\[\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\}\]\(the empty product is11wheni=1i=1\)\. Lethi​\(𝑴\):=𝔼⁡\[𝑨i⊤​𝑴​𝑨i\]h\_\{i\}\(\\bm\{M\}\):=\\mathbb\{E\}\[\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{M\}\\bm\{A\}\_\{i\}\]for symmetric𝑴∈ℝdL×dL\\bm\{M\}\\in\\mathbb\{R\}^\{d\_\{L\}\\times d\_\{L\}\}\. ThenhL​\(𝑴\)=𝑴h\_\{L\}\(\\bm\{M\}\)=\\bm\{M\}, and fori<Li<L, by[Lemma8](https://arxiv.org/html/2610.28975#Thmlemma8),

hi​\(𝑴\)=𝔼⁡\[tr⁡\(𝑨i\+1⊤​𝑴​𝑨i\+1\)\]​𝑰di=tr⁡\(hi\+1​\(𝑴\)\)​𝑰di\.h\_\{i\}\(\\bm\{M\}\)=\\mathbb\{E\}\\bigl\[\\tr\(\\bm\{A\}\_\{i\+1\}^\{\\top\}\\bm\{M\}\\bm\{A\}\_\{i\+1\}\)\\bigr\]\\bm\{I\}\_\{d\_\{i\}\}=\\tr\\bigl\(h\_\{i\+1\}\(\\bm\{M\}\)\\bigr\)\\bm\{I\}\_\{d\_\{i\}\}\.Iterating,

hi​\(𝑴\)=\{𝑴,i=L,\(∏j=i\+1L−1dj\)​tr⁡\(𝑴\)​𝑰di,i<L,h\_\{i\}\(\\bm\{M\}\)=\\begin\{cases\}\\bm\{M\},&i=L,\\\\\[2\.0pt\] \\bigl\(\\prod\_\{j=i\+1\}^\{L\-1\}d\_\{j\}\\bigr\)\\tr\(\\bm\{M\}\)\\,\\bm\{I\}\_\{d\_\{i\}\},&i<L,\\end\{cases\}with∏j=LL−1dj=1\\prod\_\{j=L\}^\{L\-1\}d\_\{j\}=1\. Taking𝑴=𝚽​𝚽⊤\\bm\{M\}=\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\(sotr⁡\(𝑴\)=‖𝚽‖F2\\tr\(\\bm\{M\}\)=\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\) and multiplying by∏j=1i−1dj\\prod\_\{j=1\}^\{i\-1\}d\_\{j\}gives𝑻i\\bm\{T\}\_\{i\}\.

Proof of[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)\.By[Lemma12](https://arxiv.org/html/2610.28975#Thmlemma12),

𝔼⁡\[𝑮i​𝑮i⊤\]=𝔼⁡\[tr⁡\(\(𝑩i​𝑩i⊤\)2\)​\(𝑨i⊤​𝑨i\)2\]\+𝔼⁡\[𝑨i⊤​𝚽​𝑩i⊤​𝑩i​𝚽⊤​𝑨i\]\.\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]=\\mathbb\{E\}\\bigl\[\\tr\(\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\)\\,\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\\bigr\]\+\\mathbb\{E\}\\bigl\[\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{\\Phi\}\\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\}\\bigr\]\.Since𝑨i,𝑩i\\bm\{A\}\_\{i\},\\bm\{B\}\_\{i\}depend on disjoint sets of𝑾j\\bm\{W\}\_\{j\}, they are independent, so by[Lemmas13](https://arxiv.org/html/2610.28975#Thmlemma13)and[14](https://arxiv.org/html/2610.28975#Thmlemma14),

𝔼⁡\[tr⁡\(\(𝑩i​𝑩i⊤\)2\)​\(𝑨i⊤​𝑨i\)2\]=𝔼⁡\[tr⁡\(\(𝑩i​𝑩i⊤\)2\)\]​𝔼​\[\(𝑨i⊤​𝑨i\)2\]=di−1​ri⋅si​𝑰di=Vi​si​𝑰di,\\mathbb\{E\}\\bigl\[\\tr\(\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\)\\,\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\\bigr\]=\\mathbb\{E\}\\bigl\[\\tr\(\(\\bm\{B\}\_\{i\}\\bm\{B\}\_\{i\}^\{\\top\}\)^\{2\}\)\\bigr\]\\,\\mathbb\{E\}\\bigl\[\(\\bm\{A\}\_\{i\}^\{\\top\}\\bm\{A\}\_\{i\}\)^\{2\}\\bigr\]=d\_\{i\-1\}r\_\{i\}\\cdot s\_\{i\}\\,\\bm\{I\}\_\{d\_\{i\}\}=V\_\{i\}s\_\{i\}\\,\\bm\{I\}\_\{d\_\{i\}\},and the second term equals𝑻i\\bm\{T\}\_\{i\}by[Lemma15](https://arxiv.org/html/2610.28975#Thmlemma15)\.[Corollary2](https://arxiv.org/html/2610.28975#Thmcorollary2)follows from the entries of \([36](https://arxiv.org/html/2610.28975#A3.E36)\): fori<Li<L,𝑻i\\bm\{T\}\_\{i\}is a multiple of𝑰di\\bm\{I\}\_\{d\_\{i\}\}so the off\-diagonal vanishes; fori=Li=L,𝑻L\\bm\{T\}\_\{L\}contributes\(𝚽​𝚽⊤\)a​b\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}entry\-wise\.

It remains to prove \([38](https://arxiv.org/html/2610.28975#A3.E38)\)\. Letπi:=∏j=0i−1dj\\pi\_\{i\}:=\\prod\_\{j=0\}^\{i\-1\}d\_\{j\}andσi:=∑m=0i−1dm−1\\sigma\_\{i\}:=\\sum\_\{m=0\}^\{i\-1\}d\_\{m\}^\{\-1\}, so thatπi=di−1​πi−1\\pi\_\{i\}=d\_\{i\-1\}\\pi\_\{i\-1\}andσi=σi−1\+di−1−1\\sigma\_\{i\}=\\sigma\_\{i\-1\}\+d\_\{i\-1\}^\{\-1\}\. SinceVi=di−1​riV\_\{i\}=d\_\{i\-1\}r\_\{i\}, the recursion \([31](https://arxiv.org/html/2610.28975#A3.E31)\)–\([32](https://arxiv.org/html/2610.28975#A3.E32)\) reads

V1=d0,v1=d02,\\displaystyle V\_\{1\}=d\_\{0\},\\qquad v\_\{1\}=d\_\{0\}^\{2\},Vi=di−1vi−1\+di−1\(di−1\+1\)Vi−1,vi=di−12vi−1\+2di−1Vi−1\(i≥2\)\.\\displaystyle V\_\{i\}=d\_\{i\-1\}v\_\{i\-1\}\+d\_\{i\-1\}\(d\_\{i\-1\}\+1\)V\_\{i\-1\},\\qquad v\_\{i\}=d\_\{i\-1\}^\{2\}v\_\{i\-1\}\+2d\_\{i\-1\}V\_\{i\-1\}\\qquad\(i\\geq 2\)\.Letai:=vi/πi2a\_\{i\}:=v\_\{i\}/\\pi\_\{i\}^\{2\}andbi:=Vi/\(πi2​σi\)b\_\{i\}:=V\_\{i\}/\(\\pi\_\{i\}^\{2\}\\sigma\_\{i\}\), soa1=b1=1a\_\{1\}=b\_\{1\}=1\. Dividing the two recursions byπi2\\pi\_\{i\}^\{2\}and byπi2​σi\\pi\_\{i\}^\{2\}\\sigma\_\{i\}gives

ai=ai−1\+2​σi−1di−1​bi−1,bi=di−1−1σi​ai−1\+σi−1σi​\(1\+di−1−1\)​bi−1\.a\_\{i\}=a\_\{i\-1\}\+\\frac\{2\\sigma\_\{i\-1\}\}\{d\_\{i\-1\}\}\\,b\_\{i\-1\},\\qquad b\_\{i\}=\\frac\{d\_\{i\-1\}^\{\-1\}\}\{\\sigma\_\{i\}\}\\,a\_\{i\-1\}\+\\frac\{\\sigma\_\{i\-1\}\}\{\\sigma\_\{i\}\}\\,\\bigl\(1\+d\_\{i\-1\}^\{\-1\}\\bigr\)b\_\{i\-1\}\.The second identity expressesbib\_\{i\}as a convex combination ofai−1a\_\{i\-1\}and\(1\+di−1−1\)​bi−1\(1\+d\_\{i\-1\}^\{\-1\}\)b\_\{i\-1\}, somin⁡\(ai−1,bi−1\)≤bi≤2​max⁡\(ai−1,bi−1\)\\min\(a\_\{i\-1\},b\_\{i\-1\}\)\\leq b\_\{i\}\\leq 2\\max\(a\_\{i\-1\},b\_\{i\-1\}\); the first givesai−1≤ai≤ai−1\+2​\(i−1\)​bi−1a\_\{i\-1\}\\leq a\_\{i\}\\leq a\_\{i\-1\}\+2\(i\-1\)b\_\{i\-1\}, usingσi−1≤i−1\\sigma\_\{i\-1\}\\leq i\-1anddi−1≥1d\_\{i\-1\}\\geq 1\. By induction,1≤ai,bi≤\(2​L\)L1\\leq a\_\{i\},b\_\{i\}\\leq\(2L\)^\{L\}for everyi≤Li\\leq L, that is,

vi=Θ⁡\(πi2\),Vi=Θ⁡\(πi2​σi\),v\_\{i\}=\\Theta\(\\pi\_\{i\}^\{2\}\),\\qquad V\_\{i\}=\\Theta\(\\pi\_\{i\}^\{2\}\\sigma\_\{i\}\),The recursion \([33](https://arxiv.org/html/2610.28975#A3.E33)\)–\([34](https://arxiv.org/html/2610.28975#A3.E34)\) has the same form read from the output side: withSi:=di​siS\_\{i\}:=d\_\{i\}s\_\{i\},ρi:=∏j=iLdj\\rho\_\{i\}:=\\prod\_\{j=i\}^\{L\}d\_\{j\}andτi:=∑m=iLdm−1\\tau\_\{i\}:=\\sum\_\{m=i\}^\{L\}d\_\{m\}^\{\-1\},

SL=dL,uL=dL2,\\displaystyle S\_\{L\}=d\_\{L\},\\qquad u\_\{L\}=d\_\{L\}^\{2\},Si−1=di−1ui\+di−1\(di−1\+1\)Si,ui−1=di−12ui\+2di−1Si\(i≤L\),\\displaystyle S\_\{i\-1\}=d\_\{i\-1\}u\_\{i\}\+d\_\{i\-1\}\(d\_\{i\-1\}\+1\)S\_\{i\},\\qquad u\_\{i\-1\}=d\_\{i\-1\}^\{2\}u\_\{i\}\+2d\_\{i\-1\}S\_\{i\}\\qquad\(i\\leq L\),and the same argument givesui=Θ⁡\(ρi2\)u\_\{i\}=\\Theta\(\\rho\_\{i\}^\{2\}\)andSi=Θ⁡\(ρi2​τi\)S\_\{i\}=\\Theta\(\\rho\_\{i\}^\{2\}\\tau\_\{i\}\)\. Therefore

Vi​si=Vi​Sidi=Θ⁡\(πi2​ρi2di​σi​τi\),πi2​ρi2di=1di​\(∏j=0Ldj\)2\.V\_\{i\}s\_\{i\}=\\frac\{V\_\{i\}S\_\{i\}\}\{d\_\{i\}\}=\\Theta\\Bigl\(\\frac\{\\pi\_\{i\}^\{2\}\\rho\_\{i\}^\{2\}\}\{d\_\{i\}\}\\,\\sigma\_\{i\}\\tau\_\{i\}\\Bigr\),\\qquad\\frac\{\\pi\_\{i\}^\{2\}\\rho\_\{i\}^\{2\}\}\{d\_\{i\}\}=\\frac\{1\}\{d\_\{i\}\}\\Bigl\(\\prod\_\{j=0\}^\{L\}d\_\{j\}\\Bigr\)^\{2\}\.Sinced0−1≤σi≤Ld\_\{0\}^\{\-1\}\\leq\\sigma\_\{i\}\\leq LanddL−1≤τi≤Ld\_\{L\}^\{\-1\}\\leq\\tau\_\{i\}\\leq L, the factorσi​τi\\sigma\_\{i\}\\tau\_\{i\}is bounded above and below, and\(∏j=0Ldj\)2/di\\bigl\(\\prod\_\{j=0\}^\{L\}d\_\{j\}\\bigr\)^\{2\}/d\_\{i\}equalsd02​dL2​di​∏j=1,j≠iL−1dj2d\_\{0\}^\{2\}d\_\{L\}^\{2\}\\,d\_\{i\}\\prod\_\{j=1,\\,j\\neq i\}^\{L\-1\}d\_\{j\}^\{2\}fori<Li<Landd02​dL​∏j=1L−1dj2d\_\{0\}^\{2\}d\_\{L\}\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}^\{2\}fori=Li=L, which gives \([38](https://arxiv.org/html/2610.28975#A3.E38)\)\.

## Appendix DTwo\-Layer ReLU Networks

### D\.1Setup

Letd0,d1,d2d\_\{0\},d\_\{1\},d\_\{2\}be positive integers\. The two\-layer ReLU network

f⁡\(𝒙\)=𝑾2​σ​\(𝑾1​𝒙\),𝑾1∈ℝd1×d0,𝑾2∈ℝd2×d1,f\(\\bm\{x\}\)=\\bm\{W\}\_\{2\}\\,\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\),\\qquad\\bm\{W\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{0\}\},\\quad\\bm\{W\}\_\{2\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{1\}\},\(45\)acts on𝒙∈ℝd0\\bm\{x\}\\in\\mathbb\{R\}^\{d\_\{0\}\}viaσ⁡\(z\)=max⁡\(z,0\)\\sigma\(z\)=\\max\(z,0\),σ′\(z\)=𝟏\[z\>0\]\\sigma^\{\\prime\}\(z\)=\\mathbf\{1\}\[z\>0\]\. Let𝒉⁡\(𝒙\):=σ⁡\(𝑾1​𝒙\)∈ℝd1\\bm\{h\}\(\\bm\{x\}\):=\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\)\\in\\mathbb\{R\}^\{d\_\{1\}\}, and let𝒘a∈ℝd0\\bm\{w\}\_\{a\}\\in\\mathbb\{R\}^\{d\_\{0\}\}denote theaa\-th row of𝑾1\\bm\{W\}\_\{1\}\. The population loss against a deterministic linear target𝚽∈ℝd2×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{0\}\}is

ℒ⁡\(𝑾1,𝑾2\):=12​𝔼𝒙​‖𝑾2​σ​\(𝑾1​𝒙\)−𝚽​𝒙‖2,\\mathcal\{L\}\(\\bm\{W\}\_\{1\},\\bm\{W\}\_\{2\}\)\\;:=\\;\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\\left\\\|\\bm\{W\}\_\{2\}\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\)\-\\bm\{\\Phi\}\\bm\{x\}\\right\\\|^\{2\},\(46\)and𝑮i:=∂ℒ/∂𝑾i\\bm\{G\}\_\{i\}:=\\partial\\mathcal\{L\}/\\partial\\bm\{W\}\_\{i\},i∈\{1,2\}i\\in\\\{1,2\\\}\. For independent𝒖,𝒗∼𝒩⁡\(0,𝑰d0\)\\bm\{u\},\\bm\{v\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{d\_\{0\}\}\)letθ:=∠⁡\(𝒖,𝒗\)∈\[0,π\]\\theta:=\\angle\(\\bm\{u\},\\bm\{v\}\)\\in\[0,\\pi\]be their angle, so thatcos⁡θ=⟨𝒖,𝒗⟩/\(‖𝒖‖​‖𝒗‖\)\\cos\\theta=\\langle\\bm\{u\},\\bm\{v\}\\rangle/\(\\left\\\|\\bm\{u\}\\right\\\|\\left\\\|\\bm\{v\}\\right\\\|\), and define

κ1\\displaystyle\\kappa\_\{1\}:=𝔼⁡\[sin2⁡θ\+\(π−θ\)24​π2\+sin⁡θ​cos⁡θ​\(π−θ\)2​π2\],\\displaystyle:=\\mathbb\{E\}\\\!\\left\[\\frac\{\\sin^\{2\}\\theta\+\(\\pi\-\\theta\)^\{2\}\}\{4\\pi^\{2\}\}\+\\frac\{\\sin\\theta\\cos\\theta\(\\pi\-\\theta\)\}\{2\\pi^\{2\}\}\\right\],\(47\)κ2\\displaystyle\\kappa\_\{2\}:=𝔼⁡\[sin⁡θ⁡\(π−θ\)\]2​π2\.\\displaystyle:=\\frac\{\\mathbb\{E\}\[\\sin\\theta\(\\pi\-\\theta\)\]\}\{2\\pi^\{2\}\}\.\(48\)

### D\.2Standing assumption

###### Assumption 7\.

\(𝑾1\)i​j\(\\bm\{W\}\_\{1\}\)\_\{ij\}and\(𝐖2\)i​j\(\\bm\{W\}\_\{2\}\)\_\{ij\}are i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\);𝐱∼𝒩⁡\(0,𝐈d0\)\\bm\{x\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{d\_\{0\}\}\);𝚽\\bm\{\\Phi\}is deterministic and independent of𝐖1,𝐖2\\bm\{W\}\_\{1\},\\bm\{W\}\_\{2\}\. Expectations𝔼⁡\[⋅\]\\mathbb\{E\}\[\\cdot\]are over𝐖1,𝐖2\\bm\{W\}\_\{1\},\\bm\{W\}\_\{2\}\(𝐱\\bm\{x\}has been integrated out inℒ\\mathcal\{L\}\)\.

### D\.3Main results restated

Theorem \([Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3), restated\)\.Under[Assumption7](https://arxiv.org/html/2610.28975#Thmassumption7), with𝐇:=𝔼𝐱​\[𝐡⁡\(𝐱\)​𝐡​\(𝐱\)⊤\]\\bm\{H\}:=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{h\}\(\\bm\{x\}\)\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\]andα:=𝔼𝐖1​\[tr⁡\(𝐇2\)\]\\alpha:=\\mathbb\{E\}\_\{\\bm\{W\}\_\{1\}\}\[\\tr\(\\bm\{H\}^\{2\}\)\],

𝔼⁡\[𝑮2​𝑮2⊤\]=α​𝑰d2\+d14​𝚽​𝚽⊤,α=Θ⁡\(d12\),\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]=\\alpha\\,\\bm\{I\}\_\{d\_\{2\}\}\+\\tfrac\{d\_\{1\}\}\{4\}\\,\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},\\qquad\\alpha=\\Theta\(d\_\{1\}^\{2\}\),\(49\)and𝔼⁡\[𝐆1​𝐆1⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]is permutation\-invariant in the hidden index with

𝔼⁡\[\(𝑮1​𝑮1⊤\)a​a\]\\displaystyle\\mathbb\{E\}\\bigl\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{aa\}\\bigr\]=d0​d224\+d0​d2​\[\(d1−1\)​κ1\+12\]\+‖𝚽‖F24,\\displaystyle=\\tfrac\{d\_\{0\}d\_\{2\}^\{2\}\}\{4\}\+d\_\{0\}d\_\{2\}\\bigl\[\(d\_\{1\}\-1\)\\kappa\_\{1\}\+\\tfrac\{1\}\{2\}\\bigr\]\+\\tfrac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\},\(50\)𝔼⁡\[\(𝑮1​𝑮1⊤\)a​c\]\\displaystyle\\mathbb\{E\}\\bigl\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{ac\}\\bigr\]=d0d2κ2\+O\(d2d0\)\(a≠c\)\.\\displaystyle=d\_\{0\}d\_\{2\}\\,\\kappa\_\{2\}\+O\(d\_\{2\}\\sqrt\{d\_\{0\}\}\)\\qquad\(a\\neq c\)\.\(51\)Asd0→∞d\_\{0\}\\to\\infty,κ1→14​π2\+116\\kappa\_\{1\}\\to\\tfrac\{1\}\{4\\pi^\{2\}\}\+\\tfrac\{1\}\{16\}andκ2→14​π\\kappa\_\{2\}\\to\\tfrac\{1\}\{4\\pi\}\.

### D\.4Preliminary lemmas

###### Lemma 16\.

Same as[Lemma1](https://arxiv.org/html/2610.28975#Thmlemma1)\.

###### Lemma 17\(Gaussian integration by parts\)\.

Let𝐱∼𝒩⁡\(0,𝐈n\)\\bm\{x\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{n\}\)andf:ℝn→ℝf:\\mathbb\{R\}^\{n\}\\to\\mathbb\{R\}be locally absolutely continuous along almost every line parallel to thejj\-th axis, with weak derivative∂jf\\partial\_\{j\}f, and𝔼​\|xj​f​\(𝐱\)\|<∞\\mathbb\{E\}\|x\_\{j\}f\(\\bm\{x\}\)\|<\\infty,𝔼​\|∂jf⁡\(𝐱\)\|<∞\\mathbb\{E\}\|\\partial\_\{j\}f\(\\bm\{x\}\)\|<\\infty\. Then

𝔼⁡\[xj​f​\(𝒙\)\]=𝔼⁡\[∂jf⁡\(𝒙\)\]\.\\mathbb\{E\}\[x\_\{j\}f\(\\bm\{x\}\)\]=\\mathbb\{E\}\[\\partial\_\{j\}f\(\\bm\{x\}\)\]\.\(52\)In particular this holds wheneverffis Lipschitz\.

Proof\.A standard identity\([Stein, 1981](https://arxiv.org/html/2610.28975#bib.bib31);[Nourdin and Peccati, 2012](https://arxiv.org/html/2610.28975#bib.bib32)\)\. The standard Gaussian densityφ\\varphisatisfies∂jφ⁡\(𝒙\)=−xj​φ​\(𝒙\)\\partial\_\{j\}\\varphi\(\\bm\{x\}\)=\-x\_\{j\}\\varphi\(\\bm\{x\}\), so by Fubini and one\-dimensional integration by parts inxjx\_\{j\}\(the boundary term vanishes by integrability\),

𝔼\[xjf\(𝒙\)\]=−∫f∂jφ=∫∂jfφ=𝔼\[∂jf\(𝒙\)\]\.\\mathbb\{E\}\[x\_\{j\}f\(\\bm\{x\}\)\]=\-\\int f\\,\\partial\_\{j\}\\varphi=\\int\\partial\_\{j\}f\\,\\varphi=\\mathbb\{E\}\[\\partial\_\{j\}f\(\\bm\{x\}\)\]\.The Lipschitz case follows from Rademacher’s theorem\.

###### Lemma 18\.

ForZ∼𝒩⁡\(0,s2\)Z\\sim\\mathcal\{N\}\(0,s^\{2\}\)withs\>0s\>0,

𝔼⁡\[σ⁡\(Z\)\]=s2​π,𝔼⁡\[σ​\(Z\)2\]=s22,𝔼⁡\[σ′​\(Z\)\]=12\.\\mathbb\{E\}\[\\sigma\(Z\)\]=\\frac\{s\}\{\\sqrt\{2\\pi\}\},\\qquad\\mathbb\{E\}\[\\sigma\(Z\)^\{2\}\]=\\frac\{s^\{2\}\}\{2\},\\qquad\\mathbb\{E\}\[\\sigma^\{\\prime\}\(Z\)\]=\\frac\{1\}\{2\}\.\(53\)

Proof\.Direct computation:𝔼⁡\[σ⁡\(Z\)\]=∫0∞z​φs​\(z\)​𝑑z=s/2​π\\mathbb\{E\}\[\\sigma\(Z\)\]=\\int\_\{0\}^\{\\infty\}z\\varphi\_\{s\}\(z\)\\,dz=s/\\sqrt\{2\\pi\};𝔼⁡\[σ​\(Z\)2\]=∫0∞z2​φs​\(z\)​𝑑z=s2/2\\mathbb\{E\}\[\\sigma\(Z\)^\{2\}\]=\\int\_\{0\}^\{\\infty\}z^\{2\}\\varphi\_\{s\}\(z\)\\,dz=s^\{2\}/2;𝔼⁡\[σ′​\(Z\)\]=Pr⁡\(Z\>0\)=1/2\\mathbb\{E\}\[\\sigma^\{\\prime\}\(Z\)\]=\\Pr\(Z\>0\)=1/2\.

###### Lemma 19\(Cho–Saul kernel\)\.

Let𝐮,𝐯∈ℝn∖\{0\}\\bm\{u\},\\bm\{v\}\\in\\mathbb\{R\}^\{n\}\\setminus\\\{0\\\},𝐱∼𝒩⁡\(0,𝐈n\)\\bm\{x\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{n\}\),θ:=∠⁡\(𝐮,𝐯\)\\theta:=\\angle\(\\bm\{u\},\\bm\{v\}\)\. Then\([Cho and Saul, 2009](https://arxiv.org/html/2610.28975#bib.bib34)\)

𝔼⁡\[σ⁡\(𝒖⊤​𝒙\)​σ​\(𝒗⊤​𝒙\)\]\\displaystyle\\mathbb\{E\}\[\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)\]=‖𝒖‖​‖𝒗‖2​π​\(sin⁡θ\+\(π−θ\)​cos⁡θ\),\\displaystyle=\\frac\{\\left\\\|\\bm\{u\}\\right\\\|\\left\\\|\\bm\{v\}\\right\\\|\}\{2\\pi\}\\bigl\(\\sin\\theta\+\(\\pi\-\\theta\)\\cos\\theta\\bigr\),\(54\)𝔼⁡\[σ′​\(𝒖⊤​𝒙\)​σ′​\(𝒗⊤​𝒙\)\]\\displaystyle\\mathbb\{E\}\[\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma^\{\\prime\}\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)\]=π−θ2​π\.\\displaystyle=\\frac\{\\pi\-\\theta\}\{2\\pi\}\.\(55\)

Proof\.Both quantities depend on\(𝒖⊤​𝒙,𝒗⊤​𝒙\)\(\\bm\{u\}^\{\\top\}\\bm\{x\},\\bm\{v\}^\{\\top\}\\bm\{x\}\), bivariate Gaussian with variances‖𝒖‖2,‖𝒗‖2\\left\\\|\\bm\{u\}\\right\\\|^\{2\},\\left\\\|\\bm\{v\}\\right\\\|^\{2\}and correlationcos⁡θ\\cos\\theta\. Equation \([55](https://arxiv.org/html/2610.28975#A4.E55)\) isPr⁡\(𝒖⊤​𝒙\>0,𝒗⊤​𝒙\>0\)\\Pr\(\\bm\{u\}^\{\\top\}\\bm\{x\}\>0,\\,\\bm\{v\}^\{\\top\}\\bm\{x\}\>0\), the angular measure of the cone\{Z1\>0,Z2\>0\}\\\{Z\_\{1\}\>0,Z\_\{2\}\>0\\\}for a standard bivariate Gaussian with correlationcos⁡θ\\cos\\theta, namely\(π−θ\)/\(2​π\)\(\\pi\-\\theta\)/\(2\\pi\)\. For \([54](https://arxiv.org/html/2610.28975#A4.E54)\), by Price’s theorem\([Price, 1958](https://arxiv.org/html/2610.28975#bib.bib33)\),

∂∂ρ​𝔼​\[σ⁡\(𝒖⊤​𝒙\)​σ​\(𝒗⊤​𝒙\)\]=‖𝒖‖​‖𝒗‖​𝔼​\[σ′​\(𝒖⊤​𝒙\)​σ′​\(𝒗⊤​𝒙\)\]=‖𝒖‖​‖𝒗‖​\(π−θ\)2​π;\\frac\{\\partial\}\{\\partial\\rho\}\\mathbb\{E\}\[\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)\]=\\left\\\|\\bm\{u\}\\right\\\|\\left\\\|\\bm\{v\}\\right\\\|\\,\\mathbb\{E\}\[\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma^\{\\prime\}\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)\]=\\frac\{\\left\\\|\\bm\{u\}\\right\\\|\\left\\\|\\bm\{v\}\\right\\\|\(\\pi\-\\theta\)\}\{2\\pi\};atρ=−1\\rho=\-1\(i\.e\.θ=π\\theta=\\pi\) the product vanishes a\.s\., and integrating∂ρ\\partial\_\{\\rho\}from−1\-1tocos⁡θ\\cos\\thetagives \([54](https://arxiv.org/html/2610.28975#A4.E54)\)\.

###### Lemma 20\(Wick/Isserlis\)\.

Same as[Lemma3](https://arxiv.org/html/2610.28975#Thmlemma3)\.

###### Lemma 21\.

Let𝐱∼𝒩⁡\(0,𝐈n\)\\bm\{x\}\\sim\\mathcal\{N\}\(0,\\bm\{I\}\_\{n\}\)and𝐮∈ℝn∖\{0\}\\bm\{u\}\\in\\mathbb\{R\}^\{n\}\\setminus\\\{0\\\}\. Then

𝔼\[𝒙𝒙⊤1\[𝒖⊤𝒙\>0\]\]=12𝑰n\.\\mathbb\{E\}\\bigl\[\\bm\{x\}\\bm\{x\}^\{\\top\}\\,\\mathbf\{1\}\[\\bm\{u\}^\{\\top\}\\bm\{x\}\>0\]\\bigr\]=\\tfrac\{1\}\{2\}\\,\\bm\{I\}\_\{n\}\.\(56\)

Proof\.Let𝑨:=𝔼\[𝒙𝒙⊤𝟏\[𝒖⊤𝒙\>0\]\]\\bm\{A\}:=\\mathbb\{E\}\[\\bm\{x\}\\bm\{x\}^\{\\top\}\\mathbf\{1\}\[\\bm\{u\}^\{\\top\}\\bm\{x\}\>0\]\], and decompose𝒙=\(𝒖⊤​𝒙/‖𝒖‖2\)​𝒖\+𝒙⟂\\bm\{x\}=\(\\bm\{u\}^\{\\top\}\\bm\{x\}/\\left\\\|\\bm\{u\}\\right\\\|^\{2\}\)\\bm\{u\}\+\\bm\{x\}\_\{\\perp\}with𝒙⟂\\bm\{x\}\_\{\\perp\}the projection onto𝒖⟂\\bm\{u\}^\{\\perp\}\. In an orthonormal basis containing𝒖/‖𝒖‖\\bm\{u\}/\\left\\\|\\bm\{u\}\\right\\\|, the off\-diagonal entries of𝑨\\bm\{A\}vanish by the reflection symmetry of𝒙⟂\\bm\{x\}\_\{\\perp\}; the entry along𝒖/‖𝒖‖\\bm\{u\}/\\left\\\|\\bm\{u\}\\right\\\|is𝔼\[\(𝒖⊤𝒙/‖𝒖‖\)2𝟏\[𝒖⊤𝒙\>0\]\]=12\\mathbb\{E\}\[\(\\bm\{u\}^\{\\top\}\\bm\{x\}/\\left\\\|\\bm\{u\}\\right\\\|\)^\{2\}\\mathbf\{1\}\[\\bm\{u\}^\{\\top\}\\bm\{x\}\>0\]\]=\\tfrac\{1\}\{2\}\(𝒖⊤​𝒙\\bm\{u\}^\{\\top\}\\bm\{x\}symmetric\), and each remaining diagonal entry is12​𝔼​\[xj2\]=12\\tfrac\{1\}\{2\}\\mathbb\{E\}\[x\_\{j\}^\{2\}\]=\\tfrac\{1\}\{2\}by independence of𝒙⟂\\bm\{x\}\_\{\\perp\}from𝒖⊤​𝒙\\bm\{u\}^\{\\top\}\\bm\{x\}andPr⁡\(𝒖⊤​𝒙\>0\)=12\\Pr\(\\bm\{u\}^\{\\top\}\\bm\{x\}\>0\)=\\tfrac\{1\}\{2\}\. Hence𝑨=12​𝑰n\\bm\{A\}=\\tfrac\{1\}\{2\}\\bm\{I\}\_\{n\}\.

### D\.5Proof of the gradient self outer\-product of𝑾2\\bm\{W\}\_\{2\}

###### Lemma 22\.

𝑮2=𝑾2​𝑯−𝚽​𝑪⊤\\bm\{G\}\_\{2\}=\\bm\{W\}\_\{2\}\\bm\{H\}\-\\bm\{\\Phi\}\\bm\{C\}^\{\\top\}, where𝐇:=𝔼𝐱​\[𝐡⁡\(𝐱\)​𝐡​\(𝐱\)⊤\]∈ℝd1×d1\\bm\{H\}:=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{h\}\(\\bm\{x\}\)\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\]\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{1\}\}and𝐂:=𝔼𝐱​\[𝐡⁡\(𝐱\)​𝐱⊤\]∈ℝd1×d0\\bm\{C\}:=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{h\}\(\\bm\{x\}\)\\bm\{x\}^\{\\top\}\]\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{0\}\}\.

Proof\.Differentiating \([46](https://arxiv.org/html/2610.28975#A4.E46)\) in𝑾2\\bm\{W\}\_\{2\},

𝑮2=𝔼𝒙​\[\(𝑾2​𝒉​\(𝒙\)−𝚽​𝒙\)​𝒉​\(𝒙\)⊤\]=𝑾2​𝔼𝒙​\[𝒉⁡\(𝒙\)​𝒉​\(𝒙\)⊤\]−𝚽​𝔼𝒙​\[𝒙​𝒉​\(𝒙\)⊤\]=𝑾2​𝑯−𝚽​𝑪⊤\.\\bm\{G\}\_\{2\}=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\(\\bm\{W\}\_\{2\}\\bm\{h\}\(\\bm\{x\}\)\-\\bm\{\\Phi\}\\bm\{x\}\)\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\]=\\bm\{W\}\_\{2\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{h\}\(\\bm\{x\}\)\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\]\-\\bm\{\\Phi\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{x\}\\bm\{h\}\(\\bm\{x\}\)^\{\\top\}\]=\\bm\{W\}\_\{2\}\\bm\{H\}\-\\bm\{\\Phi\}\\bm\{C\}^\{\\top\}\.
###### Lemma 23\.

𝑪=12​𝑾1\\bm\{C\}=\\tfrac\{1\}\{2\}\\bm\{W\}\_\{1\}\.

Proof\.Ca​j=𝔼x​\[σ⁡\(wa⊤​x\)​xj\]\\bm\{C\}\_\{aj\}=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma\(\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{x\}\)x\_\{j\}\], andf⁡\(𝒙\):=σ⁡\(𝒘a⊤​𝒙\)f\(\\bm\{x\}\):=\\sigma\(\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{x\}\)is Lipschitz with∂jf\(𝒙\)=𝟏\[𝒘a⊤𝒙\>0\]\(𝒘a\)j\\partial\_\{j\}f\(\\bm\{x\}\)=\\mathbf\{1\}\[\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{x\}\>0\]\(\\bm\{w\}\_\{a\}\)\_\{j\}a\.e\. By[Lemma17](https://arxiv.org/html/2610.28975#Thmlemma17)and[Lemma18](https://arxiv.org/html/2610.28975#Thmlemma18),

𝑪a​j=𝔼𝒙​\[∂jf⁡\(𝒙\)\]=\(𝒘a\)j​𝔼𝒙​\[σ′​\(𝒘a⊤​𝒙\)\]=12​\(𝒘a\)j\.\\bm\{C\}\_\{aj\}=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\partial\_\{j\}f\(\\bm\{x\}\)\]=\(\\bm\{w\}\_\{a\}\)\_\{j\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma^\{\\prime\}\(\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{x\}\)\]=\\tfrac\{1\}\{2\}\(\\bm\{w\}\_\{a\}\)\_\{j\}\.
Proof of \([49](https://arxiv.org/html/2610.28975#A4.E49)\)\.By[Lemma22](https://arxiv.org/html/2610.28975#Thmlemma22),

𝑮2​𝑮2⊤=𝑾2​𝑯2​𝑾2⊤−𝑾2​𝑯​𝑪​𝚽⊤−𝚽​𝑪⊤​𝑯​𝑾2⊤\+𝚽​𝑪⊤​𝑪​𝚽⊤\.\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}=\\bm\{W\}\_\{2\}\\bm\{H\}^\{2\}\\bm\{W\}\_\{2\}^\{\\top\}\-\\bm\{W\}\_\{2\}\\bm\{H\}\\bm\{C\}\\bm\{\\Phi\}^\{\\top\}\-\\bm\{\\Phi\}\\bm\{C\}^\{\\top\}\\bm\{H\}\\bm\{W\}\_\{2\}^\{\\top\}\+\\bm\{\\Phi\}\\bm\{C\}^\{\\top\}\\bm\{C\}\\bm\{\\Phi\}^\{\\top\}\.Since𝔼⁡\[𝑾2\]=0\\mathbb\{E\}\[\\bm\{W\}\_\{2\}\]=0, the cross terms vanish under𝔼𝑾2\[⋅∣𝑾1\]\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\cdot\\mid\\bm\{W\}\_\{1\}\], and by[Lemma16](https://arxiv.org/html/2610.28975#Thmlemma16)\(left identity,𝑴=𝑯2\\bm\{M\}=\\bm\{H\}^\{2\}\),

𝔼⁡\[𝑮2​𝑮2⊤∣𝑾1\]=tr⁡\(𝑯2\)​𝑰d2\+𝚽​𝑪⊤​𝑪​𝚽⊤\.\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\\mid\\bm\{W\}\_\{1\}\]=\\tr\(\\bm\{H\}^\{2\}\)\\,\\bm\{I\}\_\{d\_\{2\}\}\+\\bm\{\\Phi\}\\bm\{C\}^\{\\top\}\\bm\{C\}\\bm\{\\Phi\}^\{\\top\}\.\(57\)By[Lemma23](https://arxiv.org/html/2610.28975#Thmlemma23),𝑪⊤​𝑪=14​𝑾1⊤​𝑾1\\bm\{C\}^\{\\top\}\\bm\{C\}=\\tfrac\{1\}\{4\}\\bm\{W\}\_\{1\}^\{\\top\}\\bm\{W\}\_\{1\}, and[Lemma16](https://arxiv.org/html/2610.28975#Thmlemma16)\(right identity,𝑵=𝑰d1\\bm\{N\}=\\bm\{I\}\_\{d\_\{1\}\}\) gives𝔼𝑾1​\[𝑾1⊤​𝑾1\]=d1​𝑰d0\\mathbb\{E\}\_\{\\bm\{W\}\_\{1\}\}\[\\bm\{W\}\_\{1\}^\{\\top\}\\bm\{W\}\_\{1\}\]=d\_\{1\}\\bm\{I\}\_\{d\_\{0\}\}, so𝔼𝑾1​\[𝑪⊤​𝑪\]=d14​𝑰d0\\mathbb\{E\}\_\{\\bm\{W\}\_\{1\}\}\[\\bm\{C\}^\{\\top\}\\bm\{C\}\]=\\tfrac\{d\_\{1\}\}\{4\}\\bm\{I\}\_\{d\_\{0\}\}\. Taking𝔼𝑾1\\mathbb\{E\}\_\{\\bm\{W\}\_\{1\}\}of \([57](https://arxiv.org/html/2610.28975#A4.E57)\),

𝔼⁡\[𝑮2​𝑮2⊤\]=𝔼𝑾1​\[tr⁡\(𝑯2\)\]​𝑰d2\+d14​𝚽​𝚽⊤=α​𝑰d2\+d14​𝚽​𝚽⊤\.\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]=\\mathbb\{E\}\_\{\\bm\{W\}\_\{1\}\}\[\\tr\(\\bm\{H\}^\{2\}\)\]\\,\\bm\{I\}\_\{d\_\{2\}\}\+\\tfrac\{d\_\{1\}\}\{4\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}=\\alpha\\,\\bm\{I\}\_\{d\_\{2\}\}\+\\tfrac\{d\_\{1\}\}\{4\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\.For the order ofα\\alpha, splittr⁡\(𝑯2\)=∑aHa​a2\+∑a≠b𝑯a​b2\\tr\(\\bm\{H\}^\{2\}\)=\\sum\_\{a\}H\_\{aa\}^\{2\}\+\\sum\_\{a\\neq b\}\\bm\{H\}\_\{ab\}^\{2\}\. By[Lemma18](https://arxiv.org/html/2610.28975#Thmlemma18),𝑯a​a=12​‖𝒘a‖2\\bm\{H\}\_\{aa\}=\\tfrac\{1\}\{2\}\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}with‖𝒘a‖2∼χd02\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}\\sim\\chi^\{2\}\_\{d\_\{0\}\}, so

∑a𝔼⁡\[𝑯a​a2\]=d1​d0​\(d0\+2\)4=Θ⁡\(d1\)\.\\sum\_\{a\}\\mathbb\{E\}\[\\bm\{H\}\_\{aa\}^\{2\}\]=\\frac\{d\_\{1\}d\_\{0\}\(d\_\{0\}\+2\)\}\{4\}=\\Theta\(d\_\{1\}\)\.By \([54](https://arxiv.org/html/2610.28975#A4.E54)\),𝑯a​b=‖𝒘a‖​‖𝒘b‖​g​\(θa​b\)\\bm\{H\}\_\{ab\}=\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|\\left\\\|\\bm\{w\}\_\{b\}\\right\\\|\\,g\(\\theta\_\{ab\}\)withg⁡\(θ\)=\(sin⁡θ\+\(π−θ\)​cos⁡θ\)/\(2​π\)g\(\\theta\)=\(\\sin\\theta\+\(\\pi\-\\theta\)\\cos\\theta\)/\(2\\pi\)andθa​b=∠⁡\(𝒘a,𝒘b\)\\theta\_\{ab\}=\\angle\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{b\}\)independent of‖𝒘a‖,‖𝒘b‖\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|,\\left\\\|\\bm\{w\}\_\{b\}\\right\\\|, so𝔼⁡\[𝑯a​b2\]=d02​𝔼​\[g​\(θ\)2\]\\mathbb\{E\}\[\\bm\{H\}\_\{ab\}^\{2\}\]=d\_\{0\}^\{2\}\\,\\mathbb\{E\}\[g\(\\theta\)^\{2\}\], where𝔼⁡\[g​\(θ\)2\]\>0\\mathbb\{E\}\[g\(\\theta\)^\{2\}\]\>0, and

∑a≠b𝔼⁡\[𝑯a​b2\]=d1​\(d1−1\)​d02​𝔼​\[g​\(θ\)2\]=Θ⁡\(d12\)\.\\sum\_\{a\\neq b\}\\mathbb\{E\}\[\\bm\{H\}\_\{ab\}^\{2\}\]=d\_\{1\}\(d\_\{1\}\-1\)\\,d\_\{0\}^\{2\}\\,\\mathbb\{E\}\[g\(\\theta\)^\{2\}\]=\\Theta\(d\_\{1\}^\{2\}\)\.The off\-diagonal term dominates, soα=Θ⁡\(d12\)\\alpha=\\Theta\(d\_\{1\}^\{2\}\)\.

### D\.6Proof of the gradient self outer\-product of𝑾1\\bm\{W\}\_\{1\}

By the chain rule, with residual𝒓⁡\(𝒙\):=𝑾2​σ​\(𝑾1​𝒙\)−𝚽​𝒙\\bm\{r\}\(\\bm\{x\}\):=\\bm\{W\}\_\{2\}\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\)\-\\bm\{\\Phi\}\\bm\{x\}and𝑫⁡\(𝒙\):=diag⁡\(σ′​\(𝑾1​𝒙\)\)\\bm\{D\}\(\\bm\{x\}\):=\\diagop\(\\sigma^\{\\prime\}\(\\bm\{W\}\_\{1\}\\bm\{x\}\)\),

𝑮1=𝑮1\(1\)−𝑮1\(2\),𝑮1\(1\):=𝔼𝒙\[𝑫\(𝒙\)𝑾2⊤𝑾2σ\(𝑾1𝒙\)𝒙⊤\],𝑮1\(2\):=𝔼𝒙\[𝑫\(𝒙\)𝑾2⊤𝚽𝒙𝒙⊤\]\.\\begin\{gathered\}\\bm\{G\}\_\{1\}=\\bm\{G\}\_\{1\}^\{\(1\)\}\-\\bm\{G\}\_\{1\}^\{\(2\)\},\\\\ \\bm\{G\}\_\{1\}^\{\(1\)\}:=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{D\}\(\\bm\{x\}\)\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\)\\bm\{x\}^\{\\top\}\],\\qquad\\bm\{G\}\_\{1\}^\{\(2\)\}:=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\bm\{D\}\(\\bm\{x\}\)\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\}\\bm\{x\}\\bm\{x\}^\{\\top\}\]\.\\end\{gathered\}\(58\)𝑮1\(1\)\\bm\{G\}\_\{1\}^\{\(1\)\}is quadratic in𝑾2\\bm\{W\}\_\{2\}and𝑮1\(2\)\\bm\{G\}\_\{1\}^\{\(2\)\}is linear, so the cross terms in𝑮1​𝑮1⊤\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}are odd in𝑾2\\bm\{W\}\_\{2\}and vanish under𝔼𝑾2\[⋅∣𝑾1\]\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\cdot\\mid\\bm\{W\}\_\{1\}\]:

𝔼𝑾2​\[𝑮1​𝑮1⊤∣𝑾1\]=𝔼𝑾2​\[𝑮1\(1\)​\(𝑮1\(1\)\)⊤∣𝑾1\]\+𝔼𝑾2​\[𝑮1\(2\)​\(𝑮1\(2\)\)⊤∣𝑾1\]\.\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\\mid\\bm\{W\}\_\{1\}\]=\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\bm\{G\}\_\{1\}^\{\(1\)\}\(\\bm\{G\}\_\{1\}^\{\(1\)\}\)^\{\\top\}\\mid\\bm\{W\}\_\{1\}\]\+\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\bm\{G\}\_\{1\}^\{\(2\)\}\(\\bm\{G\}\_\{1\}^\{\(2\)\}\)^\{\\top\}\\mid\\bm\{W\}\_\{1\}\]\.\(59\)
###### Lemma 24\.

𝑮1\(2\)=12​𝑾2⊤​𝚽\\bm\{G\}\_\{1\}^\{\(2\)\}=\\tfrac\{1\}\{2\}\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\}, and𝔼𝐖2​\[𝐆1\(2\)​\(𝐆1\(2\)\)⊤∣𝐖1\]=‖𝚽‖F24​𝐈d1\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\bm\{G\}\_\{1\}^\{\(2\)\}\(\\bm\{G\}\_\{1\}^\{\(2\)\}\)^\{\\top\}\\mid\\bm\{W\}\_\{1\}\]=\\tfrac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\}\\,\\bm\{I\}\_\{d\_\{1\}\}\.

Proof\.Let𝒗a:=𝚽⊤\(𝑾2\):,a∈ℝd0\\bm\{v\}\_\{a\}:=\\bm\{\\Phi\}^\{\\top\}\(\\bm\{W\}\_\{2\}\)\_\{:,a\}\\in\\mathbb\{R\}^\{d\_\{0\}\}, so\(𝑾2⊤​𝚽\)a​b=\(𝒗a\)b\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\}\)\_\{ab\}=\(\\bm\{v\}\_\{a\}\)\_\{b\}\. Then, by[Lemma21](https://arxiv.org/html/2610.28975#Thmlemma21),

\(𝑮1\(2\)\)a​b=𝔼𝒙\[𝟏\[𝒘a⊤𝒙\>0\]xb\(𝒙⊤𝒗a\)\]=\(𝔼𝒙\[𝟏\[𝒘a⊤𝒙\>0\]𝒙𝒙⊤\]𝒗a\)b=12\(𝒗a\)b,\(\\bm\{G\}\_\{1\}^\{\(2\)\}\)\_\{ab\}=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\mathbf\{1\}\[\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{x\}\>0\]\\,x\_\{b\}\\,\(\\bm\{x\}^\{\\top\}\\bm\{v\}\_\{a\}\)\]=\\bigl\(\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\mathbf\{1\}\[\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{x\}\>0\]\\,\\bm\{x\}\\bm\{x\}^\{\\top\}\]\\,\\bm\{v\}\_\{a\}\\bigr\)\_\{b\}=\\tfrac\{1\}\{2\}\(\\bm\{v\}\_\{a\}\)\_\{b\},so𝑮1\(2\)=12​𝑾2⊤​𝚽\\bm\{G\}\_\{1\}^\{\(2\)\}=\\tfrac\{1\}\{2\}\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\}and𝑮1\(2\)​\(𝑮1\(2\)\)⊤=14​𝑾2⊤​𝚽​𝚽⊤​𝑾2\\bm\{G\}\_\{1\}^\{\(2\)\}\(\\bm\{G\}\_\{1\}^\{\(2\)\}\)^\{\\top\}=\\tfrac\{1\}\{4\}\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\\bm\{W\}\_\{2\};[Lemma16](https://arxiv.org/html/2610.28975#Thmlemma16)\(right identity,𝑵=𝚽​𝚽⊤\\bm\{N\}=\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\) gives𝔼𝑾2​\[𝑾2⊤​𝚽​𝚽⊤​𝑾2\]=‖𝚽‖F2​𝑰d1\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\\bm\{W\}\_\{2\}\]=\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bm\{I\}\_\{d\_\{1\}\}\.

###### Lemma 25\.

DefineFb​\(𝐮,𝐯\):=𝔼𝐱​\[σ′​\(𝐮⊤​𝐱\)​σ​\(𝐯⊤​𝐱\)​xb\]F\_\{b\}\(\\bm\{u\},\\bm\{v\}\):=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)x\_\{b\}\]\. ThenFb​\(𝐮,𝐮\)=12​ubF\_\{b\}\(\\bm\{u\},\\bm\{u\}\)=\\tfrac\{1\}\{2\}u\_\{b\}, and for𝐮∉span⁡\(𝐯\)\\bm\{u\}\\notin\\mathrm\{span\}\(\\bm\{v\}\),

Fb​\(𝒖,𝒗\)=ub⋅‖𝒗‖​sin⁡θ2​π​‖𝒖‖\+vb⋅π−θ2​π,θ:=∠⁡\(𝒖,𝒗\)\.F\_\{b\}\(\\bm\{u\},\\bm\{v\}\)=u\_\{b\}\\cdot\\frac\{\\left\\\|\\bm\{v\}\\right\\\|\\sin\\theta\}\{2\\pi\\left\\\|\\bm\{u\}\\right\\\|\}\+v\_\{b\}\\cdot\\frac\{\\pi\-\\theta\}\{2\\pi\},\\qquad\\theta:=\\angle\(\\bm\{u\},\\bm\{v\}\)\.\(60\)

Proof\.For𝒖=𝒗\\bm\{u\}=\\bm\{v\},σ′​\(𝒖⊤​𝒙\)​σ​\(𝒖⊤​𝒙\)=σ⁡\(𝒖⊤​𝒙\)\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)=\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)a\.e\., so by[Lemma17](https://arxiv.org/html/2610.28975#Thmlemma17)\(applied to the Lipschitz𝒙↦σ⁡\(𝒖⊤​𝒙\)\\bm\{x\}\\mapsto\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\) and[Lemma18](https://arxiv.org/html/2610.28975#Thmlemma18),Fb​\(𝒖,𝒖\)=𝔼𝒙​\[σ⁡\(𝒖⊤​𝒙\)​xb\]=𝔼𝒙​\[σ′​\(𝒖⊤​𝒙\)\]​ub=12​ubF\_\{b\}\(\\bm\{u\},\\bm\{u\}\)=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)x\_\{b\}\]=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\]u\_\{b\}=\\tfrac\{1\}\{2\}u\_\{b\}\. For𝒖≠𝒗\\bm\{u\}\\neq\\bm\{v\},f⁡\(𝒙\)=σ′​\(𝒖⊤​𝒙\)​σ​\(𝒗⊤​𝒙\)f\(\\bm\{x\}\)=\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)is discontinuous across\{𝒖⊤𝒙=0\}\\\{\\bm\{u\}^\{\\top\}\\bm\{x\}=0\\\}, so we differentiate the Cho–Saul kernel instead\. WithK⁡\(𝒖,𝒗\):=𝔼𝒙​\[σ⁡\(𝒖⊤​𝒙\)​σ​\(𝒗⊤​𝒙\)\]K\(\\bm\{u\},\\bm\{v\}\):=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)\\sigma\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)\], dominated convergence \(sinceσ\\sigmais Lipschitz with linear growth\) gives

∂K∂ub​\(𝒖,𝒗\)=𝔼𝒙​\[σ′​\(𝒖⊤​𝒙\)​xb​σ​\(𝒗⊤​𝒙\)\]=Fb​\(𝒖,𝒗\)\.\\frac\{\\partial K\}\{\\partial u\_\{b\}\}\(\\bm\{u\},\\bm\{v\}\)=\\mathbb\{E\}\_\{\\bm\{x\}\}\[\\sigma^\{\\prime\}\(\\bm\{u\}^\{\\top\}\\bm\{x\}\)x\_\{b\}\\sigma\(\\bm\{v\}^\{\\top\}\\bm\{x\}\)\]=F\_\{b\}\(\\bm\{u\},\\bm\{v\}\)\.By[Lemma19](https://arxiv.org/html/2610.28975#Thmlemma19),K⁡\(𝒖,𝒗\)=‖𝒖‖​‖𝒗‖2​π​\(sin⁡θ\+\(π−θ\)​cos⁡θ\)K\(\\bm\{u\},\\bm\{v\}\)=\\tfrac\{\\left\\\|\\bm\{u\}\\right\\\|\\left\\\|\\bm\{v\}\\right\\\|\}\{2\\pi\}\(\\sin\\theta\+\(\\pi\-\\theta\)\\cos\\theta\), smooth for𝒖∉span⁡\(𝒗\)\\bm\{u\}\\notin\\mathrm\{span\}\(\\bm\{v\}\); differentiating with∂‖𝒖‖/∂ub=ub/‖𝒖‖\\partial\\left\\\|\\bm\{u\}\\right\\\|/\\partial u\_\{b\}=u\_\{b\}/\\left\\\|\\bm\{u\}\\right\\\|,∂cos⁡θ/∂ub=1‖𝒖‖​\(vb/‖𝒗‖−cos⁡θ​ub/‖𝒖‖\)\\partial\\cos\\theta/\\partial u\_\{b\}=\\tfrac\{1\}\{\\left\\\|\\bm\{u\}\\right\\\|\}\(v\_\{b\}/\\left\\\|\\bm\{v\}\\right\\\|\-\\cos\\theta\\,u\_\{b\}/\\left\\\|\\bm\{u\}\\right\\\|\),∂θ/∂ub=−1sin⁡θ∂cosθ/∂ub\\partial\\theta/\\partial u\_\{b\}=\-\\tfrac\{1\}\{\\sin\\theta\}\\partial\\cos\\theta/\\partial u\_\{b\}, and simplifying yields \([60](https://arxiv.org/html/2610.28975#A4.E60)\)\.

###### Lemma 26\.

𝔼𝑾2​\[\(𝑾2⊤​𝑾2\)a​k​\(𝑾2⊤​𝑾2\)c​l\]=d22​δa​k​δc​l\+d2​δa​c​δk​l\+d2​δa​l​δc​k\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\)\_\{ak\}\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\)\_\{cl\}\]=d\_\{2\}^\{2\}\\,\\delta\_\{ak\}\\delta\_\{cl\}\+d\_\{2\}\\,\\delta\_\{ac\}\\delta\_\{kl\}\+d\_\{2\}\\,\\delta\_\{al\}\\delta\_\{ck\}\.

Proof\.\(W2⊤​W2\)a​k=∑i=1d2\(W2\)i​a​\(W2\)i​k\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\)\_\{ak\}=\\sum\_\{i=1\}^\{d\_\{2\}\}\(\\bm\{W\}\_\{2\}\)\_\{ia\}\(\\bm\{W\}\_\{2\}\)\_\{ik\}, and applying[Lemma20](https://arxiv.org/html/2610.28975#Thmlemma20)to the four Gaussian factors\(𝑾2\)i​a\(\\bm\{W\}\_\{2\}\)\_\{ia\},\(𝑾2\)i​k\(\\bm\{W\}\_\{2\}\)\_\{ik\},\(𝑾2\)j​c\(\\bm\{W\}\_\{2\}\)\_\{jc\},\(𝑾2\)j​l\(\\bm\{W\}\_\{2\}\)\_\{jl\}\(summed overi,ji,j\) the three pairings give

δa​k​δc​l​d22\+δa​c​δk​l​d2\+δa​l​δc​k​d2\.\\delta\_\{ak\}\\delta\_\{cl\}d\_\{2\}^\{2\}\+\\delta\_\{ac\}\\delta\_\{kl\}d\_\{2\}\+\\delta\_\{al\}\\delta\_\{ck\}d\_\{2\}\.
Proof of \([50](https://arxiv.org/html/2610.28975#A4.E50)\)–\([51](https://arxiv.org/html/2610.28975#A4.E51)\)\.Permutation invariance follows from the exchangeability of the rows of𝑾1\\bm\{W\}\_\{1\}and columns of𝑾2\\bm\{W\}\_\{2\}\. The𝑮1\(2\)\\bm\{G\}\_\{1\}^\{\(2\)\}contribution to \([59](https://arxiv.org/html/2610.28975#A4.E59)\) is‖𝚽‖F24​𝑰d1\\tfrac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\}\\bm\{I\}\_\{d\_\{1\}\}by[Lemma24](https://arxiv.org/html/2610.28975#Thmlemma24)\. For𝑮1\(1\)\\bm\{G\}\_\{1\}^\{\(1\)\}, we have\(𝑮1\(1\)\)a​b=∑k=1d1\(𝑾2⊤​𝑾2\)a​k​Fb​\(𝒘a,𝒘k\)\(\\bm\{G\}\_\{1\}^\{\(1\)\}\)\_\{ab\}=\\sum\_\{k=1\}^\{d\_\{1\}\}\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\)\_\{ak\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{k\}\), so

𝔼𝑾2​\[\(𝑮1\(1\)​\(𝑮1\(1\)\)⊤\)a​c\|𝑾1\]=∑b,k,l𝔼𝑾2​\[\(𝑾2⊤​𝑾2\)a​k​\(𝑾2⊤​𝑾2\)c​l\]​Fb​\(𝒘a,𝒘k\)​Fb​\(𝒘c,𝒘l\),\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\\bigl\[\(\\bm\{G\}\_\{1\}^\{\(1\)\}\(\\bm\{G\}\_\{1\}^\{\(1\)\}\)^\{\\top\}\)\_\{ac\}\\bigm\|\\bm\{W\}\_\{1\}\\bigr\]=\\sum\_\{b,k,l\}\\mathbb\{E\}\_\{\\bm\{W\}\_\{2\}\}\[\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\)\_\{ak\}\(\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{W\}\_\{2\}\)\_\{cl\}\]\\,F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{k\}\)F\_\{b\}\(\\bm\{w\}\_\{c\},\\bm\{w\}\_\{l\}\),and substituting[Lemma26](https://arxiv.org/html/2610.28975#Thmlemma26)givesT1\(a​c\)\+T2\(a​c\)\+T3\(a​c\)T\_\{1\}^\{\(ac\)\}\+T\_\{2\}^\{\(ac\)\}\+T\_\{3\}^\{\(ac\)\}with

T1\(a​c\)\\displaystyle T\_\{1\}^\{\(ac\)\}=d22​∑bFb​\(𝒘a,𝒘a\)​Fb​\(𝒘c,𝒘c\),\\displaystyle=d\_\{2\}^\{2\}\\textstyle\\sum\_\{b\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{a\}\)F\_\{b\}\(\\bm\{w\}\_\{c\},\\bm\{w\}\_\{c\}\),\(61\)T2\(a​c\)\\displaystyle T\_\{2\}^\{\(ac\)\}=d2​δa​c​∑k=1d1∑bFb​\(𝒘a,𝒘k\)2,\\displaystyle=d\_\{2\}\\,\\delta\_\{ac\}\\textstyle\\sum\_\{k=1\}^\{d\_\{1\}\}\\sum\_\{b\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{k\}\)^\{2\},\(62\)T3\(a​c\)\\displaystyle T\_\{3\}^\{\(ac\)\}=d2​∑bFb​\(𝒘a,𝒘c\)​Fb​\(𝒘c,𝒘a\)\.\\displaystyle=d\_\{2\}\\textstyle\\sum\_\{b\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{c\}\)F\_\{b\}\(\\bm\{w\}\_\{c\},\\bm\{w\}\_\{a\}\)\.\(63\)
*TermT1T\_\{1\}\.*By[Lemma25](https://arxiv.org/html/2610.28975#Thmlemma25),Fb​\(𝒘a,𝒘a\)=12​\(𝒘a\)bF\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{a\}\)=\\tfrac\{1\}\{2\}\(\\bm\{w\}\_\{a\}\)\_\{b\}, soT1\(a​c\)=d224​𝒘a⊤​𝒘cT\_\{1\}^\{\(ac\)\}=\\tfrac\{d\_\{2\}^\{2\}\}\{4\}\\bm\{w\}\_\{a\}^\{\\top\}\\bm\{w\}\_\{c\}, giving𝔼⁡\[T1\(a​a\)\]=d0​d224\\mathbb\{E\}\[T\_\{1\}^\{\(aa\)\}\]=\\tfrac\{d\_\{0\}d\_\{2\}^\{2\}\}\{4\}and𝔼⁡\[T1\(a​c\)\]=0\\mathbb\{E\}\[T\_\{1\}^\{\(ac\)\}\]=0fora≠ca\\neq c\.

*TermT2T\_\{2\}\.*For𝒖≠𝒗\\bm\{u\}\\neq\\bm\{v\}, letA:=‖𝒗‖​sin⁡θ2​π​‖𝒖‖A:=\\tfrac\{\\left\\\|\\bm\{v\}\\right\\\|\\sin\\theta\}\{2\\pi\\left\\\|\\bm\{u\}\\right\\\|\}andB:=π−θ2​πB:=\\tfrac\{\\pi\-\\theta\}\{2\\pi\}in \([60](https://arxiv.org/html/2610.28975#A4.E60)\); using𝒖⊤​𝒗=‖𝒖‖​‖𝒗‖​cos⁡θ\\bm\{u\}^\{\\top\}\\bm\{v\}=\\left\\\|\\bm\{u\}\\right\\\|\\left\\\|\\bm\{v\}\\right\\\|\\cos\\theta,

∑bFb​\(𝒖,𝒗\)2=A2​‖𝒖‖2\+2​A​B​\(𝒖⊤​𝒗\)\+B2​‖𝒗‖2=‖𝒗‖2​ψ​\(θ\),\\displaystyle\\sum\_\{b\}F\_\{b\}\(\\bm\{u\},\\bm\{v\}\)^\{2\}=A^\{2\}\\left\\\|\\bm\{u\}\\right\\\|^\{2\}\+2AB\(\\bm\{u\}^\{\\top\}\\bm\{v\}\)\+B^\{2\}\\left\\\|\\bm\{v\}\\right\\\|^\{2\}=\\left\\\|\\bm\{v\}\\right\\\|^\{2\}\\,\\psi\(\\theta\),ψ⁡\(θ\):=sin2⁡θ\+\(π−θ\)24​π2\+sin⁡θ​cos⁡θ​\(π−θ\)2​π2\.\\displaystyle\\psi\(\\theta\):=\\frac\{\\sin^\{2\}\\theta\+\(\\pi\-\\theta\)^\{2\}\}\{4\\pi^\{2\}\}\+\\frac\{\\sin\\theta\\cos\\theta\(\\pi\-\\theta\)\}\{2\\pi^\{2\}\}\.InT2\(a​a\)T\_\{2\}^\{\(aa\)\}the termk=ak=acontributes∑bFb​\(𝒘a,𝒘a\)2=14​‖𝒘a‖2\\sum\_\{b\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{a\}\)^\{2\}=\\tfrac\{1\}\{4\}\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}\(expectationd04\\tfrac\{d\_\{0\}\}\{4\}\), and eachk≠ak\\neq acontributes𝔼⁡\[‖𝒘k‖2\]​𝔼​\[ψ⁡\(θa​k\)\]=d0​κ1\\mathbb\{E\}\[\\left\\\|\\bm\{w\}\_\{k\}\\right\\\|^\{2\}\]\\mathbb\{E\}\[\\psi\(\\theta\_\{ak\}\)\]=d\_\{0\}\\kappa\_\{1\}, so

𝔼⁡\[T2\(a​a\)\]=d2​\[d04\+\(d1−1\)​d0​κ1\]=d0​d2​\[\(d1−1\)​κ1\+14\],\\mathbb\{E\}\[T\_\{2\}^\{\(aa\)\}\]=d\_\{2\}\\Bigl\[\\tfrac\{d\_\{0\}\}\{4\}\+\(d\_\{1\}\-1\)d\_\{0\}\\kappa\_\{1\}\\Bigr\]=d\_\{0\}d\_\{2\}\\bigl\[\(d\_\{1\}\-1\)\\kappa\_\{1\}\+\\tfrac\{1\}\{4\}\\bigr\],andT2\(a​c\)=0T\_\{2\}^\{\(ac\)\}=0fora≠ca\\neq c\.

*TermT3T\_\{3\}\.*Fora=ca=c, the sum∑bFb​\(𝒘a,𝒘a\)2\\sum\_\{b\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{a\}\)^\{2\}equals14​‖𝒘a‖2\\tfrac\{1\}\{4\}\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|^\{2\}, so𝔼⁡\[T3\(a​a\)\]=d0​d24\\mathbb\{E\}\[T\_\{3\}^\{\(aa\)\}\]=\\tfrac\{d\_\{0\}d\_\{2\}\}\{4\}\. Fora≠ca\\neq c, expandingFb​\(𝒘a,𝒘c\)​Fb​\(𝒘c,𝒘a\)F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{c\}\)F\_\{b\}\(\\bm\{w\}\_\{c\},\\bm\{w\}\_\{a\}\)via \([60](https://arxiv.org/html/2610.28975#A4.E60)\) \(withθ:=θa​c\\theta:=\\theta\_\{ac\}\) and summing overbb,

∑bFb​\(𝒘a,𝒘c\)​Fb​\(𝒘c,𝒘a\)=‖𝒘a‖​‖𝒘c‖4​π2​\[cos⁡θ⁡\(sin2⁡θ\+\(π−θ\)2\)\+2​sin⁡θ⁡\(π−θ\)\]\.\\sum\_\{b\}F\_\{b\}\(\\bm\{w\}\_\{a\},\\bm\{w\}\_\{c\}\)F\_\{b\}\(\\bm\{w\}\_\{c\},\\bm\{w\}\_\{a\}\)=\\frac\{\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|\\left\\\|\\bm\{w\}\_\{c\}\\right\\\|\}\{4\\pi^\{2\}\}\\Bigl\[\\cos\\theta\\bigl\(\\sin^\{2\}\\theta\+\(\\pi\-\\theta\)^\{2\}\\bigr\)\+2\\sin\\theta\(\\pi\-\\theta\)\\Bigr\]\.Here‖𝒘a‖​‖𝒘c‖\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|\\left\\\|\\bm\{w\}\_\{c\}\\right\\\|is independent ofθa​c\\theta\_\{ac\}with𝔼⁡\[‖𝒘a‖​‖𝒘c‖\]=d0\+O⁡\(1\)\\mathbb\{E\}\[\\left\\\|\\bm\{w\}\_\{a\}\\right\\\|\\left\\\|\\bm\{w\}\_\{c\}\\right\\\|\]=d\_\{0\}\+O\(1\); asd0→∞d\_\{0\}\\to\\infty,θa​c\\theta\_\{ac\}concentrates atπ/2\\pi/2with𝔼⁡\[cos⁡θa​c\]=0\\mathbb\{E\}\[\\cos\\theta\_\{ac\}\]=0andVar⁡\(cos⁡θa​c\)=1/d0\\Var\(\\cos\\theta\_\{ac\}\)=1/d\_\{0\}, so\|𝔼\[cosθa​cq\(θa​c\)\]\|=O\(1/d0\)\|\\mathbb\{E\}\[\\cos\\theta\_\{ac\}\\,q\(\\theta\_\{ac\}\)\]\|=O\(1/\\sqrt\{d\_\{0\}\}\)for boundedqq, while𝔼⁡\[sin⁡θa​c​\(π−θa​c\)\]=Θ⁡\(1\)\\mathbb\{E\}\[\\sin\\theta\_\{ac\}\(\\pi\-\\theta\_\{ac\}\)\]=\\Theta\(1\)\. Thecos⁡θ⁡\(⋅\)\\cos\\theta\(\\cdot\)part contributesO⁡\(d2​d0\)O\(d\_\{2\}\\sqrt\{d\_\{0\}\}\)and the2​sin⁡θ​\(π−θ\)2\\sin\\theta\(\\pi\-\\theta\)part givesd0​d2​κ2\+O⁡\(d2\)d\_\{0\}d\_\{2\}\\kappa\_\{2\}\+O\(d\_\{2\}\), so𝔼⁡\[T3\(a​c\)\]=d0​d2​κ2\+O⁡\(d2​d0\)\\mathbb\{E\}\[T\_\{3\}^\{\(ac\)\}\]=d\_\{0\}d\_\{2\}\\kappa\_\{2\}\+O\(d\_\{2\}\\sqrt\{d\_\{0\}\}\)\.

*Combining\.*Fora=ca=c, adding the three terms and the𝑮1\(2\)\\bm\{G\}\_\{1\}^\{\(2\)\}contribution and merging the twod0​d24\\tfrac\{d\_\{0\}d\_\{2\}\}\{4\}pieces,

𝔼⁡\[\(𝑮1​𝑮1⊤\)a​a\]=d0​d224\+d0​d2​\[\(d1−1\)​κ1\+12\]\+‖𝚽‖F24;\\mathbb\{E\}\\bigl\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{aa\}\\bigr\]=\\tfrac\{d\_\{0\}d\_\{2\}^\{2\}\}\{4\}\+d\_\{0\}d\_\{2\}\\bigl\[\(d\_\{1\}\-1\)\\kappa\_\{1\}\+\\tfrac\{1\}\{2\}\\bigr\]\+\\tfrac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\};fora≠ca\\neq conlyT3\(a​c\)T\_\{3\}^\{\(ac\)\}survives,𝔼⁡\[\(𝑮1​𝑮1⊤\)a​c\]=d0​d2​κ2\+O⁡\(d2​d0\)\\mathbb\{E\}\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{ac\}\]=d\_\{0\}d\_\{2\}\\kappa\_\{2\}\+O\(d\_\{2\}\\sqrt\{d\_\{0\}\}\)\. Finallyψ⁡\(π/2\)=1\+\(π/2\)24​π2=14​π2\+116\\psi\(\\pi/2\)=\\tfrac\{1\+\(\\pi/2\)^\{2\}\}\{4\\pi^\{2\}\}=\\tfrac\{1\}\{4\\pi^\{2\}\}\+\\tfrac\{1\}\{16\}andsin⁡\(π/2\)​\(π−π/2\)/\(2​π2\)=14​π\\sin\(\\pi/2\)\(\\pi\-\\pi/2\)/\(2\\pi^\{2\}\)=\\tfrac\{1\}\{4\\pi\}give the high\-dimensional limits ofκ1,κ2\\kappa\_\{1\},\\kappa\_\{2\}\.

### D\.7Discussion

At initialization the gradient of𝑾2\\bm\{W\}\_\{2\}is mostly target\-independent: theΘ⁡\(d12\)\\Theta\(d\_\{1\}^\{2\}\)leading order of𝔼⁡\[\(𝑮2​𝑮2⊤\)a​a\]\\mathbb\{E\}\[\(\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\)\_\{aa\}\]comes entirely fromtr⁡\(𝑯2\)\\tr\(\\bm\{H\}^\{2\}\), which depends only on𝑾1\\bm\{W\}\_\{1\}, while the target termd14​\(𝚽​𝚽⊤\)a​a\\tfrac\{d\_\{1\}\}\{4\}\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{aa\}is dominated unless‖𝚽‖F2\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}is exceptionally large\. The nonzeroΘ⁡\(1\)\\Theta\(1\)off\-diagonal of𝔼⁡\[𝑮1​𝑮1⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]comes fromT3T\_\{3\}and reflects ReLU’s lack of odd symmetry: in a deep linear network the symmetry𝑾1↦𝑫​𝑾1\\bm\{W\}\_\{1\}\\mapsto\\bm\{D\}\\bm\{W\}\_\{1\},𝑾2↦𝑾2​𝑫\\bm\{W\}\_\{2\}\\mapsto\\bm\{W\}\_\{2\}\\bm\{D\}for𝑫=diag⁡\(±1\)\\bm\{D\}=\\diagop\(\\pm 1\)preserves the loss and forces off\-diagonal expectations to zero \(cf\.[AppendixC](https://arxiv.org/html/2610.28975#A3)\), whereasσ⁡\(−z\)≠−σ⁡\(z\)\\sigma\(\-z\)\\neq\-\\sigma\(z\)breaks it\. The dominance is therefore asymptotic ind1d\_\{1\}rather than exact\.

## Appendix ESingle\-Layer Linear Attention

### E\.1Setup

LetT,d0,dk,dv,d2T,d\_\{0\},d\_\{k\},d\_\{v\},d\_\{2\}be positive integers\. Single\-layer linear attention with tokens𝑿∈ℝT×d0\\bm\{X\}\\in\\mathbb\{R\}^\{T\\times d\_\{0\}\}\(one per row\) and weights𝑾Q,𝑾K∈ℝdk×d0\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{0\}\},𝑾V∈ℝdv×d0\\bm\{W\}\_\{V\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{0\}\},𝑾O∈ℝd2×dv\\bm\{W\}\_\{O\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{v\}\}, and its loss against a deterministic linear target𝚽∈ℝd2×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{0\}\}, are

𝑨:=1dk​𝑿​𝑾Q⊤​𝑾K​𝑿⊤,f⁡\(𝑿\)=𝑨​𝑿​𝑾V⊤​𝑾O⊤,ℒ:=12​‖f⁡\(𝑿\)−𝑿​𝚽⊤‖F2\.\\bm\{A\}:=\\frac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\,\\bm\{X\}\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{K\}\\bm\{X\}^\{\\top\},\\qquad f\(\\bm\{X\}\)=\\bm\{A\}\\,\\bm\{X\}\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{W\}\_\{O\}^\{\\top\},\\qquad\\mathcal\{L\}:=\\tfrac\{1\}\{2\}\\,\\left\\\|f\(\\bm\{X\}\)\-\\bm\{X\}\\bm\{\\Phi\}^\{\\top\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(64\)Let

𝒁:=𝑿​𝑾V⊤,𝑹:=f⁡\(𝑿\)−𝑿​𝚽⊤,𝒀:=𝑿​𝚽⊤,𝑪:=𝑿⊤​𝑿,cp:=tr⁡\(𝑪p\),\\bm\{Z\}:=\\bm\{X\}\\bm\{W\}\_\{V\}^\{\\top\},\\qquad\\bm\{R\}:=f\(\\bm\{X\}\)\-\\bm\{X\}\\bm\{\\Phi\}^\{\\top\},\\qquad\\bm\{Y\}:=\\bm\{X\}\\bm\{\\Phi\}^\{\\top\},\\qquad\\bm\{C\}:=\\bm\{X\}^\{\\top\}\\bm\{X\},\\qquad c\_\{p\}:=\\tr\(\\bm\{C\}^\{p\}\),\(65\)withc0=tr⁡\(𝑪0\)=d0c\_\{0\}=\\tr\(\\bm\{C\}^\{0\}\)=d\_\{0\}, and define the three data moments

𝖠1:=𝔼\[c12c4\+c2c4\+c23\],𝖠2:=𝔼\[c12c22\+2c2c4\],𝖠3:=𝔼⁡\[c12​c4\+c12​c22\+3​c2​c4\+c23\]=𝖠1\+𝖠2,\\begin\{gathered\}\\mathsf\{A\}\_\{1\}:=\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\],\\qquad\\mathsf\{A\}\_\{2\}:=\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}2c\_\{2\}c\_\{4\}\],\\\\ \\mathsf\{A\}\_\{3\}:=\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}3c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\]=\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\},\\end\{gathered\}\(66\)which are explicit polynomials in\(T,d0\)\(T,d\_\{0\}\)\([Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33)\)\. The gradients𝑮∙:=∂ℒ/∂𝑾∙\\bm\{G\}\_\{\\bullet\}:=\\partial\\mathcal\{L\}/\\partial\\bm\{W\}\_\{\\bullet\},∙∈\{Q,K,V,O\}\\bullet\\in\\\{Q,K,V,O\\\}, are

𝑮Q\\displaystyle\\bm\{G\}\_\{Q\}=1dk​𝑾K​𝑿⊤​𝒁​𝑾O⊤​𝑹⊤​𝑿,\\displaystyle=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\,\\bm\{W\}\_\{K\}\\bm\{X\}^\{\\top\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{R\}^\{\\top\}\\bm\{X\},𝑮K\\displaystyle\\qquad\\bm\{G\}\_\{K\}=1dk​𝑾Q​𝑿⊤​𝑹​𝑾O​𝒁⊤​𝑿,\\displaystyle=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\,\\bm\{W\}\_\{Q\}\\bm\{X\}^\{\\top\}\\bm\{R\}\\bm\{W\}\_\{O\}\\bm\{Z\}^\{\\top\}\\bm\{X\},\(67\)𝑮V\\displaystyle\\bm\{G\}\_\{V\}=𝑾O⊤​𝑹⊤​𝑨​𝑿,\\displaystyle=\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{R\}^\{\\top\}\\bm\{A\}\\bm\{X\},𝑮O\\displaystyle\\qquad\\bm\{G\}\_\{O\}=𝑹⊤​𝑨​𝒁\.\\displaystyle=\\bm\{R\}^\{\\top\}\\bm\{A\}\\bm\{Z\}\.Muon acts per weight matrix, so the objects of study are the four gradient self outer\-products𝑮Q​𝑮Q⊤\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}and𝑮K​𝑮K⊤\\bm\{G\}\_\{K\}\\bm\{G\}\_\{K\}^\{\\top\}\(bothdk×dkd\_\{k\}\\times d\_\{k\}\),𝑮V​𝑮V⊤\\bm\{G\}\_\{V\}\\bm\{G\}\_\{V\}^\{\\top\}\(dv×dvd\_\{v\}\\times d\_\{v\}\) and𝑮O​𝑮O⊤\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\(d2×d2d\_\{2\}\\times d\_\{2\}\), as in[AppendicesB](https://arxiv.org/html/2610.28975#A2),[C](https://arxiv.org/html/2610.28975#A3)and[D](https://arxiv.org/html/2610.28975#A4)\.

### E\.2Standing assumption

###### Assumption 8\.

𝑾Q,𝑾K,𝑾V,𝑾O\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}are mutually independent with i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries; the rows of𝐗\\bm\{X\}are i\.i\.d\.𝒩⁡\(𝟎,𝐈d0\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\), independent of the weights, so𝐂∼𝒲d0​\(T,𝐈d0\)\\bm\{C\}\\sim\\mathcal\{W\}\_\{d\_\{0\}\}\(T,\\bm\{I\}\_\{d\_\{0\}\}\);𝚽\\bm\{\\Phi\}is deterministic\. Expectations𝔼⁡\[⋅\]\\mathbb\{E\}\[\\cdot\]are over data and weights jointly, and𝔼\[⋅\|𝐗\]\\mathbb\{E\}\[\\cdot\\,\|\\,\\bm\{X\}\]over the weights only\.

This is[Assumption4](https://arxiv.org/html/2610.28975#Thmassumption4)of the main text; fan\-in scaling is treated in[Remark6](https://arxiv.org/html/2610.28975#Thmremark6)\.

### E\.3Main results restated

Everydkd\_\{k\}dependence below enters through the single ratio𝖠2/𝖠1\\mathsf\{A\}\_\{2\}/\\mathsf\{A\}\_\{1\}of the moments \([66](https://arxiv.org/html/2610.28975#A5.E66)\)\.

Theorem \([Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4), restated\)\.Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8),𝔼⁡\[𝐆Q​𝐆Q⊤\]=cQ​𝐈dk\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\]=c\_\{Q\}\\bm\{I\}\_\{d\_\{k\}\},𝔼⁡\[𝐆K​𝐆K⊤\]=cK​𝐈dk\\mathbb\{E\}\[\\bm\{G\}\_\{K\}\\bm\{G\}\_\{K\}^\{\\top\}\]=c\_\{K\}\\bm\{I\}\_\{d\_\{k\}\}and𝔼⁡\[𝐆V​𝐆V⊤\]=cV​𝐈dv\\mathbb\{E\}\[\\bm\{G\}\_\{V\}\\bm\{G\}\_\{V\}^\{\\top\}\]=c\_\{V\}\\bm\{I\}\_\{d\_\{v\}\}with

cQ\\displaystyle c\_\{Q\}=1dk​\(𝔼⁡\[c2​β1\]\+dv​γ​‖𝚽‖F2\)\+1dk2​\(𝔼⁡\[c2​β1\]\+𝔼⁡\[c2​β2\]\),\\displaystyle=\\frac\{1\}\{d\_\{k\}\}\\Bigl\(\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]\+d\_\{v\}\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\Bigr\)\+\\frac\{1\}\{d\_\{k\}^\{2\}\}\\Bigl\(\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]\+\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]\\Bigr\),\(68\)cK\\displaystyle c\_\{K\}=1dk​\(𝔼⁡\[c2​β1\]\+dv​γ​‖𝚽‖F2\)\+1dk2​𝔼​\[\(c12\+c2\)​β1\],\\displaystyle=\\frac\{1\}\{d\_\{k\}\}\\Bigl\(\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]\+d\_\{v\}\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\Bigr\)\+\\frac\{1\}\{d\_\{k\}^\{2\}\}\\,\\mathbb\{E\}\[\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\)\\beta\_\{1\}\],cV\\displaystyle c\_\{V\}=d2​\(d2\+dv\+1\)​\(𝖠1\+1dk​𝖠3\)\+γ​‖𝚽‖F2,\\displaystyle=d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\Bigl\(\\mathsf\{A\}\_\{1\}\+\\tfrac\{1\}\{d\_\{k\}\}\\mathsf\{A\}\_\{3\}\\Bigr\)\+\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},whereβ1,β2\\beta\_\{1\},\\beta\_\{2\}are the polynomials \([75](https://arxiv.org/html/2610.28975#A5.E75)\); and

𝔼⁡\[𝑮O​𝑮O⊤\]=α¯O​𝑰d2\+dv​γ​𝚽​𝚽⊤,\\mathbb\{E\}\\bigl\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\\bigr\]=\\bar\{\\alpha\}\_\{O\}\\,\\bm\{I\}\_\{d\_\{2\}\}\+d\_\{v\}\\,\\gamma\\;\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},\(69\)with

γ\\displaystyle\\gamma=1d0​𝔼​\[c22\]=T⁡\[\(T​d0\+8\)​\(T\+d0\)2\+2​\(T​d0\+10\)​\(T\+d0\)\+5​\(T​d0\+4\)\],\\displaystyle=\\tfrac\{1\}\{d\_\{0\}\}\\,\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=T\\Bigl\[\(Td\_\{0\}\{\+\}8\)\(T\{\+\}d\_\{0\}\)^\{2\}\+2\(Td\_\{0\}\{\+\}10\)\(T\{\+\}d\_\{0\}\)\+5\(Td\_\{0\}\{\+\}4\)\\Bigr\],\(70\)α¯O\\displaystyle\\bar\{\\alpha\}\_\{O\}=dv​\(dv\+1\)​𝖠1\+dv​𝖠2\+dv​\(dv\+1\)dk​𝖠3\+2​dvdk​𝖠1=Θ⁡\(dv2​d06\)\.\\displaystyle=d\_\{v\}\(d\_\{v\}\{\+\}1\)\\,\\mathsf\{A\}\_\{1\}\+d\_\{v\}\\,\\mathsf\{A\}\_\{2\}\+\\frac\{d\_\{v\}\(d\_\{v\}\{\+\}1\)\}\{d\_\{k\}\}\\,\\mathsf\{A\}\_\{3\}\+\\frac\{2d\_\{v\}\}\{d\_\{k\}\}\\,\\mathsf\{A\}\_\{1\}=\\Theta\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\.\(71\)
###### Corollary 3\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8)and for fixed𝚽\\bm\{\\Phi\},

cQ≍cK=Θ⁡\(d22​dv2​d06/dk\),cV=Θ⁡\(d2​\(d2\+dv\)​d06\),α¯O=Θ⁡\(dv2​d06\),c\_\{Q\}\\asymp c\_\{K\}=\\Theta\\bigl\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}/d\_\{k\}\\bigr\),\\qquad c\_\{V\}=\\Theta\\bigl\(d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\)\\,d\_\{0\}^\{6\}\\bigr\),\\qquad\\bar\{\\alpha\}\_\{O\}=\\Theta\\bigl\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\\bigr\),withcK−cQ=\(𝔼⁡\[c12​β1\]−𝔼⁡\[c2​β2\]\)/dk2c\_\{K\}\-c\_\{Q\}=\\bigl\(\\mathbb\{E\}\[c\_\{1\}^\{2\}\\beta\_\{1\}\]\-\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]\\bigr\)/d\_\{k\}^\{2\}of relative sizeO⁡\(1/dk\)O\(1/d\_\{k\}\); and fora≠ba\\neq bwith\(𝚽​𝚽⊤\)a​b≠0\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}\\neq 0,

𝔼⁡\[\(𝑮O​𝑮O⊤\)a​a\]𝔼⁡\[\(𝑮O​𝑮O⊤\)a​b\]=Θ⁡\(dv​d03\)\.\\frac\{\\mathbb\{E\}\[\(\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\)\_\{aa\}\]\}\{\\mathbb\{E\}\[\(\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\)\_\{ab\}\]\}=\\Theta\\bigl\(d\_\{v\}d\_\{0\}^\{3\}\\bigr\)\.Thus the hidden dimensiondvd\_\{v\}and the token dimensiond0d\_\{0\}drive the diagonal dominance of𝔼⁡\[𝐆O​𝐆O⊤\]\\mathbb\{E\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\], whiledkd\_\{k\}enterscVc\_\{V\},α¯O\\bar\{\\alpha\}\_\{O\}and the𝐖O\\bm\{W\}\_\{O\}ratio only through bounded factors\.

### E\.4Preliminary lemmas

[Lemmas29](https://arxiv.org/html/2610.28975#Thmlemma29),[30](https://arxiv.org/html/2610.28975#Thmlemma30),[31](https://arxiv.org/html/2610.28975#Thmlemma31)and[32](https://arxiv.org/html/2610.28975#Thmlemma32)are stated conditionally on𝑿\\bm\{X\}, i\.e\. with𝑪\\bm\{C\}fixed andcp=tr⁡\(𝑪p\)c\_\{p\}=\\tr\(\\bm\{C\}^\{p\}\);[Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33)supplies the moments of𝑪∼𝒲d0​\(T,𝑰d0\)\\bm\{C\}\\sim\\mathcal\{W\}\_\{d\_\{0\}\}\(T,\\bm\{I\}\_\{d\_\{0\}\}\)needed to average over𝑿\\bm\{X\}\.

###### Lemma 27\.

Same as[Lemma1](https://arxiv.org/html/2610.28975#Thmlemma1)\.

###### Lemma 28\(Wick/Isserlis\)\.

Same as[Lemma3](https://arxiv.org/html/2610.28975#Thmlemma3)\.

###### Lemma 29\.

Let𝐖∈ℝn×d\\bm\{W\}\\in\\mathbb\{R\}^\{n\\times d\}have i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries, let𝐒:=𝐖⊤​𝐖∼𝒲d​\(n,𝐈d\)\\bm\{S\}:=\\bm\{W\}^\{\\top\}\\bm\{W\}\\sim\\mathcal\{W\}\_\{d\}\(n,\\bm\{I\}\_\{d\}\), and let𝐌∈ℝd×d\\bm\{M\}\\in\\mathbb\{R\}^\{d\\times d\}be deterministic symmetric\. Then

𝔼⁡\[Sa​b​Sc​d\]=n2​δa​b​δc​d\+n⁡\(δa​c​δb​d\+δa​d​δb​c\),𝔼⁡\[𝑺​𝑴​𝑺\]=n⁡\(n\+1\)​𝑴\+n​tr⁡\(𝑴\)​𝑰d,\\mathbb\{E\}\[S\_\{ab\}S\_\{cd\}\]=n^\{2\}\\delta\_\{ab\}\\delta\_\{cd\}\+n\(\\delta\_\{ac\}\\delta\_\{bd\}\+\\delta\_\{ad\}\\delta\_\{bc\}\),\\qquad\\mathbb\{E\}\[\\bm\{S\}\\bm\{M\}\\bm\{S\}\]=n\(n\{\+\}1\)\\bm\{M\}\+n\\tr\(\\bm\{M\}\)\\,\\bm\{I\}\_\{d\},\(72\)hence𝔼⁡\[tr⁡\(𝐌​𝐒​𝐌​𝐒\)\]=n⁡\(n\+1\)​tr⁡\(𝐌2\)\+n​\(tr⁡𝐌\)2\\mathbb\{E\}\[\\tr\(\\bm\{M\}\\bm\{S\}\\bm\{M\}\\bm\{S\}\)\]=n\(n\{\+\}1\)\\tr\(\\bm\{M\}^\{2\}\)\+n\(\\tr\\bm\{M\}\)^\{2\}and𝔼⁡\[𝐒2\]=n⁡\(n\+d\+1\)​𝐈d\\mathbb\{E\}\[\\bm\{S\}^\{2\}\]=n\(n\{\+\}d\{\+\}1\)\\bm\{I\}\_\{d\}\.

Proof\.SinceSa​b=∑αWα​a​Wα​bS\_\{ab\}=\\sum\_\{\\alpha\}W\_\{\\alpha a\}W\_\{\\alpha b\},[Lemma28](https://arxiv.org/html/2610.28975#Thmlemma28)gives

𝔼⁡\[Sa​b​Sc​d\]=∑α,β𝔼⁡\[Wα​a​Wα​b​Wβ​c​Wβ​d\]=∑α,β\(δa​b​δc​d\+δα​β​δa​c​δb​d\+δα​β​δa​d​δb​c\),\\mathbb\{E\}\[S\_\{ab\}S\_\{cd\}\]=\\sum\_\{\\alpha,\\beta\}\\mathbb\{E\}\[W\_\{\\alpha a\}W\_\{\\alpha b\}W\_\{\\beta c\}W\_\{\\beta d\}\]=\\sum\_\{\\alpha,\\beta\}\\bigl\(\\delta\_\{ab\}\\delta\_\{cd\}\+\\delta\_\{\\alpha\\beta\}\\delta\_\{ac\}\\delta\_\{bd\}\+\\delta\_\{\\alpha\\beta\}\\delta\_\{ad\}\\delta\_\{bc\}\\bigr\),which is the first identity\. Contracting it againstMb​cM\_\{bc\}and using𝑴=𝑴⊤\\bm\{M\}=\\bm\{M\}^\{\\top\},

𝔼⁡\[\(𝑺​𝑴​𝑺\)a​d\]=∑b,cMb​c​𝔼​\[Sa​b​Sc​d\]=n2​Ma​d\+n​Ma​d\+n​δa​d​tr⁡\(𝑴\),\\mathbb\{E\}\[\(\\bm\{S\}\\bm\{M\}\\bm\{S\}\)\_\{ad\}\]=\\sum\_\{b,c\}M\_\{bc\}\\,\\mathbb\{E\}\[S\_\{ab\}S\_\{cd\}\]=n^\{2\}M\_\{ad\}\+nM\_\{ad\}\+n\\,\\delta\_\{ad\}\\tr\(\\bm\{M\}\),which is the second identity; contracting once more againstMd​aM\_\{da\}gives the trace identity, and𝑴=𝑰d\\bm\{M\}=\\bm\{I\}\_\{d\}gives𝔼⁡\[𝑺2\]\\mathbb\{E\}\[\\bm\{S\}^\{2\}\]\.

###### Lemma 30\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8), conditionally on𝐗\\bm\{X\}, let𝐍:=𝐖Q​𝐂​𝐖Q⊤∈ℝdk×dk\\bm\{N\}:=\\bm\{W\}\_\{Q\}\\bm\{C\}\\bm\{W\}\_\{Q\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{k\}\}\. Then

𝔼⁡\[Ni​j​Nk​l\|𝑿\]=c12​δi​j​δk​l\+c2​\(δi​k​δj​l\+δi​l​δj​k\),𝔼\[\(tr𝑵\)2\|𝑿\]=dk2c12\+2dkc2,𝔼\[tr𝑵2\|𝑿\]=dk2c2\+dk\(c12\+c2\)\.\\begin\{gathered\}\\mathbb\{E\}\[N\_\{ij\}N\_\{kl\}\\,\|\\,\\bm\{X\}\]=c\_\{1\}^\{2\}\\,\\delta\_\{ij\}\\delta\_\{kl\}\+c\_\{2\}\(\\delta\_\{ik\}\\delta\_\{jl\}\+\\delta\_\{il\}\\delta\_\{jk\}\),\\\\ \\mathbb\{E\}\[\(\\tr\\bm\{N\}\)^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{k\}^\{2\}c\_\{1\}^\{2\}\+2d\_\{k\}c\_\{2\},\\qquad\\mathbb\{E\}\[\\tr\\bm\{N\}^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{k\}^\{2\}c\_\{2\}\+d\_\{k\}\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\)\.\\end\{gathered\}\(73\)The same identities hold with𝐂\\bm\{C\}replaced by any deterministic symmetric𝐄∈ℝd0×d0\\bm\{E\}\\in\\mathbb\{R\}^\{d\_\{0\}\\times d\_\{0\}\}and\(c1,c2\)\(c\_\{1\},c\_\{2\}\)by\(tr⁡𝐄,tr⁡𝐄2\)\(\\tr\\bm\{E\},\\tr\\bm\{E\}^\{2\}\), and with𝐖Q\\bm\{W\}\_\{Q\}replaced by𝐖K\\bm\{W\}\_\{K\}\.

Proof\.SinceNi​j=∑a,b\(𝑾Q\)i​a​Ca​b​\(𝑾Q\)j​bN\_\{ij\}=\\sum\_\{a,b\}\(\\bm\{W\}\_\{Q\}\)\_\{ia\}C\_\{ab\}\(\\bm\{W\}\_\{Q\}\)\_\{jb\},[Lemma28](https://arxiv.org/html/2610.28975#Thmlemma28)gives

𝔼⁡\[\(𝑾Q\)i​a​\(𝑾Q\)j​b​\(𝑾Q\)k​c​\(𝑾Q\)l​d\]=δi​j​δa​b​δk​l​δc​d\+δi​k​δa​c​δj​l​δb​d\+δi​l​δa​d​δj​k​δb​c,\\mathbb\{E\}\[\(\\bm\{W\}\_\{Q\}\)\_\{ia\}\(\\bm\{W\}\_\{Q\}\)\_\{jb\}\(\\bm\{W\}\_\{Q\}\)\_\{kc\}\(\\bm\{W\}\_\{Q\}\)\_\{ld\}\]=\\delta\_\{ij\}\\delta\_\{ab\}\\delta\_\{kl\}\\delta\_\{cd\}\+\\delta\_\{ik\}\\delta\_\{ac\}\\delta\_\{jl\}\\delta\_\{bd\}\+\\delta\_\{il\}\\delta\_\{ad\}\\delta\_\{jk\}\\delta\_\{bc\},and contracting againstCa​b​Cc​dC\_\{ab\}C\_\{cd\}, with∑a,bCa​b​Ca​b=tr⁡\(𝑪2\)=c2\\sum\_\{a,b\}C\_\{ab\}C\_\{ab\}=\\tr\(\\bm\{C\}^\{2\}\)=c\_\{2\}, gives the first line of \([73](https://arxiv.org/html/2610.28975#A5.E73)\)\. Summing it overi=ji=jandk=lk=lgives𝔼⁡\[\(tr⁡𝑵\)2\|𝑿\]\\mathbb\{E\}\[\(\\tr\\bm\{N\}\)^\{2\}\\,\|\\,\\bm\{X\}\]; summing it overj=kj=kandl=il=igives𝔼⁡\[tr⁡𝑵2\|𝑿\]\\mathbb\{E\}\[\\tr\\bm\{N\}^\{2\}\\,\|\\,\\bm\{X\}\]\. Only the symmetry of𝑪\\bm\{C\}and the law of𝑾Q\\bm\{W\}\_\{Q\}were used, which gives the last assertion\. \(These are[Lemmas11](https://arxiv.org/html/2610.28975#Thmlemma11)and[10](https://arxiv.org/html/2610.28975#Thmlemma10)withm=dkm=d\_\{k\}and𝑴=𝑪\\bm\{M\}=\\bm\{C\}\.\)

###### Lemma 31\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8), conditionally on𝐗\\bm\{X\}, let𝐍∈ℝdk×dk\\bm\{N\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{k\}\}be symmetric and independent of𝐖K\\bm\{W\}\_\{K\}, and let𝐒:=𝐖K⊤​𝐍​𝐖K\\bm\{S\}:=\\bm\{W\}\_\{K\}^\{\\top\}\\bm\{N\}\\bm\{W\}\_\{K\}\. Then𝔼\[Sa​bSc​d\|𝐗,𝐍\]=\(tr𝐍\)2δa​bδc​d\+tr\(𝐍2\)\(δa​cδb​d\+δa​dδb​c\)\\mathbb\{E\}\[S\_\{ab\}S\_\{cd\}\\,\|\\,\\bm\{X\},\\bm\{N\}\]=\(\\tr\\bm\{N\}\)^\{2\}\\delta\_\{ab\}\\delta\_\{cd\}\+\\tr\(\\bm\{N\}^\{2\}\)\(\\delta\_\{ac\}\\delta\_\{bd\}\+\\delta\_\{ad\}\\delta\_\{bc\}\), and

𝔼\[tr\(𝑺𝑪2𝑺𝑪2\)\|𝑿,𝑵\]\\displaystyle\\mathbb\{E\}\[\\tr\(\\bm\{S\}\\bm\{C\}^\{2\}\\bm\{S\}\\bm\{C\}^\{2\}\)\\,\|\\,\\bm\{X\},\\bm\{N\}\]=\(tr⁡𝑵\)2​c4\+tr⁡\(𝑵2\)​\(c4\+c22\),\\displaystyle=\(\\tr\\bm\{N\}\)^\{2\}c\_\{4\}\+\\tr\(\\bm\{N\}^\{2\}\)\(c\_\{4\}\{\+\}c\_\{2\}^\{2\}\),\(74\)𝔼\[\(tr\(𝑺𝑪2\)\)2\|𝑿,𝑵\]\\displaystyle\\mathbb\{E\}\[\(\\tr\(\\bm\{S\}\\bm\{C\}^\{2\}\)\)^\{2\}\\,\|\\,\\bm\{X\},\\bm\{N\}\]=\(tr⁡𝑵\)2​c22\+2​tr⁡\(𝑵2\)​c4\.\\displaystyle=\(\\tr\\bm\{N\}\)^\{2\}c\_\{2\}^\{2\}\+2\\tr\(\\bm\{N\}^\{2\}\)\\,c\_\{4\}\.

Proof\.SinceSa​b=∑i,j\(𝑾K\)i​a​Ni​j​\(𝑾K\)j​bS\_\{ab\}=\\sum\_\{i,j\}\(\\bm\{W\}\_\{K\}\)\_\{ia\}N\_\{ij\}\(\\bm\{W\}\_\{K\}\)\_\{jb\},[Lemma28](https://arxiv.org/html/2610.28975#Thmlemma28)applied to𝔼⁡\[\(𝑾K\)i​a​\(𝑾K\)j​b​\(𝑾K\)k​c​\(𝑾K\)l​d\]\\mathbb\{E\}\[\(\\bm\{W\}\_\{K\}\)\_\{ia\}\(\\bm\{W\}\_\{K\}\)\_\{jb\}\(\\bm\{W\}\_\{K\}\)\_\{kc\}\(\\bm\{W\}\_\{K\}\)\_\{ld\}\]as in the proof of[Lemma30](https://arxiv.org/html/2610.28975#Thmlemma30), followed by contraction againstNi​j​Nk​lN\_\{ij\}N\_\{kl\}with𝑵=𝑵⊤\\bm\{N\}=\\bm\{N\}^\{\\top\}, gives the first identity\. Contracting the first identity against\(𝑪2\)b​c​\(𝑪2\)d​a\(\\bm\{C\}^\{2\}\)\_\{bc\}\(\\bm\{C\}^\{2\}\)\_\{da\}, the three terms give\(tr⁡𝑵\)2​tr⁡\(𝑪4\)\(\\tr\\bm\{N\}\)^\{2\}\\tr\(\\bm\{C\}^\{4\}\),tr⁡\(𝑵2\)​tr⁡\(𝑪4\)\\tr\(\\bm\{N\}^\{2\}\)\\tr\(\\bm\{C\}^\{4\}\)andtr⁡\(𝑵2\)​tr⁡\(𝑪2\)2\\tr\(\\bm\{N\}^\{2\}\)\\tr\(\\bm\{C\}^\{2\}\)^\{2\}, which is the first line of \([74](https://arxiv.org/html/2610.28975#A5.E74)\); contracting against\(𝑪2\)b​a​\(𝑪2\)d​c\(\\bm\{C\}^\{2\}\)\_\{ba\}\(\\bm\{C\}^\{2\}\)\_\{dc\}instead, they give\(tr⁡𝑵\)2​tr⁡\(𝑪2\)2\(\\tr\\bm\{N\}\)^\{2\}\\tr\(\\bm\{C\}^\{2\}\)^\{2\},tr⁡\(𝑵2\)​tr⁡\(𝑪4\)\\tr\(\\bm\{N\}^\{2\}\)\\tr\(\\bm\{C\}^\{4\}\)andtr⁡\(𝑵2\)​tr⁡\(𝑪4\)\\tr\(\\bm\{N\}^\{2\}\)\\tr\(\\bm\{C\}^\{4\}\), which is the second line\.

###### Lemma 32\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8), conditionally on𝐗\\bm\{X\}, let𝐇O:=𝐖O⊤​𝐖O∼𝒲dv​\(d2,𝐈dv\)\\bm\{H\}\_\{O\}:=\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{W\}\_\{O\}\\sim\\mathcal\{W\}\_\{d\_\{v\}\}\(d\_\{2\},\\bm\{I\}\_\{d\_\{v\}\}\)and𝐄:=𝐂​𝐖V⊤​𝐇O​𝐖V​𝐂\\bm\{E\}:=\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{H\}\_\{O\}\\bm\{W\}\_\{V\}\\bm\{C\}, and define

β1\\displaystyle\\beta\_\{1\}:=d2​dv​\[\(d2​dv\+d2\+dv\+3\)​c4\+\(d2\+dv\+1\)​c22\],\\displaystyle:=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}d\_\{2\}\{\+\}d\_\{v\}\{\+\}3\)\\,c\_\{4\}\+\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\,c\_\{2\}^\{2\}\\Bigr\],\(75\)β2\\displaystyle\\beta\_\{2\}:=d2​dv​\[\(d2​dv\+2\)​c22\+2​\(d2\+dv\+1\)​c4\]\.\\displaystyle:=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}2\)\\,c\_\{2\}^\{2\}\+2\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\,c\_\{4\}\\Bigr\]\.Then𝔼⁡\[tr⁡\(𝐄2\)\|𝐗\]=β1\\mathbb\{E\}\[\\tr\(\\bm\{E\}^\{2\}\)\\,\|\\,\\bm\{X\}\]=\\beta\_\{1\}and𝔼⁡\[\(tr⁡𝐄\)2\|𝐗\]=β2\\mathbb\{E\}\[\(\\tr\\bm\{E\}\)^\{2\}\\,\|\\,\\bm\{X\}\]=\\beta\_\{2\}\.

Proof\.Let𝑺V:=𝑾V⊤​𝑯O​𝑾V\\bm\{S\}\_\{V\}:=\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{H\}\_\{O\}\\bm\{W\}\_\{V\}, so thattr⁡𝑬=tr⁡\(𝑺V​𝑪2\)\\tr\\bm\{E\}=\\tr\(\\bm\{S\}\_\{V\}\\bm\{C\}^\{2\}\)andtr⁡𝑬2=tr⁡\(𝑺V​𝑪2​𝑺V​𝑪2\)\\tr\\bm\{E\}^\{2\}=\\tr\(\\bm\{S\}\_\{V\}\\bm\{C\}^\{2\}\\bm\{S\}\_\{V\}\\bm\{C\}^\{2\}\)\. Since𝑯O\\bm\{H\}\_\{O\}is symmetric and independent of𝑾V\\bm\{W\}\_\{V\},[Lemma31](https://arxiv.org/html/2610.28975#Thmlemma31)applies with\(𝑾V,𝑯O\)\(\\bm\{W\}\_\{V\},\\bm\{H\}\_\{O\}\)in place of\(𝑾K,𝑵\)\(\\bm\{W\}\_\{K\},\\bm\{N\}\):

𝔼\[tr𝑬2\|𝑿,𝑯O\]=\(tr𝑯O\)2c4\+tr\(𝑯O2\)\(c4\+c22\),𝔼\[\(tr𝑬\)2\|𝑿,𝑯O\]=\(tr𝑯O\)2c22\+2tr\(𝑯O2\)c4\.\\begin\{gathered\}\\mathbb\{E\}\[\\tr\\bm\{E\}^\{2\}\\,\|\\,\\bm\{X\},\\bm\{H\}\_\{O\}\]=\(\\tr\\bm\{H\}\_\{O\}\)^\{2\}c\_\{4\}\+\\tr\(\\bm\{H\}\_\{O\}^\{2\}\)\(c\_\{4\}\{\+\}c\_\{2\}^\{2\}\),\\\\ \\mathbb\{E\}\[\(\\tr\\bm\{E\}\)^\{2\}\\,\|\\,\\bm\{X\},\\bm\{H\}\_\{O\}\]=\(\\tr\\bm\{H\}\_\{O\}\)^\{2\}c\_\{2\}^\{2\}\+2\\tr\(\\bm\{H\}\_\{O\}^\{2\}\)\\,c\_\{4\}\.\\end\{gathered\}By[Lemma29](https://arxiv.org/html/2610.28975#Thmlemma29)with\(n,d\)=\(d2,dv\)\(n,d\)=\(d\_\{2\},d\_\{v\}\),𝔼⁡\[\(tr⁡𝑯O\)2\]=d2​dv​\(d2​dv\+2\)\\mathbb\{E\}\[\(\\tr\\bm\{H\}\_\{O\}\)^\{2\}\]=d\_\{2\}d\_\{v\}\(d\_\{2\}d\_\{v\}\{\+\}2\)and𝔼⁡\[tr⁡𝑯O2\]=d2​dv​\(d2\+dv\+1\)\\mathbb\{E\}\[\\tr\\bm\{H\}\_\{O\}^\{2\}\]=d\_\{2\}d\_\{v\}\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\); substituting givesβ1\\beta\_\{1\}andβ2\\beta\_\{2\}\.

###### Lemma 33\(Wishart trace moments\)\.

For𝐂∼𝒲d0​\(T,𝐈d0\)\\bm\{C\}\\sim\\mathcal\{W\}\_\{d\_\{0\}\}\(T,\\bm\{I\}\_\{d\_\{0\}\}\):𝔼⁡\[c1\]=T​d0\\mathbb\{E\}\[c\_\{1\}\]=Td\_\{0\},𝔼⁡\[c2\]=T​d0​\(T\+d0\+1\)\\mathbb\{E\}\[c\_\{2\}\]=Td\_\{0\}\(T\{\+\}d\_\{0\}\{\+\}1\), and

𝔼⁡\[c4\]\\displaystyle\\mathbb\{E\}\[c\_\{4\}\]=T​d0​\[\(T\+d0\)3\+6​\(T\+d0\)2\+3​\(T​d0\+7\)​\(T\+d0\)\+5​\(T​d0\+4\)\],\\displaystyle=Td\_\{0\}\\Bigl\[\(T\{\+\}d\_\{0\}\)^\{3\}\+6\(T\{\+\}d\_\{0\}\)^\{2\}\+3\(Td\_\{0\}\{\+\}7\)\(T\{\+\}d\_\{0\}\)\+5\(Td\_\{0\}\{\+\}4\)\\Bigr\],\(76\)𝔼⁡\[c22\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=T​d0​\[\(T​d0\+8\)​\(T\+d0\)2\+2​\(T​d0\+10\)​\(T\+d0\)\+5​\(T​d0\+4\)\],\\displaystyle=Td\_\{0\}\\Bigl\[\(Td\_\{0\}\{\+\}8\)\(T\{\+\}d\_\{0\}\)^\{2\}\+2\(Td\_\{0\}\{\+\}10\)\(T\{\+\}d\_\{0\}\)\+5\(Td\_\{0\}\{\+\}4\)\\Bigr\],\(77\)𝔼⁡\[c2​c4\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]=Td0\[\(Td0\+16\)\(T\+d0\)4\+\(7Td0\+136\)\(T\+d0\)3\\displaystyle=Td\_\{0\}\\Bigl\[\(Td\_\{0\}\{\+\}16\)\(T\{\+\}d\_\{0\}\)^\{4\}\+\(7Td\_\{0\}\{\+\}136\)\(T\{\+\}d\_\{0\}\)^\{3\}=Td\_0\[\+\(3​T2​d02\+99​T​d0\+736\)​\(T\+d0\)2\+\(8​T2​d02\+305​T​d0\+1608\)​\(T\+d0\)\\displaystyle\\hphantom\{=Td\_0\\Bigl\[\}\+\\bigl\(3T^\{2\}d\_\{0\}^\{2\}\{\+\}99Td\_\{0\}\{\+\}736\\bigr\)\(T\{\+\}d\_\{0\}\)^\{2\}\+\\bigl\(8T^\{2\}d\_\{0\}^\{2\}\{\+\}305Td\_\{0\}\{\+\}1608\\bigr\)\(T\{\+\}d\_\{0\}\)=Td\_0\[\+21T2d02\+420Td0\+1344\],\\displaystyle\\hphantom\{=Td\_0\\Bigl\[\}\+21T^\{2\}d\_\{0\}^\{2\}\{\+\}420Td\_\{0\}\{\+\}1344\\Bigr\],\(78\)𝔼⁡\[c23\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}^\{3\}\]=Td0\[\(T2d02\+24Td0\+160\)\(T\+d0\)3\+\(3T2d02\+84Td0\+704\)\(T\+d0\)2\\displaystyle=Td\_\{0\}\\Bigl\[\\bigl\(T^\{2\}d\_\{0\}^\{2\}\{\+\}24Td\_\{0\}\{\+\}160\\bigr\)\(T\{\+\}d\_\{0\}\)^\{3\}\+\\bigl\(3T^\{2\}d\_\{0\}^\{2\}\{\+\}84Td\_\{0\}\{\+\}704\\bigr\)\(T\{\+\}d\_\{0\}\)^\{2\}=Td\_0\[\+\(15T2d02\+344Td0\+1664\)\(T\+d0\)\+13T2d02\+380Td0\+1312\],\\displaystyle\\hphantom\{=Td\_0\\Bigl\[\}\+\\bigl\(15T^\{2\}d\_\{0\}^\{2\}\{\+\}344Td\_\{0\}\{\+\}1664\\bigr\)\(T\{\+\}d\_\{0\}\)\+13T^\{2\}d\_\{0\}^\{2\}\{\+\}380Td\_\{0\}\{\+\}1312\\Bigr\],\(79\)𝔼⁡\[c12​c4\]\\displaystyle\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\]=\(T​d0\+8\)​\(T​d0\+10\)​𝔼​\[c4\],𝔼⁡\[c12​c22\]=\(T​d0\+8\)​\(T​d0\+10\)​𝔼​\[c22\]\.\\displaystyle=\(Td\_\{0\}\{\+\}8\)\(Td\_\{0\}\{\+\}10\)\\,\\mathbb\{E\}\[c\_\{4\}\],\\qquad\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}^\{2\}\]=\(Td\_\{0\}\{\+\}8\)\(Td\_\{0\}\{\+\}10\)\\,\\mathbb\{E\}\[c\_\{2\}^\{2\}\]\.\(80\)Consequently𝖠3=𝖠1\+𝖠2\\mathsf\{A\}\_\{3\}=\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\}\. As polynomials ind0d\_\{0\}, all six moments have positive coefficients, of degree44for𝔼⁡\[c4\],𝔼⁡\[c22\]\\mathbb\{E\}\[c\_\{4\}\],\\mathbb\{E\}\[c\_\{2\}^\{2\}\]and degree66for the other four; hence𝔼⁡\[c4\],𝔼⁡\[c22\]=Θ⁡\(d04\)\\mathbb\{E\}\[c\_\{4\}\],\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=\\Theta\(d\_\{0\}^\{4\}\),𝔼⁡\[c2​c4\],𝔼⁡\[c23\],𝔼⁡\[c12​c4\],𝔼⁡\[c12​c22\]=Θ⁡\(d06\)\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\],\\mathbb\{E\}\[c\_\{2\}^\{3\}\],\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\],\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}^\{2\}\]=\\Theta\(d\_\{0\}^\{6\}\), and thereforeγ=1d0​𝔼​\[c22\]=Θ⁡\(d03\)\\gamma=\\tfrac\{1\}\{d\_\{0\}\}\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=\\Theta\(d\_\{0\}^\{3\}\)and𝖠1,𝖠2=Θ⁡\(d06\)\\mathsf\{A\}\_\{1\},\\mathsf\{A\}\_\{2\}=\\Theta\(d\_\{0\}^\{6\}\)\.

Proof\.Throughout the proof lets:=T\+d0s:=T\{\+\}d\_\{0\}andq:=T​d0q:=Td\_\{0\}\. Since𝑪=𝑿⊤​𝑿\\bm\{C\}=\\bm\{X\}^\{\\top\}\\bm\{X\}with𝑿∈ℝT×d0\\bm\{X\}\\in\\mathbb\{R\}^\{T\\times d\_\{0\}\}standard Gaussian, every moment below is the expectation of a polynomial in the entries of𝑿\\bm\{X\}, and Gaussian integration by parts,𝔼⁡\[Xt​a​Ψ​\(𝑿\)\]=𝔼⁡\[∂Ψ/∂Xt​a\]\\mathbb\{E\}\[X\_\{ta\}\\Psi\(\\bm\{X\}\)\]=\\mathbb\{E\}\[\\partial\\Psi/\\partial X\_\{ta\}\]for polynomialΨ\\Psi, applies\.

*Step 1: a recursion in the total degree\.*Fork≥1k\\geq 1andg:=∏i=1ncmig:=\\prod\_\{i=1\}^\{n\}c\_\{m\_\{i\}\}withmi≥1m\_\{i\}\\geq 1\(g=1g=1ifn=0n=0\),

𝔼⁡\[ck​g\]=𝔼⁡\[\(\(T\+k−1\)​ck−1\+∑j=1k−1cj​ck−1−j\)​g\]\+2​∑i=1nmi​𝔼​\[ck\+mi−1​∏l≠icml\]\.\\mathbb\{E\}\[c\_\{k\}g\]=\\mathbb\{E\}\\Bigl\[\\Bigl\(\(T\{\+\}k\{\-\}1\)\\,c\_\{k\-1\}\+\\sum\_\{j=1\}^\{k\-1\}c\_\{j\}\\,c\_\{k\-1\-j\}\\Bigr\)g\\Bigr\]\+2\\sum\_\{i=1\}^\{n\}m\_\{i\}\\,\\mathbb\{E\}\\Bigl\[c\_\{k\+m\_\{i\}\-1\}\\prod\_\{l\\neq i\}c\_\{m\_\{l\}\}\\Bigr\]\.\(81\)Indeed,ck=tr⁡\(𝑿⊤​𝑿​𝑪k−1\)=∑t,aXt​a​\(𝑿​𝑪k−1\)t​ac\_\{k\}=\\tr\(\\bm\{X\}^\{\\top\}\\bm\{X\}\\bm\{C\}^\{k\-1\}\)=\\sum\_\{t,a\}X\_\{ta\}\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}, so integration by parts in eachXt​aX\_\{ta\}gives

𝔼⁡\[ck​g\]=∑t,a𝔼⁡\[∂\(𝑿​𝑪k−1\)t​a∂Xt​a​g\]\+∑t,a𝔼⁡\[\(𝑿​𝑪k−1\)t​a​∂g∂Xt​a\]\.\\mathbb\{E\}\[c\_\{k\}g\]=\\sum\_\{t,a\}\\mathbb\{E\}\\Bigl\[\\frac\{\\partial\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}\}\{\\partial X\_\{ta\}\}\\,g\\Bigr\]\+\\sum\_\{t,a\}\\mathbb\{E\}\\Bigl\[\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}\\,\\frac\{\\partial g\}\{\\partial X\_\{ta\}\}\\Bigr\]\.\(82\)From∂Cb​c/∂Xt​a=δa​b​Xt​c\+δa​c​Xt​b\\partial C\_\{bc\}/\\partial X\_\{ta\}=\\delta\_\{ab\}X\_\{tc\}\+\\delta\_\{ac\}X\_\{tb\}and the product rule, form≥1m\\geq 1,

∂\(𝑪m\)b​c∂Xt​a=∑r=0m−1\[\(𝑪r\)b​a​\(𝑿​𝑪m−1−r\)t​c\+\(𝑪r​𝑿⊤\)b​t​\(𝑪m−1−r\)a​c\],∂cm∂Xt​a=∑b∂\(𝑪m\)b​b∂Xt​a=2​m​\(𝑿​𝑪m−1\)t​a\.\\begin\{gathered\}\\frac\{\\partial\(\\bm\{C\}^\{m\}\)\_\{bc\}\}\{\\partial X\_\{ta\}\}=\\sum\_\{r=0\}^\{m\-1\}\\Bigl\[\(\\bm\{C\}^\{r\}\)\_\{ba\}\\,\(\\bm\{X\}\\bm\{C\}^\{m\-1\-r\}\)\_\{tc\}\+\(\\bm\{C\}^\{r\}\\bm\{X\}^\{\\top\}\)\_\{bt\}\\,\(\\bm\{C\}^\{m\-1\-r\}\)\_\{ac\}\\Bigr\],\\\\ \\frac\{\\partial c\_\{m\}\}\{\\partial X\_\{ta\}\}=\\sum\_\{b\}\\frac\{\\partial\(\\bm\{C\}^\{m\}\)\_\{bb\}\}\{\\partial X\_\{ta\}\}=2m\\,\(\\bm\{X\}\\bm\{C\}^\{m\-1\}\)\_\{ta\}\.\\end\{gathered\}\(83\)Since\(𝑿​𝑪k−1\)t​a=∑bXt​b​\(𝑪k−1\)b​a\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}=\\sum\_\{b\}X\_\{tb\}\(\\bm\{C\}^\{k\-1\}\)\_\{ba\}, \([83](https://arxiv.org/html/2610.28975#A5.E83)\) withm=k−1m=k\-1gives

∑t,a∂\(𝑿​𝑪k−1\)t​a∂Xt​a\\displaystyle\\sum\_\{t,a\}\\frac\{\\partial\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}\}\{\\partial X\_\{ta\}\}=∑t,a\(𝑪k−1\)a​a\+∑r=0k−2\[∑t,a\(𝑿​𝑪r\)t​a​\(𝑿​𝑪k−2−r\)t​a\]\\displaystyle=\\sum\_\{t,a\}\(\\bm\{C\}^\{k\-1\}\)\_\{aa\}\+\\sum\_\{r=0\}^\{k\-2\}\\Bigl\[\\sum\_\{t,a\}\(\\bm\{X\}\\bm\{C\}^\{r\}\)\_\{ta\}\(\\bm\{X\}\\bm\{C\}^\{k\-2\-r\}\)\_\{ta\}\\Bigr\]\+∑r=0k−2tr\(𝑿𝑪r𝑿⊤\)tr\(𝑪k−2−r\)\\displaystyle\\qquad\+\\sum\_\{r=0\}^\{k\-2\}\\tr\(\\bm\{X\}\\bm\{C\}^\{r\}\\bm\{X\}^\{\\top\}\)\\,\\tr\(\\bm\{C\}^\{k\-2\-r\}\)=T​ck−1\+\(k−1\)​ck−1\+∑r=0k−2cr\+1​ck−2−r=\(T\+k−1\)​ck−1\+∑j=1k−1cj​ck−1−j,\\displaystyle=T\\,c\_\{k\-1\}\+\(k\{\-\}1\)\\,c\_\{k\-1\}\+\\sum\_\{r=0\}^\{k\-2\}c\_\{r\+1\}\\,c\_\{k\-2\-r\}=\(T\{\+\}k\{\-\}1\)\\,c\_\{k\-1\}\+\\sum\_\{j=1\}^\{k\-1\}c\_\{j\}\\,c\_\{k\-1\-j\},since∑t,a\(𝑿​𝑪r\)t​a​\(𝑿​𝑪k−2−r\)t​a=tr⁡\(𝑪r​𝑿⊤​𝑿​𝑪k−2−r\)=ck−1\\sum\_\{t,a\}\(\\bm\{X\}\\bm\{C\}^\{r\}\)\_\{ta\}\(\\bm\{X\}\\bm\{C\}^\{k\-2\-r\}\)\_\{ta\}=\\tr\(\\bm\{C\}^\{r\}\\bm\{X\}^\{\\top\}\\bm\{X\}\\bm\{C\}^\{k\-2\-r\}\)=c\_\{k\-1\}andtr⁡\(𝑿​𝑪r​𝑿⊤\)=cr\+1\\tr\(\\bm\{X\}\\bm\{C\}^\{r\}\\bm\{X\}^\{\\top\}\)=c\_\{r\+1\}; this is the first term of \([81](https://arxiv.org/html/2610.28975#A5.E81)\)\. For the second,∂g/∂Xt​a=∑i\(∂cmi/∂Xt​a\)​∏l≠icml\\partial g/\\partial X\_\{ta\}=\\sum\_\{i\}\(\\partial c\_\{m\_\{i\}\}/\\partial X\_\{ta\}\)\\prod\_\{l\\neq i\}c\_\{m\_\{l\}\}and, by \([83](https://arxiv.org/html/2610.28975#A5.E83)\),

∑t,a\(𝑿​𝑪k−1\)t​a​∂cm∂Xt​a\\displaystyle\\sum\_\{t,a\}\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}\\,\\frac\{\\partial c\_\{m\}\}\{\\partial X\_\{ta\}\}=2​m​∑t,a\(𝑿​𝑪k−1\)t​a​\(𝑿​𝑪m−1\)t​a\\displaystyle=2m\\sum\_\{t,a\}\(\\bm\{X\}\\bm\{C\}^\{k\-1\}\)\_\{ta\}\(\\bm\{X\}\\bm\{C\}^\{m\-1\}\)\_\{ta\}=2​m​tr⁡\(𝑪k−1​𝑿⊤​𝑿​𝑪m−1\)=2​m​ck\+m−1\.\\displaystyle=2m\\,\\tr\(\\bm\{C\}^\{k\-1\}\\bm\{X\}^\{\\top\}\\bm\{X\}\\bm\{C\}^\{m\-1\}\)=2m\\,c\_\{k\+m\-1\}\.
*Step 2: moments containingc1c\_\{1\}, and \([80](https://arxiv.org/html/2610.28975#A5.E80)\)\.*Fork=1k=1the first bracket of \([81](https://arxiv.org/html/2610.28975#A5.E81)\) isT​c0=qTc\_\{0\}=qand eachcmic\_\{m\_\{i\}\}is reproduced, so

𝔼⁡\[c1​g\]=\(q\+2​deg⁡g\)​𝔼​\[g\],deg⁡g:=∑imi,\\mathbb\{E\}\[c\_\{1\}g\]=\\bigl\(q\+2\\deg g\\bigr\)\\mathbb\{E\}\[g\],\\qquad\\deg g:=\\textstyle\\sum\_\{i\}m\_\{i\},\(84\)and iterating,𝔼⁡\[c12​F\]=\(q\+2​d\+2\)​\(q\+2​d\)​𝔼​\[F\]\\mathbb\{E\}\[c\_\{1\}^\{2\}F\]=\(q\{\+\}2d\{\+\}2\)\(q\{\+\}2d\)\\,\\mathbb\{E\}\[F\]forFFhomogeneous of degreeddin𝑪\\bm\{C\};d=4d=4gives \([80](https://arxiv.org/html/2610.28975#A5.E80)\)\.

*Step 3: degree at most four\.*By \([81](https://arxiv.org/html/2610.28975#A5.E81)\) withg=1g=1andk=1,2,3k=1,2,3, and by \([84](https://arxiv.org/html/2610.28975#A5.E84)\),

𝔼⁡\[c1\]\\displaystyle\\mathbb\{E\}\[c\_\{1\}\]=q,𝔼⁡\[c12\]=\(q\+2\)​q,𝔼⁡\[c2\]=\(T\+1\+d0\)​𝔼​\[c1\]=q⁡\(s\+1\),\\displaystyle=q,\\qquad\\mathbb\{E\}\[c\_\{1\}^\{2\}\]=\(q\{\+\}2\)\\,q,\\qquad\\mathbb\{E\}\[c\_\{2\}\]=\(T\{\+\}1\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{1\}\]=q\(s\{\+\}1\),𝔼⁡\[c1​c2\]\\displaystyle\\mathbb\{E\}\[c\_\{1\}c\_\{2\}\]=\(q\+4\)​𝔼​\[c2\]=q⁡\(q\+4\)​\(s\+1\),\\displaystyle=\(q\{\+\}4\)\\mathbb\{E\}\[c\_\{2\}\]=q\(q\{\+\}4\)\(s\{\+\}1\),𝔼⁡\[c3\]\\displaystyle\\mathbb\{E\}\[c\_\{3\}\]=\(T\+2\+d0\)​𝔼​\[c2\]\+𝔼⁡\[c12\]=q⁡\[\(s\+1\)​\(s\+2\)\+q\+2\]=q⁡\[s2\+3​s\+q\+4\]\.\\displaystyle=\(T\{\+\}2\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{2\}\]\+\\mathbb\{E\}\[c\_\{1\}^\{2\}\]=q\\bigl\[\(s\{\+\}1\)\(s\{\+\}2\)\+q\+2\\bigr\]=q\\bigl\[s^\{2\}\+3s\+q\+4\\bigr\]\.Then \([81](https://arxiv.org/html/2610.28975#A5.E81)\) with\(k,g\)=\(4,1\)\(k,g\)=\(4,1\)and\(2,c2\)\(2,c\_\{2\}\)gives

𝔼⁡\[c4\]\\displaystyle\\mathbb\{E\}\[c\_\{4\}\]=\(T\+3\+d0\)​𝔼​\[c3\]\+2​𝔼​\[c1​c2\]\\displaystyle=\(T\{\+\}3\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{3\}\]\+2\\mathbb\{E\}\[c\_\{1\}c\_\{2\}\]=q⁡\[\(s\+3\)​\(s2\+3​s\+q\+4\)\+2​\(s\+1\)​\(q\+4\)\]=q⁡\[s3\+6​s2\+3​\(q\+7\)​s\+5​\(q\+4\)\],\\displaystyle=q\\bigl\[\(s\{\+\}3\)\(s^\{2\}\{\+\}3s\{\+\}q\{\+\}4\)\+2\(s\{\+\}1\)\(q\{\+\}4\)\\bigr\]=q\\bigl\[s^\{3\}\+6s^\{2\}\+3\(q\{\+\}7\)s\+5\(q\{\+\}4\)\\bigr\],𝔼⁡\[c22\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=\(T\+1\+d0\)​𝔼​\[c1​c2\]\+4​𝔼​\[c3\]\\displaystyle=\(T\{\+\}1\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{1\}c\_\{2\}\]\+4\\mathbb\{E\}\[c\_\{3\}\]=q⁡\[\(s\+1\)2​\(q\+4\)\+4​\(s2\+3​s\+q\+4\)\]=q⁡\[\(q\+8\)​s2\+2​\(q\+10\)​s\+5​\(q\+4\)\],\\displaystyle=q\\bigl\[\(s\{\+\}1\)^\{2\}\(q\{\+\}4\)\+4\(s^\{2\}\{\+\}3s\{\+\}q\{\+\}4\)\\bigr\]=q\\bigl\[\(q\{\+\}8\)s^\{2\}\+2\(q\{\+\}10\)s\+5\(q\{\+\}4\)\\bigr\],which are \([76](https://arxiv.org/html/2610.28975#A5.E76)\) and \([77](https://arxiv.org/html/2610.28975#A5.E77)\)\.

*Step 4: degree five\.*By \([81](https://arxiv.org/html/2610.28975#A5.E81)\) with\(k,g\)=\(5,1\)\(k,g\)=\(5,1\)and\(3,c2\)\(3,c\_\{2\}\), and by \([84](https://arxiv.org/html/2610.28975#A5.E84)\) for𝔼⁡\[c1​c3\]=\(q\+6\)​𝔼​\[c3\]\\mathbb\{E\}\[c\_\{1\}c\_\{3\}\]=\(q\{\+\}6\)\\mathbb\{E\}\[c\_\{3\}\]and𝔼⁡\[c12​c2\]=\(q\+4\)​\(q\+6\)​𝔼​\[c2\]\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}\]=\(q\{\+\}4\)\(q\{\+\}6\)\\mathbb\{E\}\[c\_\{2\}\],

𝔼⁡\[c5\]\\displaystyle\\mathbb\{E\}\[c\_\{5\}\]=\(T\+4\+d0\)​𝔼​\[c4\]\+2​𝔼​\[c1​c3\]\+𝔼⁡\[c22\]\\displaystyle=\(T\{\+\}4\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{4\}\]\+2\\mathbb\{E\}\[c\_\{1\}c\_\{3\}\]\+\\mathbb\{E\}\[c\_\{2\}^\{2\}\]=q⁡\[s4\+10​s3\+\(6​q\+65\)​s2\+5​\(5​q\+32\)​s\+\(q\+4\)​\(2​q\+37\)\],\\displaystyle=q\\bigl\[s^\{4\}\+10s^\{3\}\+\(6q\{\+\}65\)s^\{2\}\+5\(5q\{\+\}32\)s\+\(q\{\+\}4\)\(2q\{\+\}37\)\\bigr\],𝔼⁡\[c2​c3\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}c\_\{3\}\]=\(T\+2\+d0\)​𝔼​\[c22\]\+𝔼⁡\[c12​c2\]\+4​𝔼​\[c4\]\\displaystyle=\(T\{\+\}2\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{2\}^\{2\}\]\+\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}\]\+4\\mathbb\{E\}\[c\_\{4\}\]=q⁡\[\(q\+12\)​s3\+4​\(q\+15\)​s2\+\(q\+7\)​\(q\+24\)​s\+\(q\+4\)​\(q\+36\)\]\.\\displaystyle=q\\bigl\[\(q\{\+\}12\)s^\{3\}\+4\(q\{\+\}15\)s^\{2\}\+\(q\{\+\}7\)\(q\{\+\}24\)s\+\(q\{\+\}4\)\(q\{\+\}36\)\\bigr\]\.
*Step 5: degree six\.*By \([81](https://arxiv.org/html/2610.28975#A5.E81)\) with\(k,g\)=\(4,c2\)\(k,g\)=\(4,c\_\{2\}\)and\(2,c22\)\(2,c\_\{2\}^\{2\}\), and by \([84](https://arxiv.org/html/2610.28975#A5.E84)\) for𝔼⁡\[c1​c22\]=\(q\+8\)​𝔼​\[c22\]\\mathbb\{E\}\[c\_\{1\}c\_\{2\}^\{2\}\]=\(q\{\+\}8\)\\mathbb\{E\}\[c\_\{2\}^\{2\}\],

𝔼⁡\[c2​c4\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]=\(T\+3\+d0\)​𝔼​\[c2​c3\]\+2​𝔼​\[c1​c22\]\+4​𝔼​\[c5\]=\(s\+3\)​𝔼​\[c2​c3\]\+2​\(q\+8\)​𝔼​\[c22\]\+4​𝔼​\[c5\],\\displaystyle=\(T\{\+\}3\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{2\}c\_\{3\}\]\+2\\mathbb\{E\}\[c\_\{1\}c\_\{2\}^\{2\}\]\+4\\mathbb\{E\}\[c\_\{5\}\]=\(s\{\+\}3\)\\mathbb\{E\}\[c\_\{2\}c\_\{3\}\]\+2\(q\{\+\}8\)\\mathbb\{E\}\[c\_\{2\}^\{2\}\]\+4\\mathbb\{E\}\[c\_\{5\}\],𝔼⁡\[c23\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}^\{3\}\]=\(T\+1\+d0\)​𝔼​\[c1​c22\]\+8​𝔼​\[c2​c3\]=\(s\+1\)​\(q\+8\)​𝔼​\[c22\]\+8​𝔼​\[c2​c3\];\\displaystyle=\(T\{\+\}1\{\+\}d\_\{0\}\)\\mathbb\{E\}\[c\_\{1\}c\_\{2\}^\{2\}\]\+8\\mathbb\{E\}\[c\_\{2\}c\_\{3\}\]=\(s\{\+\}1\)\(q\{\+\}8\)\\mathbb\{E\}\[c\_\{2\}^\{2\}\]\+8\\mathbb\{E\}\[c\_\{2\}c\_\{3\}\];substituting Steps 3 and 4 and collecting powers ofssgives \([78](https://arxiv.org/html/2610.28975#A5.E78)\) and \([79](https://arxiv.org/html/2610.28975#A5.E79)\)\.

*Step 6: orders ind0d\_\{0\}\.*All coefficients above are positive, so each moment isΘ\\Thetaof its highest power ofd0d\_\{0\}, read off from the displayed formulas:d0⋅d03d\_\{0\}\\cdot d\_\{0\}^\{3\}for𝔼⁡\[c4\]\\mathbb\{E\}\[c\_\{4\}\],d0⋅d0⋅d02d\_\{0\}\\cdot d\_\{0\}\\cdot d\_\{0\}^\{2\}for𝔼⁡\[c22\]\\mathbb\{E\}\[c\_\{2\}^\{2\}\],d0⋅d0⋅d04d\_\{0\}\\cdot d\_\{0\}\\cdot d\_\{0\}^\{4\}for𝔼⁡\[c2​c4\]\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\],d0⋅d02⋅d03d\_\{0\}\\cdot d\_\{0\}^\{2\}\\cdot d\_\{0\}^\{3\}for𝔼⁡\[c23\]\\mathbb\{E\}\[c\_\{2\}^\{3\}\], andd02d\_\{0\}^\{2\}times𝔼⁡\[c4\]\\mathbb\{E\}\[c\_\{4\}\]and𝔼⁡\[c22\]\\mathbb\{E\}\[c\_\{2\}^\{2\}\]for \([80](https://arxiv.org/html/2610.28975#A5.E80)\)\. Since𝖠1\\mathsf\{A\}\_\{1\}and𝖠2\\mathsf\{A\}\_\{2\}are positive combinations of degree\-six moments by \([66](https://arxiv.org/html/2610.28975#A5.E66)\), they areΘ⁡\(d06\)\\Theta\(d\_\{0\}^\{6\}\);𝖠3=𝖠1\+𝖠2\\mathsf\{A\}\_\{3\}=\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\}holds by definition\.

### E\.5Proofs

The proof of[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)has three steps: the isotropy of𝔼⁡\[𝑮∙​𝑮∙⊤\|𝑿\]\\mathbb\{E\}\[\\bm\{G\}\_\{\\bullet\}\\bm\{G\}\_\{\\bullet\}^\{\\top\}\\,\|\\,\\bm\{X\}\]for∙∈\{Q,K,V\}\\bullet\\in\\\{Q,K,V\\\}\([Lemma34](https://arxiv.org/html/2610.28975#Thmlemma34)\); the conditional closed forms of𝔼⁡\[𝑮O​𝑮O⊤\|𝑿\]\\mathbb\{E\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\\,\|\\,\\bm\{X\}\]\([Lemma35](https://arxiv.org/html/2610.28975#Thmlemma35)\) and of the isotropic constants \([Lemma36](https://arxiv.org/html/2610.28975#Thmlemma36)\); and the average over𝑿\\bm\{X\}\.

###### Lemma 34\(Isotropy\)\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8), conditionally on𝐗\\bm\{X\},

𝔼\[𝑮Q𝑮Q⊤\|𝑿\]=cQ\(𝑪\)𝑰dk,𝔼\[𝑮K𝑮K⊤\|𝑿\]=cK\(𝑪\)𝑰dk,𝔼⁡\[𝑮V​𝑮V⊤\|𝑿\]=cV​\(𝑪\)​𝑰dv,\\begin\{gathered\}\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\\,\|\\,\\bm\{X\}\]=c\_\{Q\}\(\\bm\{C\}\)\\,\\bm\{I\}\_\{d\_\{k\}\},\\qquad\\mathbb\{E\}\[\\bm\{G\}\_\{K\}\\bm\{G\}\_\{K\}^\{\\top\}\\,\|\\,\\bm\{X\}\]=c\_\{K\}\(\\bm\{C\}\)\\,\\bm\{I\}\_\{d\_\{k\}\},\\\\ \\mathbb\{E\}\[\\bm\{G\}\_\{V\}\\bm\{G\}\_\{V\}^\{\\top\}\\,\|\\,\\bm\{X\}\]=c\_\{V\}\(\\bm\{C\}\)\\,\\bm\{I\}\_\{d\_\{v\}\},\\end\{gathered\}withcQ​\(𝐂\):=1dk​𝔼​\[‖𝐆Q‖F2\|𝐗\]c\_\{Q\}\(\\bm\{C\}\):=\\tfrac\{1\}\{d\_\{k\}\}\\mathbb\{E\}\[\\left\\\|\\bm\{G\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\],cK​\(𝐂\):=1dk​𝔼​\[‖𝐆K‖F2\|𝐗\]c\_\{K\}\(\\bm\{C\}\):=\\tfrac\{1\}\{d\_\{k\}\}\\mathbb\{E\}\[\\left\\\|\\bm\{G\}\_\{K\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]andcV​\(𝐂\):=1dv​𝔼​\[‖𝐆V‖F2\|𝐗\]c\_\{V\}\(\\bm\{C\}\):=\\tfrac\{1\}\{d\_\{v\}\}\\mathbb\{E\}\[\\left\\\|\\bm\{G\}\_\{V\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]\.

Proof\.Fix𝑿\\bm\{X\}and letW:=\(𝑾Q,𝑾K,𝑾V,𝑾O\)W:=\(\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\)\. Let𝑫∈ℝdk×dk\\bm\{D\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{k\}\}be diagonal with entries in\{±1\}\\\{\\pm 1\\\}, and letτ𝑫\\tau\_\{\\bm\{D\}\}denote the map\(𝑾Q,𝑾K,𝑾V,𝑾O\)↦\(𝑫​𝑾Q,𝑫​𝑾K,𝑾V,𝑾O\)\(\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\)\\mapsto\(\\bm\{D\}\\bm\{W\}\_\{Q\},\\bm\{D\}\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\)\. Since\(𝑫​𝑾Q\)⊤​\(𝑫​𝑾K\)=𝑾Q⊤​𝑾K\(\\bm\{D\}\\bm\{W\}\_\{Q\}\)^\{\\top\}\(\\bm\{D\}\\bm\{W\}\_\{K\}\)=\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{K\}, the matrix𝑨\\bm\{A\}in \([64](https://arxiv.org/html/2610.28975#A5.E64)\), and henceℒ\\mathcal\{L\}, is invariant:ℒ∘τ𝑫=ℒ\\mathcal\{L\}\\circ\\tau\_\{\\bm\{D\}\}=\\mathcal\{L\}\. Differentiating this identity with respect to𝑾Q\\bm\{W\}\_\{Q\}by the chain rule gives𝑫⊤​𝑮Q​\(τ𝑫​W\)=𝑮Q​\(W\)\\bm\{D\}^\{\\top\}\\bm\{G\}\_\{Q\}\(\\tau\_\{\\bm\{D\}\}W\)=\\bm\{G\}\_\{Q\}\(W\), i\.e\.

𝑮Q​\(τ𝑫​W\)=𝑫​𝑮Q​\(W\)\.\\bm\{G\}\_\{Q\}\(\\tau\_\{\\bm\{D\}\}W\)=\\bm\{D\}\\,\\bm\{G\}\_\{Q\}\(W\)\.\(85\)Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8),τ𝑫​W=dW\\tau\_\{\\bm\{D\}\}W\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}W, so

𝔼⁡\[𝑮Q​𝑮Q⊤\|𝑿\]=𝔼⁡\[𝑮Q​\(τ𝑫​W\)​𝑮Q​\(τ𝑫​W\)⊤\|𝑿\]=𝑫​𝔼​\[𝑮Q​𝑮Q⊤\|𝑿\]​𝑫\.\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\\,\|\\,\\bm\{X\}\]=\\mathbb\{E\}\\bigl\[\\bm\{G\}\_\{Q\}\(\\tau\_\{\\bm\{D\}\}W\)\\bm\{G\}\_\{Q\}\(\\tau\_\{\\bm\{D\}\}W\)^\{\\top\}\\,\\big\|\\,\\bm\{X\}\\bigr\]=\\bm\{D\}\\,\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\\,\|\\,\\bm\{X\}\]\\,\\bm\{D\}\.Fora≠ba\\neq b, choosingDa​a=1D\_\{aa\}=1andDb​b=−1D\_\{bb\}=\-1gives𝔼⁡\[\(𝑮Q​𝑮Q⊤\)a​b\|𝑿\]=−𝔼⁡\[\(𝑮Q​𝑮Q⊤\)a​b\|𝑿\]\\mathbb\{E\}\[\(\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\)\_\{ab\}\\,\|\\,\\bm\{X\}\]=\-\\mathbb\{E\}\[\(\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\)\_\{ab\}\\,\|\\,\\bm\{X\}\], hence00\. Likewise, for a permutation matrix𝑷∈ℝdk×dk\\bm\{P\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{k\}\}the map\(𝑾Q,𝑾K\)↦\(𝑷​𝑾Q,𝑷​𝑾K\)\(\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\)\\mapsto\(\\bm\{P\}\\bm\{W\}\_\{Q\},\\bm\{P\}\\bm\{W\}\_\{K\}\)preserves𝑾Q⊤​𝑾K\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{K\}and the law ofWW, and the same argument gives𝔼⁡\[𝑮Q​𝑮Q⊤\|𝑿\]=𝑷​𝔼​\[𝑮Q​𝑮Q⊤\|𝑿\]​𝑷⊤\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\\,\|\\,\\bm\{X\}\]=\\bm\{P\}\\,\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\\,\|\\,\\bm\{X\}\]\\,\\bm\{P\}^\{\\top\}; hence all diagonal entries are equal, and their common value is1dk​𝔼​\[tr⁡\(𝑮Q​𝑮Q⊤\)\|𝑿\]=cQ​\(𝑪\)\\tfrac\{1\}\{d\_\{k\}\}\\mathbb\{E\}\[\\tr\(\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\)\\,\|\\,\\bm\{X\}\]=c\_\{Q\}\(\\bm\{C\}\)\. The same argument with𝑾Q\\bm\{W\}\_\{Q\}and𝑾K\\bm\{W\}\_\{K\}exchanged gives the claim for𝑮K\\bm\{G\}\_\{K\}\.

For𝑮V\\bm\{G\}\_\{V\}, let𝑫∈ℝdv×dv\\bm\{D\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{v\}\}be diagonal with entries in\{±1\}\\\{\\pm 1\\\}and letτ𝑫\\tau\_\{\\bm\{D\}\}map\(𝑾V,𝑾O\)↦\(𝑫​𝑾V,𝑾O​𝑫\)\(\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\)\\mapsto\(\\bm\{D\}\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\\bm\{D\}\), fixing𝑾Q,𝑾K\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\. Sincef⁡\(𝑿\)=𝑨​𝑿​\(𝑾O​𝑾V\)⊤f\(\\bm\{X\}\)=\\bm\{A\}\\bm\{X\}\(\\bm\{W\}\_\{O\}\\bm\{W\}\_\{V\}\)^\{\\top\}depends on\(𝑾V,𝑾O\)\(\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\)only through the product𝑾O​𝑾V\\bm\{W\}\_\{O\}\\bm\{W\}\_\{V\}, which equals\(𝑾O​𝑫\)​\(𝑫​𝑾V\)\(\\bm\{W\}\_\{O\}\\bm\{D\}\)\(\\bm\{D\}\\bm\{W\}\_\{V\}\),ℒ∘τ𝑫=ℒ\\mathcal\{L\}\\circ\\tau\_\{\\bm\{D\}\}=\\mathcal\{L\}; differentiating with respect to𝑾V\\bm\{W\}\_\{V\}gives𝑮V​\(τ𝑫​W\)=𝑫​𝑮V​\(W\)\\bm\{G\}\_\{V\}\(\\tau\_\{\\bm\{D\}\}W\)=\\bm\{D\}\\,\\bm\{G\}\_\{V\}\(W\), and\(𝑫​𝑾V,𝑾O​𝑫\)=d\(𝑾V,𝑾O\)\(\\bm\{D\}\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\\bm\{D\}\)\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\(\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\)under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8)\. The sign and permutation arguments above apply verbatim\.

###### Lemma 35\(Conditional closed form for𝑾O\\bm\{W\}\_\{O\}\)\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8), conditionally on𝐗\\bm\{X\},

𝔼⁡\[𝑮O​𝑮O⊤\|𝑿\]=αO​\(𝑪\)​𝑰d2\+dv​c2​𝚽​𝑪2​𝚽⊤,\\mathbb\{E\}\\bigl\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\\,\\big\|\\,\\bm\{X\}\\bigr\]=\\alpha\_\{O\}\(\\bm\{C\}\)\\,\\bm\{I\}\_\{d\_\{2\}\}\+d\_\{v\}\\,c\_\{2\}\\;\\bm\{\\Phi\}\\,\\bm\{C\}^\{2\}\\,\\bm\{\\Phi\}^\{\\top\},\(86\)αO​\(𝑪\)\\displaystyle\\alpha\_\{O\}\(\\bm\{C\}\)=dv​\(dv\+1\)​\(c12​c4\+c2​c4\+c23\)\+dv​\(c12​c22\+2​c2​c4\)\\displaystyle=d\_\{v\}\(d\_\{v\}\{\+\}1\)\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\+d\_\{v\}\\bigl\(c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}2c\_\{2\}c\_\{4\}\\bigr\)\(87\)\+dv​\(dv\+1\)dk​\(c12​c4\+c12​c22\+3​c2​c4\+c23\)\+2​dvdk​\(c12​c4\+c2​c4\+c23\)\.\\displaystyle\+\\tfrac\{d\_\{v\}\(d\_\{v\}\{\+\}1\)\}\{d\_\{k\}\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}3c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\+\\tfrac\{2d\_\{v\}\}\{d\_\{k\}\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\.

Proof\.Let𝛀:=𝑨​𝒁​𝒁⊤​𝑨⊤\\bm\{\\Omega\}:=\\bm\{A\}\\bm\{Z\}\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\. Substituting𝑹=𝑨​𝒁​𝑾O⊤−𝒀\\bm\{R\}=\\bm\{A\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}\-\\bm\{Y\}into𝑮O​𝑮O⊤=𝑹⊤​𝛀​𝑹\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}=\\bm\{R\}^\{\\top\}\\bm\{\\Omega\}\\bm\{R\},

𝑮O​𝑮O⊤=𝑾O​𝒁⊤​𝑨⊤​𝛀​𝑨​𝒁​𝑾O⊤⏟T1−𝑾O​𝒁⊤​𝑨⊤​𝛀​𝒀⏟T2−𝒀⊤​𝛀​𝑨​𝒁​𝑾O⊤⏟T3\+𝒀⊤​𝛀​𝒀⏟T4\.\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}=\\underbrace\{\\bm\{W\}\_\{O\}\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{\\Omega\}\\bm\{A\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}\}\_\{\\mathrm\{T\}\_\{1\}\}\-\\underbrace\{\\bm\{W\}\_\{O\}\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{\\Omega\}\\,\\bm\{Y\}\\vphantom\{\\bm\{W\}\_\{O\}^\{\\top\}\}\}\_\{\\mathrm\{T\}\_\{2\}\}\-\\underbrace\{\\bm\{Y\}^\{\\top\}\\bm\{\\Omega\}\\bm\{A\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}\}\_\{\\mathrm\{T\}\_\{3\}\}\+\\underbrace\{\\bm\{Y\}^\{\\top\}\\bm\{\\Omega\}\\,\\bm\{Y\}\\vphantom\{\\bm\{W\}\_\{O\}^\{\\top\}\}\}\_\{\\mathrm\{T\}\_\{4\}\}\.\(88\)The termsT2,T3\\mathrm\{T\}\_\{2\},\\mathrm\{T\}\_\{3\}are linear in𝑾O\\bm\{W\}\_\{O\}, which is centered and independent of\(𝑿,𝑾Q,𝑾K,𝑾V\)\(\\bm\{X\},\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\}\), so𝔼⁡\[T2\|𝑿\]=𝔼⁡\[T3\|𝑿\]=𝟎\\mathbb\{E\}\[\\mathrm\{T\}\_\{2\}\\,\|\\,\\bm\{X\}\]=\\mathbb\{E\}\[\\mathrm\{T\}\_\{3\}\\,\|\\,\\bm\{X\}\]=\\bm\{0\}\.

*The termT4\\mathrm\{T\}\_\{4\}\.*By[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27),𝔼\[𝒁𝒁⊤\|𝑿,𝑾Q,𝑾K\]=𝑿𝔼\[𝑾V⊤𝑾V\]𝑿⊤=dv𝑿𝑿⊤\\mathbb\{E\}\[\\bm\{Z\}\\bm\{Z\}^\{\\top\}\\,\|\\,\\bm\{X\},\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\]=\\bm\{X\}\\,\\mathbb\{E\}\[\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{W\}\_\{V\}\]\\,\\bm\{X\}^\{\\top\}=d\_\{v\}\\bm\{X\}\\bm\{X\}^\{\\top\}, hence

𝔼\[T4\|𝑿,𝑾Q,𝑾K\]=dv𝚽\(𝑿⊤𝑨𝑿𝑿⊤𝑨⊤𝑿\)𝚽⊤,𝑿⊤​𝑨​𝑿​𝑿⊤​𝑨⊤​𝑿=1dk​𝑪​𝑾Q⊤​𝑾K​𝑪2​𝑾K⊤​𝑾Q​𝑪\.\\begin\{gathered\}\\mathbb\{E\}\[\\mathrm\{T\}\_\{4\}\\,\|\\,\\bm\{X\},\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\]=d\_\{v\}\\,\\bm\{\\Phi\}\\bigl\(\\bm\{X\}^\{\\top\}\\bm\{A\}\\,\\bm\{X\}\\bm\{X\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{X\}\\bigr\)\\bm\{\\Phi\}^\{\\top\},\\\\ \\bm\{X\}^\{\\top\}\\bm\{A\}\\,\\bm\{X\}\\bm\{X\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{X\}=\\tfrac\{1\}\{d\_\{k\}\}\\bm\{C\}\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{K\}\\bm\{C\}^\{2\}\\bm\{W\}\_\{K\}^\{\\top\}\\bm\{W\}\_\{Q\}\\bm\{C\}\.\\end\{gathered\}By[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27),𝔼⁡\[𝑾K​𝑪2​𝑾K⊤\|𝑿\]=c2​𝑰dk\\mathbb\{E\}\[\\bm\{W\}\_\{K\}\\bm\{C\}^\{2\}\\bm\{W\}\_\{K\}^\{\\top\}\\,\|\\,\\bm\{X\}\]=c\_\{2\}\\bm\{I\}\_\{d\_\{k\}\}and then𝔼⁡\[𝑾Q⊤​𝑾Q\]=dk​𝑰d0\\mathbb\{E\}\[\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{Q\}\]=d\_\{k\}\\bm\{I\}\_\{d\_\{0\}\}, so

𝔼⁡\[T4\|𝑿\]=dv​c2​𝚽​𝑪2​𝚽⊤\.\\mathbb\{E\}\[\\mathrm\{T\}\_\{4\}\\,\|\\,\\bm\{X\}\]=d\_\{v\}\\,c\_\{2\}\\;\\bm\{\\Phi\}\\bm\{C\}^\{2\}\\bm\{\\Phi\}^\{\\top\}\.\(89\)
*The termT1\\mathrm\{T\}\_\{1\}\.*By[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27)applied to𝑾O\\bm\{W\}\_\{O\},

𝔼\[T1\|𝑿,𝑾Q,𝑾K,𝑾V\]=tr\(𝒁⊤𝑨⊤𝛀𝑨𝒁\)𝑰d2=tr\(\(𝑨⊤𝑨𝒁𝒁⊤\)2\)𝑰d2\.\\mathbb\{E\}\[\\mathrm\{T\}\_\{1\}\\,\|\\,\\bm\{X\},\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\},\\bm\{W\}\_\{V\}\]=\\tr\\bigl\(\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{\\Omega\}\\bm\{A\}\\bm\{Z\}\\bigr\)\\bm\{I\}\_\{d\_\{2\}\}=\\tr\\bigl\(\(\\bm\{A\}^\{\\top\}\\bm\{A\}\\bm\{Z\}\\bm\{Z\}^\{\\top\}\)^\{2\}\\bigr\)\\bm\{I\}\_\{d\_\{2\}\}\.Let𝑵:=𝑾Q​𝑪​𝑾Q⊤\\bm\{N\}:=\\bm\{W\}\_\{Q\}\\bm\{C\}\\bm\{W\}\_\{Q\}^\{\\top\},𝑺:=𝑾K⊤​𝑵​𝑾K\\bm\{S\}:=\\bm\{W\}\_\{K\}^\{\\top\}\\bm\{N\}\\bm\{W\}\_\{K\},𝑯V:=𝑾V⊤​𝑾V∼𝒲d0​\(dv,𝑰d0\)\\bm\{H\}\_\{V\}:=\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{W\}\_\{V\}\\sim\\mathcal\{W\}\_\{d\_\{0\}\}\(d\_\{v\},\\bm\{I\}\_\{d\_\{0\}\}\)and𝑴:=𝑪​𝑺​𝑪\\bm\{M\}:=\\bm\{C\}\\bm\{S\}\\bm\{C\}, which is symmetric\. Then𝑨⊤​𝑨=1dk​𝑿​𝑺​𝑿⊤\\bm\{A\}^\{\\top\}\\bm\{A\}=\\tfrac\{1\}\{d\_\{k\}\}\\bm\{X\}\\bm\{S\}\\bm\{X\}^\{\\top\}and𝒁​𝒁⊤=𝑿​𝑯V​𝑿⊤\\bm\{Z\}\\bm\{Z\}^\{\\top\}=\\bm\{X\}\\bm\{H\}\_\{V\}\\bm\{X\}^\{\\top\}, so

tr⁡\(\(𝑨⊤​𝑨​𝒁​𝒁⊤\)2\)=1dk2​tr⁡\(𝑴​𝑯V​𝑴​𝑯V\)\.\\tr\\bigl\(\(\\bm\{A\}^\{\\top\}\\bm\{A\}\\bm\{Z\}\\bm\{Z\}^\{\\top\}\)^\{2\}\\bigr\)=\\tfrac\{1\}\{d\_\{k\}^\{2\}\}\\tr\(\\bm\{M\}\\bm\{H\}\_\{V\}\\bm\{M\}\\bm\{H\}\_\{V\}\)\.Since𝑯V\\bm\{H\}\_\{V\}is independent of𝑴\\bm\{M\},[Lemma29](https://arxiv.org/html/2610.28975#Thmlemma29)with\(n,d\)=\(dv,d0\)\(n,d\)=\(d\_\{v\},d\_\{0\}\)gives

𝔼\[tr\(𝑴𝑯V𝑴𝑯V\)\|𝑿,𝑾Q,𝑾K\]=dv\(dv\+1\)tr\(𝑴2\)\+dv\(tr𝑴\)2\.\\mathbb\{E\}\\bigl\[\\tr\(\\bm\{M\}\\bm\{H\}\_\{V\}\\bm\{M\}\\bm\{H\}\_\{V\}\)\\,\\big\|\\,\\bm\{X\},\\bm\{W\}\_\{Q\},\\bm\{W\}\_\{K\}\\bigr\]=d\_\{v\}\(d\_\{v\}\{\+\}1\)\\tr\(\\bm\{M\}^\{2\}\)\+d\_\{v\}\(\\tr\\bm\{M\}\)^\{2\}\.Nowtr⁡𝑴=tr⁡\(𝑺​𝑪2\)\\tr\\bm\{M\}=\\tr\(\\bm\{S\}\\bm\{C\}^\{2\}\)andtr⁡𝑴2=tr⁡\(𝑺​𝑪2​𝑺​𝑪2\)\\tr\\bm\{M\}^\{2\}=\\tr\(\\bm\{S\}\\bm\{C\}^\{2\}\\bm\{S\}\\bm\{C\}^\{2\}\), so[Lemma31](https://arxiv.org/html/2610.28975#Thmlemma31)\(conditionally on𝑵\\bm\{N\}\) followed by[Lemma30](https://arxiv.org/html/2610.28975#Thmlemma30)give

𝔼⁡\[tr⁡𝑴2\|𝑿\]\\displaystyle\\mathbb\{E\}\[\\tr\\bm\{M\}^\{2\}\\,\|\\,\\bm\{X\}\]=dk2​\(c12​c4\+c2​c4\+c23\)\+dk​\(c12​c4\+c12​c22\+3​c2​c4\+c23\),\\displaystyle=d\_\{k\}^\{2\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\+d\_\{k\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}3c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\),\(90\)𝔼⁡\[\(tr⁡𝑴\)2\|𝑿\]\\displaystyle\\mathbb\{E\}\[\(\\tr\\bm\{M\}\)^\{2\}\\,\|\\,\\bm\{X\}\]=dk2​\(c12​c22\+2​c2​c4\)\+2​dk​\(c12​c4\+c2​c4\+c23\)\.\\displaystyle=d\_\{k\}^\{2\}\\bigl\(c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}2c\_\{2\}c\_\{4\}\\bigr\)\+2d\_\{k\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\.Therefore𝔼⁡\[T1\|𝑿\]=αO​\(𝑪\)​𝑰d2\\mathbb\{E\}\[\\mathrm\{T\}\_\{1\}\\,\|\\,\\bm\{X\}\]=\\alpha\_\{O\}\(\\bm\{C\}\)\\bm\{I\}\_\{d\_\{2\}\}withαO​\(𝑪\)=1dk2​\(dv​\(dv\+1\)​𝔼​\[tr⁡𝑴2\|𝑿\]\+dv​𝔼​\[\(tr⁡𝑴\)2\|𝑿\]\)\\alpha\_\{O\}\(\\bm\{C\}\)=\\tfrac\{1\}\{d\_\{k\}^\{2\}\}\\bigl\(d\_\{v\}\(d\_\{v\}\{\+\}1\)\\mathbb\{E\}\[\\tr\\bm\{M\}^\{2\}\\,\|\\,\\bm\{X\}\]\+d\_\{v\}\\mathbb\{E\}\[\(\\tr\\bm\{M\}\)^\{2\}\\,\|\\,\\bm\{X\}\]\\bigr\), which is \([87](https://arxiv.org/html/2610.28975#A5.E87)\); adding \([89](https://arxiv.org/html/2610.28975#A5.E89)\) gives \([86](https://arxiv.org/html/2610.28975#A5.E86)\)\.

###### Lemma 36\(Conditional constants\)\.

Under[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8), conditionally on𝐗\\bm\{X\}, withtΦ:=tr⁡\(𝚽​𝐂2​𝚽⊤\)t\_\{\\Phi\}:=\\tr\(\\bm\{\\Phi\}\\bm\{C\}^\{2\}\\bm\{\\Phi\}^\{\\top\}\), the constants of[Lemma34](https://arxiv.org/html/2610.28975#Thmlemma34)are

cQ​\(𝑪\)\\displaystyle c\_\{Q\}\(\\bm\{C\}\)=c2dk​\(β1\+dv​tΦ\)\+c2dk2​\(β1\+β2\),\\displaystyle=\\frac\{c\_\{2\}\}\{d\_\{k\}\}\\bigl\(\\beta\_\{1\}\+d\_\{v\}\\,t\_\{\\Phi\}\\bigr\)\+\\frac\{c\_\{2\}\}\{d\_\{k\}^\{2\}\}\\bigl\(\\beta\_\{1\}\+\\beta\_\{2\}\\bigr\),\(91\)cK​\(𝑪\)\\displaystyle c\_\{K\}\(\\bm\{C\}\)=c2dk​\(β1\+dv​tΦ\)\+c12\+c2dk2​β1,\\displaystyle=\\frac\{c\_\{2\}\}\{d\_\{k\}\}\\bigl\(\\beta\_\{1\}\+d\_\{v\}\\,t\_\{\\Phi\}\\bigr\)\+\\frac\{c\_\{1\}^\{2\}\+c\_\{2\}\}\{d\_\{k\}^\{2\}\}\\,\\beta\_\{1\},\(92\)cV​\(𝑪\)\\displaystyle c\_\{V\}\(\\bm\{C\}\)=d2​\(d2\+dv\+1\)​\[\(c12​c4\+c2​c4\+c23\)\+1dk​\(c12​c4\+c12​c22\+3​c2​c4\+c23\)\]\+c2​tΦ\.\\displaystyle=d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\Bigl\[\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\+\\tfrac\{1\}\{d\_\{k\}\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}3c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\\Bigr\]\+c\_\{2\}\\,t\_\{\\Phi\}\.\(93\)

Proof\.Substituting𝑹=𝑨​𝒁​𝑾O⊤−𝒀\\bm\{R\}=\\bm\{A\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}\-\\bm\{Y\}into \([67](https://arxiv.org/html/2610.28975#A5.E67)\) splits each gradient as𝑮∙=𝒂∙−𝒃∙\\bm\{G\}\_\{\\bullet\}=\\bm\{a\}\_\{\\bullet\}\-\\bm\{b\}\_\{\\bullet\}, where𝒂∙\\bm\{a\}\_\{\\bullet\}collects the term containing𝑨​𝒁​𝑾O⊤\\bm\{A\}\\bm\{Z\}\\bm\{W\}\_\{O\}^\{\\top\}and𝒃∙\\bm\{b\}\_\{\\bullet\}the term containing𝒀\\bm\{Y\}\. For∙∈\{Q,K\}\\bullet\\in\\\{Q,K\\\},𝒂∙\\bm\{a\}\_\{\\bullet\}is quadratic and𝒃∙\\bm\{b\}\_\{\\bullet\}linear in𝑾O\\bm\{W\}\_\{O\}; for∙=V\\bullet=V,𝒂V\\bm\{a\}\_\{V\}is quadratic and𝒃V\\bm\{b\}\_\{V\}linear in𝑾O\\bm\{W\}\_\{O\}as well\. Since𝑾O\\bm\{W\}\_\{O\}is centered and independent of everything else,𝔼⁡\[tr⁡\(𝒂∙​𝒃∙⊤\)\|𝑿\]=0\\mathbb\{E\}\[\\tr\(\\bm\{a\}\_\{\\bullet\}\\bm\{b\}\_\{\\bullet\}^\{\\top\}\)\\,\|\\,\\bm\{X\}\]=0and

𝔼⁡\[‖𝑮∙‖F2\|𝑿\]=𝔼⁡\[‖𝒂∙‖F2\|𝑿\]\+𝔼⁡\[‖𝒃∙‖F2\|𝑿\]\.\\mathbb\{E\}\[\\left\\\|\\bm\{G\}\_\{\\bullet\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=\\mathbb\{E\}\[\\left\\\|\\bm\{a\}\_\{\\bullet\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]\+\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{\\bullet\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]\.Throughout,𝑯O:=𝑾O⊤​𝑾O\\bm\{H\}\_\{O\}:=\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{W\}\_\{O\},𝑬:=𝑪​𝑾V⊤​𝑯O​𝑾V​𝑪\\bm\{E\}:=\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{H\}\_\{O\}\\bm\{W\}\_\{V\}\\bm\{C\}\(as in[Lemma32](https://arxiv.org/html/2610.28975#Thmlemma32)\),𝑵:=𝑾Q​𝑪​𝑾Q⊤\\bm\{N\}:=\\bm\{W\}\_\{Q\}\\bm\{C\}\\bm\{W\}\_\{Q\}^\{\\top\}\(as in[Lemma30](https://arxiv.org/html/2610.28975#Thmlemma30)\), and we use the identities𝑿⊤​𝒁=𝑪​𝑾V⊤\\bm\{X\}^\{\\top\}\\bm\{Z\}=\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\}and𝑿⊤​𝑨⊤​𝑿=1dk​𝑪​𝑾K⊤​𝑾Q​𝑪\\bm\{X\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{X\}=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\bm\{C\}\\bm\{W\}\_\{K\}^\{\\top\}\\bm\{W\}\_\{Q\}\\bm\{C\}\.

*Target parts\.*From \([67](https://arxiv.org/html/2610.28975#A5.E67)\),𝒃Q=1dk​𝑾K​𝑪​𝑾V⊤​𝑾O⊤​𝚽​𝑪\\bm\{b\}\_\{Q\}=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\bm\{W\}\_\{K\}\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{\\Phi\}\\bm\{C\}\. By[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27)applied to𝑾K\\bm\{W\}\_\{K\}and cyclicity of the trace,

𝔼\[‖𝒃Q‖F2\|𝑿,𝑾V,𝑾O\]=tr\(𝑾V𝑪2𝑾V⊤⋅𝑾O⊤𝚽𝑪2𝚽⊤𝑾O\),\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\},\\bm\{W\}\_\{V\},\\bm\{W\}\_\{O\}\]=\\tr\\bigl\(\\bm\{W\}\_\{V\}\\bm\{C\}^\{2\}\\bm\{W\}\_\{V\}^\{\\top\}\\cdot\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{\\Phi\}\\bm\{C\}^\{2\}\\bm\{\\Phi\}^\{\\top\}\\bm\{W\}\_\{O\}\\bigr\),and[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27)applied to𝑾V\\bm\{W\}\_\{V\}\(𝔼⁡\[𝑾V​𝑪2​𝑾V⊤\]=c2​𝑰dv\\mathbb\{E\}\[\\bm\{W\}\_\{V\}\\bm\{C\}^\{2\}\\bm\{W\}\_\{V\}^\{\\top\}\]=c\_\{2\}\\bm\{I\}\_\{d\_\{v\}\}\) and to𝑾O\\bm\{W\}\_\{O\}\(𝔼⁡\[𝑾O⊤​𝚽​𝑪2​𝚽⊤​𝑾O\]=tΦ​𝑰dv\\mathbb\{E\}\[\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{\\Phi\}\\bm\{C\}^\{2\}\\bm\{\\Phi\}^\{\\top\}\\bm\{W\}\_\{O\}\]=t\_\{\\Phi\}\\bm\{I\}\_\{d\_\{v\}\}\) gives𝔼⁡\[‖𝒃Q‖F2\|𝑿\]=dv​c2​tΦ\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{v\}c\_\{2\}t\_\{\\Phi\}\. The same computation applies to𝒃K=1dk​𝑾Q​𝑪​𝚽⊤​𝑾O​𝑾V​𝑪\\bm\{b\}\_\{K\}=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\bm\{W\}\_\{Q\}\\bm\{C\}\\bm\{\\Phi\}^\{\\top\}\\bm\{W\}\_\{O\}\\bm\{W\}\_\{V\}\\bm\{C\}\([Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27)applied to𝑾Q\\bm\{W\}\_\{Q\}\) and to𝒃V=𝑾O⊤​𝒀⊤​𝑨​𝑿\\bm\{b\}\_\{V\}=\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{Y\}^\{\\top\}\\bm\{A\}\\bm\{X\}\([Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27)applied to𝑾O\\bm\{W\}\_\{O\}, then \([89](https://arxiv.org/html/2610.28975#A5.E89)\) for the average of𝑿⊤​𝑨​𝑿​𝑿⊤​𝑨⊤​𝑿\\bm\{X\}^\{\\top\}\\bm\{A\}\\bm\{X\}\\bm\{X\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{X\}over𝑾K,𝑾Q\\bm\{W\}\_\{K\},\\bm\{W\}\_\{Q\}\), so

𝔼⁡\[‖𝒃Q‖F2\|𝑿\]=𝔼⁡\[‖𝒃K‖F2\|𝑿\]=𝔼⁡\[‖𝒃V‖F2\|𝑿\]=dv​c2​tΦ\.\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{K\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{V\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{v\}\\,c\_\{2\}\\,t\_\{\\Phi\}\.
*Model part of𝐆Q\\bm\{G\}\_\{Q\}\.*Using𝑿⊤​𝒁=𝑪​𝑾V⊤\\bm\{X\}^\{\\top\}\\bm\{Z\}=\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\}and𝒁⊤​𝑨⊤​𝑿=𝑾V​𝑿⊤​𝑨⊤​𝑿=1dk​𝑾V​𝑪​𝑾K⊤​𝑾Q​𝑪\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{X\}=\\bm\{W\}\_\{V\}\\bm\{X\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{X\}=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\bm\{W\}\_\{V\}\\bm\{C\}\\bm\{W\}\_\{K\}^\{\\top\}\\bm\{W\}\_\{Q\}\\bm\{C\},

𝒂Q=1dk​𝑾K​𝑬​𝑾K⊤​𝑾Q​𝑪=1dk​𝑱​𝑾Q​𝑪,‖𝒂Q‖F2=1dk2​tr⁡\(𝑱2​𝑵~\),\\bm\{a\}\_\{Q\}=\\tfrac\{1\}\{d\_\{k\}\}\\bm\{W\}\_\{K\}\\bm\{E\}\\bm\{W\}\_\{K\}^\{\\top\}\\bm\{W\}\_\{Q\}\\bm\{C\}=\\tfrac\{1\}\{d\_\{k\}\}\\bm\{J\}\\bm\{W\}\_\{Q\}\\bm\{C\},\\qquad\\left\\\|\\bm\{a\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\tfrac\{1\}\{d\_\{k\}^\{2\}\}\\tr\\bigl\(\\bm\{J\}^\{2\}\\widetilde\{\\bm\{N\}\}\\bigr\),where𝑱:=𝑾K​𝑬​𝑾K⊤\\bm\{J\}:=\\bm\{W\}\_\{K\}\\bm\{E\}\\bm\{W\}\_\{K\}^\{\\top\}and𝑵~:=𝑾Q​𝑪2​𝑾Q⊤\\widetilde\{\\bm\{N\}\}:=\\bm\{W\}\_\{Q\}\\bm\{C\}^\{2\}\\bm\{W\}\_\{Q\}^\{\\top\}\. Given𝑿\\bm\{X\},𝑱\\bm\{J\}and𝑵~\\widetilde\{\\bm\{N\}\}are independent, and𝔼⁡\[𝑵~\|𝑿\]=c2​𝑰dk\\mathbb\{E\}\[\\widetilde\{\\bm\{N\}\}\\,\|\\,\\bm\{X\}\]=c\_\{2\}\\bm\{I\}\_\{d\_\{k\}\}by[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27), so𝔼⁡\[‖𝒂Q‖F2\|𝑿\]=c2dk2​𝔼​\[tr⁡𝑱2\|𝑿\]\\mathbb\{E\}\[\\left\\\|\\bm\{a\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=\\tfrac\{c\_\{2\}\}\{d\_\{k\}^\{2\}\}\\mathbb\{E\}\[\\tr\\bm\{J\}^\{2\}\\,\|\\,\\bm\{X\}\]\. By[Lemma30](https://arxiv.org/html/2610.28975#Thmlemma30)with𝑬\\bm\{E\}in place of𝑪\\bm\{C\}and𝑾K\\bm\{W\}\_\{K\}in place of𝑾Q\\bm\{W\}\_\{Q\},𝔼\[tr𝑱2\|𝑿,𝑬\]=dk2tr\(𝑬2\)\+dk\(\(tr𝑬\)2\+tr\(𝑬2\)\)\\mathbb\{E\}\[\\tr\\bm\{J\}^\{2\}\\,\|\\,\\bm\{X\},\\bm\{E\}\]=d\_\{k\}^\{2\}\\tr\(\\bm\{E\}^\{2\}\)\+d\_\{k\}\\bigl\(\(\\tr\\bm\{E\}\)^\{2\}\+\\tr\(\\bm\{E\}^\{2\}\)\\bigr\), and by[Lemma32](https://arxiv.org/html/2610.28975#Thmlemma32),𝔼⁡\[tr⁡𝑱2\|𝑿\]=dk2​β1\+dk​\(β1\+β2\)\\mathbb\{E\}\[\\tr\\bm\{J\}^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{k\}^\{2\}\\beta\_\{1\}\+d\_\{k\}\(\\beta\_\{1\}\{\+\}\\beta\_\{2\}\)\. Hence𝔼⁡\[‖𝒂Q‖F2\|𝑿\]=c2​β1\+c2dk​\(β1\+β2\)\\mathbb\{E\}\[\\left\\\|\\bm\{a\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=c\_\{2\}\\beta\_\{1\}\+\\tfrac\{c\_\{2\}\}\{d\_\{k\}\}\(\\beta\_\{1\}\{\+\}\\beta\_\{2\}\), andcQ​\(𝑪\)=1dk​\(𝔼⁡\[‖𝒂Q‖F2\|𝑿\]\+𝔼⁡\[‖𝒃Q‖F2\|𝑿\]\)c\_\{Q\}\(\\bm\{C\}\)=\\tfrac\{1\}\{d\_\{k\}\}\\bigl\(\\mathbb\{E\}\[\\left\\\|\\bm\{a\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]\+\\mathbb\{E\}\[\\left\\\|\\bm\{b\}\_\{Q\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]\\bigr\)is \([91](https://arxiv.org/html/2610.28975#A5.E91)\)\.

*Model part of𝐆K\\bm\{G\}\_\{K\}\.*Using𝑿⊤​𝑨​𝒁=1dk​𝑪​𝑾Q⊤​𝑾K​𝑪​𝑾V⊤\\bm\{X\}^\{\\top\}\\bm\{A\}\\bm\{Z\}=\\tfrac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\bm\{C\}\\bm\{W\}\_\{Q\}^\{\\top\}\\bm\{W\}\_\{K\}\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\}and𝒁⊤​𝑿=𝑾V​𝑪\\bm\{Z\}^\{\\top\}\\bm\{X\}=\\bm\{W\}\_\{V\}\\bm\{C\},

𝒂K=1dk​𝑵​𝑾K​𝑬,‖𝒂K‖F2=1dk2​tr⁡\(𝑵2​𝑾K​𝑬2​𝑾K⊤\)\.\\bm\{a\}\_\{K\}=\\tfrac\{1\}\{d\_\{k\}\}\\bm\{N\}\\bm\{W\}\_\{K\}\\bm\{E\},\\qquad\\left\\\|\\bm\{a\}\_\{K\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\tfrac\{1\}\{d\_\{k\}^\{2\}\}\\tr\\bigl\(\\bm\{N\}^\{2\}\\,\\bm\{W\}\_\{K\}\\bm\{E\}^\{2\}\\bm\{W\}\_\{K\}^\{\\top\}\\bigr\)\.Given𝑿\\bm\{X\}, the matrices𝑵\\bm\{N\},𝑾K\\bm\{W\}\_\{K\}and𝑬\\bm\{E\}are independent\. By[Lemma27](https://arxiv.org/html/2610.28975#Thmlemma27),𝔼\[𝑾K𝑬2𝑾K⊤\|𝑿,𝑬\]=tr\(𝑬2\)𝑰dk\\mathbb\{E\}\[\\bm\{W\}\_\{K\}\\bm\{E\}^\{2\}\\bm\{W\}\_\{K\}^\{\\top\}\\,\|\\,\\bm\{X\},\\bm\{E\}\]=\\tr\(\\bm\{E\}^\{2\}\)\\bm\{I\}\_\{d\_\{k\}\}; by[Lemma30](https://arxiv.org/html/2610.28975#Thmlemma30),𝔼⁡\[tr⁡𝑵2\|𝑿\]=dk2​c2\+dk​\(c12\+c2\)\\mathbb\{E\}\[\\tr\\bm\{N\}^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{k\}^\{2\}c\_\{2\}\+d\_\{k\}\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\); and by[Lemma32](https://arxiv.org/html/2610.28975#Thmlemma32),𝔼⁡\[tr⁡𝑬2\|𝑿\]=β1\\mathbb\{E\}\[\\tr\\bm\{E\}^\{2\}\\,\|\\,\\bm\{X\}\]=\\beta\_\{1\}\. Hence𝔼⁡\[‖𝒂K‖F2\|𝑿\]=\(c2\+c12\+c2dk\)​β1\\mathbb\{E\}\[\\left\\\|\\bm\{a\}\_\{K\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=\\bigl\(c\_\{2\}\+\\tfrac\{c\_\{1\}^\{2\}\+c\_\{2\}\}\{d\_\{k\}\}\\bigr\)\\beta\_\{1\}, which together with the target part gives \([92](https://arxiv.org/html/2610.28975#A5.E92)\)\.

*Model part of𝐆V\\bm\{G\}\_\{V\}\.*Here𝒂V=𝑾O⊤​𝑾O​𝒁⊤​𝑨⊤​𝑨​𝑿=𝑯O​𝒁⊤​𝑨⊤​𝑨​𝑿\\bm\{a\}\_\{V\}=\\bm\{W\}\_\{O\}^\{\\top\}\\bm\{W\}\_\{O\}\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{A\}\\bm\{X\}=\\bm\{H\}\_\{O\}\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{A\}\\bm\{X\}, so

‖𝒂V‖F2=tr⁡\(𝑯O2​𝑴V\),𝑴V:=𝒁⊤​𝑨⊤​𝑨​𝑿​𝑿⊤​𝑨⊤​𝑨​𝒁=1dk2​𝑾V​𝑪​𝑺​𝑪2​𝑺​𝑪​𝑾V⊤,\\left\\\|\\bm\{a\}\_\{V\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\tr\(\\bm\{H\}\_\{O\}^\{2\}\\bm\{M\}\_\{V\}\),\\qquad\\bm\{M\}\_\{V\}:=\\bm\{Z\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{A\}\\bm\{X\}\\bm\{X\}^\{\\top\}\\bm\{A\}^\{\\top\}\\bm\{A\}\\bm\{Z\}=\\tfrac\{1\}\{d\_\{k\}^\{2\}\}\\bm\{W\}\_\{V\}\\bm\{C\}\\bm\{S\}\\bm\{C\}^\{2\}\\bm\{S\}\\bm\{C\}\\bm\{W\}\_\{V\}^\{\\top\},with𝑺\\bm\{S\}as in the proof of[Lemma35](https://arxiv.org/html/2610.28975#Thmlemma35)\. Given𝑿\\bm\{X\},𝑯O\\bm\{H\}\_\{O\}is independent of𝑴V\\bm\{M\}\_\{V\}and𝔼⁡\[𝑯O2\]=d2​\(d2\+dv\+1\)​𝑰dv\\mathbb\{E\}\[\\bm\{H\}\_\{O\}^\{2\}\]=d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\bm\{I\}\_\{d\_\{v\}\}by[Lemma29](https://arxiv.org/html/2610.28975#Thmlemma29), so𝔼⁡\[‖𝒂V‖F2\|𝑿\]=d2​\(d2\+dv\+1\)​𝔼​\[tr⁡𝑴V\|𝑿\]\\mathbb\{E\}\[\\left\\\|\\bm\{a\}\_\{V\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\|\\,\\bm\{X\}\]=d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\mathbb\{E\}\[\\tr\\bm\{M\}\_\{V\}\\,\|\\,\\bm\{X\}\]\. Taking the expectation over𝑾V\\bm\{W\}\_\{V\}\(𝔼⁡\[𝑾V⊤​𝑾V\]=dv​𝑰d0\\mathbb\{E\}\[\\bm\{W\}\_\{V\}^\{\\top\}\\bm\{W\}\_\{V\}\]=d\_\{v\}\\bm\{I\}\_\{d\_\{0\}\}\) and then over𝑾K,𝑾Q\\bm\{W\}\_\{K\},\\bm\{W\}\_\{Q\}by[Lemmas31](https://arxiv.org/html/2610.28975#Thmlemma31)and[30](https://arxiv.org/html/2610.28975#Thmlemma30),

𝔼⁡\[tr⁡𝑴V\|𝑿\]=dvdk2​𝔼​\[tr⁡\(𝑺​𝑪2​𝑺​𝑪2\)\|𝑿\]=dv​\[\(c12​c4\+c2​c4\+c23\)\+1dk​\(c12​c4\+c12​c22\+3​c2​c4\+c23\)\],\\mathbb\{E\}\[\\tr\\bm\{M\}\_\{V\}\\,\|\\,\\bm\{X\}\]=\\tfrac\{d\_\{v\}\}\{d\_\{k\}^\{2\}\}\\,\\mathbb\{E\}\[\\tr\(\\bm\{S\}\\bm\{C\}^\{2\}\\bm\{S\}\\bm\{C\}^\{2\}\)\\,\|\\,\\bm\{X\}\]=d\_\{v\}\\Bigl\[\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\+\\tfrac\{1\}\{d\_\{k\}\}\\bigl\(c\_\{1\}^\{2\}c\_\{4\}\{\+\}c\_\{1\}^\{2\}c\_\{2\}^\{2\}\{\+\}3c\_\{2\}c\_\{4\}\{\+\}c\_\{2\}^\{3\}\\bigr\)\\Bigr\],which is the first line of \([90](https://arxiv.org/html/2610.28975#A5.E90)\) divided bydk2d\_\{k\}^\{2\}\. Adding the target part and dividing bydvd\_\{v\}gives \([93](https://arxiv.org/html/2610.28975#A5.E93)\)\.

Proof of[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)\.Take expectations over𝑿\\bm\{X\}in[Lemmas34](https://arxiv.org/html/2610.28975#Thmlemma34),[35](https://arxiv.org/html/2610.28975#Thmlemma35)and[36](https://arxiv.org/html/2610.28975#Thmlemma36); by the tower property and[Assumption8](https://arxiv.org/html/2610.28975#Thmassumption8),

𝔼\[𝑮O𝑮O⊤\]=𝔼\[αO\(𝑪\)\]𝑰d2\+dv𝚽𝔼\[c2𝑪2\]𝚽⊤,𝔼\[𝑮∙𝑮∙⊤\]=𝔼\[c∙\(𝑪\)\]𝑰\(∙∈\{Q,K,V\}\)\.\\mathbb\{E\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\]=\\mathbb\{E\}\[\\alpha\_\{O\}\(\\bm\{C\}\)\]\\,\\bm\{I\}\_\{d\_\{2\}\}\+d\_\{v\}\\,\\bm\{\\Phi\}\\,\\mathbb\{E\}\[c\_\{2\}\\bm\{C\}^\{2\}\]\\,\\bm\{\\Phi\}^\{\\top\},\\qquad\\mathbb\{E\}\[\\bm\{G\}\_\{\\bullet\}\\bm\{G\}\_\{\\bullet\}^\{\\top\}\]=\\mathbb\{E\}\[c\_\{\\bullet\}\(\\bm\{C\}\)\]\\,\\bm\{I\}\\quad\(\\bullet\\in\\\{Q,K,V\\\}\)\.For𝑸∈O⁡\(d0\)\\bm\{Q\}\\in O\(d\_\{0\}\),𝑸​𝑪​𝑸⊤=d𝑪\\bm\{Q\}\\bm\{C\}\\bm\{Q\}^\{\\top\}\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\\bm\{C\}andc2c\_\{2\}is invariant, so𝔼⁡\[c2​𝑪2\]=𝑸​𝔼​\[c2​𝑪2\]​𝑸⊤\\mathbb\{E\}\[c\_\{2\}\\bm\{C\}^\{2\}\]=\\bm\{Q\}\\,\\mathbb\{E\}\[c\_\{2\}\\bm\{C\}^\{2\}\]\\,\\bm\{Q\}^\{\\top\}for all𝑸\\bm\{Q\}, and by Schur’s lemma𝔼⁡\[c2​𝑪2\]=γ​𝑰d0\\mathbb\{E\}\[c\_\{2\}\\bm\{C\}^\{2\}\]=\\gamma\\,\\bm\{I\}\_\{d\_\{0\}\}withγ=1d0​𝔼​\[c2​tr⁡𝑪2\]=1d0​𝔼​\[c22\]\\gamma=\\tfrac\{1\}\{d\_\{0\}\}\\mathbb\{E\}\[c\_\{2\}\\tr\\bm\{C\}^\{2\}\]=\\tfrac\{1\}\{d\_\{0\}\}\\mathbb\{E\}\[c\_\{2\}^\{2\}\]; \([77](https://arxiv.org/html/2610.28975#A5.E77)\) gives the polynomial and the order in \([70](https://arxiv.org/html/2610.28975#A5.E70)\)\. This proves \([69](https://arxiv.org/html/2610.28975#A5.E69)\), and averaging \([87](https://arxiv.org/html/2610.28975#A5.E87)\) termwise with \([66](https://arxiv.org/html/2610.28975#A5.E66)\) gives the closed form ofα¯O=𝔼⁡\[αO​\(𝑪\)\]\\bar\{\\alpha\}\_\{O\}=\\mathbb\{E\}\[\\alpha\_\{O\}\(\\bm\{C\}\)\]in \([71](https://arxiv.org/html/2610.28975#A5.E71)\)\. The same isotropy argument gives𝔼⁡\[c2​tΦ\]=tr⁡\(𝚽​𝔼​\[c2​𝑪2\]​𝚽⊤\)=γ​‖𝚽‖F2\\mathbb\{E\}\[c\_\{2\}t\_\{\\Phi\}\]=\\tr\(\\bm\{\\Phi\}\\,\\mathbb\{E\}\[c\_\{2\}\\bm\{C\}^\{2\}\]\\,\\bm\{\\Phi\}^\{\\top\}\)=\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}, so averaging \([91](https://arxiv.org/html/2610.28975#A5.E91)\)–\([93](https://arxiv.org/html/2610.28975#A5.E93)\) gives \([68](https://arxiv.org/html/2610.28975#A5.E68)\), where by \([75](https://arxiv.org/html/2610.28975#A5.E75)\)

𝔼⁡\[c2​β1\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]=d2​dv​\[\(d2​dv\+d2\+dv\+3\)​𝔼​\[c2​c4\]\+\(d2\+dv\+1\)​𝔼​\[c23\]\],\\displaystyle=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}d\_\{2\}\{\+\}d\_\{v\}\{\+\}3\)\\,\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]\+\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\,\\mathbb\{E\}\[c\_\{2\}^\{3\}\]\\Bigr\],𝔼⁡\[c2​β2\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]=d2​dv​\[\(d2​dv\+2\)​𝔼​\[c23\]\+2​\(d2\+dv\+1\)​𝔼​\[c2​c4\]\],\\displaystyle=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}2\)\\,\\mathbb\{E\}\[c\_\{2\}^\{3\}\]\+2\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\,\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]\\Bigr\],\(94\)𝔼⁡\[\(c12\+c2\)​β1\]\\displaystyle\\mathbb\{E\}\[\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\)\\beta\_\{1\}\]=d2​dv​\[\(d2​dv\+d2\+dv\+3\)​\(𝔼⁡\[c12​c4\]\+𝔼⁡\[c2​c4\]\)\+\(d2\+dv\+1\)​\(𝔼⁡\[c12​c22\]\+𝔼⁡\[c23\]\)\],\\displaystyle=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}d\_\{2\}\{\+\}d\_\{v\}\{\+\}3\)\\bigl\(\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\]\{\+\}\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]\\bigr\)\+\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\bigl\(\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}^\{2\}\]\{\+\}\\mathbb\{E\}\[c\_\{2\}^\{3\}\]\\bigr\)\\Bigr\],all explicit by[Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33)\. Finally, substituting𝖠3=𝖠1\+𝖠2\\mathsf\{A\}\_\{3\}=\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\}into the closed form ofα¯O\\bar\{\\alpha\}\_\{O\},

α¯O=dv​\[\(dv\+1\)​\(1\+1dk\)\+2dk\]​𝖠1\+dv​\[1\+dv\+1dk\]​𝖠2=Θ⁡\(dv2​d06\),\\bar\{\\alpha\}\_\{O\}=d\_\{v\}\\Bigl\[\(d\_\{v\}\{\+\}1\)\\Bigl\(1\+\\tfrac\{1\}\{d\_\{k\}\}\\Bigr\)\+\\tfrac\{2\}\{d\_\{k\}\}\\Bigr\]\\mathsf\{A\}\_\{1\}\+d\_\{v\}\\Bigl\[1\+\\tfrac\{d\_\{v\}\{\+\}1\}\{d\_\{k\}\}\\Bigr\]\\mathsf\{A\}\_\{2\}=\\Theta\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\),\(95\)all terms being nonnegative and𝖠1,𝖠2=Θ⁡\(d06\)\\mathsf\{A\}\_\{1\},\\mathsf\{A\}\_\{2\}=\\Theta\(d\_\{0\}^\{6\}\)by[Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33), which is the order stated in \([71](https://arxiv.org/html/2610.28975#A5.E71)\)\.

Proof of[Corollary3](https://arxiv.org/html/2610.28975#Thmcorollary3)\.All Wishart moments of[Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33)are positive, anddv,dk≥1d\_\{v\},d\_\{k\}\\geq 1throughout\.

*Step 1: theβ\\beta\-moments\.*By \([94](https://arxiv.org/html/2610.28975#A5.E94)\), each of

𝔼⁡\[c2​β1\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]=d2​dv​\[\(d2​dv\+d2\+dv\+3\)​𝔼​\[c2​c4\]\+\(d2\+dv\+1\)​𝔼​\[c23\]\],\\displaystyle=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}d\_\{2\}\{\+\}d\_\{v\}\{\+\}3\)\\,\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]\+\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\,\\mathbb\{E\}\[c\_\{2\}^\{3\}\]\\Bigr\],𝔼⁡\[c2​β2\]\\displaystyle\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]=d2​dv​\[\(d2​dv\+2\)​𝔼​\[c23\]\+2​\(d2\+dv\+1\)​𝔼​\[c2​c4\]\],\\displaystyle=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}2\)\\,\\mathbb\{E\}\[c\_\{2\}^\{3\}\]\+2\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\,\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]\\Bigr\],𝔼⁡\[\(c12\+c2\)​β1\]\\displaystyle\\mathbb\{E\}\[\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\)\\beta\_\{1\}\]=d2​dv​\[\(d2​dv\+d2\+dv\+3\)​\(𝔼⁡\[c12​c4\]\+𝔼⁡\[c2​c4\]\)\+\(d2\+dv\+1\)​\(𝔼⁡\[c12​c22\]\+𝔼⁡\[c23\]\)\]\\displaystyle=d\_\{2\}d\_\{v\}\\Bigl\[\(d\_\{2\}d\_\{v\}\{\+\}d\_\{2\}\{\+\}d\_\{v\}\{\+\}3\)\\bigl\(\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\]\{\+\}\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\]\\bigr\)\+\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\bigl\(\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}^\{2\}\]\{\+\}\\mathbb\{E\}\[c\_\{2\}^\{3\}\]\\bigr\)\\Bigr\]isd2​dvd\_\{2\}d\_\{v\}times a polynomial of degree one indvd\_\{v\}whose coefficients are positive combinations of the degree\-six moments𝔼⁡\[c2​c4\],𝔼⁡\[c23\],𝔼⁡\[c12​c4\],𝔼⁡\[c12​c22\]=Θ⁡\(d06\)\\mathbb\{E\}\[c\_\{2\}c\_\{4\}\],\\mathbb\{E\}\[c\_\{2\}^\{3\}\],\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{4\}\],\\mathbb\{E\}\[c\_\{1\}^\{2\}c\_\{2\}^\{2\}\]=\\Theta\(d\_\{0\}^\{6\}\)of[Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33); sinced2​dv≥12​\(d2\+dv\)d\_\{2\}d\_\{v\}\\geq\\tfrac\{1\}\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\), each is

𝔼⁡\[c2​β1\]=Θ⁡\(d22​dv2​d06\),𝔼⁡\[c2​β2\]=Θ⁡\(d22​dv2​d06\),𝔼⁡\[\(c12\+c2\)​β1\]=Θ⁡\(d22​dv2​d06\)\.\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]=\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\),\\qquad\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]=\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\),\\qquad\\mathbb\{E\}\[\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\)\\beta\_\{1\}\]=\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\.\(96\)
*Step 2:cQc\_\{Q\}andcKc\_\{K\}\.*Substituting \([96](https://arxiv.org/html/2610.28975#A5.E96)\) into \([68](https://arxiv.org/html/2610.28975#A5.E68)\),

cQ\\displaystyle c\_\{Q\}=1dk​\(𝔼⁡\[c2​β1\]⏟Θ⁡\(d22​dv2​d06\)\+dv​γ​‖𝚽‖F2⏟Θ⁡\(dv​d03\)\)\+1dk2​\(𝔼⁡\[c2​β1\]\+𝔼⁡\[c2​β2\]\)⏟Θ⁡\(d22​dv2​d06\),\\displaystyle=\\frac\{1\}\{d\_\{k\}\}\\Bigl\(\\underbrace\{\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]\}\_\{\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\}\+\\underbrace\{d\_\{v\}\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\_\{\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)\}\\Bigr\)\+\\frac\{1\}\{d\_\{k\}^\{2\}\}\\underbrace\{\\bigl\(\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]\+\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]\\bigr\)\}\_\{\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\},cK\\displaystyle c\_\{K\}=1dk​\(𝔼⁡\[c2​β1\]⏟Θ⁡\(d22​dv2​d06\)\+dv​γ​‖𝚽‖F2⏟Θ⁡\(dv​d03\)\)\+1dk2​𝔼⁡\[\(c12\+c2\)​β1\]⏟Θ⁡\(d22​dv2​d06\),\\displaystyle=\\frac\{1\}\{d\_\{k\}\}\\Bigl\(\\underbrace\{\\mathbb\{E\}\[c\_\{2\}\\beta\_\{1\}\]\}\_\{\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\}\+\\underbrace\{d\_\{v\}\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\_\{\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)\}\\Bigr\)\+\\frac\{1\}\{d\_\{k\}^\{2\}\}\\underbrace\{\\mathbb\{E\}\[\(c\_\{1\}^\{2\}\{\+\}c\_\{2\}\)\\beta\_\{1\}\]\}\_\{\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\},whereγ=Θ⁡\(d03\)\\gamma=\\Theta\(d\_\{0\}^\{3\}\)by[Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33)\. All terms are positive, and sinced2,dv,dk,d0≥1d\_\{2\},d\_\{v\},d\_\{k\},d\_\{0\}\\geq 1the second and third terms are at most a constant multiple of the first; hence

cQ≍cK=Θ⁡\(d22​dv2​d06/dk\),cK−cQ=𝔼⁡\[c12​β1\]−𝔼⁡\[c2​β2\]dk2=O⁡\(d22​dv2​d06dk2\)=O⁡\(1dk\)​cQ\.c\_\{Q\}\\asymp c\_\{K\}=\\Theta\\bigl\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}/d\_\{k\}\\bigr\),\\qquad c\_\{K\}\-c\_\{Q\}=\\frac\{\\mathbb\{E\}\[c\_\{1\}^\{2\}\\beta\_\{1\}\]\-\\mathbb\{E\}\[c\_\{2\}\\beta\_\{2\}\]\}\{d\_\{k\}^\{2\}\}=O\\Bigl\(\\frac\{d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}\}\{d\_\{k\}^\{2\}\}\\Bigr\)=O\\Bigl\(\\frac\{1\}\{d\_\{k\}\}\\Bigr\)\\,c\_\{Q\}\.
*Step 3:cVc\_\{V\}\.*By \([68](https://arxiv.org/html/2610.28975#A5.E68)\) with𝖠3=𝖠1\+𝖠2\\mathsf\{A\}\_\{3\}=\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\},

cV=d2​\(d2\+dv\+1\)​\(𝖠1\+𝖠3dk\)\+γ​‖𝚽‖F2,d2​𝖠1≤d2​\(𝖠1\+𝖠3dk\)≤d2​\(𝖠1\+𝖠3\),c\_\{V\}=d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\{\+\}1\)\\Bigl\(\\mathsf\{A\}\_\{1\}\+\\frac\{\\mathsf\{A\}\_\{3\}\}\{d\_\{k\}\}\\Bigr\)\+\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\},\\qquad d\_\{2\}\\mathsf\{A\}\_\{1\}\\leq d\_\{2\}\\Bigl\(\\mathsf\{A\}\_\{1\}\+\\frac\{\\mathsf\{A\}\_\{3\}\}\{d\_\{k\}\}\\Bigr\)\\leq d\_\{2\}\(\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{3\}\),socVc\_\{V\}is affine indvd\_\{v\}with slope betweend2​𝖠1d\_\{2\}\\mathsf\{A\}\_\{1\}andd2​\(𝖠1\+𝖠3\)d\_\{2\}\(\\mathsf\{A\}\_\{1\}\{\+\}\\mathsf\{A\}\_\{3\}\)and intercept betweend2​\(d2\+1\)​𝖠1d\_\{2\}\(d\_\{2\}\{\+\}1\)\\mathsf\{A\}\_\{1\}andd2​\(d2\+1\)​\(𝖠1\+𝖠3\)\+γ​‖𝚽‖F2d\_\{2\}\(d\_\{2\}\{\+\}1\)\(\\mathsf\{A\}\_\{1\}\{\+\}\\mathsf\{A\}\_\{3\}\)\+\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}; since𝖠1,𝖠2=Θ⁡\(d06\)\\mathsf\{A\}\_\{1\},\\mathsf\{A\}\_\{2\}=\\Theta\(d\_\{0\}^\{6\}\)andγ=Θ⁡\(d03\)\\gamma=\\Theta\(d\_\{0\}^\{3\}\)\([Lemma33](https://arxiv.org/html/2610.28975#Thmlemma33)\),cV=Θ⁡\(d2​\(d2\+dv\)​d06\)c\_\{V\}=\\Theta\\bigl\(d\_\{2\}\(d\_\{2\}\{\+\}d\_\{v\}\)\\,d\_\{0\}^\{6\}\\bigr\)\.

*Step 4:α¯O\\bar\{\\alpha\}\_\{O\}\.*Letα¯O∞:=dv​\(dv\+1\)​𝖠1\+dv​𝖠2\\bar\{\\alpha\}\_\{O\}^\{\\infty\}:=d\_\{v\}\(d\_\{v\}\{\+\}1\)\\mathsf\{A\}\_\{1\}\+d\_\{v\}\\mathsf\{A\}\_\{2\}be thedkd\_\{k\}\-free part of \([95](https://arxiv.org/html/2610.28975#A5.E95)\)\. Then

α¯O−α¯O∞\\displaystyle\\bar\{\\alpha\}\_\{O\}\-\\bar\{\\alpha\}\_\{O\}^\{\\infty\}=1dk​\[dv​\(dv\+1\)​𝖠3\+2​dv​𝖠1\]≤1dk​\[dv​\(dv\+1\)​\(𝖠1\+𝖠2\)\+2​dv​\(dv\+1\)​𝖠1\]\\displaystyle=\\frac\{1\}\{d\_\{k\}\}\\Bigl\[d\_\{v\}\(d\_\{v\}\{\+\}1\)\\mathsf\{A\}\_\{3\}\+2d\_\{v\}\\mathsf\{A\}\_\{1\}\\Bigr\]\\leq\\frac\{1\}\{d\_\{k\}\}\\Bigl\[d\_\{v\}\(d\_\{v\}\{\+\}1\)\(\\mathsf\{A\}\_\{1\}\{\+\}\\mathsf\{A\}\_\{2\}\)\+2d\_\{v\}\(d\_\{v\}\{\+\}1\)\\mathsf\{A\}\_\{1\}\\Bigr\]=3​𝖠1\+𝖠2𝖠1⋅dv​\(dv\+1\)​𝖠1dk≤3​𝖠1\+𝖠2𝖠1⋅α¯O∞dk,\\displaystyle=\\frac\{3\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\}\}\{\\mathsf\{A\}\_\{1\}\}\\cdot\\frac\{d\_\{v\}\(d\_\{v\}\{\+\}1\)\\mathsf\{A\}\_\{1\}\}\{d\_\{k\}\}\\leq\\frac\{3\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\}\}\{\\mathsf\{A\}\_\{1\}\}\\cdot\\frac\{\\bar\{\\alpha\}\_\{O\}^\{\\infty\}\}\{d\_\{k\}\},so that

α¯O∞≤α¯O≤\(1\+3​𝖠1\+𝖠2𝖠1​dk\)​α¯O∞,dv2​𝖠1≤α¯O∞≤2​dv2​𝖠1\+dv2​𝖠2,\\bar\{\\alpha\}\_\{O\}^\{\\infty\}\\leq\\bar\{\\alpha\}\_\{O\}\\leq\\Bigl\(1\+\\frac\{3\\mathsf\{A\}\_\{1\}\+\\mathsf\{A\}\_\{2\}\}\{\\mathsf\{A\}\_\{1\}d\_\{k\}\}\\Bigr\)\\bar\{\\alpha\}\_\{O\}^\{\\infty\},\\qquad d\_\{v\}^\{2\}\\mathsf\{A\}\_\{1\}\\leq\\bar\{\\alpha\}\_\{O\}^\{\\infty\}\\leq 2d\_\{v\}^\{2\}\\mathsf\{A\}\_\{1\}\+d\_\{v\}^\{2\}\\mathsf\{A\}\_\{2\},\(97\)and therefore, with𝖠1,𝖠2=Θ⁡\(d06\)\\mathsf\{A\}\_\{1\},\\mathsf\{A\}\_\{2\}=\\Theta\(d\_\{0\}^\{6\}\),α¯O=Θ⁡\(dv2​d06\)\\bar\{\\alpha\}\_\{O\}=\\Theta\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\)\.

*Step 5: the𝐖O\\bm\{W\}\_\{O\}ratio\.*By \([69](https://arxiv.org/html/2610.28975#A5.E69)\), fora≠ba\\neq bwith\(𝚽​𝚽⊤\)a​b≠0\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}\\neq 0,

𝔼⁡\[\(𝑮O​𝑮O⊤\)a​a\]=α¯O\+dv​γ​\(𝚽​𝚽⊤\)a​a=Θ⁡\(dv2​d06\),\\displaystyle\\mathbb\{E\}\[\(\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\)\_\{aa\}\]=\\bar\{\\alpha\}\_\{O\}\+d\_\{v\}\\gamma\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{aa\}=\\Theta\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\),𝔼⁡\[\(𝑮O​𝑮O⊤\)a​b\]=dv​γ​\(𝚽​𝚽⊤\)a​b=Θ⁡\(dv​d03\),\\displaystyle\\mathbb\{E\}\[\(\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\)\_\{ab\}\]=d\_\{v\}\\gamma\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\}=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\),and dividing gives the ratioΘ⁡\(dv​d03\)\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)\. Finally, Steps 3 and 4 show thatdkd\_\{k\}enterscVc\_\{V\}andα¯O\\bar\{\\alpha\}\_\{O\}only through the factors1\+𝖠3/\(𝖠1​dk\)∈\[1,1\+𝖠3/𝖠1\]1\+\\mathsf\{A\}\_\{3\}/\(\\mathsf\{A\}\_\{1\}d\_\{k\}\)\\in\[1,\\,1\+\\mathsf\{A\}\_\{3\}/\\mathsf\{A\}\_\{1\}\]and the bracket in \([97](https://arxiv.org/html/2610.28975#A5.E97)\), both bounded above and below; the same is then true of the ratio\.

### E\.6Discussion

## Appendix FAsymptotic Approximation between Orthogonalization and Row Normalization

What does diagonal dominance say about the orthogonalization step itself? Suppose𝑽t​𝑽t⊤\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}is diagonal\. Then𝑽t​𝑽t⊤=diag⁡\(𝑽t​𝑽t⊤\)\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}=\\diagop\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\), so\(𝑽t𝑽t⊤\)−1/2=diag\(𝑽t𝑽t⊤\)−1/2\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}=\\diagop\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}, and sincediag\(𝑽t𝑽t⊤\)i​i=‖\(𝑽t\)i,:‖22\\diagop\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)\_\{ii\}=\\left\\\|\(\\bm\{V\}\_\{t\}\)\_\{i,:\}\\right\\\|\_\{2\}^\{2\},

\[\(𝑽t𝑽t⊤\)−1/2𝑽t\]i,:=\(𝑽t\)i,:‖\(𝑽t\)i,:‖2\.\\bigl\[\(\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}\)^\{\-1/2\}\\bm\{V\}\_\{t\}\\bigr\]\_\{i,:\}=\\frac\{\(\\bm\{V\}\_\{t\}\)\_\{i,:\}\}\{\\left\\\|\(\\bm\{V\}\_\{t\}\)\_\{i,:\}\\right\\\|\_\{2\}\}\.\(98\)That is, when𝑽t​𝑽t⊤\\bm\{V\}\_\{t\}\\bm\{V\}\_\{t\}^\{\\top\}is diagonal the orthogonalization in \([1](https://arxiv.org/html/2610.28975#S1.E1)\) is just each row of the momentum divided by its ownℓ2\\ell\_\{2\}norm, which is exactly the row normalization used by several recent stateless optimizers\([Ma et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib21);[Scetbon et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib22);[Glentis et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib23);[Wen et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib24);[Pethick et al\., 2025](https://arxiv.org/html/2610.28975#bib.bib13);[Deng et al\., 2026](https://arxiv.org/html/2610.28975#bib.bib27)\); equivalently, its implicit preconditioner isdiag⁡\(𝑷t\)1/2⊗𝑰n\\diagop\(\\bm\{P\}\_\{t\}\)^\{1/2\}\\otimes\\bm\{I\}\_\{n\}, which equalsHMuonH\_\{\\mathrm\{Muon\}\}when𝑷t\\bm\{P\}\_\{t\}is diagonal\. Putting \([98](https://arxiv.org/html/2610.28975#A6.E98)\) together with[Theorems1](https://arxiv.org/html/2610.28975#Thmtheorem1),[2](https://arxiv.org/html/2610.28975#Thmtheorem2),[3](https://arxiv.org/html/2610.28975#Thmtheorem3)and[4](https://arxiv.org/html/2610.28975#Thmtheorem4): at a Gaussian initialization, as the hidden dimension grows, Muon’sΘ⁡\(m​n​min⁡\(m,n\)\)\\Theta\(mn\\min\(m,n\)\)orthogonalization collapses to aΘ⁡\(m​n\)\\Theta\(mn\)row\-wiseℓ2\\ell\_\{2\}normalization with the same preconditioner structure, and for the query, key and value weights of attention the collapse is exact in expectation\.[Deng et al\. \(2026\)](https://arxiv.org/html/2610.28975#bib.bib27)check that this equivalence holds throughout the training of GPT\-2 and LLaMA, and that the dominance ratio grows with model size, in line with the hidden\-dimension dependence in[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)and[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)\. Carrying the analysis past initialization, and to deep nonlinear networks, is left for future work\.

## Appendix GSimulation of Theoretical Results

We numerically verify the dimension\-scaling predictions of[Theorems1](https://arxiv.org/html/2610.28975#Thmtheorem1),[2](https://arxiv.org/html/2610.28975#Thmtheorem2),[3](https://arxiv.org/html/2610.28975#Thmtheorem3)and[4](https://arxiv.org/html/2610.28975#Thmtheorem4)under the Gaussian initialization of[Section3\.2](https://arxiv.org/html/2610.28975#S3.SS2)\. For each model, we perform two experiments\. First, at a representative configuration, we estimate the expected gradient self outer\-product and examine its diagonal and off\-diagonal structure\. Second, we vary the relevant model dimensions and measure how the diagonal magnitude, off\-diagonal magnitude, and their ratio change\.

To summarize the structure of a matrix𝑷∈ℝm×m\\bm\{P\}\\in\\mathbb\{R\}^\{m\\times m\}, we define the mean diagonal magnitude, mean off\-diagonal magnitude, and diagonal\-dominance ratio as

D⁡\(𝑷\)=1m​∑i=1m\|Pi​i\|,OD⁡\(𝑷\)=1m⁡\(m−1\)​∑i≠j\|Pi​j\|,R⁡\(𝑷\)=D⁡\(𝑷\)OD⁡\(𝑷\)\.D\(\\bm\{P\}\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\|P\_\{ii\}\|,\\qquad\\mathrm\{OD\}\(\\bm\{P\}\)=\\frac\{1\}\{m\(m\-1\)\}\\sum\_\{i\\neq j\}\|P\_\{ij\}\|,\\qquad R\(\\bm\{P\}\)=\\frac\{D\(\\bm\{P\}\)\}\{\\mathrm\{OD\}\(\\bm\{P\}\)\}\.\(99\)
GivenSSindependent initializations with gradients𝑮\(1\),…,𝑮\(S\)\\bm\{G\}^\{\(1\)\},\\ldots,\\bm\{G\}^\{\(S\)\}, we estimate the expected gradient self outer\-product by

𝔼^​\[𝑮​𝑮⊤\]=1S​∑s=1S𝑮\(s\)​\(𝑮\(s\)\)⊤\.\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]=\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}\\bm\{G\}^\{\(s\)\}\(\\bm\{G\}^\{\(s\)\}\)^\{\\top\}\.\(100\)
We useD^\\widehat\{D\},OD^\\widehat\{\\mathrm\{OD\}\}, andR^\\widehat\{R\}to denote the corresponding empirical estimates ofD⁡\(𝔼⁡\[𝑮​𝑮⊤\]\)D\(\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\),OD⁡\(𝔼⁡\[𝑮​𝑮⊤\]\)\\mathrm\{OD\}\(\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\), andR⁡\(𝔼⁡\[𝑮​𝑮⊤\]\)R\(\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\), respectively\.

### G\.1Symmetric Matrix Factorization

#### Setup\.

We next verify the predictions of[Theorems1](https://arxiv.org/html/2610.28975#Thmtheorem1)and[1](https://arxiv.org/html/2610.28975#Thmcorollary1)for symmetric matrix factorization\. As defined in \([16](https://arxiv.org/html/2610.28975#A2.E16)\), the gradient is

𝑮=\(𝑾​𝑾⊤−𝑴⋆\)​𝑾,\\bm\{G\}=\(\\bm\{W\}\\bm\{W\}^\{\\top\}\-\\bm\{M\}^\{\\star\}\)\\bm\{W\},where𝑾∈ℝd×k\\bm\{W\}\\in\\mathbb\{R\}^\{d\\times k\}and𝑴⋆∈ℝd×d\\bm\{M\}^\{\\star\}\\in\\mathbb\{R\}^\{d\\times d\}is symmetric positive semidefinite\. Under[Assumption5](https://arxiv.org/html/2610.28975#Thmassumption5),[Theorem1](https://arxiv.org/html/2610.28975#Thmtheorem1)gives

𝔼⁡\[𝑮​𝑮⊤\]=α3​𝑰d−2​α2​𝑴⋆\+k​\(𝑴⋆\)2,\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]=\\alpha\_\{3\}\\bm\{I\}\_\{d\}\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\+k\(\\bm\{M\}^\{\\star\}\)^\{2\},\(101\)whereα2=k⁡\(k\+d\+1\)\\alpha\_\{2\}=k\(k\+d\+1\)andα3=k3\+3​\(d\+1\)​k2\+\(d2\+3​d\+4\)​k\\alpha\_\{3\}=k^\{3\}\+3\(d\+1\)k^\{2\}\+\(d^\{2\}\+3d\+4\)k\.[Corollary1](https://arxiv.org/html/2610.28975#Thmcorollary1)gives the corresponding diagonal and off\-diagonal entries,

𝔼⁡\[\(𝑮​𝑮⊤\)i​i\]\\displaystyle\\mathbb\{E\}\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ii\}\]=α3−2​α2​𝑴i​i⋆\+k​\(\(𝑴⋆\)2\)i​i,\\displaystyle=\\alpha\_\{3\}\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\_\{ii\}\+k\\bigl\(\(\\bm\{M\}^\{\\star\}\)^\{2\}\\bigr\)\_\{ii\},\(102\)𝔼⁡\[\(𝑮​𝑮⊤\)i​j\]\\displaystyle\\mathbb\{E\}\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ij\}\]=−2α2𝑴⋆i​j\+k\(\(𝑴⋆\)2\)i​j,i≠j\.\\displaystyle=\-2\\alpha\_\{2\}\\bm\{M\}^\{\\star\}\_\{ij\}\+k\\bigl\(\(\\bm\{M\}^\{\\star\}\)^\{2\}\\bigr\)\_\{ij\},\\qquad i\\neq j\.For fixedddand a fixed target with nonzero off\-diagonal structure, this givesD⁡\(𝔼⁡\[𝑮​𝑮⊤\]\)=Θ⁡\(k3\)D\(\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\)=\\Theta\(k^\{3\}\)andOD⁡\(𝔼⁡\[𝑮​𝑮⊤\]\)=Θ⁡\(k2\)\\mathrm\{OD\}\(\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\)=\\Theta\(k^\{2\}\), and henceR⁡\(𝔼⁡\[𝑮​𝑮⊤\]\)=Θ⁡\(k\)R\(\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]\)=\\Theta\(k\)\.

#### Empirical structure of the expected gradient Gram\.

We first examine the structure of the expected gradient Gram at a representative hidden dimension\. We setd=25d=25,k=80k=80, and construct a fixed rank\-5 target𝑴⋆=𝑸​𝚲​𝑸⊤\\bm\{M\}^\{\\star\}=\\bm\{Q\}\\bm\{\\Lambda\}\\bm\{Q\}^\{\\top\}, where𝑸∈ℝ25×5\\bm\{Q\}\\in\\mathbb\{R\}^\{25\\times 5\}has orthonormal columns obtained from the QR factorization of a Gaussian matrix and𝚲=diag⁡\(3,2\.5,2,1\.5,1\)\\bm\{\\Lambda\}=\\operatorname\{diag\}\(3,2\.5,2,1\.5,1\)\. We drawS=10,000S=10\{,\}000independent initializations𝑾\(1\),…,𝑾\(S\)\\bm\{W\}^\{\(1\)\},\\ldots,\\bm\{W\}^\{\(S\)\}, with i\.i\.d\. entries from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\. For eachs∈\{1,…,S\}s\\in\\\{1,\\ldots,S\\\}, let𝑮\(s\)\\bm\{G\}^\{\(s\)\}denote the gradient corresponding to𝑾\(s\)\\bm\{W\}^\{\(s\)\}\. Using \([100](https://arxiv.org/html/2610.28975#A7.E100)\), we estimate𝔼⁡\[𝑮​𝑮⊤\]\\mathbb\{E\}\[\\bm\{G\}\\bm\{G\}^\{\\top\}\]from these10,00010\{,\}000initializations\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/exp21_mf_empirical_gram.png)Figure 7:Empirical expected gradient self outer\-product for symmetric matrix factorization withd=25d=25,k=80k=80, and a fixed rank\-5 target, estimated over10,00010\{,\}000independent initializations\. Each heatmap is normalized by its empirical mean diagonal magnitude\. The panel uses the logarithmic color scale as shown\.Figure[7](https://arxiv.org/html/2610.28975#A7.F7)shows a strongly diagonal expected gradient Gram, with substantially smaller off\-diagonal entries\. This is consistent with the diagonal\-dominance behavior predicted by[Theorem1](https://arxiv.org/html/2610.28975#Thmtheorem1)\.

#### Hidden\-dimension scaling\.

We next varykkover 15 values from55to640640, while keepingd=25d=25and𝑴⋆\\bm\{M\}^\{\\star\}fixed, usingS=10,000S=10\{,\}000independent initializations at each hidden dimension\.

To reduce Monte Carlo noise in the smaller off\-diagonal entries, we subtract a term whose expected off\-diagonal contribution is zero\. By[Lemma4](https://arxiv.org/html/2610.28975#Thmlemma4), setting𝑴⋆=𝟎\\bm\{M\}^\{\\star\}=\\bm\{0\}gives𝑮⁡\(𝟎\)​𝑮​\(𝟎\)⊤=𝑺3\\bm\{G\}\(\\bm\{0\}\)\\bm\{G\}\(\\bm\{0\}\)^\{\\top\}=\\bm\{S\}^\{3\}, where𝑺=𝑾​𝑾⊤\\bm\{S\}=\\bm\{W\}\\bm\{W\}^\{\\top\}\. Moreover,[Lemma7](https://arxiv.org/html/2610.28975#Thmlemma7)gives𝔼⁡\[𝑺3\]=α3​𝑰d\\mathbb\{E\}\[\\bm\{S\}^\{3\}\]=\\alpha\_\{3\}\\bm\{I\}\_\{d\}, so this term has zero expected off\-diagonal entries\. We therefore define

𝚫MF\(s\)=𝑮\(s\)​\(𝑴⋆\)​𝑮\(s\)​\(𝑴⋆\)⊤−𝑮\(s\)​\(𝟎\)​𝑮\(s\)​\(𝟎\)⊤\.\\bm\{\\Delta\}\_\{\\mathrm\{MF\}\}^\{\(s\)\}=\\bm\{G\}^\{\(s\)\}\(\\bm\{M\}^\{\\star\}\)\\bm\{G\}^\{\(s\)\}\(\\bm\{M\}^\{\\star\}\)^\{\\top\}\-\\bm\{G\}^\{\(s\)\}\(\\bm\{0\}\)\\bm\{G\}^\{\(s\)\}\(\\bm\{0\}\)^\{\\top\}\.\(103\)Fori≠ji\\neq j,

𝔼⁡\[\(𝚫MF\(s\)\)i​j\]=𝔼⁡\[\(𝑮​𝑮⊤\)i​j\]\.\\mathbb\{E\}\\\!\\left\[\(\\bm\{\\Delta\}\_\{\\mathrm\{MF\}\}^\{\(s\)\}\)\_\{ij\}\\right\]=\\mathbb\{E\}\\\!\\left\[\(\\bm\{G\}\\bm\{G\}^\{\\top\}\)\_\{ij\}\\right\]\.Thus, the paired difference removes the𝑺3\\bm\{S\}^\{3\}contribution while preserving the expected off\-diagonal entries\.

For each hidden dimension, we computeD^\\widehat\{D\}from the full empirical average \([100](https://arxiv.org/html/2610.28975#A7.E100)\)\. ForOD^\\widehat\{\\mathrm\{OD\}\}, we first average𝚫MF\(s\)\\bm\{\\Delta\}\_\{\\mathrm\{MF\}\}^\{\(s\)\}over the sameSSinitializations and then applyOD⁡\(⋅\)\\mathrm\{OD\}\(\\cdot\)\. We then computeR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}\.

Figure 8:Hidden\-dimension scaling for symmetric matrix factorization withd=25d=25and a fixed rank\-5 target\. From left to right, the panels showD^\\widehat\{D\},OD^\\widehat\{\\mathrm\{OD\}\}, andR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}as the factorization hidden dimensionkkis varied\. Empirical estimates use10,00010\{,\}000independent initializations at each hidden dimension;OD^\\widehat\{\\mathrm\{OD\}\}uses the paired estimator in \([103](https://arxiv.org/html/2610.28975#A7.E103)\)\.Figure[8](https://arxiv.org/html/2610.28975#A7.F8)shows that the diagonal magnitude grows faster withkkthan the off\-diagonal magnitude, whileR^\\widehat\{R\}grows approximately linearly withkk\. This is consistent with the theoretical predictionsD=Θ⁡\(k3\)D=\\Theta\(k^\{3\}\),OD=Θ⁡\(k2\)\\mathrm\{OD\}=\\Theta\(k^\{2\}\), andR=Θ⁡\(k\)R=\\Theta\(k\)\.

### G\.2Deep Linear Network

#### Setup\.

We next verify the predictions of[Theorems2](https://arxiv.org/html/2610.28975#Thmtheorem2)and[2](https://arxiv.org/html/2610.28975#Thmcorollary2)for the deep linear network\. Under the Gaussian initialization of[Assumption2](https://arxiv.org/html/2610.28975#Thmassumption2),[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)gives

𝔼⁡\[𝑮i​𝑮i⊤\]=Vi​si​𝑰di\+𝑻i,𝑻i=\{\(∏j=1L−1dj\)​𝚽​𝚽⊤,i=L,\(∏1≤j≤L−1j≠idj\)​‖𝚽‖F2​𝑰di,i<L\.\\mathbb\{E\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]=V\_\{i\}s\_\{i\}\\bm\{I\}\_\{d\_\{i\}\}\+\\bm\{T\}\_\{i\},\\qquad\\bm\{T\}\_\{i\}=\\begin\{cases\}\\left\(\\displaystyle\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\right\)\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},&i=L,\\\\\[6\.0pt\] \\left\(\\displaystyle\\prod\_\{\\begin\{subarray\}\{c\}1\\leq j\\leq L\-1\\\\ j\\neq i\\end\{subarray\}\}d\_\{j\}\\right\)\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bm\{I\}\_\{d\_\{i\}\},&i<L\.\\end\{cases\}\(104\)[Corollary2](https://arxiv.org/html/2610.28975#Thmcorollary2)therefore gives zero expected off\-diagonal entries for every hidden layeri<Li<L, while for the output layer

𝔼⁡\[\(𝑮L​𝑮L⊤\)a​b\]=\(∏j=1L−1dj\)​\(𝚽​𝚽⊤\)a​b,a≠b\.\\mathbb\{E\}\[\(\\bm\{G\}\_\{L\}\\bm\{G\}\_\{L\}^\{\\top\}\)\_\{ab\}\]=\\left\(\\prod\_\{j=1\}^\{L\-1\}d\_\{j\}\\right\)\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\_\{ab\},\\qquad a\\neq b\.\(105\)

#### Empirical structure of the expected gradient Gram\.

We first examine the empirical expected gradient Grams at a representative configuration\. We use depthL=4L=4with dimensions\[d0,d1,d2,d3,d4\]=\[512,80,80,80,512\]\[d\_\{0\},d\_\{1\},d\_\{2\},d\_\{3\},d\_\{4\}\]=\[512,80,80,80,512\]and a fixed target𝚽∈ℝ512×512\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{512\\times 512\}with entries sampled from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\. We useS=10,000S=10\{,\}000independent Gaussian weight initializations and estimate the expected Gram for each𝑮1,…,𝑮4\\bm\{G\}\_\{1\},\\ldots,\\bm\{G\}\_\{4\}using \([100](https://arxiv.org/html/2610.28975#A7.E100)\)\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/exp21_deep_linear_empirical_grams.png)Figure 9:Empirical expected gradient self outer\-products for the depth\-44linear network with baseline dimensions\[512,80,80,80,512\]\[512,80,80,80,512\], estimated from10,00010\{,\}000independent initializations\. The four panels show𝔼^​\[𝑮i​𝑮i⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{i\}\\bm\{G\}\_\{i\}^\{\\top\}\]fori=1,…,4i=1,\\ldots,4\. Each heatmap is normalized by its empirical mean diagonal magnitude\. A common logarithmic color scale is used across all panels\.Figure[9](https://arxiv.org/html/2610.28975#A7.F9)shows strongly diagonal expected Grams for𝑮1,𝑮2,𝑮3\\bm\{G\}\_\{1\},\\bm\{G\}\_\{2\},\\bm\{G\}\_\{3\}, consistent with[Corollary2](https://arxiv.org/html/2610.28975#Thmcorollary2), which gives zero expected off\-diagonal entries for the hidden layers\. The output\-layer Gram retains visible off\-diagonal structure, consistent with the𝚽​𝚽⊤\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}term in[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)\.

#### Dimension scaling\.

We next vary each hidden dimensiond1,d2,d3d\_\{1\},d\_\{2\},d\_\{3\}individually while holding the remaining dimensions fixed at the baseline values above\. Each hidden dimension is varied over a 12\-point grid\{5,10,20,40,60,80,120,160,200,240,280,320\}\\\{5,10,20,40,60,80,120,160,200,240,280,320\\\}, and we evaluate𝑮1,…,𝑮4\\bm\{G\}\_\{1\},\\ldots,\\bm\{G\}\_\{4\}\. A fixed512×512512\\times 512target is sampled with entries from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\. We useS=10,000S=10\{,\}000independent initializations for every configuration\.

Because the expected off\-diagonal entries are much smaller than the diagonal entries, direct Monte Carlo estimation ofOD\\mathrm\{OD\}has high variance\. We therefore use a paired construction to reduce this Monte Carlo variance while preserving the expected off\-diagonal entries\. From \([29](https://arxiv.org/html/2610.28975#A3.E29)\) and \([30](https://arxiv.org/html/2610.28975#A3.E30)\), for fixed weights, the gradient is affine in𝚽\\bm\{\\Phi\}\. For eachs∈\{1,…,S\}s\\in\\\{1,\\ldots,S\\\}, let𝑮i\(s\)​\(𝚽\)\\bm\{G\}\_\{i\}^\{\(s\)\}\(\\bm\{\\Phi\}\)denote the layer\-iigradient for thess\-th weight initialization and target𝚽\\bm\{\\Phi\}\. Using the same weights, we evaluate the gradient at𝚽\\bm\{\\Phi\},−𝚽\-\\bm\{\\Phi\}, and𝟎\\bm\{0\}, and define

𝚫i\(s\)=12​\[𝑮i\(s\)​\(𝚽\)​𝑮i\(s\)​\(𝚽\)⊤\+𝑮i\(s\)​\(−𝚽\)​𝑮i\(s\)​\(−𝚽\)⊤\]−𝑮i\(s\)​\(𝟎\)​𝑮i\(s\)​\(𝟎\)⊤\.\\bm\{\\Delta\}\_\{i\}^\{\(s\)\}=\\frac\{1\}\{2\}\\left\[\\bm\{G\}\_\{i\}^\{\(s\)\}\(\\bm\{\\Phi\}\)\\bm\{G\}\_\{i\}^\{\(s\)\}\(\\bm\{\\Phi\}\)^\{\\top\}\+\\bm\{G\}\_\{i\}^\{\(s\)\}\(\-\\bm\{\\Phi\}\)\\bm\{G\}\_\{i\}^\{\(s\)\}\(\-\\bm\{\\Phi\}\)^\{\\top\}\\right\]\-\\bm\{G\}\_\{i\}^\{\(s\)\}\(\\bm\{0\}\)\\bm\{G\}\_\{i\}^\{\(s\)\}\(\\bm\{0\}\)^\{\\top\}\.\(106\)The symmetric combination cancels the target\-independent and target\-linear terms\. The remaining term is

𝚫i\(s\)=\(𝑨i\(s\)\)⊤​𝚽​\(𝑩i\(s\)\)⊤​𝑩i\(s\)​𝚽⊤​𝑨i\(s\)\.\\bm\{\\Delta\}\_\{i\}^\{\(s\)\}=\(\\bm\{A\}\_\{i\}^\{\(s\)\}\)^\{\\top\}\\bm\{\\Phi\}\(\\bm\{B\}\_\{i\}^\{\(s\)\}\)^\{\\top\}\\bm\{B\}\_\{i\}^\{\(s\)\}\\bm\{\\Phi\}^\{\\top\}\\bm\{A\}\_\{i\}^\{\(s\)\}\.\(107\)By[Lemma15](https://arxiv.org/html/2610.28975#Thmlemma15),

𝔼⁡\[𝚫i\(s\)\]=𝑻i\.\\mathbb\{E\}\[\\bm\{\\Delta\}\_\{i\}^\{\(s\)\}\]=\\bm\{T\}\_\{i\}\.\(108\)Since the remaining termVi​si​𝑰diV\_\{i\}s\_\{i\}\\bm\{I\}\_\{d\_\{i\}\}in[Theorem2](https://arxiv.org/html/2610.28975#Thmtheorem2)is diagonal, \([106](https://arxiv.org/html/2610.28975#A7.E106)\) has exactly the same expected off\-diagonal entries as the full gradient Gram\.

For each configuration, we computeD^\\widehat\{D\}from the full empirical expected Gram\. ForOD^\\widehat\{\\mathrm\{OD\}\}, we first average𝚫i\(s\)\\bm\{\\Delta\}\_\{i\}^\{\(s\)\}over the sameSSinitializations and then applyOD⁡\(⋅\)\\mathrm\{OD\}\(\\cdot\)\. We then computeR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}\.

Figure 10:Empirical dimension scaling for the depth\-44linear network\. Columns varyd1,d2,d\_\{1\},d\_\{2\},andd3d\_\{3\}individually while holding the remaining dimensions fixed at their baseline values\. Rows showD^\\widehat\{D\},OD^\\widehat\{\\mathrm\{OD\}\}, andR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}for𝑮1,…,𝑮4\\bm\{G\}\_\{1\},\\ldots,\\bm\{G\}\_\{4\}, each normalized by its value at the baseline configuration\[d0,d1,d2,d3,d4\]=\[512,80,80,80,512\]\[d\_\{0\},d\_\{1\},d\_\{2\},d\_\{3\},d\_\{4\}\]=\[512,80,80,80,512\]to facilitate comparison across layers with different absolute scales\. Empirical estimates use10,00010\{,\}000independent initializations per configuration\. The off\-diagonal statistic is estimated using \([106](https://arxiv.org/html/2610.28975#A7.E106)\)\.Figure[10](https://arxiv.org/html/2610.28975#A7.F10)shows the layer\-dependent scaling predicted by[Theorems2](https://arxiv.org/html/2610.28975#Thmtheorem2)and[2](https://arxiv.org/html/2610.28975#Thmcorollary2)\. Whendjd\_\{j\}is varied,D^\\widehat\{D\}grows more slowly for the corresponding gradient𝑮j\\bm\{G\}\_\{j\}than for the other hidden\-layer gradients\. This is consistent withD=Θ⁡\(dj\)D=\\Theta\(d\_\{j\}\)for𝑮j\\bm\{G\}\_\{j\}, compared withD=Θ⁡\(dj2\)D=\\Theta\(d\_\{j\}^\{2\}\)for𝑮i\\bm\{G\}\_\{i\},i≠ji\\neq j, when the remaining dimensions are fixed\.

[Corollary2](https://arxiv.org/html/2610.28975#Thmcorollary2)gives zero expected off\-diagonal entries fori<Li<L, and the remaining nonzero values are finite\-sample Monte Carlo variances\. For the output layer, the normalizedD^\\widehat\{D\}curves show approximately quadratic growth, while the normalizedOD^\\widehat\{\\mathrm\{OD\}\}curves grow approximately linearly when any one ofd1,d2,d3d\_\{1\},d\_\{2\},d\_\{3\}is varied\. This is consistent withD=Θ⁡\(dj2\)D=\\Theta\(d\_\{j\}^\{2\}\),OD=Θ⁡\(dj\)\\mathrm\{OD\}=\\Theta\(d\_\{j\}\), and henceR=Θ⁡\(dj\)R=\\Theta\(d\_\{j\}\)\.

### G\.3Two\-Layer ReLU Network

#### Setup\.

We next verify the predictions of[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)for the two\-layer ReLU networkf⁡\(𝒙\)=𝑾2​σ​\(𝑾1​𝒙\)f\(\\bm\{x\}\)=\\bm\{W\}\_\{2\}\\sigma\(\\bm\{W\}\_\{1\}\\bm\{x\}\), with hidden dimensiond1d\_\{1\}\. Unlike the hidden layers of the deep linear network, the expected gradient self outer\-products are not exactly diagonal\.

Under the Gaussian initialization of[Assumption3](https://arxiv.org/html/2610.28975#Thmassumption3),[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)gives, for𝑮1\\bm\{G\}\_\{1\},

𝔼⁡\[\(𝑮1​𝑮1⊤\)a​a\]=d0​d224\+d0​d2​\(\(d1−1\)​κ1\+12\)\+‖𝚽‖F24,\\mathbb\{E\}\\\!\\left\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{aa\}\\right\]=\\frac\{d\_\{0\}d\_\{2\}^\{2\}\}\{4\}\+d\_\{0\}d\_\{2\}\\left\(\(d\_\{1\}\-1\)\\kappa\_\{1\}\+\\frac\{1\}\{2\}\\right\)\+\\frac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\},\(109\)while fora≠ca\\neq c,

𝔼⁡\[\(𝑮1​𝑮1⊤\)a​c\]=d0​d2​κ2\+O⁡\(d2​d0\),\\mathbb\{E\}\\\!\\left\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{ac\}\\right\]=d\_\{0\}d\_\{2\}\\kappa\_\{2\}\+O\(d\_\{2\}\\sqrt\{d\_\{0\}\}\),\(110\)whereκ1\\kappa\_\{1\}andκ2\\kappa\_\{2\}are defined in[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)and depend only ond0d\_\{0\}\. Holdingd0d\_\{0\}andd2d\_\{2\}fixed while varyingd1d\_\{1\}therefore gives

D⁡\(𝔼⁡\[𝑮1​𝑮1⊤\]\)=Θ⁡\(d1\),OD⁡\(𝔼⁡\[𝑮1​𝑮1⊤\]\)=Θ⁡\(1\),R⁡\(𝔼⁡\[𝑮1​𝑮1⊤\]\)=Θ⁡\(d1\)\.D\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]\\right\)=\\Theta\(d\_\{1\}\),\\qquad\\mathrm\{OD\}\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]\\right\)=\\Theta\(1\),\\qquad R\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]\\right\)=\\Theta\(d\_\{1\}\)\.\(111\)
For𝑮2\\bm\{G\}\_\{2\},[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)gives

𝔼⁡\[𝑮2​𝑮2⊤\]=α​𝑰d2\+d14​𝚽​𝚽⊤,α=Θ⁡\(d12\)\.\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]=\\alpha\\bm\{I\}\_\{d\_\{2\}\}\+\\frac\{d\_\{1\}\}\{4\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},\\qquad\\alpha=\\Theta\(d\_\{1\}^\{2\}\)\.\(112\)Thus, for fixedd0d\_\{0\},d2d\_\{2\}, and a fixed target with nonzero off\-diagonal structure,

D⁡\(𝔼⁡\[𝑮2​𝑮2⊤\]\)=Θ⁡\(d12\),OD⁡\(𝔼⁡\[𝑮2​𝑮2⊤\]\)=Θ⁡\(d1\),R⁡\(𝔼⁡\[𝑮2​𝑮2⊤\]\)=Θ⁡\(d1\)\.D\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]\\right\)=\\Theta\(d\_\{1\}^\{2\}\),\\qquad\\mathrm\{OD\}\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]\\right\)=\\Theta\(d\_\{1\}\),\\qquad R\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]\\right\)=\\Theta\(d\_\{1\}\)\.\(113\)

#### Empirical structure of the expected gradient Gram\.

We first examine the empirical expected gradient Grams at a representative configuration\. We use\[d0,d1,d2\]=\[512,80,12\]\[d\_\{0\},d\_\{1\},d\_\{2\}\]=\[512,80,12\]and a fixed target𝚽∈ℝ12×512\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{12\\times 512\}whose entries are sampled from𝒩⁡\(0,9\)\\mathcal\{N\}\(0,9\)\. For each ofS=10,000S=10\{,\}000independent Gaussian weight initializations, we estimate the population gradients using128,000128\{,\}000independent inputs sampled from𝒩⁡\(𝟎,𝑰d0\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\), and estimate the expected gradient Grams using \([100](https://arxiv.org/html/2610.28975#A7.E100)\)\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/exp21_relu_empirical_grams.png)Figure 11:Empirical expected gradient self outer\-products for the two\-layer ReLU network with dimensions\[d0,d1,d2\]=\[512,80,12\]\[d\_\{0\},d\_\{1\},d\_\{2\}\]=\[512,80,12\], estimated from10,00010\{,\}000independent weight initializations\. The panels show𝔼^​\[𝑮1​𝑮1⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]and𝔼^​\[𝑮2​𝑮2⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\]\. Each heatmap is normalized by its empirical mean diagonal magnitude\. A common logarithmic color scale is used across all panels\.Figure[11](https://arxiv.org/html/2610.28975#A7.F11)shows diagonally dominant empirical expected Grams for both𝑮1\\bm\{G\}\_\{1\}and𝑮2\\bm\{G\}\_\{2\}, with smaller but nonzero off\-diagonal entries\. This is consistent with[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3), which predicts asymptotic diagonal dominance\.

#### Dimension scaling\.

We setd0=512d\_\{0\}=512andd2=12d\_\{2\}=12, and varyd1∈\{5,10,20,40,60,80,120,160,200,240,280,320,400,480,640\}d\_\{1\}\\in\\\{5,\\allowbreak 10,\\allowbreak 20,\\allowbreak 40,\\allowbreak 60,\\allowbreak 80,\\allowbreak 120,\\allowbreak 160,\\allowbreak 200,\\allowbreak 240,\\allowbreak 280,\\allowbreak 320,\\allowbreak 400,\\allowbreak 480,\\allowbreak 640\\\}\. The same fixed target is used throughout the sweep, and each population gradient is estimated from128,000128\{,\}000Gaussian inputs for each ofS=10,000S=10\{,\}000independent weight initializations\.

Because the expected off\-diagonal entries are much smaller than the diagonal entries, direct Monte Carlo estimation ofOD\\mathrm\{OD\}is noisy\. We therefore use the following expressions to reduce this Monte Carlo variance while preserving the expected off\-diagonal entries\.

For𝑮1\\bm\{G\}\_\{1\}, \([58](https://arxiv.org/html/2610.28975#A4.E58)\) gives

𝑮1=𝑮1\(1\)−𝑮1\(2\)\.\\bm\{G\}\_\{1\}=\\bm\{G\}\_\{1\}^\{\(1\)\}\-\\bm\{G\}\_\{1\}^\{\(2\)\}\.\(114\)By \([59](https://arxiv.org/html/2610.28975#A4.E59)\), the cross terms vanish in expectation, and[Lemma24](https://arxiv.org/html/2610.28975#Thmlemma24)gives

𝑮1\(2\)=12​𝑾2⊤​𝚽,𝔼⁡\[𝑮1\(2\)​\(𝑮1\(2\)\)⊤\]=‖𝚽‖F24​𝑰d1\.\\bm\{G\}\_\{1\}^\{\(2\)\}=\\frac\{1\}\{2\}\\bm\{W\}\_\{2\}^\{\\top\}\\bm\{\\Phi\},\\qquad\\mathbb\{E\}\\\!\\left\[\\bm\{G\}\_\{1\}^\{\(2\)\}\(\\bm\{G\}\_\{1\}^\{\(2\)\}\)^\{\\top\}\\right\]=\\frac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\}\\bm\{I\}\_\{d\_\{1\}\}\.\(115\)Hence,

𝔼⁡\[𝑮1​𝑮1⊤\]=𝔼⁡\[𝑮1\(1\)​\(𝑮1\(1\)\)⊤\]\+‖𝚽‖F24​𝑰d1\.\\mathbb\{E\}\[\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\]=\\mathbb\{E\}\\\!\\left\[\\bm\{G\}\_\{1\}^\{\(1\)\}\(\\bm\{G\}\_\{1\}^\{\(1\)\}\)^\{\\top\}\\right\]\+\\frac\{\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{4\}\\bm\{I\}\_\{d\_\{1\}\}\.\(116\)The second term is diagonal, so all expected off\-diagonal entries come from𝑮1\(1\)\\bm\{G\}\_\{1\}^\{\(1\)\}\. Moreover, setting𝚽=𝟎\\bm\{\\Phi\}=\\bm\{0\}in \([58](https://arxiv.org/html/2610.28975#A4.E58)\) gives𝑮1​\(𝟎\)=𝑮1\(1\)\\bm\{G\}\_\{1\}\(\\bm\{0\}\)=\\bm\{G\}\_\{1\}^\{\(1\)\}\. For eachs∈\{1,…,S\}s\\in\\\{1,\\ldots,S\\\}, we therefore define

𝚫1\(s\)=𝑮1\(s\)​\(𝟎\)​𝑮1\(s\)​\(𝟎\)⊤\.\\bm\{\\Delta\}\_\{1\}^\{\(s\)\}=\\bm\{G\}\_\{1\}^\{\(s\)\}\(\\bm\{0\}\)\\bm\{G\}\_\{1\}^\{\(s\)\}\(\\bm\{0\}\)^\{\\top\}\.\(117\)Fora≠ca\\neq c,

𝔼⁡\[\(𝚫1\(s\)\)a​c\]=𝔼⁡\[\(𝑮1​𝑮1⊤\)a​c\]\.\\mathbb\{E\}\\\!\\left\[\(\\bm\{\\Delta\}\_\{1\}^\{\(s\)\}\)\_\{ac\}\\right\]=\\mathbb\{E\}\\\!\\left\[\(\\bm\{G\}\_\{1\}\\bm\{G\}\_\{1\}^\{\\top\}\)\_\{ac\}\\right\]\.\(118\)Thus, \([117](https://arxiv.org/html/2610.28975#A7.E117)\) preserves the expected off\-diagonal entries while removing the diagonal target contribution\.

For𝑮2\\bm\{G\}\_\{2\},[Lemma22](https://arxiv.org/html/2610.28975#Thmlemma22)gives𝑮2=𝑾2​𝑯−𝚽​𝑪⊤\.\\bm\{G\}\_\{2\}=\\bm\{W\}\_\{2\}\\bm\{H\}\-\\bm\{\\Phi\}\\bm\{C\}^\{\\top\}\.For eachs∈\{1,…,S\}s\\in\\\{1,\\ldots,S\\\}, let𝑮2\(s\)​\(𝚽\)\\bm\{G\}\_\{2\}^\{\(s\)\}\(\\bm\{\\Phi\}\)denote the output\-layer gradient for thess\-th weight initialization and target𝚽\\bm\{\\Phi\}\. Using the same weights and input samples, we evaluate the gradient at𝚽\\bm\{\\Phi\},−𝚽\-\\bm\{\\Phi\}, and𝟎\\bm\{0\}, and define

𝚫2\(s\)=12​\[𝑮2\(s\)​\(𝚽\)​𝑮2\(s\)​\(𝚽\)⊤\+𝑮2\(s\)​\(−𝚽\)​𝑮2\(s\)​\(−𝚽\)⊤\]−𝑮2\(s\)​\(𝟎\)​𝑮2\(s\)​\(𝟎\)⊤\.\\bm\{\\Delta\}\_\{2\}^\{\(s\)\}=\\frac\{1\}\{2\}\\left\[\\bm\{G\}\_\{2\}^\{\(s\)\}\(\\bm\{\\Phi\}\)\\bm\{G\}\_\{2\}^\{\(s\)\}\(\\bm\{\\Phi\}\)^\{\\top\}\+\\bm\{G\}\_\{2\}^\{\(s\)\}\(\-\\bm\{\\Phi\}\)\\bm\{G\}\_\{2\}^\{\(s\)\}\(\-\\bm\{\\Phi\}\)^\{\\top\}\\right\]\-\\bm\{G\}\_\{2\}^\{\(s\)\}\(\\bm\{0\}\)\\bm\{G\}\_\{2\}^\{\(s\)\}\(\\bm\{0\}\)^\{\\top\}\.\(119\)Substituting[Lemma22](https://arxiv.org/html/2610.28975#Thmlemma22)into \([119](https://arxiv.org/html/2610.28975#A7.E119)\) cancels the terms independent of𝚽\\bm\{\\Phi\}and linear in𝚽\\bm\{\\Phi\}, leaving

𝚫2\(s\)=𝚽​\(𝑪\(s\)\)⊤​𝑪\(s\)​𝚽⊤\.\\bm\{\\Delta\}\_\{2\}^\{\(s\)\}=\\bm\{\\Phi\}\(\\bm\{C\}^\{\(s\)\}\)^\{\\top\}\\bm\{C\}^\{\(s\)\}\\bm\{\\Phi\}^\{\\top\}\.\(120\)By[Lemma23](https://arxiv.org/html/2610.28975#Thmlemma23),𝑪=12​𝑾1\\bm\{C\}=\\tfrac\{1\}\{2\}\\bm\{W\}\_\{1\}, and[Lemma16](https://arxiv.org/html/2610.28975#Thmlemma16)gives𝔼⁡\[𝑾1⊤​𝑾1\]=d1​𝑰d0\\mathbb\{E\}\[\\bm\{W\}\_\{1\}^\{\\top\}\\bm\{W\}\_\{1\}\]=d\_\{1\}\\bm\{I\}\_\{d\_\{0\}\}\. Hence

𝔼⁡\[𝚫2\(s\)\]=d14​𝚽​𝚽⊤\.\\mathbb\{E\}\[\\bm\{\\Delta\}\_\{2\}^\{\(s\)\}\]=\\frac\{d\_\{1\}\}\{4\}\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\.\(121\)Since the remaining termα​𝑰d2\\alpha\\bm\{I\}\_\{d\_\{2\}\}in \([112](https://arxiv.org/html/2610.28975#A7.E112)\) is diagonal, \([119](https://arxiv.org/html/2610.28975#A7.E119)\) has exactly the same expected off\-diagonal entries as the full𝑮2​𝑮2⊤\\bm\{G\}\_\{2\}\\bm\{G\}\_\{2\}^\{\\top\}\.

For both layers, we computeD^\\widehat\{D\}from the full empirical expected Gram\. For𝑮1\\bm\{G\}\_\{1\}, we computeOD^\\widehat\{\\mathrm\{OD\}\}from the zero\-target empirical Gram\. For𝑮2\\bm\{G\}\_\{2\}, we first average𝚫2\(s\)\\bm\{\\Delta\}\_\{2\}^\{\(s\)\}over the sameSSinitializations and then applyOD⁡\(⋅\)\\mathrm\{OD\}\(\\cdot\)\. We then computeR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}\.

Figure 12:Hidden\-dimension scaling for the two\-layer ReLU network withd0=512d\_\{0\}=512andd2=12d\_\{2\}=12\. The top and bottom rows correspond to𝑮1\\bm\{G\}\_\{1\}and𝑮2\\bm\{G\}\_\{2\}, respectively, and showD^\\widehat\{D\},OD^\\widehat\{\\mathrm\{OD\}\}, andR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}asd1d\_\{1\}is varied\. For𝑮1\\bm\{G\}\_\{1\},OD^\\widehat\{\\mathrm\{OD\}\}is estimated from the zero\-target Gram; for𝑮2\\bm\{G\}\_\{2\}, it is estimated using \([119](https://arxiv.org/html/2610.28975#A7.E119)\)\.Figure[12](https://arxiv.org/html/2610.28975#A7.F12)shows the different hidden\-dimension scalings predicted by[Theorem3](https://arxiv.org/html/2610.28975#Thmtheorem3)\. For𝑮1\\bm\{G\}\_\{1\},D^\\widehat\{D\}grows approximately linearly withd1d\_\{1\}, whileOD^\\widehat\{\\mathrm\{OD\}\}remains approximately constant, consistent withD=Θ⁡\(d1\)D=\\Theta\(d\_\{1\}\)andOD=Θ⁡\(1\)\\mathrm\{OD\}=\\Theta\(1\)in \([111](https://arxiv.org/html/2610.28975#A7.E111)\)\. For𝑮2\\bm\{G\}\_\{2\},D^\\widehat\{D\}grows approximately quadratically withd1d\_\{1\}, whileOD^\\widehat\{\\mathrm\{OD\}\}grows approximately linearly, consistent with \([113](https://arxiv.org/html/2610.28975#A7.E113)\)\. Accordingly,R^\\widehat\{R\}increases withd1d\_\{1\}for both layers, consistent with the predictedR=Θ⁡\(d1\)R=\\Theta\(d\_\{1\}\)\.

### G\.4Single\-Layer Attention

#### Setup\.

Finally, we verify the predictions of[Theorems4](https://arxiv.org/html/2610.28975#Thmtheorem4)and[3](https://arxiv.org/html/2610.28975#Thmcorollary3)for single\-layer attention defined in \([8](https://arxiv.org/html/2610.28975#S3.E8)\)\. We consider the gradients𝑮Q,𝑮K,𝑮V,𝑮O\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{V\},\\bm\{G\}\_\{O\}of the query, key, value, and output projections, respectively\. Under the Gaussian initialization of[Assumption4](https://arxiv.org/html/2610.28975#Thmassumption4),[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)gives

𝔼⁡\[𝑮Q​𝑮Q⊤\]\\displaystyle\\mathbb\{E\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\]=cQ​𝑰dk,\\displaystyle=c\_\{Q\}\\bm\{I\}\_\{d\_\{k\}\},\(122\)𝔼⁡\[𝑮K​𝑮K⊤\]\\displaystyle\\mathbb\{E\}\[\\bm\{G\}\_\{K\}\\bm\{G\}\_\{K\}^\{\\top\}\]=cK​𝑰dk,\\displaystyle=c\_\{K\}\\bm\{I\}\_\{d\_\{k\}\},𝔼⁡\[𝑮V​𝑮V⊤\]\\displaystyle\\mathbb\{E\}\[\\bm\{G\}\_\{V\}\\bm\{G\}\_\{V\}^\{\\top\}\]=cV​𝑰dv,\\displaystyle=c\_\{V\}\\bm\{I\}\_\{d\_\{v\}\},𝔼⁡\[𝑮O​𝑮O⊤\]\\displaystyle\\mathbb\{E\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\]=α¯O​𝑰d2\+dv​γ​𝚽​𝚽⊤,\\displaystyle=\\bar\{\\alpha\}\_\{O\}\\bm\{I\}\_\{d\_\{2\}\}\+d\_\{v\}\\gamma\\,\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\},wherecQ,cK,cV,α¯Oc\_\{Q\},c\_\{K\},c\_\{V\},\\bar\{\\alpha\}\_\{O\}, andγ\\gammaare given explicitly in[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)and[SectionE\.3](https://arxiv.org/html/2610.28975#A5.SS3)\. Thus, the expected gradient self outer\-products for the query, key, and value projections are exactly diagonal,

OD⁡\(𝔼⁡\[𝑮p​𝑮p⊤\]\)=0,p∈\{Q,K,V\}\.\\mathrm\{OD\}\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{p\}\\bm\{G\}\_\{p\}^\{\\top\}\]\\right\)=0,\\qquad p\\in\\\{Q,K,V\\\}\.\(123\)Combining[Theorems4](https://arxiv.org/html/2610.28975#Thmtheorem4)and[3](https://arxiv.org/html/2610.28975#Thmcorollary3), for a fixed target withOD⁡\(𝚽​𝚽⊤\)\>0\\mathrm\{OD\}\(\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\)\>0, we obtain

R⁡\(𝔼⁡\[𝑮O​𝑮O⊤\]\)=Θ⁡\(dv​d03\)\.R\\\!\\left\(\\mathbb\{E\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\]\\right\)=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)\.\(124\)Hence increasing the hidden dimensiondvd\_\{v\}or the token dimensiond0d\_\{0\}makes the expected output\-projection Gram increasingly diagonally dominant, while the hidden dimensiondkd\_\{k\}enters the ratio only through a bounded factor\.

#### Empirical structure of the expected gradient Gram\.

We first examine the empirical expected gradient Grams at a representative configuration\. We use linear attention withT=16T=16,d0=32d\_\{0\}=32,dk=dv=20d\_\{k\}=d\_\{v\}=20, andd2=12d\_\{2\}=12\. The entries of the fixed target𝚽∈ℝd2×d0\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\_\{0\}\}are sampled from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\. We useS=10,000S=10\{,\}000independent initializations\. For each initialization, the rows of𝑿∈ℝT×d0\\bm\{X\}\\in\\mathbb\{R\}^\{T\\times d\_\{0\}\}are sampled independently from𝒩⁡\(𝟎,𝑰d0\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\), independently of the Gaussian weights, as specified in[Assumption4](https://arxiv.org/html/2610.28975#Thmassumption4)\. We estimate the expected gradient Gram for each projection using \([100](https://arxiv.org/html/2610.28975#A7.E100)\)\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/exp21_attention_empirical_grams.png)Figure 13:Empirical expected gradient self outer\-products for single\-layer attention withT=16T=16,d0=32d\_\{0\}=32,dk=dv=20d\_\{k\}=d\_\{v\}=20, andd2=12d\_\{2\}=12, estimated from10,00010\{,\}000independent initializations\. The four panels show𝔼^​\[𝑮p​𝑮p⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{p\}\\bm\{G\}\_\{p\}^\{\\top\}\]forp∈\{Q,K,V,O\}p\\in\\\{Q,K,V,O\\\}\. Each heatmap is normalized by its empirical mean diagonal magnitude\. A common logarithmic color scale is used across all panels\.Figure[13](https://arxiv.org/html/2610.28975#A7.F13)shows strongly diagonal empirical expected Grams for the query, key, and value projections, consistent with the exact isotropy in[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)\. The output\-projection Gram is also diagonally dominant but retains nonzero off\-diagonal structure, consistent with thedv​γ​𝚽​𝚽⊤d\_\{v\}\\gamma\\,\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}term in \([122](https://arxiv.org/html/2610.28975#A7.E122)\)\.

#### Dimension scaling\.

We examine four dimension sweeps:dk=dvd\_\{k\}=d\_\{v\}jointly,dkd\_\{k\}withdv=80d\_\{v\}=80,dvd\_\{v\}withdk=80d\_\{k\}=80, andd0=d2d\_\{0\}=d\_\{2\}jointly\. For the first three sweeps, we setT=32T=32,d0=512d\_\{0\}=512, andd2=12d\_\{2\}=12, and use the 12\-point grid\{5,10,20,40,60,80,120,160,200,240,280,320\}\\\{5,10,20,40,60,80,120,160,200,240,280,320\\\}\. For the jointd0=d2d\_\{0\}=d\_\{2\}sweep, we setT=16T=16anddk=dv=20d\_\{k\}=d\_\{v\}=20, and use the 13\-point grid\{32,64,128,192,256,320,384,448,512,640,768,896,1024\}\\\{32,64,128,192,256,320,384,448,512,640,768,896,1024\\\}\. Further, for the jointd0=d2d\_\{0\}=d\_\{2\}sweep, we rescale the target𝚽\\bm\{\\Phi\}at each dimension to have the same Frobenius norm as the32×3232\\times 32reference target\. This controls for the explicit target dependence in[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4): the diagonal constantscQ,cK,cVc\_\{Q\},c\_\{K\},c\_\{V\}contain terms proportional to‖𝚽‖F2\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}in \([68](https://arxiv.org/html/2610.28975#A5.E68)\), while the output Gram contains the termdv​γ​𝚽​𝚽⊤d\_\{v\}\\gamma\\,\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}in \([69](https://arxiv.org/html/2610.28975#A5.E69)\)\. Holding‖𝚽‖F\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}fixed therefore isolates the effect of increasingd0=d2d\_\{0\}=d\_\{2\}from the increase in target magnitude from the larger target matrix\. For every configuration, we useS=10,000S=10\{,\}000independent initializations\.For each initialization, the rows of𝑿\\bm\{X\}are sampled independently from𝒩⁡\(𝟎,𝑰d0\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\_\{d\_\{0\}\}\), independently of the weights\.

Since the expected off\-diagonal entries are much smaller than the diagonal entries, direct Monte Carlo estimation ofOD\\mathrm\{OD\}is noisy\. We therefore use the following expressions to reduce this Monte Carlo variance while preserving the expected off\-diagonal entries\. From \([65](https://arxiv.org/html/2610.28975#A5.E65)\) and \([67](https://arxiv.org/html/2610.28975#A5.E67)\), each gradient is affine in𝚽\\bm\{\\Phi\}when the weights and𝑿\\bm\{X\}are fixed\. For eachs∈\{1,…,S\}s\\in\\\{1,\\ldots,S\\\}, let𝑮p\(s\)​\(𝚽\)\\bm\{G\}\_\{p\}^\{\(s\)\}\(\\bm\{\\Phi\}\)denote the gradient for projectionp∈\{Q,K,V,O\}p\\in\\\{Q,K,V,O\\\}under thess\-th initialization and target𝚽\\bm\{\\Phi\}\. Using the same weights and𝑿\\bm\{X\}, we evaluate the gradient at𝚽\\bm\{\\Phi\},−𝚽\-\\bm\{\\Phi\}, and𝟎\\bm\{0\}, and define

𝚫p\(s\)=12​\[𝑮p\(s\)​\(𝚽\)​𝑮p\(s\)​\(𝚽\)⊤\+𝑮p\(s\)​\(−𝚽\)​𝑮p\(s\)​\(−𝚽\)⊤\]−𝑮p\(s\)​\(𝟎\)​𝑮p\(s\)​\(𝟎\)⊤\.\\bm\{\\Delta\}\_\{p\}^\{\(s\)\}=\\frac\{1\}\{2\}\\left\[\\bm\{G\}\_\{p\}^\{\(s\)\}\(\\bm\{\\Phi\}\)\\bm\{G\}\_\{p\}^\{\(s\)\}\(\\bm\{\\Phi\}\)^\{\\top\}\+\\bm\{G\}\_\{p\}^\{\(s\)\}\(\-\\bm\{\\Phi\}\)\\bm\{G\}\_\{p\}^\{\(s\)\}\(\-\\bm\{\\Phi\}\)^\{\\top\}\\right\]\-\\bm\{G\}\_\{p\}^\{\(s\)\}\(\\bm\{0\}\)\\bm\{G\}\_\{p\}^\{\(s\)\}\(\\bm\{0\}\)^\{\\top\}\.\(125\)The symmetric combination cancels the terms independent of𝚽\\bm\{\\Phi\}and linear in𝚽\\bm\{\\Phi\}, leaving the contribution quadratic in the target\. Combining this cancellation with the target\-dependent terms in \([68](https://arxiv.org/html/2610.28975#A5.E68)\) and \([69](https://arxiv.org/html/2610.28975#A5.E69)\) gives

𝔼⁡\[𝚫Q\(s\)\]\\displaystyle\\mathbb\{E\}\[\\bm\{\\Delta\}\_\{Q\}^\{\(s\)\}\]=dv​γdk​‖𝚽‖F2​𝑰dk,\\displaystyle=\\frac\{d\_\{v\}\\gamma\}\{d\_\{k\}\}\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bm\{I\}\_\{d\_\{k\}\},\(126\)𝔼⁡\[𝚫K\(s\)\]\\displaystyle\\mathbb\{E\}\[\\bm\{\\Delta\}\_\{K\}^\{\(s\)\}\]=dv​γdk​‖𝚽‖F2​𝑰dk,\\displaystyle=\\frac\{d\_\{v\}\\gamma\}\{d\_\{k\}\}\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bm\{I\}\_\{d\_\{k\}\},𝔼⁡\[𝚫V\(s\)\]\\displaystyle\\mathbb\{E\}\[\\bm\{\\Delta\}\_\{V\}^\{\(s\)\}\]=γ​‖𝚽‖F2​𝑰dv,\\displaystyle=\\gamma\\left\\\|\\bm\{\\Phi\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\bm\{I\}\_\{d\_\{v\}\},𝔼⁡\[𝚫O\(s\)\]\\displaystyle\\mathbb\{E\}\[\\bm\{\\Delta\}\_\{O\}^\{\(s\)\}\]=dv​γ​𝚽​𝚽⊤\.\\displaystyle=d\_\{v\}\\gamma\\,\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\top\}\.Thus,𝚫p\(s\)\\bm\{\\Delta\}\_\{p\}^\{\(s\)\}in \([125](https://arxiv.org/html/2610.28975#A7.E125)\) has zero expected off\-diagonal entries forp∈\{Q,K,V\}p\\in\\\{Q,K,V\\\}, exactly as in \([123](https://arxiv.org/html/2610.28975#A7.E123)\), while forp=Op=Oit preserves the off\-diagonal term in \([122](https://arxiv.org/html/2610.28975#A7.E122)\)\.

For each configuration, we computeD^\\widehat\{D\}from the full empirical expected Gram\. ForOD^\\widehat\{\\mathrm\{OD\}\}, we first average𝚫p\(s\)\\bm\{\\Delta\}\_\{p\}^\{\(s\)\}over the sameSSinitializations and then applyOD⁡\(⋅\)\\mathrm\{OD\}\(\\cdot\)\. We then computeR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}\.

Figure 14:Empirical dimension scaling for single\-layer attention\. Rows varydk=dvd\_\{k\}=d\_\{v\}jointly,dkd\_\{k\},dvd\_\{v\}, andd0=d2d\_\{0\}=d\_\{2\}\(top to bottom\)\. Columns showD^\\widehat\{D\},OD^\\widehat\{\\mathrm\{OD\}\}, andR^=D^/OD^\\widehat\{R\}=\\widehat\{D\}/\\widehat\{\\mathrm\{OD\}\}for𝑮Q,𝑮K,𝑮V,\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{V\},and𝑮O\\bm\{G\}\_\{O\}, each normalized by its value at the baseline configuration \(first three rows:T=32T=32,d0=512d\_\{0\}=512,dk=dv=80d\_\{k\}=d\_\{v\}=80,d2=12d\_\{2\}=12; fourth row:T=16,d0=d2=32,dk=dv=20T=16,d\_\{0\}=d\_\{2\}=32,d\_\{k\}=d\_\{v\}=20\), to facilitate comparison across projections with different absolute scales\. Empirical estimates use10,00010\{,\}000independent initializations per configuration; the off\-diagonal statistics are estimated using \([125](https://arxiv.org/html/2610.28975#A7.E125)\)\.Figure[14](https://arxiv.org/html/2610.28975#A7.F14)shows the dimension\-dependent diagonal scaling predicted by[Theorems4](https://arxiv.org/html/2610.28975#Thmtheorem4)and[3](https://arxiv.org/html/2610.28975#Thmcorollary3)\. Whendkd\_\{k\}is varied withdvd\_\{v\}fixed,D^\\widehat\{D\}for𝑮Q\\bm\{G\}\_\{Q\}and𝑮K\\bm\{G\}\_\{K\}decreases, consistent withD=Θ⁡\(d22​dv2​d06/dk\)D=\\Theta\(d\_\{2\}^\{2\}d\_\{v\}^\{2\}d\_\{0\}^\{6\}/d\_\{k\}\), while𝑮V\\bm\{G\}\_\{V\}and𝑮O\\bm\{G\}\_\{O\}depend only weakly ondkd\_\{k\}\. Increasingdvd\_\{v\}increasesD^\\widehat\{D\}for all four projections; the stronger growth of𝑮Q,𝑮K,𝑮O\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{O\}relative to𝑮V\\bm\{G\}\_\{V\}is consistent with their quadratic versus linear dependence ondvd\_\{v\}\. Whendk=dvd\_\{k\}=d\_\{v\}are increased jointly,D^\\widehat\{D\}grows approximately linearly for𝑮Q,𝑮K,𝑮V\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{V\}and more rapidly for𝑮O\\bm\{G\}\_\{O\}, again matching the predicted orders\.

ForQQ,KK, andVV, the population off\-diagonal entries are exactly zero by[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4); the nonzeroOD^\\widehat\{\\mathrm\{OD\}\}values are finite\-sample Monte Carlo variances\. For the output projection,OD^\\widehat\{\\mathrm\{OD\}\}grows approximately linearly withdvd\_\{v\}and is nearly unchanged when onlydkd\_\{k\}is varied, whileR^\\widehat\{R\}grows approximately linearly withdvd\_\{v\}, consistent with[Corollary3](https://arxiv.org/html/2610.28975#Thmcorollary3), which givesD=Θ⁡\(dv2​d06\)D=\\Theta\(d\_\{v\}^\{2\}d\_\{0\}^\{6\}\),OD=Θ⁡\(dv​d03\)\\mathrm\{OD\}=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\), andR=Θ⁡\(dv​d03\)R=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)\. For the jointd0=d2=md\_\{0\}=d\_\{2\}=msweep,[Corollary3](https://arxiv.org/html/2610.28975#Thmcorollary3)givesD=Θ⁡\(m8\)D=\\Theta\(m^\{8\}\)for𝑮Q,𝑮K,𝑮V\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{V\}, while for𝑮O\\bm\{G\}\_\{O\},D=Θ⁡\(m6\)D=\\Theta\(m^\{6\}\),OD=Θ⁡\(m3\)\\mathrm\{OD\}=\\Theta\(m^\{3\}\), and henceR=Θ⁡\(m3\)R=\\Theta\(m^\{3\}\)\.

## Appendix HEmpirical Evaluation on Large Language Models

We next test whether the diagonal dominance behavior of[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)persists in transformer language models\. We consider Qwen3\.5\-0\.8B\-Base\([Qwen Team, 2026](https://arxiv.org/html/2610.28975#bib.bib80)\), Granite\-4\.0\-1B\-Base\([IBM Granite Team, 2025](https://arxiv.org/html/2610.28975#bib.bib81)\), and Llama\-3\.2\-1B\([Meta AI, 2024](https://arxiv.org/html/2610.28975#bib.bib82)\)\. We perform two experiments\. First, we examine the expected gradient Grams of individual attention projections at random initialization\. Second, we measure how their diagonal dominance changes with model width across the three models\.

### H\.1Gradient Gram Structure at Initialization

We first examine the gradient Gram of attention projections in Qwen3\.5\-0\.8B\-Base, Granite\-4\.0\-1B\-Base, and Llama\-3\.2\-1B at random initialization\. The models are instantiated from their released Hugging Face configurations\([Wolf et al\., 2020](https://arxiv.org/html/2610.28975#bib.bib90)\)without pretrained weights\. We set the sequence length toT=256T=256, retain each model’s released model width and head dimension, and set the number of key/value heads equal to the number of query heads\. Native RoPE behavior is used throughout\.

We analyze head00in block33for Qwen and in block00for Granite and Llama\. For Qwen, DeltaNet blocks are excluded, so block33is the first full\-attention block considered\. For each projectionp∈\{Q,K,V,O\}p\\in\\\{Q,K,V,O\\\}, we estimate

𝑷^p=1NW​NX​∑w=1NW∑x=1NX𝑮p,w,x​𝑮p,w,x⊤,\\widehat\{\\bm\{P\}\}\_\{p\}=\\frac\{1\}\{N\_\{W\}N\_\{X\}\}\\sum\_\{w=1\}^\{N\_\{W\}\}\\sum\_\{x=1\}^\{N\_\{X\}\}\\bm\{G\}\_\{p,w,x\}\\bm\{G\}\_\{p,w,x\}^\{\\top\},\(127\)usingNW=25N\_\{W\}=25independent random initializations andNX=64N\_\{X\}=64fixed FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.28975#bib.bib83)\)sequences\. We useNW=25N\_\{W\}=25independent random initializations obtained from each model’s native Hugging Face initialization scheme, which uses zero\-mean Gaussian initialization for the attention projection weights with model\-specific scales\. The same input sequences are reused across all initializations\. For Qwen, the gating rows of the query projection are excluded when forming the query Gram\.

Figure[15](https://arxiv.org/html/2610.28975#A8.F15)shows the resulting empirical expected Grams\. All four attention projections exhibit a clear diagonal component across all three models\. This indicates that the diagonal structure predicted for the single\-layer attention model in[Theorem4](https://arxiv.org/html/2610.28975#Thmtheorem4)also appears in deeper transformer architectures at random initialization\.

![Refer to caption](https://arxiv.org/html/2610.28975v1/llm_head0_QKVO_empirical_grams_4x3.png)Figure 15:Empirical expected gradient self outer\-products for Qwen3\.5\-0\.8B\-Base, Granite\-4\.0\-1B\-Base, and Llama\-3\.2\-1B \(columns\), estimated from2525independent random initializations and6464fixed FineWeb\-Edu sequences of lengthT=256T=256\. The rows show𝔼^​\[𝑮Q​𝑮Q⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{Q\}\\bm\{G\}\_\{Q\}^\{\\top\}\],𝔼^​\[𝑮K​𝑮K⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{K\}\\bm\{G\}\_\{K\}^\{\\top\}\],𝔼^​\[𝑮V​𝑮V⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{V\}\\bm\{G\}\_\{V\}^\{\\top\}\], and𝔼^​\[𝑮O​𝑮O⊤\]\\widehat\{\\mathbb\{E\}\}\[\\bm\{G\}\_\{O\}\\bm\{G\}\_\{O\}^\{\\top\}\]\. We use head00of block33for Qwen and head00of block00for Granite and Llama\. Each heatmap is normalized by its empirical mean diagonal magnitude\. A common logarithmic color scale is used across all panels\.
### H\.2Model Width Scaling

We next examine how diagonal dominance changes with model width\. Using the same random initialization protocol as above, we vary the transformer hidden width \(hidden\_size\) while holding the head dimension \(head\_dim\) fixed at its released value\. In the notation of our single\-layer attention model, this corresponds to varying the residual input and output dimensionsd0=d2d\_\{0\}=d\_\{2\}while keepingdk=dvd\_\{k\}=d\_\{v\}fixed\. The released head dimensions are256256for Qwen3\.5\-0\.8B\-Base,128128for Granite\-4\.0\-1B\-Base, and6464for Llama\-3\.2\-1B\. The query\-head count \(num\_attention\_heads\) is held fixed, and the number of key/value heads \(num\_key\_value\_heads\) is set equal to the number of query heads to enable analogous dimension sweep in the models\. The sequence length is fixed atT=256T=256\.

We consider the model\-width grid

\{128,256,512,768,1024,1536,2048,3072,4096,6144,8192,12288,16384\}\.\\\{128,256,512,768,1024,1536,2048,3072,4096,6144,8192,12288,16384\\\}\.At each width, we use2525independent random initializations \(obtained from each model’s native Hugging Face initialization scheme, which uses zero\-mean Gaussian initialization for the attention projection weights with model\-specific scales\) and the same6464fixed FineWeb\-Edu sequences\. We analyze head00in block33for Qwen and in block00for Granite and Llama\. For each projectionp∈\{Q,K,V,O\}p\\in\\\{Q,K,V,O\\\}, we report the empirical diagonal\-dominance ratioR^p=R⁡\(𝑷^p\)\\widehat\{R\}\_\{p\}=R\(\\widehat\{\\bm\{P\}\}\_\{p\}\), withR⁡\(⋅\)R\(\\cdot\)defined in \([99](https://arxiv.org/html/2610.28975#A7.E99)\)\.

Figure 16:Model width scaling of the empirical diagonal\-dominance ratioR^\\widehat\{R\}for𝑮Q,𝑮K,𝑮V,\\bm\{G\}\_\{Q\},\\bm\{G\}\_\{K\},\\bm\{G\}\_\{V\},and𝑮O\\bm\{G\}\_\{O\}in Qwen3\.5\-0\.8B\-Base, Granite\-4\.0\-1B\-Base, and Llama\-3\.2\-1B\. Model width \(hidden\_size\) is varied while the head dimension \(head\_dim\) is held fixed at its released value\. We analyze head00in block33for Qwen and in block00for Granite and Llama\. Each point is estimated from2525independent random initializations and6464fixed FineWeb\-Edu sequences of lengthT=256T=256\.Figure[16](https://arxiv.org/html/2610.28975#A8.F16)shows thatR^\\widehat\{R\}increases with model width for all four attention projections across all three models\. For𝑮O\\bm\{G\}\_\{O\}, this is consistent with theR=Θ⁡\(dv​d03\)R=\\Theta\(d\_\{v\}d\_\{0\}^\{3\}\)scaling in[Corollary3](https://arxiv.org/html/2610.28975#Thmcorollary3)\. The increase is strongest for𝑮Q\\bm\{G\}\_\{Q\}and𝑮K\\bm\{G\}\_\{K\}in Qwen, and for𝑮V\\bm\{G\}\_\{V\}and𝑮O\\bm\{G\}\_\{O\}in Granite and Llama\.