Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

# Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
Source: [https://arxiv.org/html/2606.02596](https://arxiv.org/html/2606.02596)
###### Abstract

The curvature exponentα\\alphainhk∝σkαh\_\{k\}\\propto\\sigma\_\{k\}^\{\\alpha\}—governing how Hessian eigenvalues scale with gradient singular values—varies systematically across layer types \(α≈2\\alpha\\approx 2for convolutions,≈1\\approx 1for transformer attention,<1<1for MLP up\-projections\)\. Why? We prove theSpectral Alignment Decomposition:α=2\+d​log⁡Φk/d​log⁡σk\\alpha=2\+d\\log\\Phi\_\{k\}/d\\log\\sigma\_\{k\}, whereΦk\\Phi\_\{k\}measures alignment between Kronecker factor eigenbases and gradient singular directions\. This reduces “why doesα\\alphavary?” to a geometric question we answer for LayerNorm, residual connections, and softmax heads\. The decomposition implies aspectral transfer identitys=α​γs=\\alpha\\gammalinking curvature exponent, effective gradient rank\-decayγ\\gamma, and Hessian decay exponentss\[[1](https://arxiv.org/html/2606.02596#bib.bib1)\]\. The identity is algebraic; its empirical content is thatα\\alphaandγ\\gamma, fit on*independent*data \(HVPs vs\. SVD\), recoverssto≈\\approx2% median error across 93 layers, five architectures, and three datasets—with no free parameters\. A zeta\-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer\. As a proof of concept, we derive the architecture\-adaptive preconditionerT​\(σ;α\)T\(\\sigma;\\alpha\)and show thatSpectral Newton—implementingTTin the gradient singular basis—outperforms AdamW on vision benchmarks whereα≈2\\alpha\\approx 2\.

## 1Introduction

The eigenvalue spectrum of the neural network loss Hessian encodes how optimization navigates parameter space\. Recent work has established*what*the spectrum looks like: Hessian eigenvalues decay as power laws with exponentss\[[1](https://arxiv.org/html/2606.02596#bib.bib1),[5](https://arxiv.org/html/2606.02596#bib.bib5)\], weight matrices exhibit heavy\-tailed spectral densities correlated with generalization\[[2](https://arxiv.org/html/2606.02596#bib.bib2)\], and Kronecker\-factored approximations capture much of the Hessian structure\[[3](https://arxiv.org/html/2606.02596#bib.bib3),[4](https://arxiv.org/html/2606.02596#bib.bib4)\]\. These results describe*different projections*of the same underlying object, but the mechanistic question—*why*sstakes the values it does, and how it connects to gradient structure—remains unanswered\.

We study curvature*along gradient singular directions*\. For layerℓ\\ellwith gradientGℓ=U​Σ​V⊤G\_\{\\ell\}=U\\Sigma V^\{\\top\}, lethkh\_\{k\}denote the exact Hessian eigenvalue along directionuk​vk⊤u\_\{k\}v\_\{k\}^\{\\top\}, measured via Hessian\-vector products \(HVPs\)\. Empirically,hkh\_\{k\}follows a power law in the gradient singular valueσk\\sigma\_\{k\}:

hk=c⋅σkαh\_\{k\}=c\\cdot\\sigma\_\{k\}^\{\\alpha\}\(1\)The exponentα\\alphais*not universal*: it depends on layer type, architecture, and task \(Section[3](https://arxiv.org/html/2606.02596#S3)\)\. Understandingα\\alphais the key to predicting which preconditioner a layer requires\.

Our central contribution is theSpectral Alignment Decomposition\(Theorem[1](https://arxiv.org/html/2606.02596#Thmtheorem1)\): an exact decomposition ofhkh\_\{k\}into Kronecker factor eigenvalues and measurable alignment ratios, yielding an identity forα\\alphain terms of log\-log slopes of these quantities\. We then:

1. 1\.Derive mechanism\-specific predictions for LayerNorm \(α≈1\\alpha\\approx 1\), residual connections, and softmax heads \(α\>2\\alpha\>2\), validated by controlled ablations\.
2. 2\.Establish thespectral transfer identitys=α​γs=\\alpha\\gamma: an algebraic link betweenα\\alpha, an effective rank\-decay exponentγ\\gammafit on gradient singular values, and Hessian decayss\[[1](https://arxiv.org/html/2606.02596#bib.bib1)\], validated on 89 layers across three datasets \(Section[7](https://arxiv.org/html/2606.02596#S7)\)\.
3. 3\.Bound the curvature participation ratio via the Riemann zeta function, showing learning is effectively one\-dimensional per layer\.
4. 4\.Prescribe architecture\-adaptive spectral transfer functionsT​\(σ;α\)T\(\\sigma;\\alpha\)and validateSpectral Newtonas their optimizer instantiation \(Section[7\.3](https://arxiv.org/html/2606.02596#S7.SS3)\)\.

This is a spectral asymptotics result in the spirit of Weyl’s law: the decay exponent encodes geometric information \(alignment structure\) of the loss landscape, not merely a fitted phenomenological parameter\.

## 2Setup and Measurement

For layerℓ\\ellwith weightWℓ∈ℝm×nW\_\{\\ell\}\\in\\mathbb\{R\}^\{m\\times n\}, gradientGℓ=∇WℓℒG\_\{\\ell\}=\\nabla\_\{W\_\{\\ell\}\}\\mathcal\{L\}, and mini\-batch activationsA∈ℝB×nA\\in\\mathbb\{R\}^\{B\\times n\}, errorsδ∈ℝB×m\\delta\\in\\mathbb\{R\}^\{B\\times m\}, the gradient factors asGℓ=1B​δ⊤​AG\_\{\\ell\}=\\frac\{1\}\{B\}\\delta^\{\\top\}A\. Under the Gauss–Newton \(GN\) approximation, the per\-layer Hessian satisfiesℋGN\(ℓ\)≈Cδ⊗CA\\mathcal\{H\}\_\{\\mathrm\{GN\}\}^\{\(\\ell\)\}\\approx C\_\{\\delta\}\\otimes C\_\{A\}withCδ=1B​δ⊤​δC\_\{\\delta\}=\\frac\{1\}\{B\}\\delta^\{\\top\}\\deltaandCA=1B​A⊤​AC\_\{A\}=\\frac\{1\}\{B\}A^\{\\top\}A\.

#### Measurement protocol\.

We train models on CIFAR\-10 and Tiny\-ImageNet\-200 to convergence, and also evaluate pretrained ImageNet\-1K weights \(IMAGENET1K\_V1\)\. For each, we fix a batch ofB=2048B=2048–40964096samples, compute top\-kkgradient singular directions via SVD, and measurehk=vk⊤​ℋ​vkh\_\{k\}=v\_\{k\}^\{\\top\}\\mathcal\{H\}v\_\{k\}via double backpropagation \(exact HVP\)\. We fitα\\alphaby log\-log regression on\(σk,hk\)\(\\sigma\_\{k\},h\_\{k\}\)\. All results use exact HVPs; finite\-difference Hessians yieldR2≈0\.17R^\{2\}\\approx 0\.17with 50% spurious negative curvatures and are excluded\.

#### Exponentsα\\alphaandγ\\gamma\.

We fitα\\alphafrom exact HVPs on\(σk,hk\)\(\\sigma\_\{k\},h\_\{k\}\)\. We defineγ\\gammaas the negated log\-log slope of rank\-orderedσk\\sigma\_\{k\}on top\-kkgradient singular values \(typicallyk=20k\{=\}20;k=100k\{=\}100for extended spectra in Appendix[G](https://arxiv.org/html/2606.02596#A7)\)\. This is an*effective*decay exponent over a finite rank window, not a claim thatσk\\sigma\_\{k\}follows a global power law\.

## 3The Curvature Exponent: An Empirical Landscape

Figure[1](https://arxiv.org/html/2606.02596#S3.F1)visualizes the core empirical lawhk∝σkαh\_\{k\}\\propto\\sigma\_\{k\}^\{\\alpha\}on representative layers; Figure[2](https://arxiv.org/html/2606.02596#S3.F2)shows thatα\\alphais not random noise but tracks architecture and depth\. Table[1](https://arxiv.org/html/2606.02596#S3.T1)summarizes means across models: convolutional layers universally exhibitα≈2\\alpha\\approx 2; transformer layers cluster nearα≈1\\alpha\\approx 1; output heads can exceedα=4\\alpha=4\.

![Refer to caption](https://arxiv.org/html/2606.02596v1/x1.png)Figure 1:Exact HVP curvaturehkh\_\{k\}vs\. gradient singular valueσk\\sigma\_\{k\}\(log–log\)\.\(a–d\)Representative layers: convolutions followα≈2\\alpha\\approx 2; FC and transformer MLP layers deviate\. Fit exponents shown in\-panel; dashed grid omitted on log axes for clarity\.![Refer to caption](https://arxiv.org/html/2606.02596v1/x2.png)Figure 2:Per\-layer curvature exponentα\\alphavs\. depth index across ResNet\-18, VGG\-11, and GPT\-2 \(CIFAR\-10\)\. Shaded band:α∈\[1\.85,2\.15\]\\alpha\\in\[1\.85,2\.15\]\. Interior convolutions cluster nearα=2\\alpha=2; transformers and boundary layers deviate\.Table 1:Mean curvature exponentα\\alphaby architecture and layer type\. CIFAR\-10 unless noted;†Tiny\-ImageNet\-200;‡ImageNet\-1K pretrained \(IMAGENET1K\_V1\)\.Table 2:Transformer layer detail \(mini GPT\-2, 5 epochs, synthetic LM\)\.Conv layers maintainα≈2\\alpha\\approx 2throughout training \(emerging from∼1\.5\\sim 1\.5at initialization; Appendix[E](https://arxiv.org/html/2606.02596#A5)\)\. This holds at scale: ResNet\-50 on Tiny\-ImageNet \(200 classes, 64×\\times64\) yields medianα=1\.93\\alpha=1\.93across 49 conv layers withR2≥0\.88R^\{2\}\\geq 0\.88\.On ImageNet\-1Kwith pretrained weights \(IMAGENET1K\_V1\), 14 conv layers have medianα=2\.15\\alpha=2\.15withR2≥0\.97R^\{2\}\\geq 0\.97and spectral transfer error 1\.6% \(Table[1](https://arxiv.org/html/2606.02596#S3.T1)\)\. The FC head exhibits opposite trends at different scales:α=0\.90\\alpha=0\.90with 200 classes \(Tiny\-ImageNet\) vs\.α=2\.83\\alpha=2\.83with 1000 classes \(ImageNet\-1K\)\. Both are consistent with the theory: 200 classes produce sparse anisotropicCδC\_\{\\delta\}, pullingα\\alphabelow 2; 1000 classes giveCδC\_\{\\delta\}richer structure that strengthens alignment and pushesα\\alphaabove 2 \(Section[6](https://arxiv.org/html/2606.02596#S6)\)\. Full spectrum analysis showsα\\alphais stable across quartiles of the singular value indexkk\(Appendix[F](https://arxiv.org/html/2606.02596#A6)\)\.

For convolutional layers,hk∝σk2h\_\{k\}\\propto\\sigma\_\{k\}^\{2\}holds withR2=0\.98R^\{2\}=0\.98\(median across 21 ResNet\-18 layers; full table in Appendix[A](https://arxiv.org/html/2606.02596#A1)\)\.

## 4Related Work

#### Hessian structure and closed\-form spectra\.

Tang et al\.\[[1](https://arxiv.org/html/2606.02596#bib.bib1)\]*discover*power\-law Hessian eigenvalue decayhk∝k−sh\_\{k\}\\propto k^\{\-s\}across CNNs and LLMs and use it to predict generalization\. Their contribution is the observation and its statistical characterization; they do not explain*why*sstakes the values it does or howssrelates to gradient structure\. Our work*decomposes*ssvia the spectral transfer identitys=α​γs=\\alpha\\gamma: the Hessian decay exponent is the product of a curvature–gradient alignment exponentα\\alpha\(which we trace to Kronecker factor eigenbases, Theorem[1](https://arxiv.org/html/2606.02596#Thmtheorem1)\) and an effective gradient rank\-decay exponentγ\\gamma\. In short, Tang et al\. measuress; we explainss\. Wu et al\.\[[5](https://arxiv.org/html/2606.02596#bib.bib5)\]identify shared Hessian structure across architectures\. Recent work derives*closed\-form*Hessian spectra for shallow networks\[[12](https://arxiv.org/html/2606.02596#bib.bib12)\], complementing our layer\-wise*structural*decomposition for deep trained models\.

#### Kronecker factorization and layer\-wise preconditioning\.

K\-FAC\[[3](https://arxiv.org/html/2606.02596#bib.bib3),[4](https://arxiv.org/html/2606.02596#bib.bib4)\]and Shampoo\[[14](https://arxiv.org/html/2606.02596#bib.bib14)\]approximate curvature with Kronecker factorsCδ⊗CAC\_\{\\delta\}\\otimes C\_\{A\}\. Zhang et al\.\[[13](https://arxiv.org/html/2606.02596#bib.bib13)\]prove that layer\-wise Kronecker preconditioning is*provably necessary*for feature learning in certain regimes—our Spectral Alignment Decomposition gives a per\-layer spectral explanation \(alignment ratiosρk\\rho\_\{k\},cos2⁡θk\\cos^\{2\}\\theta\_\{k\}\) for when the approximation is accurate\. ESO\[[15](https://arxiv.org/html/2606.02596#bib.bib15)\]builds efficient spectral preconditioners via truncated Shampoo; ours=α​γs=\\alpha\\gammaidentity predicts which layers tolerate low\-rank spectral truncation \(steepss, concentrated curvature\) vs\. which require richer structure\.

#### Heavy\-tailed self\-regularization\.

Martin–Mahoney\[[2](https://arxiv.org/html/2606.02596#bib.bib2)\]relate weight matrix tail exponents to generalization\. Corollary[8](https://arxiv.org/html/2606.02596#Thmtheorem8)connects theirαweight\\alpha\_\{\\mathrm\{weight\}\}to our\(α,γ,s\)\(\\alpha,\\gamma,s\)triad\.

#### Spectral methods in optimization\.

Muon\[[8](https://arxiv.org/html/2606.02596#bib.bib8)\]flattens gradient singular values; SAM\[[9](https://arxiv.org/html/2606.02596#bib.bib9)\]explicitly minimizes sharpness\. Our decomposition shows the architecture\-appropriate spectral weighting isT​\(σ;α\)=σ/\(σα\+d\)T\(\\sigma;\\alpha\)=\\sigma/\(\\sigma^\{\\alpha\}\+d\)—a consequence of alignment geometry\.Spectral Newtonimplements this transfer in the gradient singular basis; Section[7\.3](https://arxiv.org/html/2606.02596#S7.SS3)validates it on vision benchmarks whereα≈2\\alpha\\approx 2\.

## 5The Spectral Alignment Decomposition

###### Definition 1\(Spectral alignment ratios\)\.

LetG=U​Σ​V⊤G=U\\Sigma V^\{\\top\}be the gradient SVD, andCδ=Qδ​Λδ​Qδ⊤C\_\{\\delta\}=Q\_\{\\delta\}\\Lambda\_\{\\delta\}Q\_\{\\delta\}^\{\\top\},CA=QA​ΛA​QA⊤C\_\{A\}=Q\_\{A\}\\Lambda\_\{A\}Q\_\{A\}^\{\\top\}eigendecompositions with eigenvalues in decreasing order\. Define:

ρk\(δ\)=uk⊤​Cδ​uk\[Λδ\]k​k,ρk\(A\)=vk⊤​CA​vk\[ΛA\]k​k\\rho\_\{k\}^\{\(\\delta\)\}=\\frac\{u\_\{k\}^\{\\top\}C\_\{\\delta\}u\_\{k\}\}\{\[\\Lambda\_\{\\delta\}\]\_\{kk\}\},\\qquad\\rho\_\{k\}^\{\(A\)\}=\\frac\{v\_\{k\}^\{\\top\}C\_\{A\}v\_\{k\}\}\{\[\\Lambda\_\{A\}\]\_\{kk\}\}

###### Theorem 1\(Spectral Alignment Decomposition\)\.

UnderℋGN\(ℓ\)≈Cδ⊗CA\\mathcal\{H\}\_\{\\mathrm\{GN\}\}^\{\(\\ell\)\}\\approx C\_\{\\delta\}\\otimes C\_\{A\}:

1. 1\.\(Exact decomposition\.\)hk=ρk\(δ\)⋅ρk\(A\)⋅\[Λδ\]k​k⋅\[ΛA\]k​kh\_\{k\}=\\rho\_\{k\}^\{\(\\delta\)\}\\cdot\\rho\_\{k\}^\{\(A\)\}\\cdot\[\\Lambda\_\{\\delta\}\]\_\{kk\}\\cdot\[\\Lambda\_\{A\}\]\_\{kk\}\.
2. 2\.\(α\\alphaas alignment slope\.\)WithΦk=ρk\(δ\)​ρk\(A\)\\Phi\_\{k\}=\\rho\_\{k\}^\{\(\\delta\)\}\\rho\_\{k\}^\{\(A\)\}and power\-law fithk=c​σkαh\_\{k\}=c\\sigma\_\{k\}^\{\\alpha\}: α=d​log⁡Λkd​log⁡σk\+d​log⁡Φkd​log⁡σk,Λk=\[Λδ\]k​k​\[ΛA\]k​k\\alpha=\\frac\{d\\log\\Lambda\_\{k\}\}\{d\\log\\sigma\_\{k\}\}\+\\frac\{d\\log\\Phi\_\{k\}\}\{d\\log\\sigma\_\{k\}\},\\quad\\Lambda\_\{k\}=\[\\Lambda\_\{\\delta\}\]\_\{kk\}\[\\Lambda\_\{A\}\]\_\{kk\}\(2\)
3. 3\.\(Canonical form\.\)WhenQδ=UQ\_\{\\delta\}=U,QA=VQ\_\{A\}=V\(perfect alignment\),\[Λδ\]k​k​\[ΛA\]k​k=σk2/cos2⁡θk\[\\Lambda\_\{\\delta\}\]\_\{kk\}\[\\Lambda\_\{A\}\]\_\{kk\}=\\sigma\_\{k\}^\{2\}/\\cos^\{2\}\\theta\_\{k\}and α=2\+d​log⁡Φkd​log⁡σk−d​log⁡cos2⁡θkd​log⁡σk\\alpha=2\+\\frac\{d\\log\\Phi\_\{k\}\}\{d\\log\\sigma\_\{k\}\}\-\\frac\{d\\log\\cos^\{2\}\\\!\\theta\_\{k\}\}\{d\\log\\sigma\_\{k\}\}\(3\)wherecos2⁡θk=hkexact/hkKron\\cos^\{2\}\\theta\_\{k\}=h\_\{k\}^\{\\mathrm\{exact\}\}/h\_\{k\}^\{\\mathrm\{Kron\}\}measures the GN\-to\-exact gap\.

###### Proof\.

Part 1\.The curvature alonguk​vk⊤u\_\{k\}v\_\{k\}^\{\\top\}ishk=vec\(ukvk⊤\)⊤\(Cδ⊗CA\)vec\(ukvk⊤\)=\(uk⊤Cδuk\)\(vk⊤CAvk\)h\_\{k\}=\\operatorname\{vec\}\(u\_\{k\}v\_\{k\}^\{\\top\}\)^\{\\top\}\(C\_\{\\delta\}\\otimes C\_\{A\}\)\\operatorname\{vec\}\(u\_\{k\}v\_\{k\}^\{\\top\}\)=\(u\_\{k\}^\{\\top\}C\_\{\\delta\}u\_\{k\}\)\(v\_\{k\}^\{\\top\}C\_\{A\}v\_\{k\}\)\. By Definition[1](https://arxiv.org/html/2606.02596#Thmdefinition1),uk⊤​Cδ​uk=ρk\(δ\)​\[Λδ\]k​ku\_\{k\}^\{\\top\}C\_\{\\delta\}u\_\{k\}=\\rho\_\{k\}^\{\(\\delta\)\}\[\\Lambda\_\{\\delta\}\]\_\{kk\}andvk⊤​CA​vk=ρk\(A\)​\[ΛA\]k​kv\_\{k\}^\{\\top\}C\_\{A\}v\_\{k\}=\\rho\_\{k\}^\{\(A\)\}\[\\Lambda\_\{A\}\]\_\{kk\}, giving the decomposition\.

Part 2\.Taking logs:log⁡hk=log⁡Φk\+log⁡Λk\\log h\_\{k\}=\\log\\Phi\_\{k\}\+\\log\\Lambda\_\{k\}\. Ifhk=c​σkαh\_\{k\}=c\\sigma\_\{k\}^\{\\alpha\}, thend​log⁡hk/d​log⁡σk=αd\\log h\_\{k\}/d\\log\\sigma\_\{k\}=\\alpha\. Differentiating the decomposition yields Eq\. \([2](https://arxiv.org/html/2606.02596#S5.E2)\)\.

Part 3\.WhenQδ=UQ\_\{\\delta\}=UandQA=VQ\_\{A\}=V, we haveρk\(δ\)=ρk\(A\)=1\\rho\_\{k\}^\{\(\\delta\)\}=\\rho\_\{k\}^\{\(A\)\}=1, soΦk=1\\Phi\_\{k\}=1\. Under the alignment condition of Theorem[2](https://arxiv.org/html/2606.02596#Thmtheorem2),\[Λδ\]k​k​\[ΛA\]k​k=σk2/cos2⁡θk\[\\Lambda\_\{\\delta\}\]\_\{kk\}\[\\Lambda\_\{A\}\]\_\{kk\}=\\sigma\_\{k\}^\{2\}/\\cos^\{2\}\\theta\_\{k\}, henced​log⁡Λk/d​log⁡σk=2−d​log⁡cos2⁡θk/d​log⁡σkd\\log\\Lambda\_\{k\}/d\\log\\sigma\_\{k\}=2\-d\\log\\cos^\{2\}\\theta\_\{k\}/d\\log\\sigma\_\{k\}\. Substituting into Part 2 withd​log⁡Φk=0d\\log\\Phi\_\{k\}=0gives Eq\. \([3](https://arxiv.org/html/2606.02596#S5.E3)\)\. ∎

###### Theorem 2\(Baseline:α=2\\alpha=2under perfect alignment\)\.

WhenCδC\_\{\\delta\},CAC\_\{A\}share the singular/eigen\-basis ofGGand per\-sample\(uk⊤​δi\)2\(u\_\{k\}^\{\\top\}\\delta\_\{i\}\)^\{2\},\(vk⊤​ai\)2\(v\_\{k\}^\{\\top\}a\_\{i\}\)^\{2\}are approximately independent across samples,hk∝σk2h\_\{k\}\\propto\\sigma\_\{k\}^\{2\}\.

## 6Architectural Mechanisms

We now show how each architectural component produces a specific pattern inΦk\\Phi\_\{k\}vs\.σk\\sigma\_\{k\}\.

### 6\.1LayerNorm

###### Theorem 3\(LayerNorm flattens activation alignment\)\.

Fory=W⋅LN​\(x\)y=W\\cdot\\mathrm\{LN\}\(x\), the activation covarianceCALN=Γ​C^​ΓC\_\{A\}^\{\\mathrm\{LN\}\}=\\Gamma\\hat\{C\}\\GammawithC^\\hat\{C\}the normalized\-input covariance\. The condition numberκ​\(C^\|𝟏⟂\)=1\+O​\(n/B\)\\kappa\(\\hat\{C\}\|\_\{\\mathbf\{1\}^\{\\perp\}\}\)=1\+O\(\\sqrt\{n/B\}\), andρk\(A\)\\rho\_\{k\}^\{\(A\)\}is confined to aσk\\sigma\_\{k\}\-independent band\. Thusd​log⁡ρk\(A\)/d​log⁡σk≈0d\\log\\rho\_\{k\}^\{\(A\)\}/d\\log\\sigma\_\{k\}\\approx 0, reducingα\\alphafrom22toward11\.

###### Corollary 4\.

Withcos2⁡θk\\cos^\{2\}\\theta\_\{k\}approximately constant,α≈1\+d​log⁡ρk\(δ\)/d​log⁡σk\\alpha\\approx 1\+d\\log\\rho\_\{k\}^\{\(\\delta\)\}/d\\log\\sigma\_\{k\}\. Ifuk⊤​Cδ​uk∝σku\_\{k\}^\{\\top\}C\_\{\\delta\}u\_\{k\}\\propto\\sigma\_\{k\}, thenα≈1\\alpha\\approx 1\.

### 6\.2Residual connections

###### Proposition 5\(Residual decorrelation\)\.

Iny=x\+f​\(x\)y=x\+f\(x\)with skip\-dominated regime \(‖Jf⊤​δout‖≪‖δout‖\\\|J\_\{f\}^\{\\top\}\\delta\_\{\\mathrm\{out\}\}\\\|\\ll\\\|\\delta\_\{\\mathrm\{out\}\}\\\|\),Cδ≈CδoutC\_\{\\delta\}\\approx C\_\{\\delta\_\{\\mathrm\{out\}\}\}whose eigenvectors bear no relation to current\-layeruku\_\{k\}\. Thusρk\(δ\)\\rho\_\{k\}^\{\(\\delta\)\}has no systematicσk\\sigma\_\{k\}dependence andd​log⁡ρk\(δ\)/d​log⁡σk≈0d\\log\\rho\_\{k\}^\{\(\\delta\)\}/d\\log\\sigma\_\{k\}\\approx 0\.

Combined with LayerNorm,α≈2−d​log⁡cos2⁡θk/d​log⁡σk\\alpha\\approx 2\-d\\log\\cos^\{2\}\\theta\_\{k\}/d\\log\\sigma\_\{k\}, explaining MLP up\-projections withα≈0\.8\\alpha\\approx 0\.8\.

### 6\.3Softmax concentration

###### Proposition 6\(Softmax createsα\>2\\alpha\>2\)\.

For output layer withccclasses,CδC\_\{\\delta\}has rank≤c\\leq c\. Top singular vectors ofGGalign with top eigenvectors ofCδC\_\{\\delta\}, givingρk\(δ\)≈1\\rho\_\{k\}^\{\(\\delta\)\}\\approx 1fork≤ck\\leq candρk\(δ\)≪1\\rho\_\{k\}^\{\(\\delta\)\}\\ll 1beyond—a sharp drop creatingd​log⁡Φk/d​log⁡σk\>0d\\log\\Phi\_\{k\}/d\\log\\sigma\_\{k\}\>0andα\>2\\alpha\>2\.

### 6\.4The conv gap: GN vs\. Kronecker

For conv layers,cos2⁡θk∼103\\cos^\{2\}\\theta\_\{k\}\\sim 10^\{3\}–10710^\{7\}yetα≈2\\alpha\\approx 2\. The*concentration ratio*h¯SoP/h¯PoS≈1\\bar\{h\}^\{\\mathrm\{SoP\}\}/\\bar\{h\}^\{\\mathrm\{PoS\}\}\\approx 1\(Kronecker factorization is accurate\), whilecos2⁡θk≫1\\cos^\{2\}\\theta\_\{k\}\\gg 1\(GN\-to\-exact gap from weight\-sharing\)\. What matters forα\\alphais the*slope*d​log⁡cos2⁡θk/d​log⁡σk≈0d\\log\\cos^\{2\}\\theta\_\{k\}/d\\log\\sigma\_\{k\}\\approx 0, not the magnitude \(Appendix[D](https://arxiv.org/html/2606.02596#A4)\)\.

### 6\.5Refined prediction: LayerNorm is necessary but not sufficient

ViT\-Tiny on CIFAR\-10 has LayerNorm yetα≈2\\alpha\\approx 2\. Controlled ablations \(Appendix[C](https://arxiv.org/html/2606.02596#A3)\) show that large output dimension creates sparse, anisotropicCδC\_\{\\delta\}, pullingα\\alphabelow22on CIFAR\-100; switching GPT\-2 from LM to binary classification raisesα\\alphatoward22\. The corrected condition:α<2\\alpha<2requiresbothLayerNormandsufficiently large output dimension\.

## 7Spectral Transfer and Intrinsic Dimension

### 7\.1The Alpha Triangle

Three exponents summarize the spectral structure: curvatureα\\alpha\(hk∝σkαh\_\{k\}\\propto\\sigma\_\{k\}^\{\\alpha\}\), effective gradient rank decayγ\\gamma\(slope oflog⁡σk\\log\\sigma\_\{k\}vs\.log⁡k\\log kon top\-kkdirections\), and Hessian decayss\(λk∼k−s\\lambda\_\{k\}\\sim k^\{\-s\}along comparable directions\)\[[1](https://arxiv.org/html/2606.02596#bib.bib1)\]\.

###### Theorem 7\(Spectral transfer identity\)\.

Supposeσk∼k−γ\\sigma\_\{k\}\\sim k^\{\-\\gamma\}andhk=c​σkαh\_\{k\}=c\\sigma\_\{k\}^\{\\alpha\}on a rank interval\. Thenλk∼k−s\\lambda\_\{k\}\\sim k^\{\-s\}withs=α​γs=\\alpha\\gamma\.

###### Proof\.

λk=hk=c​\(c′​k−γ\)α=c′′​k−α​γ\\lambda\_\{k\}=h\_\{k\}=c\(c^\{\\prime\}k^\{\-\\gamma\}\)^\{\\alpha\}=c^\{\\prime\\prime\}k^\{\-\\alpha\\gamma\}\. ∎

![Refer to caption](https://arxiv.org/html/2606.02596v1/x3.png)Figure 3:Measured Hessian decayssvs\. predicteds=α​γs=\\alpha\\gammaacross 93 layers and five architectures/datasets\. Points near the diagonal validate the spectral transfer identity; shaded band is±10%\\pm 10\\%\. Color indicates data source\.![Refer to caption](https://arxiv.org/html/2606.02596v1/x4.png)Figure 4:Gradient singular value rank profiles on ResNet\-18 with BIC\-selected models\.\(a–c\)Measuredσk\\sigma\_\{k\}\(points\) vs\. power\-law, exponential, and log\-normal fits \(lines\); bold curve is the BIC winner\. Log\-normal fits are clipped to the data range on the FC head panel\.###### Corollary 8\(Connection to HT\-SR\)\.

Martin–Mahoney’s weight tail exponentαweight\\alpha\_\{\\mathrm\{weight\}\}\[[2](https://arxiv.org/html/2606.02596#bib.bib2)\]is constrained bys=α​γs=\\alpha\\gammagoverning the Hessian spectrum\. We findαweight<s\\alpha\_\{\\mathrm\{weight\}\}<sconsistently, consistent with heavier weight tails correlating with better generalization\.

### 7\.2Near\-rank\-one curvature

###### Theorem 9\(Zeta function bound\)\.

Withhk∼k−α​γh\_\{k\}\\sim k^\{\-\\alpha\\gamma\}, the participation ratioPRh=\(∑hk\)2/∑hk2≤ζ​\(α​γ\)2/ζ​\(2​α​γ\)\\mathrm\{PR\}\_\{h\}=\(\\sum h\_\{k\}\)^\{2\}/\\sum h\_\{k\}^\{2\}\\leq\\zeta\(\\alpha\\gamma\)^\{2\}/\\zeta\(2\\alpha\\gamma\), bounded forα​γ\>1\\alpha\\gamma\>1\. Forα​γ=2\\alpha\\gamma=2:PRh≲2\.5\\mathrm\{PR\}\_\{h\}\\lesssim 2\.5\.

Despite hundreds of singular directions, curvature concentrates ontoPRh∈\[1\.0,1\.8\]\\mathrm\{PR\}\_\{h\}\\in\[1\.0,1\.8\]while gradients span moderate\-dimensional subspaces \(PRgrad∈\[1\.8,58\]\\mathrm\{PR\}\_\{\\mathrm\{grad\}\}\\in\[1\.8,58\]; Appendix[B](https://arxiv.org/html/2606.02596#A2)\)\. Width scaling fromn=64n=64to512512changes PR by only 0\.7 \(Appendix[I](https://arxiv.org/html/2606.02596#A9)\)\.

### 7\.3From decomposition to Spectral Newton

The decomposition predicts*which layers*admit Kronecker\-style curvature approximations: when\|α−2\|\|\\alpha\-2\|is small and the concentration ratioh¯SoP/h¯PoS≈1\\bar\{h\}^\{\\mathrm\{SoP\}\}/\\bar\{h\}^\{\\mathrm\{PoS\}\}\\approx 1, the Kronecker product\-of\-means matches the exact mean\-of\-products Hessian along gradient directions \(Figure[5](https://arxiv.org/html/2606.02596#S7.F5)\)\. Conversely, layers withα≪2\\alpha\\ll 2\(LayerNorm \+ large output dimension\) orα≫2\\alpha\\gg 2\(softmax heads\) break alignment; full Hessian or task\-specific preconditioning is required, consistent with K\-FAC/Shampoo using richer structure than diagonal Adam\.

#### Prescriptive transfer function\.

Whenhk∝σkαh\_\{k\}\\propto\\sigma\_\{k\}^\{\\alpha\}, the optimal step along singular directionkkscales asT​\(σk;α\)=σk/\(σkα\+d\)T\(\\sigma\_\{k\};\\alpha\)=\\sigma\_\{k\}/\(\\sigma\_\{k\}^\{\\alpha\}\+d\)rather than a uniform spectral flattening \(Muon\) or a single scalar per layer\. For convolutional layers withα≈2\\alpha\\approx 2, this reduces to Spectral Newton \(SN\):Δ​W=U​diag⁡\(σk/\(σk2\+d\)\)​V⊤\\Delta W=U\\,\\operatorname\{diag\}\(\\sigma\_\{k\}/\(\\sigma\_\{k\}^\{2\}\+d\)\)\\,V^\{\\top\}forG=U​Σ​V⊤G=U\\Sigma V^\{\\top\}\. Per\-layerα\\alphafrom Table[1](https://arxiv.org/html/2606.02596#S3.T1)selects the family member—α≈1\\alpha\\approx 1for transformers suggestsT​\(σ\)=σ/\(σ\+d\)T\(\\sigma\)=\\sigma/\(\\sigma\+d\)instead\.

![Refer to caption](https://arxiv.org/html/2606.02596v1/x5.png)Figure 5:Kronecker factorization quality vs\.\|α−2\|\|\\alpha\-2\|on ResNet\-18 conv layers \(\+ FC control\)\.\(a\)Deviation of concentration ratio from 1 \(marker size∝\\proptoaccuracy\)\.\(b\)R2R^\{2\}ofhk∝σkαh\_\{k\}\\propto\\sigma\_\{k\}^\{\\alpha\}; dotted line at 0\.97\.
#### Directional evidence\.

As a proof of concept, we test SN on ResNet\-18/50 vision tasks where medianα≈2\\alpha\\approx 2\(Table[1](https://arxiv.org/html/2606.02596#S3.T1)\)\. Using matrix\-function recovery after polar decomposition at∼\\sim1\.8×\\timesMuon cost, SN outperforms AdamW at matched epoch budgets on all three benchmarks \(Table[3](https://arxiv.org/html/2606.02596#S7.T3)\), with gains of\+1\.1%\+1\.1\\%on CIFAR\-10 and\+2\.4\+2\.4–3\.5%3\.5\\%on the harder CIFAR\-100\. These are single\-seed runs reported as directional evidence that the decomposition yields a useful optimizer, not as a comprehensive benchmark; multi\-seed evaluation and LLM\-scale testing are deferred to future work\. Crucially,*scalar*per\-layer learning\-rate scaling froms=α​γs=\\alpha\\gammadoes*not*reproduce this improvement \(Appendix[M](https://arxiv.org/html/2606.02596#A13)\): curvature correction is directional, not a single scale per layer\.

Table 3:Spectral Newton vs\. baselines \(test accuracy %, matched epochs, seed 42\)\. SN uses dampingd=0\.1d\{=\}0\.1\.

## 8Discussion

#### Spectral asymptotics interpretation\.

The identityα=2\+d​log⁡Φk/d​log⁡σk\\alpha=2\+d\\log\\Phi\_\{k\}/d\\log\\sigma\_\{k\}is a Weyl\-law analog: the decay exponent of curvature along gradient directions encodes the*alignment geometry*of the loss Hessian restricted to the gradient subspace\. Just asλk∼k2/d\\lambda\_\{k\}\\sim k^\{2/d\}reveals manifold dimensiondd,α\\alphareveals how Kronecker factor eigenbases align with gradient singular directions\.

#### Optimizer implications\.

The decomposition yields curvature\-adaptive transfer functionsT​\(σ;α\)=σ/\(σα\+d\)T\(\\sigma;\\alpha\)=\\sigma/\(\\sigma^\{\\alpha\}\+d\)with architecture\-dependentα\\alpha\. As directional evidence, Spectral Newton implements this prescription in the gradient singular basis and outperforms AdamW on vision benchmarks whereα≈2\\alpha\\approx 2\(Table[3](https://arxiv.org/html/2606.02596#S7.T3); single\-seed, proof of concept\)\. The deeper structural insight is that Kronecker\-style methods are justified when\|α−2\|\|\\alpha\-2\|is small \(Figure[5](https://arxiv.org/html/2606.02596#S7.F5)\), and that scalar per\-layer scaling froms=α​γs=\\alpha\\gammafails because eigenvector alignment \(Theorem[1](https://arxiv.org/html/2606.02596#Thmtheorem1)\) is directional—the theory prescribesT​\(σk;αℓ\)T\(\\sigma\_\{k\};\\alpha\_\{\\ell\}\)per singular direction, not one learning rate per layer\.

#### Gradient rank profile and phase transitions\.

BIC favors log\-normal\-in\-rank over strict power\-law on most CIFAR layers \(Figure[4](https://arxiv.org/html/2606.02596#S7.F4)\)\. During training, the winning rank model shifts: Figure[6](https://arxiv.org/html/2606.02596#S8.F6)shows VGG\-16 on Tiny\-ImageNet transitioning from log\-normal dominance at initialization toward mixed power\-law/exponential regimes by epoch 15–25 \(slower than ResNet\-50, where the transition peaks by epoch 5\)\. Thusγ\\gammais an effective exponent; Appendix[L](https://arxiv.org/html/2606.02596#A12)shows that the same Kronecker factors predict the rank\-profile shape\.

![Refer to caption](https://arxiv.org/html/2606.02596v1/x6.png)Figure 6:Fraction of VGG\-16 layers whose top\-kkgradient spectrum is best fit by each BIC model vs\. training epoch \(Tiny\-ImageNet\)\. Log\-normal dominance at init gives way to power\-law and exponential wins—the*spectral phase transition*\.
#### Limitations and open questions\.

Scale\.Our largest models are GPT\-2 \(6\-layer, 85M\) and ResNet\-50; the theory is architecture\-generic and makes concrete, falsifiable predictions at larger scales—for instance, Theorem[1](https://arxiv.org/html/2606.02596#Thmtheorem1)predicts that ViT\-Large attention layers will exhibitα≈1\\alpha\\approx 1\(driven by LayerNorm\-induced alignment collapse, Section[6](https://arxiv.org/html/2606.02596#S6)\) and that LLaMA\-7B MLP layers will followα<1\.2\\alpha<1\.2with the same Kronecker decomposition—but we have not yet verified these\. Validation at the 1–10B parameter regime is the most important next step\.Spectral Newton\.SN results \(Table[3](https://arxiv.org/html/2606.02596#S7.T3)\) are single\-seed, single\-architecture, and serve as directional evidence rather than a comprehensive optimizer comparison\. Multi\-seed evaluation with wall\-clock accounting is needed\.

1. 1\.Extend the zeta PR bound to log\-normal or exponential rank spectra via Tauberian theorems\.
2. 2\.Validate the full Kronecker spectral theory at LLM scale \(ViT\-Large, LLaMA\-7B, GPT\-3 class\)\.
3. 3\.Connect spectral phase\-transition timing to gradient flow on\(Cδ​\(t\),CA​\(t\)\)\(C\_\{\\delta\}\(t\),C\_\{A\}\(t\)\)\.

## Reproducibility Statement

All empirical claims are reproduced from scripts that measure exact Hessian\-vector products per layer\. The full reproducibility package is available at[https://github\.com/9D\-Labs/9d\-spectral\-alignment\-decomposition](https://github.com/9D-Labs/9d-spectral-alignment-decomposition):

- •Frozen outputs:15 JSON files inresults/covering all tables and figures\.
- •Claim verification:verify\_claims\.pychecks 27 quantitative claims against JSON; exits 1 on mismatch\.
- •Figures:generate\_figures\.pyandanalyze\_kfac\_prediction\.pybuild all 6 main\-text plots from JSON \(no GPU\)\.
- •One command:make allverifies claims, regenerates figures, and compiles the PDF\.
- •Full reproduction:EXPERIMENTS\.mddocuments the measurement protocol for all 10 experiments \(Modal A10G,∼\\sim3–4 h total\)\.

## Acknowledgments

We thank the 9D Labs team for compute infrastructure and experimental support\.

## References

- \[1\]Q\.\-Y\. Tang et al\., “Investigating the Overlooked Hessian Structure: From CNNs to LLMs,” ICML, 2025\.
- \[2\]C\. H\. Martin, M\. W\. Mahoney, “Implicit Self\-Regularization in Deep Neural Networks,” JMLR, 2021\.
- \[3\]J\. Martens, R\. Grosse, “Optimizing Neural Networks with Kronecker\-factored Approximate Curvature,” ICML, 2015\.
- \[4\]R\. Grosse, J\. Martens, “A Kronecker\-factored approximate Fisher matrix for convolution layers,” ICML, 2016\.
- \[5\]Y\. Wu et al\., “Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks,” arXiv:2010\.04261, 2022\.
- \[6\]Z\. Bai, Y\. Yin, “Limit of the smallest eigenvalue of a large dimensional sample covariance matrix,” Ann\. Probab\., 1993\.
- \[7\]B\. Hanin, M\. Nica, “Products of Many Large Random Matrices and Gradients in Deep Neural Networks,” arXiv:1812\.05994, 2018\.
- \[8\]K\. Jordan, “Muon: An optimizer for hidden layers in neural networks,” GitHub, 2024\.
- \[9\]P\. Foret et al\., “Sharpness\-aware minimization,” ICLR, 2021\.
- \[10\]J\. Bernstein, “Old optimizer, new norm: An anthology,” arXiv, 2024\.
- \[11\]R\. Couillet, W\. Hachem, “Analysis of the limiting spectral measure of large random matrices of the separable covariance type,” Random Matrices: Theory Appl\., 2014\.
- \[12\]S\. P\. Singh, W\. Ormaniec, T\. Hofmann, “Cracking the Hessian: Closed\-Form Hessian Spectra for Fundamental Neural Networks,” under review, 2026\.
- \[13\]Y\. Zhang et al\., “On the Concurrence of Layer\-wise Preconditioning Methods and Provable Feature Learning,” ICML, 2025\.
- \[14\]V\. Gupta et al\., “Shampoo: Preconditioned stochastic tensor optimization,” ICML, 2018\.
- \[15\]Y\. Chen, J\. He, Z\. Zheng, J\. Lavaei, “ESO: Efficient Spectral Preconditioner for Second\-Order Optimization,” 2026\.

## Appendix AHVP Validation Table

## Appendix BParticipation Ratio Table

## Appendix CControlled Ablations

## Appendix DConv Gap Table

## Appendix ETraining Dynamics ofα\\alpha

ResNet\-18 on CIFAR\-10,α\\alphaat epochs 0, 1, 3, 5, 10, 15, 20:

Conv layers converge toα≈2\\alpha\\approx 2; FC layers decrease as softmax concentration develops\.

## Appendix FFull Spectrum Analysis

α\\alphaby quartile of singular indexkk\(ResNet\-18, top\-50 directions\):

## Appendix GGradient Rank Profile andγ\\gamma

We compare three rank\-ordered models forσk\\sigma\_\{k\}: power lawσk∝k−γ\\sigma\_\{k\}\\propto k^\{\-\\gamma\}, exponentialσk∝e−β​k\\sigma\_\{k\}\\propto e^\{\-\\beta k\}, and log\-normal in rank \(log⁡σk\\log\\sigma\_\{k\}quadratic inlog⁡k\\log k\), using BIC\.

Exponential BIC wins increase during training \(6→\\to15 layers from epoch 0 to 25\) but do not dominate at initialization\. Fittedγ\\gammais therefore aneffectivelog\-log slope over the measurement window; the Alpha Triangle uses thisγ\\gammaconsistently\. See the reproducibility package for reproduction details\.

## Appendix HFull Alpha Triangle Table

CIFAR\-10 experiments: 20 layers, median relative error 1\.9% \(n=19n\{=\}19well\-conditioned at≤\\leq10% rel\. error\)\. Tiny\-ImageNet ResNet\-50: 54 layers, median 1\.0%\. CNN layers:Rγ2\>0\.88R^\{2\}\_\{\\gamma\}\>0\.88on CIFAR; GPT\-2: strongα\\alphafits, weakerγ\\gammapower\-lawR2R^\{2\}but identity still holds\.

## Appendix IWidth Scaling of Participation Ratio

Single\-hidden\-layer MLPs on CIFAR\-10: PR grows from 2\.5 \(n=64n=64\) to 3\.3 \(n=512n=512\) despite 8×\\timeswidth—bounded, not linear innn\.

## Appendix JProof Details

### J\.1LayerNorm \(Theorem[3](https://arxiv.org/html/2606.02596#Thmtheorem3)\)

CALN=Γ​C^​ΓC\_\{A\}^\{\\mathrm\{LN\}\}=\\Gamma\\hat\{C\}\\Gammawithx^i⊤​𝟏=0\\hat\{x\}\_\{i\}^\{\\top\}\\mathbf\{1\}=0\. By Bai–Yin\[[6](https://arxiv.org/html/2606.02596#bib.bib6)\],κ​\(C^\|𝟏⟂\)=1\+O​\(n/B\)\\kappa\(\\hat\{C\}\|\_\{\\mathbf\{1\}^\{\\perp\}\}\)=1\+O\(\\sqrt\{n/B\}\)\. The Rayleigh quotientρk\(A\)\\rho\_\{k\}^\{\(A\)\}is bounded in akk\-independent interval, sod​log⁡ρk\(A\)/d​log⁡σk→0d\\log\\rho\_\{k\}^\{\(A\)\}/d\\log\\sigma\_\{k\}\\to 0\.

### J\.2Softmax \(Proposition[6](https://arxiv.org/html/2606.02596#Thmtheorem6)\)

CδC\_\{\\delta\}has rank≤c\\leq c\. Top\-ccleft singular vectors ofG=1B​δ⊤​AG=\\frac\{1\}\{B\}\\delta^\{\\top\}Aalign with eigenvectors ofCδC\_\{\\delta\}when predictions are concentrated, givingρk\(δ\)≈1\\rho\_\{k\}^\{\(\\delta\)\}\\approx 1fork≤ck\\leq cand≪1\\ll 1beyond—positived​log⁡Φk/d​log⁡σkd\\log\\Phi\_\{k\}/d\\log\\sigma\_\{k\}\.

### J\.3Emergence during training

α\\alphaconverges to layer\-type attractors because alignment ratios evolve predictably: conv layers tighten Kronecker alignment \(Φk→1\\Phi\_\{k\}\\to 1,α→2\\alpha\\to 2\); FC layers develop softmax anti\-alignment \(α→1\\alpha\\to 1\); transformer layers maintain the LayerNorm barrier \(α≈1\\alpha\\approx 1throughout\)\.

## Appendix KVGG\-16 Spectral Trajectory

We repeat the spectral trajectory analysis \(Appendix[G](https://arxiv.org/html/2606.02596#A7)\) on VGG\-16 / Tiny\-ImageNet \(15 conv layers, 25 epochs, top\-k=100k\{=\}100\)\. BIC model counts by epoch:

VGG\-16 exhibits the same spectral phase transition as ResNet\-50: log\-normal dominance at initialization gives way to a mixed power\-law/exponential regime after training\. The transition is*slower*\(power\-law peaks at epoch 15–20, vs\. epoch 5 in ResNet\-50\), consistent with the absence of skip connections delaying spectral mixing through the backward pass\. This confirms that the phase transition is architecture\-general, not an artifact of residual structure\.

## Appendix LKronecker Spectral Identity

The Spectral Alignment Decomposition \(Theorem[1](https://arxiv.org/html/2606.02596#Thmtheorem1)\) determinesα\\alpha; we show here that the same Kronecker factors also predict the*shape*of the gradient rank profile\.

#### Separable variance structure\.

In the Kronecker eigenbasis, the rotated gradientG~=Qδ⊤​G​QA\\widetilde\{G\}=Q\_\{\\delta\}^\{\\top\}GQ\_\{A\}has entry varianceVar​\(G~i​j\)∝λi\(δ\)​λj\(A\),\\mathrm\{Var\}\(\\widetilde\{G\}\_\{ij\}\)\\propto\\lambda\_\{i\}^\{\(\\delta\)\}\\lambda\_\{j\}^\{\(A\)\},a rank\-one separable profile\. The singular value distribution of matrices with separable variance is characterized by the Couillet–Hachem Stieltjes transform\[[11](https://arxiv.org/html/2606.02596#bib.bib11)\]: the limiting spectral measure ofG~⊤​G~\\widetilde\{G\}^\{\\top\}\\widetilde\{G\}depends only on the empirical distributions of\{λi\(δ\)\}\\\{\\lambda\_\{i\}^\{\(\\delta\)\}\\\}and\{λj\(A\)\}\\\{\\lambda\_\{j\}^\{\(A\)\}\\\}\.

#### Empirical validation\.

On 15 ResNet\-50 layers \(Tiny\-ImageNet, 25 epochs\), we compute Kronecker factorsCδC\_\{\\delta\},CAC\_\{A\}via backward hooks \(im2col for conv layers\) and compare their spectra to the gradient SVD:

When one Kronecker factor has low effective rank \(reff<10r\_\{\\mathrm\{eff\}\}<10\), the gradient spectrum is exponential \(hard spectral cutoff\); when both havereff\>10r\_\{\\mathrm\{eff\}\}\>10, the gradient spectrum is log\-normal \(CLT on the separable product\)\. The factor BIC model matches the gradient’s in 87% of layers, withCAC\_\{A\}being the dominant driver \(73%\)\.

#### Theoretical bridge\.

Three supporting results formalize the mechanism: \(i\) a*rank\-truncation bound*\(Weyl inequality\) for exponentialσk\\sigma\_\{k\}when one factor has lowreffr\_\{\\mathrm\{eff\}\}; \(ii\) an*effective rank bound*onKeff​\(G\)K\_\{\\mathrm\{eff\}\}\(G\); \(iii\) a*log\-normal rank profile theorem*: nonzero Couillet–Hachem quantile curvature implies BIC selects log\-normal over power\-law whenmin⁡\(reff\)≳10\\min\(r\_\{\\mathrm\{eff\}\}\)\\gtrsim 10\(50/50 synthetic verification\)\. Full proofs are deferred to a forthcoming manuscript\.

## Appendix MScalar spectral preconditioner \(negative control\)

A cheaper alternative to SN is Adam with per\-layer learning rateηℓ∝\(sℓ/s¯\)β\\eta\_\{\\ell\}\\propto\(s\_\{\\ell\}/\\bar\{s\}\)^\{\\beta\}using measuredsℓ=αℓ​γℓs\_\{\\ell\}=\\alpha\_\{\\ell\}\\gamma\_\{\\ell\}\. Acrossβ∈\{0\.5,1\.0,1\.5\}\\beta\\in\\\{0\.5,1\.0,1\.5\\\}, three seeds, and controls \(inverse scaling, random per\-layer noise\), best test accuracy on ResNet\-18 / CIFAR\-10 \(100 epochs\) differs from Adam by at most∼\\sim0\.2%, within seed variance:

This confirms that knowing the eigenvalue profile viassis insufficient without applyingT​\(σk;α\)T\(\\sigma\_\{k\};\\alpha\)along each gradient singular direction\.