扩散模型中的去噪得分匹配：秩约束对泛化能力的影响

# 1Introduction
Source: [https://arxiv.org/html/2610.20850](https://arxiv.org/html/2610.20850)
Denoising Score Matching in Diffusion Models: The Effect of Rank Constraints on Generalization

Anand Jerry GeorgeNicolas Macris

École Polytechnique Fédérale de Lausanne \(EPFL\),Lab for Statistical Mechanics of Inference in Large Systems \(SMILS\),CH\-1015 Lausanne,Switzerland

###### Abstract

Architectural biases restrict the class of functions that a diffusion score network can represent, but their effect on generalization remains poorly understood\. We study denoising score matching with score networks having a rank constrained non\-linear term and a linear skip connection\. Numerical experiments with trainable three\-layer networks on data supported on an embedded sphere show that increasing rank lowers training error, but its effect on test error depends on sample size and data dimension\. To explain this behavior, we analyze a random\-feature score model under rank constraint with Gaussian data supported on a low\-dimensional subspace\. We characterize the optimal rank\-constrained estimator through a truncated singular value decomposition and derive asymptotically precise limiting training and test errors when sample size\(n\)\(n\), feature width\(p\)\(p\), data dimension\(D\)\(D\), and rank\(r\)\(r\)scale proportionally with the ambient dimension\(d\)\(d\)\. The resulting spectral characterization distinguishes eigenmodes that improve the empirical fit from those that improve population accuracy\. It also identifies the dimension of the orthogonal complement of the data subspace as a natural rank threshold\. The theoretical learning curves qualitatively reproduce the observed rank\-dependent overfitting, providing a tractable account of how bottlenecks or rank constraints can regularize score estimation and why their effectiveness depends on data dimension and sample size\.

## 1Introduction

Diffusion models have achieved remarkable success in high\-dimensional generative modeling, yet their ability to memorize training data remains an important theoretical and practical concern\([Somepalli et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib47);[Carlini et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib58)\)\. In the highly overparameterized regime, score networks can in principle fit increasingly fine details of the training dataset, and at sufficiently small noise levels the learned score may become strongly influenced by individual training samples\([Gu et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib34);[Yoon et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib27)\)\. At the same time, practical diffusion architectures often generate novel samples rather than simply reproducing their training sets\. Understanding which architectural biases prevent a score model from collapsing toward such memorizing solutions is therefore important for explaining the generalization behavior of diffusion models\.

A prominent feature of practical score networks is their encoder–decoder structure\([Ronneberger et al\., 2015](https://arxiv.org/html/2610.20850#bib.bib21)\)\. Although these models may contain a very large number of parameters, information is repeatedly compressed into lower\-dimensional representations before being decoded back to the ambient data space\. This suggests that architectural bottlenecks may act as a restriction on the class of score functions that can be learned\. In particular, they may prevent the model from independently fitting the score around every training example, while retaining the dominant directions needed to represent the underlying data distribution\. This motivates the hypothesis that a form of effective low\-rank structure in the learned score can suppress overfitting\.

In this work, we isolate this mechanism through two complementary models of the score function\. First, we numerically study the learning curves of a three\-layer neural network with a bottleneck structure\. Second, we consider a random\-feature parameterization with an explicit rank constraint on the learned second\-layer weight matrix\. Both models comprise an additional linear skip connection term\. The rank constraint in the random\-feature model acts as an idealized encoder–decoder bottleneck, forcing the learned vector field to factor through a low\-dimensional latent space\. For this model, we derive analytical expressions for the limiting learning curves\. By comparing these theoretical predictions with our empirical findings, we investigate how the rank of the learned score affects generalization in a controlled setting\.

Our analysis shows how restricting the rank can qualitatively change the learning behavior of denoising score matching\. High\-rank models have sufficient flexibility to exploit sample\-specific directions and can approach solutions strongly tied to the empirical dataset, whereas reducing the rank can suppresses these directions and force the estimator to retain only collective structure shared across samples\. The resulting framework provides a simple theoretical model for studying how bottlenecks in practical score architectures can act as an regularizer against memorization/overfitting\.

### 1\.1Main Findings

Figure 1:A qualitative illustration of our findings\.Our findings reveal how rank of the nonlinear score component, sample size, and data dimension, all affect generalization\. This is summarized schematically in Fig\.[1](https://arxiv.org/html/2610.20850#S1.F1)\. The horizontal axis represents the rankrr, while the vertical axis represents the number of training samplesnn\. For data spanning aDD\-dimensional subspace ofℝd\\mathbb\{R\}^\{d\}, two reference lines help organize the observed behavior:n=Dn=D, andr=d−Dr=d\-D\.

- •Whenn\>Dn\>D, the training samples span the data subspace almost surely in the models considered\. Increasing rank belowd−Dd\-Dimproves both training and test errors by allowing the model to learn more of the score component orthogonal to the data subspace\. This behavior motivates us to interpret the upper\-left part of Fig\.[1](https://arxiv.org/html/2610.20850#S1.F1)as generalization region\.
- •Increasing rank beyondd−Dd\-Dintroduces additional flexibility beyond that required by the data distribution\. Training error continues to decrease, while test error can increase or plateau\. The upper\-right region represents this tendency toward overfitting\. Additional training samples can mitigate this effect\.
- •Whenn<Dn<D, the training samples leave some directions of genuine data variation unobserved\. The model can mistake these directions for normal space to the data subspace, so even a score with rank less thand−Dd\-Dcan generalize poorly\. This explains the overfitting region below the horizontal boundary and why restricting rank alone is insufficient\.
- •Combining limited sample size with a large rank can further increase the separation between training and test errors, motivating the severe\-overfitting region in the lower\-right corner\.

These regions describe qualitative tendencies rather than sharp phase boundaries\. In particular, for certain values ofn/d,D/dn/d,D/d, additional rank can be detrimental even belowd−Dd\-D, including whenn\>Dn\>D\. Our random\-feature analysis characterizes this dependence through limiting training and test errors, while the numerical study illustrate how it appears in networks with learned features\.

## 2Related Works

##### Diffusion models and score estimation\.

Diffusion models learn a generative process through denoising and score estimation\([Sohl\-Dickstein et al\., 2015](https://arxiv.org/html/2610.20850#bib.bib48);[Song and Ermon, 2019](https://arxiv.org/html/2610.20850#bib.bib46);[Ho et al\., 2020](https://arxiv.org/html/2610.20850#bib.bib52);[Song et al\., 2020](https://arxiv.org/html/2610.20850#bib.bib30)\)\. Their sampling accuracy has been studied under assumptions on the error of the learned score\([Chen et al\., 2022](https://arxiv.org/html/2610.20850#bib.bib39);[Benton et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib43);[Chen et al\., 2023a](https://arxiv.org/html/2610.20850#bib.bib57);[Bortoli, 2022](https://arxiv.org/html/2610.20850#bib.bib54)\)\. These guarantees motivate a complementary question: how do finite data and restrictions on the score network determine that error? Analyses of score learning and distribution estimation address this question in several tractable settings\([Cui et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib38);[Shah et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib49);[Han et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib45);[Chen et al\., 2023b](https://arxiv.org/html/2610.20850#bib.bib40);[Cui et al\., 2025](https://arxiv.org/html/2610.20850#bib.bib24)\)\. Our focus is the effect of an explicit architectural rank constraint on denoising score matching and its generalization error\.

##### Memorization, data geometry, and architectural biases

Empirical evidence of training\-data replication\([Somepalli et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib47);[Carlini et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib58)\)has motivated theoretical studies of the empirical optimal score and the resulting sampling dynamics\([Biroli et al\., 2024](https://arxiv.org/html/2610.20850#bib.bib55);[Raya and Ambrogioni, 2023](https://arxiv.org/html/2610.20850#bib.bib31);[Ambrogioni, 2024](https://arxiv.org/html/2610.20850#bib.bib28);[Achilli et al\., 2024](https://arxiv.org/html/2610.20850#bib.bib44)\)\. Manifold and geometric perspectives further reveal how data structure influences memorization\([Pidstrigach, 2022](https://arxiv.org/html/2610.20850#bib.bib22);[Ross et al\., 2024](https://arxiv.org/html/2610.20850#bib.bib17);[Achilli et al\., 2025](https://arxiv.org/html/2610.20850#bib.bib20);[Wang and Pehlevan, 2025](https://arxiv.org/html/2610.20850#bib.bib10);[George et al\., 2025](https://arxiv.org/html/2610.20850#bib.bib12);[George and Macris, 2026](https://arxiv.org/html/2610.20850#bib.bib6);[Merger and Goldt, 2026](https://arxiv.org/html/2610.20850#bib.bib9)\)\. Architectural constraints can favor shared structure over memorization\.[Kamb and Ganguli \(2024\)](https://arxiv.org/html/2610.20850#bib.bib32)show that locality and equivariance enable novel recombinations of training\-image patches, while[Buchanan et al\. \(2025\)](https://arxiv.org/html/2610.20850#bib.bib14)identify a capacity\-dependent crossover between memorizing and generalizing models’ training losses\.[Ye et al\. \(2025\)](https://arxiv.org/html/2610.20850#bib.bib4)distinguish empirical from population score estimation and their network\-size requirements, supporting capacity control as a means to reduce memorization\.[Kadkhodaie et al\. \(2023\)](https://arxiv.org/html/2610.20850#bib.bib33)connect generalization to learned denoisers that perform shrinkage in geometry\-adaptive harmonic bases, highlighting how architectural inductive biases shape the directions used for denoising\.

##### Random features and precise learning curves\.

Random\-feature score models provide a tractable setting for studying finite\-sample generalization\([Li et al\., 2023](https://arxiv.org/html/2610.20850#bib.bib19);[Saha and Tran, 2025](https://arxiv.org/html/2610.20850#bib.bib15)\)\.[George et al\. \(2026\)](https://arxiv.org/html/2610.20850#bib.bib8)derive asymptotically precise test and train errors for Gaussian data, identifying the joint influence of sample size, feature width, and the number of noise realizations used in training\.[Bonnaire et al\. \(2025\)](https://arxiv.org/html/2610.20850#bib.bib16)identify a separation between the training times at which generalization and memorization emerge\. More recently,[Latourelle\-Vigeant et al\. \(2026\)](https://arxiv.org/html/2610.20850#bib.bib7)derive precise risk trajectories for kernel score under gradient flow, distinguishing generalizing, interpolating, and memorizing regimes\. Closer to our work,[Urfin et al\. \(2026\)](https://arxiv.org/html/2610.20850#bib.bib1)connect generalization and memorization to distinct spectral components of the neural tangent kernel Gram matrix and investigate spectral truncation as a regularizer\. In contrast, we impose explicit output rank constraints on a nonlinear random\-feature model for data supported on a low\-dimensional subspace\.

## 3Preliminaries

We briefly discuss some basics of diffusion models, and the score models that we use\.

### 3\.1Diffusion Models

Givennni\.i\.d\. samplesx1,x2,⋯,xnx\_\{1\},x\_\{2\},\\cdots,x\_\{n\}from an unknown distributionP0P\_\{0\}onℝd\\mathbb\{R\}^\{d\}, diffusion models draw new samples fromP0P\_\{0\}by time reversing a stochastic process\. This stochastic process gradually transformsP0P\_\{0\}into a tractable reference distribution, such as the isotropic Gaussian\. We use an Ornstein\-Uhlenbeck \(OU\) process\([Gardiner, 2009](https://arxiv.org/html/2610.20850#bib.bib23)\)for this forward diffusion:

dXt=−Xtdt\+2dBt,X0∼P0,\\differential X\_\{t\}=\-X\_\{t\}\\;\\differential t\+\\sqrt\{2\}\\;\\differential B\_\{t\}\\;,\\quad X\_\{0\}\\sim P\_\{0\}\\;,\(1\)whereBtB\_\{t\}is a standard Brownian motion inℝd\\mathbb\{R\}^\{d\}\. Conditional onX0X\_\{0\}, the distribution ofXtX\_\{t\}is𝒩⁡\(at​X0,ht​Id\)\\mathcal\{N\}\\left\(a\_\{t\}X\_\{0\},h\_\{t\}I\_\{d\}\\right\), whereat=e−ta\_\{t\}=e^\{\-t\}andht=1−e−2​th\_\{t\}=1\-e^\{\-2t\}, so thatat2\+ht=1a\_\{t\}^\{2\}\+h\_\{t\}=1\. Ast→∞t\\to\\infty,at→0a\_\{t\}\\to 0andht→1h\_\{t\}\\to 1, and the distribution ofXtX\_\{t\}converges to thedd\-dimensional standard Gaussian\. Fort\>0t\>0, its marginal density is

Pt\(x\)=\(2πht\)−d/2∫ℝddP0\(x0\)e−‖x−at​x0‖22​ht\.P\_\{t\}\(x\)=\{\(2\\pi h\_\{t\}\)^\{\-d/2\}\}\\int\_\{\\mathbb\{R\}^\{d\}\}\\differential P\_\{0\}\(x\_\{0\}\)\\;e^\{\-\\frac\{\\norm\{x\-a\_tx\_0\}^\{2\}\}\{2h\_\{t\}\}\}\\;\.\(2\)For a fixedT\>0T\>0, the reverse process starts fromYT∼PTY\_\{T\}\\sim P\_\{T\}and evolves backward in time according to

−dYt=\(Yt\+2∇logPt\(Yt\)\)dt\+2dB~t,\-\\differential Y\_\{t\}=\\left\(Y\_\{t\}\+2\\nabla\\log P\_\{t\}\(Y\_\{t\}\)\\right\)\\;\\differential t\+\\sqrt\{2\}\\;\\differential\\tilde\{B\}\_\{t\}\\;,\(3\)whereB~t\\tilde\{B\}\_\{t\}is a different instance of standard Brownian motion\. Classical results state thatYtY\_\{t\}andXtX\_\{t\}have the same marginal distribution at everytt\([Anderson, 1982](https://arxiv.org/html/2610.20850#bib.bib56);[Haussmann and Pardoux, 1986](https://arxiv.org/html/2610.20850#bib.bib53)\), yieldingY0∼P0Y\_\{0\}\\sim P\_\{0\}\. In practice, the unknown terminal distributionPTP\_\{T\}is approximated by𝒩⁡\(0,Id\)\\mathcal\{N\}\\left\(0,I\_\{d\}\\right\)for sufficiently largeTT\. Implementing the reverse process also requires estimating the unknown*score function*∇log⁡Pt\\nabla\\log P\_\{t\}\.

Denoising score matching \(DSM\)\([Vincent, 2011](https://arxiv.org/html/2610.20850#bib.bib29)\)estimates this score using noisy observationsyt=at​x\+ht​zy\_\{t\}=a\_\{t\}x\+\\sqrt\{h\_\{t\}\}z, wherex∼P0x\\sim P\_\{0\}andz∼𝒩⁡\(0,Id\)z\\sim\\mathcal\{N\}\\left\(0,I\_\{d\}\\right\)are independent\. The conditional score provides the tractable target∇logPt\(yt∣x\)=−z/ht\\nabla\\log P\_\{t\}\(y\_\{t\}\\mid x\)=\-z/\\sqrt\{h\_\{t\}\}fort\>0t\>0, suggesting the population objective

ℒDSM​\(st\)=1d​𝔼x0,z​‖ht​st​\(yt\)\+z‖2\.\\mathcal\{L\}\_\{\\text\{DSM\}\}\(s\_\{t\}\)=\\frac\{1\}\{d\}\\mathbb\{E\}\_\{x\_\{0\},z\}\{\\norm\{\\sqrt\{h\_t\}s\_t\(y\_t\)\+z\}^\{2\}\}\.Replacing the expectation overP0P\_\{0\}by the training samples gives the empirical loss studied in this work:

ℒ⁡\(st\)\\displaystyle\\mathcal\{L\}\(s\_\{t\}\)=1d​n​∑i=1n𝔼z​‖ht​st​\(at​xi\+ht​z\)\+z‖2\.\\displaystyle=\\frac\{1\}\{dn\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\norm\{\\sqrt\{h\_t\}s\_t\(a\_tx\_i\+\\sqrt\{h\_t\}z\)\+z\}^\{2\}\}\.\(4\)In practice, the score functionssis chosen from a parametric function class and trained by minimizing this empirical objective, subject to the model constraints or regularization\. We next introduce the rank\-constrained score models used in our study\.

### 3\.2Score Model for Numerical Study: Rank Constrained 3\-Layer NN

For each diffusion timett, we consider the score model

st​\(y\)=−y\+Btr​Atp​ϱ​\(Wtd​y\+bt\),s\_\{t\}\(y\)=\-y\+\\frac\{B\_\{t\}\}\{\\sqrt\{r\}\}\\frac\{A\_\{t\}\}\{\\sqrt\{p\}\}\\varrho\\left\(\\frac\{W\_\{t\}\}\{\\sqrt\{d\}\}y\+b\_\{t\}\\right\),\(5\)whereWt∈ℝp×d,Bt∈ℝd×r,At∈ℝr×p,bt∈ℝpW\_\{t\}\\in\\mathbb\{R\}^\{p\\times d\},\\;B\_\{t\}\\in\\mathbb\{R\}^\{d\\times r\},\\;A\_\{t\}\\in\\mathbb\{R\}^\{r\\times p\},\\;b\_\{t\}\\in\\mathbb\{R\}^\{p\}are learnable parameters, andϱ\\varrhois a non\-linear activation function\. Here,ppis the hidden\-layer width andrrcontrols the rank of the nonlinear component\. The parameters are learned independently at eachtt\.

The first term,−x\-x, acts as a linear skip connection\. The remaining term provides a nonlinear, learnable correction whose output belongs to the column space ofBtB\_\{t\}\. Sincerank⁡\(Bt​At\)≤r,\\operatorname\{rank\}\(B\_\{t\}A\_\{t\}\)\\leq r,this correction is restricted to an output subspace of dimension at mostrr\. Small values ofrrtherefore impose a strong bottleneck, whereas larger values allow the score model to represent a richer collection of directions\.

This parameterization allows us to study whether rank constraining helps in capturing the population\-level geometry of the data while preventing the model from fitting sample\-specific fluctuations\. In particular, we investigate whether increasingrrreduces the training error while eventually increasing the train–test gap, indicating a transition from population\-level learning to finite\-sample overfitting\.

### 3\.3Theoretical Score Model: Random Features with a Skip Connection

For our theoretical analysis, we consider a random\-feature neural network\([Rahimi and Recht, 2007](https://arxiv.org/html/2610.20850#bib.bib50)\)with fixed first\-layer weights, a trainable second layer, and a linear skip connection\. In particular, the score parameterization fromℝd\\mathbb\{R\}^\{d\}toℝd\\mathbb\{R\}^\{d\}given by

st​\(y\|Wt\)=−y\+At​ϱ​\(Wt​y/d\),s\_\{t\}\(y\|W\_\{t\}\)=\-y\+A\_\{t\}\\varrho\\left\(W\_\{t\}y/\\sqrt\{d\}\\right\),\(6\)whereWt∈ℝp×dW\_\{t\}\\in\\mathbb\{R\}^\{p\\times d\}has entries drawn i\.i\.d\. from𝒩⁡\(0,1\)\\mathcal\{N\}\\left\(0,1\\right\),ϱ\\varrhois an activation function applied element\-wise, andAt∈ℝd×pA\_\{t\}\\in\\mathbb\{R\}^\{d\\times p\}contains the trainable weights\.

We impose the constraintrank​\(At\)≤r\\text\{rank\}\(A\_\{t\}\)\\leq r, which restricts the nonlinear correction to an output subspace of dimension at mostrr\. This provides a tractable counterpart to the bottleneck architecture used in our numerical study\. The feature widthppcontrols the size of the fixed nonlinear representation, whilerrcontrols the rank of the learned component\.

Random\-feature models admit precise theoretical analyses while reproducing phenomena observed in more complex neural networks, including double descent in overparameterized regimes\([Mei and Montanari, 2022](https://arxiv.org/html/2610.20850#bib.bib13);[Hu et al\., 2024](https://arxiv.org/html/2610.20850#bib.bib2);[Bodin and Macris, 2021](https://arxiv.org/html/2610.20850#bib.bib42);[Bodin and Macris, 2022](https://arxiv.org/html/2610.20850#bib.bib11)\)\.

## 4Numerical Study

For our numerical experiments, the data distributionP0P\_\{0\}is uniform distribution on a sphere embedded inℝd\\mathbb\{R\}^\{d\}:x=Θ​ξ,Θ⊤​Θ=ID,ξ=D​g‖g‖,g∼𝒩⁡\(0,ID\)\.x=\\Theta\\xi,\\;\\Theta^\{\\top\}\\Theta=I\_\{D\},\\;\\xi=\\sqrt\{D\}\\,\\frac\{g\}\{\\\|g\\\|\},\\;g\\sim\\mathcal\{N\}\(0,I\_\{D\}\)\.Thus the data manifold has intrinsic dimensionD−1D\-1, and is embedded in a linear subspace of dimensionDD\.

For each rankrr, we train a separate score model given in \([5](https://arxiv.org/html/2610.20850#S3.E5)\)\. We sweeprrand optimize the denoising score\-matching objective \([4](https://arxiv.org/html/2610.20850#S3.E4)\)\. Fig\.[2](https://arxiv.org/html/2610.20850#S4.F2)reports train and test error estimates of this same objective after3030k Adam steps, averaged over three random seeds\. More details on the simulation can be found in Appendix[C](https://arxiv.org/html/2610.20850#A3)\.

Figure 2:Simulation results for our numerical model, withtanh\\tanhactivation att=0\.01t=0\.01,p/d=2p/d=2\.First, we observe that the train error decreases monotonically with increasingrracross all settings\. However, the behavior of test error depends on the samples sizenn, and data dimensionDD\. Forn\>Dn\>D, the test error decreases as we increaserrtilld−Dd\-D\. Beyondd−Dd\-D, we observe an abrupt change in behavior: with additional rank, the test error increases or plateaus, indicating finite\-sample overfitting\. Increasingnnsuppresses this effect strongly at smallerDD, but does not eliminate it at largerDD\. Whenn<Dn<D, the sudden change in test error atr=d−Dr=d\-Ddisappears\. This is because the clean training samples cannot span the full data subspace, making the unseen data subspace directions a source of poor generalization\.

The observed rank dependence is consistent with learning the score component in the space orthogonal to the data span, whose dimension isd−Dd\-D\. Section[6](https://arxiv.org/html/2610.20850#S6)discusses these aspects in detail\.

## 5Theoretical Study

We analyze the random\-feature score model given in \([6](https://arxiv.org/html/2610.20850#S3.E6)\) at a fixed diffusion time\. Suppressing the time index, we write

sA​\(y\)=−y\+A​ϕ​\(y\),ϕ⁡\(y\)=ϱ⁡\(W​y/d\),s\_\{A\}\(y\)=\-y\+A\\phi\(y\),\\qquad\\phi\(y\)=\\varrho\\left\(Wy/\\sqrt\{d\}\\right\),whereWWis held fixed andAAis learned subject to a rank constraint\. We consider Gaussian datax∼𝒩⁡\(0,𝒞\)x\\sim\\mathcal\{N\}\(0,\\mathcal\{C\}\)with𝒞=diag⁡\(ID,0\)\\mathcal\{C\}=\\operatorname\{diag\}\(I\_\{D\},0\), providing a tractable model of data supported on aDD\-dimensional subspace of the ambient space\.

Given independent training samplesx1,…,xnx\_\{1\},\\ldots,x\_\{n\}, letyi=a​xi\+h​zy\_\{i\}=ax\_\{i\}\+\\sqrt\{h\}z, wherez∼𝒩⁡\(0,Id\)z\\sim\\mathcal\{N\}\(0,I\_\{d\}\)is independent of the data\. The rank\-constrained DSM estimator solves

minrank​\(A\)≤r⁡ℒ⁡\(A\)=∑i=1n𝔼z​‖h​\(A​ϕ​\(yi\)−yi\)\+z‖2\.\\min\_\{\\text\{rank\}\(A\)\\leq r\}\\mathcal\{L\}\(A\)=\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\norm\{\\sqrt\{h\}\(A\\phi\(y\_i\)\-y\_i\)\+z\}^\{2\}\}\.\(7\)
To characterize the solution, define matrices

U\\displaystyle U=1n​∑i=1n𝔼z​\[ϕ⁡\(yi\)​ϕ​\(yi\)T\],\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\left\[\\phi\(y\_\{i\}\)\\phi\(y\_\{i\}\)^\{T\}\\right\]\},V\\displaystyle V=1n​∑i=1n𝔼z​\[ϕ⁡\(yi\)​\(yi−z/h\)T\]\.\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\left\[\\phi\(y\_\{i\}\)\(y\_\{i\}\-z/\\sqrt\{h\}\)^\{T\}\\right\]\}\.Here,UUandVVare the empirical feature second moment and feature–target cross moment, respectively\. The following theorem gives an exact finite\-dimensional solution to \([7](https://arxiv.org/html/2610.20850#S5.E7)\) through a truncated singular value decomposition\.

###### Theorem 1\(Rank\-constrained Random Features estimator\)\.

Fixt\>0t\>0, leta=e−ta=e^\{\-t\}andh=1−e−2​th=1\-e^\{\-2t\}, and condition on the training samples andWW\. Assume thatU≻0U\\succ 0\. Letq=min⁡\(d,p\)q=\\min\(d,p\)andr∈\{0,…,q\}r\\in\\\{0,\\ldots,q\\\}\. WriteVTU−1/2=SΣRT,V^\{T\}U^\{\-1/2\}=S\\Sigma R^\{T\},with singular valuess1≥⋯≥sq≥0s\_\{1\}\\geq\\cdots\\geq s\_\{q\}\\geq 0, and letΣr\\Sigma\_\{r\}retain therrlargest singular values\. Then a solution to \([7](https://arxiv.org/html/2610.20850#S5.E7)\) is

A^r=SΣrRTU−1/2\.\\widehat\{A\}\_\{r\}=S\\Sigma\_\{r\}R^\{T\}U^\{\-1/2\}\.\(8\)

###### Proof Sketch\.

The normal equationALS​U=VTA\_\{\\mathrm\{LS\}\}U=V^\{T\}gives

ℒtrain​\(A\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{train\}\}\(A\)=ℒtrain​\(ALS\)\+hd​‖\(A−ALS\)​U1/2‖F2\.\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{train\}\}\(A\_\{\\mathrm\{LS\}\}\)\+\\frac\{h\}\{d\}\\\|\(A\-A\_\{\\mathrm\{LS\}\}\)U^\{1/2\}\\\|\_\{F\}^\{2\}\.Since multiplication byU1/2U^\{1/2\}preserves rank, the Eckart–Young theorem yields \([8](https://arxiv.org/html/2610.20850#S5.E8)\)\. ∎

Equation[8](https://arxiv.org/html/2610.20850#S5.E8)shows that rank truncation retains the leading singular modes after accounting for the empirical feature covariance\.

We next state the limiting characterization of test and train errors for the rank constrained solution\. First, we define test and train loss as follows:

ℒtest​\(A\)\\displaystyle\\mathcal\{L\}\_\{\\text\{test\}\}\(A\)=1d​𝔼x0,z​‖ht​sA​\(at​x0\+ht​z\)\+z‖2,\\displaystyle=\\frac\{1\}\{d\}\\mathbb\{E\}\_\{x\_\{0\},z\}\{\\norm\{\\sqrt\{h\_t\}s\_A\(a\_tx\_0\+\\sqrt\{h\_t\}z\)\+z\}^\{2\}\},ℒtrain​\(A\)\\displaystyle\\mathcal\{L\}\_\{\\text\{train\}\}\(A\)=1d​n​∑i=1n𝔼z​‖ht​sA​\(at​xi\+ht​z\)\+z‖2\.\\displaystyle=\\frac\{1\}\{dn\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\norm\{\\sqrt\{h\_t\}s\_A\(a\_tx\_i\+\\sqrt\{h\_t\}z\)\+z\}^\{2\}\}\.
###### Assumption 1\(Activation function\)\.

We assume that forg∼𝒩⁡\(0,1\)g\\sim\\mathcal\{N\}\(0,1\), and for allκ\>0\\kappa\>0,𝔼g​\[ϱ⁡\(κ​g\)\]=0\\mathbb\{E\}\_\{g\}\{\\left\[\\varrho\(\\kappa g\)\\right\]\}=0\. Setσ2=a2​ψD\+h\\sigma^\{2\}=a^\{2\}\\psi\_\{D\}\+h, andμ=1σ​𝔼g​\[g​ϱ​\(σ​g\)\]\\mu=\\frac\{1\}\{\\sigma\}\\mathbb\{E\}\_\{g\}\{\\left\[g\\varrho\(\\sigma g\)\\right\]\}\. Constantss,v,s,v,andv0v\_\{0\}are as defined in eqns \([17](https://arxiv.org/html/2610.20850#A1.E17)\), \([18](https://arxiv.org/html/2610.20850#A1.E18)\), \([19](https://arxiv.org/html/2610.20850#A1.E19)\) in Appendix[A\.2](https://arxiv.org/html/2610.20850#A1.SS2)\. We assume thatμ\\muandssare positive\.

###### Theorem 2\(Limiting characterization of test and train errors\)\.

Fixt\>0t\>0and letd→∞d\\to\\inftywithψ⋄=⋄d\\psi\_\{\\diamond\}=\\frac\{\\diamond\}\{d\}, for⋄∈\{n,p,D,r\}\\diamond\\in\\\{n,p,D,r\\\}held fixed\. Let the training samples be independent draws from𝒩⁡\(0,diag⁡\(ID,0\)\)\\mathcal\{N\}\(0,\\operatorname\{diag\}\(I\_\{D\},0\)\), independent of the i\.i\.d\. standard Gaussian entries ofWW\. Letϱ\\varrhosatisfy Assumption[1](https://arxiv.org/html/2610.20850#Thmassumption1), and suppose that the Gaussian equivalence principle given in Assumption[2](https://arxiv.org/html/2610.20850#Thmassumption2)of Appendix[A\.2](https://arxiv.org/html/2610.20850#A1.SS2)\. is valid\.

LetG00,G17,G22,G44G\_\{00\},G\_\{17\},G\_\{22\},G\_\{44\}be obtained by solving the system of fixed point equations \([34](https://arxiv.org/html/2610.20850#A1.E34)\) corresponding to the limiting block resolvent of \([28](https://arxiv.org/html/2610.20850#A1.E28)\)\. ForImz\>0\\imaginary z\>0, define

e1​\(z\)\\displaystyle e\_\{1\}\(z\)=−ψps2​z​G00​\(z\),e2​\(z\)=−ψDa2​μ​G17​\(z\),\\displaystyle=\-\\frac\{\\psi\_\{p\}\}\{s^\{2\}z\}G\_\{00\}\(z\),\\quad e\_\{2\}\(z\)=\-\\frac\{\\psi\_\{D\}\}\{a^\{2\}\\mu\}G\_\{17\}\(z\),e3​\(z\)\\displaystyle e\_\{3\}\(z\)=1−ψDa4​μ2​\(1−G22​\(z\)\)\.\\displaystyle=\\frac\{1\-\\psi\_\{D\}\}\{a^\{4\}\\mu^\{2\}\}\\bigl\(1\-G\_\{22\}\(z\)\\bigr\)\.\(9\)The limiting eigenvalue distributionν\\nuofM=VT​U−1​VM=V^\{T\}U^\{\-1\}Vhas Stieltjes transform

sν​\(z\)=ψD​\(1−G44​\(z\)\)\+\(1−ψD\)​\(1−G22​\(z\)\)−1z\.\\textbf\{s\}\_\{\\nu\}\(z\)=\\frac\{\\psi\_\{D\}\(1\-G\_\{44\}\(z\)\)\+\(1\-\\psi\_\{D\}\)\(1\-G\_\{22\}\(z\)\)\-1\}\{z\}\.\(10\)Suppose there is aΓψr\>0\\Gamma\_\{\\psi\_\{r\}\}\>0satisfying

ν⁡\(\(Γψr,∞\)\)=ψr\.\\nu\(\(\\Gamma\_\{\\psi\_\{r\}\},\\infty\)\)=\\psi\_\{r\}\.\(11\)Define

m​\(z\)=μ2​e2​\(z\)\+\(h​μ2−2​a4​μ2z\)​e3​\(z\)\+v2​e1​\(z\)\.\\textbf\{m\}\(z\)=\\mu^\{2\}e\_\{2\}\(z\)\+\\left\(h\\mu^\{2\}\-\\frac\{2a^\{4\}\\mu^\{2\}\}\{z\}\\right\)e\_\{3\}\(z\)\+v^\{2\}e\_\{1\}\(z\)\.\(12\)Then, withω=1\+h⁡\(σ2−2\)\\omega=1\+h\(\\sigma^\{2\}\-2\), the estimator \([8](https://arxiv.org/html/2610.20850#S5.E8)\) satisfies

ℒtest​\(A^r\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{test\}\}\(\\widehat\{A\}\_\{r\}\)→ℙω\+hπlimK→∞limη↓0∫ΓψrKxImm\(x\+iη\)dx,\\displaystyle\\xrightarrow\{\\mathbb\{P\}\}\\omega\+\\frac\{h\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}x\\,\\imaginary\\textbf\{m\}\(x\+i\\eta\)\\,\\differential x,ℒtrain​\(A^r\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{train\}\}\(\\widehat\{A\}\_\{r\}\)→ℙω−hπlimK→∞limη↓0∫ΓψrKxImsν\(x\+iη\)dx\.\\displaystyle\\xrightarrow\{\\mathbb\{P\}\}\\omega\-\\frac\{h\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}x\\,\\imaginary\\textbf\{s\}\_\{\\nu\}\(x\+i\\eta\)\\,\\differential x\.

###### Proof sketch\.

We outline the proof here\. Complete steps can be found in Appendix[A\.2](https://arxiv.org/html/2610.20850#A1.SS2)\. Expanding the DSM losses gives quadratic, cross, andA^r\\widehat\{A\}\_\{r\}independent terms; see \([15](https://arxiv.org/html/2610.20850#A1.E15)\) and \([27](https://arxiv.org/html/2610.20850#A1.E27)\)\. The test error involves population moments, while the training error involves the empirical moments that define the optimizer\.

The first step is to obtain tractable representations of these moment matrices\. Using Hermite expansions and truncation up to leading order give \([16](https://arxiv.org/html/2610.20850#A1.E16)\)\. TheGaussian equivalence principle\([Gerace et al\., 2020](https://arxiv.org/html/2610.20850#bib.bib36);[Goldt et al\., 2022](https://arxiv.org/html/2610.20850#bib.bib35);[Hu and Lu, 2023](https://arxiv.org/html/2610.20850#bib.bib51);[Montanari and Saeed, 2022](https://arxiv.org/html/2610.20850#bib.bib3)\)then replaces the nonlinear noise\-averaged feature matrixH=ϱ0​\(W​X/d\)H=\\varrho\_\{0\}\(WX/\\sqrt\{d\}\), for the limiting trace quantities under consideration, by

HG=a​μ​W​X/d\+v0​Ω,H^\{G\}=a\\mu WX/\\sqrt\{d\}\+v\_\{0\}\\Omega,whereΩ\\Omegahas independent standard Gaussian entries and is independent ofWWandXX\. The linear term preserves the feature–data correlations, whilev0​Ωv\_\{0\}\\Omegarepresents the residual feature variance\. This is an asymptotic equivalence for the relevant observables, rather than an entrywise identity\. It reduces the moment matrices to expressions involving Gaussian random matrices and the coefficientsμ,s,v,v0\\mu,s,v,v\_\{0\}\.

Next, Theorem[1](https://arxiv.org/html/2610.20850#Thmtheorem1)implements the rank constraint by retaining the leading singular modes ofVTU−1/2V^\{T\}U^\{\-1/2\}\. Their squared singular values are the nonzero eigenvalues shared byM=VT​U−1​VM=V^\{T\}U^\{\-1\}VandM~=U−1/2VVTU−1/2\\tilde\{M\}=U^\{\-1/2\}VV^\{T\}U^\{\-1/2\}\. Stone’s formula converts this truncation into a resolvent integral\. For example, with a cutoffΓ\\Gammaseparating the retained eigenvalues and an upper endpointKKabove the spectrum,

SΣrΣrTST=1πlimη↓0∫ΓKxIm\(M−\(x\+iη\)I\)−1dx\.S\\Sigma\_\{r\}\\Sigma\_\{r\}^\{T\}S^\{T\}=\\frac\{1\}\{\\pi\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\}^\{K\}x\\,\\imaginary\(M\-\(x\+i\\eta\)I\)^\{\-1\}\\,\\differential x\.The analogous formula forM~\\tilde\{M\}handles the quadratic term\. In the proportional limit, the rank fraction selects the upper\-tail cutoffΓψr\\Gamma\_\{\\psi\_\{r\}\}through \([11](https://arxiv.org/html/2610.20850#S5.E11)\)\.

These representations reduce the losses to normalized traces of rational expressions in random matrices\. SetQ⁡\(z\)=\(V​VT−z​U\)−1Q\(z\)=\(VV^\{T\}\-zU\)^\{\-1\}\. The identity

\(M−z​I\)−1=z−1​\(VT​Q​\(z\)​V−I\)\(M\-zI\)^\{\-1\}=z^\{\-1\}\\bigl\(V^\{T\}Q\(z\)V\-I\\bigr\)expresses the Stieltjes transform ofMMthroughE0​\(z\)=d−1​tr⁡\(VT​Q​\(z\)​V\)E\_\{0\}\(z\)=d^\{\-1\}\\tr\{V^TQ\(z\)V\}, while the population weights produce the tracesE1,E2,E3E\_\{1\},E\_\{2\},E\_\{3\}in \([25](https://arxiv.org/html/2610.20850#A1.E25)\)\. To compute their limits, thelinear\-penciltechnique embeds these rational expressions into selected inverse blocks of the larger matrixL⁡\(z\)L\(z\)in \([28](https://arxiv.org/html/2610.20850#A1.E28)\)\. Its random blocks depend linearly on the underlying Gaussian matrices, and block matrix inverse recovers the required resolvents\. A computation rooted in operator\-valued free probability theory\([Far et al\., 2006](https://arxiv.org/html/2610.20850#bib.bib37);[Helton et al\., 2018](https://arxiv.org/html/2610.20850#bib.bib25);[Adlam and Pennington, 2020](https://arxiv.org/html/2610.20850#bib.bib26);[Bodin, 2024](https://arxiv.org/html/2610.20850#bib.bib41);[Subramonian and Dohmatob, 2025](https://arxiv.org/html/2610.20850#bib.bib5)\)yields the fixed\-point system \([34](https://arxiv.org/html/2610.20850#A1.E34)\) for the limiting normalized block traces\. Extracting the appropriate blocks gives \([9](https://arxiv.org/html/2610.20850#S5.E9)\) and \([10](https://arxiv.org/html/2610.20850#S5.E10)\)\.

Finally, summing up the quadratic \(μ2​e2\+h​μ2​e3\+v2​e1\\mu^\{2\}e\_\{2\}\+h\\mu^\{2\}e\_\{3\}\+v^\{2\}e\_\{1\}\) and cross \(−2a4μ2e3/z\-2a^\{4\}\\mu^\{2\}e\_\{3\}/z\) contributions to test error gives \([12](https://arxiv.org/html/2610.20850#S5.E12)\)\. For training error, the empirical normal equation combines the quadratic and cross terms into a single term\. Applying spectral integration with the rank cutoff and adding the constant term1\+h⁡\(σ2−2\)1\+h\(\\sigma^\{2\}\-2\)gives the stated learning curves\. The spectral integrals are understood with a finite upper endpoint beyond the spectral support, before takingη↓0\\eta\\downarrow 0\. ∎

The characterization reduces the computation into solving the fixed\-point equations \([34](https://arxiv.org/html/2610.20850#A1.E34)\), finding the spectral cutoff \([11](https://arxiv.org/html/2610.20850#S5.E11)\), and evaluating two integrals\. Rank enters through the cutoffΓψr\\Gamma\_\{\\psi\_\{r\}\}, whileψn\\psi\_\{n\},ψp\\psi\_\{p\}, andψD\\psi\_\{D\}shape the resolvent functionsmandsν\\textbf\{s\}\_\{\\nu\}\.

Figure 3:Analytical and finite dimensional\(d=200\)\(d=200\)simulation results for our theoretical model withtanh\\tanhactivation att=0\.01t=0\.01,ψp=2\.0\\psi\_\{p\}=2\.0\. Top line: test and train errors\. Bottom line: Spectral selection thresholdsΓψr\\Gamma\_\{\\psi\_\{r\}\}\.Fig\.[3](https://arxiv.org/html/2610.20850#S5.F3)presents the theoretical learning curves and finite\-dimensional simulations, which qualitatively reproduce the numerical study’s dependence on rank, sample size, and data dimension\. Training error decreases with rank, whereas test error exhibits distinct behaviors\. For smallerψD\\psi\_\{D\}andψn\\psi\_\{n\}, test error initially decreases before rising sharply atψr=1−ψD\\psi\_\{r\}=1\-\\psi\_\{D\}, while increasingψn\\psi\_\{n\}substantially suppresses this deterioration\. At largerψD\\psi\_\{D\}and smallerψn\\psi\_\{n\}, test error increases throughout the rank range, showing that additional rank can be detrimental even belowψr=1−ψD\\psi\_\{r\}=1\-\\psi\_\{D\}\. For largeψn\\psi\_\{n\}however, the variation in test error is small\. Section[6](https://arxiv.org/html/2610.20850#S6)discusses the mechanisms underlying these observations\. Fig\.[4](https://arxiv.org/html/2610.20850#S5.F4)shows the limiting spectrum ofMM, whose leading modes are retained by the rank constraint\.

Figure 4:Spectrum ofMMwithtanh\\tanhactivation att=0\.01t=0\.01,ψp=2\.0\\psi\_\{p\}=2\.0\.
## 6Discussion

Figs\.[2](https://arxiv.org/html/2610.20850#S4.F2)and[3](https://arxiv.org/html/2610.20850#S5.F3)\(and Figs\.[6](https://arxiv.org/html/2610.20850#A2.F6)and[7](https://arxiv.org/html/2610.20850#A2.F7)in Appendix[B\.1](https://arxiv.org/html/2610.20850#A2.SS1)\) show that increasing the rank of the nonlinear score component improves the fit to the training dataset without always improving generalization\. We organize these observations around the four regimes in Fig\.[1](https://arxiv.org/html/2610.20850#S1.F1), determined by the sample thresholdn=Dn=Dand the reference rankr=d−Dr=d\-D\. These thresholds help explain the competing effects of model expressivity and estimation errors, but do not define sharp boundaries between decreasing and increasing test error\.

##### Sufficient samples and restricted rank:n\>Dn\>D,r<d−Dr<d\-D\.

Whenn\>Dn\>D, the clean training samples span the data subspace almost surely for the distributions considered here\. In this regime, increasing rank can improve generalization by allowing the nonlinear score component to represent more directions orthogonal to that subspace\. The underlying geometry is particularly transparent in the Gaussian case: the noisy covariance is𝒞~=diag​\(ID,h​Id−D\)\\tilde\{\\mathcal\{C\}\}=\\text\{diag\}\(I\_\{D\},hI\_\{d\-D\}\), and the ideal score is∇log⁡Pt​\(y\)=−y−a2h​diag​\(0,Id−D\)​y\\nabla\\log P\_\{t\}\(y\)=\-y\-\\frac\{a^\{2\}\}\{h\}\\text\{diag\}\(0,I\_\{d\-D\}\)y\. The skip connection already gives the correct score along the data subspace, while the remaining component acts in its normal space and has rankd−Dd\-D\. A rank belowd−Dd\-Dcannot represent this component fully, and the factora2/ha^\{2\}/hmakes the missing normal directions especially consequential at small diffusion time\. This mechanism explains the initial decrease of both training and test errors observed at smallerD/dD/d\. The spectra in Fig\.[4](https://arxiv.org/html/2610.20850#S5.F4)offer a complementary perspective: atD/d=0\.30D/d=0\.30, an upper band is separated from a lower component, with clearer separation for largern/dn/d\. In the population setting with an unconstrained linear score, the analogue of matrixMMis given by\(I−𝒞~−1\)2​𝒞~\(I\-\\tilde\{\\mathcal\{C\}\}^\{\-1\}\)^\{2\}\\tilde\{\\mathcal\{C\}\}\. It hasd−Dd\-Dnonzero eigenvalues, all equal toa4/ha^\{4\}/h\. This reference spectrum suggests that the upper band in Fig\.[4](https://arxiv.org/html/2610.20850#S5.F4)primarily reflects normal\-space directions\. However, it should be noted that the empirical setting and the random\-feature representation alter both the spectrum and the retained directions\. Consequently, even whenn\>Dn\>D, additional rank belowd−Dd\-Dneed not always improve test error\.

##### Sufficient samples and excess rank:n\>Dn\>D,r\>d−Dr\>d\-D\.

Beyondd−Dd\-D, additional rank is no longer needed to represent the normal\-space component of the Gaussian population score\. It nevertheless provides further flexibility to fit the training samples\. Theorem[1](https://arxiv.org/html/2610.20850#Thmtheorem1)makes this distinction precise: the rank constraint retains the largest singular modes ofVTU−1/2V^\{T\}U^\{\-1/2\}, or equivalently the largest eigenvalues ofMM, and admitting the next mode decreases the training loss byh​sr\+12/dhs\_\{r\+1\}^\{2\}/d\. The effect on test error also depends on the direction of that mode and the population feature moments\. In Theorem[2](https://arxiv.org/html/2610.20850#Thmtheorem2), this distinction appears through functionm​\(z\)\\textbf\{m\}\(z\)which is different from the pure spectral objectsν\\textbf\{s\}\_\{\\nu\}\. Thus, a large empirical eigenvalue establishes a training benefit without ensuring a corresponding test benefit\. Additional modes can capture sample\-specific structure, leading to a plateau or an increase in test error despite continued improvement in training error\. This interpretation is consistent with the approximate plateau observed for the learned network at smallerD/dD/dandn/d=2n/d=2, and with the more pronounced train–test separation at largerD/dD/d\. Increasing sample size reduces estimation uncertainty, but does not eliminate the effects of excessive rank at the sample sizes considered\.

##### Insufficient samples and restricted rank:n<Dn<D,r<d−Dr<d\-D\.

Restricting rank is less effective when the training samples do not span the true data subspace\. Whenn<Dn<D, at leastD−nD\-ndirections of genuine data variation remain unobserved\. A model can confuse these directions with the normal space and erroneously learn a strong denoising response along these directions\. Due to its larger magnitude, the learned score component along the unseen directions are retained even under rank constraint\. This mechanism is consistent with the increasing test error at largerD/dD/dandn/d=0\.5n/d=0\.5, even within the ranger<d−Dr<d\-D\. AtD/d=0\.69D/d=0\.69andn/d=0\.5n/d=0\.5, Fig\.[4](https://arxiv.org/html/2610.20850#S5.F4)shows appreciable density between the lower and upper parts of the spectrum, consistent with a less distinct separation of spectral components\. The spectrum alone does not identify the alignment of individual eigenmodes, but it illustrates why selecting eigenmodes that benefit the empirical training need not isolate the directions required for generalization\.

##### Insufficient samples and excess rank:n<Dn<D,r\>d−Dr\>d\-D\.

Combining insufficient samples with a large rank allows both sources of overfitting to act together\. The training data leave part of the true data subspace unresolved, while the additional rank permits the model to fit more directions beyond those needed for the population normal\-space component\. Training error continues to decrease, but this added flexibility can strengthen responses along unobserved data directions and capture further sample\-specific structure, widening the train–test gap\. This combination motivates the severe\-overfitting region in Fig\.[1](https://arxiv.org/html/2610.20850#S1.F1)and is consistent with the substantial deterioration at higher ranks for largerD/dD/dandn/d=0\.5n/d=0\.5\.

Fig\.[5](https://arxiv.org/html/2610.20850#S6.F5), which plots training and test errors againstψn\\psi\_\{n\}for two values ofψr\\psi\_\{r\}, further illustrates how sample size changes the consequences of a given rank constraint\. Oncen\>Dn\>D, the ambiguity caused by an unobserved part of the data span essentially disappears\.

Figure 5:Theoretical results withtanh\\tanhactivation att=0\.01t=0\.01,ψp=2\.0\\psi\_\{p\}=2\.0\.Together, the numerical and theoretical results show that a useful rank constraint depends on both the data dimension and the sample size available to estimate the data geometry\. The random\-feature model isolates a mechanism for rank\-dependent overfitting: empirical spectral selection rewards training fit, whereas generalization also depends on the population alignment of the selected directions\.

## 7Conclusion, Limitations, and Future Work

We studied how output\-rank constraints affect generalization in denoising score matching\. Experiments with trainable networks and a proportional\-limit random\-feature analysis show that additional rank can improve training fit while worsening test error\. Conversely, a rank\-constraint can indeed suppress overfitting\. The spectral characterization ofMMexplains this tradeoff through the population contributions of the retained modes, and highlights the joint role of data geometry and sample size\. These findings establish rank constraints as a useful, but data\-dependent, form of regularization for score estimation\.

Our study isolates the effect of rank, which requires a tractable framework for rank control\. The theoretical analysis considers fixed random features and Gaussian data supported on a linear subspace, while the numerical experiments use trainable networks with data on an embedded sphere\. Their qualitative agreement supports the proposed mechanism, although the theory is not intended to predict every learning curve of a network with learned features\. Moreover, the errors are evaluated at a fixed diffusion timett\. Connecting these results to generated\-sample quality and training\-data replication requires studying the complete reverse process\. Explicit output\-rank constraints thus provide a tractable model of an architectural bottleneck, whose relationship to practical encoder–decoder networks remains to be explored\.

Future work includes extending the analysis to learned features and more general data geometries, and developing frameworks to study low\-rank effects in practical networks\.

#### Acknowledgements

The work of A\. J\. G\. has been supported by Swiss National Science Foundation grant number 200021\-204119\.

### AI use statement

In this work, we have not used generative AI tools for problem formulation, mathematical analysis, and the rest of the required disclosure tasks are not applicable to this work\. We used generative AI tools for 1\) developing codes required for simulations and plotting, and 2\) assist in writing the paper\. We have reviewed all AI\-assisted work\. LLM\-generated code was verified and tested for correctness by an author\. Portions of text written with the help of AI tools were reviewed carefully by the authors\. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI\.

## References

- B\. Achilli, L\. Ambrogioni, C\. Lucibello, M\. Mézard, and E\. VenturaMemorization and Generalization in Generative Diffusion under the Manifold Hypothesis\.arXiv\.Note:arXiv:2502\.09578 \[cond\-mat\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2502.09578)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Achilliet al\.\(2024\)B\. Achilli, E\. Ventura, G\. Silvestri, B\. Pham, G\. Raya, D\. Krotov, C\. Lucibello, and L\. AmbrogioniLosing dimensions: Geometric memorization in generative diffusion\.arXiv\.Note:arXiv:2410\.08727Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Adlam and Pennington \(2020\)B\. Adlam and J\. PenningtonThe Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi\-Scale Theory of Generalization\.InProceedings of the 37th International Conference on Machine Learning,pp\. 74–84\(en\)\.External Links:ISSN 2640\-3498Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p10.2.3)\.
- Ambrogioni \(2024\)L\. AmbrogioniIn Search of Dispersed Memories: Generative Diffusion Models Are Associative Memory Networks\.Entropy26\(5\),pp\. 381\(en\)\.External Links:ISSN 1099\-4300,[Document](https://dx.doi.org/10.3390/e26050381)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Anderson \(1982\)B\. D\. O\. AndersonReverse\-time diffusion equation models\.Stochastic Processes and their Applications12\(3\),pp\. 313–326\.Cited by:[§3\.1](https://arxiv.org/html/2610.20850#S3.SS1.p1.4)\.
- Bentonet al\.\(2023\)J\. Benton, V\. D\. Bortoli, A\. Doucet, and G\. DeligiannidisNearly d\-Linear Convergence Bounds for Diffusion Models via Stochastic Localization\.InThe Twelfth International Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Biroliet al\.\(2024\)G\. Biroli, T\. Bonnaire, V\. de Bortoli, and M\. MézardDynamical regimes of diffusion models\.Nature Communications15\(1\),pp\. 9957\(en\)\.External Links:ISSN 2041\-1723,[Document](https://dx.doi.org/10.1038/s41467-024-54281-3)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Bodin and Macris \(2021\)A\. Bodin and N\. MacrisModel, sample, and epoch\-wise descents: exact solution of gradient flow in the random feature model\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 21605–21617\.Cited by:[§3\.3](https://arxiv.org/html/2610.20850#S3.SS3.p3.1)\.
- Bodin and Macris \(2022\)A\. Bodin and N\. MacrisGradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures\.arXiv\.Note:arXiv:2212\.06757 \[stat\]External Links:[Link](http://arxiv.org/abs/2212.06757),[Document](https://dx.doi.org/10.48550/arXiv.2212.06757)Cited by:[§3\.3](https://arxiv.org/html/2610.20850#S3.SS3.p3.1)\.
- Bodin \(2024\)A\. P\. M\. BodinRandom matrix methods for high\-dimensional machine learning models\.Ph\.D\. Thesis,EPFL, \(en\)\.External Links:[Document](https://dx.doi.org/10.5075/epfl-thesis-10524)Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p10.2.3)\.
- Bonnaireet al\.\(2025\)T\. Bonnaire, R\. Urfin, G\. Biroli, and M\. MézardWhy Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training\.arXiv\.Note:arXiv:2505\.17638 \[cs\]External Links:[Link](http://arxiv.org/abs/2505.17638),[Document](https://dx.doi.org/10.48550/arXiv.2505.17638)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px3.p1.1)\.
- Bortoli \(2022\)V\. D\. BortoliConvergence of denoising diffusion models under the manifold hypothesis\.Transactions on Machine Learning Research\(en\)\.External Links:ISSN 2835\-8856Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Buchananet al\.\(2025\)S\. Buchanan, D\. Pai, Y\. Ma, and V\. D\. BortoliOn the Edge of Memorization in Diffusion Models\.arXiv\.Note:arXiv:2508\.17689 \[cs\]External Links:[Link](http://arxiv.org/abs/2508.17689),[Document](https://dx.doi.org/10.48550/arXiv.2508.17689)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Carliniet al\.\(2023\)N\. Carlini, J\. Hayes, M\. Nasr, M\. Jagielski, V\. Sehwag, F\. Tramèr, B\. Balle, D\. Ippolito, and E\. WallaceExtracting training data from diffusion models\.InProceedings of the 32nd USENIX Conference on Security Symposium,SEC ’23,USA,pp\. 5253–5270\.External Links:ISBN 978\-1\-939133\-37\-3Cited by:[§1](https://arxiv.org/html/2610.20850#S1.p1.1),[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023a\)H\. Chen, H\. Lee, and J\. LuImproved analysis of score\-based generative modeling: user\-friendly bounds under minimal smoothness assumptions\.InProceedings of the 40th International Conference on Machine Learning,ICML’23, Vol\.202,Honolulu, Hawaii, USA,pp\. 4735–4763\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2023b\)M\. Chen, K\. Huang, T\. Zhao, and M\. WangScore Approximation, Estimation and Distribution Recovery of Diffusion Models on Low\-Dimensional Data\.InProceedings of the 40th International Conference on Machine Learning,pp\. 4672–4712\(en\)\.External Links:ISSN 2640\-3498Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2022\)S\. Chen, S\. Chewi, J\. Li, Y\. Li, A\. Salim, and A\. ZhangSampling is as easy as learning the score: theory for diffusion models with minimal data assumptions\.InThe Eleventh International Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Cuiet al\.\(2023\)H\. Cui, F\. Krzakala, E\. Vanden\-Eijnden, and L\. ZdeborovaAnalysis of Learning a Flow\-based Generative Model from Limited Sample Complexity\.InThe Twelfth International Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Cuiet al\.\(2025\)H\. Cui, C\. Pehlevan, and Y\. M\. LuA precise asymptotic analysis of learning diffusion models: theory and insights\.arXiv\.Note:arXiv:2501\.03937 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.03937)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Faret al\.\(2006\)R\. R\. Far, T\. Oraby, W\. Bryc, and R\. SpeicherSpectra of large block matrices\.arXiv\.Note:arXiv:cs/0610045Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p10.2.3)\.
- Gardiner \(2009\)C\. W\. GardinerStochastic methods: a handbook for the natural and social sciences\.4th ed edition,Springer series in synergetics,Springer,Berlin Heidelberg\(eng\)\.External Links:ISBN 978\-3\-642\-08962\-6Cited by:[§3\.1](https://arxiv.org/html/2610.20850#S3.SS1.p1.1)\.
- George and Macris \(2026\)A\. J\. George and N\. MacrisAsymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data\.arXiv\.Note:arXiv:2603\.22962 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2603.22962),[Document](https://dx.doi.org/10.48550/arXiv.2603.22962)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Georgeet al\.\(2025\)A\. J\. George, R\. Veiga, and N\. MacrisAnalysis of Diffusion Models for Manifold Data\.In2025 IEEE International Symposium on Information Theory \(ISIT\),pp\. 1–6\.Note:ISSN: 2157\-8117External Links:ISSN 2157\-8117,[Document](https://dx.doi.org/10.1109/ISIT63088.2025.11195641)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Georgeet al\.\(2026\)A\. J\. George, R\. Veiga, and N\. MacrisDenoising score matching with random features: insights on diffusion models from precise learning curves\*\.Journal of Statistical Mechanics: Theory and Experiment2026\(8\),pp\. 084009\(en\)\.External Links:ISSN 1742\-5468,[Link](https://doi.org/10.1088/1742-5468/ae9379),[Document](https://dx.doi.org/10.1088/1742-5468/ae9379)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px3.p1.1)\.
- Geraceet al\.\(2020\)F\. Gerace, B\. Loureiro, F\. Krzakala, M\. Mezard, and L\. ZdeborovaGeneralisation error in learning with random features and the hidden manifold model\.InProceedings of the 37th International Conference on Machine Learning,pp\. 3452–3462\(en\)\.External Links:ISSN 2640\-3498Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p8.1.3)\.
- Goldtet al\.\(2022\)S\. Goldt, B\. Loureiro, G\. Reeves, F\. Krzakala, M\. Mezard, and L\. ZdeborovaThe Gaussian equivalence of generative models for learning with shallow neural networks\.InProceedings of the 2nd Mathematical and Scientific Machine Learning Conference,pp\. 426–471\(en\)\.External Links:ISSN 2640\-3498Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p8.1.3)\.
- Guet al\.\(2023\)X\. Gu, C\. Du, T\. Pang, C\. Li, M\. Lin, and Y\. WangOn Memorization in Diffusion Models\.arXiv\.Note:arXiv:2310\.02664 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.02664)Cited by:[§1](https://arxiv.org/html/2610.20850#S1.p1.1)\.
- Hanet al\.\(2023\)Y\. Han, M\. Razaviyayn, and R\. XuNeural Network\-Based Score Estimation in Diffusion Models: Optimization and Generalization\.InThe Twelfth International Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Haussmann and Pardoux \(1986\)U\. G\. Haussmann and E\. PardouxTime Reversal of Diffusions\.The Annals of Probability14\(4\),pp\. 1188–1205\.Cited by:[§3\.1](https://arxiv.org/html/2610.20850#S3.SS1.p1.4)\.
- Heltonet al\.\(2018\)J\. W\. Helton, T\. Mai, and R\. SpeicherApplications of realizations \(aka linearizations\) to free probability\.Journal of Functional Analysis274\(1\),pp\. 1–79\.Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p10.2.3)\.
- Hoet al\.\(2020\)J\. Ho, A\. Jain, and P\. AbbeelDenoising Diffusion Probabilistic Models\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 6840–6851\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2024\)H\. Hu, Y\. M\. Lu, and T\. MisiakiewiczAsymptotics of Random Feature Regression Beyond the Linear Scaling Regime\.arXiv\(en\)\.Note:arXiv:2403\.08160 \[stat\]External Links:[Link](http://arxiv.org/abs/2403.08160),[Document](https://dx.doi.org/10.48550/arXiv.2403.08160)Cited by:[§3\.3](https://arxiv.org/html/2610.20850#S3.SS3.p3.1)\.
- Hu and Lu \(2023\)H\. Hu and Y\. M\. LuUniversality Laws for High\-Dimensional Learning With Random Features\.IEEE Transactions on Information Theory69\(3\),pp\. 1932–1964\.Note:Conference Name: IEEE Transactions on Information TheoryCited by:[§5](https://arxiv.org/html/2610.20850#S5.p8.1.3)\.
- Kadkhodaieet al\.\(2023\)Z\. Kadkhodaie, F\. Guth, E\. P\. Simoncelli, and S\. MallatGeneralization in diffusion models arises from geometry\-adaptive harmonic representations\.InThe Twelfth International Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Kamb and Ganguli \(2024\)M\. Kamb and S\. GanguliAn analytic theory of creativity in convolutional diffusion models\.arXiv\.Note:arXiv:2412\.20292 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.20292)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Kingma and Ba \(2017\)D\. P\. Kingma and J\. BaAdam: A Method for Stochastic Optimization\.arXiv\.Note:arXiv:1412\.6980 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.1412.6980)Cited by:[Appendix C](https://arxiv.org/html/2610.20850#A3.SS0.SSS0.Px2.p1.1)\.
- Latourelle\-Vigeantet al\.\(2026\)H\. Latourelle\-Vigeant, S\. Chewi, A\. Pooladian, J\. Sous, and T\. MisiakiewiczGeneralization, memorization, and overfitting for diffusion models trained in the lazy high\-dimensional regime\.arXiv\.Note:arXiv:2608\.23938 \[stat\.ML\]External Links:[Link](http://arxiv.org/abs/2608.23938),[Document](https://dx.doi.org/10.48550/arXiv.2608.23938)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2023\)P\. Li, Z\. Li, H\. Zhang, and J\. BianOn the Generalization Properties of Diffusion Models\.Advances in Neural Information Processing Systems36,pp\. 2097–2127\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px3.p1.1)\.
- Mei and Montanari \(2022\)S\. Mei and A\. MontanariThe Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve\.Communications on Pure and Applied Mathematics75\(4\),pp\. 667–766\(en\)\.Cited by:[§3\.3](https://arxiv.org/html/2610.20850#S3.SS3.p3.1)\.
- Merger and Goldt \(2026\)C\. Merger and S\. GoldtGeneralization Dynamics of Linear Diffusion Models\.arXiv\.Note:arXiv:2505\.24769 \[stat\.ML\]External Links:[Link](http://arxiv.org/abs/2505.24769),[Document](https://dx.doi.org/10.48550/arXiv.2505.24769)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Montanari and Saeed \(2022\)A\. Montanari and B\. N\. SaeedUniversality of empirical risk minimization\.InProceedings of Thirty Fifth Conference on Learning Theory,pp\. 4310–4312\(en\)\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v178/montanari22a.html)Cited by:[§5](https://arxiv.org/html/2610.20850#S5.p8.1.3)\.
- Pidstrigach \(2022\)J\. PidstrigachScore\-Based Generative Models Detect Manifolds\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 35852–35865\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Rahimi and Recht \(2007\)A\. Rahimi and B\. RechtRandom Features for Large\-Scale Kernel Machines\.InAdvances in Neural Information Processing Systems,Vol\.20\.Cited by:[§3\.3](https://arxiv.org/html/2610.20850#S3.SS3.p1.1)\.
- Raya and Ambrogioni \(2023\)G\. Raya and L\. AmbrogioniSpontaneous symmetry breaking in generative diffusion models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 66377–66389\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Ronnebergeret al\.\(2015\)O\. Ronneberger, P\. Fischer, and T\. BroxU\-Net: Convolutional Networks for Biomedical Image Segmentation\.InMedical Image Computing and Computer\-Assisted Intervention – MICCAI 2015,N\. Navab, J\. Hornegger, W\. M\. Wells, and A\. F\. Frangi \(Eds\.\),Cham,pp\. 234–241\(en\)\.External Links:ISBN 978\-3\-319\-24574\-4,[Document](https://dx.doi.org/10.1007/978-3-319-24574-4%5F28)Cited by:[§1](https://arxiv.org/html/2610.20850#S1.p2.1)\.
- Rosset al\.\(2024\)B\. L\. Ross, H\. Kamkari, T\. Wu, R\. Hosseinzadeh, Z\. Liu, G\. Stein, J\. C\. Cresswell, and G\. Loaiza\-GanemA Geometric Framework for Understanding Memorization in Generative Models\.InThe Thirteenth International Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Saha and Tran \(2025\)E\. Saha and G\. TranGeneralization Bound for Diffusion Models using Random Features\.arXiv\.Note:arXiv:2310\.04417 \[stat\]External Links:[Link](http://arxiv.org/abs/2310.04417),[Document](https://dx.doi.org/10.48550/arXiv.2310.04417)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px3.p1.1)\.
- Shahet al\.\(2023\)K\. Shah, S\. Chen, and A\. KlivansLearning Mixtures of Gaussians Using the DDPM Objective\.Advances in Neural Information Processing Systems36,pp\. 19636–19649\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Sohl\-Dicksteinet al\.\(2015\)J\. Sohl\-Dickstein, E\. Weiss, N\. Maheswaranathan, and S\. GanguliDeep Unsupervised Learning using Nonequilibrium Thermodynamics\.InProceedings of the 32nd International Conference on Machine Learning,pp\. 2256–2265\(en\)\.External Links:ISSN 1938\-7228Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Somepalliet al\.\(2023\)G\. Somepalli, V\. Singla, M\. Goldblum, J\. Geiping, and T\. GoldsteinDiffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models\.In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6048–6058\.External Links:ISSN 2575\-7075Cited by:[§1](https://arxiv.org/html/2610.20850#S1.p1.1),[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Song and Ermon \(2019\)Y\. Song and S\. ErmonGenerative Modeling by Estimating Gradients of the Data Distribution\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Songet al\.\(2020\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-Based Generative Modeling through Stochastic Differential Equations\.InInternational Conference on Learning Representations,\(en\)\.Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px1.p1.1)\.
- Subramonian and Dohmatob \(2025\)A\. Subramonian and E\. DohmatobAuto\-fpt: Automating Free Probability Theory Calculations for Machine Learning Theory\.arXiv\.Note:arXiv:2504\.10754 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2504.10754),[Document](https://dx.doi.org/10.48550/arXiv.2504.10754)Cited by:[§A\.2\.4](https://arxiv.org/html/2610.20850#A1.SS2.SSS4.p1.1),[§5](https://arxiv.org/html/2610.20850#S5.p10.2.3)\.
- Urfinet al\.\(2026\)R\. Urfin, T\. Bonnaire, G\. Biroli, and M\. MézardFirst Learn, Then Memorize: The Spectral Bias of Diffusion Models\.arXiv\.Note:arXiv:2609\.35377 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2609.35377),[Document](https://dx.doi.org/10.48550/arXiv.2609.35377)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px3.p1.1)\.
- Vincent \(2011\)P\. VincentA Connection Between Score Matching and Denoising Autoencoders\.Neural Computation23\(7\),pp\. 1661–1674\(en\)\.External Links:ISSN 0899\-7667, 1530\-888X,[Document](https://dx.doi.org/10.1162/NECO%5Fa%5F00142)Cited by:[§3\.1](https://arxiv.org/html/2610.20850#S3.SS1.p2.1)\.
- Wang and Pehlevan \(2025\)B\. Wang and C\. PehlevanAn Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=SDhOClkyqC)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Yeet al\.\(2025\)Z\. Ye, Q\. Zhu, M\. Tao, and M\. ChenProvable Separations between Memorization and Generalization in Diffusion Models\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=42gfTZzyvV)Cited by:[§2](https://arxiv.org/html/2610.20850#S2.SS0.SSS0.Px2.p1.1)\.
- Yoonet al\.\(2023\)T\. Yoon, J\. Y\. Choi, S\. Kwon, and E\. K\. RyuDiffusion Probabilistic Models Generalize when They Fail to Memorize\.InICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling,\(en\)\.Cited by:[§1](https://arxiv.org/html/2610.20850#S1.p1.1)\.

Supplementary Materials

## Appendix AProofs

We first derive the finite\-dimensional rank\-constrained optimizer\. We then express its training and test errors as spectral integrals and compute the required limiting transforms using a linear pencil\. The moment calculations underlying this reduction are collected in Section[A\.2\.6](https://arxiv.org/html/2610.20850#A1.SS2.SSS6)\.

### A\.1Proof of Theorem[1](https://arxiv.org/html/2610.20850#Thmtheorem1)

Letϕ⁡\(x\)=ϱ⁡\(Wd​x\)\\phi\(x\)=\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}x\\right\)\. Considers⁡\(x\)=−x\+A​ϕ​\(x\)s\(x\)=\-x\+A\\phi\(x\)andx∼𝒩⁡\(0,𝒞\)x\\sim\\mathcal\{N\}\\left\(0,\\mathcal\{C\}\\right\)\. Letyi=a​xi\+h​zy\_\{i\}=ax\_\{i\}\+\\sqrt\{h\}z\. We have the following minimization problem

minrank​\(A\)≤r⁡ℒ⁡\(A\)=∑i=1n𝔼z​‖h​\(A​ϕ​\(yi\)−yi\)\+z‖2\.\\min\_\{\\text\{rank\}\(A\)\\leq r\}\\mathcal\{L\}\(A\)=\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\norm\{\\sqrt\{h\}\(A\\phi\(y\_i\)\-y\_i\)\+z\}^\{2\}\}\.Condition on the training samples andWW, and assumeU≻0U\\succ 0as in Theorem[1](https://arxiv.org/html/2610.20850#Thmtheorem1)\. Differentiating the quadratic objective gives the normal equationALS​U=\(Vϕ​y−1h​Vϕ​z\)TA\_\{\\text\{LS\}\}U=\\left\(V\_\{\\phi y\}\-\\frac\{1\}\{\\sqrt\{h\}\}V\_\{\\phi z\}\\right\)^\{T\}, so the unconstrained optimizer is

ALS=\(Vϕ​y−1h​Vϕ​z\)T⏟:=VTU−1,A\_\{\\text\{LS\}\}=\\underbrace\{\\left\(V\_\{\\phi y\}\-\\frac\{1\}\{\\sqrt\{h\}\}V\_\{\\phi z\}\\right\)^\{T\}\}\_\{:=V^\{T\}\}U^\{\-1\},\(13\)where

U\\displaystyle U=1n​∑i=1n𝔼z​\[ϕ⁡\(yi\)​ϕ​\(yi\)T\],\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\left\[\\phi\(y\_\{i\}\)\\phi\(y\_\{i\}\)^\{T\}\\right\]\},\(14\)Vϕ​z\\displaystyle V\_\{\\phi z\}=1n​∑i=1n𝔼z​\[ϕ⁡\(yi\)​zT\],\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\left\[\\phi\(y\_\{i\}\)z^\{T\}\\right\]\},Vϕ​y\\displaystyle V\_\{\\phi y\}=1n​∑i=1n𝔼z​\[ϕ⁡\(yi\)​yiT\]\.\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\left\[\\phi\(y\_\{i\}\)y\_\{i\}^\{T\}\\right\]\}\.To impose the rank constraint, expand the objective aroundALSA\_\{\\text\{LS\}\}\. The mixed term vanishes by the normal equation associated with \([13](https://arxiv.org/html/2610.20850#A1.E13)\):

ℒ⁡\(A\)\\displaystyle\\mathcal\{L\}\(A\)=∑i=1n𝔼z​‖h​\(A−ALS\)​ϕ​\(a​xi\+h​z\)\+h​\(ALS​ϕ​\(a​xi\+h​z\)−\(a​xi\+h​z\)\)\+z‖2,\\displaystyle=\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\norm\{\\sqrt\{h\}\(A\-A\_\{\\text\{LS\}\}\)\\phi\(ax\_i\+\\sqrt\{h\}z\)\+\\sqrt\{h\}\(A\_\{\\text\{LS\}\}\\phi\(ax\_i\+\\sqrt\{h\}z\)\-\(ax\_i\+\\sqrt\{h\}z\)\)\+z\}^\{2\}\},=∑i=1n𝔼z​‖h​\(A−ALS\)​ϕ​\(a​xi\+h​z\)‖2\+∑i=1n𝔼z​‖h​\(ALS​ϕ​\(a​xi\+h​z\)−\(a​xi\+h​z\)\)\+z‖2,\\displaystyle=\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\\norm\{\\sqrt\{h\}\(A\-A\_\{\\text\{LS\}\}\)\\phi\(ax\_i\+\\sqrt\{h\}z\)\}^\{2\}\+\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\\norm\{\\sqrt\{h\}\(A\_\{\\text\{LS\}\}\\phi\(ax\_i\+\\sqrt\{h\}z\)\-\(ax\_i\+\\sqrt\{h\}z\)\)\+z\}^\{2\},The residual term is independent ofAA\. Dropping this term and positive multiplicative constanthh, and using \([14](https://arxiv.org/html/2610.20850#A1.E14)\), reduces the optimization to a matrix approximation problem:

arg⁡minrank​\(A\)≤r​ℒ​\(A\)\\displaystyle\\arg\\min\_\{\\text\{rank\}\(A\)\\leq r\}\\mathcal\{L\}\(A\)=arg⁡min⁡∑i=1nrank​\(A\)≤r⁡𝔼z​‖\(A−ALS\)​ϕ​\(a​xi\+h​z\)‖2,\\displaystyle=\\arg\\min\_\{\\text\{rank\}\(A\)\\leq r\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\\norm\{\(A\-A\_\{\\text\{LS\}\}\)\\phi\(ax\_i\+\\sqrt\{h\}z\)\}^\{2\},=arg⁡minrank​\(A\)≤r⁡‖\(A−ALS\)​U1/2‖\.\\displaystyle=\\arg\\min\_\{\\text\{rank\}\(A\)\\leq r\}\\norm\{\(A\-A\_\{\\text\{LS\}\}\)U^\{1/2\}\}\.LetALS​U1/2=S​Σ​RTA\_\{\\text\{LS\}\}U^\{1/2\}=S\\Sigma R^\{T\}be the singular value decomposition\. The Eckart–Young theorem in the Frobenius norm therefore gives

A^r=SΣrRTU−1/2,\\hat\{A\}\_\{r\}=S\\Sigma\_\{r\}R^\{T\}U^\{\-1/2\},whereΣr\\Sigma\_\{r\}retains therrlargest singular values and sets the others to zero\.

### A\.2Proof of Theorem[2](https://arxiv.org/html/2610.20850#Thmtheorem2)

The proof proceeds by reducing the losses to weighted resolvent traces, obtaining their limiting values from \([34](https://arxiv.org/html/2610.20850#A1.E34)\), and recovering the rank\-truncated errors by Stieltjes inversion\. Barred moments \(U¯,V¯ϕ​y,V¯ϕ​z\\bar\{U\},\\bar\{V\}\_\{\\phi y\},\\bar\{V\}\_\{\\phi z\}\) average over an independent test sample and its noise; empirical moments condition on the clean training samples\.

#### A\.2\.1Test Error

We assume that𝔼⁡\[ϱ⁡\(g\)\]=0\\mathbb\{E\}\{\\left\[\\varrho\\left\(g\\right\)\\right\]\}=0\. The data covariance is diagonal with firstDDdiagonal entries equal to 1 and the remainingd−Dd\-Dentries equal to 0\. Letσ02=ψD\\sigma\_\{0\}^\{2\}=\\psi\_\{D\}, the normalized trace of the covariance matrix\. Accordingly, writeX=\[X1T,0\]TX=\[X\_\{1\}^\{T\},0\]^\{T\}, whereX1X\_\{1\}is aD×nD\\times nGaussian random matrix, and partitionW=\[W1,W2\]W=\[W\_\{1\},W\_\{2\}\]into its data\-subspace and normal\-space blocks\. Expanding the squared residual separates the test error into a quadratic term, a cross term, and an estimator\-independent term\. Withy=a​x\+h​zy=ax\+\\sqrt\{h\}z, we have

ℰtest​\(A^r\)\\displaystyle\\mathcal\{E\}\_\{\\text\{test\}\}\(\\hat\{A\}\_\{r\}\)=1d​𝔼x∼P0,z∼𝒩⁡\(0,1\)​‖h​\(A^r​ϕ​\(y\)−y\)\+z‖2,\\displaystyle=\\frac\{1\}\{d\}\\mathbb\{E\}\_\{x\\sim P\_\{0\},z\\sim\\mathcal\{N\}\\left\(0,1\\right\)\}\{\\norm\{\\sqrt\{h\}\(\\hat\{A\}\_r\\phi\(y\)\-y\)\+z\}^\{2\}\},\(15\)=hd​tr⁡\(A^rT​A^r​U¯\)\+2​hd​tr⁡\(A^r​\(V¯ϕ​zh−V¯ϕ​y\)\)\+hd​tr⁡\(a2​𝒞\+h​I\)\+\(1−2​h\),\\displaystyle=\\frac\{h\}\{d\}\\tr\{\\hat\{A\}\_r^T\\hat\{A\}\_r\\bar\{U\}\}\+\\frac\{2h\}\{d\}\\tr\{\\hat\{A\}\_r\\left\(\\frac\{\\bar\{V\}\_\{\\phi z\}\}\{\\sqrt\{h\}\}\-\\bar\{V\}\_\{\\phi y\}\\right\)\}\+\\frac\{h\}\{d\}\\tr\{a^2\\cC\+hI\}\+\\left\(1\-2h\\right\),whereU¯=𝔼x,z​\[ϕ⁡\(a​x\+h​z\)​ϕ​\(a​x\+h​z\)⊤\]\\bar\{U\}=\\mathbb\{E\}\_\{x,z\}\{\\left\[\\phi\(ax\+\\sqrt\{h\}z\)\\phi\(ax\+\\sqrt\{h\}z\)^\{\\top\}\\right\]\},V¯ϕ​y=𝔼x,z​\[ϕ⁡\(a​x\+h​z\)​\(a​x\+h​z\)⊤\]\\bar\{V\}\_\{\\phi y\}=\\mathbb\{E\}\_\{x,z\}\{\\left\[\\phi\(ax\+\\sqrt\{h\}z\)\(ax\+\\sqrt\{h\}z\)^\{\\top\}\\right\]\}, andV¯ϕ​z=𝔼x,z​\[ϕ⁡\(a​x\+h​z\)​z⊤\]\\bar\{V\}\_\{\\phi z\}=\\mathbb\{E\}\_\{x,z\}\{\\left\[\\phi\(ax\+\\sqrt\{h\}z\)z^\{\\top\}\\right\]\}\.

We need to computeVϕ​y,Vϕ​z,U,V¯ϕ​y,V¯ϕ​z,U¯V\_\{\\phi y\},V\_\{\\phi z\},U,\\bar\{V\}\_\{\\phi y\},\\bar\{V\}\_\{\\phi z\},\\bar\{U\}in order to get an expression forℰtest\\mathcal\{E\}\_\{\\text\{test\}\}\. Note that, in order to derive an asymptotically precise expression forℰtest\\mathcal\{E\}\_\{\\text\{test\}\}, it suffices to obtain these moment matrices up to leading order\. In particular, we can neglectO⁡\(1/d1−ϵ\)O\(1/d^\{1\-\\epsilon\}\)terms entry\-wise in these matrices, for0≤ϵ<1/20\\leq\\epsilon<1/2and for the diagonal termsϵ\\epsiloncan be in\[0,1\)\[0,1\)\. This can be justifies as follows\. By union bound, such an error matrixΔ\\Deltawill have Frobenius‖Δ‖F=O⁡\(dϵ\)\\norm\{\\Delta\}\_\{F\}=O\(d^\{\\epsilon\}\)with probability tending to11faster than any power\. In the expressions for test and train errors, these error matrices typically appears as1d​tr⁡\(Δ​A\)\\frac\{1\}\{d\}\\tr\{\\Delta A\}, whereAAis a matrix with operator normO⁡\(1\)\{O\\left\(1\\right\)\}w\.r\.t\.dd\. Using standard inequalities, we can upper bound1d​tr⁡\(Δ​A\)\\frac\{1\}\{d\}\\tr\{\\Delta A\}as

1d​tr⁡\(Δ​A\)\\displaystyle\\frac\{1\}\{d\}\\tr\{\\Delta A\}≤1d​‖Δ‖∗​‖A‖op\\displaystyle\\leq\\frac\{1\}\{d\}\\norm\{\\Delta\}\_\{\\ast\}\\norm\{A\}\_\{\\text\{op\}\}≤1d​rank​\(Δ\)​‖Δ‖F​‖A‖op\\displaystyle\\leq\\frac\{1\}\{d\}\\sqrt\{\\text\{rank\}\(\\Delta\)\}\\norm\{\\Delta\}\_\{F\}\\norm\{A\}\_\{\\text\{op\}\}=O⁡\(1d\)​‖Δ‖F=O⁡\(dϵ−1/2\)\.\\displaystyle=\{O\\left\(\\frac\{1\}\{\\sqrt\{d\}\}\\right\)\}\\norm\{\\Delta\}\_\{F\}=O\(d^\{\\epsilon\-1/2\}\)\.where the last equality holds with probability tending to11faster than any inverse power\. Thus, asymptotically these error terms vanish with high probability\. Section[A\.2\.6](https://arxiv.org/html/2610.20850#A1.SS2.SSS6)develops the moment calculations underlying the following asymptotic representations, which we use in the spectral reduction below:

V¯ϕ​y\\displaystyle\\bar\{V\}\_\{\\phi y\}=μ1​\(σ\)σ​W​𝒞~d,\\displaystyle=\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\\tilde\{\\mathcal\{C\}\}\}\{\\sqrt\{d\}\},\(16\)V¯ϕ​z\\displaystyle\\bar\{V\}\_\{\\phi z\}=h​μ1​\(σ\)σ​Wd,\\displaystyle=\\sqrt\{h\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\}\{\\sqrt\{d\}\},Vϕ​y\\displaystyle V\_\{\\phi y\}=a​Hn​XTn\+h​V¯ϕ​z,\\displaystyle=a\\frac\{H\}\{\\sqrt\{n\}\}\\frac\{X^\{T\}\}\{\\sqrt\{n\}\}\+\\sqrt\{h\}\\bar\{V\}\_\{\\phi z\},Vϕ​z\\displaystyle V\_\{\\phi z\}=h​μ1​\(σ\)σ​Wd=V¯ϕ​z,\\displaystyle=\\sqrt\{h\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\}\{\\sqrt\{d\}\}=\\bar\{V\}\_\{\\phi z\},U¯\\displaystyle\\bar\{U\}=μ12​\(σ\)σ2​Wd​𝒞~​WTd\+v2​\(σ\)​Ip,\\displaystyle=\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{W\}\{\\sqrt\{d\}\}\\tilde\{\\mathcal\{C\}\}\\frac\{W^\{T\}\}\{\\sqrt\{d\}\}\+v^\{2\}\(\\sigma\)I\_\{p\}\\;,U\\displaystyle U=Hn​HTn\+h​μ12​\(σ\)σ2​Wd​WTd\+s2​Ip,\\displaystyle=\\frac\{H\}\{\\sqrt\{n\}\}\\frac\{H^\{T\}\}\{\\sqrt\{n\}\}\+h\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{W\}\{\\sqrt\{d\}\}\\frac\{W^\{T\}\}\{\\sqrt\{d\}\}\+s^\{2\}I\_\{p\}\\;,where𝒞~=a2​𝒞\+h​I\\tilde\{\\mathcal\{C\}\}=a^\{2\}\\mathcal\{C\}\+hI,H=ϱ0​\(Wd​X\)H=\\varrho\_\{0\}\\left\(\\frac\{W\}\{\\sqrt\{d\}\}X\\right\), and

s2\\displaystyle s^\{2\}=‖ϱ\(σ⋅\)‖2−c⁡\(a2​σ02/σ2,σ\)−h​μ12​\(σ\)σ2,\\displaystyle=\\norm\{\\varrho\(\\sigma\\;\\cdot\)\}^\{2\}\-c\(a^\{2\}\\sigma\_\{0\}^\{2\}/\\sigma^\{2\},\\sigma\)\-h\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\;,\(17\)v2​\(σ\)\\displaystyle v^\{2\}\(\\sigma\)=‖ϱ\(σ⋅\)‖2−μ12​\(σ\)\.\\displaystyle=\\norm\{\\varrho\(\\sigma\\;\\cdot\)\}^\{2\}\-\\mu\_\{1\}^\{2\}\(\\sigma\)\.\(18\)Here, for standard Gaussian random variables\(u,v\)\(u,v\)with correlationγ\\gamma, we definec⁡\(γ,κ\)=𝔼u,v​\[ϱ⁡\(κ​u\)​ϱ​\(κ​v\)\]c\(\\gamma,\\kappa\)=\\mathbb\{E\}\_\{u,v\}\{\\left\[\\varrho\(\\kappa u\)\\varrho\(\\kappa v\)\\right\]\}\.

###### Assumption 2\(Gaussian equivalence\)\.

Setμ=μ1​\(σ\)/σ\\mu=\\mu\_\{1\}\(\\sigma\)/\\sigmaand define the Gaussian surrogate

HG=a​μ​W​Xd\+v0​Ω,H^\{\\mathrm\{G\}\}=a\\mu\\frac\{WX\}\{\\sqrt\{d\}\}\+v\_\{0\}\\Omega\\;,whereΩ∈ℝp×n\\Omega\\in\\mathbb\{R\}^\{p\\times n\}has i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\\left\(0,1\\right\)entries and is independent ofWWandXX, and

v02\\displaystyle v\_\{0\}^\{2\}=c⁡\(a2​σ02/σ2,σ\)−\(a​σ0​μ1​\(σ\)σ\)2\.\\displaystyle=c\(a^\{2\}\\sigma\_\{0\}^\{2\}/\\sigma^\{2\},\\sigma\)\-\\left\(a\\sigma\_\{0\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\right\)^\{2\}\\;\.\(19\)ForF∈\{H,HG\}F\\in\\\{H,H^\{\\mathrm\{G\}\}\\\}, define

UF\\displaystyle U\_\{F\}=F​FTn\+h​μ2​W​WTd\+s2​Ip,\\displaystyle=\\frac\{FF^\{T\}\}\{n\}\+h\\mu^\{2\}\\frac\{WW^\{T\}\}\{d\}\+s^\{2\}I\_\{p\},VF\\displaystyle V\_\{F\}=a​F​XTn−a2​μ​Wd,\\displaystyle=\\frac\{aFX^\{T\}\}\{n\}\-a^\{2\}\\mu\\frac\{W\}\{\\sqrt\{d\}\},QF​\(z\)\\displaystyle Q\_\{F\}\(z\)=\(VF​VFT−z​UF\)−1,\\displaystyle=\(V\_\{F\}V\_\{F\}^\{T\}\-zU\_\{F\}\)^\{\-1\},and the normalized traces

E0,F​\(z\)\\displaystyle E\_\{0,F\}\(z\)=1d​tr⁡\(VFT​QF​\(z\)​VF\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{V\_F^TQ\_F\(z\)V\_F\},E1,F​\(z\)\\displaystyle E\_\{1,F\}\(z\)=1d​tr⁡\(QF​\(z\)\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{Q\_F\(z\)\},E2,F​\(z\)\\displaystyle E\_\{2,F\}\(z\)=1d​tr⁡\(QF​\(z\)​W1​W1Td\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{Q\_F\(z\)\\frac\{W\_1W\_1^T\}\{d\}\},E3,F​\(z\)\\displaystyle E\_\{3,F\}\(z\)=1d​tr⁡\(QF​\(z\)​W2​W2Td\)\.\\displaystyle=\\frac\{1\}\{d\}\\tr\{Q\_F\(z\)\\frac\{W\_2W\_2^T\}\{d\}\}\.Assumes2\>0s^\{2\}\>0and that, in the proportional limit of Theorem[2](https://arxiv.org/html/2610.20850#Thmtheorem2), for every fixedz∈ℂz\\in\\mathbb\{C\}withImz\>0\\imaginary z\>0,

Ej,H​\(z\)−Ej,HG​\(z\)→ℙ0,j∈\{0,1,2,3\}\.E\_\{j,H\}\(z\)\-E\_\{j,H^\{\\mathrm\{G\}\}\}\(z\)\\xrightarrow\{\\mathbb\{P\}\}0,\\qquad j\\in\\\{0,1,2,3\\\}\.

Conditioning on this assumption, permits replacingHHbyHGH^\{\\mathrm\{G\}\}in these four limiting traces, with the sameWWandXXin both models\.

Let𝒞~=a2​𝒞\+h​I\\tilde\{\\mathcal\{C\}\}=a^\{2\}\\mathcal\{C\}\+hI\. Note that𝒞~\\tilde\{\\mathcal\{C\}\}is diagonal with firstDDdiagonal entries equal to 1 and the remainingd−Dd\-Dentries equal tohh\. To incorporate rank truncation into \([15](https://arxiv.org/html/2610.20850#A1.E15)\), introduce the two Gram matrices associated with the singular value decomposition ofVTU−1/2V^\{T\}U^\{\-1/2\}:

M\\displaystyle M=S​Σ​ΣT​ST=ALS​U​ALST=VT​U−1​V,\\displaystyle=S\\Sigma\\Sigma^\{T\}S^\{T\}=A\_\{\\text\{LS\}\}UA\_\{\\text\{LS\}\}^\{T\}=V^\{T\}U^\{\-1\}V,\(20\)M~\\displaystyle\\tilde\{M\}=RΣTΣRT=U1/2ALSTALSU1/2=U−1/2VVTU−1/2\.\\displaystyle=R\\Sigma^\{T\}\\Sigma R^\{T\}=U^\{1/2\}A\_\{\\text\{LS\}\}^\{T\}A\_\{\\text\{LS\}\}U^\{1/2\}=U^\{\-1/2\}VV^\{T\}U^\{\-1/2\}\.Recall thatψp=p/d\\psi\_\{p\}=p/d,ψn=n/d\\psi\_\{n\}=n/d,ψD=D/d\\psi\_\{D\}=D/d, andψr=r/d\\psi\_\{r\}=r/d\. The matrices in \([20](https://arxiv.org/html/2610.20850#A1.E20)\) have dimensionsd×dd\\times dandp×pp\\times p, respectively, and share the same nonzero eigenvalues\. LetΓψr\\Gamma\_\{\\psi\_\{r\}\}be the threshold such thatψr\\psi\_\{r\}fraction of eigenvalues ofMMare aboveΓψr\\Gamma\_\{\\psi\_\{r\}\}\. We can use Stone’s integration representation to get the rank\-truncated versions ofMMandM~\\tilde\{M\}\. LetSiS\_\{i\}be theiith column ofSS, andsis\_\{i\}be the corresponding singular value:

S​Σr​ΣrT​ST\\displaystyle S\\Sigma\_\{r\}\\Sigma\_\{r\}^\{T\}S^\{T\}=∑i≤rsi2​Si​SiT,\\displaystyle=\\sum\_\{i\\leq r\}s\_\{i\}^\{2\}S\_\{i\}S\_\{i\}^\{T\},\(21\)=1πlimK→∞limη↓0∫ΓψrKdxxIm\(M−\(x\+iη\)I\)−1\.\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\(M\-\(x\+i\\eta\)I\)^\{\-1\}\.Similarly,

RΣrTΣrRT=1πlimK→∞limη↓0∫ΓψrKdxxIm\(M~−\(x\+iη\)I\)−1\.\\displaystyle R\\Sigma\_\{r\}^\{T\}\\Sigma\_\{r\}R^\{T\}=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\(\\tilde\{M\}\-\(x\+i\\eta\)I\)^\{\-1\}\.\(22\)Using \([20](https://arxiv.org/html/2610.20850#A1.E20)\), the output\-space resolvent can be expressed through the feature\-space matrixV​VT−z​UVV^\{T\}\-zU:

\(M−z​I\)−1\\displaystyle\(M\-zI\)^\{\-1\}=\(VT​U−1​V−z​I\)−1,\\displaystyle=\(V^\{T\}U^\{\-1\}V\-zI\)^\{\-1\},\(23\)=1z​\(\(VT​U−1​V−z​I\)−1​VT​U−1​V−I\),\\displaystyle=\\frac\{1\}\{z\}\\left\(\(V^\{T\}U^\{\-1\}V\-zI\)^\{\-1\}V^\{T\}U^\{\-1\}V\-I\\right\),=1z​\(VT​\(V​VT−z​U\)−1​V−I\),\\displaystyle=\\frac\{1\}\{z\}\\left\(V^\{T\}\(VV^\{T\}\-zU\)^\{\-1\}V\-I\\right\),LetQ⁡\(z\):=\(V​VT−z​U\)−1Q\(z\):=\(VV^\{T\}\-zU\)^\{\-1\}\. Substituting \([16](https://arxiv.org/html/2610.20850#A1.E16)\) into the definition ofVVand usingh−1=−a2h\-1=\-a^\{2\}gives

V\\displaystyle V=Vϕ​y−1h​Vϕ​z\\displaystyle=V\_\{\\phi y\}\-\\frac\{1\}\{\\sqrt\{h\}\}V\_\{\\phi z\}=a​Hn​XTn\+\(h−1\)​μ1​\(σ\)σ​Wd,\\displaystyle=a\\frac\{H\}\{\\sqrt\{n\}\}\\frac\{X^\{T\}\}\{\\sqrt\{n\}\}\+\(h\-1\)\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\}\{\\sqrt\{d\}\},=a​Hn​XTn−a2​μ1​\(σ\)σ​Wd\.\\displaystyle=a\\frac\{H\}\{\\sqrt\{n\}\}\\frac\{X^\{T\}\}\{\\sqrt\{n\}\}\-a^\{2\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\}\{\\sqrt\{d\}\}\.BecauseXT​\(𝒞−I\)=0X^\{T\}\(\\mathcal\{C\}\-I\)=0, multiplication by𝒞−I\\mathcal\{C\}\-Iremoves the data\-dependent term:

V⁡\(𝒞−I\)\\displaystyle V\(\\mathcal\{C\}\-I\)=a2​μ1​\(σ\)σ​\[0,W2\]d\.\\displaystyle=a^\{2\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{\[0,W\_\{2\}\]\}\{\\sqrt\{d\}\}\.and

V¯ϕ​zh−V¯ϕ​y\\displaystyle\\frac\{\\bar\{V\}\_\{\\phi z\}\}\{\\sqrt\{h\}\}\-\\bar\{V\}\_\{\\phi y\}=μ1​\(σ\)σ​W⁡\(I−𝒞~\)d,\\displaystyle=\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\(I\-\\tilde\{\\mathcal\{C\}\}\)\}\{\\sqrt\{d\}\},\(24\)=\(1−h\)​μ1​\(σ\)σ​\[0,W2\]d,\\displaystyle=\(1\-h\)\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{\[0,W\_\{2\}\]\}\{\\sqrt\{d\}\},=a2​μ1​\(σ\)σ​\[0,W2\]d,\\displaystyle=a^\{2\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{\[0,W\_\{2\}\]\}\{\\sqrt\{d\}\},=V⁡\(𝒞−I\)\.\\displaystyle=V\(\\mathcal\{C\}\-I\)\.Next, for the quadratic term in \([15](https://arxiv.org/html/2610.20850#A1.E15)\), we substitute the optimizer and apply \([21](https://arxiv.org/html/2610.20850#A1.E21)\)\. MovingU−1/2U^\{\-1/2\}through the resolvent producesQQ; inserting \([16](https://arxiv.org/html/2610.20850#A1.E16)\) then separates the contributions of the two blocks ofWW:

1d​tr⁡\(A^rT​A^r​U¯\)\\displaystyle\\frac\{1\}\{d\}\\tr\{\\hat\{A\}\_r^T\\hat\{A\}\_r\\bar\{U\}\}=1d​tr\(U−1/2RΣrTΣrRTU−1/2U¯\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{U^\{\-1/2\}R\\Sigma\_r^T\\Sigma\_r R^TU^\{\-1/2\}\\bar\{U\}\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr\(U−1/2\(M~−\(x\+iη\)I\)−1U−1/2U¯\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{U^\{\-1/2\}\(\\tilde\{M\}\-\(x\+i\\eta\)I\)^\{\-1\}U^\{\-1/2\}\\bar\{U\}\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr\(U−1/2\(U−1/2VVTU−1/2−\(x\+iη\)I\)−1U−1/2U¯\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{U^\{\-1/2\}\(U^\{\-1/2\}VV^TU^\{\-1/2\}\-\(x\+i\\eta\)I\)^\{\-1\}U^\{\-1/2\}\\bar\{U\}\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(\(V​VT−\(x\+i​η\)​U\)−1​U¯\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\(VV^T\-\(x\+i\\eta\)U\)^\{\-1\}\\bar\{U\}\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(Q⁡\(x\+i​η\)​\(μ12​\(σ\)σ2​Wd​𝒞~​WTd\+v2​\(σ\)​Ip\)\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{Q\(x\+i\\eta\)\\left\(\\frac\{\\mu\_1^2\(\\sigma\)\}\{\\sigma^2\}\\rat\{W\}\{d\}\\tilde\{\\cC\}\\rat\{W^T\}\{d\} \+ v^2\(\\sigma\) I\_p\\right\)\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(Q⁡\(x\+i​η\)​\(μ12​\(σ\)σ2​\(W1d​W1Td\+h​W2d​W2Td\)\+v2​\(σ\)​Ip\)\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{Q\(x\+i\\eta\)\\left\(\\frac\{\\mu\_1^2\(\\sigma\)\}\{\\sigma^2\}\\left\(\\rat\{W\_1\}\{d\}\\rat\{W\_1^T\}\{d\}\+h\\rat\{W\_2\}\{d\}\\rat\{W\_2^T\}\{d\}\\right\) \+ v^2\(\\sigma\) I\_p\\right\)\},=1πlimK→∞limη↓0∫ΓψrKdxxIm\(μ12​\(σ\)σ2E2\(x\+iη\)\+h​μ12​\(σ\)σ2E3\(x\+iη\)\+v2\(σ\)E1\(x\+iη\)\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\left\(\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}E\_\{2\}\(x\+i\\eta\)\+\\frac\{h\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}E\_\{3\}\(x\+i\\eta\)\+\{v^\{2\}\(\\sigma\)\}E\_\{1\}\(x\+i\\eta\)\\right\),where

E1​\(z\)\\displaystyle E\_\{1\}\(z\)=1d​tr⁡\(Q⁡\(z\)\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{Q\(z\)\},\(25\)E2​\(z\)\\displaystyle E\_\{2\}\(z\)=1d​tr⁡\(W1Td​Q​\(z\)​W1d\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{\\rat\{W\_1^T\}\{d\}Q\(z\)\\rat\{W\_1\}\{d\}\},E3​\(z\)\\displaystyle E\_\{3\}\(z\)=1d​tr⁡\(W2Td​Q​\(z\)​W2d\)\.\\displaystyle=\\frac\{1\}\{d\}\\tr\{\\rat\{W\_2^T\}\{d\}Q\(z\)\\rat\{W\_2\}\{d\}\}\.For the cross term in \([15](https://arxiv.org/html/2610.20850#A1.E15)\), use \([24](https://arxiv.org/html/2610.20850#A1.E24)\), followed by \([22](https://arxiv.org/html/2610.20850#A1.E22)\) and \([23](https://arxiv.org/html/2610.20850#A1.E23)\)\.

1d​tr⁡\(A^r​\(V¯ϕ​zh−V¯ϕ​y\)\)\\displaystyle\\frac\{1\}\{d\}\\tr\{\\hat\{A\}\_r\\left\(\\frac\{\\bar\{V\}\_\{\\phi z\}\}\{\\sqrt\{h\}\}\-\\bar\{V\}\_\{\\phi y\}\\right\)\}=1d​tr\(SΣrRTU−1/2V\(𝒞−I\)\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r R^TU^\{\-1/2\}V\(\\cC\-I\)\},=1d​tr⁡\(S​Σr​RT​U1/2​ALST​\(𝒞−I\)\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r R^TU^\{1/2\}A\_\{\\text\{LS\}\}^T\(\\cC\-I\)\},=1d​tr⁡\(S​Σr​ΣT​ST​\(𝒞−I\)\)=1d​tr⁡\(S​Σr​ΣrT​ST​\(𝒞−I\)\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r \\Sigma^TS^T\(\\cC\-I\)\}=\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r \\Sigma\_r^TS^T\(\\cC\-I\)\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(\(M−\(x\+i​η\)​I\)−1​\(𝒞−I\)\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\(M\-\(x\+i\\eta\)I\)^\{\-1\}\(\\cC\-I\)\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(1\(x\+i​η\)​VT​\(V​VT−\(x\+i​η\)​U\)−1​V​\(𝒞−I\)\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\\frac\{1\}\{\(x\+i\\eta\)\}V^T\(VV^T\-\(x\+i\\eta\)U\)^\{\-1\}V\(\\cC\-I\)\},=−a4μ12​\(σ\)σ21πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(1\(x\+i​η\)​W2Td​Q​\(x\+i​η\)​W2d\),\\displaystyle=\-a^\{4\}\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\\frac\{1\}\{\(x\+i\\eta\)\}\\rat\{W\_2^T\}\{d\}Q\(x\+i\\eta\)\\rat\{W\_2\}\{d\}\},=−a4μ12​\(σ\)σ21πlimK→∞limη↓0∫ΓψrKdxxImE3​\(x\+i​η\)\(x\+i​η\)\.\\displaystyle=\-a^\{4\}\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{E\_\{3\}\(x\+i\\eta\)\}\{\(x\+i\\eta\)\}\.The test error therefore requires the limiting weighted traces in \([25](https://arxiv.org/html/2610.20850#A1.E25)\)\. DeterminingΓψr\\Gamma\_\{\\psi\_\{r\}\}also requires the spectrum ofMM\. Taking the normalized trace of \([23](https://arxiv.org/html/2610.20850#A1.E23)\) gives its Stieltjes transform:

sM​\(z\)=1d​tr⁡\(\(M−z​I\)−1\)=1z​d​tr⁡\(VT​Q​\(z\)​V\)−1z=1z​\(E0​\(z\)−1\),\\textbf\{s\}\_\{M\}\(z\)=\\frac\{1\}\{d\}\\tr\{\(M\-zI\)^\{\-1\}\}=\\frac\{1\}\{zd\}\\tr\{V^TQ\(z\)V\}\-\\frac\{1\}\{z\}=\\frac\{1\}\{z\}\(E\_\{0\}\(z\)\-1\),\(26\)where

E0​\(z\)=1d​tr⁡\(VT​Q​\(z\)​V\)\.E\_\{0\}\(z\)=\\frac\{1\}\{d\}\\tr\{V^TQ\(z\)V\}\.To summarize, we need the four tracesE0,E1,E2,E3E\_\{0\},E\_\{1\},E\_\{2\},E\_\{3\}to compute the test error\. After expressing training error also in terms of these traces, we later construct a linear pencil that encodes the four traces\.

#### A\.2\.2Train Error

The training error has the same quadratic structure as \([15](https://arxiv.org/html/2610.20850#A1.E15)\), with empirical moments replacing population moments\. These terms simplify because the same empirical moments defineA^r\\hat\{A\}\_\{r\}\.

ℰtrain​\(A^r\)\\displaystyle\\mathcal\{E\}\_\{\\text\{train\}\}\(\\hat\{A\}\_\{r\}\)=1d​n​∑i=1n𝔼z​‖h​\(A^r​ϕ​\(yi\)−yi\)\+z‖2,\\displaystyle=\\frac\{1\}\{dn\}\\sum\_\{i=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\norm\{\\sqrt\{h\}\(\\hat\{A\}\_r\\phi\(y\_i\)\-y\_i\)\+z\}^\{2\}\},\(27\)=hd​tr⁡\(A^rT​A^r​U\)\+2​hd​tr⁡\(A^r​\(Vϕ​zh−Vϕ​y\)\)\+h​σ2\+\(1−2​h\),\\displaystyle=\\frac\{h\}\{d\}\\tr\{\\hat\{A\}\_r^T\\hat\{A\}\_r U\}\+\\frac\{2h\}\{d\}\\tr\{\\hat\{A\}\_r\\left\(\\frac\{V\_\{\\phi z\}\}\{\\sqrt\{h\}\}\-V\_\{\\phi y\}\\right\)\}\+h\\sigma^\{2\}\+\\left\(1\-2h\\right\),
1d​tr⁡\(A^rT​A^r​U\)\\displaystyle\\frac\{1\}\{d\}\\tr\{\\hat\{A\}\_r^T\\hat\{A\}\_rU\}=1d​tr\(U−1/2RΣrTΣrRTU−1/2U\),\\displaystyle=\\frac\{1\}\{d\}\\tr\{U^\{\-1/2\}R\\Sigma\_r^T\\Sigma\_r R^TU^\{\-1/2\}U\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(\(M~−\(x\+i​η\)​I\)−1\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\(\\tilde\{M\}\-\(x\+i\\eta\)I\)^\{\-1\}\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr\(\(U−1/2VVTU−1/2−\(x\+iη\)I\)−1\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\(U^\{\-1/2\}VV^TU^\{\-1/2\}\-\(x\+i\\eta\)I\)^\{\-1\}\},=1πlimK→∞limη↓0∫ΓψrKdxxIm1d\(tr⁡\(\(M−\(x\+i​η\)​I\)−1\)−\(p−d\)1x\+i​η\),\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\left\(\\tr\{\(M\-\(x\+i\\eta\)I\)^\{\-1\}\}\-\(p\-d\)\\frac\{1\}\{x\+i\\eta\}\\right\),=1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(\(M−\(x\+i​η\)​I\)−1\)\.\\displaystyle=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\(M\-\(x\+i\\eta\)I\)^\{\-1\}\}\.The extra term involvingp−dp\-daccounts for the different numbers of zero eigenvalues ofMMandM~\\tilde\{M\}\. It contributes no mass above the positive cutoff\. For the cross term, useVT=ALS​UV^\{T\}=A\_\{\\mathrm\{LS\}\}Ufrom \([13](https://arxiv.org/html/2610.20850#A1.E13)\):

1d​tr⁡\(A^r​\(Vϕ​zh−Vϕ​y\)\)\\displaystyle\\frac\{1\}\{d\}\\tr\{\\hat\{A\}\_r\\left\(\\frac\{V\_\{\\phi z\}\}\{\\sqrt\{h\}\}\-V\_\{\\phi y\}\\right\)\}=−1d​tr\(SΣrRTU−1/2V\),\\displaystyle=\-\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r R^TU^\{\-1/2\}V\},=−1d​tr⁡\(S​Σr​RT​U1/2​ALST\),\\displaystyle=\-\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r R^TU^\{1/2\}A\_\{\\text\{LS\}\}^T\},=−1d​tr⁡\(S​Σr​ΣT​ST\)=−1d​tr⁡\(S​Σr​ΣrT​ST\),\\displaystyle=\-\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r \\Sigma^TS^T\}=\-\\frac\{1\}\{d\}\\tr\{S\\Sigma\_r \\Sigma\_r^TS^T\},=−1πlimK→∞limη↓0∫ΓψrKdxxIm1dtr⁡\(\(M−\(x\+i​η\)​I\)−1\)\.\\displaystyle=\-\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\frac\{1\}\{d\}\\tr\{\(M\-\(x\+i\\eta\)I\)^\{\-1\}\}\.
Combining the two contributions with their coefficients in \([27](https://arxiv.org/html/2610.20850#A1.E27)\) leaves a negative upper\-tail spectral integral\. Unlike the test error, the training error requires only the eigenvalue distribution ofMM\.

#### A\.2\.3Linear Pencil

The pencil replaces products of random matrices inQ⁡\(z\)Q\(z\)by a larger matrix with blocks linear inW1,W2,X1W\_\{1\},W\_\{2\},X\_\{1\}, andΩ\\Omega\. Selected inverse blocks recover the traces in \([25](https://arxiv.org/html/2610.20850#A1.E25)\) and \([26](https://arxiv.org/html/2610.20850#A1.E26)\)\. Setμ:=μ1​\(σ\)/σ\\mu:=\\mu\_\{1\}\(\\sigma\)/\\sigma\. With block sizes\(p,D,d−D,n,D,n,D,D,d−D\)\(p,D,d\-D,n,D,n,D,D,d\-D\), the pencil is

L⁡\(z\)=\(Ip−h​μ2​W1da4​μ2​W2z​d−v0​Ωn−a2​μ​W1z​da​v0​Ωz​n−a​μ​W1da2​μ​W1z​d−h​μ2​W2dW1Ts2​dID0000000W2Ts2​d0Id−D000000v0​ΩTs2​n−a​μ​X1Tn0In00000−a2​μ​W1Ts2​d00−a​X1nID00000000−X1TnIn000000−X1n00ID0000000−X1n0ID0W2Ts2​d0000000Id−D\)\.\\displaystyle L\(z\)=\\begin\{pmatrix\}I\_\{p\}&\-\\frac\{h\\mu^\{2\}W\_\{1\}\}\{\\sqrt\{d\}\}&\\frac\{a^\{4\}\\mu^\{2\}W\_\{2\}\}\{z\\sqrt\{d\}\}&\-\\frac\{v\_\{0\}\\Omega\}\{\\sqrt\{n\}\}&\-\\frac\{a^\{2\}\\mu W\_\{1\}\}\{z\\sqrt\{d\}\}&\\frac\{av\_\{0\}\\Omega\}\{z\\sqrt\{n\}\}&\-\\frac\{a\\mu W\_\{1\}\}\{\\sqrt\{d\}\}&\\frac\{a^\{2\}\\mu W\_\{1\}\}\{z\\sqrt\{d\}\}&\-\\frac\{h\\mu^\{2\}W\_\{2\}\}\{\\sqrt\{d\}\}\\\\ \\frac\{W\_\{1\}^\{T\}\}\{s^\{2\}\\sqrt\{d\}\}&I\_\{D\}&0&0&0&0&0&0&0\\\\ \\frac\{W\_\{2\}^\{T\}\}\{s^\{2\}\\sqrt\{d\}\}&0&I\_\{d\-D\}&0&0&0&0&0&0\\\\ \\frac\{v\_\{0\}\\Omega^\{T\}\}\{s^\{2\}\\sqrt\{n\}\}&\-\\frac\{a\\mu X\_\{1\}^\{T\}\}\{\\sqrt\{n\}\}&0&I\_\{n\}&0&0&0&0&0\\\\ \-\\frac\{a^\{2\}\\mu W\_\{1\}^\{T\}\}\{s^\{2\}\\sqrt\{d\}\}&0&0&\-\\frac\{aX\_\{1\}\}\{\\sqrt\{n\}\}&I\_\{D\}&0&0&0&0\\\\ 0&0&0&0&\-\\frac\{X\_\{1\}^\{T\}\}\{\\sqrt\{n\}\}&I\_\{n\}&0&0&0\\\\ 0&0&0&\-\\frac\{X\_\{1\}\}\{\\sqrt\{n\}\}&0&0&I\_\{D\}&0&0\\\\ 0&0&0&0&0&\-\\frac\{X\_\{1\}\}\{\\sqrt\{n\}\}&0&I\_\{D\}&0\\\\ \\frac\{W\_\{2\}^\{T\}\}\{s^\{2\}\\sqrt\{d\}\}&0&0&0&0&0&0&0&I\_\{d\-D\}\\end\{pmatrix\}\.

\(28\)Here, each zero has the corresponding block dimensions\. Eliminating the auxiliary blocks of \([28](https://arxiv.org/html/2610.20850#A1.E28)\) by block inversion recoversQ⁡\(z\)Q\(z\)and gives the following identities\. Block indices in this display start at one:

E0​\(z\)\\displaystyle E\_\{0\}\(z\)=\(1−1D​tr⁡\(L55−1\)\)​ψD\+\(1−1d−D​tr⁡\(L33−1\)\)​\(1−ψD\),\\displaystyle=\\left\(1\-\\frac\{1\}\{D\}\\tr\{L^\{\-1\}\_\{55\}\}\\right\)\\psi\_\{D\}\+\\left\(1\-\\frac\{1\}\{d\-D\}\\tr\{L^\{\-1\}\_\{33\}\}\\right\)\(1\-\\psi\_\{D\}\),E1​\(z\)\\displaystyle E\_\{1\}\(z\)=−1d​s2​z​tr⁡\(L11−1\),\\displaystyle=\-\\frac\{1\}\{ds^\{2\}z\}\\tr\{L^\{\-1\}\_\{11\}\},E2​\(z\)\\displaystyle E\_\{2\}\(z\)=−1d​a2​μ​tr⁡\(L28−1\),\\displaystyle=\-\\frac\{1\}\{da^\{2\}\\mu\}\\tr\{L^\{\-1\}\_\{28\}\},E3​\(z\)\\displaystyle E\_\{3\}\(z\)=1a4​μ2​\(1−1d−D​tr⁡\(L33−1\)\)​\(1−ψD\)\.\\displaystyle=\\frac\{1\}\{a^\{4\}\\mu^\{2\}\}\\left\(1\-\\frac\{1\}\{d\-D\}\\tr\{L^\{\-1\}\_\{33\}\}\\right\)\(1\-\\psi\_\{D\}\)\.Operator\-valued free probability theory then characterizes the limiting normalized block traces through a closed system of fixed\-point equations, from which the limits of\{Ei​\(z\)\}i=0,⋯,3\\\{E\_\{i\}\(z\)\\\}\_\{i=0,\\cdots,3\}are recovered\.

#### A\.2\.4Fixed\-Point Equations

We use theauto\-fptlibrary of[Subramonian and Dohmatob \(2025\)](https://arxiv.org/html/2610.20850#bib.bib5)to symbolically generate the fixed\-point equations from the linear pencil in \([28](https://arxiv.org/html/2610.20850#A1.E28)\)\. The library automates free\-probability calculations for normalized traces of rational expressions in large random matrices\. Applied to our pencil, it produces equations for the limiting block traces needed to recovere1,e2,e3e\_\{1\},e\_\{2\},e\_\{3\}and the Stieltjes transform ofMM\. We present the resulting system below after eliminating redundant variables\.

Letψp=p/d\\psi\_\{p\}=p/d,ψn=n/d\\psi\_\{n\}=n/d, andψD=D/d\\psi\_\{D\}=D/d\. We use zero\-based block indices and writeGi​j=Gi​j​\(z\)G\_\{ij\}=G\_\{ij\}\(z\)for the scalar unknowns which are the appropriately normalized traces of square matrix blocks\(L−1\)i​j\(L^\{\-1\}\)\_\{ij\}\. ThusG00G\_\{00\}corresponds to the first diagonal block in the preceding display\. The relations

G16=−za​G17,G46=−za​G47,G28=−h​za4​G82G\_\{16\}=\-\\frac\{z\}\{a\}G\_\{17\},\\qquad G\_\{46\}=\-\\frac\{z\}\{a\}G\_\{47\},\\qquad G\_\{28\}=\-\\frac\{hz\}\{a^\{4\}\}G\_\{82\}\(29\)reduce the system to eighteen equations\. Only the shared denominators are abbreviated:

𝒟0\\displaystyle\\mathcal\{D\}\_\{0\}=μ3​ψD​a2​h​z​G14\+μ2​ψD​\[a4​\(G44−G74\)\+a3​z​G64−h​z​G11\]\\displaystyle=\{\}\\mu^\{3\}\\psi\_\{D\}a^\{2\}hzG\_\{14\}\+\\mu^\{2\}\\psi\_\{D\}\\big\[a^\{4\}\(G\_\{44\}\-G\_\{74\}\)\+a^\{3\}zG\_\{64\}\-hzG\_\{11\}\\big\]\+μ2​\(1−ψD\)​\[a4​G22−h​z​\(2​G82\+G88\)\]\\displaystyle\\quad\+\\mu^\{2\}\(1\-\\psi\_\{D\}\)\\big\[a^\{4\}G\_\{22\}\-hz\(2G\_\{82\}\+G\_\{88\}\)\\big\]\+μ​ψD​\[a2​\(G71−G41\)−a​z​G61\]\+v02​\(a​G53−z​G33\)−s2​z,\\displaystyle\\quad\+\\mu\\psi\_\{D\}\\big\[a^\{2\}\(G\_\{71\}\-G\_\{41\}\)\-azG\_\{61\}\\big\]\+v\_\{0\}^\{2\}\(aG\_\{53\}\-zG\_\{33\}\)\-s^\{2\}z,\(30\)𝒟1\\displaystyle\\mathcal\{D\}\_\{1\}=μ2ψpG00\[a4\(\(G33−1\)\(G55−1\)−G35G53\)\\displaystyle=\{\}\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\\Big\[a^\{4\}\\big\(\(G\_\{33\}\-1\)\(G\_\{55\}\-1\)\-G\_\{35\}G\_\{53\}\\big\)\+a3\(zG35\+G53\)−a2zG33\+ahzG35−hz\]\\displaystyle\\qquad\+a^\{3\}\(zG\_\{35\}\+G\_\{53\}\)\-a^\{2\}zG\_\{33\}\+ahzG\_\{35\}\-hz\\Big\]\+s2​z​\(a​G35−1\),\\displaystyle\\quad\+s^\{2\}z\(aG\_\{35\}\-1\),\(31\)𝒟2\\displaystyle\\mathcal\{D\}\_\{2\}=μ​ψD2​a2​s2​z​\(G14​G47−G17​G44\)−μ​ψD​ψn​a2​s2​z​G14\\displaystyle=\{\}\\mu\\psi\_\{D\}^\{2\}a^\{2\}s^\{2\}z\(G\_\{14\}G\_\{47\}\-G\_\{17\}G\_\{44\}\)\-\\mu\\psi\_\{D\}\\psi\_\{n\}a^\{2\}s^\{2\}zG\_\{14\}\+μ​ψD​ψn​s2​z2​G17−ψD​ψn​s2​z​G47−ψD​ψp​a2​v02​G00​G44\\displaystyle\\quad\+\\mu\\psi\_\{D\}\\psi\_\{n\}s^\{2\}z^\{2\}G\_\{17\}\-\\psi\_\{D\}\\psi\_\{n\}s^\{2\}zG\_\{47\}\-\\psi\_\{D\}\\psi\_\{p\}a^\{2\}v\_\{0\}^\{2\}G\_\{00\}G\_\{44\}\+ψn2​s2​z\+ψn​ψp​v02​z​G00,\\displaystyle\\quad\+\\psi\_\{n\}^\{2\}s^\{2\}z\+\\psi\_\{n\}\\psi\_\{p\}v\_\{0\}^\{2\}zG\_\{00\},\(32\)𝒟3\\displaystyle\\mathcal\{D\}\_\{3\}=s2​z\+μ2​ψp​\(h​z−a4\)​G00\.\\displaystyle=\{\}s^\{2\}z\+\\mu^\{2\}\\psi\_\{p\}\(hz\-a^\{4\}\)G\_\{00\}\.\(33\)The fixed\-point equations are as follows:

G00\\displaystyle G\_\{00\}=−s2​z𝒟0,\\displaystyle=\-\\frac\{s^\{2\}z\}\{\\mathcal\{D\}\_\{0\}\},\(34a\)G11\\displaystyle G\_\{11\}=μ2​ψp​G00​\[a4​\(1−G55\)\+a3​z​G35\]\+s2​z​\(a​G35−1\)𝒟1,\\displaystyle=\\frac\{\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\\big\[a^\{4\}\(1\-G\_\{55\}\)\+a^\{3\}zG\_\{35\}\\big\]\+s^\{2\}z\(aG\_\{35\}\-1\)\}\{\\mathcal\{D\}\_\{1\}\},\(34b\)G14\\displaystyle G\_\{14\}=μ​ψp​a​G00​\[a⁡\(1−G55\)\+z​G35\]𝒟1,\\displaystyle=\\frac\{\\mu\\psi\_\{p\}aG\_\{00\}\\big\[a\(1\-G\_\{55\}\)\+zG\_\{35\}\\big\]\}\{\\mathcal\{D\}\_\{1\}\},\(34c\)G17\\displaystyle G\_\{17\}=μ​ψp​a2​G00​\(a​G35−1\)𝒟1,\\displaystyle=\\frac\{\\mu\\psi\_\{p\}a^\{2\}G\_\{00\}\(aG\_\{35\}\-1\)\}\{\\mathcal\{D\}\_\{1\}\},\(34d\)G41\\displaystyle G\_\{41\}=−μ​a2​\[μ2​ψp​G00​\(−a3​G53\+a2​z​G33\+h​z\)\+s2​z​G33\]𝒟1,\\displaystyle=\-\\frac\{\\mu a^\{2\}\\big\[\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\(\-a^\{3\}G\_\{53\}\+a^\{2\}zG\_\{33\}\+hz\)\+s^\{2\}zG\_\{33\}\\big\]\}\{\\mathcal\{D\}\_\{1\}\},\(34e\)G44\\displaystyle G\_\{44\}=−μ2​ψp​G00​\(−a3​G53\+a2​z​G33\+h​z\)\+s2​z𝒟1,\\displaystyle=\-\\frac\{\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\(\-a^\{3\}G\_\{53\}\+a^\{2\}zG\_\{33\}\+hz\)\+s^\{2\}z\}\{\\mathcal\{D\}\_\{1\}\},\(34f\)G47\\displaystyle G\_\{47\}=−μ2​ψp​a4​G00​\(G33−1\)𝒟1,\\displaystyle=\-\\frac\{\\mu^\{2\}\\psi\_\{p\}a^\{4\}G\_\{00\}\(G\_\{33\}\-1\)\}\{\\mathcal\{D\}\_\{1\}\},\(34g\)G61\\displaystyle G\_\{61\}=−μ​a𝒟1\[μ2ψpG00\(a4\(G33G55−G33−G35G53\)\+ahzG35\)\\displaystyle=\-\\frac\{\\mu a\}\{\\mathcal\{D\}\_\{1\}\}\\Big\[\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\\big\(a^\{4\}\(G\_\{33\}G\_\{55\}\-G\_\{33\}\-G\_\{35\}G\_\{53\}\)\+ahzG\_\{35\}\\big\)\+s2zG33\],\\displaystyle\\hskip 85\.35826pt\+s^\{2\}zG\_\{33\}\\Big\],\(34h\)G64\\displaystyle G\_\{64\}=−1𝒟1\[μ2ψpG00\(a3\(G33G55−G33−G35G53\)\+hzG35\)\\displaystyle=\-\\frac\{1\}\{\\mathcal\{D\}\_\{1\}\}\\Big\[\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\\big\(a^\{3\}\(G\_\{33\}G\_\{55\}\-G\_\{33\}\-G\_\{35\}G\_\{53\}\)\+hzG\_\{35\}\\big\)\+s2zG35\],\\displaystyle\\hskip 85\.35826pt\+s^\{2\}zG\_\{35\}\\Big\],\(34i\)G71\\displaystyle G\_\{71\}=−μ​a𝒟1\[μ2ψpG00\(−a4G53\+a3z\(G33G55−G35G53\)\+ahzG55\)\\displaystyle=\-\\frac\{\\mu a\}\{\\mathcal\{D\}\_\{1\}\}\\Big\[\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\\big\(\-a^\{4\}G\_\{53\}\+a^\{3\}z\(G\_\{33\}G\_\{55\}\-G\_\{35\}G\_\{53\}\)\+ahzG\_\{55\}\\big\)\+s2z\(a\(G33G55−G35G53\)\+G53\)\],\\displaystyle\\hskip 85\.35826pt\+s^\{2\}z\\big\(a\(G\_\{33\}G\_\{55\}\-G\_\{35\}G\_\{53\}\)\+G\_\{53\}\\big\)\\Big\],\(34j\)G74\\displaystyle G\_\{74\}=−1𝒟1\[μ2ψpG00\(−a3G53\+a2z\(G33G55−G35G53\)\+hzG55\)\\displaystyle=\-\\frac\{1\}\{\\mathcal\{D\}\_\{1\}\}\\Big\[\\mu^\{2\}\\psi\_\{p\}G\_\{00\}\\big\(\-a^\{3\}G\_\{53\}\+a^\{2\}z\(G\_\{33\}G\_\{55\}\-G\_\{35\}G\_\{53\}\)\+hzG\_\{55\}\\big\)\+s2zG55\],\\displaystyle\\hskip 85\.35826pt\+s^\{2\}zG\_\{55\}\\Big\],\(34k\)G33\\displaystyle G\_\{33\}=ψn​s2​z​\(ψn−ψD​G47\)𝒟2,\\displaystyle=\\frac\{\\psi\_\{n\}s^\{2\}z\(\\psi\_\{n\}\-\\psi\_\{D\}G\_\{47\}\)\}\{\\mathcal\{D\}\_\{2\}\},\(34l\)G35\\displaystyle G\_\{35\}=ψn​a​\(μ​ψD​s2​z​G17\+ψp​v02​G00\)𝒟2,\\displaystyle=\\frac\{\\psi\_\{n\}a\\big\(\\mu\\psi\_\{D\}s^\{2\}zG\_\{17\}\+\\psi\_\{p\}v\_\{0\}^\{2\}G\_\{00\}\\big\)\}\{\\mathcal\{D\}\_\{2\}\},\(34m\)G53\\displaystyle G\_\{53\}=ψD​ψn​s2​z​\(a​G44−za​G47\)𝒟2,\\displaystyle=\\frac\{\\psi\_\{D\}\\psi\_\{n\}s^\{2\}z\\left\(aG\_\{44\}\-\\frac\{z\}\{a\}G\_\{47\}\\right\)\}\{\\mathcal\{D\}\_\{2\}\},\(34n\)G55\\displaystyle G\_\{55\}=ψn​z​\[ψn​s2\+ψp​v02​G00−μ​ψD​a2​s2​G14\+μ​ψD​s2​z​G17\]𝒟2,\\displaystyle=\\frac\{\\psi\_\{n\}z\\big\[\\psi\_\{n\}s^\{2\}\+\\psi\_\{p\}v\_\{0\}^\{2\}G\_\{00\}\-\\mu\\psi\_\{D\}a^\{2\}s^\{2\}G\_\{14\}\+\\mu\\psi\_\{D\}s^\{2\}zG\_\{17\}\\big\]\}\{\\mathcal\{D\}\_\{2\}\},\(34o\)G22\\displaystyle G\_\{22\}=s2​z\+μ2​ψp​h​z​G00𝒟3,\\displaystyle=\\frac\{s^\{2\}z\+\\mu^\{2\}\\psi\_\{p\}hzG\_\{00\}\}\{\\mathcal\{D\}\_\{3\}\},\(34p\)G82\\displaystyle G\_\{82\}=μ2​ψp​a4​G00𝒟3,\\displaystyle=\\frac\{\\mu^\{2\}\\psi\_\{p\}a^\{4\}G\_\{00\}\}\{\\mathcal\{D\}\_\{3\}\},\(34q\)G88\\displaystyle G\_\{88\}=s2​z−μ2​ψp​a4​G00𝒟3\.\\displaystyle=\\frac\{s^\{2\}z\-\\mu^\{2\}\\psi\_\{p\}a^\{4\}G\_\{00\}\}\{\\mathcal\{D\}\_\{3\}\}\.\(34r\)

#### A\.2\.5Recovering the Limiting Training and Test Errors

We now combine \([34](https://arxiv.org/html/2610.20850#A1.E34)\) with the spectral representations of test and train errors\.

Step 1: ComputeΓψr\\Gamma\_\{\\psi\_\{r\}\} To computeΓψr\\Gamma\_\{\\psi\_\{r\}\}, we need the limiting spectrum ofMM, for which we use the Stieltjes transform inversion formula\. Limiting Stieltjes transform ofMMis given by

sν​\(z\)=limd→∞sM​\(z\)=1z​\(\(1−G44​\(z\)\)​ψD\+\(1−G22​\(z\)\)​\(1−ψD\)−1\)\.\\textbf\{s\}\_\{\\nu\}\(z\)=\\lim\_\{d\\to\\infty\}\\textbf\{s\}\_\{M\}\(z\)=\\frac\{1\}\{z\}\\left\(\\left\(1\-G\_\{44\}\(z\)\\right\)\\psi\_\{D\}\+\\left\(1\-G\_\{22\}\(z\)\\right\)\(1\-\\psi\_\{D\}\)\-1\\right\)\.Stieltjes inversion gives the upper\-tail massFM​\(x\)F\_\{M\}\(x\), rather than the cumulative mass belowxx:

FM\(x\)=1πlimK→∞limη↓0Im∫xKsM\(u\+iη\)du\.F\_\{M\}\(x\)=\\frac\{1\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\imaginary\\int\_\{x\}^\{K\}\\textbf\{s\}\_\{M\}\(u\+i\\eta\)du\.Then,Γψr\\Gamma\_\{\\psi\_\{r\}\}is obtained by solving

FM​\(Γψr\)=ψr\.F\_\{M\}\(\\Gamma\_\{\\psi\_\{r\}\}\)=\\psi\_\{r\}\.\(35\)
Step 2: Use the limiting expressions for\{Ei\}i=1,2,3\\\{E\_\{i\}\\\}\_\{i=1,2,3\} Letei=limd→∞Eie\_\{i\}=\\lim\_\{d\\to\\infty\}E\_\{i\}fori=1,2,3i=1,2,3\. Translating the block\-inverse identities into the zero\-based indices of \([34](https://arxiv.org/html/2610.20850#A1.E34)\) gives

e1​\(z\)\\displaystyle e\_\{1\}\(z\)=−ψps2​z​G00​\(z\),\\displaystyle=\-\\frac\{\\psi\_\{p\}\}\{s^\{2\}z\}G\_\{00\}\(z\),e2​\(z\)\\displaystyle e\_\{2\}\(z\)=−ψDa2​μ​G17,\\displaystyle=\-\\frac\{\\psi\_\{D\}\}\{a^\{2\}\\mu\}G\_\{17\},e3​\(z\)\\displaystyle e\_\{3\}\(z\)=1a4​μ2​\(1−G22\)​\(1−ψD\)\.\\displaystyle=\\frac\{1\}\{a^\{4\}\\mu^\{2\}\}\\left\(1\-G\_\{22\}\\right\)\(1\-\\psi\_\{D\}\)\.Substituting these limits into the quadratic and cross terms of \([15](https://arxiv.org/html/2610.20850#A1.E15)\), and using \([26](https://arxiv.org/html/2610.20850#A1.E26)\) for the training error, yields the following expressions with the cutoff from \([35](https://arxiv.org/html/2610.20850#A1.E35)\):

limd→∞ℰtest​\(A^r\)\\displaystyle\\lim\_\{d\\to\\infty\}\\mathcal\{E\}\_\{\\text\{test\}\}\(\\hat\{A\}\_\{r\}\)=hσ2\+\(1−2h\)\+hπlimK→∞limη↓0∫ΓψrKdxxImm\(x\+iη\),\\displaystyle=h\\sigma^\{2\}\+\\left\(1\-2h\\right\)\+\\frac\{h\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\textbf\{m\}\(x\+i\\eta\),limd→∞ℰtrain​\(A^r\)\\displaystyle\\lim\_\{d\\to\\infty\}\\mathcal\{E\}\_\{\\text\{train\}\}\(\\hat\{A\}\_\{r\}\)=hσ2\+\(1−2h\)−hπlimK→∞limη↓0∫ΓψrKdxxImsν\(x\+iη\),\\displaystyle=h\\sigma^\{2\}\+\\left\(1\-2h\\right\)\-\\frac\{h\}\{\\pi\}\\lim\_\{K\\to\\infty\}\\lim\_\{\\eta\\downarrow 0\}\\int\_\{\\Gamma\_\{\\psi\_\{r\}\}\}^\{K\}dx\\;x\\imaginary\\textbf\{s\}\_\{\\nu\}\(x\+i\\eta\),where

m​\(z\)=μ12​\(σ\)σ2​e2​\(z\)\+\(h​μ12​\(σ\)σ2−2​a4​μ12​\(σ\)σ2​1z\)​e3​\(z\)\+v2​\(σ\)​e1​\(z\)\.\\displaystyle\\textbf\{m\}\(z\)=\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}e\_\{2\}\(z\)\+\\left\(\\frac\{h\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\-2a^\{4\}\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{1\}\{z\}\\right\)e\_\{3\}\(z\)\+\{v^\{2\}\(\\sigma\)\}e\_\{1\}\(z\)\.The transformmcombines the quadratic contribution with twice the cross contribution, recovering the error characterization of Theorem[2](https://arxiv.org/html/2610.20850#Thmtheorem2)\.

#### A\.2\.6Derivation of the Relevant Moments

We collect the Gaussian moment calculations used in the spectral reduction\. The common tool is Mehler’s expansion in Hermite polynomials: for a jointly standard Gaussian pair with lawPγP^\{\\gamma\}, correlations of nonlinear functions are expanded in powers ofγ\\gamma\. In cross moments with a linear second factor, Hermite orthogonality selects the first coefficient\. Feature second moments require separate diagonal and off\-diagonal calculations\. \(i\) We will first considerV¯ϕ​y\\bar\{V\}\_\{\\phi y\}:We have

V¯ϕ​y=𝔼x,z​\[ϱ⁡\(Wd​\(a​x\+h​z\)\)​\(a​x\+h​z\)T\]\.\\bar\{V\}\_\{\\phi y\}=\\mathbb\{E\}\_\{x,z\}\{\\left\[\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}\(ax\+\\sqrt\{h\}z\)\\right\)\(ax\+\\sqrt\{h\}z\)^\{T\}\\right\]\}\\;\.Letwiw\_\{i\}denote theithi^\{\\text\{th\}\}row ofWWand letx⁡\(j\)x\(j\)denote thejt​hj^\{th\}component ofxx\. For largedd,‖wi‖2d\\frac\{\\norm\{w\_i\}^\{2\}\}\{d\}concentrates to11\. Note that, conditioned onWW,wiT​\(a​x\+h​z\)d\\frac\{w\_\{i\}^\{T\}\(ax\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}is a zero\-mean Gaussian random variable with variance𝔼x,z​\[\(wiT​\(a​x\+h​z\)\)2d\]=1d​wiT​\(a2​𝒞\+h​I\)​wi\\mathbb\{E\}\_\{x,z\}\{\\left\[\\frac\{\(w\_\{i\}^\{T\}\(ax\+\\sqrt\{h\}z\)\)^\{2\}\}\{d\}\\right\]\}=\\frac\{1\}\{d\}w\_\{i\}^\{T\}\(a^\{2\}\\mathcal\{C\}\+hI\)w\_\{i\}\. By the concentration property of quadratic forms, the variance concentrates toσ2=1d​tr⁡\(a2​𝒞\+h​I\)\\sigma^\{2\}=\\frac\{1\}\{d\}\\tr\{a^2\\cC\+hI\}asd→∞d\\to\\infty\. Let𝒞~=a2​𝒞\+h​I\\tilde\{\\mathcal\{C\}\}=a^\{2\}\\mathcal\{C\}\+hI\. Letγi​j=1σ​𝒞~j​j​wiT​𝒞~⋅jd\\gamma\_\{ij\}=\\frac\{1\}\{\\sigma\\sqrt\{\\tilde\{\\mathcal\{C\}\}\_\{jj\}\}\}\\frac\{w\_\{i\}^\{T\}\\tilde\{\\mathcal\{C\}\}\_\{\\cdot j\}\}\{\\sqrt\{d\}\}

\[V¯ϕ​y\]i​j\\displaystyle\[\\bar\{V\}\_\{\\phi y\}\]\_\{ij\}=𝔼x,z​\[ϱ⁡\(wiT​\(a​x\+h​z\)d\)​\(a​xj\+h​zj\)\],\\displaystyle=\\mathbb\{E\}\_\{x,z\}\{\\left\[\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)\(ax\_\{j\}\+\\sqrt\{h\}z\_\{j\}\)\\right\]\},=𝒞~j​j​𝔼\(u,v\)∼Pγi​j​\[ϱ⁡\(σ​u\)​v\],\\displaystyle=\\sqrt\{\\tilde\{\\mathcal\{C\}\}\_\{jj\}\}\\mathbb\{E\}\_\{\(u,v\)\\sim P^\{\\gamma\_\{ij\}\}\}\{\\left\[\\varrho\\left\(\\sigma u\\right\)v\\right\]\},=𝒞~j​j​∑k=0∞\(γi​j\)kk\!​𝔼u​\[ϱ⁡\(σ​u\)​Hek​\(u\)\]​𝔼v​\[v​Hek​\(v\)\],\\displaystyle=\\sqrt\{\\tilde\{\\mathcal\{C\}\}\_\{jj\}\}\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(\\gamma\_\{ij\}\)^\{k\}\}\{k\!\}\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\\left\(\\sigma u\\right\)\\text\{He\}\_\{k\}\(u\)\\right\]\}\\mathbb\{E\}\_\{v\}\{\\left\[v\\text\{He\}\_\{k\}\(v\)\\right\]\},=𝒞~j​j​γi​j​𝔼u​\[ϱ⁡\(σ​u\)​u\],\\displaystyle=\\sqrt\{\\tilde\{\\mathcal\{C\}\}\_\{jj\}\}\\gamma\_\{ij\}\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\\left\(\\sigma u\\right\)u\\right\]\},=μ1​\(σ\)σ​\[W​𝒞~\]i​jd\.\\displaystyle=\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{\[W\\tilde\{\\mathcal\{C\}\}\]\_\{ij\}\}\{\\sqrt\{d\}\}\\;\.We haveV¯ϕ​y=μ1​\(σ\)σ​W​𝒞~d\\bar\{V\}\_\{\\phi y\}=\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\\tilde\{\\mathcal\{C\}\}\}\{\\sqrt\{d\}\}\.

\(ii\) Next, considerV¯ϕ​z\\bar\{V\}\_\{\\phi z\}:

\[V¯ϕ​z\]i​j\\displaystyle\[\\bar\{V\}\_\{\\phi z\}\]\_\{ij\}=𝔼x,z​\[ϱ⁡\(wiT​\(a​x\+h​z\)d\)​zj\],\\displaystyle=\\mathbb\{E\}\_\{x,z\}\{\\left\[\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)z\_\{j\}\\right\]\},=𝔼\(x,u,v\)∼Pwi​jd​\[ϱ⁡\(a​wiT​xd\+h​u\)​v\],\\displaystyle=\\mathbb\{E\}\_\{\(x,u,v\)\\sim P^\{\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\}\}\{\\left\[\\varrho\\left\(\\frac\{aw\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)v\\right\]\},=∑k=0∞\(wi​jd\)kk\!​𝔼x,u​\[ϱ⁡\(a​wiT​xd\+h​u\)​Hek​\(u\)\]​𝔼v​\[v​Hek​\(v\)\],\\displaystyle=\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\)^\{k\}\}\{k\!\}\\mathbb\{E\}\_\{x,u\}\{\\left\[\\varrho\\left\(\\frac\{aw\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)\\text\{He\}\_\{k\}\(u\)\\right\]\}\\mathbb\{E\}\_\{v\}\{\\left\[v\\text\{He\}\_\{k\}\(v\)\\right\]\},=wi​jd​𝔼x,u​\[ϱ⁡\(a​wiT​xd\+h​u\)​u\],\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\mathbb\{E\}\_\{x,u\}\{\\left\[\\varrho\\left\(\\frac\{aw\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)u\\right\]\},=wi​jd​𝔼g,u​\[ϱ⁡\(a​σ0​g\+h​u\)​u\],\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\mathbb\{E\}\_\{g,u\}\{\\left\[\\varrho\(a\\sigma\_\{0\}g\+\\sqrt\{h\}u\)u\\right\]\},=h​μ1​\(σ\)σ​wi​jd\.\\displaystyle=\\sqrt\{h\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\;\.We haveV¯ϕ​z=h​μ1​\(σ\)σ​Wd\\bar\{V\}\_\{\\phi z\}=\\sqrt\{h\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\}\{\\sqrt\{d\}\}\.

\(iii\) Now we will considerU¯\\bar\{U\}:When consideringU¯\\bar\{U\}, a similar argument allows to write\(u~0,u~1\)=\(1σ​wiT​\(a​x\+h​z\)d,1σ​wjT​\(a​x\+h​z\)d\)∼Pγ~i​j\(\\tilde\{u\}\_\{0\},\\tilde\{u\}\_\{1\}\)=\(\\frac\{1\}\{\\sigma\}\\frac\{w\_\{i\}^\{T\}\(ax\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\},\\frac\{1\}\{\\sigma\}\\frac\{w\_\{j\}^\{T\}\(ax\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\)\\sim P^\{\\tilde\{\\gamma\}\_\{ij\}\}withγ~i​j=1σ2​wiT​𝒞~​wjd\\tilde\{\\gamma\}\_\{ij\}=\\frac\{1\}\{\\sigma^\{2\}\}\\frac\{w\_\{i\}^\{T\}\\tilde\{\\mathcal\{C\}\}w\_\{j\}\}\{d\}, leading to

U¯i​j\\displaystyle\\bar\{U\}\_\{ij\}=𝔼x​\[ϱ⁡\(wiT​xd\)​ϱ​\(wjT​xd\)\]\\displaystyle=\\mathbb\{E\}\_\{x\}\{\\left\[\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\varrho\\left\(\\frac\{w\_\{j\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\right\]\}=𝔼\(u~0,u~1\)∼Pγ~i​j​\[ϱ⁡\(σ​u~0\)​ϱ​\(σ​u~1\)\]\\displaystyle=\\mathbb\{E\}\_\{\(\\tilde\{u\}\_\{0\},\\tilde\{u\}\_\{1\}\)\\sim P^\{\\tilde\{\\gamma\}\_\{ij\}\}\}\{\\left\[\\varrho\(\\sigma\\tilde\{u\}\_\{0\}\)\\varrho\(\\sigma\\tilde\{u\}\_\{1\}\)\\right\]\}=∑k=0∞γ~i​jkk\!​𝔼u~0​\[ϱ⁡\(σ​u~0\)​Hek​\(u~0\)\]​𝔼u~1​\[ϱ⁡\(σ​u~1\)​Hek​\(u~1\)\]\\displaystyle=\\sum\_\{k=0\}^\{\\infty\}\\frac\{\\tilde\{\\gamma\}\_\{ij\}^\{k\}\}\{k\!\}\\;\\mathbb\{E\}\_\{\\tilde\{u\}\_\{0\}\}\{\\left\[\\varrho\(\\sigma\\tilde\{u\}\_\{0\}\)\\text\{He\}\_\{k\}\(\\tilde\{u\}\_\{0\}\)\\right\]\}\\mathbb\{E\}\_\{\\tilde\{u\}\_\{1\}\}\{\\left\[\\varrho\(\\sigma\\tilde\{u\}\_\{1\}\)\\text\{He\}\_\{k\}\(\\tilde\{u\}\_\{1\}\)\\right\]\}=∑k=0∞γ~i​jkk\!​𝔼u~0​\[ϱ⁡\(σ​u~0\)​Hek​\(u~0\)\]2\.\\displaystyle=\\sum\_\{k=0\}^\{\\infty\}\\frac\{\\tilde\{\\gamma\}\_\{ij\}^\{k\}\}\{k\!\}\\;\\mathbb\{E\}\_\{\\tilde\{u\}\_\{0\}\}\{\\left\[\\varrho\(\\sigma\\tilde\{u\}\_\{0\}\)\\text\{He\}\_\{k\}\(\\tilde\{u\}\_\{0\}\)\\right\]\}^\{2\}\\;\.Separating diagonal and off\-diagonal entries gives:

U¯i​j=\{μ02​\(σ\)\+μ12​\(σ\)σ2​wiT​𝒞~​wjd\+O⁡\(1/d\)if​i≠j,‖ϱ\(σ⋅\)‖2if​i=j\.\\bar\{U\}\_\{ij\}=\\begin\{cases\}\\mu\_\{0\}^\{2\}\(\\sigma\)\+\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{w\_\{i\}^\{T\}\\tilde\{\\mathcal\{C\}\}w\_\{j\}\}\{d\}\+O\(1/d\)\\quad&\\text\{if \}i\\neq j\\;,\\\\ \\norm\{\\varrho\(\\sigma\\;\\cdot\)\}^\{2\}\\quad&\\text\{if \}i=j\\;\.\\end\{cases\}TheO⁡\(1/d\)\{O\\left\(1/d\\right\)\}contribution cannot give rise to aO⁡\(1\)\{O\\left\(1\\right\)\}change to the normalized trace quantities we are interested in\. Hence, we neglect it\. We have

U¯=μ02​\(σ\)​1p​1pT\+μ12​\(σ\)σ2​Wd​𝒞~​WTd\+v2​\(σ\)​Ip,\\bar\{U\}=\\mu\_\{0\}^\{2\}\(\\sigma\)\\textbf\{1\}\_\{p\}\\textbf\{1\}\_\{p\}^\{T\}\+\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{W\}\{\\sqrt\{d\}\}\\tilde\{\\mathcal\{C\}\}\\frac\{W^\{T\}\}\{\\sqrt\{d\}\}\+v^\{2\}\(\\sigma\)I\_\{p\}\\;,wherev2​\(κ\)=‖ϱ\(κ⋅\)‖2−μ02​\(κ\)−μ12​\(κ\)v^\{2\}\(\\kappa\)=\\norm\{\\varrho\(\\kappa\\;\\cdot\)\}^\{2\}\-\\mu\_\{0\}^\{2\}\(\\kappa\)\-\\mu\_\{1\}^\{2\}\(\\kappa\)\.

\(iv\) Now we will considerVϕ​zV\_\{\\phi z\}:

Let

Vϕ​zl=𝔼z​\[ϱ⁡\(Wd​\(a​xl\+h​z\)\)​zT\]\.V\_\{\\phi z\}^\{l\}=\\mathbb\{E\}\_\{z\}\{\\left\[\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\\right\)z^\{T\}\\right\]\}\\;\.Herexlx\_\{l\}is fixed and only the noise is averaged\. Applying Mehler’s formula as in the population cross\-moment calculation gives

\[Vϕ​zl\]i​j\\displaystyle\[V\_\{\\phi z\}^\{l\}\]\_\{ij\}=𝔼z​\[ϱ⁡\(wiT​\(a​xl\+h​z\)d\)​zj\],\\displaystyle=\\mathbb\{E\}\_\{z\}\{\\left\[\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)z\_\{j\}\\right\]\},=𝔼\(u,v\)∼Pwi​jd​\[ϱ⁡\(a​wiT​xld\+h​u\)​v\],\\displaystyle=\\mathbb\{E\}\_\{\(u,v\)\\sim P^\{\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\}\}\{\\left\[\\varrho\\left\(\\frac\{aw\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)v\\right\]\},=∑k=0∞\(wi​jd\)kk\!​𝔼u​\[ϱ⁡\(a​wiT​xld\+h​u\)​Hek​\(u\)\]​𝔼v​\[v​Hek​\(v\)\],\\displaystyle=\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\)^\{k\}\}\{k\!\}\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\\left\(\\frac\{aw\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)\\text\{He\}\_\{k\}\(u\)\\right\]\}\\mathbb\{E\}\_\{v\}\{\\left\[v\\text\{He\}\_\{k\}\(v\)\\right\]\},=wi​jd​𝔼u​\[ϱ⁡\(a​wiT​xld\+h​u\)​u\]\+O⁡\(1/d\),\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\\left\(\\frac\{aw\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)u\\right\]\}\+O\(1/d\),=wi​jd​ϱ1​\(wiT​xld\)\+O⁡\(1/d\),\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+O\(1/d\)\\;,whereϱ1​\(y\)=𝔼u​\[ϱ⁡\(a​y\+h​u\)​u\]\\varrho\_\{1\}\(y\)=\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\(ay\+\\sqrt\{h\}u\)u\\right\]\}\. Summing over thenndata samples:

\[Vϕ​z\]i​j\\displaystyle\[V\_\{\\phi z\}\]\_\{ij\}=1n​∑l=1nVi​jl,\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}V^\{l\}\_\{ij\},=wi​jd​1n​∑l=1nϱ1​\(wiT​xld\),\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\),=wi​jd​𝔼x​\[ϱ1​\(wiT​xd\)\]\+O⁡\(1/d\),\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\mathbb\{E\}\_\{x\}\{\\left\[\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\right\]\}\+O\(1/\\sqrt\{d\}\),=wi​jd​𝔼g​\[ϱ1​\(σ0​g\)\]\+O⁡\(1/d\),\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\mathbb\{E\}\_\{g\}\{\\left\[\\varrho\_\{1\}\(\\sigma\_\{0\}g\)\\right\]\}\+O\(1/\\sqrt\{d\}\),=wi​jd​𝔼g,u​\[ϱ⁡\(a​σ0​g\+h​u\)​u\]\+O⁡\(1/d\),\\displaystyle=\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\\mathbb\{E\}\_\{g,u\}\{\\left\[\\varrho\(a\\sigma\_\{0\}g\+\\sqrt\{h\}u\)u\\right\]\}\+O\(1/\\sqrt\{d\}\),=h​μ1​\(σ\)σ​wi​jd\+O⁡\(1/d\)\.\\displaystyle=\\sqrt\{h\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{w\_\{ij\}\}\{\\sqrt\{d\}\}\+O\(1/\\sqrt\{d\}\)\\;\.NeglectingO⁡\(1/d\)O\(1/\\sqrt\{d\}\)terms, we haveVϕ​z=h​μ1​\(σ\)σ​Wd=V¯ϕ​zV\_\{\\phi z\}=\\sqrt\{h\}\\frac\{\\mu\_\{1\}\(\\sigma\)\}\{\\sigma\}\\frac\{W\}\{\\sqrt\{d\}\}=\\bar\{V\}\_\{\\phi z\}\.

\(v\) Now we will considerVϕ​yV\_\{\\phi y\}:

Let

Vϕ​yl\\displaystyle V\_\{\\phi y\}^\{l\}=𝔼z​\[ϱ⁡\(Wd​\(a​xl\+h​z\)\)​\(a​xl\+h​z\)T\],\\displaystyle=\\mathbb\{E\}\_\{z\}\{\\left\[\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\\right\)\(ax\_\{l\}\+\\sqrt\{h\}z\)^\{T\}\\right\]\}\\;,=a​𝔼z​\[ϱ⁡\(Wd​\(a​xl\+h​z\)\)\]​xlT\+h​Vϕ​zl,\\displaystyle=a\\mathbb\{E\}\_\{z\}\{\\left\[\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\\right\)\\right\]\}x\_\{l\}^\{T\}\+\\sqrt\{h\}V\_\{\\phi z\}^\{l\}\\;,=a​ϱ0​\(Wd​xl\)​xlT\+h​Vϕ​zl\.\\displaystyle=a\\varrho\_\{0\}\\left\(\\frac\{W\}\{\\sqrt\{d\}\}x\_\{l\}\\right\)x\_\{l\}^\{T\}\+\\sqrt\{h\}V\_\{\\phi z\}^\{l\}\\;\.Therefore, summing over thenndata samples:

Vϕ​y\\displaystyle V\_\{\\phi y\}=1n​∑l=1nVϕ​yl,\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}V^\{l\}\_\{\\phi y\}\\;,=a​1n​ϱ0​\(Wd​X\)​XT\+h​Vϕ​z\.\\displaystyle=a\\frac\{1\}\{n\}\\varrho\_\{0\}\\left\(\\frac\{W\}\{\\sqrt\{d\}\}X\\right\)X^\{T\}\+\\sqrt\{h\}V\_\{\\phi z\}\\;\.
\(vi\) Now, let’s considerUU:Let

Ul\\displaystyle U^\{l\}=𝔼z​\[ϱ⁡\(Wd​\(a​xl\+h​z\)\)​ϱ​\(Wd​\(a​xl\+h​z\)\)T\]\.\\displaystyle=\\mathbb\{E\}\_\{z\}\{\\left\[\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\\right\)\\varrho\\left\(\\frac\{W\}\{\\sqrt\{d\}\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\\right\)^\{T\}\\right\]\}\\;\.Fori≠ji\\neq j, expand in the correlation between the two noise projections\. The zeroth\-order term gives the product of noise\-averaged features, while the first\-order term involvesϱ1\\varrho\_\{1\}:

Ui​jl\\displaystyle U^\{l\}\_\{ij\}=𝔼z​\[ϱ⁡\(wiT​\(a​xl\+h​z\)d\)​ϱ​\(wjT​\(a​xl\+h​z\)d\)\]\\displaystyle=\\mathbb\{E\}\_\{z\}\{\\left\[\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)\\varrho\\left\(\\frac\{w\_\{j\}^\{T\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)\\right\]\}=𝔼\(u,v\)∼PwiT​wjd​\[ϱ⁡\(a​wiT​xld\+h​u\)​ϱ​\(a​wjT​xld\+h​v\)\]\\displaystyle=\\mathbb\{E\}\_\{\(u,v\)\\sim P^\{\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\}\}\{\\left\[\\varrho\\left\(a\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)\\varrho\\left\(a\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}v\\right\)\\right\]\}=∑k=0∞\(wiT​wjd\)kk\!​𝔼u​\[ϱ⁡\(a​wiT​xld\+h​u\)​Hek​\(u\)\]​𝔼v​\[ϱ⁡\(a​wjT​xld\+h​v\)​Hek​\(v\)\]\\displaystyle=\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\)^\{k\}\}\{k\!\}\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\\left\(a\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}u\\right\)\\text\{He\}\_\{k\}\(u\)\\right\]\}\\mathbb\{E\}\_\{v\}\{\\left\[\\varrho\\left\(a\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\+\\sqrt\{h\}v\\right\)\\text\{He\}\_\{k\}\(v\)\\right\]\}=ϱ0​\(wiT​xld\)​ϱ0​\(wjT​xld\)\+wiT​wjd​ϱ1​\(wiT​xld\)​ϱ1​\(wjT​xld\)\+O⁡\(1/d\),\\displaystyle=\\varrho\_\{0\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{0\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{1\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+O\(1/d\)\\;,whereϱ0​\(y\)=𝔼u​\[ϱ⁡\(a​y\+h​u\)\]\\varrho\_\{0\}\(y\)=\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\(ay\+\\sqrt\{h\}u\)\\right\]\}andϱ1​\(y\)=𝔼u​\[ϱ⁡\(a​y\+h​u\)​u\]\\varrho\_\{1\}\(y\)=\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\(ay\+\\sqrt\{h\}u\)u\\right\]\}\. Summing over thenndata samples:

Ui​j\\displaystyle U\_\{ij\}=1n​∑l=1nUi​jl\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}U^\{l\}\_\{ij\}=1n​∑l=1nϱ0​\(wiT​xld\)​ϱ0​\(wjT​xld\)\+wiT​wjd​1n​∑l=1nϱ1​\(wiT​xld\)​ϱ1​\(wjT​xld\)\+O⁡\(1/d\)\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\varrho\_\{0\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{0\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{1\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+O\(1/d\)=1n​∑l=1nϱ0​\(wiT​xld\)​ϱ0​\(wjT​xld\)\+wiT​wjd​𝔼x​\[ϱ1​\(wiT​xd\)​ϱ1​\(wjT​xd\)\]\+O⁡\(1/d\)\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\varrho\_\{0\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{0\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\\mathbb\{E\}\_\{x\}\{\\left\[\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{1\}\\left\(\\frac\{w\_\{j\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\right\]\}\+O\(1/d\)=\(a\)1n​∑l=1nϱ0​\(wiT​xld\)​ϱ0​\(wjT​xld\)\+wiT​wjd​𝔼g​\[ϱ1​\(σ0​g\)\]2\+O⁡\(1/d\)\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{=\}\}\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\varrho\_\{0\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{0\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\\mathbb\{E\}\_\{g\}\{\\left\[\\varrho\_\{1\}\(\\sigma\_\{0\}g\)\\right\]\}^\{2\}\+O\(1/d\)=1n​∑l=1nϱ0​\(wiT​xld\)​ϱ0​\(wjT​xld\)\+h​μ12​\(σ\)σ2​wiT​wjd\+O⁡\(1/d\)\.\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\varrho\_\{0\}\\left\(\\frac\{w\_\{i\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{0\}\\left\(\\frac\{w\_\{j\}^\{T\}x\_\{l\}\}\{\\sqrt\{d\}\}\\right\)\+h\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{w\_\{i\}^\{T\}w\_\{j\}\}\{d\}\+O\(1/d\)\\;\.In step\(a\)\(a\), we apply Mehler’s formula to𝔼x​\[ϱ1​\(wiT​xd\)​ϱ1​\(wjT​xd\)\]\\mathbb\{E\}\_\{x\}\{\\left\[\\varrho\_\{1\}\\left\(\\frac\{w\_\{i\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\varrho\_\{1\}\\left\(\\frac\{w\_\{j\}^\{T\}x\}\{\\sqrt\{d\}\}\\right\)\\right\]\}and neglected terms that will lead toO⁡\(1/d\)O\(1/d\)terms\. The diagonal requires a separate calculation because a feature is perfectly correlated with itself\. Fori=ji=j, we have:

Ui​il\\displaystyle U^\{l\}\_\{ii\}=𝔼z​\[\(ϱ⁡\(wiT​\(a​xl\+h​z\)d\)\)2\],\\displaystyle=\\mathbb\{E\}\_\{z\}\{\\left\[\\left\(\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)\\right\)^\{2\}\\right\]\}\\;,and

Ui​i\\displaystyle U\_\{ii\}=1n​∑l=1n𝔼z​\[\(ϱ⁡\(wiT​\(a​xl\+h​z\)d\)\)2\]\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{l=1\}^\{n\}\\mathbb\{E\}\_\{z\}\{\\left\[\\left\(\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\_\{l\}\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)\\right\)^\{2\}\\right\]\}=𝔼z,x​\[\(ϱ⁡\(wiT​\(a​x\+h​z\)d\)\)2\]\+O⁡\(1/d\)\\displaystyle=\\mathbb\{E\}\_\{z,x\}\{\\left\[\\left\(\\varrho\\left\(\\frac\{w\_\{i\}^\{T\}\(ax\+\\sqrt\{h\}z\)\}\{\\sqrt\{d\}\}\\right\)\\right\)^\{2\}\\right\]\}\+O\(1/\\sqrt\{d\}\)=‖ϱ\(σ⋅\)‖2\+O⁡\(1/d\)\.\\displaystyle=\\norm\{\\act\{\\sigma\\;\\cdot\}\}^\{2\}\+O\(1/\\sqrt\{d\}\)\\;\.TheO⁡\(1/d\)\{O\\left\(1/\\sqrt\{d\}\\right\)\}term in the above equation can be neglected, since there are onlyO⁡\(d\)\{O\\left\(d\\right\)\}terms on the diagonal\. LetX=\[x1,x2,⋯,xn\]∈ℝd×nX=\[x\_\{1\},x\_\{2\},\\cdots,x\_\{n\}\]\\in\\mathbb\{R\}^\{d\\times n\}\. We can writeUUas:

U=ϱ0​\(Wd​X\)n​ϱ0​\(Wd​X\)Tn\+h​μ12​\(σ\)σ2​Wd​WTd\+s2​Ip,U=\\frac\{\\varrho\_\{0\}\\left\(\\frac\{W\}\{\\sqrt\{d\}\}X\\right\)\}\{\\sqrt\{n\}\}\\frac\{\\varrho\_\{0\}\\left\(\\frac\{W\}\{\\sqrt\{d\}\}X\\right\)^\{T\}\}\{\\sqrt\{n\}\}\+h\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{W\}\{\\sqrt\{d\}\}\\frac\{W^\{T\}\}\{\\sqrt\{d\}\}\+s^\{2\}I\_\{p\}\\;,where

s2\\displaystyle s^\{2\}=‖ϱ\(σ⋅\)‖2−𝔼g​\[ϱ0​\(σ0​g\)2\]−hσ2​μ12​\(σ\)\\displaystyle=\\norm\{\\varrho\(\\sigma\\;\\cdot\)\}^\{2\}\-\\mathbb\{E\}\_\{g\}\{\\left\[\\varrho\_\{0\}\(\\sigma\_\{0\}g\)^\{2\}\\right\]\}\-\\frac\{h\}\{\\sigma^\{2\}\}\\mu\_\{1\}^\{2\}\(\\sigma\)=‖ϱ\(σ⋅\)‖2−𝔼g​\[𝔼u​\[ϱ⁡\(a​σ0​g\+h​u\)\]2\]−hσ2​μ12​\(σ\)\\displaystyle=\\norm\{\\varrho\(\\sigma\\;\\cdot\)\}^\{2\}\-\\mathbb\{E\}\_\{g\}\{\\left\[\\mathbb\{E\}\_\{u\}\{\\left\[\\varrho\(a\\sigma\_\{0\}g\+\\sqrt\{h\}u\)\\right\]\}^\{2\}\\right\]\}\-\\frac\{h\}\{\\sigma^\{2\}\}\\mu\_\{1\}^\{2\}\(\\sigma\)=‖ϱ\(σ⋅\)‖2−c⁡\(a2​σ02/σ2,σ\)−hσ2​μ12​\(σ\),\\displaystyle=\\norm\{\\varrho\(\\sigma\\;\\cdot\)\}^\{2\}\-c\(a^\{2\}\\sigma\_\{0\}^\{2\}/\\sigma^\{2\},\\sigma\)\-\\frac\{h\}\{\\sigma^\{2\}\}\\mu\_\{1\}^\{2\}\(\\sigma\)\\;,withc⁡\(γ,κ\)=𝔼u,v∼Pγ​\[ϱ⁡\(κ​u\)​ϱ​\(κ​v\)\]c\(\\gamma,\\kappa\)=\\mathbb\{E\}\_\{u,v\\sim P^\{\\gamma\}\}\{\\left\[\\varrho\(\\kappa u\)\\varrho\(\\kappa v\)\\right\]\}\. LetH:=ϱ0​\(Wd​X\)H:=\\varrho\_\{0\}\\left\(\\frac\{W\}\{\\sqrt\{d\}\}X\\right\)\. We have found:

U=Hn​HTn\+h​μ12​\(σ\)σ2​Wd​WTd\+s2​Ip,U=\\frac\{H\}\{\\sqrt\{n\}\}\\frac\{H^\{T\}\}\{\\sqrt\{n\}\}\+h\\frac\{\\mu\_\{1\}^\{2\}\(\\sigma\)\}\{\\sigma^\{2\}\}\\frac\{W\}\{\\sqrt\{d\}\}\\frac\{W^\{T\}\}\{\\sqrt\{d\}\}\+s^\{2\}I\_\{p\}\\;,

## Appendix BAdditional Numerical Results

### B\.1Test and train errors for different diffusion timett

Figures[6](https://arxiv.org/html/2610.20850#A2.F6)and[7](https://arxiv.org/html/2610.20850#A2.F7)extend the learning curves tot=0\.1t=0\.1\. The main rank\-dependent trends persist at this larger diffusion time: training error decreases with rank, while test error depends on both sample size and data dimension\. In the displayed settings withn\>Dn\>D, test error reaches a minimum nearr=d−Dr=d\-Dand then increases, whereas forn<Dn<Dit increases throughout the displayed rank range\. The random\-feature predictions agree closely with the simulated training and test errors and reproduce these qualitative trends in the trained networks, supporting the relevance of the geometric reference rank beyondt=0\.01t=0\.01\.

Figure 6:Simulation results for our numerical model, withtanh\\tanhactivation att=0\.1t=0\.1,p/d=2p/d=2\.Figure 7:Analytical and simulation results for our theoretical model withtanh\\tanhactivation att=0\.1t=0\.1,ψp=2\.0\\psi\_\{p\}=2\.0\.

## Appendix CSimulation Conditions

We describe the simulations of the trainable score network in \([5](https://arxiv.org/html/2610.20850#S3.E5)\), reported in Figures[2](https://arxiv.org/html/2610.20850#S4.F2)and[6](https://arxiv.org/html/2610.20850#A2.F6)\.

##### Data and parameter sweep\.

We set the ambient dimension tod=256d=256and the hidden\-layer width top=512p=512\. For each dataset seed, an orthonormal basisΘ∈ℝd×D\\Theta\\in\\mathbb\{R\}^\{d\\times D\}is obtained by a QR decomposition of a standard Gaussian matrix\. Independent training and test samples are drawn asx=D​Θ​g/‖g‖x=\\sqrt\{D\}\\,\\Theta g/\\\|g\\\|, withg∼𝒩⁡\(0,ID\)g\\sim\\mathcal\{N\}\(0,I\_\{D\}\)\. We useD∈\{78,176\}D\\in\\\{78,176\\\}andn∈\{128,512\}n\\in\\\{128,512\\\}, corresponding toD/d≈0\.305,0\.688D/d\\approx 0\.305,0\.688andn/d=0\.5,2n/d=0\.5,2\. Each test set contains512512independent clean samples from the same distribution\.

##### Initialization and training\.

The activation istanh\\tanh, and the factorsWt,At,BtW\_\{t\},A\_\{t\},B\_\{t\}are initialized with independent standard Gaussian entries, with the normalization factors specified in \([5](https://arxiv.org/html/2610.20850#S3.E5)\)\. Biases are initialized to zero\. All three weight matrices and the biases are trained for30,00030\{,\}000Adam\([Kingma and Ba, 2017](https://arxiv.org/html/2610.20850#bib.bib18)\)updates with a constant learning rate of10−210^\{\-2\}, batch size512512, no weight decay, and global gradient\-norm clipping at1010\. Each update samples clean training points uniformly with replacement and draws fresh standard Gaussian noise, formingy=at​x\+ht​zy=a\_\{t\}x\+\\sqrt\{h\_\{t\}\}z\. The optimized loss is the mean of‖ht​st​\(y\)\+z‖2/d\\\|\\sqrt\{h\_\{t\}\}s\_\{t\}\(y\)\+z\\\|^\{2\}/dover the minibatch\. Within each seed and diffusion time, models at different ranks share initial weight entries, minibatch indices, and noise realizations, but maintain separate trainable parameters and optimizer states\. The rank constraint is implemented by retaining the firstrrterms of the factorized output map\.

##### Evaluation and averaging\.

Training and test errors evaluate the same normalized DSM objective on all clean training points and on the independent test set, respectively\. Each evaluation averages over1616fresh noise realizations per clean sample, processed in batches of256256; evaluation noise is separate from training noise and shared across ranks at a fixed diffusion time\. Plotted curves use the final evaluation at30,00030\{,\}000updates\. We average over three paired dataset and initialization seeds,\(0,0\)\(0,0\),\(1,1\)\(1,1\), and\(2,2\)\(2,2\)\. The implementation uses JAX, Flax, and Optax with double\-precision arrays\.