A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-norm

# A Data-dependent Early Stopping Rule using Rademacher Complexity with 𝐿_1-norm
Source: [https://arxiv.org/html/2608.24210](https://arxiv.org/html/2608.24210)
Duy Hoang, Bastien Berret, Olivier Bruneau, and Laurent Fribourg

Duy Hoang hoangduy@lmf\.cnrs\.frAffiliation:Université Paris\-Saclay, CNRS, ENS Paris\-Saclay, LMFAffiliation:Gif\-sur\-Yvette, 91190, FranceBastien Berret bastien\.berret@universite\-paris\-saclay\.frOlivier Bruneau olivier\.bruneau@ens\-paris\-saclay\.frAffiliation:Université Paris\-Saclay, ENS Paris\-Saclay, LURPAAffiliation:Gif\-sur\-Yvette, 91190, FranceLaurent Fribourg fribourg@lmf\.cnrs\.frAffiliation:Université Paris\-Saclay, CNRS, ENS Paris\-Saclay, LMFAffiliation:Gif\-sur\-Yvette, 91190, France

###### Abstract

Training neural networks requires balancing the trade\-off between fitting the training data and achieving robust performance on unseen inputs\. This ability, commonly referred to as generalizability, is determined by the gap between the empirical risk on the training set \(“empirical loss”\) and the expected risk over the data distribution \(“generalization error”\)\. Existing approaches typically estimate the generalization error numerically, requiring gradient descent training and an “early stopping” strategy\. In this work, we introduce an analytic framework that estimates the optimal time of early stopping without the need for training\. Several works in the literature also give such analytical estimations, but they are generally based on random matrix theory and often make assumptions on the distribution of the data or the eigenvalue distribution of the covariance matrix\. In contrast, our work is based on Rademacher complexity \(RC\) without needing such probabilistic assumptions\. For both theoretical and numerical reasons, it is more relevant to express RC with theL1L\_\{1\}\-norm rather than with theL2L\_\{2\}\-norm\. We focus on the case of linear models and the problem of linear regression\. Thanks to the “linear probing” method, our results can, however, be successfully applied to nonlinear neural networks, as illustrated in the classification MNIST example\.

††heading:23 2026 1\-1/21; Revised 5/22 9/22 21\-0000††shortheadings:A Data\-dependent Early Stopping Rule using Rademacher Complexity withL1L\_\{1\}\-norm / Hoang, Berret, Bruneau, and Fribourg††firstpage:1###### keywords

generalization, bias\-variance trade\-off, linear regression\.

## 1Introduction

Given a distribution𝒟\\cal\{D\}of input/output pairs, the expected loss across𝒟\\cal\{D\}is referred to as a “population loss”, and denotedL𝒟L\_\{\{\\cal D\}\}\. Using a neural network \(NN\) and a process of gradient descent \(GD\), one can estimateL𝒟L\_\{\{\\cal D\}\}on a setSSofnnsamples randomly selected from𝒟\\cal\{D\}\. This is called the “empirical loss”, and denotedLSL\_\{S\}\. The generalization loss, denotedLGL\_\{G\}, is the difference betweenL𝒟L\_\{\\cal\{D\}\}andLSL\_\{S\}, and accounts for the data located outsideSS\. At the beginning of the GD process,LSL\_\{S\}tends to decrease whileLGL\_\{G\}tends to increase \(“bias\-variance” tradeoff\), so theearly stoppingstrategy seeks to halt GD at a timet∗t^\{\*\}minimizingLS\+LGL\_\{S\}\+L\_\{G\}\. Classically,t∗t^\{\*\}is estimatednumericallyas the timett​e​s​tt\_\{test\}that minimizes the empirical loss on a separate datasetSt​e​s​tS\_\{test\}\(see, e\.g\.,[Prechelt \(2002\)](https://arxiv.org/html/2608.24210#bib.bib4)\)\. Usingrandom matrix theory\(RMT\), several works estimatet∗t^\{\*\}analytically, without needing to perform GD \(e\.g\.,[Raskutti et al\. \(2014\)](https://arxiv.org/html/2608.24210#bib.bib12);[Liao and Couillet \(2018\)](https://arxiv.org/html/2608.24210#bib.bib1);[Ali et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib9);[Advani et al\. \(2020\)](https://arxiv.org/html/2608.24210#bib.bib24)\)\. However these works often assume the distribution𝒟\{\\cal D\}to be Gaussian, or the eigenvalue distribution of the data covariance matrix to be Marchenko–Pastur \([Le Cun et al\. \(1991\)](https://arxiv.org/html/2608.24210#bib.bib8)\)\. In contrast here, we estimatet∗t^\{\*\}analytically usingRademacher complexity\(RC\) theory \([Bartlett and Mendelson \(2002\)](https://arxiv.org/html/2608.24210#bib.bib20)\), and do not make any assumption about the data distribution\. Our method is based on a data\-dependent criterion of the form𝒞⁡\(s\)\{\\cal C\}\(s\)that ensuresLS​\(t\)\+LG​\(t\)L\_\{S\}\(t\)\+L\_\{G\}\(t\)todecreasefor allt≤st\\leq s\. We show that the highestsssatisfying𝒞⁡\(s\)\{\\cal C\}\(s\)is a low estimatet\+t^\{\+\}oft∗t^\{\*\}\(see Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\)\. We also give sufficient conditions that ensuret\+=t∗t^\{\+\}=t^\{\*\}\. In order to eliminate a factorMMdependent on𝒟\{\\cal D\}, we estimate RC using theL1L\_\{1\}\-norm rather than theL2L\_\{2\}\-norm \(see Remark[2](https://arxiv.org/html/2608.24210#Thmtheorem2)\)\. On the examples \(see Section[4](https://arxiv.org/html/2608.24210#S4)\), we check that theL1L\_\{1\}\-norm yields a value oft\+t^\{\+\}much closer to the numerical stopping timett​e​s​tt\_\{test\}than theL2L\_\{2\}\-norm\.

We also give an analytic form for the value ofLS\+LGL\_\{S\}\+L\_\{G\}att=∞t=\\infty\(see Section[3\.3](https://arxiv.org/html/2608.24210#S3.SS3)\)\. By comparing it withLS\+LGL\_\{S\}\+L\_\{G\}att=t\+t=t^\{\+\}, we determine whether the early stopping strategy should be applied or not \(see, e\.g\.,[Sonthalia et al\. \(2024\)](https://arxiv.org/html/2608.24210#bib.bib14);[Bartlett et al\. \(2020\)](https://arxiv.org/html/2608.24210#bib.bib13);[Belkin et al\. \(2018\)](https://arxiv.org/html/2608.24210#bib.bib11)\)\. Our work focuses onlinearmodels\. However, our results can be applied tononlinearNNs thanks to the linear probing strategy \(see[Alain and Bengio \(2016\)](https://arxiv.org/html/2608.24210#bib.bib5)\)\. This is illustrated on MNIST classification examples \(Examples[11](https://arxiv.org/html/2608.24210#Thmtheorem11)and[12](https://arxiv.org/html/2608.24210#Thmtheorem12), Section[4](https://arxiv.org/html/2608.24210#S4)\)\. The proofs of our results are given in Appendix\.

### Comparison with related work

As mentioned earlier, several works obtain analytical upper bounds on the generalization loss using RMT, and used them to estimate the optimal stopping time\. More precisely, letλ1≥λ2≥⋯≥λn≥0\\lambda\_\{1\}\\geq\\lambda\_\{2\}\\geq\\cdots\\geq\\lambda\_\{n\}\\geq 0be the eigenvalues of the data covariance matrix\. In[Advani et al\. \(2020\)](https://arxiv.org/html/2608.24210#bib.bib24)for example, they evaluate a stopping time that minimizes the average generalization dynamics, by determining the error associated with each modeii\(i∈\[n\]i\\in\[n\]\)\. For modeii, they find an optimal stopping time of the form

topt=1λi​ln⁡\(1\+λi⋅S​N​R\),t^\{\\text\{opt\}\}=\\frac\{1\}\{\\lambda\_\{i\}\}\\ln\(1\+\\lambda\_\{i\}\\cdot SNR\),whereS​N​RSNRis a signal\-to\-noise ratio\. Here, without making any probabilistic assumptions, we find a similar result using Rademacher complexity, viz\., an estimate of the optimal stopping time of the form \(see Equation \([27](https://arxiv.org/html/2608.24210#S3.E27)\)\)

t\+≈1λ1​ln⁡Γ⁡\(0\)Ω⁡\(0\)\.t^\{\+\}\\approx\\frac\{1\}\{\\lambda\_\{1\}\}\\ln\\frac\{\\Gamma\(0\)\}\{\\Omega\(0\)\}\.The numeratorΓ\\Gammadepends on the higher eigenvaluesλ1,…,λα\\lambda\_\{1\},\\dots,\\lambda\_\{\\alpha\}, and contains the “informative” part \(see, e\.g\.,[Oymak et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib22)\)\. The denominatorΩ\\Omegadepends on the lower eigenvaluesλα\+1,…,λn\\lambda\_\{\\alpha\+1\},\\dots,\\lambda\_\{n\}, and contains the “nuisance” part\. So the quotientΓ/Ω\\Gamma/\\Omegacan be interpreted as a form of SNR ratio\.

In the literature, Rademacher complexity \(together with “Neural Tangent Kernel” theory\) has often been used to find analytical bounds on the population loss: see, e\.g\.,[Jacot et al\. \(2018\)](https://arxiv.org/html/2608.24210#bib.bib25);[Du et al\. \(2018\)](https://arxiv.org/html/2608.24210#bib.bib28);[Arora et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib26);[Allen\-Zhu et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib6);[Oymak et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib22);[Li et al\. \(2020\)](https://arxiv.org/html/2608.24210#bib.bib27)\. However, these studies require the numbermmof NN parameters be large compared to the numbernnof samples \(overparameterization\)\. In this case, a phenomenon of “benign overfitting” or “epochwise double descent” appears \(see, e\.g\.,[Heckel and Yilmaz \(2020\)](https://arxiv.org/html/2608.24210#bib.bib18);[Stephenson and Lee \(2021\)](https://arxiv.org/html/2608.24210#bib.bib16);[Nakkiran et al\. \(2021\)](https://arxiv.org/html/2608.24210#bib.bib15)\): The lossLt​e​s​t​\(t\)L\_\{test\}\(t\)reaches a first local minimum att=tt​e​s​tt=t\_\{test\}, then increases before going down later, converging towards a minimum lower than att=tt​e​s​tt=t\_\{test\}\. In this context, the strategy of early stopping is not “beneficial”\. In contrast, our method is well adapted to theunderparameterizedcase \(i\.e\.,m≤nm\\leq n\)\.

### Notation

In this paper,ℝ\\mathbb\{R\}andℕ\\mathbb\{N\}refer to the sets of real and natural numbers, respectively\. We denote byℝp\\mathbb\{R\}^\{p\}app\-dimensional Euclidean space, and byℝp×q\\mathbb\{R\}^\{p\\times q\}a space of real matrices withpprows andqqcolumns\. We use bold letters for vectors and bold capital letters for matrices\. For a given matrix𝑴∈ℝp×q\\boldsymbol\{M\}\\in\\mathbb\{R\}^\{p\\times q\}, we use𝑴⊤\\boldsymbol\{M\}^\{\\top\}for its transpose, and𝑴†\\boldsymbol\{M\}^\{\\dagger\}for its Moore\-Penrose \(pseudo\)inverse\. TheLp​\-normL\_\{p\}\\text\{\-norm\}of a vector𝒗\\boldsymbol\{v\}is denoted by‖𝒗‖p\\\|\\boldsymbol\{v\}\\\|\_\{p\}\. We usei\.i\.d\.to indicate the set of independent and identically distributed random variables\. We use\[n\]\[n\]for\{1,…,n\}\\\{1,\\dots,n\\\}, and𝑰\\boldsymbol\{I\}for the identity matrix\. Forv∈ℝv\\in\\mathbb\{R\}, we usesgn​\(v\)\\text\{sgn\}\(v\)to denote11ifv≥0v\\geq 0, or−1\-1ifv<0v<0\. For𝒗=\(v1,…,vn\)∈ℝn\\boldsymbol\{v\}=\(v\_\{1\},\\dots,v\_\{n\}\)\\in\\mathbb\{R\}^\{n\}, we usesgn​\(𝒗\)\\text\{sgn\}\(\\boldsymbol\{v\}\)for\(sgn​\(v1\),…,sgn​\(vn\)\)\(\\text\{sgn\}\(v\_\{1\}\),\\dots,\\text\{sgn\}\(v\_\{n\}\)\)\.

## 2Preliminary Results

We consider a distribution𝒟\\cal\{D\}overℋ×𝒴\{\\cal H\}\\times\\cal\{Y\}whereℋ⊂\{\\cal H\}\\subsetℝm\\mathbb\{R\}^\{m\}is the input space of all possible instances𝒉\\boldsymbol\{h\}, and𝒴⊂ℛ\\cal\{Y\}\\subset\\mathbb\{R\}the space of the corresponding outputs\.

### 2\.1Population loss and Rademacher complexity

The “training set”SSis a set ofnninput/output pairs\{\(𝒉1,y1\),…,\(𝒉n,yn\)\}\\\{\(\\boldsymbol\{h\}\_\{1\},y\_\{1\}\),\\dots,\(\\boldsymbol\{h\}\_\{n\},y\_\{n\}\)\\\}made ofnnsamples selectedi\.i\.d\.from𝒟\\cal\{D\}\. Let𝒚=\(y1,…,yn\)∈ℝn\\boldsymbol\{y\}=\(y\_\{1\},\\dots,y\_\{n\}\)\\in\\mathbb\{R\}^\{n\}\. Let𝑲\\boldsymbol\{K\}be then×mn\\times mmatrix having𝒉1,…,𝒉n∈ℝm\\boldsymbol\{h\}\_\{1\},\\dots,\\boldsymbol\{h\}\_\{n\}\\in\\mathbb\{R\}^\{m\}as columns\. The data covariance matrix𝑯∈ℝn×n\\boldsymbol\{H\}\\in\\mathbb\{R\}^\{n\\times n\}is defined by

𝑯=𝑲​𝑲⊤,\\boldsymbol\{H\}=\\boldsymbol\{K\}\\boldsymbol\{K\}^\{\\top\},i\.e\., the\(i,j\)\(i,j\)\-entry of𝑯\\boldsymbol\{H\}isHi,j=𝒉i⊤​𝒉jH\_\{i,j\}=\\boldsymbol\{h\}\_\{i\}^\{\\top\}\\boldsymbol\{h\}\_\{j\}\. \(See, e\.g\.,[Du et al\. \(2018\)](https://arxiv.org/html/2608.24210#bib.bib28)\.\) As in[Martin Xavier et al\. \(2025\)](https://arxiv.org/html/2608.24210#bib.bib23), we focus here on theL1L\_\{1\}\-norm \(see Remark[2](https://arxiv.org/html/2608.24210#Thmtheorem2)\)\. In this context, thepopulation lossL𝒟​\[f\]L\_\{\\cal D\}\[f\]\(more simply denoted asL𝒟L\_\{\\cal\{D\}\}\) over data distribution𝒟\{\\cal D\}is:

L𝒟​\[f\]=𝔼\(𝒉,y\)∼𝒟​\[\|f⁡\(𝒉\)−y\|\],L\_\{\{\\cal D\}\}\[f\]=\\mathbb\{E\}\_\{\(\\boldsymbol\{h\},y\)\\sim\{\\cal D\}\}\[\|f\(\\boldsymbol\{h\}\)\-y\|\],wheref:ℝm→ℝf:\\mathbb\{R\}^\{m\}\\rightarrow\\mathbb\{R\}is a given function\. As we focus here on the linear regression problem,ffis of the form:

f⁡\(𝒉\)=𝒂⊤​𝒉f\(\\boldsymbol\{h\}\)=\\boldsymbol\{a\}^\{\\top\}\\boldsymbol\{h\}where𝒂=\(a1,…,am\)∈ℝm\\boldsymbol\{a\}=\(a\_\{1\},\\dots,a\_\{m\}\)\\in\\mathbb\{R\}^\{m\}\. Theempirical lossLSL\_\{S\}overSSis defined by:

LS=1n​∑i∈\[n\]\|𝒂⊤​𝒉i−yi\|=1n​∑i∈\[n\]\|vi\|=1n​‖𝒗‖1L\_\{S\}=\\frac\{1\}\{n\}\\sum\_\{i\\in\[n\]\}\|\\boldsymbol\{a\}^\{\\top\}\\boldsymbol\{h\}\_\{i\}\-y\_\{i\}\|=\\frac\{1\}\{n\}\\sum\_\{i\\in\[n\]\}\|v\_\{i\}\|=\\frac\{1\}\{n\}\\\|\\boldsymbol\{v\}\\\|\_\{1\}\(1\)with

vi=\\displaystyle v\_\{i\}=𝒂⊤​𝒉i−yi∈ℝ\(i∈\[n\]\),\\displaystyle\\;\\boldsymbol\{a\}^\{\\top\}\\boldsymbol\{h\}\_\{i\}\-y\_\{i\}\\in\\mathbb\{R\}\\ \\ \\ \(i\\in\[n\]\),\(2\)𝒗=\\displaystyle\\boldsymbol\{v\}=\(v1,…,vn\)=𝑲​𝒂−𝒚∈ℝn\.\\displaystyle\\;\(v\_\{1\},\\dots,v\_\{n\}\)=\\boldsymbol\{K\}\\boldsymbol\{a\}\-\\boldsymbol\{y\}\\in\\mathbb\{R\}^\{n\}\.\(3\)The vector𝒗\\boldsymbol\{v\}is called theempirical error vector\(or thetraining error vector\)\. We are searching for a vector𝒂\\boldsymbol\{a\}that minimizes the population lossL𝒟L\_\{\{\\cal D\}\}across the entire data distribution𝒟\\cal\{D\}\. Since𝒟\\cal Dis unknown, our objective is actually to minimize anupper boundonL𝒟L\_\{\{\\cal D\}\}\. LetMMandCCbe two positive reals satisfying respectively:

\|𝒂⊤​𝒉−y\|≤M∀\(𝒉,y\)∈ℋ×𝒴\|\\boldsymbol\{a\}^\{\\top\}\\boldsymbol\{h\}\-y\|\\leq M\\ \\ \\ \\forall\(\\boldsymbol\{h\},y\)\\in\{\\cal H\}\\times\{\\cal Y\}\\\\\(4\)𝔼𝒉∼𝒟​\[‖𝒉‖22\]≤C2\.\\ \\ \\ \\mathbb\{E\}\_\{\\boldsymbol\{h\}\\sim\{\\cal D\}\}\\left\[\\\|\\boldsymbol\{h\}\\\|\_\{2\}^\{2\}\\right\]\\leq C^\{2\}\.\(5\)LetLG∗L\_\{G\}^\{\*\}andL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}be defined as:

LG∗=\\displaystyle L\_\{G\}^\{\*\}=2​‖𝒂‖2​Cn\\displaystyle\\;\\frac\{2\\\|\\boldsymbol\{a\}\\\|\_\{2\}C\}\{\\sqrt\{n\}\}\(6\)L𝒟∗=\\displaystyle L\_\{\{\\cal D\}\}^\{\*\}=LS\+LG∗=1n​‖𝒗‖1\+2​‖𝒂‖2​Cn\.\\displaystyle\\;L\_\{S\}\+L\_\{G\}^\{\*\}=\\frac\{1\}\{n\}\\\|\\boldsymbol\{v\}\\\|\_\{1\}\+\\frac\{2\\\|\\boldsymbol\{a\}\\\|\_\{2\}C\}\{\\sqrt\{n\}\}\.\(7\)
We follow an approach based on RC \(see[Martin Xavier et al\. \(2025\)](https://arxiv.org/html/2608.24210#bib.bib23)\)\. In the linear setting, the result is as follows\.

###### Proposition 1\.

\(cf\. Proposition 3 of[Martin Xavier et al\. \(2025\)](https://arxiv.org/html/2608.24210#bib.bib23)\) With probability at least1−δ1\-\\deltaover the sampleSSof sizenn, the population lossL𝒟L\_\{\{\\cal D\}\}satisfies:

L𝒟≤L𝒟∗\+ϵL\_\{\{\\cal D\}\}\\leq L\_\{\{\\cal D\}\}^\{\*\}\+\\epsilon\(8\)where

ϵ=3​M​log⁡2δ2​n\.\\epsilon=3M\\sqrt\{\\frac\{\\log\\frac\{2\}\{\\delta\}\}\{2n\}\}\.\(9\)

### 2\.2Gradient flow for linear models

For simplicity, we express the problem in the continuous\-time setting, and consider the gradient flow process \(GF\) instead of discrete\-time GD\. The idea is to apply GF to𝒂\\boldsymbol\{a\}, and stop the process at the timet∗t^\{\*\}whereL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}is expected to reach its minimum \(strategy of “early stopping”\)\. Classically, one estimatest∗t^\{\*\}by considering a separate set of dataSt​e​s​tS\_\{test\}, and determine the timett​e​s​tt\_\{test\}where GF reaches its minimum on that set \(see,e\.g\.,[Prechelt \(2002\)](https://arxiv.org/html/2608.24210#bib.bib4)\)\. We give here a method that allows us to estimatet∗t^\{\*\}analytically without needing a separate set or having to apply GF\. More formally, we consider the problem of finding𝒂=\(a1,…,am\)∈ℝm\\boldsymbol\{a\}=\(a\_\{1\},\.\.\.,a\_\{m\}\)\\in\\mathbb\{R\}^\{m\}that minimizes the following quadratic loss function:

ℒ⁡\(𝒂\)=12​∑i=1n\(𝒂⊤​𝒉i−𝒚i\)2=12​∑i=1nvi2=12​‖𝒗‖22\.\{\\cal L\}\(\\boldsymbol\{a\}\)=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}\(\\boldsymbol\{a\}^\{\\top\}\\boldsymbol\{h\}\_\{i\}\-\\boldsymbol\{y\}\_\{i\}\)^\{2\}=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}v\_\{i\}^\{2\}=\\frac\{1\}\{2\}\\\|\\boldsymbol\{v\}\\\|\_\{2\}^\{2\}\.\(10\)The vector𝒂\\boldsymbol\{a\}that minimizesℒ⁡\(𝒂\)\{\\cal L\}\(\\boldsymbol\{a\}\)on𝒟\{\\cal D\}is found by considering a given training setSScontainingnnsamples\(𝒉i,yi\)\\left\(\\boldsymbol\{h\}\_\{i\},y\_\{i\}\\right\)drawn randomly i\.i\.d\. from𝒟\\cal\{D\}\. The vector𝒂\\boldsymbol\{a\}is initialized to 0, and is updated via GF onSSas follows:

d​𝒂d​t=−∂ℒ∂𝒂\.\\frac\{d\\boldsymbol\{a\}\}\{dt\}=\-\\frac\{\\partial\{\\cal L\}\}\{\\partial\\boldsymbol\{a\}\}\.Hence, using \([2](https://arxiv.org/html/2608.24210#S2.E2)\), \([3](https://arxiv.org/html/2608.24210#S2.E3)\) and \([10](https://arxiv.org/html/2608.24210#S2.E10)\), we have:

d​𝒂d​t=\\displaystyle\\frac\{d\\boldsymbol\{a\}\}\{dt\}=−∂ℒ∂𝒂=−12∑i=1n∂vi2∂𝒂=−∑i=1nvi𝒉i=−𝑲⊤𝒗\.\\displaystyle\-\\frac\{\\partial\{\\cal L\}\}\{\\partial\\boldsymbol\{a\}\}=\-\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}\\frac\{\\partial v\_\{i\}^\{2\}\}\{\\partial\\boldsymbol\{a\}\}=\-\\sum\_\{i=1\}^\{n\}v\_\{i\}\\boldsymbol\{h\}\_\{i\}=\-\\boldsymbol\{K\}^\{\\top\}\\boldsymbol\{v\}\.\(11\)On the other hand, the dynamic of the training error vector𝒗\\boldsymbol\{v\}during GF is expressed by \(see, e\.g\.,[Du et al\. \(2018\)](https://arxiv.org/html/2608.24210#bib.bib28)\):

d​𝒗d​t=−𝑯​𝒗\.\\frac\{d\\boldsymbol\{v\}\}\{dt\}=\-\\boldsymbol\{H\}\\boldsymbol\{v\}\.\(12\)The GF process makes the first term‖𝒗‖1/n\\\|\\boldsymbol\{v\}\\\|\_\{1\}/nofL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}\(see \([7](https://arxiv.org/html/2608.24210#S2.E7)\)\) decrease, and the second term2​‖𝒂‖2​C/n2\\\|\\boldsymbol\{a\}\\\|\_\{2\}C/\\sqrt\{n\}increase\. The curveL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}is thus ‘‘U\-shaped’’111At least in a first phase, since the curve may decline later on \(“epoch\-wise double descent”\)\.\. It reaches a first local minimum att=t∗t=t^\{\*\}, which is the first time whend​L𝒟∗​\(t\)/d​t≥0dL\_\{\{\\cal D\}\}^\{\*\}\(t\)/dt\\geq 0:

t∗=inft≥0\{t:d​L𝒟∗​\(t\)d​t≥0\}\.t^\{\*\}=\\inf\_\{t\\geq 0\}\\\{t:\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(t\)\}\{dt\}\\geq 0\\\}\.\(13\)

## 3A Data\-dependent Estimatet\+t^\{\+\}oft∗t^\{\*\}

We now explain how to estimate the first local minimum ofL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}without needing to apply GF\.

### 3\.1Analytic form ofd​L𝒟∗/d​tdL\_\{\{\\cal D\}\}^\{\*\}/dt

###### Proposition 5\.

The derivative ofL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}is given by:

d​L𝒟∗​\(t\)d​t=\\displaystyle\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(t\)\}\{dt\}=−1n​\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒗​\(t\)\+Ψ⁡\(t\)\\displaystyle\\;\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{v\}\(t\)\+\\Psi\(t\)\(14\)withΨ⁡\(t\)=\\displaystyle\\Psi\(t\)=−2​Cn​\(𝑲†​\(𝒗⁡\(t\)\+𝒚\)\)⊤‖𝑲†​\(𝒗⁡\(t\)\+𝒚\)‖2​𝑲⊤​𝒗​\(t\)∈ℝ,\\displaystyle\\;\-\\frac\{2C\}\{\\sqrt\{n\}\}\\frac\{\(\\boldsymbol\{K\}^\{\\dagger\}\(\\boldsymbol\{v\}\(t\)\+\\boldsymbol\{y\}\)\)^\{\\top\}\}\{\\\|\\boldsymbol\{K\}^\{\\dagger\}\\left\(\\boldsymbol\{v\}\(t\)\+\\boldsymbol\{y\}\\right\)\\\|\_\{2\}\}\\boldsymbol\{K\}^\{\\top\}\\boldsymbol\{v\}\(t\)\\in\\mathbb\{R\},\(15\)where𝐊†∈ℝm×n\\boldsymbol\{K\}^\{\\dagger\}\\in\\mathbb\{R\}^\{m\\times n\}is the pseudoinverse of𝐊\\boldsymbol\{K\}\.

### 3\.2Identification of an area whereL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}decreases

We now consider the set𝒱\{\\cal V\}of eigenvalues of𝑯\\boldsymbol\{H\}:λ1≥λ2≥⋯λn≥0\\lambda\_\{1\}\\geq\\lambda\_\{2\}\\geq\\cdots\\lambda\_\{n\}\\geq 0\. This set typically breaks down into a set𝒱1=\{λ1,…,λα\}\{\\cal\{V\}\}\_\{1\}=\\\{\\lambda\_\{1\},\\dots,\\lambda\_\{\\alpha\}\\\}made of a small number of large values, and the remaining set𝒱2=\{λα\+1,…,λn\}\{\\cal\{V\}\}\_\{2\}=\\\{\\lambda\_\{\\alpha\+1\},\\dots,\\lambda\_\{n\}\\\}made of a “bulk” of low values\. See, e\.g\.,[Oymak et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib22);[Ghorbani et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib10);[Advani et al\. \(2020\)](https://arxiv.org/html/2608.24210#bib.bib24);[Murray et al\. \(2023\)](https://arxiv.org/html/2608.24210#bib.bib19)\.

Let𝑷∈ℝn×n\\boldsymbol\{P\}\\in\\mathbb\{R\}^\{n\\times n\}the transition matrix satisfying

𝑯=𝑷​diag​\(λ1,…,λn\)​𝑷⊤\\boldsymbol\{H\}=\\boldsymbol\{P\}\\text\{diag\}\(\\lambda\_\{1\},\\dots,\\lambda\_\{n\}\)\\boldsymbol\{P\}^\{\\top\}and𝑷i\\boldsymbol\{P\}\_\{i\}theii\-th column of𝑷\\boldsymbol\{P\}\. Let𝒖0=𝑷⊤​𝒗​\(0\)∈ℝn\\boldsymbol\{u\}\_\{0\}=\\boldsymbol\{P\}^\{\\top\}\\boldsymbol\{v\}\(0\)\\in\\mathbb\{R\}^\{n\}andUi∈ℝU\_\{i\}\\in\\mathbb\{R\}be theii\-th component of𝒖0\\boldsymbol\{u\}\_\{0\}\. We have:

𝒗⁡\(t\)=∑i=1n𝒘i​e−λi​t\\boldsymbol\{v\}\(t\)=\\sum\_\{i=1\}^\{n\}\\boldsymbol\{w\}\_\{i\}e^\{\-\\lambda\_\{i\}t\}\(16\)with𝒘i=Ui​𝑷i\\boldsymbol\{w\}\_\{i\}=U\_\{i\}\\boldsymbol\{P\}\_\{i\}\. Let

Γi​\(t\)=\\displaystyle\\Gamma\_\{i\}\(t\)=\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒘i\(i∈\[α\]\),\\displaystyle\\;\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{w\}\_\{i\}\\ \\ \(i\\in\[\\alpha\]\),Γ⁡\(t\)=\\displaystyle\\Gamma\(t\)=∑i∈\[α\]Γi​\(t\),\\displaystyle\\;\\sum\_\{i\\in\[\\alpha\]\}\\Gamma\_\{i\}\(t\),Δ⁡\(t\)=\\displaystyle\\Delta\(t\)=\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​∑j=α\+1n𝒘j​e−λj​t,\\displaystyle\\;\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\sum\_\{j=\\alpha\+1\}^\{n\}\\boldsymbol\{w\}\_\{j\}e^\{\-\\lambda\_\{j\}t\},Ω⁡\(t\)=\\displaystyle\\Omega\(t\)=n​Ψ​\(t\)−Δ⁡\(t\)\.\\displaystyle\\;n\\Psi\(t\)\-\\Delta\(t\)\.
We decompose the time space into contiguous time intervals𝒯1\{\\cal T\}\_\{1\},𝒯2,…\{\\cal T\}\_\{2\},\\dotsover which the signs of eachΓi\\Gamma\_\{i\}\(i∈\[α\]i\\in\[\\alpha\]\) andΩ\\Omegaare invariant \(“sign\-invariance”\)\. So over each time interval𝒯\{\\cal T\}and eachi∈\[α\]i\\in\[\\alpha\], we have:

- •Γi​\(t\)\>0​∀t∈𝒯\\Gamma\_\{i\}\(t\)\>0\\ \\forall t\\in\{\\cal T\}orΓi​\(t\)≤0​∀t∈𝒯\\Gamma\_\{i\}\(t\)\\leq 0\\ \\forall t\\in\{\\cal T\}, and
- •Ω⁡\(t\)\>0​∀t∈𝒯\\Omega\(t\)\>0\\ \\forall t\\in\{\\cal T\}orΩ⁡\(t\)≤0​∀t∈𝒯\\Omega\(t\)\\leq 0\\ \\forall t\\in\{\\cal T\}\.

We define

I\+​\(𝒯\)=\\displaystyle I\_\{\+\}\(\{\\cal T\}\)=\{i∈\[α\]:Γi​\(t\)\>0​∀t∈𝒯\}\\displaystyle\\;\\\{i\\in\[\\alpha\]:\\Gamma\_\{i\}\(t\)\>0\\ \\forall t\\in\{\\cal T\}\\\}I−​\(𝒯\)=\\displaystyle I\_\{\-\}\(\{\\cal T\}\)=\{i∈\[α\]:Γi​\(t\)≤0​∀t∈𝒯\}\.\\displaystyle\\;\\\{i\\in\[\\alpha\]:\\Gamma\_\{i\}\(t\)\\leq 0\\ \\forall t\\in\{\\cal T\}\\\}\.Let

Φ⁡\(t\)=∑i∈\[α\]Γi​\(t\)​e−λi​t−Ω⁡\(t\)\.\\Phi\(t\)=\\sum\_\{i\\in\[\\alpha\]\}\\Gamma\_\{i\}\(t\)e^\{\-\\lambda\_\{i\}t\}\-\\Omega\(t\)\.\(17\)Note that the only time\-varying terms ofΦ⁡\(t\)\\Phi\(t\)are𝒗⁡\(t\)\\boldsymbol\{v\}\(t\)ande−λi​te^\{\-\\lambda\_\{i\}t\}\(i∈\[n\]i\\in\[n\]\)\. Since𝒗⁡\(t\)\\boldsymbol\{v\}\(t\)is itself a linear combination ofe−λi​te^\{\-\\lambda\_\{i\}t\}\(see \([16](https://arxiv.org/html/2608.24210#S3.E16)\)\), the time\-varying terms ofΦ⁡\(t\)\\Phi\(t\)are just \(products of\)e−λi​te^\{\-\\lambda\_\{i\}t\}\. Let us consider interval𝒯1\{\\cal T\}\_\{1\}\(assumed to be of the form\[0,τ1\)\[0,\\tau\_\{1\}\), and consider the timet\+t^\{\+\}defined, usingΦ⁡\(t\)\\Phi\(t\), as:

t\+=sups∈𝒯1\{s:Φ\(t\)\>0,∀t∈\[0,s\)\}\.t^\{\+\}=\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:\\Phi\(t\)\>0,\\ \\forall t\\in\[0,s\)\\right\\\}\.\(18\)We supposeΦ⁡\(0\)\>0\\Phi\(0\)\>0\. \(Otherwise,t∗=0t^\{\*\}=0and our method fails\.\) We will show thatt\+t^\{\+\}is such that:d​L𝒟∗​\(s\)d​t<0\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(s\)\}\{dt\}<0for alls∈\[0,t\+\)s\\in\[0,t^\{\+\}\)\(see \([20](https://arxiv.org/html/2608.24210#S3.E20)\)\)\. Hence,L𝒟∗L\_\{\{\\cal D\}\}^\{\*\}is decreasing on\[0,t\+\)\[0,t^\{\+\}\)\. We can thus taket\+t^\{\+\}as a low estimate oft∗t^\{\*\}\(which is the first timesssuch thatd​L𝒟​\(s\)/d​t≥0dL\_\{\{\\cal D\}\}\(s\)/dt\\geq 0\)\. We havet\+≤t∗t^\{\+\}\\leq t^\{\*\}and, under certain conditions:t\+=t∗t^\{\+\}=t^\{\*\}\(case 1 of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\)\. Formally:

###### Proposition 7\.

We have:

d​L𝒟∗​\(t\)d​t​<0iffΦ⁡\(t\)\>​0,\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(t\)\}\{dt\}<0\\ \\ \\ \\ \\text\{ iff \}\\ \\ \\Phi\(t\)\>0,\\\(19\)t\+=\\displaystyle t^\{\+\}=sups∈𝒯1\{s:d​L𝒟∗​\(s\)d​t<0​∀t∈\[0,s\)\},\\displaystyle\\;\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(s\)\}\{dt\}<0\\,\\ \\forall t\\in\[0,s\)\\right\\\},\(20\)t\+≤\\displaystyle t^\{\+\}\\leqt∗\.\\displaystyle\\;t^\{\*\}\.\(21\)There are two cases:

- •ift∗∈𝒯1t^\{\*\}\\in\{\\cal T\}\_\{1\}\(case 1\), we have t\+=t∗<\\displaystyle t^\{\+\}=t^\{\*\}<τ1\.\\displaystyle\\;\\tau\_\{1\}\.\(22\)
- •ift∗∉𝒯1t^\{\*\}\\not\\in\{\\cal T\}\_\{1\}\(case 2\), we have t\+=τ1≤\\displaystyle t^\{\+\}=\\tau\_\{1\}\\leqt∗\.\\displaystyle\\;t^\{\*\}\.\(23\)

Suppose furthermore:

I−​\(𝒯1\)=∅​andΓ⁡\(0\)\>Ω⁡\(0\)\>0\.I\_\{\-\}\(\{\\cal T\}\_\{1\}\)=\\emptyset\\ \\mbox\{ and \}\\ \\ \\Gamma\(0\)\>\\Omega\(0\)\>0\.\(24\)Then:

t1\+≤t\+≤\\displaystyle t\_\{1\}^\{\+\}\\leq t^\{\+\}\\leqtα\+\\displaystyle\\;t\_\{\\alpha\}^\{\+\}\(25\)where, fori∈\{1,α\}i\\in\\\{1,\\alpha\\\},ti\+t\_\{i\}^\{\+\}is defined as:

ti\+\\displaystyle t\_\{i\}^\{\+\}=sups∈𝒯1\{s:t<1λilnΓ⁡\(t\)Ω⁡\(t\)∀t∈\[0,s\)\}\.\\displaystyle=\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:t<\\frac\{1\}\{\\lambda\_\{i\}\}\\ln\\frac\{\\Gamma\(t\)\}\{\\Omega\(t\)\}\\ \\ \\forall t\\in\[0,s\)\\right\\\}\.\(26\)

### 3\.3Beneficial early stopping

The first local minimum reached byL𝒟∗​\(t\)L\_\{\{\\cal D\}\}^\{\*\}\(t\)att=t∗t=t^\{\*\}corresponds to the minimum of the “U\-shaped” phase ofL𝒟∗=LS\+LG∗L\_\{\{\\cal D\}\}^\{\*\}=L\_\{S\}\+L\_\{G\}^\{\*\}\(whereLSL\_\{S\}decreases whileLG∗L\_\{G\}^\{\*\}increases\)\. Later, the curveL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}can fall again, and pass through other local minima\. This is related to the phenomenon of “epoch\-wise double descent” \(see, e\.g\.,[Heckel and Yilmaz \(2020\)](https://arxiv.org/html/2608.24210#bib.bib18);[Stephenson and Lee \(2021\)](https://arxiv.org/html/2608.24210#bib.bib16);[Nakkiran et al\. \(2021\)](https://arxiv.org/html/2608.24210#bib.bib15)\)\. It is thus interesting to compare the value ofL𝒟∗​\(t\)L\_\{\{\\cal D\}\}^\{\*\}\(t\)att=t∗t=t^\{\*\}or att=t\+t=t^\{\+\}\(as given by \([18](https://arxiv.org/html/2608.24210#S3.E18)\)\) withL𝒟∗​\(∞\)L\_\{\{\\cal D\}\}^\{\*\}\(\\infty\)\. IfL𝒟∗​\(t\+\)<L𝒟∗​\(∞\)L\_\{\{\\cal D\}\}^\{\*\}\(t^\{\+\}\)<L\_\{\{\\cal D\}\}^\{\*\}\(\\infty\), the early stopping strategy is said to be “beneficial”; otherwise, a phenomenon of “benign overfitting” may occur, and it may be interesting to continue the training aftert\+t^\{\+\}\(see, e\.g\.,[Bartlett et al\. \(2020\)](https://arxiv.org/html/2608.24210#bib.bib13);[Sonthalia et al\. \(2024\)](https://arxiv.org/html/2608.24210#bib.bib14)\)\. We now explain how to obtain an analytic form forL𝒟∗​\(∞\)L\_\{\{\\cal D\}\}^\{\*\}\(\\infty\)\(see \([29](https://arxiv.org/html/2608.24210#S3.E29)\)\)\. Sincea⁡\(t\)a\(t\)has been initialized to 0, we can show using \([3](https://arxiv.org/html/2608.24210#S2.E3)\) that att=∞t=\\infty\(cf\.[Björck and Golub \(1973\)](https://arxiv.org/html/2608.24210#bib.bib3)\):

𝒂⁡\(∞\)=\\displaystyle\\boldsymbol\{a\}\(\\infty\)=𝑲†​𝒚,\\displaystyle\\;\\boldsymbol\{K\}^\{\\dagger\}\\boldsymbol\{y\},𝒗⁡\(∞\)=\\displaystyle\\boldsymbol\{v\}\(\\infty\)=\(𝑲​𝑲†−𝑰\)​𝒚\.\\displaystyle\\;\(\\boldsymbol\{K\}\\boldsymbol\{K\}^\{\\dagger\}\-\\boldsymbol\{I\}\)\\boldsymbol\{y\}\.Att=∞t=\\infty, Equation \([7](https://arxiv.org/html/2608.24210#S2.E7)\) gives:

L𝒟∗​\(∞\)=\\displaystyle L\_\{\{\\cal D\}\}^\{\*\}\(\\infty\)=1n​‖𝒗⁡\(∞\)‖1\+2​C​‖𝒂⁡\(∞\)‖2n,\\displaystyle\\;\\frac\{1\}\{n\}\\\|\\boldsymbol\{v\}\(\\infty\)\\\|\_\{1\}\+\\frac\{2C\\\|\\boldsymbol\{a\}\(\\infty\)\\\|\_\{2\}\}\{\\sqrt\{n\}\},\(28\)hence:L𝒟∗​\(∞\)=\\displaystyle L\_\{\{\\cal D\}\}^\{\*\}\(\\infty\)=1n​‖\(𝑲​𝑲†−𝑰\)​𝒚‖1\+2​C​‖𝑲†​𝒚‖2n\.\\displaystyle\\\>\\frac\{1\}\{n\}\\\|\(\\boldsymbol\{K\}\\boldsymbol\{K^\{\\dagger\}\}\-\\boldsymbol\{I\}\)\\boldsymbol\{y\}\\\|\_\{1\}\+\\frac\{2C\\\|\\boldsymbol\{K^\{\\dagger\}\}\\boldsymbol\{y\}\\\|\_\{2\}\}\{\\sqrt\{n\}\}\.\(29\)Note that the epochwise double descent typically happens in the overparametrization case \(i\.e\.,m≫nm\\gg n\)\.

## 4Examples

We consider a training set of the formS=\{𝒉1,…,𝒉n\}S=\\\{\\boldsymbol\{h\}\_\{1\},\\dots,\\boldsymbol\{h\}\_\{n\}\\\}and a separate test set of the formSt​e​s​t=\{𝒈1,…,𝒈nt​e​s​t\}S\_\{test\}=\\\{\\boldsymbol\{g\}\_\{1\},\\dots,\\boldsymbol\{g\}\_\{n\_\{test\}\}\\\}\. Forp∈\{1,2\}p\\in\\\{1,2\\\}, let

L𝒟∗Lp=\\displaystyle L\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}=1n​∑i=1n‖𝒉i‖pp\+2​p​Mp−1​C​‖𝒂‖2n,\\displaystyle\\;\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\\|\\boldsymbol\{h\}\_\{i\}\\\|\_\{p\}^\{p\}\+2pM^\{p\-1\}\\frac\{C\\\|\\boldsymbol\{a\}\\\|\_\{2\}\}\{\\sqrt\{n\}\},Lt​e​s​tLp=\\displaystyle L\_\{test\}^\{L\_\{p\}\}=1n​∑i=1nt​e​s​t‖𝒈i‖pp\.\\displaystyle\\;\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\_\{test\}\}\\\|\\boldsymbol\{g\}\_\{i\}\\\|\_\{p\}^\{p\}\.Lettt​e​s​tLp=argmint​Lt​e​s​tLpt\_\{test\}^\{L\_\{p\}\}=\\text\{argmin\}\_\{t\}\\ L\_\{test\}^\{L\_\{p\}\}andtLp∗=argmint​L𝒟∗Lpt^\{\*\}\_\{L\_\{p\}\}=\\text\{argmin\}\_\{t\}\\ L\_\{\{\{\\cal D\}^\{\*\}\}\}^\{L\_\{p\}\}\. \(We haveL𝒟∗≡L𝒟∗L1L\_\{\{\\cal D\}\}^\{\*\}\\equiv L\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{1\}\},Lt​e​s​t≡Lt​e​s​tL1L\_\{test\}\\equiv L\_\{test\}^\{L\_\{1\}\},t∗≡tL1∗t^\{\*\}\\equiv t^\{\*\}\_\{L\_\{1\}\},tt​e​s​t≡tt​e​s​tL1t\_\{test\}\\equiv t\_\{test\}^\{L\_\{1\}\}\.\) The valuett​e​s​tLpt\_\{test\}^\{L\_\{p\}\}represents a good estimate of theoptimalearly stopping time with respect toLpL\_\{p\}\-norm\. On the other hand, it follows from Equation \([8](https://arxiv.org/html/2608.24210#S2.E8)\) \(and its counter part forL2L\_\{2\}\-norm\) that, with probability at least1−δ1\-\\delta:

Lt​e​s​tLp≤L𝒟∗Lp\+ϵp\.L\_\{test\}^\{L\_\{p\}\}\\leq L\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\+\\epsilon\_\{p\}\.\(30\)This is verified on the subsequent examples\. We also check thatt∗≡tL1∗t^\{\*\}\\equiv t^\{\*\}\_\{L\_\{1\}\}is closer thantL2∗t^\{\*\}\_\{L\_\{2\}\}tott​e​s​tLpt\_\{test\}^\{L\_\{p\}\}for bothp=1p=1andp=2p=2\. This confirms that our RC\-based method works better withL1L\_\{1\}\-norm than withL2L\_\{2\}\-norm\. We focus on the casem≤nm\\leq n\(underparameterization\)\. Form\>nm\>n, we haveL𝒟∗​\(t\)\>0L\_\{\{\\cal D\}\}^\{\*\}\(t\)\>0almost immediately forttequal or close to00, and our method fails \(t∗≈0t^\{\*\}\\approx 0\)\. In the examples, we check that our stopping rule is beneficial \(i\.e\.,L𝒟∗​\(t\+\)<L𝒟∗​\(∞\)L\_\{\{\\cal D\}\}^\{\*\}\(t^\{\+\}\)<L\_\{\{\\cal D\}\}^\{\*\}\(\\infty\)\)\. As shown in Example[13](https://arxiv.org/html/2608.24210#Thmtheorem13), the method applies equally to various kinds of data distribution \(Gaussian, uniform, Pareto\)\. We observe that it works better \(i\.e\.,t∗t^\{\*\}closer tott​e​s​tt\_\{test\}\) when the ration/mn/mis larger\. The numerical results are obtained using GD with a step size of10−610^\{\-6\}\. For the sake of clarity, the values of timest1\+,tα\+,t\+,…t\_\{1\}^\{\+\},t\_\{\\alpha\}^\{\+\},t^\{\+\},\\dotsare expressed as the number of GD steps\.

###### Example 10\.

\(Gaussian distribution\) We consider the example of[Liao and Couillet \(2018\)](https://arxiv.org/html/2608.24210#bib.bib1)in a binary classification setup\. The input vector𝐡1,…,𝐡n∈ℝm\\boldsymbol\{h\}\_\{1\},\.\.\.,\\boldsymbol\{h\}\_\{n\}\\in\\mathbb\{R\}^\{m\}withm=256m=256are sampled from two distribution classes𝒞1\{\\cal C\}\_\{1\}and𝒞2\{\\cal C\}\_\{2\}\. A vector𝐡i\\boldsymbol\{h\}\_\{i\}belongs to the class𝒞j\{\\cal C\}\_\{j\}satisfies:

𝒉i=\(−1\)j​𝝁\+𝒛i\\boldsymbol\{h\}\_\{i\}=\(\-1\)^\{j\}\\boldsymbol\{\\mu\}\+\\boldsymbol\{z\}\_\{i\}\(31\)forj=\{1,2\}j=\\\{1,2\\\},𝛍=\[2;𝟎m−1\]\\boldsymbol\{\\mu\}=\\left\[2;\\boldsymbol\{0\}\_\{m\-1\}\\right\], and the noise vector𝐳i∼𝒩⁡\(𝟎m,𝐈m\)\\boldsymbol\{z\}\_\{i\}\\sim\{\\cal\{N\}\}\(\\boldsymbol\{0\}\_\{m\},\\boldsymbol\{I\}\_\{m\}\)\. To distinguish the two distribution, an input vector𝐡i\\boldsymbol\{h\}\_\{i\}is labeled byyi=1y\_\{i\}=1if𝐡i\\boldsymbol\{h\}\_\{i\}is in classes𝒞1\{\\cal C\}\_\{1\}and byyi=−1y\_\{i\}=\-1if𝐡i\\boldsymbol\{h\}\_\{i\}is in classes𝒞2\{\\cal C\}\_\{2\}\. We first consider a training set ofn=512n=512samples, with216216samples from𝒞1\{\\cal C\}\_\{1\}and the remaining216216from𝒞2\{\\cal C\}\_\{2\}\. We calculate the eigenvalues of the corresponding transition matrix𝐇\\boldsymbol\{H\}\. We have𝒱1:\{λ1=λα=2865\}\\mathcal\{V\}\_\{1\}:\\\{\\lambda\_\{1\}=\\lambda\_\{\\alpha\}=2865\\\}and𝒱2:\{λ2=1475,…,λn\}\\mathcal\{V\}\_\{2\}:\\\{\\lambda\_\{2\}=1475,\\dots,\\lambda\_\{n\}\\\}\. We see thatd​L𝒟∗​\(t\)/d​tdL\_\{\{\\cal D\}\}^\{\*\}\(t\)/dt\(computed using \([14](https://arxiv.org/html/2608.24210#S3.E14)\)\) becomes 0 att=t∗=27<τ1=100t=t^\{\*\}=27<\\tau\_\{1\}=100\. This corresponds to case 1 of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\(t∗∈𝒯1=\[0,τ1\)t^\{\*\}\\in\{\\cal T\}\_\{1\}=\[0,\\tau\_\{1\}\)\)\. The estimatet\+=27t^\{\+\}=27\(computed using \([18](https://arxiv.org/html/2608.24210#S3.E18)\)\) satisfiest\+=t∗=27t^\{\+\}=t^\{\*\}=27in accordance with Equation \([22](https://arxiv.org/html/2608.24210#S3.E22)\)\. See Figure[1](https://arxiv.org/html/2608.24210#S4.F1)\. We see in Figure[2](https://arxiv.org/html/2608.24210#S4.F2)thatt∗≡tL1∗t^\{\*\}\\equiv t\_\{L\_\{1\}\}^\{\*\}is closer thantL2∗t\_\{L\_\{2\}\}^\{\*\}tott​e​s​tLpt^\{L\_\{p\}\}\_\{test\}for bothp=1p=1andp=2p=2\.

Similarly, forn=16384n=16384andm=256m=256, the derivatived​L𝒟∗​\(t\)/d​tdL\_\{\{\\cal D\}\}^\{\*\}\(t\)/dtbecomes 0 att=t∗=64<τ1=100t=t^\{\*\}=64<\\tau\_\{1\}=100\. This corresponds again to case 1 of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\. The estimatet\+t^\{\+\}satisfiest\+=t∗=64<τ1t^\{\+\}=t^\{\*\}=64<\\tau\_\{1\}, in accordance with Equation \([22](https://arxiv.org/html/2608.24210#S3.E22)\)\. See Figure[3](https://arxiv.org/html/2608.24210#S4.F3)\. Likewise in Figure[4](https://arxiv.org/html/2608.24210#S4.F4), we see thatt∗≡tL1∗t^\{\*\}\\equiv t^\{\*\}\_\{L\_\{1\}\}is closer thantL2∗t^\{\*\}\_\{L\_\{2\}\}tott​e​s​tLpt\_\{test\}^\{L\_\{p\}\}for bothp=1p=1andp=2p=2\.

Figure 1:Top: the sign\-invariant subinterval𝒯1=\[0,100\)\{\\cal T\}\_\{1\}=\\left\[0,100\\right\)with curvesΓ1,Ω\\Gamma\_\{1\},\\Omega\. Middle: curved​L𝒟∗/d​tdL\_\{\{\\cal D\}\}^\{\*\}/dt\. Bottom: curveΦ⁡\(t\)\\Phi\(t\)\. We havet\+=t∗=27<τ1=100t^\{\+\}=t^\{\*\}=27<\\tau\_\{1\}=100\(case 1 of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\)\.Figure 2:CurvesL𝒟∗LpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}andLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}for Gaussian distribution \(m=256m=256,n=512n=512\)\. We check thatLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}is belowL𝒟∗LpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\(p=1,2p=1,2\), andtL1∗t^\{\*\}\_\{L\_\{1\}\}closer thantL2∗t^\{\*\}\_\{L\_\{2\}\}tott​e​s​tLpt\_\{test\}^\{L\_\{p\}\}forp=1p=1and22\.Figure 3:Top: the sign\-invariant subinterval𝒯1=\[0,100\)\{\\cal T\}\_\{1\}=\\left\[0,100\\right\)with curvesΓ1,Ω\\Gamma\_\{1\},\\Omega\. Middle: curved​L𝒟∗/d​tdL\_\{\{\\cal D\}\}^\{\*\}/dt\. Bottom: curveΦ⁡\(t\)\\Phi\(t\)\. We havet\+=t∗=64<τ1t^\{\+\}=t^\{\*\}=64<\\tau\_\{1\}\(case 1\)\.Figure 4:CurvesL𝒟∗LpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}andLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}for Gaussian distribution \(m=256m=256,n=16384n=16384\)\. We check thatLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}is belowL𝒟∗LpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\(p=1,2p=1,2\), andtL1∗t^\{\*\}\_\{L\_\{1\}\}closer thantL2∗t^\{\*\}\_\{L\_\{2\}\}tott​e​s​tLpt\_\{test\}^\{L\_\{p\}\}forp=1p=1and22\.
###### Example 11\.

\(MNIST classification 3\-5\) We consider a problem of binary classification between classes 3 and 5 of the MNIST dataset[LeCun \(1998\)](https://arxiv.org/html/2608.24210#bib.bib7)\. We consider an NN with ReLU activation function, 4 hidden layers of widthm=10m=10and output layer𝐚∈ℝm\\boldsymbol\{a\}\\in\\mathbb\{R\}^\{m\}\. Using Pytorch[Paszke et al\. \(2019\)](https://arxiv.org/html/2608.24210#bib.bib17), we select a training setSSofn=104n=10^\{4\}samples of the form\(𝐱i,yi\)\(\\boldsymbol\{x\}\_\{i\},y\_\{i\}\)\(with𝐱i∈ℝ784\\boldsymbol\{x\}\_\{i\}\\in\\mathbb\{R\}^\{784\}\) and a test setSt​e​s​tS\_\{test\}of19001900samples\. Following the linear probing method \(see[Alain and Bengio \(2016\)](https://arxiv.org/html/2608.24210#bib.bib5)\), we reduce the NN to a linear model as follows\. We first freeze each weightaia\_\{i\}\(i∈\[m\]i\\in\[m\]\) of𝐚\\boldsymbol\{a\}arbitrarily to either11or−1\-1, and pre\-train the resulting modelN​NNNusing GD \(with a step sizeη=10−6\\eta=10^\{\-6\}\)\. The weights of the hidden layers are then frozen themselves, which yieldsnnfixed vectors of the form𝐡1=N​N​\(𝐱1\),…,𝐡n=N​N​\(𝐱n\)∈ℝm\\boldsymbol\{h\}\_\{1\}=NN\(\\boldsymbol\{x\}\_\{1\}\),\\dots,\\boldsymbol\{h\}\_\{n\}=NN\(\\boldsymbol\{x\}\_\{n\}\)\\in\\mathbb\{R\}^\{m\}\. We then regard𝐡1,…,𝐡n\\boldsymbol\{h\}\_\{1\},\\dots,\\boldsymbol\{h\}\_\{n\}as input vectors, and consider the linear model consisting only of the output layer𝐚\\boldsymbol\{a\}reinitialized to 0\. We calculate the eigenvalues of the corresponding transition matrix𝐇\\boldsymbol\{H\}\. We have𝒱1:\{λ1=12919,λ2≡λα=11384\{\\cal\{V\}\}\_\{1\}:\\\{\\lambda\_\{1\}=12919,\\lambda\_\{2\}\\equiv\\lambda\_\{\\alpha\}=11384\}, and𝒱2:\{λ3=48,…,λn\}\{\\cal\{V\}\}\_\{2\}:\\\{\\lambda\_\{3\}=48,\\dots,\\lambda\_\{n\}\\\}\.

Figure 5:Top: sign\-invariant subintervals𝒯1=\[0,343\)\{\\cal T\}\_\{1\}=\\left\[0,343\\right\),𝒯2=\[343,407\)\{\\cal T\}\_\{2\}=\\left\[343,407\\right\),𝒯3=\[407,600\)\{\\cal T\}\_\{3\}=\\left\[407,600\\right\)with curvesΓ1,Γ2,Ω\\Gamma\_\{1\},\\Gamma\_\{2\},\\Omega\. Middle: curved​L𝒟∗/d​tdL\_\{\{\\cal D\}\}^\{\*\}/dt\. Bottom: curveΦ⁡\(t\)\\Phi\(t\)\. We havet\+=342≈τ1=343≤t∗=357t^\{\+\}=342\\approx\\tau\_\{1\}=343\\leq t^\{\*\}=357\(case 2 of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\)\.We consider the time intervalT=\[0,600\)T=\\left\[0,600\\right\), which divides it into three sign\-invariant subintervals:𝒯1=\[0,343\)\{\\cal T\}\_\{1\}=\\left\[0,343\\right\)\(in red, the top plot of Figure[5](https://arxiv.org/html/2608.24210#S4.F5)\),𝒯2=\[343,407\)\{\\cal T\}\_\{2\}=\\left\[343,407\\right\)\(in blue\), and𝒯3=\[407,600\)\{\\cal T\}\_\{3\}=\\left\[407,600\\right\)\(in green\)\. We findC=1\.572C=1\.572,M=2M=2andϵ=0\.0814\\epsilon=0\.0814\. Using \([14](https://arxiv.org/html/2608.24210#S3.E14)\), we findd​L𝒟∗​\(t\)/d​t≥0dL\_\{\{\\cal D\}\}^\{\*\}\(t\)/dt\\geq 0for the first time att∗=357t^\{\*\}=357\(see the middle plot of Figure[5](https://arxiv.org/html/2608.24210#S4.F5)\)\. On the other hand, we have:I−=∅I\_\{\-\}=\\emptysetandΓ⁡\(0\)\>Ω⁡\(0\)\>0\\Gamma\(0\)\>\\Omega\(0\)\>0on𝒯1\{\\cal T\}\_\{1\}, so \([24](https://arxiv.org/html/2608.24210#S3.E24)\) is satisfied\. We findt\+=342t^\{\+\}=342using Equation \([18](https://arxiv.org/html/2608.24210#S3.E18)\) \(see the bottom plot of Figure[5](https://arxiv.org/html/2608.24210#S4.F5)\), and verifyt1\+=342≤t\+≤tα\+=342t^\{\+\}\_\{1\}=342\\leq t^\{\+\}\\leq t^\{\+\}\_\{\\alpha\}=342in accordance with Equation \([25](https://arxiv.org/html/2608.24210#S3.E25)\)\. Sincet∗=357\>τ1=343t^\{\*\}=357\>\\tau\_\{1\}=343, we havet∗∉𝒯1t^\{\*\}\\not\\in\{\\cal T\}\_\{1\}\(case 2 of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\)\. Sot\+=342≈τ1=343≤t∗=357t^\{\+\}=342\\approx\\tau\_\{1\}=343\\leq t^\{\*\}=357, in accordance with \([23](https://arxiv.org/html/2608.24210#S3.E23)\)\.

Figure 6:CurvesL𝒟∗Lp\+ϵpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\+\\epsilon\_\{p\}andLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}for classification 3\-5\. We check thatLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}is belowL𝒟∗Lp\+ϵpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\+\\epsilon\_\{p\}\(p=1,2p=1,2\), andtL1∗t^\{\*\}\_\{L\_\{1\}\}closer thantL2∗t^\{\*\}\_\{L\_\{2\}\}tott​e​s​tLpt\_\{test\}^\{L\_\{p\}\}forp=1p=1andp=2p=2\.Figure[6](https://arxiv.org/html/2608.24210#S4.F6)shows that, forδ=0\.05\\delta=0\.05, curveLt​e​s​tL\_\{test\}is always belowL𝒟∗\+ϵL\_\{\{\\cal D\}\}^\{\*\}\+\\epsilon, in accordance with equation \([30](https://arxiv.org/html/2608.24210#S4.E30)\)\. We see thattt​e​s​t=356t\_\{test\}=356almost coincides witht∗=357t^\{\*\}=357\. So curvesLt​e​s​tL\_\{test\}andL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}reach their minima at almost the same time\. The values ofLt​e​s​t​\(t\)L\_\{test\}\(t\)att=t\+,t∗,tt​e​s​tt=t^\{\+\},t^\{\*\},t\_\{test\}are identical \(equal to0\.11660\.1166\), and almost equal to its value att=tapprox\+t=t\_\{\\text\{approx\}\}^\{\+\}\(equal to0\.11670\.1167\)\. See Table 1\. This shows an excellent agreement between analytic and numerical estimates of the optimal stopping time\. Besides, we have:L𝒟∗​\(t\+\)\+ϵ=0\.2138<L𝒟∗​\(∞\)\+ϵ=0\.2274L\_\{\\cal\{D\}\}^\{\*\}\(t^\{\+\}\)\+\\epsilon=0\.2138<L\_\{\\cal\{D\}\}^\{\*\}\(\\infty\)\+\\epsilon=0\.2274, which shows that early stopping is beneficial\.

Table 1:The values oft1\+t^\{\+\}\_\{1\},tα\+t^\{\+\}\_\{\\alpha\}\(α=2\\alpha=2\),t\+t^\{\+\},…\\dots,tt​e​s​tt\_\{test\}with correspondingLt​e​s​t​\(t\)L\_\{test\}\(t\)andL𝒟∗​\(t\)\+ϵL\_\{\\cal\{D\}\}^\{\*\}\(t\)\+\\epsilon\(Example[11](https://arxiv.org/html/2608.24210#Thmtheorem11)\)\.
###### Example 12\.

\(MNIST classification 0\-1\) We use the same framework as in Example[11](https://arxiv.org/html/2608.24210#Thmtheorem11)with a 4\-hidden\-layer NN of widthm=10m=10, and a training setSSofn=104n=10^\{4\}examples, corresponding to classes 0 and 1 \(instead of 3 and 5\)\. We calculate the eigenvalues of the corresponding transition matrix𝐇\\boldsymbol\{H\}\. We have𝒱1:\{λ1=11432,λ2≡λα=10639\{\\cal\{V\}\}\_\{1\}:\\\{\\lambda\_\{1\}=11432,\\lambda\_\{2\}\\equiv\\lambda\_\{\\alpha\}=10639\}and𝒱2\{\\cal\{V\}\}\_\{2\}:\{λ3=0\.005,…,λn\}\\\{\\lambda\_\{3\}=0\.005,\\dots,\\lambda\_\{n\}\\\}\.

Figure 7:Top: sign\-invariant subintervals𝒯1=\[0,421\)\{\\cal T\}\_\{1\}=\\left\[0,421\\right\),𝒯2=\[421,493\)\{\\cal T\}\_\{2\}=\\left\[421,493\\right\),𝒯3=\[493,600\)\{\\cal T\}\_\{3\}=\\left\[493,600\\right\)with curvesΓ1,Γ2,Ω\\Gamma\_\{1\},\\Gamma\_\{2\},\\Omega\. Middle: curved​L𝒟∗/d​tdL\_\{\{\\cal D\}\}^\{\*\}/dt\. Bottom: curveΦ⁡\(t\)\\Phi\(t\)\. We havet\+=t∗=415<τ1=421t^\{\+\}=t^\{\*\}=415<\\tau\_\{1\}=421\(case 1\)\.The time intervalT=\[0,600\)T=\\left\[0,600\\right\)divides into sign\-invariant subintervals𝒯1=\[0,421\)\{\\cal T\}\_\{1\}=\\left\[0,421\\right\)\(in red, the top plot of Figure[7](https://arxiv.org/html/2608.24210#S4.F7)\),𝒯2=\[421,493\)\{\\cal T\}\_\{2\}=\\left\[421,493\\right\)\(in blue\), and𝒯3=\[493,600\)\{\\cal T\}\_\{3\}=\\left\[493,600\\right\)\(in green\)\. We findC=1\.495C=1\.495,M=2M=2andϵ=0\.0814\\epsilon=0\.0814\. Using \([14](https://arxiv.org/html/2608.24210#S3.E14)\), we findd​L𝒟∗/d​t=0dL\_\{\{\\cal D\}\}^\{\*\}/dt=0att∗=415t^\{\*\}=415\(see the middle plot Figure[7](https://arxiv.org/html/2608.24210#S4.F7)\)\. Here again, \([24](https://arxiv.org/html/2608.24210#S3.E24)\) holds on𝒯1\{\\cal T\}\_\{1\}\. We findt\+=415t^\{\+\}=415\(using Equation \([18](https://arxiv.org/html/2608.24210#S3.E18)\)\), and verifyt1\+=414≤t\+≤tα\+≡t2\+=417t\_\{1\}^\{\+\}=414\\leq t^\{\+\}\\leq t\_\{\\alpha\}^\{\+\}\\equiv t\_\{2\}^\{\+\}=417, in accordance with Equation \([25](https://arxiv.org/html/2608.24210#S3.E25)\)\. See the bottom plot of Figure[7](https://arxiv.org/html/2608.24210#S4.F7)\. Sincet∗=415<τ1=421t^\{\*\}=415<\\tau\_\{1\}=421, we havet∗∈𝒯1t^\{\*\}\\in\{\\cal T\}\_\{1\}\(case 1 of Prop\.[7](https://arxiv.org/html/2608.24210#Thmtheorem7)\)\. We have:t\+=t∗=415<τ1=421t^\{\+\}=t^\{\*\}=415<\\tau\_\{1\}=421, in accordance with \([22](https://arxiv.org/html/2608.24210#S3.E22)\)\.

Figure 8:CurvesL𝒟∗Lp\+ϵpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\+\\epsilon\_\{p\}andLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}for classification 0\-1\. We check thatLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}is belowL𝒟∗Lp\+ϵpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\+\\epsilon\_\{p\}\(p=1,2p=1,2\), andtL1∗t^\{\*\}\_\{L\_\{1\}\}closer thantL2∗t^\{\*\}\_\{L\_\{2\}\}tott​e​s​tLpt\_\{test\}^\{L\_\{p\}\}forp=1p=1andp=2p=2\.Figure[8](https://arxiv.org/html/2608.24210#S4.F8)shows that curveLt​e​s​tL\_\{test\}is again belowL𝒟∗\+ϵL\_\{\{\\cal D\}\}^\{\*\}\+\\epsilonin accordance with \([30](https://arxiv.org/html/2608.24210#S4.E30)\)\. We havett​e​s​t=418t\_\{test\}=418, which is close tot∗=415t^\{\*\}=415\. SoLt​e​s​tL\_\{test\}andL𝒟∗L\_\{\{\\cal D\}\}^\{\*\}reach their minima at around the same time \(t\+=t∗=415≈tt​e​s​tL1=418t^\{\+\}=t^\{\*\}=415\\approx t\_\{test\}^\{L\_\{1\}\}=418\)\. The values ofLt​e​s​t​\(t\)L\_\{test\}\(t\)att=t\+,t∗,tt​e​s​tt=t^\{\+\},t^\{\*\},t\_\{test\}are identical \(=0\.0582=0\.0582\), and almost equal to the value att=tapprox\+t=t\_\{\\text\{approx\}\}^\{\+\}\(=0\.0586=0\.0586\)\. See Table 2\. Here again, there is an excellent agreement between analytic and numerical estimates of the optimal stopping time\.

Table 2:The values oft1\+t\_\{1\}^\{\+\},tα\+t\_\{\\alpha\}^\{\+\}\(α=2\\alpha=2\),t\+t^\{\+\},…\\dots,tt​e​s​tt\_\{test\}with correspondingLt​e​s​t​\(t\)L\_\{test\}\(t\)andL𝒟∗​\(t\)\+ϵL\_\{\\cal\{D\}\}^\{\*\}\(t\)\+\\epsilon\(Example[12](https://arxiv.org/html/2608.24210#Thmtheorem12)\)\.Finally, we have:L𝒟∗​\(t\+\)\+ϵ=0\.1703<L𝒟∗​\(∞\)\+ϵ=0\.1713L\_\{\\cal\{D\}\}^\{\*\}\(t^\{\+\}\)\+\\epsilon=0\.1703<L\_\{\\cal\{D\}\}^\{\*\}\(\\infty\)\+\\epsilon=0\.1713, which proves that early stopping is again beneficial\.

###### Example 13\.

\(Various input data distributions\) For each kind of distribution, we give a figure displaying the curvesL𝒟∗Lp\+ϵpL\_\{\{\\cal D\}^\{\*\}\}^\{L\_\{p\}\}\+\\epsilon\_\{p\}andLt​e​s​tLpL\_\{test\}^\{L\_\{p\}\}with two couples\(m,n\)\(m,n\)of different ratiom/nm/n: Figure[9](https://arxiv.org/html/2608.24210#S4.F9)for Gaussian distribution, Figure[10](https://arxiv.org/html/2608.24210#S4.F10)for uniform distribution, Figure[11](https://arxiv.org/html/2608.24210#S4.F11)for Pareto distribution\. For all these examples, we haveα=1\\alpha=1\. The values ofλ1\\lambda\_\{1\}andλ2\\lambda\_\{2\}are given in caption\. We see that the method works better \(i\.e\.,t∗t^\{\*\}closer tott​e​s​tt\_\{test\}\) when the ration/mn/mis larger\. For example, in Figure[11](https://arxiv.org/html/2608.24210#S4.F11), the method fails form=256,n=512m=256,n=512\(top plot:t\+=t∗=0t^\{\+\}=t^\{\*\}=0\) while it succeeds form=256,n=16384m=256,n=16384\(bottom plot:t\+=t∗=45≈tt​e​s​t=46t^\{\+\}=t^\{\*\}=45\\approx t\_\{test\}=46\)\.

Figure 9:Gaussian distribution\. Top:m=512m=512,n=512n=512\(λ1≡λα=2291\\lambda\_\{1\}\\equiv\\lambda\_\{\\alpha\}=2291,λ2=486\\lambda\_\{2\}=486\)\. Bottom:m=256m=256,n=16384n=16384\(λ1≡λα=8\.23⋅104\\lambda\_\{1\}\\equiv\\lambda\_\{\\alpha\}=8\.23\\cdot 10^\{4\},λ2=2\.05⋅104\\lambda\_\{2\}=2\.05\\cdot 10^\{4\}\)\.Figure 10:Uniform distribution\. Top:m=512m=512,n=512n=512\(λ1≡λα=2413\\lambda\_\{1\}\\equiv\\lambda\_\{\\alpha\}=2413,λ2=679\\lambda\_\{2\}=679\)\. Bottom:m=256m=256,n=512n=512\(λ1≡λα=2291\\lambda\_\{1\}\\equiv\\lambda\_\{\\alpha\}=2291,λ2=486\\lambda\_\{2\}=486\)\.Figure 11:Pareto distribution\. Top:m=256m=256,n=512n=512\. \(λ1≡λα=5⋅105\\lambda\_\{1\}\\equiv\\lambda\_\{\\alpha\}=5\\cdot 10^\{5\},λ2=1⋅105\\lambda\_\{2\}=1\\cdot 10^\{5\}\)\. Bottom:m=256m=256,n=16384n=16384\. \(λ1≡λα=17⋅106\\lambda\_\{1\}\\equiv\\lambda\_\{\\alpha\}=17\\cdot 10^\{6\},λ2=3⋅106\\lambda\_\{2\}=3\\cdot 10^\{6\}\)\.

## 5Final Remarks

We have proposed a data\-dependent definition of timet\+t^\{\+\}as a stopping time for the gradient flow process\. This timet\+t^\{\+\}is a low estimate of the timet∗t^\{\*\}at which an upper bound on the population lossL𝒟L\_\{\{\\cal D\}\}reaches its minimum\. The advantage is thatt\+t^\{\+\}\(and even more so its approximationtapprox\+t^\{\+\}\_\{\\text\{approx\}\}\) is easily calculated without needing to apply gradient flow\. Our method is well suited to the underparameterized context \(m≤nm\\leq n\), with results improving as the ration/mn/mincreases\. It applies equally to different types of data distributions without requiring any specific knowledge on them\.

We have focused on linear models\. Although the method, in combination with linear probing, has been applied successfully to nonlinear examples, it would be interesting to apply it directly to nonlinear models\. On the other hand, we have considered only scalar outputs \(y∈ℝy\\in\\mathbb\{R\}\)\. Another extension would be to consider vectorial outputs \(y∈ℝqy\\in\\mathbb\{R\}^\{q\}withq≥2q\\geq 2\)\. Finally, it would be interesting to treat input data affected by random noise\. All these extensions will be the subject of future work\.

## Appendix AProof of Proposition[1](https://arxiv.org/html/2608.24210#Thmtheorem1)

###### Proof\.

The proof is a simple adaptation of the proof of Proposition 3 of[Martin Xavier et al\. \(2025\)](https://arxiv.org/html/2608.24210#bib.bib23)in the case of linear models\. By Theorem \(11\.3\) of[Mohri et al\. \(2018\)](https://arxiv.org/html/2608.24210#bib.bib21), we have with probability1−δ1\-\\delta:

L𝒟−LS≤2​ℛn\+ϵL\_\{\{\\cal D\}\}\-L\_\{S\}\\leq 2\{\\cal R\}\_\{n\}\+\\epsilonwhereℛn\{\\cal R\}\_\{n\}is the Rademacher complexity of the linear class\. By Theorem 5\.5 of[Ma \(2022\)](https://arxiv.org/html/2608.24210#bib.bib2), we haveℛn≤‖𝒂‖2​C/n\{\\cal R\}\_\{n\}\\leq\\\|\\boldsymbol\{a\}\\\|\_\{2\}C/\\sqrt\{n\}\. It then follows:

L𝒟≤LS\+2​C​‖𝒂‖2n\+ϵ=L𝒟∗\+ϵ,L\_\{\{\\cal D\}\}\\leq L\_\{S\}\+\\frac\{2C\\\|\\boldsymbol\{a\}\\\|\_\{2\}\}\{\\sqrt\{n\}\}\+\\epsilon=L\_\{\{\\cal D\}\}^\{\*\}\+\\epsilon,i\.e\. \([8](https://arxiv.org/html/2608.24210#S2.E8)\)\. ∎

## Appendix BProof of Proposition[5](https://arxiv.org/html/2608.24210#Thmtheorem5)

###### Proof\.

Using \([1](https://arxiv.org/html/2608.24210#S2.E1)\) and \([12](https://arxiv.org/html/2608.24210#S2.E12)\), the derivative ofLSL\_\{S\}is:

d​LS​\(t\)d​t=\\displaystyle\\frac\{dL\_\{S\}\(t\)\}\{dt\}=1n​d​‖𝒗⁡\(t\)‖1d​t=1n​∂‖𝒗‖1∂𝒗​d​𝒗​\(t\)d​t=−1n​\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒗​\(t\)\.\\displaystyle\\frac\{1\}\{n\}\\frac\{d\\\|\\boldsymbol\{v\}\(t\)\\\|\_\{1\}\}\{dt\}=\\frac\{1\}\{n\}\\frac\{\\partial\\\|\\boldsymbol\{v\}\\\|\_\{1\}\}\{\\partial\\boldsymbol\{v\}\}\\frac\{d\\boldsymbol\{v\}\(t\)\}\{dt\}=\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{v\}\(t\)\.\(32\)
Besides, using \([6](https://arxiv.org/html/2608.24210#S2.E6)\), the derivative ofLG∗L\_\{G\}^\{\*\}is:

d​LG∗​\(t\)d​t=\\displaystyle\\frac\{dL\_\{G\}^\{\*\}\(t\)\}\{dt\}=2​C​𝒂⊤n​‖𝒂‖2​d​𝒂d​t\.\\displaystyle\\frac\{2C\\boldsymbol\{a^\{\\top\}\}\}\{\\sqrt\{n\}\\\|\\boldsymbol\{a\}\\\|\_\{2\}\}\\frac\{d\\boldsymbol\{a\}\}\{dt\}\.\(33\)From \([3](https://arxiv.org/html/2608.24210#S2.E3)\), we have:𝑲​𝒂​\(t\)=𝒗⁡\(t\)\+𝒚\\boldsymbol\{K\}\\boldsymbol\{a\}\(t\)=\\boldsymbol\{v\}\(t\)\+\\boldsymbol\{y\}\. Since𝒂⁡\(t\)\\boldsymbol\{a\}\(t\)is initialized to 0, know that the vector𝒂⁡\(t\)\\boldsymbol\{a\}\(t\)updated by GF, satisfies \(see, e\.g\.,[Björck and Golub \(1973\)](https://arxiv.org/html/2608.24210#bib.bib3)\):

𝒂⁡\(t\)=𝑲†​\(𝒗⁡\(t\)\+𝒚\)\.\\boldsymbol\{a\}\(t\)=\\boldsymbol\{K\}^\{\\dagger\}\\left\(\\boldsymbol\{v\}\(t\)\+\\boldsymbol\{y\}\\right\)\.\(34\)So, using \([11](https://arxiv.org/html/2608.24210#S2.E11)\), \([15](https://arxiv.org/html/2608.24210#S3.E15)\) and \([34](https://arxiv.org/html/2608.24210#A2.E34)\), Equation \([33](https://arxiv.org/html/2608.24210#A2.E33)\) becomes :

d​LG∗​\(t\)d​t=\\displaystyle\\frac\{dL\_\{G\}^\{\*\}\(t\)\}\{dt\}=−2​Cn​\(𝑲†​\(𝒗⁡\(t\)\+𝒚\)\)⊤‖𝑲†​\(𝒗⁡\(t\)\+𝒚\)‖2​𝑲⊤​v​\(t\)=Ψ⁡\(t\)\.\\displaystyle\-\\frac\{2C\}\{\\sqrt\{n\}\}\\frac\{\(\\boldsymbol\{K\}^\{\\dagger\}\(\\boldsymbol\{v\}\(t\)\+\\boldsymbol\{y\}\)\)^\{\\top\}\}\{\\\|\\boldsymbol\{K\}^\{\\dagger\}\(\\boldsymbol\{v\}\(t\)\+\\boldsymbol\{y\}\)\\\|\_\{2\}\}\\boldsymbol\{K\}^\{\\top\}v\(t\)=\\Psi\(t\)\.\(35\)
It then follows from \([7](https://arxiv.org/html/2608.24210#S2.E7)\), \([32](https://arxiv.org/html/2608.24210#A2.E32)\) and \([35](https://arxiv.org/html/2608.24210#A2.E35)\):

d​L𝒟∗​\(t\)d​t=\\displaystyle\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(t\)\}\{dt\}=−1n​\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒗​\(t\)\+Ψ⁡\(t\),\\displaystyle\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{v\}\(t\)\+\\Psi\(t\),i\.e\. \([14](https://arxiv.org/html/2608.24210#S3.E14)\)\. ∎

## Appendix CProof of Proposition[7](https://arxiv.org/html/2608.24210#Thmtheorem7)

###### Proof\.

We have

d​LS​\(t\)d​t=\\displaystyle\\frac\{dL\_\{S\}\(t\)\}\{dt\}=−1n​\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒗​\(t\)\\displaystyle\\;\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{v\}\(t\)=\\displaystyle=−1n\(sgn\(𝒗\(t\)\)\)⊤𝑯∑i=1α𝒘ie−λi​t\\displaystyle\\;\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\sum\_\{i=1\}^\{\\alpha\}\\boldsymbol\{w\}\_\{i\}e^\{\-\\lambda\_\{i\}t\}−1n\(sgn\(𝒗\(t\)\)\)⊤𝑯∑j=α\+1n𝒘je−λj​t\\displaystyle\\;\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\sum\_\{j=\\alpha\+1\}^\{n\}\\boldsymbol\{w\}\_\{j\}e^\{\-\\lambda\_\{j\}t\}=\\displaystyle=−1n\(sgn\(𝒗\(t\)\)\)⊤𝑯∑i=1α𝒘ie−λi​t−1nΔ\(t\)\.\\displaystyle\\;\-\\frac\{1\}\{n\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\sum\_\{i=1\}^\{\\alpha\}\\boldsymbol\{w\}\_\{i\}e^\{\-\\lambda\_\{i\}t\}\-\\frac\{1\}\{n\}\\Delta\(t\)\.
Hence, using \([14](https://arxiv.org/html/2608.24210#S3.E14)\), we have

d​L𝒟∗​\(t\)/d​t=1n​d​LS​\(t\)/d​t\+Ψ⁡\(t\)<0dL\_\{\{\\cal D\}\}^\{\*\}\(t\)/dt=\\frac\{1\}\{n\}dL\_\{S\}\(t\)/dt\+\\Psi\(t\)<0iff:

−\\displaystyle\-∑i=1α\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒘i​e−λi​t−Δ⁡\(t\)\+n​Ψ​\(t\)<0\\displaystyle\\sum\_\{i=1\}^\{\\alpha\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{w\}\_\{i\}e^\{\-\\lambda\_\{i\}t\}\-\\Delta\(t\)\+n\\Psi\(t\)<0i\.e\.:−\\displaystyle\-∑i=1α\(sgn​\(𝒗⁡\(t\)\)\)⊤​𝑯​𝒘i​e−λi​t\+Ω⁡\(t\)<0\\displaystyle\\sum\_\{i=1\}^\{\\alpha\}\\left\(\\text\{sgn\}\(\\boldsymbol\{v\}\(t\)\)\\right\)^\{\\top\}\\boldsymbol\{H\}\\boldsymbol\{w\}\_\{i\}e^\{\-\\lambda\_\{i\}t\}\+\\Omega\(t\)<0i\.e\.:∑i∈\[α\]Γi​\(t\)​e−λi​t−Ω⁡\(t\)\>0\\displaystyle\\sum\_\{i\\in\[\\alpha\]\}\\Gamma\_\{i\}\(t\)e^\{\-\\lambda\_\{i\}t\}\-\\Omega\(t\)\>0i\.e\.:Φ⁡\(t\)\>0\.\\displaystyle\\;\\Phi\(t\)\>0\.Henced​L𝒟∗​\(t\)/d​t<0dL\_\{\{\\cal D\}\}^\{\*\}\(t\)/dt<0iffΦ⁡\(t\)\>0\\Phi\(t\)\>0, i\.e\.: \([19](https://arxiv.org/html/2608.24210#S3.E19)\)\. From \([18](https://arxiv.org/html/2608.24210#S3.E18)\) and \([19](https://arxiv.org/html/2608.24210#S3.E19)\), it then follows:

t\+=sups∈𝒯1\{s:d​L𝒟∗​\(s\)d​t<0​∀t∈\[0,s\)\},t^\{\+\}=\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:\\frac\{dL\_\{\{\\cal D\}\}^\{\*\}\(s\)\}\{dt\}<0\\,\\ \\forall t\\in\[0,s\)\\right\\\},i\.e\. \([20](https://arxiv.org/html/2608.24210#S3.E20)\)\. On the other hand,t∗t^\{\*\}is the least values≥0s\\geq 0such thatd​L𝒟∗​\(s\)/d​t=0dL\_\{\{\\cal D\}\}^\{\*\}\(s\)/dt=0\(see \([13](https://arxiv.org/html/2608.24210#S2.E13)\)\)\. Hencet\+≤t∗t^\{\+\}\\leq t^\{\*\}, i\.e\. \([21](https://arxiv.org/html/2608.24210#S3.E21)\)\.

There are now two cases: Eithert∗t^\{\*\}belongs to𝒯1\{\\cal T\}\_\{1\}, i\.e\.t∗<τ1t^\{\*\}<\\tau\_\{1\}\(case 1\), or not, i\.e\.t∗≥τ1t^\{\*\}\\geq\\tau\_\{1\}\(case 2\)\.

Case 1 \(OPENt∗<τ1\)t^\{\*\}<\\tau\_\{1\}\): Sincet∗t^\{\*\}is the first timesssuch thatd​L𝒟∗​\(s\)/d​t≥0dL\_\{\{\\cal D\}\}^\{\*\}\(s\)/dt\\geq 0\(see \([13](https://arxiv.org/html/2608.24210#S2.E13)\)\), we haved​L𝒟∗​\(s\)/d​t<0dL\_\{\{\\cal D\}\}^\{\*\}\(s\)/dt<0for alls<t∗<τ1s<t^\{\*\}<\\tau\_\{1\}\. It follows from \([20](https://arxiv.org/html/2608.24210#S3.E20)\):t∗≤t\+t^\{\*\}\\leq t^\{\+\}\. Sincet\+≤t∗t^\{\+\}\\leq t^\{\*\}\(see \([21](https://arxiv.org/html/2608.24210#S3.E21)\)\), we have:t\+=t∗t^\{\+\}=t^\{\*\}, i\.e\. \([22](https://arxiv.org/html/2608.24210#S3.E22)\)\.

Case 2 \(t∗≥τ1t^\{\*\}\\geq\\tau\_\{1\}\): Sincet∗t^\{\*\}is the first timesssuch thatd​L𝒟∗​\(s\)/d​t≥0dL\_\{\{\\cal D\}\}^\{\*\}\(s\)/dt\\geq 0\(see \([13](https://arxiv.org/html/2608.24210#S2.E13)\)\) andt∗≥τ1t^\{\*\}\\geq\\tau\_\{1\}, we haved​L𝒟∗​\(s\)/d​t<0dL\_\{\{\\cal D\}\}^\{\*\}\(s\)/dt<0for alls<τ1s<\\tau\_\{1\}\. It follows from \([20](https://arxiv.org/html/2608.24210#S3.E20)\):τ1≤t\+\\tau\_\{1\}\\leq t^\{\+\}\. Besides, as a supremum of a set of elements<τ1<\\tau\_\{1\},t\+t^\{\+\}satisfies:t\+≤τ1t^\{\+\}\\leq\\tau\_\{1\}\. Sot\+=τ1t^\{\+\}=\\tau\_\{1\}, i\.e\. \([23](https://arxiv.org/html/2608.24210#S3.E23)\)\.

Suppose furthermore \([24](https://arxiv.org/html/2608.24210#S3.E24)\)\. ThenΓ⁡\(t\)=Σi∈\[α\]​Γi​\(t\)=Σi∈ℐ\+​Γi​\(t\)\\Gamma\(t\)=\\Sigma\_\{i\\in\[\\alpha\]\}\\Gamma\_\{i\}\(t\)=\\Sigma\_\{i\\in\{\\cal I\}^\{\+\}\}\\Gamma\_\{i\}\(t\), withΓi​\(t\)\>0\\Gamma\_\{i\}\(t\)\>0for alli∈\[α\]i\\in\[\\alpha\]\. Then, using \([17](https://arxiv.org/html/2608.24210#S3.E17)\), we have

Γ⁡\(t\)​e−λ1​t−Ω⁡\(t\)≤Φ⁡\(t\)≤Γ⁡\(t\)​e−λα​t−Ω⁡\(t\)\.\\Gamma\(t\)e^\{\-\\lambda\_\{1\}t\}\-\\Omega\(t\)\\leq\\Phi\(t\)\\leq\\Gamma\(t\)e^\{\-\\lambda\_\{\\alpha\}t\}\-\\Omega\(t\)\.Using \([18](https://arxiv.org/html/2608.24210#S3.E18)\):

t\+=sups∈𝒯1\{s:Φ\(t\)\>0,∀t∈\[0,s\)\},t^\{\+\}=\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:\\Phi\(t\)\>0,\\ \\forall t\\in\[0,s\)\\right\\\},it follows:

sups∈𝒯1\{s:Γ\(t\)e−λ1​t−Ω\(t\)\>0,∀t∈\[0,s\)\}≤t\+≤sups∈𝒯1\{s:Γ\(t\)e−λα​t−Ω\(t\)\>0,∀t∈\[0,s\)\},\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:\\Gamma\(t\)e^\{\-\\lambda\_\{1\}t\}\-\\Omega\(t\)\>0,\\ \\forall t\\in\[0,s\)\\right\\\}\\leq t^\{\+\}\\leq\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:\\Gamma\(t\)e^\{\-\\lambda\_\{\\alpha\}t\}\-\\Omega\(t\)\>0,\\ \\forall t\\in\[0,s\)\\right\\\},i\.e\.:

sups∈𝒯1\{s:t<1λ1lnΓ⁡\(t\)Ω⁡\(t\),∀t∈\[0,s\)\}≤t\+≤sups∈𝒯1\{s:t<1λαlnΓ⁡\(t\)Ω⁡\(t\),∀t∈\[0,s\)\},\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:t<\\frac\{1\}\{\\lambda\_\{1\}\}\\ln\\frac\{\\Gamma\(t\)\}\{\\Omega\(t\)\},\\ \\forall t\\in\[0,s\)\\right\\\}\\leq t^\{\+\}\\leq\\sup\_\{s\\in\{\\cal T\}\_\{1\}\}\\left\\\{s:t<\\frac\{1\}\{\\lambda\_\{\\alpha\}\}\\ln\\frac\{\\Gamma\(t\)\}\{\\Omega\(t\)\},\\ \\forall t\\in\[0,s\)\\right\\\},i\.e\.:

t1\+≤t\+≤tα\+,t^\{\+\}\_\{1\}\\leq t^\{\+\}\\leq t^\{\+\}\_\{\\alpha\},i\.e\. \([25](https://arxiv.org/html/2608.24210#S3.E25)\)\. ∎

## References

- Advaniet al\.\(2020\)M\. S\. Advani, A\. M\. Saxe, and H\. SompolinskyHigh\-dimensional dynamics of generalization error in neural networks\.Neural Networks132,pp\. 428–446\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p1.1),[§1](https://arxiv.org/html/2608.24210#S1.p1.1),[§3\.2](https://arxiv.org/html/2608.24210#S3.SS2.p1.1)\.
- Alain and Bengio \(2016\)G\. Alain and Y\. BengioUnderstanding intermediate layers using linear classifier probes\.arXiv preprint arXiv:1610\.01644\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p2.1),[Example 11](https://arxiv.org/html/2608.24210#Thmtheorem11.p1.1.1)\.
- Aliet al\.\(2019\)A\. Ali, J\. Z\. Kolter, and R\. J\. TibshiraniA continuous\-time view of early stopping for least squares regression\.InThe 22nd international conference on artificial intelligence and statistics,pp\. 1370–1378\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p1.1)\.
- Allen\-Zhuet al\.\(2019\)Z\. Allen\-Zhu, Y\. Li, and Y\. LiangLearning and generalization in overparameterized neural networks, going beyond two layers\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1)\.
- Aroraet al\.\(2019\)S\. Arora, S\. S\. Du, W\. Hu, Z\. Li, and R\. WangFine\-grained analysis of optimization and generalization for overparameterized two\-layer neural networks\.InICML 2019, Long Beach, California, USA,Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1)\.
- Bartlettet al\.\(2020\)P\. L\. Bartlett, P\. M\. Long, G\. Lugosi, and A\. TsiglerBenign overfitting in linear regression\.Proceedings of the National Academy of Sciences117\(48\),pp\. 30063–30070\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p2.1),[§3\.3](https://arxiv.org/html/2608.24210#S3.SS3.p1.1)\.
- Bartlett and Mendelson \(2002\)P\. L\. Bartlett and S\. MendelsonRademacher and gaussian complexities: risk bounds and structural results\.Journal of Machine Learning Research3,pp\. 463–482\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p1.1)\.
- Belkinet al\.\(2018\)M\. Belkin, D\. Hsu, S\. Ma, and S\. MandalReconciling modern machine learning and the bias\-variance trade\-off\.CoRRabs/1812\.11118\.External Links:[Link](http://arxiv.org/abs/1812.11118),1812\.11118Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p2.1)\.
- Björck and Golub \(1973\)A\. Björck and G\. H\. GolubNumerical methods for computing angles between linear subspaces\.Mathematics of computation27\(123\),pp\. 579–594\.Cited by:[Appendix B](https://arxiv.org/html/2608.24210#A2.p2.4.1),[§3\.3](https://arxiv.org/html/2608.24210#S3.SS3.p1.1)\.
- Duet al\.\(2018\)S\. S\. Du, X\. Zhai, B\. Póczos, and A\. SinghGradient descent provably optimizes over\-parameterized neural networks\.CoRRabs/1810\.02054\.External Links:1810\.02054Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1),[§2\.1](https://arxiv.org/html/2608.24210#S2.SS1.p1.2),[§2\.2](https://arxiv.org/html/2608.24210#S2.SS2.p1.5)\.
- Ghorbaniet al\.\(2019\)B\. Ghorbani, S\. Krishnan, and Y\. XiaoAn investigation into neural net optimization via hessian eigenvalue density\.InInternational Conference on Machine Learning,pp\. 2232–2241\.Cited by:[§3\.2](https://arxiv.org/html/2608.24210#S3.SS2.p1.1)\.
- Heckel and Yilmaz \(2020\)R\. Heckel and F\. F\. YilmazEarly stopping in deep networks: double descent and how to eliminate it\.arXiv preprint arXiv:2007\.10099\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1),[§3\.3](https://arxiv.org/html/2608.24210#S3.SS3.p1.1)\.
- Jacotet al\.\(2018\)A\. Jacot, C\. Hongler, and F\. GabrielNeural tangent kernel: convergence and generalization in neural networks\.InNeurIPS 2018, December 3\-8, 2018, Montréal, Canada,pp\. 8580–8589\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1)\.
- Le Cunet al\.\(1991\)Y\. Le Cun, I\. Kanter, and S\. A\. SollaEigenvalues of covariance matrices: application to neural\-network learning\.Physical review letters66\(18\),pp\. 2396\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p1.1)\.
- LeCun \(1998\)Y\. LeCunThe mnist database of handwritten digits\.http://yann\. lecun\. com/exdb/mnist/\.Cited by:[Example 11](https://arxiv.org/html/2608.24210#Thmtheorem11.p1.1.1)\.
- Liet al\.\(2020\)M\. Li, M\. Soltanolkotabi, and S\. OymakGradient descent with early stopping is provably robust to label noise for overparameterized neural networks\.InAISTATS 2020, 26\-28 August 2020,Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1)\.
- Liao and Couillet \(2018\)Z\. Liao and R\. CouilletThe dynamics of learning: A random matrix approach\.InProceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10\-15, 2018,J\. G\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research,pp\. 3078–3087\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p1.1),[Example 10](https://arxiv.org/html/2608.24210#Thmtheorem10.p1.1.1)\.
- Ma \(2022\)T\. MaLecture notes for machine learning theory \(cs229m/stats214\)\.June\.Cited by:[Appendix A](https://arxiv.org/html/2608.24210#A1.p1.2.1),[Remark 2](https://arxiv.org/html/2608.24210#Thmtheorem2.p1.2.1)\.
- Martin Xavieret al\.\(2025\)D\. Martin Xavier, L\. Chamoin, and L\. FribourgEarly stopping strategy using neural tangent kernel theory and Rademacher complexity\.In2025 American Control Conference \(ACC\),pp\. 1301–1306\.Cited by:[Appendix A](https://arxiv.org/html/2608.24210#A1.p1.1.1),[§2\.1](https://arxiv.org/html/2608.24210#S2.SS1.p1.2),[§2\.1](https://arxiv.org/html/2608.24210#S2.SS1.p3.1),[Proposition 1](https://arxiv.org/html/2608.24210#Thmtheorem1.p1.1.1)\.
- Mohriet al\.\(2018\)M\. Mohri, A\. Rostamizadeh, and A\. TalwalkarFoundations of machine learning\.MIT press\.Cited by:[Appendix A](https://arxiv.org/html/2608.24210#A1.p1.1.1),[Remark 2](https://arxiv.org/html/2608.24210#Thmtheorem2.p1.1.1)\.
- Murrayet al\.\(2023\)M\. Murray, H\. Jin, B\. Bowman, and G\. MontúfarCharacterizing the spectrum of the NTK via a power series expansion\.InICLR 2023, Kigali, Rwanda, May 1\-5, 2023,Cited by:[§3\.2](https://arxiv.org/html/2608.24210#S3.SS2.p1.1)\.
- Nakkiranet al\.\(2021\)P\. Nakkiran, G\. Kaplun, Y\. Bansal, T\. Yang, B\. Barak, and I\. SutskeverDeep double descent: where bigger models and more data hurt\*\.Journal of Statistical Mechanics: Theory and Experiment2021\(12\),pp\. 124003\.External Links:[Document](https://dx.doi.org/10.1088/1742-5468/ac3a74),[Link](https://doi.org/10.1088/1742-5468/ac3a74)Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1),[§3\.3](https://arxiv.org/html/2608.24210#S3.SS3.p1.1)\.
- Oymaket al\.\(2019\)S\. Oymak, Z\. Fabian, M\. Li, and M\. SoltanolkotabiGeneralization guarantees for neural networks via harnessing the low\-rank structure of the jacobian\.CoRRabs/1906\.05392\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p1.3),[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1),[§3\.2](https://arxiv.org/html/2608.24210#S3.SS2.p1.1),[Remark 6](https://arxiv.org/html/2608.24210#Thmtheorem6.p1.1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, and F\. M\. et al\.PyTorch: An imperative style, high\-performance deep learning library\.InNeurIPS 2019, December, Vancouver, BC, Canada,Cited by:[Example 11](https://arxiv.org/html/2608.24210#Thmtheorem11.p1.1.1)\.
- Prechelt \(2002\)L\. PrecheltEarly stopping\-but when?\.InNeural Networks: Tricks of the trade,pp\. 55–69\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.24210#S2.SS2.p1.2)\.
- Raskuttiet al\.\(2014\)G\. Raskutti, M\. J\. Wainwright, and B\. YuEarly stopping and non\-parametric regression: an optimal data\-dependent stopping rule\.Journal of Machine Learning Research15\(11\),pp\. 335–366\.Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p1.1)\.
- Sonthaliaet al\.\(2024\)R\. Sonthalia, J\. Lok, and E\. RebrovaOn regularization via early stopping for least squares regression\.External Links:2406\.04425,[Link](https://arxiv.org/abs/2406.04425)Cited by:[§1](https://arxiv.org/html/2608.24210#S1.p2.1),[§3\.3](https://arxiv.org/html/2608.24210#S3.SS3.p1.1)\.
- Stephenson and Lee \(2021\)C\. Stephenson and T\. LeeWhen and how epochwise double descent happens\.External Links:2108\.12006,[Link](https://arxiv.org/abs/2108.12006)Cited by:[§1](https://arxiv.org/html/2608.24210#S1.SSx1.p2.1),[§3\.3](https://arxiv.org/html/2608.24210#S3.SS3.p1.1)\.