线性逼近类的一般频率原理

# General Frequency Principle for Linear Approximation Classes
Source: [https://arxiv.org/html/2610.41179](https://arxiv.org/html/2610.41179)
MSC Classification \(2020\)\]41A25, 33C10, 42B10, 65D15, 68T07

Yujun WangEmail:[yujunwang@std\.uestc\.edu\.cn](mailto:[email protected])Affiliation:School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, 611731, ChinaYiqi GuEmail:[yiqigu@uestc\.edu\.cn](mailto:[email protected])Affiliation:School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, 611731, China

###### Abstract

In this paper, we study a general frequency principle under gradient descent training for the finite linear approximation classes\. Under suitable assumptions, we prove that the learner amplitude relative to the target amplitude vanishes at high frequencies, uniformly throughout training\. We also establish sufficient conditions for low\-frequency priority, covering single\-frequency targets trained separately and two frequency components trained in the same run\. This general frequency principle does not imply that lower frequencies always converge faster\. Instead, for two frequencies that satisfy certain comparison conditions, one component will take longer to learn than the other; and in common cases, the longer component has higher frequency\. Numerical experiments with Legendre and trigonometric bases validate the theoretical results\.

###### keywords

frequency principle, spectral bias, approximation capacity, learning time, linear approximation classes, gradient descent

## 1Introduction

Gradient\-based training of neural networks often fits low\-frequency target components before high\-frequency ones\. This behavior, known as the frequency principle or spectral bias, has been observed across network architectures and activation functions[Xu et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib2);[Rahaman et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib1);[Xu et al\. \(2020\)](https://arxiv.org/html/2610.41179#bib.bib3);[Xu et al\. \(2025\)](https://arxiv.org/html/2610.41179#bib.bib4)\. A related question is how frequency\-dependent learning can be characterized for linear approximation classes with fixed basis functions\. In such classes, a small high\-frequency response may result from limited representation or slow convergence toward the best available approximation\. Distinguishing these effects is necessary for interpreting the learning order\.

One approach to the frequency principle uses Fourier decay\. For shallow tanh networks with small input weights, Xu et al\.[Xu et al\. \(2020\)](https://arxiv.org/html/2610.41179#bib.bib3)compared frequency contributions to parameter gradients and, in an idealized two\-frequency setting, loss decay rates\. Rahaman et al\.[Rahaman et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib1)analyzed the Fourier decay of ReLU networks and their parameter derivatives through the piecewise linear structure\. Luo et al\.[Luo et al\. \(2021\)](https://arxiv.org/html/2610.41179#bib.bib20)estimated high\-frequency contributions to loss or output changes under regularity and bounded\-trajectory assumptions\. Their gradient\-flow results cover initial and intermediate training, with final\-stage estimates under convergence to a nondegenerate global minimizer\. These analyses concern gradients or changes during training, which differ from the time required for an individual Fourier error to reach a prescribed threshold\.

A second approach studies the spectrum of the training operator\. For fixed linear classes, coefficient gradient descent is Landweber iteration, with convergence governed by the Gram spectrum[Landweber \(1951\)](https://arxiv.org/html/2610.41179#bib.bib19);[Engl et al\. \(1996\)](https://arxiv.org/html/2610.41179#bib.bib18)\. Neural tangent kernel \(NTK\) theory gives a related description of wide networks near their initial linearization[Jacot et al\. \(2018\)](https://arxiv.org/html/2610.41179#bib.bib5);[Du et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib6);[Allen\-Zhu et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib11);[Arora et al\. \(2019a\)](https://arxiv.org/html/2610.41179#bib.bib12);[Lee et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib7)\. Kernel eigenvalues control residual decay[Arora et al\. \(2019b\)](https://arxiv.org/html/2610.41179#bib.bib13);[Cao et al\. \(2021\)](https://arxiv.org/html/2610.41179#bib.bib14); under uniform spherical data, spherical harmonics relate these rates to frequency[Basri et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib8);[Cao et al\. \(2021\)](https://arxiv.org/html/2610.41179#bib.bib14)\. Nonuniform data density also affects convergence[Basri et al\. \(2020\)](https://arxiv.org/html/2610.41179#bib.bib9)\. Fourier descriptions have been derived through mean\-field linearization[Zhang et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib15)and infinite\-width NTK dynamics[Luo et al\. \(2022\)](https://arxiv.org/html/2610.41179#bib.bib17)\. At finite width, Zhang et al\.[Zhang et al\. \(2025\)](https://arxiv.org/html/2610.41179#bib.bib16)studied Gram spectra, finite\-precision approximation limits, and lower bounds on learning times for generalized Fourier coefficients\. Bowman and Montúfar[Bowman and Montúfar \(2022\)](https://arxiv.org/html/2610.41179#bib.bib10)bounded deviations from NTK dynamics under width and sample\-size conditions\. For general fixed bases, Fourier modes need not be eigenfunctions of the training operator\. Comparing their learning times therefore requires the distribution of each represented mode among the eigendirections, together with its attainable approximation error\.

In this paper, we study a general frequency principle for finite linear approximation classes on a bounded box\. LetΩ=\[a,b\]d⊂ℝd\\Omega=\[a,b\]^\{d\}\\subset\\mathbb\{R\}^\{d\}be a bounded domain andf∈L2​\(Ω\)f\\in L^\{2\}\(\\Omega\)a possibly complex\-valued target\. We consider using a learner with linear structure to approximateff, i\.e\.,

h𝒄​\(𝒙\)=∑s=1ncs​ϕs​\(𝒙\),𝒙∈Ω,h\_\{\\bm\{c\}\}\(\\bm\{x\}\)=\\sum\_\{s=1\}^\{n\}c\_\{s\}\\phi\_\{s\}\(\\bm\{x\}\),\\qquad\\bm\{x\}\\in\\Omega,\(1\.1\)whereϕ1,…,ϕn∈L2​\(Ω\)\\phi\_\{1\},\\ldots,\\phi\_\{n\}\\in L^\{2\}\(\\Omega\)are fixed and linearly independent and only the coefficients𝒄=\(c1,…,cn\)∈ℂn\\bm\{c\}=\(c\_\{1\},\\ldots,c\_\{n\}\)\\in\\mathbb\{C\}^\{n\}are trained\.

Our main contributions are as follows\. First, we establish bounds on high\-frequency relative movement\. Under zero initialization and suitable Fourier decay assumptions, Theorem[3\.1](https://arxiv.org/html/2610.41179#S3.Thmtheorem1)shows that the learner amplitude relative to the target amplitude tends to zero at high frequencies along the prescribed frequency set\. This holds at each fixed iteration, and Corollary[3\.2](https://arxiv.org/html/2610.41179#S3.Thmtheorem2)gives a uniform bound throughout training under a stable step size\.

Second, we derive learning\-time bounds relative to the best approximation available in the class\. Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)gives sufficient conditions for strict learning order between two single\-frequency targets trained separately through comparisons of learning\-time bounds, spectral distributions, and spectral means\. For lower\- and higher\-frequency pairs satisfying the corresponding conditions, the lower frequency reaches the prescribed relative error first\.

Third, we extend the analysis to general targets\. We give sufficient conditions for the order in which two components in the same training run reach and remain below a prescribed relative error\. Experiments with Legendre and trigonometric bases illustrate the high\-frequency estimates and learning\-time results\.

These results characterize high\-frequency relative movement and conditional learning order within a common linear framework\. Here, ”general” refers to establishing learning order by comparing bounds: if an upper learning\-time bound for one mode is below a lower bound for another, the first mode is learned earlier\. Under suitable conditions on the basis, this comparison also implies that the first mode has the smaller frequency magnitude, recovering low\-frequency priority for such pairs\. Table[1](https://arxiv.org/html/2610.41179#S1.T1)compares this analysis with representative results in the literature\.

Table 1:Comparison of representative results on frequency\-dependent learning\.This paper is organized as follows\. Section[2](https://arxiv.org/html/2610.41179#S2)introduces the training operator and its spectral representation\. Section[3](https://arxiv.org/html/2610.41179#S3)studies high\-frequency relative movement, and Section[4](https://arxiv.org/html/2610.41179#S4)derives learning\-time bounds and conditions for learning order\. Both sections include numerical experiments\. Section[5](https://arxiv.org/html/2610.41179#S5)concludes\.

## 2Notation and preliminary results

This section introduces the Fourier coordinates and the training operator used throughout the paper\. We first identify the attainable least\-squares limit, then relate each Fourier mode to the capacity and spectrum of the approximation class\.

### 2\.1Approximation classes and Fourier coordinates

Throughout the paper, we use linear structure \([1\.1](https://arxiv.org/html/2610.41179#S1.E1)\) to approximateff\. We callV=span⁡\{ϕ1,…,ϕn\}V=\\operatorname\{span\}\\\{\\phi\_\{1\},\\ldots,\\phi\_\{n\}\\\}the approximation class and denote byPVP\_\{V\}the orthogonal projection operator fromL2​\(Ω\)L^\{2\}\(\\Omega\)ontoVV\. Therefore, the projectionPV​fP\_\{V\}fis the part offfthat can be truly learned fromVV, which we refer to as the represented part offf\.

Foru∈L2​\(Ω\)u\\in L^\{2\}\(\\Omega\), extenduuonℝd\\mathbb\{R\}^\{d\}with zero inℝd\\Ω\\mathbb\{R\}^\{d\}\\backslash\\Omegaand define the unitary Fourier transform

u^\(𝒌\)=\(2π\)−d/2∫Ωu\(𝒙\)e−i𝒌⋅𝒙d𝒙,𝒌∈ℝd\.\\widehat\{u\}\(\\bm\{k\}\)=\(2\\pi\)^\{\-d/2\}\\int\_\{\\Omega\}u\(\\bm\{x\}\)\\mathrm\{e\}^\{\-\\mathrm\{i\}\\bm\{k\}\\cdot\\bm\{x\}\}~\\mathrm\{d\}\\bm\{x\},\\qquad\\bm\{k\}\\in\\mathbb\{R\}^\{d\}\.\(2\.1\)Fixϑ∈ℝd\\vartheta\\in\\mathbb\{R\}^\{d\}and set

Λ=2​πb−a\(ℤd\+ϑ\),χ𝒌\(𝒙\)=\|Ω\|−1/2ei​𝒌⋅𝒙,𝒌∈Λ\.\\Lambda=\\frac\{2\\pi\}\{b\-a\}\\left\(\\mathbb\{Z\}^\{d\}\+\\vartheta\\right\),\\qquad\\chi\_\{\\bm\{k\}\}\(\\bm\{x\}\)=\\left\|\\Omega\\right\|^\{\-1/2\}\\mathrm\{e\}^\{\\mathrm\{i\}\\bm\{k\}\\cdot\\bm\{x\}\},\\qquad\\bm\{k\}\\in\\Lambda\.\(2\.2\)For𝒌∈Λ\\bm\{k\}\\in\\Lambda,‖𝒌‖\\left\\\|\\bm\{k\}\\right\\\|denotes its Euclidean norm,‖𝒌‖=\(∑j=1d\|kj\|2\)1/2\\left\\\|\\bm\{k\}\\right\\\|=\(\\sum\_\{j=1\}^\{d\}\\left\|k\_\{j\}\\right\|^\{2\}\)^\{1/2\}\. The functions\{χ𝒌\}𝒌∈Λ\\\{\\chi\_\{\\bm\{k\}\}\\\}\_\{\\bm\{k\}\\in\\Lambda\}form an orthonormal basis ofL2​\(Ω\)L^\{2\}\(\\Omega\)\. We further write

u𝒌=⟨u,χ𝒌⟩Ω=\(2​πb−a\)d/2​u^​\(𝒌\),𝒌∈Λ,uΛ=\(u𝒌\)𝒌∈Λ\.u\_\{\\bm\{k\}\}=\\left\\langle u,\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}=\\left\(\\frac\{2\\pi\}\{b\-a\}\\right\)^\{d/2\}\\widehat\{u\}\(\\bm\{k\}\),\\qquad\\bm\{k\}\\in\\Lambda,\\qquad u^\{\\Lambda\}=\(u\_\{\\bm\{k\}\}\)\_\{\\bm\{k\}\\in\\Lambda\}\.\(2\.3\)Thus the lattice coefficients are the values of the Fourier transform \([2\.1](https://arxiv.org/html/2610.41179#S2.E1)\) onΛ\\Lambda, up to a fixed factor\. By Parseval’s identity, the mapu↦uΛu\\mapsto u^\{\\Lambda\}is unitary fromL2​\(Ω\)L^\{2\}\(\\Omega\)ontoℓ2​\(Λ\)\\ell^\{2\}\(\\Lambda\), and

⟨pΛ,qΛ⟩Λ=∑𝒌∈Λp𝒌​q𝒌¯=⟨p,q⟩Ω,p,q∈L2​\(Ω\)\.\\left\\langle p^\{\\Lambda\},q^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}=\\sum\_\{\\bm\{k\}\\in\\Lambda\}p\_\{\\bm\{k\}\}\\overline\{q\_\{\\bm\{k\}\}\}=\\left\\langle p,q\\right\\rangle\_\{\\Omega\},\\qquad p,q\\in L^\{2\}\(\\Omega\)\.\(2\.4\)
Now we consider the regression problem\. For a targetf∈L2​\(Ω\)f\\in L^\{2\}\(\\Omega\)and approximationh𝒄h\_\{\\bm\{c\}\}, the loss function is given by

J⁡\(𝒄\)=12​‖h𝒄−f‖L2​\(Ω\)2\.J\(\\bm\{c\}\)=\\frac\{1\}\{2\}\\left\\\|h\_\{\\bm\{c\}\}\-f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}^\{2\}\.\(2\.5\)Using gradient descent with learning rateη\>0\\eta\>0leads to

cs\(t\+1\)=cs\(t\)−η⟨h\(t\)−f,ϕs⟩Ω,t=0,1,…,c\_\{s\}^\{\(t\+1\)\}=c\_\{s\}^\{\(t\)\}\-\\eta\\left\\langle h^\{\(t\)\}\-f,\\phi\_\{s\}\\right\\rangle\_\{\\Omega\},\\quad t=0,1,\\dots,\(2\.6\)wherecs\(t\)c\_\{s\}^\{\(t\)\}andcs\(t\+1\)c\_\{s\}^\{\(t\+1\)\}denotecsc\_\{s\}at iterationsttandt\+1t\+1, respectively, andh\(t\)h^\{\(t\)\}denotesh𝒄h\_\{\\bm\{c\}\}at iterationtt\. Note thath\(0\)h^\{\(0\)\}is the initialized learner\. In matrix form, \([2\.6](https://arxiv.org/html/2610.41179#S2.E6)\) reads

𝒄\(t\+1\)=𝒄\(t\)−η​G​𝒄\(t\)\+η​\(⟨f,ϕr⟩Ω\)r=1n,\\bm\{c\}^\{\(t\+1\)\}=\\bm\{c\}^\{\(t\)\}\-\\eta G\\bm\{c\}^\{\(t\)\}\+\\eta\\bigl\(\\left\\langle f,\\phi\_\{r\}\\right\\rangle\_\{\\Omega\}\\bigr\)\_\{r=1\}^\{n\},\(2\.7\)whereGGis the Gram matrix with entriesGr​s=⟨ϕs,ϕr⟩ΩG\_\{rs\}=\\left\\langle\\phi\_\{s\},\\phi\_\{r\}\\right\\rangle\_\{\\Omega\}\.

Sincecs∈ℂc\_\{s\}\\in\\mathbb\{C\}, we writecs=ξs\+i​ζsc\_\{s\}=\\xi\_\{s\}\+\\mathrm\{i\}\\zeta\_\{s\}withξs,ζs∈ℝ\\xi\_\{s\},\\zeta\_\{s\}\\in\\mathbb\{R\}\. Differentiation with respect toξs,ζs\\xi\_\{s\},\\zeta\_\{s\}gives

∂J∂ξs=Re⁡⟨h𝒄−f,ϕs⟩Ω,∂J∂ζs=Im⁡⟨h𝒄−f,ϕs⟩Ω\.\\frac\{\\partial J\}\{\\partial\\xi\_\{s\}\}=\\operatorname\{Re\}\\left\\langle h\_\{\\bm\{c\}\}\-f,\\phi\_\{s\}\\right\\rangle\_\{\\Omega\},\\qquad\\frac\{\\partial J\}\{\\partial\\zeta\_\{s\}\}=\\operatorname\{Im\}\\left\\langle h\_\{\\bm\{c\}\}\-f,\\phi\_\{s\}\\right\\rangle\_\{\\Omega\}\.\(2\.8\)

### 2\.2Training operator and convergence

We express the Fourier error dynamics as an operator iteration and determine its limit\. Fors=1,…,ns=1,\\ldots,n, letϕsΛ=\(\(ϕs\)𝒌\)𝒌∈Λ\\phi\_\{s\}^\{\\Lambda\}=\(\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\)\_\{\\bm\{k\}\\in\\Lambda\}denote the lattice coefficients \([2\.3](https://arxiv.org/html/2610.41179#S2.E3)\) of the basis functions\. Define𝒮:ℓ2​\(Λ\)→ℓ2​\(Λ\)\\mathcal\{S\}:\\ell^\{2\}\(\\Lambda\)\\to\\ell^\{2\}\(\\Lambda\)by

\(𝒮​uΛ\)𝒌\\displaystyle\(\\mathcal\{S\}u^\{\\Lambda\}\)\_\{\\bm\{k\}\}=∑𝒍∈Λ𝒦𝒌​𝒍u𝒍,u∈L2\(Ω\),\\displaystyle=\\sum\_\{\\bm\{l\}\\in\\Lambda\}\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{l\}\}u\_\{\\bm\{l\}\},\\qquad u\\in L^\{2\}\(\\Omega\),\(2\.9\)𝒦𝒌​𝒍\\displaystyle\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{l\}\}=∑s=1n\(ϕs\)𝒌​\(ϕs\)𝒍¯,𝒌,𝒍∈Λ\.\\displaystyle=\\sum\_\{s=1\}^\{n\}\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\overline\{\(\\phi\_\{s\}\)\_\{\\bm\{l\}\}\},\\qquad\\bm\{k\},\\bm\{l\}\\in\\Lambda\.The kernel𝒦\\mathcal\{K\}is the Gram matrix of the sequencesϕ1Λ,…,ϕnΛ\\phi\_\{1\}^\{\\Lambda\},\\ldots,\\phi\_\{n\}^\{\\Lambda\}, indexed by frequency\. Equivalently,𝒮\\mathcal\{S\}represents the integral operator onL2​\(Ω\)L^\{2\}\(\\Omega\)with kernel∑s=1nϕs​\(𝒙\)​ϕs​\(𝒚\)¯\\sum\_\{s=1\}^\{n\}\\phi\_\{s\}\(\\bm\{x\}\)\\overline\{\\phi\_\{s\}\(\\bm\{y\}\)\}in the orthonormal basis\{χ𝒌\}\\\{\\chi\_\{\\bm\{k\}\}\\\}\. A finite\-rank singular value expansion gives the following spectral properties[Engl et al\. \(1996\)](https://arxiv.org/html/2610.41179#bib.bib18)\.

###### Proposition 2\.1\.

The operator𝒮\\mathcal\{S\}is finite\-rank, self\-adjoint, and positive semidefinite\. Moreover, for everyu∈L2​\(Ω\)u\\in L^\{2\}\(\\Omega\),

\(𝒮​uΛ\)𝒌=∑s=1n\(ϕs\)𝒌​⟨uΛ,ϕsΛ⟩Λ,⟨𝒮​uΛ,uΛ⟩Λ=∑s=1n\|⟨uΛ,ϕsΛ⟩Λ\|2\.\(\\mathcal\{S\}u^\{\\Lambda\}\)\_\{\\bm\{k\}\}=\\sum\_\{s=1\}^\{n\}\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\left\\langle u^\{\\Lambda\},\\phi\_\{s\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\},\\qquad\\left\\langle\\mathcal\{S\}u^\{\\Lambda\},u^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}=\\sum\_\{s=1\}^\{n\}\\left\|\\left\\langle u^\{\\Lambda\},\\phi\_\{s\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}\\right\|^\{2\}\.\(2\.10\)In particular,range⁡𝒮=\{vΛ:v∈V\}\\operatorname\{range\}\\mathcal\{S\}=\\\{v^\{\\Lambda\}:v\\in V\\\}, and the nonzero eigenvalues of𝒮\\mathcal\{S\}andGGcoincide\.

###### Proof\.

By \([2\.9](https://arxiv.org/html/2610.41179#S2.E9)\),𝒮​uΛ=∑s=1n⟨uΛ,ϕsΛ⟩Λ​ϕsΛ\\mathcal\{S\}u^\{\\Lambda\}=\\sum\_\{s=1\}^\{n\}\\left\\langle u^\{\\Lambda\},\\phi\_\{s\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}\\phi\_\{s\}^\{\\Lambda\}, which gives \([2\.10](https://arxiv.org/html/2610.41179#S2.E10)\)\. This representation shows that𝒮\\mathcal\{S\}is self\-adjoint andrange⁡𝒮⊂\{vΛ:v∈V\}\\operatorname\{range\}\\mathcal\{S\}\\subset\\\{v^\{\\Lambda\}:v\\in V\\\}, so𝒮\\mathcal\{S\}is finite\-rank\. The second identity in \([2\.10](https://arxiv.org/html/2610.41179#S2.E10)\) gives positive semidefiniteness\.

Note thatrank⁡G=n\\operatorname\{rank\}G=n, andλ1≥⋯≥λn\>0\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{n\}\>0be the eigenvalues ofGGwith orthonormal eigenvectors𝒗1,…,𝒗n∈ℂn\\bm\{v\}\_\{1\},\\ldots,\\bm\{v\}\_\{n\}\\in\\mathbb\{C\}^\{n\}\. By Gram\-Schmit process, we can write an orthonormal basis ofVV, i\.e\.\{q1,…,qn\}\\\{q\_\{1\},\\ldots,q\_\{n\}\\\}, where

qj:=λj−1/2∑s=1n\(𝒗j\)sϕs,j=1,…,n\.q\_\{j\}:=\\lambda\_\{j\}^\{\-1/2\}\\sum\_\{s=1\}^\{n\}\(\\bm\{v\}\_\{j\}\)\_\{s\}\\phi\_\{s\},\\qquad j=1,\\ldots,n\.\(2\.11\)Then \([2\.10](https://arxiv.org/html/2610.41179#S2.E10)\) and \([2\.4](https://arxiv.org/html/2610.41179#S2.E4)\) give𝒮qjΛ=λj−1/2∑s=1n\(G𝒗j\)sϕsΛ=λjqjΛ\\mathcal\{S\}q\_\{j\}^\{\\Lambda\}=\\lambda\_\{j\}^\{\-1/2\}\\sum\_\{s=1\}^\{n\}\(G\\bm\{v\}\_\{j\}\)\_\{s\}\\phi\_\{s\}^\{\\Lambda\}=\\lambda\_\{j\}q\_\{j\}^\{\\Lambda\}for alljj\. Also, it is trivial to see that𝒮\\mathcal\{S\}has no nonzero eigenvalues outside of\{λj\}j=1n\\\{\\lambda\_\{j\}\\\}\_\{j=1\}^\{n\}\. So,λ1,…,λn\\lambda\_\{1\},\\dots,\\lambda\_\{n\}are exactly the nonzero eigenvalues of𝒮\\mathcal\{S\}\. ∎

The coefficient update \([2\.6](https://arxiv.org/html/2610.41179#S2.E6)\) gives the following iteration for the Fourier error\.

###### Theorem 2\.2\.

Lete\(t\)=\(h\(t\)−f\)Λe^\{\(t\)\}=\(h^\{\(t\)\}\-f\)^\{\\Lambda\}be the sequence of lattice coefficients of the residualh\(t\)−fh^\{\(t\)\}\-fat iterationtt\. Then

e\(t\+1\)=\(I−η​𝒮\)​e\(t\),e^\{\(t\+1\)\}=\(I\-\\eta\\mathcal\{S\}\)e^\{\(t\)\},\(2\.12\)whereIIis the identity operator onℓ2​\(Λ\)\\ell^\{2\}\(\\Lambda\)\.

###### Proof\.

By \([2\.4](https://arxiv.org/html/2610.41179#S2.E4)\) and \([2\.6](https://arxiv.org/html/2610.41179#S2.E6)\),

cs\(t\+1\)=cs\(t\)−η​⟨h\(t\)−f,ϕs⟩Ω=cs\(t\)−η​⟨e\(t\),ϕsΛ⟩Λ\.c\_\{s\}^\{\(t\+1\)\}=c\_\{s\}^\{\(t\)\}\-\\eta\\left\\langle h^\{\(t\)\}\-f,\\phi\_\{s\}\\right\\rangle\_\{\\Omega\}=c\_\{s\}^\{\(t\)\}\-\\eta\\left\\langle e^\{\(t\)\},\\phi\_\{s\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}\.\(2\.13\)Therefore, for every𝒌∈Λ\\bm\{k\}\\in\\Lambda,

h𝒌\(t\+1\)=∑s=1ncs\(t\+1\)​\(ϕs\)𝒌=h𝒌\(t\)−η​∑s=1n\(ϕs\)𝒌​⟨e\(t\),ϕsΛ⟩Λ=h𝒌\(t\)−η​\(𝒮​e\(t\)\)𝒌\.h^\{\(t\+1\)\}\_\{\\bm\{k\}\}=\\sum\_\{s=1\}^\{n\}c\_\{s\}^\{\(t\+1\)\}\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}=h^\{\(t\)\}\_\{\\bm\{k\}\}\-\\eta\\sum\_\{s=1\}^\{n\}\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\left\\langle e^\{\(t\)\},\\phi\_\{s\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}=h^\{\(t\)\}\_\{\\bm\{k\}\}\-\\eta\(\\mathcal\{S\}e^\{\(t\)\}\)\_\{\\bm\{k\}\}\.\(2\.14\)Subtractingf𝒌f\_\{\\bm\{k\}\}givese𝒌\(t\+1\)=e𝒌\(t\)−η​\(𝒮​e\(t\)\)𝒌e^\{\(t\+1\)\}\_\{\\bm\{k\}\}=e^\{\(t\)\}\_\{\\bm\{k\}\}\-\\eta\(\\mathcal\{S\}e^\{\(t\)\}\)\_\{\\bm\{k\}\}, which is \([2\.12](https://arxiv.org/html/2610.41179#S2.E12)\)\. ∎

The convergence of gradient descent and the limit follow from the spectral decomposition given as follows\.

###### Theorem 2\.3\.

LetPRP\_\{R\}andPNP\_\{N\}denote the orthogonal projections onto the range and the null space of𝒮\\mathcal\{S\}, respectively\. If0<η<2λ10<\\eta<\\frac\{2\}\{\\lambda\_\{1\}\}, then

limt→∞PR​e\(t\)=0,PN​e\(t\)=PN​e\(0\)\.\\lim\_\{t\\to\\infty\}P\_\{R\}e^\{\(t\)\}=0,\\qquad P\_\{N\}e^\{\(t\)\}=P\_\{N\}e^\{\(0\)\}\.\(2\.15\)Moreover,e\(t\)e^\{\(t\)\}converges inℓ2​\(Λ\)\\ell^\{2\}\(\\Lambda\)to

e\(∞\)=−\(f−PV​f\)Λ\.e^\{\(\\infty\)\}=\-\\left\(f\-P\_\{V\}f\\right\)^\{\\Lambda\}\.\(2\.16\)In particular,e\(t\)→0e^\{\(t\)\}\\to 0if and only iff∈Vf\\in V\.

###### Proof\.

By \([2\.12](https://arxiv.org/html/2610.41179#S2.E12)\),e\(t\)=\(I−η​𝒮\)t​e\(0\)e^\{\(t\)\}=\(I\-\\eta\\mathcal\{S\}\)^\{t\}e^\{\(0\)\}\. Sinceq1Λ,…,qnΛq\_\{1\}^\{\\Lambda\},\\ldots,q\_\{n\}^\{\\Lambda\}form an orthonormal basis ofrange⁡𝒮\\operatorname\{range\}\\mathcal\{S\}andPN​𝒮=0P\_\{N\}\\mathcal\{S\}=0, we have

PR​e\(t\)=∑j=1n\(1−η​λj\)t​⟨e\(0\),qjΛ⟩Λ​qjΛ,PN​e\(t\)=PN​e\(0\)\.P\_\{R\}e^\{\(t\)\}=\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle e^\{\(0\)\},q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}q\_\{j\}^\{\\Lambda\},\\qquad P\_\{N\}e^\{\(t\)\}=P\_\{N\}e^\{\(0\)\}\.\(2\.17\)The condition0<η<2/λ10<\\eta<2/\\lambda\_\{1\}implies\|1−η​λj\|<1\\left\|1\-\\eta\\lambda\_\{j\}\\right\|<1for1≤j≤n1\\leq j\\leq n, which proves \([2\.15](https://arxiv.org/html/2610.41179#S2.E15)\)\. Sincerange⁡𝒮=\{vΛ:v∈V\}\\operatorname\{range\}\\mathcal\{S\}=\\\{v^\{\\Lambda\}:v\\in V\\\}, we havePR​wΛ=\(PV​w\)ΛP\_\{R\}w^\{\\Lambda\}=\(P\_\{V\}w\)^\{\\Lambda\}andPN​wΛ=\(\(I−PV\)​w\)ΛP\_\{N\}w^\{\\Lambda\}=\(\(I\-P\_\{V\}\)w\)^\{\\Lambda\}for everyw∈L2​\(Ω\)w\\in L^\{2\}\(\\Omega\)\. Finally,h\(0\)∈Vh^\{\(0\)\}\\in VgivesPN​e\(0\)=−\(\(I−PV\)​f\)ΛP\_\{N\}e^\{\(0\)\}=\-\(\(I\-P\_\{V\}\)f\)^\{\\Lambda\}, which proves \([2\.16](https://arxiv.org/html/2610.41179#S2.E16)\)\. ∎

The functionsqjq\_\{j\}are generally not Fourier modes\. We therefore write the iteration \([2\.12](https://arxiv.org/html/2610.41179#S2.E12)\) at a fixed frequency and separate same\-frequency damping from frequency mixing\.

For a Fourier frequency𝒌∈Λ\\bm\{k\}\\in\\Lambda, define the functionalℳ𝒌:ℓ2​\(Λ\)→ℂ\\mathcal\{M\}\_\{\\bm\{k\}\}:\\ell^\{2\}\(\\Lambda\)\\rightarrow\\mathbb\{C\}by

ℳ𝒌​uΛ=\(𝒮​uΛ\)𝒌−𝒦𝒌​𝒌​u𝒌=∑𝒍≠𝒌𝒦𝒌​𝒍​u𝒍,u∈L2​\(Ω\),\\mathcal\{M\}\_\{\\bm\{k\}\}u^\{\\Lambda\}=\(\\mathcal\{S\}u^\{\\Lambda\}\)\_\{\\bm\{k\}\}\-\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}u\_\{\\bm\{k\}\}=\\sum\_\{\\bm\{l\}\\neq\\bm\{k\}\}\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{l\}\}u\_\{\\bm\{l\}\},\\qquad u\\in L^\{2\}\(\\Omega\),\(2\.18\)where𝒦𝒌​𝒌\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}acts as the damping coefficient at frequency𝒌\\bm\{k\}, andℳ𝒌​e\(t\)\\mathcal\{M\}\_\{\\bm\{k\}\}e^\{\(t\)\}is the mixing contribution from the other residual modes\. Equation \([2\.12](https://arxiv.org/html/2610.41179#S2.E12)\) gives the exact pointwise decomposition

e𝒌\(t\+1\)=\(1−η​𝒦𝒌​𝒌\)​e𝒌\(t\)−η​ℳ𝒌​e\(t\)\.e\_\{\\bm\{k\}\}^\{\(t\+1\)\}=\(1\-\\eta\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}\)e\_\{\\bm\{k\}\}^\{\(t\)\}\-\\eta\\mathcal\{M\}\_\{\\bm\{k\}\}e^\{\(t\)\}\.\(2\.19\)Here\(1−η​𝒦𝒌​𝒌\)​e𝒌\(t\)\(1\-\\eta\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}\)e\_\{\\bm\{k\}\}^\{\(t\)\}is the same\-frequency contribution, and−η​ℳ𝒌​e\(t\)\-\\eta\\mathcal\{M\}\_\{\\bm\{k\}\}e^\{\(t\)\}accounts for frequency mixing\. Since the lattice modes form an orthonormal basis, \([2\.19](https://arxiv.org/html/2610.41179#S2.E19)\) describes the exact training dynamics without discretization\. The Fourier dynamics are diagonal only if𝒦𝒌​𝒍=0\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{l\}\}=0for𝒌≠𝒍\\bm\{k\}\\neq\\bm\{l\}, equivalently, if every lattice mode is an eigenvector of𝒮\\mathcal\{S\}\.

### 2\.3Capacity and local spectral measures

To distinguish representational capacity from learning speed, we use the orthonormal basis\{qj\}\\\{q\_\{j\}\\\}ofVVfrom Proposition[2\.1](https://arxiv.org/html/2610.41179#S2.Thmtheorem1)and define the capacity at frequency𝒌\\bm\{k\}by

ℓ𝒌=‖PV​χ𝒌‖L2​\(Ω\)2=∑j=1n\|\(qj\)𝒌\|2𝒌∈Λ,\\ell\_\{\\bm\{k\}\}=\\left\\\|P\_\{V\}\\chi\_\{\\bm\{k\}\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}^\{2\}=\\sum\_\{j=1\}^\{n\}\\left\|\(q\_\{j\}\)\_\{\\bm\{k\}\}\\right\|^\{2\}\\qquad\\bm\{k\}\\in\\Lambda,\(2\.20\)where\(qj\)𝒌=⟨qj,χ𝒌⟩Ω\(q\_\{j\}\)\_\{\\bm\{k\}\}=\\left\\langle q\_\{j\},\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}\. The capacity satisfies0≤ℓ𝒌≤10\\leq\\ell\_\{\\bm\{k\}\}\\leq 1\. Sinceq1Λ,…,qnΛq\_\{1\}^\{\\Lambda\},\\ldots,q\_\{n\}^\{\\Lambda\}form an orthonormal basis ofrange⁡𝒮\\operatorname\{range\}\\mathcal\{S\}and satisfy𝒮​qjΛ=λj​qjΛ\\mathcal\{S\}q\_\{j\}^\{\\Lambda\}=\\lambda\_\{j\}q\_\{j\}^\{\\Lambda\}, the self\-adjointness of𝒮\\mathcal\{S\}yields

𝒮​w=∑j=1nλj​⟨w,qjΛ⟩Λ​qjΛ,w∈ℓ2​\(Λ\)\.\\mathcal\{S\}w=\\sum\_\{j=1\}^\{n\}\\lambda\_\{j\}\\left\\langle w,q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}q\_\{j\}^\{\\Lambda\},\\qquad w\\in\\ell^\{2\}\(\\Lambda\)\.\(2\.21\)Applying \([2\.9](https://arxiv.org/html/2610.41179#S2.E9)\), \([2\.10](https://arxiv.org/html/2610.41179#S2.E10)\) and \([2\.21](https://arxiv.org/html/2610.41179#S2.E21)\) to the unit vectore𝒌∈ℓ2​\(Λ\)e\_\{\\bm\{k\}\}\\in\\ell^\{2\}\(\\Lambda\)gives

𝒦𝒌​𝒌=∑s=1n\|\(ϕs\)𝒌\|2=⟨𝒮​e𝒌,e𝒌⟩Λ=∑j=1nλj​\|\(qj\)𝒌\|2\.\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}=\\sum\_\{s=1\}^\{n\}\\left\|\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\right\|^\{2\}=\\left\\langle\\mathcal\{S\}e\_\{\\bm\{k\}\},e\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Lambda\}=\\sum\_\{j=1\}^\{n\}\\lambda\_\{j\}\\left\|\(q\_\{j\}\)\_\{\\bm\{k\}\}\\right\|^\{2\}\.\(2\.22\)Hence

λn​ℓ𝒌≤𝒦𝒌​𝒌≤λ1​ℓ𝒌\.\\lambda\_\{n\}\\ell\_\{\\bm\{k\}\}\\leq\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}\\leq\\lambda\_\{1\}\\ell\_\{\\bm\{k\}\}\.\(2\.23\)Forℓ𝒌\>0\\ell\_\{\\bm\{k\}\}\>0, we define the local spectral measure of𝒌\\bm\{k\}through the weights

ϖj\(𝒌\)=\|\(qj\)𝒌\|2ℓ𝒌,j=1,…,n\.\\varpi\_\{j\}\(\\bm\{k\}\)=\\frac\{\\left\|\(q\_\{j\}\)\_\{\\bm\{k\}\}\\right\|^\{2\}\}\{\\ell\_\{\\bm\{k\}\}\},\\qquad j=1,\\ldots,n\.\(2\.24\)By \([2\.20](https://arxiv.org/html/2610.41179#S2.E20)\), these weights are nonnegative and sum to one\. We call their mean

λ¯𝒌=∑j=1nϖj​\(𝒌\)​λj=𝒦𝒌​𝒌ℓ𝒌\\bar\{\\lambda\}\_\{\\bm\{k\}\}=\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(\\bm\{k\}\)\\lambda\_\{j\}=\\frac\{\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}\}\{\\ell\_\{\\bm\{k\}\}\}\(2\.25\)the speed at frequency𝒌\\bm\{k\}\. The second equality follows from \([2\.22](https://arxiv.org/html/2610.41179#S2.E22)\)\. Thus the same\-frequency damping coefficient in \([2\.19](https://arxiv.org/html/2610.41179#S2.E19)\) factors as

𝒦𝒌​𝒌=ℓ𝒌​λ¯𝒌\.\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}=\\ell\_\{\\bm\{k\}\}\\,\\bar\{\\lambda\}\_\{\\bm\{k\}\}\.\(2\.26\)The speedλ¯𝒌\\bar\{\\lambda\}\_\{\\bm\{k\}\}measures the average learning rate of the represented part\. Note that∂h𝒄∂cs=ϕs\\frac\{\\partial h\_\{\\bm\{c\}\}\}\{\\partial c\_\{s\}\}=\\phi\_\{s\}, the coefficient𝒦𝒌​𝒌\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}is the sum of the squared moduli of the derivatives of\(h𝒄\)𝒌\(h\_\{\\bm\{c\}\}\)\_\{\\bm\{k\}\}with respect tocsc\_\{s\}\. Its high\-frequency decay underlies the explanation in[Rahaman et al\. \(2019\)](https://arxiv.org/html/2610.41179#bib.bib1)\. The factorization \([2\.26](https://arxiv.org/html/2610.41179#S2.E26)\) separates this decay into effects on capacity and speed\.

### 2\.4Boundary traces and capacity bounds

We begin with an integration\-by\-parts estimate relating the order of the first nonzero boundary trace to the Fourier decay rate\. Forj∈\{1,…,d\}j\\in\\\{1,\\ldots,d\\\}and𝒙∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}, let𝒙j′\\bm\{x\}^\{\\prime\}\_\{j\}denote the vector obtained from𝒙\\bm\{x\}by removing itsjj\-th entry, and setΩj′=\[a,b\]d−1\\Omega^\{\\prime\}\_\{j\}=\[a,b\]^\{d\-1\}\. Fort∈\[a,b\]t\\in\[a,b\]and𝒙j′∈Ωj′\\bm\{x\}^\{\\prime\}\_\{j\}\\in\\Omega^\{\\prime\}\_\{j\}, we use the shorthand

u⁡\(t,𝒙j′\):=u⁡\(x1,…,xj−1,t,xj\+1,…,xd\),u\(t,\\bm\{x\}^\{\\prime\}\_\{j\}\):=u\(x\_\{1\},\\ldots,x\_\{j\-1\},t,x\_\{j\+1\},\\ldots,x\_\{d\}\),\(2\.27\)which replaces thejj\-th coordinate byttwhile keeping the other coordinates unchanged\. We apply one\-dimensional integration by parts inxjx\_\{j\}with the remaining variables fixed\. For a nonnegative integermm, letWm,1​\(Ω\)W^\{m,1\}\(\\Omega\)denote the Sobolev space of functions inL1​\(Ω\)L^\{1\}\(\\Omega\)whose weak derivatives up to ordermmalso belong toL1​\(Ω\)L^\{1\}\(\\Omega\)\.

###### Lemma 2\.4\.

Letν∈ℕ0=\{0,1,2,…\}\\nu\\in\\mathbb\{N\}\_\{0\}=\\\{0,1,2,\\ldots\\\}andu∈Wν\+1,1​\(Ω\)u\\in W^\{\\nu\+1,1\}\(\\Omega\)\. Assume that, for everyj∈\{1,…,d\}j\\in\\\{1,\\ldots,d\\\}and every integer0≤p<ν0\\leq p<\\nu,

∂xjpu⁡\(a,𝒙j′\)=∂xjpu⁡\(b,𝒙j′\)=0for almost every​𝒙j′∈Ωj′\.\\partial\_\{x\_\{j\}\}^\{p\}u\(a,\\bm\{x\}^\{\\prime\}\_\{j\}\)=\\partial\_\{x\_\{j\}\}^\{p\}u\(b,\\bm\{x\}^\{\\prime\}\_\{j\}\)=0\\qquad\\text\{for almost every \}\\bm\{x\}^\{\\prime\}\_\{j\}\\in\\Omega^\{\\prime\}\_\{j\}\.\(2\.28\)Then there isBu\>0B\_\{u\}\>0, depending only onuu, such that

\|u^​\(𝒌\)\|≤Bu​‖𝒌‖−\(ν\+1\),𝒌∈ℝd∖\{0\}\.\\left\|\\widehat\{u\}\(\\bm\{k\}\)\\right\|\\leq B\_\{u\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\},\\qquad\\bm\{k\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{0\\\}\.\(2\.29\)

###### Proof\.

Let𝒌≠0\\bm\{k\}\\neq 0and choosejjsuch that\|kj\|=‖𝒌‖∞\\left\|k\_\{j\}\\right\|=\\left\\\|\\bm\{k\}\\right\\\|\_\{\\infty\}\. Sinceu∈Wν\+1,1​\(Ω\)u\\in W^\{\\nu\+1,1\}\(\\Omega\), for almost every fixed𝒙j′∈Ωj′\\bm\{x\}^\{\\prime\}\_\{j\}\\in\\Omega^\{\\prime\}\_\{j\}the functionsxj↦∂xjpu⁡\(𝒙\)x\_\{j\}\\mapsto\\partial\_\{x\_\{j\}\}^\{p\}u\(\\bm\{x\}\),0≤p≤ν0\\leq p\\leq\\nu, are absolutely continuous on\[a,b\]\[a,b\]after a change on a null set\. Their derivatives arexj↦∂xjp\+1u⁡\(𝒙\)x\_\{j\}\\mapsto\\partial\_\{x\_\{j\}\}^\{p\+1\}u\(\\bm\{x\}\), and their values ataaandbbequal the traces of∂xjpu\\partial\_\{x\_\{j\}\}^\{p\}u\. For such𝒙j′\\bm\{x\}^\{\\prime\}\_\{j\}, we integrate by partsν\+1\\nu\+1times inxjx\_\{j\}:

∫abu⁡\(𝒙\)​e−i​kj​xj​d​xj=∑p=0ν\(−1\)p\(−i​kj\)p\+1​\[∂xjpu⁡\(𝒙\)​e−i​kj​xj\]xj=axj=b\+\(−1\)ν\+1\(−i​kj\)ν\+1∫ab∂xjν\+1u\(𝒙\)e−i​kj​xjdxj\.\\int\_\{a\}^\{b\}u\(\\bm\{x\}\)\\mathrm\{e\}^\{\-\\mathrm\{i\}k\_\{j\}x\_\{j\}\}~\\mathrm\{d\}x\_\{j\}=\\sum\_\{p=0\}^\{\\nu\}\\frac\{\(\-1\)^\{p\}\}\{\(\-\\mathrm\{i\}k\_\{j\}\)^\{p\+1\}\}\\Bigl\[\\partial\_\{x\_\{j\}\}^\{p\}u\(\\bm\{x\}\)\\mathrm\{e\}^\{\-\\mathrm\{i\}k\_\{j\}x\_\{j\}\}\\Bigr\]\_\{x\_\{j\}=a\}^\{x\_\{j\}=b\}\\\\ \+\\frac\{\(\-1\)^\{\\nu\+1\}\}\{\(\-\\mathrm\{i\}k\_\{j\}\)^\{\\nu\+1\}\}\\int\_\{a\}^\{b\}\\partial\_\{x\_\{j\}\}^\{\\nu\+1\}u\(\\bm\{x\}\)\\mathrm\{e\}^\{\-\\mathrm\{i\}k\_\{j\}x\_\{j\}\}~\\mathrm\{d\}x\_\{j\}\.\(2\.30\)The boundary terms withp<νp<\\nuvanish by \([2\.28](https://arxiv.org/html/2610.41179#S2.E28)\)\. Taking absolute values of the remaining terms and integrating overΩj′\\Omega^\{\\prime\}\_\{j\}gives

\|u^​\(𝒌\)\|≤\(2π\)−d/2\|kj\|ν\+1​\(∫Ωj′\|∂xjνu⁡\(b,𝒙j′\)\|\+\|∂xjνu⁡\(a,𝒙j′\)\|​d​𝒙j′\+‖∂xjν\+1u‖L1​\(Ω\)\)≤Cu​\|kj\|−\(ν\+1\),\\left\|\\widehat\{u\}\(\\bm\{k\}\)\\right\|\\leq\\frac\{\(2\\pi\)^\{\-d/2\}\}\{\\left\|k\_\{j\}\\right\|^\{\\nu\+1\}\}\\left\(\\int\_\{\\Omega^\{\\prime\}\_\{j\}\}\\left\|\\partial\_\{x\_\{j\}\}^\{\\nu\}u\(b,\\bm\{x\}^\{\\prime\}\_\{j\}\)\\right\|\+\\left\|\\partial\_\{x\_\{j\}\}^\{\\nu\}u\(a,\\bm\{x\}^\{\\prime\}\_\{j\}\)\\right\|~\\mathrm\{d\}\\bm\{x\}^\{\\prime\}\_\{j\}\+\\left\\\|\\partial\_\{x\_\{j\}\}^\{\\nu\+1\}u\\right\\\|\_\{L^\{1\}\(\\Omega\)\}\\right\)\\\\ \\leq C\_\{u\}\\left\|k\_\{j\}\\right\|^\{\-\(\\nu\+1\)\},\(2\.31\)whereCuC\_\{u\}is\(2π\)−d/2\(2\\pi\)^\{\-d/2\}times the maximum overj=1,…,dj=1,\\ldots,dof the expression in parentheses\. Since\|kj\|=‖𝒌‖∞≥d−1/2‖𝒌‖\\left\|k\_\{j\}\\right\|=\\left\\\|\\bm\{k\}\\right\\\|\_\{\\infty\}\\geq d^\{\-1/2\}\\left\\\|\\bm\{k\}\\right\\\|, we have

\|u^​\(𝒌\)\|≤Cu​d\(ν\+1\)/2​‖𝒌‖−\(ν\+1\)\.\\left\|\\widehat\{u\}\(\\bm\{k\}\)\\right\|\\leq C\_\{u\}d^\{\(\\nu\+1\)/2\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\}\.\(2\.32\)SettingBu=Cu​d\(ν\+1\)/2B\_\{u\}=C\_\{u\}d^\{\(\\nu\+1\)/2\}proves \([2\.29](https://arxiv.org/html/2610.41179#S2.E29)\)\. ∎

The next proposition uses basis\-function decay to bound the capacity \([2\.20](https://arxiv.org/html/2610.41179#S2.E20)\) and the change of high\-frequency modes uniformly in the training time\.

###### Proposition 2\.5\.

Fixν∈ℕ0\\nu\\in\\mathbb\{N\}\_\{0\}and a set𝒜⊂Λ∖\{0\}\\mathcal\{A\}\\subset\\Lambda\\setminus\\\{0\\\}\. Assume that

\|\(ϕs\)𝒌\|≤Bϕ‖𝒌‖−\(ν\+1\),s=1,…,n,𝒌∈𝒜,\\left\|\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\right\|\\leq B\_\{\\phi\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\},\\qquad s=1,\\ldots,n,\\quad\\bm\{k\}\\in\\mathcal\{A\},\(2\.33\)whereBϕ\>0B\_\{\\phi\}\>0is independent of𝐤\\bm\{k\}\. Then

𝒦𝒌​𝒌≤n​Bϕ2​‖𝒌‖−2​\(ν\+1\),ℓ𝒌≤n​Bϕ2λn​‖𝒌‖−2​\(ν\+1\),𝒌∈𝒜\.\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}\\leq nB\_\{\\phi\}^\{2\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-2\(\\nu\+1\)\},\\qquad\\ell\_\{\\bm\{k\}\}\\leq\\frac\{nB\_\{\\phi\}^\{2\}\}\{\\lambda\_\{n\}\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-2\(\\nu\+1\)\},\\qquad\\bm\{k\}\\in\\mathcal\{A\}\.\(2\.34\)Moreover, ifh\(0\)=0h^\{\(0\)\}=0and0<η<2/λ10<\\eta<2/\\lambda\_\{1\}, then, for everyt∈ℕ0t\\in\\mathbb\{N\}\_\{0\},

\|e𝒌\(t\)−e𝒌\(0\)\|≤2​‖f‖L2​\(Ω\)​Bϕ​nλn​‖𝒌‖−\(ν\+1\),𝒌∈𝒜\.\\left\|e\_\{\\bm\{k\}\}^\{\(t\)\}\-e\_\{\\bm\{k\}\}^\{\(0\)\}\\right\|\\leq 2\\left\\\|f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}B\_\{\\phi\}\\sqrt\{\\frac\{n\}\{\\lambda\_\{n\}\}\}\\,\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\},\\qquad\\bm\{k\}\\in\\mathcal\{A\}\.\(2\.35\)

###### Proof\.

The first inequality in \([2\.34](https://arxiv.org/html/2610.41179#S2.E34)\) follows from \([2\.22](https://arxiv.org/html/2610.41179#S2.E22)\) and \([2\.33](https://arxiv.org/html/2610.41179#S2.E33)\)\. The second follows fromℓ𝒌≤𝒦𝒌​𝒌/λn\\ell\_\{\\bm\{k\}\}\\leq\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}/\\lambda\_\{n\}\.

By the proof of Theorem[2\.3](https://arxiv.org/html/2610.41179#S2.Thmtheorem3),\(h\(t\)−PV​f\)Λ=PR​e\(t\)\(h^\{\(t\)\}\-P\_\{V\}f\)^\{\\Lambda\}=P\_\{R\}e^\{\(t\)\}\. According to \([2\.17](https://arxiv.org/html/2610.41179#S2.E17)\) and the orthonormality ofqjΛq\_\{j\}^\{\\Lambda\},

‖PR​e\(t\)‖ℓ2​\(Λ\)\\displaystyle\\left\\\|P\_\{R\}e^\{\(t\)\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}=‖∑j=1n\(1−η​λj\)t​⟨e\(0\),qjΛ⟩Λ​qjΛ‖ℓ2​\(Λ\)\\displaystyle=\\left\\\|\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle e^\{\(0\)\},q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}q\_\{j\}^\{\\Lambda\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}\(2\.36\)=\(∑j=1n\|1−η​λj\|2​t​\|⟨e\(0\),qjΛ⟩Λ\|2\)1/2\\displaystyle=\\left\(\\sum\_\{j=1\}^\{n\}\\left\|1\-\\eta\\lambda\_\{j\}\\right\|^\{2t\}\\left\|\\left\\langle e^\{\(0\)\},q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}\\right\|^\{2\}\\right\)^\{1/2\}≤\(∑j=1n\|⟨e\(0\),qjΛ⟩Λ\|2\)1/2\\displaystyle\\leq\\left\(\\sum\_\{j=1\}^\{n\}\\left\|\\left\\langle e^\{\(0\)\},q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}\\right\|^\{2\}\\right\)^\{1/2\}=‖PR​e\(0\)‖ℓ2​\(Λ\)\.\\displaystyle=\\left\\\|P\_\{R\}e^\{\(0\)\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}\.Hence, by Parseval’s identity \([2\.4](https://arxiv.org/html/2610.41179#S2.E4)\) andh\(0\)=0h^\{\(0\)\}=0,

‖h\(t\)−PV​f‖L2​\(Ω\)\\displaystyle\\left\\\|h^\{\(t\)\}\-P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}=\(∑𝒌∈Λ\|⟨h\(t\)−PV​f,χ𝒌⟩Ω\|2\)1/2\\displaystyle=\\left\(\\sum\_\{\\bm\{k\}\\in\\Lambda\}\\left\|\\left\\langle h^\{\(t\)\}\-P\_\{V\}f,\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}\\right\|^\{2\}\\right\)^\{1/2\}\(2\.37\)=‖\(h\(t\)−PV​f\)Λ‖ℓ2​\(Λ\)=‖PR​e\(t\)‖ℓ2​\(Λ\)\\displaystyle=\\left\\\|\(h^\{\(t\)\}\-P\_\{V\}f\)^\{\\Lambda\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}=\\left\\\|P\_\{R\}e^\{\(t\)\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}≤‖PR​e\(0\)‖ℓ2​\(Λ\)=‖\(h\(0\)−PV​f\)Λ‖ℓ2​\(Λ\)\\displaystyle\\leq\\left\\\|P\_\{R\}e^\{\(0\)\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}=\\left\\\|\(h^\{\(0\)\}\-P\_\{V\}f\)^\{\\Lambda\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\}=‖h\(0\)−PV​f‖L2​\(Ω\)=‖PV​f‖L2​\(Ω\)\.\\displaystyle=\\left\\\|h^\{\(0\)\}\-P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}=\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\.Consequently,

‖h\(t\)‖L2​\(Ω\)≤‖PV​f‖L2​\(Ω\)\+‖h\(t\)−PV​f‖L2​\(Ω\)≤2​‖PV​f‖L2​\(Ω\)≤2​‖f‖L2​\(Ω\),t∈ℕ0\.\\left\\\|h^\{\(t\)\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\leq\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\+\\left\\\|h^\{\(t\)\}\-P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\leq 2\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\leq 2\\left\\\|f\\right\\\|\_\{L^\{2\}\(\\Omega\)\},\\qquad t\\in\\mathbb\{N\}\_\{0\}\.\(2\.38\)Sinceh\(0\)=0h^\{\(0\)\}=0, we have

\|e𝒌\(t\)−e𝒌\(0\)\|\\displaystyle\\left\|e\_\{\\bm\{k\}\}^\{\(t\)\}\-e\_\{\\bm\{k\}\}^\{\(0\)\}\\right\|=\|⟨h\(t\),χ𝒌⟩Ω\|=\|⟨h\(t\),PV​χ𝒌⟩Ω\|\\displaystyle=\\left\|\\left\\langle h^\{\(t\)\},\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}\\right\|=\\left\|\\left\\langle h^\{\(t\)\},P\_\{V\}\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}\\right\|\(2\.39\)≤‖h\(t\)‖L2​\(Ω\)​‖PV​χ𝒌‖L2​\(Ω\)\\displaystyle\\leq\\left\\\|h^\{\(t\)\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\left\\\|P\_\{V\}\\chi\_\{\\bm\{k\}\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}=‖h\(t\)‖L2​\(Ω\)​ℓ𝒌\\displaystyle=\\left\\\|h^\{\(t\)\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\sqrt\{\\ell\_\{\\bm\{k\}\}\}≤2​‖f‖L2​\(Ω\)​Bϕ​nλn​‖𝒌‖−\(ν\+1\),\\displaystyle\\leq 2\\left\\\|f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}B\_\{\\phi\}\\sqrt\{\\frac\{n\}\{\\lambda\_\{n\}\}\}\\,\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\},This proves \([2\.35](https://arxiv.org/html/2610.41179#S2.E35)\)\. ∎

The spectral condition \([2\.33](https://arxiv.org/html/2610.41179#S2.E33)\) admit many familiar examples\. To illustrate the basis and target assumptions, considerΩ=\[0,1\]\\Omega=\[0,1\]andΛ=2​π​ℤ\\Lambda=2\\pi\\mathbb\{Z\}\. Any fixed finite family of polynomials, including Legendre and Chebyshev polynomials, or fixed Gaussian functions restricted to this interval satisfies \([3\.2](https://arxiv.org/html/2610.41179#S3.E2)\) withν=0\\nu=0, with a constant depending on the chosen family\.

Proposition[2\.5](https://arxiv.org/html/2610.41179#S2.Thmtheorem5)gives a bound uniform intt, including the limitt→∞t\\to\\infty, without further regularity assumptions on the targetff\. The upper bound in \([2\.35](https://arxiv.org/html/2610.41179#S2.E35)\) is independent ofttand depends on𝒌\\bm\{k\}through‖𝒌‖−\(ν\+1\)\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\}, but it does not determine which frequency will be learned first\. Moreover, the potential large factor\(n/λn\)1/2\(n/\\lambda\_\{n\}\)^\{1/2\}can make the bound large at moderate frequencies\.

### 2\.5Spectral representation of the attainable error

The next proposition gives a closed\-form error for each frequency𝒌\\bm\{k\}\. Recall thate\(∞\)e^\{\(\\infty\)\}given by \([2\.16](https://arxiv.org/html/2610.41179#S2.E16)\) is best error we can obtain when usingVVas the approximation space, ande\(t\)−e\(∞\)e^\{\(t\)\}\-e^\{\(\\infty\)\}measures the gap between the current approximator and the best one\.

###### Proposition 2\.6\.

Let0<η<2/λ10<\\eta<2/\\lambda\_\{1\}\. Then, for everyt∈ℕ0t\\in\\mathbb\{N\}\_\{0\}and every𝐤∈Λ\\bm\{k\}\\in\\Lambda,

e𝒌\(t\)−e𝒌\(∞\)=∑j=1n\(1−η​λj\)t​⟨h\(0\)−f,qj⟩Ω​\(qj\)𝒌\.e\_\{\\bm\{k\}\}^\{\(t\)\}\-e\_\{\\bm\{k\}\}^\{\(\\infty\)\}=\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle h^\{\(0\)\}\-f,q\_\{j\}\\right\\rangle\_\{\\Omega\}\(q\_\{j\}\)\_\{\\bm\{k\}\}\.\(2\.40\)

###### Proof\.

By the proof of Theorem[2\.3](https://arxiv.org/html/2610.41179#S2.Thmtheorem3),e\(∞\)=PN​e\(0\)=PN​e\(t\)e^\{\(\\infty\)\}=P\_\{N\}e^\{\(0\)\}=P\_\{N\}e^\{\(t\)\}\. Hencee\(t\)−e\(∞\)=PR​e\(t\)e^\{\(t\)\}\-e^\{\(\\infty\)\}=P\_\{R\}e^\{\(t\)\}, and \([2\.17](https://arxiv.org/html/2610.41179#S2.E17)\) gives

e\(t\)−e\(∞\)=∑j=1n\(1−η​λj\)t​⟨e\(0\),qjΛ⟩Λ​qjΛ\.e^\{\(t\)\}\-e^\{\(\\infty\)\}=\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle e^\{\(0\)\},q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}q\_\{j\}^\{\\Lambda\}\.\(2\.41\)By \([2\.4](https://arxiv.org/html/2610.41179#S2.E4)\),⟨e\(0\),qjΛ⟩Λ=⟨h\(0\)−f,qj⟩Ω\\left\\langle e^\{\(0\)\},q\_\{j\}^\{\\Lambda\}\\right\\rangle\_\{\\Lambda\}=\\left\\langle h^\{\(0\)\}\-f,q\_\{j\}\\right\\rangle\_\{\\Omega\}\. Taking the entry at𝒌\\bm\{k\}gives \([2\.40](https://arxiv.org/html/2610.41179#S2.E40)\)\. ∎

Formula \([2\.40](https://arxiv.org/html/2610.41179#S2.E40)\) shows that the distance betweene𝒌\(t\)e^\{\(t\)\}\_\{\\bm\{k\}\}ande𝒌\(∞\)e^\{\(\\infty\)\}\_\{\\bm\{k\}\}consist of two parts:\(1−η​λj\)t\(1\-\\eta\\lambda\_\{j\}\)^\{t\}involves the time and\(qj\)𝒌\(q\_\{j\}\)\_\{\\bm\{k\}\}is the𝒌\\bm\{k\}\-th Fourier coefficient ofqjq\_\{j\}\. Thus𝒌\\bm\{k\}is learned slowly when the part ofχ𝒌\\chi\_\{\\bm\{k\}\}inVVlies mainly along eigenfunctionsqjq\_\{j\}with small eigenvaluesλj\\lambda\_\{j\}\.

## 3High\-frequency movement and representational limits

### 3\.1Bounds on high\-frequency relative changes

We bound high\-frequency changes in the Fourier error relative to its initial amplitude\. Under suitable Fourier decay assumptions, the bounds hold at each fixed iteration and, for stable gradient descent, uniformly throughout training\. The uniform bound distinguishes limited approximation capacity from slow convergence\.

Let𝒜⊂\{𝒌∈Λ:‖𝒌‖≥1\}\\mathcal\{A\}\\subset\\\{\\bm\{k\}\\in\\Lambda:\\left\\\|\\bm\{k\}\\right\\\|\\geq 1\\\}be unbounded\. For zero initialization,e𝒌\(0\)=−f𝒌e\_\{\\bm\{k\}\}^\{\(0\)\}=\-f\_\{\\bm\{k\}\}\. At some frequency𝒌\\bm\{k\}wheref𝒌≠0f\_\{\\bm\{k\}\}\\neq 0, define the relative change during the training by

Γt​\(𝒌\):=\|h𝒌\(t\)\|\|f𝒌\|=\|h𝒌\(t\)−h𝒌\(0\)\|\|f𝒌\|=\|e𝒌\(t\)−e𝒌\(0\)\|\|e𝒌\(0\)\|\.\\Gamma\_\{t\}\(\\bm\{k\}\):=\\frac\{\\left\|h\_\{\\bm\{k\}\}^\{\(t\)\}\\right\|\}\{\\left\|f\_\{\\bm\{k\}\}\\right\|\}=\\frac\{\\left\|h\_\{\\bm\{k\}\}^\{\(t\)\}\-h\_\{\\bm\{k\}\}^\{\(0\)\}\\right\|\}\{\\left\|f\_\{\\bm\{k\}\}\\right\|\}=\\frac\{\\left\|e\_\{\\bm\{k\}\}^\{\(t\)\}\-e\_\{\\bm\{k\}\}^\{\(0\)\}\\right\|\}\{\\left\|e\_\{\\bm\{k\}\}^\{\(0\)\}\\right\|\}\.\(3\.1\)This quantity measures how far the current learner is from initialization at frequency𝒌\\bm\{k\}\. Intuitively, smallΓt​\(𝒌\)\\Gamma\_\{t\}\(\\bm\{k\}\)means the learner is trained slowly at this frequency\. From the third equality in \([3\.1](https://arxiv.org/html/2610.41179#S3.E1)\), smallΓt​\(𝒌\)\\Gamma\_\{t\}\(\\bm\{k\}\)also means that the Fourier error remains close to its initial value on this relative scale, but it does not assert that the mode is close to its attainable limit\. Moreover, the special caseΓt​\(𝒌\)=1\\Gamma\_\{t\}\(\\bm\{k\}\)=1means equal amplitudes ofh𝒌\(t\)h\_\{\\bm\{k\}\}^\{\(t\)\}andf𝒌f\_\{\\bm\{k\}\}, but does not ensure the accurate learning, i\.e\.,h𝒌\(t\)=f𝒌h\_\{\\bm\{k\}\}^\{\(t\)\}=f\_\{\\bm\{k\}\}\.

###### Theorem 3\.1\.

Letν∈ℕ0\\nu\\in\\mathbb\{N\}\_\{0\}and suppose that, for someBϕ\>0B\_\{\\phi\}\>0,

\|\(ϕs\)𝒌\|≤Bϕ‖𝒌‖−\(ν\+1\),s=1,…,n,𝒌∈𝒜\.\\left\|\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\right\|\\leq B\_\{\\phi\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\},\\qquad s=1,\\ldots,n,\\quad\\bm\{k\}\\in\\mathcal\{A\}\.\(3\.2\)Assume also that there are0<α<ν\+10<\\alpha<\\nu\+1and0<cf≤Cf0<c\_\{f\}\\leq C\_\{f\}such that

cf​‖𝒌‖−α≤\|f𝒌\|≤Cf​‖𝒌‖−α,𝒌∈𝒜\.c\_\{f\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\\alpha\}\\leq\\left\|f\_\{\\bm\{k\}\}\\right\|\\leq C\_\{f\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\\alpha\},\\qquad\\bm\{k\}\\in\\mathcal\{A\}\.\(3\.3\)Leth\(0\)=0h^\{\(0\)\}=0andη\>0\\eta\>0\. For every fixed integert≥1t\\geq 1, there are constantsC1,t,C2,t\>0C\_\{1,t\},C\_\{2,t\}\>0, independent of𝐤\\bm\{k\}, such that

Γt​\(𝒌\)≤C1,t​‖𝒌‖−2​\(ν\+1\)\+C2,t​‖𝒌‖α−\(ν\+1\),𝒌∈𝒜\.\\Gamma\_\{t\}\(\\bm\{k\}\)\\leq C\_\{1,t\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-2\(\\nu\+1\)\}\+C\_\{2,t\}\\left\\\|\\bm\{k\}\\right\\\|^\{\\alpha\-\(\\nu\+1\)\},\\qquad\\bm\{k\}\\in\\mathcal\{A\}\.\(3\.4\)Consequently,Γt​\(𝐤\)→0\\Gamma\_\{t\}\(\\bm\{k\}\)\\to 0as‖𝐤‖→∞\\left\\\|\\bm\{k\}\\right\\\|\\to\\inftyfor every fixedtt\.

###### Proof\.

Writep=ν\+1p=\\nu\+1\. By \([3\.2](https://arxiv.org/html/2610.41179#S3.E2)\),

0≤𝒦𝒌​𝒌=∑s=1n\|\(ϕs\)𝒌\|2≤n​Bϕ2​‖𝒌‖−2​p\.0\\leq\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}=\\sum\_\{s=1\}^\{n\}\\left\|\(\\phi\_\{s\}\)\_\{\\bm\{k\}\}\\right\|^\{2\}\\leq nB\_\{\\phi\}^\{2\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-2p\}\.\(3\.5\)The off\-diagonal functionalℳ𝒌\\mathcal\{M\}\_\{\\bm\{k\}\}in \([2\.19](https://arxiv.org/html/2610.41179#S2.E19)\) satisfies

‖ℳ𝒌‖2=∑𝒍≠𝒌\|𝒦𝒌​𝒍\|2≤∑𝒍∈Λ\|𝒦𝒌​𝒍\|2≤λ1​𝒦𝒌​𝒌\.\\left\\\|\\mathcal\{M\}\_\{\\bm\{k\}\}\\right\\\|^\{2\}=\\sum\_\{\\bm\{l\}\\neq\\bm\{k\}\}\\left\|\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{l\}\}\\right\|^\{2\}\\leq\\sum\_\{\\bm\{l\}\\in\\Lambda\}\\left\|\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{l\}\}\\right\|^\{2\}\\leq\\lambda\_\{1\}\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}\.\(3\.6\)Here the norm is fromℓ2​\(Λ\)\\ell^\{2\}\(\\Lambda\)toℂ\\mathbb\{C\}\. The last inequality follows from𝒮2≤λ1​𝒮\\mathcal\{S\}^\{2\}\\leq\\lambda\_\{1\}\\mathcal\{S\}, applied to the unit coordinate vector at𝒌\\bm\{k\}\. Hence, with

Et=max0≤q<t⁡‖e\(q\)‖ℓ2​\(Λ\),E\_\{t\}=\\max\_\{0\\leq q<t\}\\left\\\|e^\{\(q\)\}\\right\\\|\_\{\\ell^\{2\}\(\\Lambda\)\},\(3\.7\)we have

\|ℳ𝒌​e\(q\)\|≤Bϕ​n​λ1​Et​‖𝒌‖−p,0≤q<t\.\\left\|\\mathcal\{M\}\_\{\\bm\{k\}\}e^\{\(q\)\}\\right\|\\leq B\_\{\\phi\}\\sqrt\{n\\lambda\_\{1\}\}\\,E\_\{t\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-p\},\\qquad 0\\leq q<t\.\(3\.8\)The numberEtE\_\{t\}is finite because𝒮\\mathcal\{S\}is bounded andttis fixed; no stability restriction onη\\etais needed here\.

Seta𝒌=1−η​𝒦𝒌​𝒌a\_\{\\bm\{k\}\}=1\-\\eta\\mathcal\{K\}\_\{\\bm\{k\}\\bm\{k\}\}andAη=1\+η​n​Bϕ2A\_\{\\eta\}=1\+\\eta nB\_\{\\phi\}^\{2\}\. Since‖𝒌‖≥1\\left\\\|\\bm\{k\}\\right\\\|\\geq 1, we have\|a𝒌\|≤Aη\\left\|a\_\{\\bm\{k\}\}\\right\|\\leq A\_\{\\eta\}\. Iterating \([2\.19](https://arxiv.org/html/2610.41179#S2.E19)\) gives

e𝒌\(t\)=a𝒌t​e𝒌\(0\)−η​∑q=0t−1a𝒌t−1−q​ℳ𝒌​e\(q\)\.e\_\{\\bm\{k\}\}^\{\(t\)\}=a\_\{\\bm\{k\}\}^\{t\}e\_\{\\bm\{k\}\}^\{\(0\)\}\-\\eta\\sum\_\{q=0\}^\{t\-1\}a\_\{\\bm\{k\}\}^\{t\-1\-q\}\\mathcal\{M\}\_\{\\bm\{k\}\}e^\{\(q\)\}\.\(3\.9\)The identitya𝒌t−1=\(a𝒌−1\)​∑q=0t−1a𝒌qa\_\{\\bm\{k\}\}^\{t\}\-1=\(a\_\{\\bm\{k\}\}\-1\)\\sum\_\{q=0\}^\{t\-1\}a\_\{\\bm\{k\}\}^\{q\}and \([3\.5](https://arxiv.org/html/2610.41179#S3.E5)\) imply

\|a𝒌t−1\|≤η​t​Aηt−1​n​Bϕ2​‖𝒌‖−2​p\.\\left\|a\_\{\\bm\{k\}\}^\{t\}\-1\\right\|\\leq\\eta tA\_\{\\eta\}^\{t\-1\}nB\_\{\\phi\}^\{2\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-2p\}\.\(3\.10\)Using \([3\.8](https://arxiv.org/html/2610.41179#S3.E8)\) in the sum in \([3\.9](https://arxiv.org/html/2610.41179#S3.E9)\), and dividing by\|f𝒌\|≥cf​‖𝒌‖−α\\left\|f\_\{\\bm\{k\}\}\\right\|\\geq c\_\{f\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\\alpha\}, therefore yields

Γt​\(𝒌\)≤η​t​Aηt−1​n​Bϕ2​‖𝒌‖−2​p\+η​t​Aηt−1​Bϕ​n​λ1​Etcf​‖𝒌‖α−p\.\\displaystyle\\Gamma\_\{t\}\(\\bm\{k\}\)\\leq\\eta tA\_\{\\eta\}^\{t\-1\}nB\_\{\\phi\}^\{2\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-2p\}\+\\frac\{\\eta tA\_\{\\eta\}^\{t\-1\}B\_\{\\phi\}\\sqrt\{n\\lambda\_\{1\}\}\\,E\_\{t\}\}\{c\_\{f\}\}\\left\\\|\\bm\{k\}\\right\\\|^\{\\alpha\-p\}\.\(3\.11\)This proves \([3\.4](https://arxiv.org/html/2610.41179#S3.E4)\), with the displayed coefficients asC1,tC\_\{1,t\}andC2,tC\_\{2,t\}\. Both powers tend to zero because0<α<p0<\\alpha<p\. ∎

For the target condition \([3\.3](https://arxiv.org/html/2610.41179#S3.E3)\), examples include smooth functions with unequal endpoint values, such asf⁡\(x\)=xf\(x\)=x\(α=1\\alpha=1\); smooth functions with matching endpoint values but unequal endpoint derivatives, such asf⁡\(x\)=x⁡\(1−x\)f\(x\)=x\(1\-x\)\(α=2\\alpha=2\); functions with an interior jump, such asf​\(x\)=𝟏\(0,1/2\)​\(x\)f\(x\)=\\mathbf\{1\}\_\{\(0,1/2\)\}\(x\)\(α=1\\alpha=1\); and functions with a weak endpoint singularity, such asf⁡\(x\)=x−βf\(x\)=x^\{\-\\beta\},0<β<1/20<\\beta<1/2\(α=1−β\\alpha=1\-\\beta\)\. For these four examples, one may take all sufficiently large nonzero lattice frequencies, except that the jump example requires𝒜=\{2​π​m:m∈ℤ​is odd\}\\mathcal\{A\}=\\\{2\\pi m:m\\in\\mathbb\{Z\}\\text\{ is odd\}\\\}to exclude zero coefficients\. The conditionα<ν\+1\\alpha<\\nu\+1must also hold: the first and third examples requireν≥1\\nu\\geq 1, the second requiresν≥2\\nu\\geq 2, and the fourth is compatible withν=0\\nu=0\.

The estimates in Theorem[3\.1](https://arxiv.org/html/2610.41179#S3.Thmtheorem1)should be interpreted as an asymptotic high\-frequency form of the frequency principle, not as a monotone ordering of all Fourier modes\. In other words, for fixed timett, given some frequency𝒌0∈𝒜\\bm\{k\}\_\{0\}\\in\\mathcal\{A\}withΓt​\(𝒌0\)≠0\\Gamma\_\{t\}\(\\bm\{k\}\_\{0\}\)\\neq 0, there existsK\>0K\>0such thatΓt​\(𝒌\)<Γt​\(𝒌0\)\\Gamma\_\{t\}\(\\bm\{k\}\)<\\Gamma\_\{t\}\(\\bm\{k\}\_\{0\}\)for all‖𝒌‖\>K\\left\\\|\\bm\{k\}\\right\\\|\>K; namely, for all sufficiently large frequencies, their relative changes must be smaller than that of𝒌0\\bm\{k\}\_\{0\}\. Next, we prove a stronger result uniformly fortt\.

###### Corollary 3\.2\.

Assume \([3\.2](https://arxiv.org/html/2610.41179#S3.E2)\) and the lower bound

\|f𝒌\|≥cf​‖𝒌‖−α,𝒌∈𝒜,\\left\|f\_\{\\bm\{k\}\}\\right\|\\geq c\_\{f\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\\alpha\},\\qquad\\bm\{k\}\\in\\mathcal\{A\},\(3\.12\)wherecf\>0c\_\{f\}\>0and0<α<ν\+10<\\alpha<\\nu\+1\. Leth\(0\)=0h^\{\(0\)\}=0and0<η<2/λ10<\\eta<2/\\lambda\_\{1\}\. Then

supt∈ℕ0Γt​\(𝒌\)≤2​‖PV​f‖L2​\(Ω\)​Bϕcf​nλn​‖𝒌‖α−\(ν\+1\),𝒌∈𝒜\.\\sup\_\{t\\in\\mathbb\{N\}\_\{0\}\}\\Gamma\_\{t\}\(\\bm\{k\}\)\\leq\\frac\{2\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}B\_\{\\phi\}\}\{c\_\{f\}\}\\sqrt\{\\frac\{n\}\{\\lambda\_\{n\}\}\}\\,\\left\\\|\\bm\{k\}\\right\\\|^\{\\alpha\-\(\\nu\+1\)\},\\qquad\\bm\{k\}\\in\\mathcal\{A\}\.\(3\.13\)If0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\}, the factor22can be replaced by11\. Consequently,supt∈ℕ0Γt​\(𝐤\)→0\\sup\_\{t\\in\\mathbb\{N\}\_\{0\}\}\\Gamma\_\{t\}\(\\bm\{k\}\)\\to 0as‖𝐤‖→∞\\left\\\|\\bm\{k\}\\right\\\|\\to\\infty\.

###### Proof\.

The spectral representation in Proposition[2\.6](https://arxiv.org/html/2610.41179#S2.Thmtheorem6), withh\(0\)=0h^\{\(0\)\}=0, gives

h\(t\)=∑j=1n\[1−\(1−η​λj\)t\]​⟨f,qj⟩Ω​qj\.h^\{\(t\)\}=\\sum\_\{j=1\}^\{n\}\\bigl\[1\-\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\bigr\]\\left\\langle f,q\_\{j\}\\right\\rangle\_\{\\Omega\}q\_\{j\}\.\(3\.14\)Since\|1−η​λj\|<1\\left\|1\-\\eta\\lambda\_\{j\}\\right\|<1, orthonormality implies

‖h\(t\)‖L2​\(Ω\)≤2​‖PV​f‖L2​\(Ω\)\.\\left\\\|h^\{\(t\)\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\leq 2\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\.\(3\.15\)For0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\}, every multiplier1−\(1−η​λj\)t1\-\(1\-\\eta\\lambda\_\{j\}\)^\{t\}lies in\[0,1\]\[0,1\], so the factor22is unnecessary\. Moreover,h\(t\)∈Vh^\{\(t\)\}\\in Vand \([2\.20](https://arxiv.org/html/2610.41179#S2.E20)\) give

\|h𝒌\(t\)\|=\|⟨h\(t\),PV​χ𝒌⟩Ω\|≤‖h\(t\)‖L2​\(Ω\)​ℓ𝒌1/2\.\\left\|h\_\{\\bm\{k\}\}^\{\(t\)\}\\right\|=\\left\|\\left\\langle h^\{\(t\)\},P\_\{V\}\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}\\right\|\\leq\\left\\\|h^\{\(t\)\}\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\ell\_\{\\bm\{k\}\}^\{1/2\}\.\(3\.16\)By \([2\.34](https://arxiv.org/html/2610.41179#S2.E34)\),

ℓ𝒌1/2≤Bϕ​nλn​‖𝒌‖−\(ν\+1\)\.\\ell\_\{\\bm\{k\}\}^\{1/2\}\\leq B\_\{\\phi\}\\sqrt\{\\frac\{n\}\{\\lambda\_\{n\}\}\}\\,\\left\\\|\\bm\{k\}\\right\\\|^\{\-\(\\nu\+1\)\}\.\(3\.17\)Combining these estimates and dividing by\|f𝒌\|≥cf​‖𝒌‖−α\\left\|f\_\{\\bm\{k\}\}\\right\|\\geq c\_\{f\}\\left\\\|\\bm\{k\}\\right\\\|^\{\-\\alpha\}proves \([3\.13](https://arxiv.org/html/2610.41179#S3.E13)\) uniformly intt\. ∎

Note that the estimate in Corollary[3\.2](https://arxiv.org/html/2610.41179#S3.Thmtheorem2)holds uniformly over allt∈ℕ0t\\in\\mathbb\{N\}\_\{0\}, whereas Theorem[3\.1](https://arxiv.org/html/2610.41179#S3.Thmtheorem1)gives an estimate for each fixedtt\. The uniform estimate shows thatΓt​\(𝒌\)\\Gamma\_\{t\}\(\\bm\{k\}\)remains uniformly small forttat sufficiently high frequencies𝒌∈𝒜\\bm\{k\}\\in\\mathcal\{A\}\. Ast→∞t\\to\\infty, the approximation converges toPV​fP\_\{V\}f, and the limiting ratio\|\(PV​f\)𝒌\|/\|f𝒌\|\\left\|\(P\_\{V\}f\)\_\{\\bm\{k\}\}\\right\|/\\left\|f\_\{\\bm\{k\}\}\\right\|satisfies the same bound\. The termsupt∈ℕ0Γt​\(𝒌\)\\sup\_\{t\\in\\mathbb\{N\}\_\{0\}\}\\Gamma\_\{t\}\(\\bm\{k\}\)can be viewed as the maximal relative change of frequencykkthroughout training\. Corollary[3\.2](https://arxiv.org/html/2610.41179#S3.Thmtheorem2)implies that given a timet0≥1t\_\{0\}\\geq 1and a frequency𝒌0∈𝒜\\bm\{k\}\_\{0\}\\in\\mathcal\{A\}withΓt0​\(𝒌0\)\>0\\Gamma\_\{t\_\{0\}\}\(\\bm\{k\}\_\{0\}\)\>0, there existsK\>0K\>0, independent oftt, such thatΓt​\(𝒌\)<Γt0​\(𝒌0\)\\Gamma\_\{t\}\(\\bm\{k\}\)<\\Gamma\_\{t\_\{0\}\}\(\\bm\{k\}\_\{0\}\)for allt∈ℕ0t\\in\\mathbb\{N\}\_\{0\}and all𝒌∈𝒜\\bm\{k\}\\in\\mathcal\{A\}with‖𝒌‖\>K\\left\\\|\\bm\{k\}\\right\\\|\>K\. In other words, high\-frequency errors remain nearly unchanged throughout training\.

These statements concern a fixed finite linear class\. The factorBϕ​n/λnB\_\{\\phi\}\\sqrt\{n/\\lambda\_\{n\}\}in \([3\.13](https://arxiv.org/html/2610.41179#S3.E13)\) may grow when the class is enlarged or its Gram matrix becomes ill\-conditioned\. The estimate is consequently not uniform over families of increasing dimension without additional assumptions\.

The estimates above show thatΓt​\(𝒌\)\\Gamma\_\{t\}\(\\bm\{k\}\)can remain small even after convergence toPV​fP\_\{V\}f, so small relative change alone does not necessarily imply slow learning\. In the next section, we compare how quicklyh𝒌\(t\)h\_\{\\bm\{k\}\}^\{\(t\)\}approaches\(PV​f\)𝒌\(P\_\{V\}f\)\_\{\\bm\{k\}\}at different frequencies\.

### 3\.2Numerical experiments

#### 3\.2\.1Legendre bases

We use the orthonormal tensor product Legendre basis onΩd=\[−1,1\]d\\Omega\_\{d\}=\[\-1,1\]^\{d\}, ford=1,2,3d=1,2,3,

ϕ𝒑​\(𝒙\)=∏j=1d2​pj\+12​Ppj​\(xj\),𝒑∈\{0,…,Md−1\}d,\\phi\_\{\\bm\{p\}\}\(\\bm\{x\}\)=\\prod\_\{j=1\}^\{d\}\\sqrt\{\\frac\{2p\_\{j\}\+1\}\{2\}\}P\_\{p\_\{j\}\}\(x\_\{j\}\),\\qquad\\bm\{p\}\\in\\\{0,\\ldots,M\_\{d\}\-1\\\}^\{d\},\(3\.18\)wherePpP\_\{p\}is the Legendre polynomial of degreepp\. We take\(M1,M2,M3\)=\(12,10,8\)\(M\_\{1\},M\_\{2\},M\_\{3\}\)=\(12,10,8\), giving\(n1,n2,n3\)=\(12,100,512\)\(n\_\{1\},n\_\{2\},n\_\{3\}\)=\(12,100,512\)basis functions, and choose the targets

f1​\(x1\)\\displaystyle f\_\{1\}\(x\_\{1\}\)=\|x1\|−0\.30,\\displaystyle=\\left\|x\_\{1\}\\right\|^\{\-0\.30\},\(3\.19\)f2​\(x1,x2\)\\displaystyle f\_\{2\}\(x\_\{1\},x\_\{2\}\)=\|x1\|−0\.35​\|x2−0\.27\|−0\.15,\\displaystyle=\\left\|x\_\{1\}\\right\|^\{\-0\.35\}\\left\|x\_\{2\}\-0\.27\\right\|^\{\-0\.15\},\(3\.20\)f3​\(x1,x2,x3\)\\displaystyle f\_\{3\}\(x\_\{1\},x\_\{2\},x\_\{3\}\)=\|x1\|−0\.40​\|x2−0\.27\|−0\.15​\|x3\+0\.31\|−0\.20\.\\displaystyle=\\left\|x\_\{1\}\\right\|^\{\-0\.40\}\\left\|x\_\{2\}\-0\.27\\right\|^\{\-0\.15\}\\left\|x\_\{3\}\+0\.31\\right\|^\{\-0\.20\}\.\(3\.21\)We apply full batch gradient descent from zero initialization withη=0\.01\\eta=0\.01and recordt=1,5,20,100,500,2000t=1,5,20,100,500,2000\. The integrals are evaluated by 1800\-point Gauss–Legendre quadrature after a change of variables removes the target singularities\. Along the frequency ray

𝝎k=\(π⁡\(k\+12\),0,…,0\),k≥1,\\bm\{\\omega\}\_\{k\}=\\bigl\(\\pi\(k\+\\tfrac\{1\}\{2\}\),0,\\ldots,0\\bigr\),\\qquad k\\geq 1,\(3\.22\)we samplek=1,…,Kdk=1,\\ldots,K\_\{d\}, with\(K1,K2,K3\)=\(180,200,220\)\(K\_\{1\},K\_\{2\},K\_\{3\}\)=\(180,200,220\)\. The low\- and high\-frequency bands arek≤⌊0\.15​Kd⌋k\\leq\\lfloor 0\.15K\_\{d\}\\rfloorandk≥⌈0\.75​Kd⌉k\\geq\\lceil 0\.75K\_\{d\}\\rceil, respectively\.

For these finite Legendre classes, Lemma[2\.4](https://arxiv.org/html/2610.41179#S2.Thmtheorem4)gives the basis decay in \([3\.2](https://arxiv.org/html/2610.41179#S3.E2)\) withν=0\\nu=0\. Along \([3\.22](https://arxiv.org/html/2610.41179#S3.E22)\), the target spectra satisfy

cd​‖𝝎k‖−βd≤\|fd^​\(𝝎k\)\|≤Cd​‖𝝎k‖−βd,\(β1,β2,β3\)=\(0\.70,0\.65,0\.60\),c\_\{d\}\\left\\\|\\bm\{\\omega\}\_\{k\}\\right\\\|^\{\-\\beta\_\{d\}\}\\leq\\left\|\\widehat\{f\_\{d\}\}\(\\bm\{\\omega\}\_\{k\}\)\\right\|\\leq C\_\{d\}\\left\\\|\\bm\{\\omega\}\_\{k\}\\right\\\|^\{\-\\beta\_\{d\}\},\\qquad\(\\beta\_\{1\},\\beta\_\{2\},\\beta\_\{3\}\)=\(0\.70,0\.65,0\.60\),\(3\.23\)for all sufficiently largekk, with0<cd≤Cd0<c\_\{d\}\\leq C\_\{d\}\. Substituting these exponents into Theorem[3\.1](https://arxiv.org/html/2610.41179#S3.Thmtheorem1)gives

Γt\(d\)​\(𝝎k\)≤C1,t\(d\)​‖𝝎k‖−2\+C2,t\(d\)​‖𝝎k‖βd−1=Ot​\(k−2\+kβd−1\)\.\\Gamma\_\{t\}^\{\(d\)\}\(\\bm\{\\omega\}\_\{k\}\)\\leq C\_\{1,t\}^\{\(d\)\}\\left\\\|\\bm\{\\omega\}\_\{k\}\\right\\\|^\{\-2\}\+C\_\{2,t\}^\{\(d\)\}\\left\\\|\\bm\{\\omega\}\_\{k\}\\right\\\|^\{\\beta\_\{d\}\-1\}=O\_\{t\}\\\!\\left\(k^\{\-2\}\+k^\{\\beta\_\{d\}\-1\}\\right\)\.\(3\.24\)Thus the predicted decay rates areOt​\(k−0\.30\)O\_\{t\}\(k^\{\-0\.30\}\),Ot​\(k−0\.35\)O\_\{t\}\(k^\{\-0\.35\}\), andOt​\(k−0\.40\)O\_\{t\}\(k^\{\-0\.40\}\)ford=1,2,3d=1,2,3, respectively\.

Figure[1](https://arxiv.org/html/2610.41179#S3.F1)shows that the relative movement increases withtt, while its high\-frequency envelope decreases withkk\. Att=2000t=2000, the ratios of the mean movement in the low\-frequency band to that in the high\-frequency band are4\.144\.14,2\.322\.32, and6\.046\.04ford=1,2,3d=1,2,3, respectively\. These results agree with \([3\.24](https://arxiv.org/html/2610.41179#S3.E24)\): high\-frequency movement is smaller on average, with local oscillations across the sampled frequencies\.

![Refer to caption](https://arxiv.org/html/2610.41179v1/linear_theorem26_gamma.png)Figure 1:Pointwise relative Fourier movement for the one\-, two\-, and three\-dimensional tensor product Legendre approximation classes\.
#### 3\.2\.2trigonometric bases

We use the periodic box𝕋d=\[0,2​π\]d\\mathbb\{T\}^\{d\}=\[0,2\\pi\]^\{d\}, ford=1,2,3d=1,2,3, and the orthonormal trigonometric basis

ϕ~𝒑\(𝒙\)=\(2π\)−d/2ei​𝒑⋅𝒙,𝒑∈ℐd:=\{−20,…,20\}d\.\\tilde\{\\phi\}\_\{\\bm\{p\}\}\(\\bm\{x\}\)=\(2\\pi\)^\{\-d/2\}\\mathrm\{e\}^\{\\mathrm\{i\}\\bm\{p\}\\cdot\\bm\{x\}\},\\qquad\\bm\{p\}\\in\\mathcal\{I\}\_\{d\}:=\\\{\-20,\\ldots,20\\\}^\{d\}\.\(3\.25\)The target is

f~d=∑𝒑∈ℐdb𝒑​ϕ~𝒑,b𝒑=\(1\+‖𝒑‖\)−2\.\\tilde\{f\}\_\{d\}=\\sum\_\{\\bm\{p\}\\in\\mathcal\{I\}\_\{d\}\}b\_\{\\bm\{p\}\}\\tilde\{\\phi\}\_\{\\bm\{p\}\},\\qquad b\_\{\\bm\{p\}\}=\(1\+\\left\\\|\\bm\{p\}\\right\\\|\)^\{\-2\}\.\(3\.26\)We apply full batch gradient descent from zero initialization withη=0\.01\\eta=0\.01and recordt=1,5,20,100,500,2000t=1,5,20,100,500,2000\. For the band averages, we use0<r≤rmax/40<r\\leq r\_\{\\max\}/4andr≥3​rmax/4r\\geq 3r\_\{\\max\}/4, wherer=‖𝒑‖r=\\left\\\|\\bm\{p\}\\right\\\|andrmax=20​dr\_\{\\max\}=20\\sqrt\{d\}\.

Since the Gram matrix is the identity and the target belongs to the approximation class, every represented Fourier mode satisfies

e𝒑\(t\)=0\.99t​e𝒑\(0\),Γt​\(𝒑\)=1−0\.99t\.e\_\{\\bm\{p\}\}^\{\(t\)\}=0\.99^\{t\}e\_\{\\bm\{p\}\}^\{\(0\)\},\\qquad\\Gamma\_\{t\}\(\\bm\{p\}\)=1\-0\.99^\{t\}\.\(3\.27\)At the six recorded iterations, the theoretical movement values are approximately0\.010\.01,0\.0490100\.049010,0\.1820930\.182093,0\.6339680\.633968,0\.9934300\.993430, and0\.99999999810\.9999999981, respectively\. These values are independent of frequency and dimension\.

Figure[2](https://arxiv.org/html/2610.41179#S3.F2)shows the computed movement averaged over unit\-width radial shells\. The curves are flat to machine precision in all three dimensions, with a maximum discrepancy of1\.11×10−151\.11\\times 10^\{\-15\}from \([3\.27](https://arxiv.org/html/2610.41179#S3.E27)\)\. Att=2000t=2000, the ratio of the low\- to high\-frequency mean movement is11in each dimension\. Thus all represented frequencies have the same relative movement, and this periodic class exhibits no frequency ordering\.

![Refer to caption](https://arxiv.org/html/2610.41179v1/trigonometric_remark27_gamma.png)Figure 2:Relative Fourier movement for the one\-, two\-, and three\-dimensional periodic trigonometric classes\.

## 4Learning times and frequency order

Section[3](https://arxiv.org/html/2610.41179#S3)bounds the output amplitudes relative to the target amplitudes at high frequencies, showing that high\-frequency learner components usually change slowly under gradient descent\. In this section, we derive bounds on learning times relative to the attainable limit and examine how local spectral measures and frequency mixing affect the learning order\.

### 4\.1Single\-frequency targets

To measure the learning progress of a single Fourier mode withℓ𝒌\>0\\ell\_\{\\bm\{k\}\}\>0, we first consider the single\-frequency targetf=χ𝒌f=\\chi\_\{\\bm\{k\}\}\. We also assumeh\(0\)=0h^\{\(0\)\}=0, thene𝒌\(0\)=−1e\_\{\\bm\{k\}\}^\{\(0\)\}=\-1, and \([2\.16](https://arxiv.org/html/2610.41179#S2.E16)\) givese𝒌\(∞\)=−⟨χ𝒌−PV​χ𝒌,χ𝒌⟩Ω=−\(1−ℓ𝒌\)e\_\{\\bm\{k\}\}^\{\(\\infty\)\}=\-\\left\\langle\\chi\_\{\\bm\{k\}\}\-P\_\{V\}\\chi\_\{\\bm\{k\}\},\\chi\_\{\\bm\{k\}\}\\right\\rangle\_\{\\Omega\}=\-\(1\-\\ell\_\{\\bm\{k\}\}\)\. For0<ε<10<\\varepsilon<1, we define the learning time required to reduce the relative error belowε\\varepsilonby

Tε​\(𝒌\)=min⁡\{t∈ℕ0:ρ𝒌​\(t\)≤ε\},T\_\{\\varepsilon\}\(\\bm\{k\}\)=\\min\\left\\\{t\\in\\mathbb\{N\}\_\{0\}:\\rho\_\{\\bm\{k\}\}\(t\)\\leq\\varepsilon\\right\\\},\(4\.1\)whereρ𝒌​\(t\):=e𝒌\(t\)−e𝒌\(∞\)e𝒌\(0\)−e𝒌\(∞\)\\rho\_\{\\bm\{k\}\}\(t\):=\\frac\{e\_\{\\bm\{k\}\}^\{\(t\)\}\-e\_\{\\bm\{k\}\}^\{\(\\infty\)\}\}\{e\_\{\\bm\{k\}\}^\{\(0\)\}\-e\_\{\\bm\{k\}\}^\{\(\\infty\)\}\}is the relative error\. By Proposition[2\.6](https://arxiv.org/html/2610.41179#S2.Thmtheorem6)and \([2\.24](https://arxiv.org/html/2610.41179#S2.E24)\), the relative errorρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)takes the form

ρ𝒌​\(t\)=∑j=1n\(1−η​λj\)t​⟨−χ𝒌,qj⟩Ω​\(qj\)𝒌−ℓ𝒌=1ℓ𝒌​∑j=1n\|\(qj\)𝒌\|2​\(1−η​λj\)t=∑j=1nϖj​\(𝒌\)​\(1−η​λj\)t\.\\rho\_\{\\bm\{k\}\}\(t\)=\\frac\{\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle\-\\chi\_\{\\bm\{k\}\},q\_\{j\}\\right\\rangle\_\{\\Omega\}\(q\_\{j\}\)\_\{\\bm\{k\}\}\}\{\-\\ell\_\{\\bm\{k\}\}\}\\\\ =\\frac\{1\}\{\\ell\_\{\\bm\{k\}\}\}\\sum\_\{j=1\}^\{n\}\\left\|\(q\_\{j\}\)\_\{\\bm\{k\}\}\\right\|^\{2\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}=\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(\\bm\{k\}\)\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\.\(4\.2\)
Thusρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)is the mean of the Landweber residual polynomial\(1−η​λ\)t\(1\-\\eta\\lambda\)^\{t\}[Engl et al\. \(1996\)](https://arxiv.org/html/2610.41179#bib.bib18)under the local spectral measure\. For0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\}, it is real and decreases monotonically from one to zero\. Inρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\), each decay factor\(1−η​λj\)t\(1\-\\eta\\lambda\_\{j\}\)^\{t\}contributes with weightϖj​\(𝒌\)\\varpi\_\{j\}\(\\bm\{k\}\)\. By Theorem[2\.3](https://arxiv.org/html/2610.41179#S2.Thmtheorem3), the gradient descent converges, ande𝒌\(∞\)e\_\{\\bm\{k\}\}^\{\(\\infty\)\}characterizes the approximation error between the targetffand the approximation classVV\. Specifically,e𝒌\(∞\)=0e\_\{\\bm\{k\}\}^\{\(\\infty\)\}=0iff∈Vf\\in V\. So, in the numerator ofρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\), subtractinge𝒌\(∞\)e\_\{\\bm\{k\}\}^\{\(\\infty\)\}removes the approximation error, and thereforeρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)measures how quickly the attainable limit is reached\.

For a single\-frequency target,Γt​\(𝒌\)\\Gamma\_\{t\}\(\\bm\{k\}\)andρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)satisfy an exact relation\. If0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\}, then

e𝒌\(t\)=−\(1−ℓ𝒌\+ℓ𝒌​ρ𝒌​\(t\)\),Γt​\(𝒌\)=ℓ𝒌​\(1−ρ𝒌​\(t\)\)\.e\_\{\\bm\{k\}\}^\{\(t\)\}=\-\\bigl\(1\-\\ell\_\{\\bm\{k\}\}\+\\ell\_\{\\bm\{k\}\}\\rho\_\{\\bm\{k\}\}\(t\)\\bigr\),\\qquad\\Gamma\_\{t\}\(\\bm\{k\}\)=\\ell\_\{\\bm\{k\}\}\\bigl\(1\-\\rho\_\{\\bm\{k\}\}\(t\)\\bigr\)\.\(4\.3\)Indeed, the first identity follows frome𝒌\(0\)=−1e\_\{\\bm\{k\}\}^\{\(0\)\}=\-1ande𝒌\(∞\)=−\(1−ℓ𝒌\)e\_\{\\bm\{k\}\}^\{\(\\infty\)\}=\-\(1\-\\ell\_\{\\bm\{k\}\}\)\. The second follows fromf𝒌=1f\_\{\\bm\{k\}\}=1andh𝒌\(t\)=e𝒌\(t\)\+1=ℓ𝒌​\(1−ρ𝒌​\(t\)\)≥0h\_\{\\bm\{k\}\}^\{\(t\)\}=e\_\{\\bm\{k\}\}^\{\(t\)\}\+1=\\ell\_\{\\bm\{k\}\}\(1\-\\rho\_\{\\bm\{k\}\}\(t\)\)\\geq 0, since0≤ρ𝒌​\(t\)≤10\\leq\\rho\_\{\\bm\{k\}\}\(t\)\\leq 1\. Therefore, small capacityℓ𝒌\\ell\_\{\\bm\{k\}\}can keep the relative changeΓt​\(𝒌\)\\Gamma\_\{t\}\(\\bm\{k\}\)small throughout training, even when the represented part is learned quickly \(i\.e\.,ρ𝒌​\(t\)→0\\rho\_\{\\bm\{k\}\}\(t\)\\rightarrow 0quickly\)\.

The next theorem gives two\-sided learning\-time bounds in terms of the local spectral measure\.

###### Theorem 4\.1\.

Let0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\},𝐤∈Λ\\bm\{k\}\\in\\Lambdawithℓ𝐤\>0\\ell\_\{\\bm\{k\}\}\>0, and0<ε<10<\\varepsilon<1\. Then the following statements hold\.

1. \(a\)The learning time satisfies Tε​\(𝒌\)≥log⁡\(1/ε\)−log⁡\(1−η​λ¯𝒌\)≥\(1η​λ¯𝒌−1\)​log⁡\(1/ε\)\.T\_\{\\varepsilon\}\(\\bm\{k\}\)\\geq\\frac\{\\log\(1/\\varepsilon\)\}\{\-\\log\(1\-\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}\)\}\\geq\\left\(\\frac\{1\}\{\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}\}\-1\\right\)\\log\(1/\\varepsilon\)\.\(4\.4\)
2. \(b\)For0<μ≤λ10<\\mu\\leq\\lambda\_\{1\}: if∑λj≤μϖj​\(𝒌\)\>ε\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\>\\varepsilon, then Tε​\(𝒌\)≥log⁡\(ε−1​∑λj≤μϖj​\(𝒌\)\)−log⁡\(1−η​μ\);T\_\{\\varepsilon\}\(\\bm\{k\}\)\\geq\\frac\{\\log\\bigl\(\\varepsilon^\{\-1\}\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\bigr\)\}\{\-\\log\(1\-\\eta\\mu\)\};\(4\.5\)if∑λj<μϖj​\(𝒌\)<ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)<\\varepsilon, then Tε​\(𝒌\)≤1\+∑λj≥μϖj​\(𝒌\)/λje​η​\(ε−∑λj<μϖj​\(𝒌\)\),T\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq 1\+\\frac\{\\sum\_\{\\lambda\_\{j\}\\geq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)/\\lambda\_\{j\}\}\{\\mathrm\{e\}\\,\\eta\\,\\bigl\(\\varepsilon\-\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\bigr\)\},\(4\.6\)and moreover, if∑λj<μϖj​\(𝒌\)≤c​ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\leq c\\varepsilonwith0≤c<10\\leq c<1, then Tε​\(𝒌\)≤1\+log⁡\(1/\(\(1−c\)​ε\)\)η​μ\.T\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq 1\+\\frac\{\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)\}\{\\eta\\mu\}\.\(4\.7\)

The lower bounds forTε​\(𝐤\)T\_\{\\varepsilon\}\(\\bm\{k\}\)in \(a\) and \(b\) are understood as zero ifη​λ¯𝐤=1\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}=1orη​μ=1\\eta\\mu=1, respectively\.

###### Proof\.

\(a\) Sinceρ𝒌​\(0\)=1\>ε\\rho\_\{\\bm\{k\}\}\(0\)=1\>\\varepsilon, we haveTε​\(𝒌\)≥1T\_\{\\varepsilon\}\(\\bm\{k\}\)\\geq 1\. For each integert≥1t\\geq 1, the functiongt​\(λ\)=\(1−η​λ\)tg\_\{t\}\(\\lambda\)=\(1\-\\eta\\lambda\)^\{t\}is convex on\[0,1/η\]\[0,1/\\eta\]\. It is affine whent=1t=1; fort≥2t\\geq 2, its second derivative ist⁡\(t−1\)​η2​\(1−η​λ\)t−2≥0t\(t\-1\)\\eta^\{2\}\(1\-\\eta\\lambda\)^\{t\-2\}\\geq 0on the interior, and it extends continuously to the endpoints\. All eigenvaluesλj\\lambda\_\{j\}lie in\(0,λ1\]⊂\(0,1/η\]\(0,\\lambda\_\{1\}\]\\subset\(0,1/\\eta\]\. Since the weightsϖj​\(𝒌\)\\varpi\_\{j\}\(\\bm\{k\}\)are nonnegative and sum to one, Jensen’s inequality and \([2\.25](https://arxiv.org/html/2610.41179#S2.E25)\) give

ρ𝒌​\(t\)=∑j=1nϖj​\(𝒌\)​gt​\(λj\)≥gt​\(λ¯𝒌\)=\(1−η​λ¯𝒌\)t,t≥1\.\\rho\_\{\\bm\{k\}\}\(t\)=\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(\\bm\{k\}\)g\_\{t\}\(\\lambda\_\{j\}\)\\geq g\_\{t\}\(\\bar\{\\lambda\}\_\{\\bm\{k\}\}\)=\(1\-\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}\)^\{t\},\\qquad t\\geq 1\.\(4\.8\)Ifη​λ¯𝒌<1\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}<1, evaluating this inequality att=Tε​\(𝒌\)t=T\_\{\\varepsilon\}\(\\bm\{k\}\)gives\(1−η​λ¯𝒌\)Tε​\(𝒌\)≤ε\(1\-\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}\)^\{T\_\{\\varepsilon\}\(\\bm\{k\}\)\}\\leq\\varepsilon\. Taking logarithms proves the first inequality in \([4\.4](https://arxiv.org/html/2610.41179#S4.E4)\)\. The second follows from−log⁡\(1−z\)≤z/\(1−z\)\-\\log\(1\-z\)\\leq z/\(1\-z\)for0≤z<10\\leq z<1\. Ifη​λ¯𝒌=1\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}\}=1, both lower bounds in \([4\.4](https://arxiv.org/html/2610.41179#S4.E4)\) are zero under the stated convention, so the conclusion also holds\.

\(b\) Fix0<μ≤λ10<\\mu\\leq\\lambda\_\{1\}and an integert≥1t\\geq 1\. Every term in \([4\.2](https://arxiv.org/html/2610.41179#S4.E2)\) is nonnegative\. Forλj≤μ\\lambda\_\{j\}\\leq\\mu, we have\(1−η​λj\)t≥\(1−η​μ\)t\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\geq\(1\-\\eta\\mu\)^\{t\}, which gives a lower bound forρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)after dropping the terms withλj\>μ\\lambda\_\{j\}\>\\mu\. For an upper bound, we use\(1−η​λj\)t≤1\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\leq 1whenλj<μ\\lambda\_\{j\}<\\muand\(1−η​λj\)t≤\(1−η​μ\)t\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\leq\(1\-\\eta\\mu\)^\{t\}whenλj≥μ\\lambda\_\{j\}\\geq\\mu, together with∑λj≥μϖj​\(𝒌\)≤1\\sum\_\{\\lambda\_\{j\}\\geq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\leq 1\. Thus

\(∑λj≤μϖj​\(𝒌\)\)​\(1−η​μ\)t≤ρ𝒌​\(t\)≤∑λj<μϖj​\(𝒌\)\+\(1−η​μ\)t\.\\biggl\(\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\biggr\)\(1\-\\eta\\mu\)^\{t\}\\leq\\rho\_\{\\bm\{k\}\}\(t\)\\leq\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\+\(1\-\\eta\\mu\)^\{t\}\.\(4\.9\)If∑λj≤μϖj​\(𝒌\)\>ε\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\>\\varepsilonandη​μ<1\\eta\\mu<1, evaluating the lower bound att=Tε​\(𝒌\)t=T\_\{\\varepsilon\}\(\\bm\{k\}\)and usingρ𝒌​\(Tε​\(𝒌\)\)≤ε\\rho\_\{\\bm\{k\}\}\(T\_\{\\varepsilon\}\(\\bm\{k\}\)\)\\leq\\varepsilongives \([4\.5](https://arxiv.org/html/2610.41179#S4.E5)\) after taking logarithms\. Ifη​μ=1\\eta\\mu=1, the stated lower bound is zero, so \([4\.5](https://arxiv.org/html/2610.41179#S4.E5)\) also holds\.

To prove \([4\.6](https://arxiv.org/html/2610.41179#S4.E6)\), lett≥1t\\geq 1\. For0≤z≤10\\leq z\\leq 1, we havez​\(1−z\)t≤tt/\(t\+1\)t\+1≤1/\(e​t\)z\(1\-z\)^\{t\}\\leq t^\{t\}/\(t\+1\)^\{t\+1\}\\leq 1/\(\\mathrm\{e\}t\), since\(1\+1/t\)t\+1≥e\(1\+1/t\)^\{t\+1\}\\geq\\mathrm\{e\}\. Withz=η​λjz=\\eta\\lambda\_\{j\}, this gives\(1−η​λj\)t≤1/\(e​η​t​λj\)\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\leq 1/\(\\mathrm\{e\}\\eta t\\lambda\_\{j\}\)\. Applying this estimate to the terms withλj≥μ\\lambda\_\{j\}\\geq\\muand using\(1−η​λj\)t≤1\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\leq 1for the remaining terms yields

ρ𝒌​\(t\)≤∑λj<μϖj​\(𝒌\)\+1e​η​t​∑λj≥μϖj​\(𝒌\)λj\.\\rho\_\{\\bm\{k\}\}\(t\)\\leq\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\+\\frac\{1\}\{\\mathrm\{e\}\\eta t\}\\sum\_\{\\lambda\_\{j\}\\geq\\mu\}\\frac\{\\varpi\_\{j\}\(\\bm\{k\}\)\}\{\\lambda\_\{j\}\}\.\(4\.10\)If∑λj<μϖj​\(𝒌\)<ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)<\\varepsilon, this impliesρ𝒌​\(t\)≤ε\\rho\_\{\\bm\{k\}\}\(t\)\\leq\\varepsilonfor every integert≥1t\\geq 1with

t≥∑λj≥μϖj​\(𝒌\)/λje​η​\(ε−∑λj<μϖj​\(𝒌\)\)\.t\\geq\\frac\{\\sum\_\{\\lambda\_\{j\}\\geq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)/\\lambda\_\{j\}\}\{\\mathrm\{e\}\\,\\eta\\,\\bigl\(\\varepsilon\-\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\bigr\)\}\.\(4\.11\)The right\-hand side is positive\. Rounding it up to an integer increases it by at most one, which proves \([4\.6](https://arxiv.org/html/2610.41179#S4.E6)\)\.

Finally, let0≤c<10\\leq c<1and suppose that∑λj<μϖj​\(𝒌\)≤c​ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\leq c\\varepsilon\. The upper bound in \([4\.9](https://arxiv.org/html/2610.41179#S4.E9)\) gives

ρ𝒌​\(t\)≤c​ε\+\(1−η​μ\)t≤c​ε\+e−η​μ​t,t≥1\.\\rho\_\{\\bm\{k\}\}\(t\)\\leq c\\varepsilon\+\(1\-\\eta\\mu\)^\{t\}\\leq c\\varepsilon\+\\mathrm\{e\}^\{\-\\eta\\mu t\},\\qquad t\\geq 1\.\(4\.12\)Thusρ𝒌​\(t\)≤ε\\rho\_\{\\bm\{k\}\}\(t\)\\leq\\varepsilonwhenevere−η​μ​t≤\(1−c\)​ε\\mathrm\{e\}^\{\-\\eta\\mu t\}\\leq\(1\-c\)\\varepsilon, which holds for every integert≥log⁡\(1/\(\(1−c\)​ε\)\)/\(η​μ\)t\\geq\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)/\(\\eta\\mu\)\. Since0<\(1−c\)​ε<10<\(1\-c\)\\varepsilon<1, this threshold is positive\. Rounding it up to an integer proves \([4\.7](https://arxiv.org/html/2610.41179#S4.E7)\)\. ∎

This theorem provides upper and lower bounds forTε​\(𝒌\)T\_\{\\varepsilon\}\(\\bm\{k\}\)under certain conditions\. Note thatλ¯𝒌≤λ1\\bar\{\\lambda\}\_\{\\bm\{k\}\}\\leq\\lambda\_\{1\}, Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)\(a\) means that the required time to make the errorρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)reachε\\varepsilonis at least\(1η​λ1−1\)​log⁡\(1/ε\)\\left\(\\frac\{1\}\{\\eta\\lambda\_\{1\}\}\-1\\right\)\\log\(1/\\varepsilon\)\. In Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)\(b\), the sum∑λj≤μϖj​\(𝒌\)\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)represents the proportion ofPV​χ𝒌P\_\{V\}\\chi\_\{\\bm\{k\}\}in directions with eigenvalues at mostμ\\mu\. If this proportion isΘ⁡\(1\)\\Theta\(1\)\(greater thanε\\varepsilon\), then it has to costΘ\(−log\(ε−1\)/log\(1−ημ\)\)\\Theta\\left\(\-\\log\\bigl\(\\varepsilon^\{\-1\}\\bigr\)/\{\\log\(1\-\\eta\\mu\)\}\\right\)time to make the error reachε\\varepsilon; if∑λj<μϖj​\(𝒌\)<Θ⁡\(ε\)\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)<\\Theta\(\\varepsilon\)with constant less than 1, reducing the error belowε\\varepsilonrequiresmin⁡\{Θ⁡\(1η​ϵ\),Θ⁡\(log⁡\(1/ϵ\)η​μ\)\}\\min\\\{\\Theta\\left\(\\frac\{1\}\{\\eta\\epsilon\}\\right\),\\Theta\\left\(\\frac\{\\log\(1/\\epsilon\)\}\{\\eta\\mu\}\\right\)\\\}time\. Here we writeA=Θ⁡\(B\)A=\\Theta\(B\)if there exist constants0<c1≤c20<c\_\{1\}\\leq c\_\{2\}, independent of the varying parameters, such thatc1​B≤A≤c2​Bc\_\{1\}B\\leq A\\leq c\_\{2\}B\.

Fix0≤c<10\\leq c<1\. LetUε​\(𝒌\)U\_\{\\varepsilon\}\(\\bm\{k\}\)be the minimum of the upper bounds \([4\.7](https://arxiv.org/html/2610.41179#S4.E7)\) and \([4\.6](https://arxiv.org/html/2610.41179#S4.E6)\) overμ∈\(0,λ1\]\\mu\\in\(0,\\lambda\_\{1\}\]satisfying∑λj<μϖj​\(𝒌\)≤c​ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\leq c\\varepsilonand∑λj<μϖj​\(𝒌\)<ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)<\\varepsilon, respectively\. Similarly, letLε​\(𝒌\)L\_\{\\varepsilon\}\(\\bm\{k\}\)be the maximum of the lower bound \([4\.4](https://arxiv.org/html/2610.41179#S4.E4)\) and the lower bounds \([4\.5](https://arxiv.org/html/2610.41179#S4.E5)\) overμ∈\(0,λ1\]\\mu\\in\(0,\\lambda\_\{1\}\]satisfying∑λj≤μϖj​\(𝒌\)\>ε\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\>\\varepsilon\. Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)gives

Lε​\(𝒌\)≤Tε​\(𝒌\)≤Uε​\(𝒌\)\.L\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq T\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq U\_\{\\varepsilon\}\(\\bm\{k\}\)\.\(4\.13\)
By linear independence, it is trivial that if the represented partPV​χ𝒌P\_\{V\}\\chi\_\{\\bm\{k\}\}is learned accurately, i\.e\.,h𝒄𝒌=PV​χ𝒌h\_\{\\bm\{c\}\_\{\\bm\{k\}\}\}=P\_\{V\}\\chi\_\{\\bm\{k\}\}, then the unique coefficient vector is𝒄𝒌=∑j=1nλj−1/2\(qj\)𝒌¯𝒗j\\bm\{c\}\_\{\\bm\{k\}\}=\\sum\_\{j=1\}^\{n\}\\lambda\_\{j\}^\{\-1/2\}\\,\\overline\{\(q\_\{j\}\)\_\{\\bm\{k\}\}\}\\,\\bm\{v\}\_\{j\}\. Its norm satisfies‖𝒄𝒌‖2=ℓ𝒌​∑j=1nϖj​\(𝒌\)/λj\\left\\\|\\bm\{c\}\_\{\\bm\{k\}\}\\right\\\|^\{2\}=\\ell\_\{\\bm\{k\}\}\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(\\bm\{k\}\)/\\lambda\_\{j\}\. We define the harmonic meanλ𝒌H=\(∑j=1nϖj​\(𝒌\)/λj\)−1=ℓ𝒌/‖𝒄𝒌‖2\\lambda^\{H\}\_\{\\bm\{k\}\}=\(\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(\\bm\{k\}\)/\\lambda\_\{j\}\)^\{\-1\}=\\ell\_\{\\bm\{k\}\}/\\left\\\|\\bm\{c\}\_\{\\bm\{k\}\}\\right\\\|^\{2\}\.

Now we compare the harmonic meanλkH\\lambda^\{H\}\_\{k\}and the arithmetic meanλ¯k′\\bar\{\\lambda\}\_\{k^\{\\prime\}\}\. Specifically, onΩ=\[−1,1\]\\Omega=\[\-1,1\], we take

ϕ1​\(x\)=32​x,ϕ2​\(x\)=δN​\(x2−d0\),\\phi\_\{1\}\(x\)=\\sqrt\{\\frac\{3\}\{2\}\}\\,x,\\qquad\\phi\_\{2\}\(x\)=\\frac\{\\sqrt\{\\delta\}\}\{N\}\(x^\{2\}\-d\_\{0\}\),\(4\.14\)where

δ=10−6,d0=1−8π2,N=\(25−4​d03\+2​d02\)1/2\.\\delta=10^\{\-6\},\\qquad d\_\{0\}=1\-\\frac\{8\}\{\\pi^\{2\}\},\\qquad N=\\left\(\\frac\{2\}\{5\}\-\\frac\{4d\_\{0\}\}\{3\}\+2d\_\{0\}^\{2\}\\right\)^\{1/2\}\.\(4\.15\)The two basis functions are orthogonal, and the Gram eigenvalues are11andδ\\delta\. Figure[3](https://arxiv.org/html/2610.41179#S4.F3)showsλkH\\lambda^\{H\}\_\{k\}atk=π⁡\(m\+1/2\)k=\\pi\(m\+1/2\),m=0,…,40m=0,\\ldots,40, andλ¯k′\\bar\{\\lambda\}\_\{k^\{\\prime\}\}atk′=π⁡\(m′\+1/2\)k^\{\\prime\}=\\pi\(m^\{\\prime\}\+1/2\),m′=0,…,1000m^\{\\prime\}=0,\\ldots,1000\. Both means decrease strictly along the positive half\-integer lattice and converge toδ\\delta\. The harmonic mean is already close toδ\\deltaatk=3​π/2k=3\\pi/2, whereas the arithmetic mean approaches this limit more gradually\.

![Refer to caption](https://arxiv.org/html/2610.41179v1/two_polynomial_lambda_harmonic.png)

![Refer to caption](https://arxiv.org/html/2610.41179v1/two_polynomial_lambda_arithmetic.png)

Figure 3:Comparison betweenλkH\\lambda^\{H\}\_\{k\}\(left\) andλ¯k′\\bar\{\\lambda\}\_\{k^\{\\prime\}\}\(right\) for the two\-polynomial basis\.The next theorem compares two separate training runs, one with targetχ𝒌\\chi\_\{\\bm\{k\}\}and one with targetχ𝒌′\\chi\_\{\\bm\{k\}^\{\\prime\}\}, using the same basis functions, learning rate, and zero initialization\.

###### Theorem 4\.2\.

Let0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\},0<ε<10<\\varepsilon<1, and0≤c<10\\leq c<1, and let𝐤,𝐤′∈Λ\\bm\{k\},\\bm\{k\}^\{\\prime\}\\in\\Lambdawithℓ𝐤\>0\\ell\_\{\\bm\{k\}\}\>0andℓ𝐤′\>0\\ell\_\{\\bm\{k\}^\{\\prime\}\}\>0\.

1. \(a\)IfUε​\(𝒌\)<Lε​\(𝒌′\)U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), thenTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\.
2. \(b\)Assume that there are0<μ′<μ≤λ10<\\mu^\{\\prime\}<\\mu\\leq\\lambda\_\{1\}andε<p≤1\\varepsilon<p\\leq 1such that∑λj<μϖj​\(𝒌\)≤c​ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(\\bm\{k\}\)\\leq c\\varepsilon,∑λj≤μ′ϖj​\(𝒌′\)≥p\\sum\_\{\\lambda\_\{j\}\\leq\\mu^\{\\prime\}\}\\varpi\_\{j\}\(\\bm\{k\}^\{\\prime\}\)\\geq p, and μ′μ<log⁡\(p/ε\)log⁡\(p/ε\)\+log⁡\(1/\(\(1−c\)​ε\)\)\+1\.\\frac\{\\mu^\{\\prime\}\}\{\\mu\}<\\frac\{\\log\(p/\\varepsilon\)\}\{\\log\(p/\\varepsilon\)\+\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)\+1\}\.\(4\.16\)ThenUε​\(𝒌\)<Lε​\(𝒌′\)U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), and henceTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\.
3. \(c\)If λ¯𝒌′<κ⁡\(ε\)​λ𝒌H,κ⁡\(ε\)=e​ε​log⁡\(1/ε\)1\+e​ε​\(1\+log⁡\(1/ε\)\),\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}<\\kappa\(\\varepsilon\)\\,\\lambda^\{H\}\_\{\\bm\{k\}\},\\qquad\\kappa\(\\varepsilon\)=\\frac\{\\mathrm\{e\}\\varepsilon\\log\(1/\\varepsilon\)\}\{1\+\\mathrm\{e\}\\varepsilon\\,\(1\+\\log\(1/\\varepsilon\)\)\},\(4\.17\)thenUε​\(𝒌\)<Lε​\(𝒌′\)U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), and henceTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\.

###### Proof\.

\(a\) By \([4\.13](https://arxiv.org/html/2610.41179#S4.E13)\),Tε​\(𝒌\)≤Uε​\(𝒌\)<Lε​\(𝒌′\)≤Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\\leq T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\.

\(b\) By \([4\.7](https://arxiv.org/html/2610.41179#S4.E7)\),Uε​\(𝒌\)≤log⁡\(1/\(\(1−c\)​ε\)\)/\(η​μ\)\+1U\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)/\(\\eta\\mu\)\+1\. By \([4\.5](https://arxiv.org/html/2610.41179#S4.E5)\),∑λj≤μ′ϖj​\(𝒌′\)≥p\>ε\\sum\_\{\\lambda\_\{j\}\\leq\\mu^\{\\prime\}\}\\varpi\_\{j\}\(\\bm\{k\}^\{\\prime\}\)\\geq p\>\\varepsilon, and−log⁡\(1−z\)≤z/\(1−z\)\-\\log\(1\-z\)\\leq z/\(1\-z\), we have

Lε​\(𝒌′\)≥log⁡\(p/ε\)−log⁡\(1−η​μ′\)≥\(1−η​μ′\)​log⁡\(p/ε\)η​μ′\.L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\\geq\\frac\{\\log\(p/\\varepsilon\)\}\{\-\\log\(1\-\\eta\\mu^\{\\prime\}\)\}\\geq\\frac\{\(1\-\\eta\\mu^\{\\prime\}\)\\log\(p/\\varepsilon\)\}\{\\eta\\mu^\{\\prime\}\}\.\(4\.18\)Sinceη​μ≤1\\eta\\mu\\leq 1, we haveη​μ′≤μ′/μ\\eta\\mu^\{\\prime\}\\leq\\mu^\{\\prime\}/\\mu\. Hence

η​μ′​\(Lε​\(𝒌′\)−Uε​\(𝒌\)\)\\displaystyle\\eta\\mu^\{\\prime\}\\bigl\(L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\-U\_\{\\varepsilon\}\(\\bm\{k\}\)\\bigr\)≥\(1−η​μ′\)​log⁡\(p/ε\)−μ′μ​log⁡\(1/\(\(1−c\)​ε\)\)−η​μ′\\displaystyle\\geq\(1\-\\eta\\mu^\{\\prime\}\)\\log\(p/\\varepsilon\)\-\\frac\{\\mu^\{\\prime\}\}\{\\mu\}\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)\-\\eta\\mu^\{\\prime\}\(4\.19\)≥\(1−μ′μ\)​log⁡\(p/ε\)−μ′μ​\(log⁡\(1/\(\(1−c\)​ε\)\)\+1\),\\displaystyle\\geq\\Bigl\(1\-\\frac\{\\mu^\{\\prime\}\}\{\\mu\}\\Bigr\)\\log\(p/\\varepsilon\)\-\\frac\{\\mu^\{\\prime\}\}\{\\mu\}\\Bigl\(\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)\+1\\Bigr\),and the right\-hand side is positive by \([4\.16](https://arxiv.org/html/2610.41179#S4.E16)\)\.

\(c\) Takeμ=λn\\mu=\\lambda\_\{n\}in \([4\.6](https://arxiv.org/html/2610.41179#S4.E6)\)\. Since∑λj<λnϖj​\(𝒌\)=0\\sum\_\{\\lambda\_\{j\}<\\lambda\_\{n\}\}\\varpi\_\{j\}\(\\bm\{k\}\)=0and∑j=1nϖj​\(𝒌\)/λj=1/λ𝒌H\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(\\bm\{k\}\)/\\lambda\_\{j\}=1/\\lambda^\{H\}\_\{\\bm\{k\}\}, we haveUε​\(𝒌\)≤1\+1/\(e​η​ε​λ𝒌H\)U\_\{\\varepsilon\}\(\\bm\{k\}\)\\leq 1\+1/\(\\mathrm\{e\}\\eta\\varepsilon\\lambda^\{H\}\_\{\\bm\{k\}\}\)\. By \([4\.4](https://arxiv.org/html/2610.41179#S4.E4)\),Lε​\(𝒌′\)≥\(1−η​λ¯𝒌′\)​log⁡\(1/ε\)/\(η​λ¯𝒌′\)L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\\geq\(1\-\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}\)\\log\(1/\\varepsilon\)/\(\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}\)\. Letx=η​λ¯𝒌′x=\\eta\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}andy=η​λ𝒌Hy=\\eta\\lambda^\{H\}\_\{\\bm\{k\}\}\. Theny≤1y\\leq 1, sinceλ𝒌H≤λ1\\lambda^\{H\}\_\{\\bm\{k\}\}\\leq\\lambda\_\{1\}, andx<κ⁡\(ε\)​y<1x<\\kappa\(\\varepsilon\)y<1\. Usingx≤x/yx\\leq x/y, we obtain

x⁡\(Lε​\(𝒌′\)−Uε​\(𝒌\)\)≥\(1−x\)​log⁡\(1/ε\)−x−xe​ε​y≥log⁡\(1/ε\)−xy​\(log⁡\(1/ε\)\+1\+1e​ε\)\.x\\bigl\(L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\-U\_\{\\varepsilon\}\(\\bm\{k\}\)\\bigr\)\\geq\(1\-x\)\\log\(1/\\varepsilon\)\-x\-\\frac\{x\}\{\\mathrm\{e\}\\varepsilon y\}\\geq\\log\(1/\\varepsilon\)\-\\frac\{x\}\{y\}\\Bigl\(\\log\(1/\\varepsilon\)\+1\+\\frac\{1\}\{\\mathrm\{e\}\\varepsilon\}\\Bigr\)\.\(4\.20\)Sinceκ⁡\(ε\)=log⁡\(1/ε\)/\(log⁡\(1/ε\)\+1\+1/\(e​ε\)\)\\kappa\(\\varepsilon\)=\\log\(1/\\varepsilon\)/\(\\log\(1/\\varepsilon\)\+1\+1/\(\\mathrm\{e\}\\varepsilon\)\), the right\-hand side is positive by \([4\.17](https://arxiv.org/html/2610.41179#S4.E17)\)\. ∎

Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)gives sufficient conditions for the learning order in these two single\-frequency runs\. Result \(a\) gives the general form; results \(b\) and \(c\) give sufficient conditions for \(a\)\. We remark that the conditions in \(b\) and \(c\) are not strict, which can be satisfied by many basis classes, e\.g\., unnormalized Legendre bases and weighted Fourier bases for suitable frequency pairs\. See Section[4\.3\.2](https://arxiv.org/html/2610.41179#S4.SS3.SSS2)for numerical examples\.

We usually study the frequency principle by comparing two frequencies with‖𝒌‖<‖𝒌′‖\\left\\\|\\bm\{k\}\\right\\\|<\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|and asking whetherTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\. Here Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(c\) uses the conditionλ¯𝒌′<κ⁡\(ε\)​λ𝒌H\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}<\\kappa\(\\varepsilon\)\\lambda^\{H\}\_\{\\bm\{k\}\}as a sufficient criterion in place ofk<k′k<k^\{\\prime\}\. In general,λ¯𝒌′\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}does not tend to zero; instead, it approaches a positive limit\. Thus takingk′k^\{\\prime\}sufficiently large does not guarantee \([4\.17](https://arxiv.org/html/2610.41179#S4.E17)\) for every fixedkkandε\\varepsilon\. For suitable choices ofkkandε\\varepsilon, however, the condition can hold for all sufficiently largek′k^\{\\prime\}\. In this case,Tε​\(k\)<Tε​\(k′\)T\_\{\\varepsilon\}\(k\)<T\_\{\\varepsilon\}\(k^\{\\prime\}\), giving a limited form of the frequency principle\.

For many classes, the condition in Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(a\) also implies the frequency order\. For example, takeϕs=\(1\+‖𝒌s‖\)−p​χ𝒌s\\phi\_\{s\}=\(1\+\\left\\\|\\bm\{k\}\_\{s\}\\right\\\|\)^\{\-p\}\\chi\_\{\\bm\{k\}\_\{s\}\},s=1,…,ns=1,\\ldots,n, wherep\>0p\>0and𝒌1,…,𝒌n∈Λ\\bm\{k\}\_\{1\},\\ldots,\\bm\{k\}\_\{n\}\\in\\Lambdaare distinct\. Under the assumptions of Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2), each selected frequency𝒌\\bm\{k\}hasℓ𝒌=1\\ell\_\{\\bm\{k\}\}=1andρ𝒌​\(t\)=\[1−η​\(1\+‖𝒌‖\)−2​p\]t\\rho\_\{\\bm\{k\}\}\(t\)=\\bigl\[1\-\\eta\(1\+\\left\\\|\\bm\{k\}\\right\\\|\)^\{\-2p\}\\bigr\]^\{t\}\. Henceρ𝒌​\(t\)\\rho\_\{\\bm\{k\}\}\(t\)andTε​\(𝒌\)T\_\{\\varepsilon\}\(\\bm\{k\}\)are nondecreasing in‖𝒌‖\\left\\\|\\bm\{k\}\\right\\\|\. Consequently, for𝒌,𝒌′∈\{𝒌1,…,𝒌n\}\\bm\{k\},\\bm\{k\}^\{\\prime\}\\in\\\{\\bm\{k\}\_\{1\},\\ldots,\\bm\{k\}\_\{n\}\\\}, ifUε​\(𝒌\)<Lε​\(𝒌′\)U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), then‖𝒌‖<‖𝒌′‖\\left\\\|\\bm\{k\}\\right\\\|<\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|\. This example clarifies the general frequency principle\. The usual low\-frequency priority states thatTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)whenever‖𝒌‖<‖𝒌′‖\\left\\\|\\bm\{k\}\\right\\\|<\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|\. In our general formulation, the conditionUε​\(𝒌\)<Lε​\(𝒌′\)U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)guaranteesTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)under the assumptions of Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\. Under additional conditions on the basis, as in the example above, the same comparison also implies‖𝒌‖<‖𝒌′‖\\left\\\|\\bm\{k\}\\right\\\|<\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|\. Thus, the comparison condition determines both the learning\-time order and the frequency order in these classes\.

The following corollary connects this condition to frequency magnitude through the decay of the basis functions\.

###### Corollary 4\.3\.

Let0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\}and0<ε<10<\\varepsilon<1, and assume \([2\.33](https://arxiv.org/html/2610.41179#S2.E33)\)\. Let𝐤∈Λ\\bm\{k\}\\in\\Lambdawithℓ𝐤\>0\\ell\_\{\\bm\{k\}\}\>0, and let𝐜𝐤\\bm\{c\}\_\{\\bm\{k\}\}be the unique coefficient vector satisfyingh𝐜𝐤=PV​χ𝐤h\_\{\\bm\{c\}\_\{\\bm\{k\}\}\}=P\_\{V\}\\chi\_\{\\bm\{k\}\}\. If𝐤′∈𝒜\\bm\{k\}^\{\\prime\}\\in\\mathcal\{A\}satisfies

ℓ𝒌′​‖𝒌′‖2​\(ν\+1\)\>n​Bϕ2​‖𝒄𝒌‖2κ⁡\(ε\)​ℓ𝒌,\\ell\_\{\\bm\{k\}^\{\\prime\}\}\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|^\{2\(\\nu\+1\)\}\>\\frac\{nB\_\{\\phi\}^\{2\}\\,\\left\\\|\\bm\{c\}\_\{\\bm\{k\}\}\\right\\\|^\{2\}\}\{\\kappa\(\\varepsilon\)\\,\\ell\_\{\\bm\{k\}\}\},\(4\.21\)thenTε​\(𝐤\)<Tε​\(𝐤′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\)\.

###### Proof\.

By \([2\.25](https://arxiv.org/html/2610.41179#S2.E25)\) and \([2\.34](https://arxiv.org/html/2610.41179#S2.E34)\),λ¯𝒌′=𝒦𝒌′​𝒌′/ℓ𝒌′≤n​Bϕ2​‖𝒌′‖−2​\(ν\+1\)/ℓ𝒌′\\bar\{\\lambda\}\_\{\\bm\{k\}^\{\\prime\}\}=\\mathcal\{K\}\_\{\\bm\{k\}^\{\\prime\}\\bm\{k\}^\{\\prime\}\}/\\ell\_\{\\bm\{k\}^\{\\prime\}\}\\leq nB\_\{\\phi\}^\{2\}\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|^\{\-2\(\\nu\+1\)\}/\\ell\_\{\\bm\{k\}^\{\\prime\}\}\. By \([4\.21](https://arxiv.org/html/2610.41179#S4.E21)\) andλ𝒌H=ℓ𝒌/‖𝒄𝒌‖2\\lambda^\{H\}\_\{\\bm\{k\}\}=\\ell\_\{\\bm\{k\}\}/\\left\\\|\\bm\{c\}\_\{\\bm\{k\}\}\\right\\\|^\{2\}, this is smaller thanκ⁡\(ε\)​λ𝒌H\\kappa\(\\varepsilon\)\\lambda^\{H\}\_\{\\bm\{k\}\}\. Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(c\) gives the claim\. ∎

Corollary[4\.3](https://arxiv.org/html/2610.41179#S4.Thmtheorem3)gives a sufficient condition in terms of frequency for lower frequencies to converge faster than higher frequencies\. For a fixed𝒌\\bm\{k\}, condition \([4\.21](https://arxiv.org/html/2610.41179#S4.E21)\) requiresℓ𝒌′​‖𝒌′‖2​\(ν\+1\)\\ell\_\{\\bm\{k\}^\{\\prime\}\}\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|^\{2\(\\nu\+1\)\}to be sufficiently large relative to‖𝒄𝒌‖2/ℓ𝒌\\left\\\|\\bm\{c\}\_\{\\bm\{k\}\}\\right\\\|^\{2\}/\\ell\_\{\\bm\{k\}\}\. Thus a larger‖𝒌′‖\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|helps satisfy the condition, provided thatℓ𝒌′\\ell\_\{\\bm\{k\}^\{\\prime\}\}does not become too small\. For a pair with‖𝒌‖<‖𝒌′‖\\left\\\|\\bm\{k\}\\right\\\|<\\left\\\|\\bm\{k\}^\{\\prime\}\\right\\\|satisfying this condition, the lower frequency reaches the prescribed relative error first\.

### 4\.2General targets

Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)compares two separate training runs, one with targetf=χ𝒌f=\\chi\_\{\\bm\{k\}\}and the other with targetf=χ𝒌′f=\\chi\_\{\\bm\{k\}^\{\\prime\}\}\. We now consider a single training run whose target contains several frequencies\.

We first extend \([4\.2](https://arxiv.org/html/2610.41179#S4.E2)\) from the single\-frequency targetχ𝒌\\chi\_\{\\bm\{k\}\}to any functionf∈L2​\(Ω\)f\\in L^\{2\}\(\\Omega\)\. Note thatPV​fP\_\{V\}fis the represented part that can be actually learned\. We assume thatℓf:=‖PV​f‖L2​\(Ω\)2\>0\\ell\_\{f\}:=\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}^\{2\}\>0\. We also defineϖj​\(f\)=\|⟨qj,f⟩Ω\|2ℓf=\|⟨qj,PV​f⟩Ω\|2ℓf\\varpi\_\{j\}\(f\)=\\frac\{\\left\|\\left\\langle q\_\{j\},f\\right\\rangle\_\{\\Omega\}\\right\|^\{2\}\}\{\\ell\_\{f\}\}=\\frac\{\\left\|\\left\\langle q\_\{j\},P\_\{V\}f\\right\\rangle\_\{\\Omega\}\\right\|^\{2\}\}\{\\ell\_\{f\}\}, which is the square of the ratio between the coefficient ofPV​fP\_\{V\}fin terms ofqjq\_\{j\}and its length\. Hence,ϖj​\(f\)∈\[0,1\]\\varpi\_\{j\}\(f\)\\in\[0,1\]and∑jϖj​\(f\)=1\\sum\_\{j\}\\varpi\_\{j\}\(f\)=1\.

Here we still consider zero initializationh\(0\)=0h^\{\(0\)\}=0\. We write erroref\(t\)=⟨h\(t\)−f,f⟩Ωe\_\{f\}^\{\(t\)\}=\\left\\langle h^\{\(t\)\}\-f,f\\right\\rangle\_\{\\Omega\}andef\(∞\)=⟨PV​f−f,f⟩Ωe\_\{f\}^\{\(\\infty\)\}=\\left\\langle P\_\{V\}f\-f,f\\right\\rangle\_\{\\Omega\}\. Then define the relative error

ρf​\(t\):=ef\(t\)−ef\(∞\)ef\(0\)−ef\(∞\)=⟨h\(t\),f⟩Ω−ℓf−ℓf=1−⟨h\(t\),f⟩Ωℓf\.\\rho\_\{f\}\(t\):=\\frac\{e\_\{f\}^\{\(t\)\}\-e\_\{f\}^\{\(\\infty\)\}\}\{e\_\{f\}^\{\(0\)\}\-e\_\{f\}^\{\(\\infty\)\}\}=\\frac\{\\langle h^\{\(t\)\},f\\rangle\_\{\\Omega\}\-\\ell\_\{f\}\}\{\-\\ell\_\{f\}\}=1\-\\frac\{\\langle h^\{\(t\)\},f\\rangle\_\{\\Omega\}\}\{\\ell\_\{f\}\}\.\(4\.22\)By \([3\.14](https://arxiv.org/html/2610.41179#S3.E14)\), we have

ρf​\(t\)=1−⟨h\(t\),f⟩Ωℓf=∑j=1nϖj​\(f\)​\(1−η​λj\)t\.\\rho\_\{f\}\(t\)=1\-\\frac\{\\left\\langle h^\{\(t\)\},f\\right\\rangle\_\{\\Omega\}\}\{\\ell\_\{f\}\}=\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(f\)\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\.\(4\.23\)For0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\},ρf​\(t\)\\rho\_\{f\}\(t\)is real and decreases monotonically from one to zero\. Similar to the notations in previous sections, we define the arithmetic mean by

λ¯f=∑j=1nϖj​\(f\)​λj,\\bar\{\\lambda\}\_\{f\}=\\sum\_\{j=1\}^\{n\}\\varpi\_\{j\}\(f\)\\lambda\_\{j\},\(4\.24\)and the learning time required to reduce the relative error belowε\\varepsilonby

Tε​\(f\)=min⁡\{t∈ℕ0:ρf​\(t\)≤ε\}\.T\_\{\\varepsilon\}\(f\)=\\min\\left\\\{t\\in\\mathbb\{N\}\_\{0\}:\\rho\_\{f\}\(t\)\\leq\\varepsilon\\right\\\}\.\(4\.25\)Let0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\}and0<ε<10<\\varepsilon<1, and fix0≤c<10\\leq c<1\. DefineUε​\(f\)U\_\{\\varepsilon\}\(f\)as the minimum of

1\+log⁡\(1/\(\(1−c\)​ε\)\)η​μand1\+∑λj≥μϖj​\(f\)/λje​η​\(ε−∑λj<μϖj​\(f\)\)1\+\\frac\{\\log\\bigl\(1/\(\(1\-c\)\\varepsilon\)\\bigr\)\}\{\\eta\\mu\}\\qquad\\text\{and\}\\qquad 1\+\\frac\{\\sum\_\{\\lambda\_\{j\}\\geq\\mu\}\\varpi\_\{j\}\(f\)/\\lambda\_\{j\}\}\{\\mathrm\{e\}\\,\\eta\\,\\bigl\(\\varepsilon\-\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(f\)\\bigr\)\}\(4\.26\)overμ∈\(0,λ1\]\\mu\\in\(0,\\lambda\_\{1\}\]satisfying∑λj<μϖj​\(f\)≤c​ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(f\)\\leq c\\varepsilonand∑λj<μϖj​\(f\)<ε\\sum\_\{\\lambda\_\{j\}<\\mu\}\\varpi\_\{j\}\(f\)<\\varepsilon, respectively\. DefineLε​\(f\)L\_\{\\varepsilon\}\(f\)as the maximum of

log⁡\(1/ε\)−log⁡\(1−η​λ¯f\)andlog⁡\(ε−1​∑λj≤μϖj​\(f\)\)−log⁡\(1−η​μ\),\\frac\{\\log\(1/\\varepsilon\)\}\{\-\\log\(1\-\\eta\\bar\{\\lambda\}\_\{f\}\)\}\\qquad\\text\{and\}\\qquad\\frac\{\\log\\bigl\(\\varepsilon^\{\-1\}\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(f\)\\bigr\)\}\{\-\\log\(1\-\\eta\\mu\)\},\(4\.27\)where the second expression is maximized overμ∈\(0,λ1\]\\mu\\in\(0,\\lambda\_\{1\}\]satisfying∑λj≤μϖj​\(f\)\>ε\\sum\_\{\\lambda\_\{j\}\\leq\\mu\}\\varpi\_\{j\}\(f\)\>\\varepsilon\. The two expressions definingLε​\(f\)L\_\{\\varepsilon\}\(f\)are taken to be zero ifη​λ¯f=1\\eta\\bar\{\\lambda\}\_\{f\}=1orη​μ=1\\eta\\mu=1, respectively\.

For a general targetff, the next theorem bounds the absolute error\.

###### Theorem 4\.4\.

Let0<η<2/λ10<\\eta<2/\\lambda\_\{1\},f∈L2​\(Ω\)f\\in L^\{2\}\(\\Omega\), andh\(0\)=0h^\{\(0\)\}=0\. The following statements hold for allt∈ℕ0t\\in\\mathbb\{N\}\_\{0\}\.

1. \(a\)For everyffwith‖f‖L2​\(Ω\)=1\\left\\\|f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}=1, \|ef\(t\)−ef\(∞\)\|≤‖PV​f‖L2​\(Ω\)​\(ℓf​ρf​\(2​t\)\)1/2\.\\left\|e\_\{f\}^\{\(t\)\}\-e\_\{f\}^\{\(\\infty\)\}\\right\|\\leq\\left\\\|P\_\{V\}f\\right\\\|\_\{L^\{2\}\(\\Omega\)\}\\bigl\(\\ell\_\{f\}\\,\\rho\_\{f\}\(2t\)\\bigr\)^\{1/2\}\.\(4\.28\)
2. \(b\)Iff=∑l=1Kal​ulf=\\sum\_\{l=1\}^\{K\}a\_\{l\}u\_\{l\}, whereK∈ℤ\+K\\in\\mathbb\{Z\}^\{\+\},al∈ℂa\_\{l\}\\in\\mathbb\{C\}andu1,…,uKu\_\{1\},\\ldots,u\_\{K\}are orthonormal functions under⟨⋅,⋅⟩Ω\\left\\langle\\cdot,\\cdot\\right\\rangle\_\{\\Omega\}withℓul\>0\\ell\_\{u\_\{l\}\}\>0\(l=1,…,Kl=1,\\dots,K\), then eul\(t\)−eul\(∞\)\\displaystyle e\_\{u\_\{l\}\}^\{\(t\)\}\-e\_\{u\_\{l\}\}^\{\(\\infty\)\}=−al​ℓul​ρul​\(t\)−∑i≠lai​Ci​l​\(t\),\\displaystyle=\-a\_\{l\}\\,\\ell\_\{u\_\{l\}\}\\rho\_\{u\_\{l\}\}\(t\)\-\\sum\_\{i\\neq l\}a\_\{i\}\\,C\_\{il\}\(t\),\(4\.29\)Ci​l​\(t\)\\displaystyle C\_\{il\}\(t\)=∑j=1n\(1−η​λj\)t​⟨ui,qj⟩Ω​⟨qj,ul⟩Ω\.\\displaystyle=\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle u\_\{i\},q\_\{j\}\\right\\rangle\_\{\\Omega\}\\left\\langle q\_\{j\},u\_\{l\}\\right\\rangle\_\{\\Omega\}\.The cross terms satisfy \|Ci​l​\(t\)\|\\displaystyle\\left\|C\_\{il\}\(t\)\\right\|≤\(ℓui​ℓul\)1/2​min⁡\{ρui​\(2​t\),ρul​\(2​t\)\}1/2,\\displaystyle\\leq\(\\ell\_\{u\_\{i\}\}\\ell\_\{u\_\{l\}\}\)^\{1/2\}\\min\\\{\\rho\_\{u\_\{i\}\}\(2t\),\\rho\_\{u\_\{l\}\}\(2t\)\\\}^\{1/2\},\(4\.30\)\|Ci​l​\(0\)\|\\displaystyle\\left\|C\_\{il\}\(0\)\\right\|≤\(\(1−ℓui\)​\(1−ℓul\)\)1/2\.\\displaystyle\\leq\\bigl\(\(1\-\\ell\_\{u\_\{i\}\}\)\(1\-\\ell\_\{u\_\{l\}\}\)\\bigr\)^\{1/2\}\.Ifη≤1/λ1\\eta\\leq 1/\\lambda\_\{1\}, then moreover\|Ci​l​\(t\)−Ci​l​\(0\)\|≤\(ℓui​ℓul\)1/2​min⁡\{1−ρui​\(t\),1−ρul​\(t\)\}1/2\\left\|C\_\{il\}\(t\)\-C\_\{il\}\(0\)\\right\|\\leq\(\\ell\_\{u\_\{i\}\}\\ell\_\{u\_\{l\}\}\)^\{1/2\}\\min\\\{1\-\\rho\_\{u\_\{i\}\}\(t\),1\-\\rho\_\{u\_\{l\}\}\(t\)\\\}^\{1/2\}\.

###### Proof\.

Expandingh\(t\)−PV​fh^\{\(t\)\}\-P\_\{V\}fin the basisqjq\_\{j\}as in Proposition[2\.6](https://arxiv.org/html/2610.41179#S2.Thmtheorem6)gives

ef\(t\)−ef\(∞\)=⟨h\(t\)−PV​f,f⟩Ω=∑j=1n\(1−η​λj\)t​⟨h\(0\)−f,qj⟩Ω​⟨qj,f⟩Ω\.e\_\{f\}^\{\(t\)\}\-e\_\{f\}^\{\(\\infty\)\}=\\left\\langle h^\{\(t\)\}\-P\_\{V\}f,f\\right\\rangle\_\{\\Omega\}=\\sum\_\{j=1\}^\{n\}\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\\left\\langle h^\{\(0\)\}\-f,q\_\{j\}\\right\\rangle\_\{\\Omega\}\\left\\langle q\_\{j\},f\\right\\rangle\_\{\\Omega\}\.\(4\.31\)Part \(a\) follows by the Cauchy–Schwarz inequality\. Substitutingf=∑lal​ulf=\\sum\_\{l\}a\_\{l\}u\_\{l\}andh\(0\)=0h^\{\(0\)\}=0gives \([4\.29](https://arxiv.org/html/2610.41179#S4.E29)\)\. Applying the same inequality toCi​l​\(t\)C\_\{il\}\(t\), with the factor\(1−η​λj\)t\(1\-\\eta\\lambda\_\{j\}\)^\{t\}in either sum, gives the first bound in \([4\.30](https://arxiv.org/html/2610.41179#S4.E30)\)\. The second follows fromCi​l​\(0\)=−⟨\(I−PV\)​ul,\(I−PV\)​ui⟩ΩC\_\{il\}\(0\)=\-\\left\\langle\(I\-P\_\{V\}\)u\_\{l\},\(I\-P\_\{V\}\)u\_\{i\}\\right\\rangle\_\{\\Omega\}fori≠li\\neq l\. Ifη≤1/λ1\\eta\\leq 1/\\lambda\_\{1\}, then\(1−\(1−η​λj\)t\)2≤1−\(1−η​λj\)t\(1\-\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\)^\{2\}\\leq 1\-\(1\-\\eta\\lambda\_\{j\}\)^\{t\}\. Another application of the Cauchy–Schwarz inequality therefore gives\|Ci​l​\(t\)−Ci​l​\(0\)\|≤\(ℓui​ℓul​\(1−ρul​\(t\)\)\)1/2\\left\|C\_\{il\}\(t\)\-C\_\{il\}\(0\)\\right\|\\leq\\bigl\(\\ell\_\{u\_\{i\}\}\\ell\_\{u\_\{l\}\}\(1\-\\rho\_\{u\_\{l\}\}\(t\)\)\\bigr\)^\{1/2\}, and interchangingiiandllcompletes the proof\. ∎

Letf=a​u\+a′​u′f=au\+a^\{\\prime\}u^\{\\prime\}witha,a′≠0a,a^\{\\prime\}\\neq 0, whereu,u′∈L2​\(Ω\)u,u^\{\\prime\}\\in L^\{2\}\(\\Omega\)are orthonormal withℓu,ℓu′\>0\\ell\_\{u\},\\ell\_\{u^\{\\prime\}\}\>0\. Define

β=\|a′\|​ℓu′1/2\|a\|​ℓu1/2,γ=\(\(1−ℓu\)​\(1−ℓu′\)ℓu​ℓu′\)1/2\.\\beta=\\frac\{\\left\|a^\{\\prime\}\\right\|\\,\\ell\_\{u^\{\\prime\}\}^\{1/2\}\}\{\\left\|a\\right\|\\,\\ell\_\{u\}^\{1/2\}\},\\qquad\\gamma=\\Bigl\(\\frac\{\(1\-\\ell\_\{u\}\)\(1\-\\ell\_\{u^\{\\prime\}\}\)\}\{\\ell\_\{u\}\\ell\_\{u^\{\\prime\}\}\}\\Bigr\)^\{1/2\}\.\(4\.32\)Forv∈\{u,u′\}v\\in\\\{u,u^\{\\prime\}\\\}withev\(0\)≠ev\(∞\)e\_\{v\}^\{\(0\)\}\\neq e\_\{v\}^\{\(\\infty\)\}, also defineRv​\(t\)=\|ev\(t\)−ev\(∞\)\|/\|ev\(0\)−ev\(∞\)\|R\_\{v\}\(t\)=\\left\|e\_\{v\}^\{\(t\)\}\-e\_\{v\}^\{\(\\infty\)\}\\right\|/\\left\|e\_\{v\}^\{\(0\)\}\-e\_\{v\}^\{\(\\infty\)\}\\right\|\. For0<ε<10<\\varepsilon<1, letTε​\(v,f\)=min⁡\{t∈ℕ0:Rv​\(s\)≤ε​for all​s≥t\}T\_\{\\varepsilon\}\(v;f\)=\\min\\\{t\\in\\mathbb\{N\}\_\{0\}:R\_\{v\}\(s\)\\leq\\varepsilon\\text\{ for all \}s\\geq t\\\}\. This is the first iteration from which the relative error stays at mostε\\varepsilon; the error need not decrease monotonically\. The next theorem combines these estimates with the amplitude ratio to compare two frequencies in one run\.

###### Theorem 4\.5\.

For the targetffdefined above, assume0<η≤1/λ10<\\eta\\leq 1/\\lambda\_\{1\},0<ε<10<\\varepsilon<1,h\(0\)=0h^\{\(0\)\}=0,β​γ<1\\beta\\gamma<1, andγ<β\\gamma<\\beta\. Then, for allt∈ℕ0t\\in\\mathbb\{N\}\_\{0\},

Ru​\(t\)\\displaystyle R\_\{u\}\(t\)≤ρu​\(t\)\+β​ρu​\(2​t\)1/21−β​γ,\\displaystyle\\leq\\frac\{\\rho\_\{u\}\(t\)\+\\beta\\rho\_\{u\}\(2t\)^\{1/2\}\}\{1\-\\beta\\gamma\},\(4\.33\)Ru′​\(t\)\\displaystyle R\_\{u^\{\\prime\}\}\(t\)≥ρu′​\(t\)−β−1​min⁡\{ρu​\(2​t\)1/2,γ\+\(1−ρu′​\(t\)\)1/2\}1\+β−1​γ\.\\displaystyle\\geq\\frac\{\\rho\_\{u^\{\\prime\}\}\(t\)\-\\beta^\{\-1\}\\min\\bigl\\\{\\rho\_\{u\}\(2t\)^\{1/2\},\\,\\gamma\+\(1\-\\rho\_\{u^\{\\prime\}\}\(t\)\)^\{1/2\}\\bigr\\\}\}\{1\+\\beta^\{\-1\}\\gamma\}\.\(4\.34\)In particular, if the right\-hand side of \([4\.33](https://arxiv.org/html/2610.41179#S4.E33)\) is at mostε\\varepsilonand the right\-hand side of \([4\.34](https://arxiv.org/html/2610.41179#S4.E34)\) is larger thanε\\varepsilonfor somet1∈ℕ0t\_\{1\}\\in\\mathbb\{N\}\_\{0\}, thenTε​\(u,f\)≤t1<Tε​\(u′,f\)T\_\{\\varepsilon\}\(u;f\)\\leq t\_\{1\}<T\_\{\\varepsilon\}\(u^\{\\prime\};f\)\.

###### Proof\.

By Theorem[4\.4](https://arxiv.org/html/2610.41179#S4.Thmtheorem4)\(b\) att=0t=0, we have\|eu\(0\)−eu\(∞\)\|≥\|a\|​ℓu​\(1−β​γ\)\\left\|e\_\{u\}^\{\(0\)\}\-e\_\{u\}^\{\(\\infty\)\}\\right\|\\geq\\left\|a\\right\|\\ell\_\{u\}\(1\-\\beta\\gamma\)and\|a′\|​ℓu′​\(1−β−1​γ\)≤\|eu′\(0\)−eu′\(∞\)\|≤\|a′\|​ℓu′​\(1\+β−1​γ\)\\left\|a^\{\\prime\}\\right\|\\ell\_\{u^\{\\prime\}\}\(1\-\\beta^\{\-1\}\\gamma\)\\leq\\left\|e\_\{u^\{\\prime\}\}^\{\(0\)\}\-e\_\{u^\{\\prime\}\}^\{\(\\infty\)\}\\right\|\\leq\\left\|a^\{\\prime\}\\right\|\\ell\_\{u^\{\\prime\}\}\(1\+\\beta^\{\-1\}\\gamma\)\. Thus both denominators are positive\. Applying \([4\.30](https://arxiv.org/html/2610.41179#S4.E30)\) to the two terms in \([4\.29](https://arxiv.org/html/2610.41179#S4.E29)\) gives \([4\.33](https://arxiv.org/html/2610.41179#S4.E33)\)\. For \([4\.34](https://arxiv.org/html/2610.41179#S4.E34)\), also use\|C12​\(t\)\|≤\|C12​\(0\)\|\+\|C12​\(t\)−C12​\(0\)\|\\left\|C\_\{12\}\(t\)\\right\|\\leq\\left\|C\_\{12\}\(0\)\\right\|\+\\left\|C\_\{12\}\(t\)\-C\_\{12\}\(0\)\\right\|and the last bound of Proposition[4\.4](https://arxiv.org/html/2610.41179#S4.Thmtheorem4)\(b\), followed by the reverse triangle inequality\. Finally, the upper bound in \([4\.33](https://arxiv.org/html/2610.41179#S4.E33)\) is nonincreasing\. HenceRu​\(s\)≤εR\_\{u\}\(s\)\\leq\\varepsilonfor alls≥t1s\\geq t\_\{1\}, whileRu′​\(t1\)\>εR\_\{u^\{\\prime\}\}\(t\_\{1\}\)\>\\varepsilongivesTε​\(u′,f\)\>t1T\_\{\\varepsilon\}\(u^\{\\prime\};f\)\>t\_\{1\}\. ∎

Whenℓu\\ell\_\{u\}andℓu′\\ell\_\{u^\{\\prime\}\}are close to one,γ\\gammais small\. If\|a\|\|a\|and\|a′\|\|a^\{\\prime\}\|are also comparable, thenβ\\betais of order one\. In this case,β​γ<1\\beta\\gamma<1andγ<β\\gamma<\\betaare satisfied\. A specific example will be presented in Section[4\.3\.3](https://arxiv.org/html/2610.41179#S4.SS3.SSS3)\. Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)compares two separate training runs, whereas Theorem[4\.5](https://arxiv.org/html/2610.41179#S4.Thmtheorem5)compares two components learned in the same run\. In the latter case, each component can affect the error in the other direction through the cross terms\. The conditions in Theorem[4\.5](https://arxiv.org/html/2610.41179#S4.Thmtheorem5)also control these effects to establish the learning order\. A large difference between\|a\|\|a\|and\|a′\|\|a^\{\\prime\}\|can change this order\.

Here,Tε​\(v,f\)T\_\{\\varepsilon\}\(v;f\)is the smallestttsuch thatRv​\(s\)≤εR\_\{v\}\(s\)\\leq\\varepsilonfor everys≥ts\\geq t\. Thus,Tε​\(u,f\)<Tε​\(u′,f\)T\_\{\\varepsilon\}\(u;f\)<T\_\{\\varepsilon\}\(u^\{\\prime\};f\)means that the error in directionuustays belowε\\varepsilonearlier\. It does not mean that this error first falls belowε\\varepsilonearlier: the error in directionu′u^\{\\prime\}may fall belowε\\varepsilonand then rise above it again\. The assumptionsβ​γ<1\\beta\\gamma<1andγ<β\\gamma<\\betaensure that the denominators inRu​\(t\)R\_\{u\}\(t\)andRu′​\(t\)R\_\{u^\{\\prime\}\}\(t\)are nonzero\. They also implyγ2<1\\gamma^\{2\}<1, which is equivalent toℓu\+ℓu′\>1\\ell\_\{u\}\+\\ell\_\{u^\{\\prime\}\}\>1\. This restriction is part of the sufficient conditions in the theorem; the same learning order may still hold when it is not satisfied\.

### 4\.3Numerical experiments

#### 4\.3\.1Verification of Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)

We examine Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)onΩ=\[−1,1\]2\\Omega=\[\-1,1\]^\{2\}using tensor products of the one\-dimensional basis families

ℬL\\displaystyle\\mathcal\{B\}\_\{\\mathrm\{L\}\}=\{Pr:r=0,…,63\},\\displaystyle=\\\{P\_\{r\}:r=0,\\ldots,63\\\},\(4\.35\)ℬW\\displaystyle\\mathcal\{B\}\_\{\\mathrm\{W\}\}=\{12\}∪\{cos⁡\(m​π​x\)1\+m,sin⁡\(m​π​x\)1\+m:m=1,…,32\},\\displaystyle=\\left\\\{\\frac\{1\}\{\\sqrt\{2\}\}\\right\\\}\\cup\\left\\\{\\frac\{\\cos\(m\\pi x\)\}\{\\sqrt\{1\+m\}\},\\frac\{\\sin\(m\\pi x\)\}\{\\sqrt\{1\+m\}\}:m=1,\\ldots,32\\right\\\},\(4\.36\)wherePrP\_\{r\}is the standard Legendre polynomial of degreerr\. The two classes contain40964096and42254225basis functions, respectively\. We setϑ=\(1/2,1/2\)\\vartheta=\(1/2,1/2\),η=1/4\\eta=1/4,ε=0\.1\\varepsilon=0\.1, andc=1/2c=1/2, and train each targetf=χ𝒌=2−1​ei​𝒌⋅𝒙f=\\chi\_\{\\bm\{k\}\}=2^\{\-1\}\\mathrm\{e\}^\{\\mathrm\{i\}\\bm\{k\}\\cdot\\bm\{x\}\}separately from zero initialization\. The largest Gram eigenvalues are44and11, so the step size satisfiesη≤1/λ1\\eta\\leq 1/\\lambda\_\{1\}for both classes\. We test the twelve frequencies

𝒌/π∈\{\\displaystyle\\bm\{k\}/\\pi\\in\\bigl\\\{\(0\.5,0\.5\),\(0\.5,1\.5\),\(1\.5,1\.5\),\(2\.5,2\.5\),\\displaystyle\(0\.5,0\.5\),\(0\.5,1\.5\),\(1\.5,1\.5\),\(2\.5,2\.5\),\(4\.37\)\(0\.5,5\.5\),\(5\.5,0\.5\),\(5\.5,5\.5\),\(5\.5,9\.5\),\\displaystyle\(0\.5,5\.5\),\(5\.5,0\.5\),\(5\.5,5\.5\),\(5\.5,9\.5\),\(10\.5,10\.5\),\(15\.5,15\.5\),\(24\.5,24\.5\),\(31\.5,31\.5\)\}\.\\displaystyle\(10\.5,10\.5\),\(15\.5,15\.5\),\(24\.5,24\.5\),\(31\.5,31\.5\)\\bigr\\\}\.
Using the corresponding Gram spectra and local spectral weights, we maximize \([4\.5](https://arxiv.org/html/2610.41179#S4.E5)\) and minimize \([4\.6](https://arxiv.org/html/2610.41179#S4.E6)\) and \([4\.7](https://arxiv.org/html/2610.41179#S4.E7)\) separately over their admissible cutoffs among the distinct Gram eigenvalues\. For example, at𝒌=π⁡\(5\.5,9\.5\)\\bm\{k\}=\\pi\(5\.5,9\.5\), the right\-hand sides of \([4\.5](https://arxiv.org/html/2610.41179#S4.E5)\)–\([4\.7](https://arxiv.org/html/2610.41179#S4.E7)\) are2166\.9752166\.975,4681\.7504681\.750, and6031\.4096031\.409for the Legendre class, and516\.224516\.224,1042\.0221042\.022, and1199\.2931199\.293for the weighted trigonometric class, respectively, to three decimal places\. We then computeTε​\(𝒌\)T\_\{\\varepsilon\}\(\\bm\{k\}\)by full batch coefficient gradient descent, using the exact Gram matrix and target projections evaluated by 384\-point Gauss–Legendre quadrature in each coordinate\. The learning time is the first iteration satisfyingρ𝒌​\(t\)≤ε\\rho\_\{\\bm\{k\}\}\(t\)\\leq\\varepsilon, with the relative error measured against the attainable Fourier limit as in \([4\.1](https://arxiv.org/html/2610.41179#S4.E1)\)\.

Figure[4](https://arxiv.org/html/2610.41179#S4.F4)compares the measured learning times \(black stars\) with the two lower and two upper bounds in \([4\.4](https://arxiv.org/html/2610.41179#S4.E4)\)–\([4\.7](https://arxiv.org/html/2610.41179#S4.E7)\)\. The bounds follow the variation in learning time across the sampled frequencies\. Both lower bounds lie belowTε​\(𝒌\)T\_\{\\varepsilon\}\(\\bm\{k\}\), and both upper bounds lie above it, for all2424runs, in agreement with Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)\. At𝒌=π⁡\(5\.5,9\.5\)\\bm\{k\}=\\pi\(5\.5,9\.5\), gradient descent givesTε​\(𝒌\)=3090T\_\{\\varepsilon\}\(\\bm\{k\}\)=3090and654654for the Legendre and weighted trigonometric classes, respectively\.

![Refer to caption](https://arxiv.org/html/2610.41179v1/theorem41_legendre_bounds.png)

![Refer to caption](https://arxiv.org/html/2610.41179v1/theorem41_weighted_trigonometric_bounds.png)

Figure 4:Verification of Theorem[4\.1](https://arxiv.org/html/2610.41179#S4.Thmtheorem1)for the two\-dimensional unnormalized Legendre basis \(left\) and weighted trigonometric basis \(right\)\.
#### 4\.3\.2Verification of Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)

- •Regions satisfying Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(a\)\. We use the Legendre and weighted trigonometric bases \([4\.35](https://arxiv.org/html/2610.41179#S4.E35)\) and \([4\.36](https://arxiv.org/html/2610.41179#S4.E36)\) onΩ=\[−1,1\]\\Omega=\[\-1,1\]\. We setϑ=1/2\\vartheta=1/2,η=1/2\\eta=1/2,ε=0\.1\\varepsilon=0\.1, andc=1/2c=1/2, and consider all frequency pairs inℱ2\\mathcal\{F\}^\{2\}, whereℱ=\{π\(j\+1/2\):j=0,…,31\}\\mathcal\{F\}=\\\{\\pi\(j\+1/2\):j=0,\\ldots,31\\\}\. Each targetf=χk=2−1/2ei​k​xf=\\chi\_\{k\}=2^\{\-1/2\}\\mathrm\{e\}^\{\\mathrm\{i\}kx\}is trained separately from zero initialization\. We compute the optimized boundsUεU\_\{\\varepsilon\},LεL\_\{\\varepsilon\}, and the learning times as in Section[4\.3\.1](https://arxiv.org/html/2610.41179#S4.SS3.SSS1)\. Figure[5](https://arxiv.org/html/2610.41179#S4.F5)shows the sampled regions whereUε​\(k\)<Lε​\(k′\)U\_\{\\varepsilon\}\(k\)<L\_\{\\varepsilon\}\(k^\{\\prime\}\)\. This condition holds for206206of the10241024pairs with the Legendre class and317317pairs with the weighted trigonometric class\. The colored cells all correspond to sampled pairs satisfyingk<k′k<k^\{\\prime\}\. For each such pair,Uε​\(k\)<Lε​\(k′\)U\_\{\\varepsilon\}\(k\)<L\_\{\\varepsilon\}\(k^\{\\prime\}\), and Theorem 4\.2\(a\) givesTε​\(k\)<Tε​\(k′\)T\_\{\\varepsilon\}\(k\)<T\_\{\\varepsilon\}\(k^\{\\prime\}\), i\.e\., frequencykkcan be learned faster\. Thus both orderings hold for these pairs, although the experiment does not establish a causal relation between them\. ![Refer to caption](https://arxiv.org/html/2610.41179v1/theorem42a_legendre_region.png) ![Refer to caption](https://arxiv.org/html/2610.41179v1/theorem42a_weighted_trigonometric_region.png) Figure 5:Frequency pairs satisfyingUε​\(k\)<Lε​\(k′\)U\_\{\\varepsilon\}\(k\)<L\_\{\\varepsilon\}\(k^\{\\prime\}\)\(teal\) for the Legendre class \(left\) and weighted trigonometric class \(right\)\.
- •Tests of Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(b\) and \(c\)\. We use the corresponding tensor product bases onΩ=\[−1,1\]2\\Omega=\[\-1,1\]^\{2\}, containing40964096and42254225basis functions, respectively\. We setϑ=\(1/2,1/2\)\\vartheta=\(1/2,1/2\),η=1/4\\eta=1/4,ε=0\.1\\varepsilon=0\.1,c=1/2c=1/2, andp=0\.8p=0\.8in Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(b\)\. Each targetf=χ𝒌=2−1​ei​𝒌⋅𝒙f=\\chi\_\{\\bm\{k\}\}=2^\{\-1\}\\mathrm\{e\}^\{\\mathrm\{i\}\\bm\{k\}\\cdot\\bm\{x\}\}is trained separately from zero initialization\. We test the two frequency pairs Case𝒌/π𝒌′/πA\(0\.5,1\.5\)\(5\.5,9\.5\)B\(2\.5,2\.5\)\(10\.5,10\.5\)\\begin\{array\}\[\]\{c\|cc\}\\text\{Case\}&\\bm\{k\}/\\pi&\\bm\{k\}^\{\\prime\}/\\pi\\\\ \\hline\\cr\\mathrm\{A\}&\(0\.5,1\.5\)&\(5\.5,9\.5\)\\\\ \\mathrm\{B\}&\(2\.5,2\.5\)&\(10\.5,10\.5\)\\end\{array\}\(4\.38\) Table[2](https://arxiv.org/html/2610.41179#S4.T2)reportsUε​\(𝒌\)U\_\{\\varepsilon\}\(\\bm\{k\}\),Lε​\(𝒌′\)L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), and the measured learning times for both classes\. Both frequency pairs satisfy the conditions in Theorem[4\.2](https://arxiv.org/html/2610.41179#S4.Thmtheorem2)\(b\) and \(c\) for each class\. In all four tests, the bounds satisfyUε​\(𝒌\)<Lε​\(𝒌′\)U\_\{\\varepsilon\}\(\\bm\{k\}\)<L\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), and gradient descent givesTε​\(𝒌\)<Tε​\(𝒌′\)T\_\{\\varepsilon\}\(\\bm\{k\}\)<T\_\{\\varepsilon\}\(\\bm\{k\}^\{\\prime\}\), in agreement with the predicted strict learning order\. Table 2:Bounds and learning times\.

#### 4\.3\.3Verification of Theorem[4\.5](https://arxiv.org/html/2610.41179#S4.Thmtheorem5)

We examine Theorem[4\.5](https://arxiv.org/html/2610.41179#S4.Thmtheorem5)onΩ=\[−1,1\]\\Omega=\[\-1,1\]using the unnormalized Legendre and weighted trigonometric bases \([4\.35](https://arxiv.org/html/2610.41179#S4.E35)\) and \([4\.36](https://arxiv.org/html/2610.41179#S4.E36)\), containing6464and6565functions, respectively\. We setϑ=1/2\\vartheta=1/2,η=1/2\\eta=1/2, andε=0\.1\\varepsilon=0\.1, and trainf=u\+u′f=u\+u^\{\\prime\}from zero initialization, whereu=2−1/2ei​k​xu=2^\{\-1/2\}\\mathrm\{e\}^\{\\mathrm\{i\}kx\}andu′=2−1/2ei​k′​xu^\{\\prime\}=2^\{\-1/2\}\\mathrm\{e\}^\{\\mathrm\{i\}k^\{\\prime\}x\}\. The frequency pairs\(k/π,k′/π\)\(k/\\pi,k^\{\\prime\}/\\pi\)are\(0\.5,9\.5\)\(0\.5,9\.5\)in Case A and\(2\.5,15\.5\)\(2\.5,15\.5\)in Case B\. The two modes are orthonormal, andη≤1/λ1\\eta\\leq 1/\\lambda\_\{1\}holds for both classes\. We use coefficient gradient descent with the exact Gram matrix and target projections evaluated by 384\-point Gauss–Legendre quadrature\. Each learning time is one plus the last iteration above the threshold, with a decreasing spectral envelope controlling the uncomputed tail\.

Table[3](https://arxiv.org/html/2610.41179#S4.T3)reports the right\-hand sides of \([4\.33](https://arxiv.org/html/2610.41179#S4.E33)\) and \([4\.34](https://arxiv.org/html/2610.41179#S4.E34)\) att1t\_\{1\}, together with the measured learning times\. All four tests satisfyβ​γ<1\\beta\\gamma<1andγ<β\\gamma<\\beta, and both bounds hold at every computed iteration\. At the listedt1t\_\{1\}, the right\-hand side of \([4\.33](https://arxiv.org/html/2610.41179#S4.E33)\) is belowε\\varepsilon, whereas that of \([4\.34](https://arxiv.org/html/2610.41179#S4.E34)\) is aboveε\\varepsilon\. Thus the lower\-frequency component reaches and remains below the prescribed relative error earlier, in agreement with Theorem[4\.5](https://arxiv.org/html/2610.41179#S4.Thmtheorem5)\.

Table 3:Bounds and learning times forf=u\+u′f=u\+u^\{\\prime\}\.

## 5Conclusion

We have studied a general frequency principle for gradient descent in finite linear approximation classes on a bounded box\. Our results describe two complementary aspects of frequency\-dependent learning\. Under zero initialization and suitable Fourier decay assumptions, the output amplitude relative to the target amplitude vanishes at high frequencies along the prescribed frequency set\. This holds at every fixed iteration and, under a stable step size, uniformly throughout training\. Such small relative movement can reflect limited approximation capacity even when convergence to the attainable limit is fast\.

To characterize convergence separately, we have derived learning\-time bounds after removing the limiting approximation error\. Comparing these bounds and the associated spectral quantities gives sufficient conditions for strict learning order between single\-frequency targets trained separately\. The analysis also gives conditions for two frequency components in the same run to reach and remain below a prescribed relative\-error threshold in a specified order, accounting for their interaction\.

The Legendre experiments illustrate the high\-frequency movement bounds\. Experiments with unnormalized Legendre and weighted trigonometric bases support the learning\-time bounds and the predicted order for the tested pairs\. The orthonormal periodic trigonometric example gives equal relative convergence rates for all represented frequencies\. These examples show how the framework characterizes different learning behaviors according to the basis and its scaling\.

Future work may focus on characterizing broader classes of basis functions for which the comparison condition also determines frequency order\. Sharper learning\-time bounds and estimates of interactions between components may extend the ordering results to targets with more than two components\. Another direction is to study models whose basis functions evolve during training, where changes in the training operator must be incorporated into the comparison criteria

#### Acknowledgments

This work was supported by the Major Research Plan of the National Natural Science Foundation of China \(92370101\)\.

#### Data availability

The data and code supporting the numerical experiments in this study are available from the corresponding author upon reasonable request\.

#### AI use declaration

ChatGPT \(OpenAI, GPT\-6\.1\-sol\) was used to generate the initial draft of the code for the numerical experiments and to identify the two\-polynomial example presented in Eq\. \([4\.14](https://arxiv.org/html/2610.41179#S4.E14)\)\. The authors take full responsibility for the final code, the example, and the numerical results reported in this manuscript\.

## References

- Z\. Allen\-Zhu, Y\. Li, and Z\. SongA convergence theory for deep learning via over\-parameterization\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 242–252\.Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Aroraet al\.\(2019a\)S\. Arora, S\. S\. Du, W\. Hu, Z\. Li, R\. Salakhutdinov, and R\. WangOn exact computation with an infinitely wide neural net\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:1904\.11955Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Aroraet al\.\(2019b\)S\. Arora, S\. S\. Du, W\. Hu, Z\. Li, and R\. WangFine\-grained analysis of optimization and generalization for overparameterized two\-layer neural networks\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 322–332\.Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Basriet al\.\(2020\)R\. Basri, M\. Galun, A\. Geifman, D\. Jacobs, Y\. Kasten, and S\. KritchmanFrequency bias in neural networks for input of non\-uniform density\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 685–694\.Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Basriet al\.\(2019\)R\. Basri, D\. Jacobs, Y\. Kasten, and S\. KritchmanThe convergence rate of neural networks for learned functions of different frequencies\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:1906\.00425Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Bowman and Montúfar \(2022\)B\. Bowman and G\. MontúfarImplicit bias of MSE gradient optimization in underparameterized neural networks\.InInternational Conference on Learning Representations,External Links:2201\.04738Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Caoet al\.\(2021\)Y\. Cao, Z\. Fang, Y\. Wu, D\. Zhou, and Q\. GuTowards understanding the spectral bias of deep learning\.InProceedings of the Thirtieth International Joint Conference on Artificial Intelligence,pp\. 2205–2211\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2021/304)Cited by:[Table 1](https://arxiv.org/html/2610.41179#S1.T1.2.5.1.1.1),[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Duet al\.\(2019\)S\. S\. Du, X\. Zhai, B\. Poczos, and A\. SinghGradient descent provably optimizes over\-parameterized neural networks\.InInternational Conference on Learning Representations,External Links:1810\.02054Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Englet al\.\(1996\)H\. W\. Engl, M\. Hanke, and A\. NeubauerRegularization of inverse problems\.Springer\.Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.41179#S2.SS2.p1.3),[§4\.1](https://arxiv.org/html/2610.41179#S4.SS1.p2.1)\.
- Jacotet al\.\(2018\)A\. Jacot, F\. Gabriel, and C\. HonglerNeural tangent kernel: convergence and generalization in neural networks\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:1806\.07572Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Landweber \(1951\)L\. LandweberAn iteration formula for Fredholm integral equations of the first kind\.American Journal of Mathematics73\(3\),pp\. 615–624\.External Links:[Document](https://dx.doi.org/10.2307/2372313)Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Leeet al\.\(2019\)J\. Lee, L\. Xiao, S\. S\. Schoenholz, Y\. Bahri, R\. Novak, J\. Sohl\-Dickstein, and J\. PenningtonWide neural networks of any depth evolve as linear models under gradient descent\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:1902\.06720Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Luoet al\.\(2021\)T\. Luo, Z\. Ma, Z\. J\. Xu, and Y\. ZhangTheory of the frequency principle for general deep neural networks\.CSIAM Transactions on Applied Mathematics2\(3\),pp\. 484–507\.External Links:[Document](https://dx.doi.org/10.4208/csiam-am.SO-2020-0005)Cited by:[Table 1](https://arxiv.org/html/2610.41179#S1.T1.2.4.1.1.1),[§1](https://arxiv.org/html/2610.41179#S1.p2.1)\.
- Luoet al\.\(2022\)T\. Luo, Z\. Ma, Z\. J\. Xu, and Y\. ZhangOn the exact computation of linear frequency principle dynamics and its generalization\.SIAM Journal on Mathematics of Data Science4\(4\),pp\. 1272–1292\.External Links:[Document](https://dx.doi.org/10.1137/21M1444400),2010\.08153,[Link](https://doi.org/10.1137/21M1444400)Cited by:[Table 1](https://arxiv.org/html/2610.41179#S1.T1.2.6.1.1.1),[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Rahamanet al\.\(2019\)N\. Rahaman, A\. Baratin, D\. Arpit, F\. Draxler, M\. Lin, F\. Hamprecht, Y\. Bengio, and A\. CourvilleOn the spectral bias of neural networks\.InProceedings of the International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 5301–5310\.Cited by:[Table 1](https://arxiv.org/html/2610.41179#S1.T1.2.3.1.1.1),[§1](https://arxiv.org/html/2610.41179#S1.p1.1),[§1](https://arxiv.org/html/2610.41179#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.41179#S2.SS3.p1.8)\.
- Xuet al\.\(2020\)Z\. J\. Xu, Y\. Zhang, T\. Luo, Y\. Xiao, and Z\. MaFrequency principle: Fourier analysis sheds light on deep neural networks\.Communications in Computational Physics28\(5\),pp\. 1746–1767\.External Links:[Document](https://dx.doi.org/10.4208/cicp.OA-2020-0085),1901\.06523Cited by:[Table 1](https://arxiv.org/html/2610.41179#S1.T1.2.2.1.1.1),[§1](https://arxiv.org/html/2610.41179#S1.p1.1),[§1](https://arxiv.org/html/2610.41179#S1.p2.1)\.
- Xuet al\.\(2025\)Z\. J\. Xu, Y\. Zhang, and T\. LuoOverview frequency principle/spectral bias in deep learning\.Communications on Applied Mathematics and Computation7\(3\),pp\. 827–864\.External Links:[Document](https://dx.doi.org/10.1007/s42967-024-00398-7)Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p1.1)\.
- Xuet al\.\(2019\)Z\. J\. Xu, Y\. Zhang, and Y\. XiaoTraining behavior of deep neural network in frequency domain\.InNeural Information Processing \(ICONIP 2019\),Lecture Notes in Computer Science, Vol\.11953,pp\. 264–274\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-36708-4%5F22),1807\.01251Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p1.1)\.
- Zhanget al\.\(2025\)S\. Zhang, H\. Zhao, Y\. Zhong, and H\. ZhouWhy shallow networks struggle to approximate and learn high frequencies\.Information and Inference: A Journal of the IMA14\(3\),pp\. iaaf022\.External Links:[Document](https://dx.doi.org/10.1093/imaiai/iaaf022),[Link](https://doi.org/10.1093/imaiai/iaaf022)Cited by:[Table 1](https://arxiv.org/html/2610.41179#S1.T1.2.7.1.1.1),[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.
- Zhanget al\.\(2019\)Y\. Zhang, Z\. J\. Xu, T\. Luo, and Z\. MaExplicitizing an implicit bias of the frequency principle in two\-layer neural networks\.Note:arXiv:1905\.10264External Links:arXiv:1905\.10264Cited by:[§1](https://arxiv.org/html/2610.41179#S1.p3.1)\.