A Trust-region Framework for Moment Estimation

# A Trust-region Framework for Moment Estimation
Source: [https://arxiv.org/html/2608.04026](https://arxiv.org/html/2608.04026)
###### Abstract

In this paper, we develop a trust\-region framework for understanding the behavior of adaptive moment estimation mechanisms, such asAdam, in stochastic gradient optimization\. Specifically, in this framework, the magnitude of the update step for each individual weight is constrained within a trust\-region governed by a moment constraint of orderp∈\[2,4\]p\\in\[2,4\]\. The resulting derivation then leads to a family of learning\-rate mechanisms based on second\-moment estimation and a normalizedpp\-th moment estimation\. Whenp=4p=4, this involves kurtosis\-like estimation\. The general mechanism, referred to asGmake, provides a unified interpretation of normalization by moment estimation, learning\-rate scheduling, spectral lowpass filtering as momentum, and operator\-level spectral normalization within a common trust\-region framework\. Experiments on GPT2\-124M trained on FineWeb\-Edu and TinyStories suggest that the fourth\-moment realization provides its greatest benefit when trust\-region constraints are weak\. As progressively stronger trust\-region controls are introduced, the second\-moment realization becomes increasingly competitive, often achieving slightly lower validation loss than its corresponding fourth\-moment realization\.

## 1Introduction

A natural question arising from the popular adaptive moment estimation \(Adam,Kingma and Ba \([2015](https://arxiv.org/html/2608.04026#bib.bib324)\)\) mechanism in stochastic gradient optimization is whether its moment\-normalized update components can be understood from a rigorous trust\-region principle\.

In this paper, we address this question by developing a moment\-constrained trust\-region control framework for stochastic gradient optimization\. The resulting derivation leads to a family of learning\-rate mechanisms governed bypp\-th moment trust\-region constraints on the update step, for2≤p≤42\\leq p\\leq 4\. In the special casep=4p=4, the mechanism involves both moment estimation, and a normalized kurtosis estimation of the gradient process\. Hence, we refer, loosely, to realizations of this trust\-region framework asGmake\.

Beyond the derivation of a family of learning\-rate mechanisms, the framework reveals that several mechanisms traditionally studied separately can be understood within a common trust\-region perspective on the update step\. In particular, common learning\-rate schedules arise naturally as solutions to a trust\-region variational problem, while momentum and spectral normalization emerge as complementary mechanisms for progressively strengthening enforcement of an underlying trust\-region constraint\. Viewed through this lens, these mechanisms can be understood as complementary forms of trust\-region control on the update process\.

### 1\.1A Trust\-region problem

Let the update stepΔ​\(t\+1\)\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}of the stochastic gradient algorithm applied to an individual weightw​\(t\)∈ℝw\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\in\{\\mathbb\{R\}\}at iterationttbe defined as

Δ​\(t\+1\)=w​\(t\+1\)−w​\(t\),\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\-w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\(1\)and letg​\(t\)∈ℝg\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\in\{\\mathbb\{R\}\}denote the associated gradient component obtained by minimizing a scalar\-valued loss functionf​\(t\)f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}with respect to aw​\(t\)w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}over a window of iterationst∈\{0,1,…,τ\}t\\in\\\{0,1,\\ldots,\\tau\\\}whereτ≫0\\tau\\gg 0\.

Throughout, we use𝔼​\{⋅\}\\mathbb\{E\}\\\{\\cdot\\\}to denote expectation with respect to the underlying stochastic process generating the gradients\. No specific objective function, probabilistic structure, or distribution of the gradients is assumed\. We make only the following assumptions for allt∈\{0,1,…,τ\}t\\in\\\{0,1,\\ldots,\\tau\\\}:

Assumption 1 \(Lipschitz Regularity\): The functionf​\(t\)f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}is at least twice continuously differentiable, and bothf​\(t\)f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}andg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}are Lipschitz continuous inw​\(t\)w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\(Bottouet al\.,[2018](https://arxiv.org/html/2608.04026#bib.bib75)\)\.

Assumption 2 \(Boundedpp\-th Moment\): The stochastic signalg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}satisfies‖g​\(t\)‖∞<∞\\\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{\\infty\}<\\infty\. Hence, for everyp∈\[2,4\]p\\in\[2,4\],𝔼​\{\|g​\(t\)\|p\}≤‖g​\(t\)‖∞p<∞\\mathbb\{E\}\\\{\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\}\\\}\\leq\\\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{\\infty\}^\{p\}<\\infty\.

Each iteration of the learning algorithm \([1](https://arxiv.org/html/2608.04026#S1.E1)\) is constructed to at least, ensure𝔼​\{\|Δ​\(t\+1\)\|\}≤μ\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert\\\}\\leq\\mu\. In other words, the expected magnitude of the update step is bounded within a maximum trust region radiusμ∈\(0,1\)\\mu\\in\(0,1\)\. The trust\-region radius of the update step is characterized, at each iteration, by itspp\-th moment

δpp​\(t\)≡𝔼​\{\|Δ​\(t\+1\)\|p\}≤μp,\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\equiv\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}\\leq\\mu^\{p\},\(2\)whereμ\\mu, the maximum allowable update step\-size, is a real constant, whileδp​\(t\)≤μ\{\\delta\_\{p\}\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu, withδp​\(t\)→0\{\\delta\_\{p\}\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0ast→τt\\to\\tauis a trust\-region radius shaping function governing variation of the update step\.

### 1\.2Overview of the Algorithm

The basic form of the learning algorithm, with aς​\(t\)∈\[0,1\]\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\in\[0,1\], andς​\(t\)→0\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0ast→τt\\to\\tauis

δp​\(t\)=μ​ς​\(t\)1p,\\displaystyle\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu\\,\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\frac\{1\}\{p\}\},\(3\)g¯​\(t\)=g​\(t\)𝔼​\{\|g​\(t\)\|2\}12,\\displaystyle\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\frac\{g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{2\}\\\}^\{\\frac\{1\}\{2\}\}\},\(4\)g¯​\(t\)←g¯​\(t\)𝔼​\{\|g¯​\(t\)\|p\}1p,\\displaystyle\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leftarrow\\frac\{\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\|\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\},\(5\)Δ​\(t\+1\)=−δp​\(t\)​g¯​\(t\)\.\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(6\)Letℍ:ℓ2→ℓ2\{\\mathbb\{H\}\}\\colon\\\!\\ell^\{2\}\\\!\\to\\\!\\ell^\{2\}be a linear, iteration\-invariant operator whose transfer functionℍ​\(z\)\{\\mathbb\{H\}\}\(z\), withz=ej​ωz=e^\{j\\omega\}, over all frequenciesω∈\[−π,π\]\\omega\\in\[\-\\pi,\\pi\]satisfiesℍ​\(1\)=1\{\\mathbb\{H\}\}\(1\)=1,\|ℍ​\(ej​ω1\)\|≥\|H​\(ej​ω2\)\|\\lvert\{\\mathbb\{H\}\}\(e^\{j\\omega\_\{1\}\}\)\\rvert\\geq\\lvert H\(e^\{j\\omega\_\{2\}\}\)\\rvertfor all0≤\|ω1\|≤\|ω2\|≤π0\\leq\\lvert\\omega\_\{1\}\\rvert\\leq\\lvert\\omega\_\{2\}\\rvert\\leq\\piand

‖ℍ‖∞=supω∈\[−π,π\]\|ℍ​\(ej​ω\)\|≤1\.\\displaystyle\\\|\{\\mathbb\{H\}\}\\\|\_\{\\infty\}=\\sup\_\{\\omega\\in\[\-\\pi,\\pi\]\}\\lvert\{\\mathbb\{H\}\}\(e^\{j\\omega\}\)\\rvert\\leq 1\.\(7\)An example of such an operator is a lowpass filter\. The first two conditions ensure strong attenuation of high\-frequency components, while the last guarantees dissipativity in energy\. Then, regularizingg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}withℍ\{\\mathbb\{H\}\}strengthens satisfaction of the moment\-based trust\-region constraint\. Since‖ℍ​\{g​\(t\)\}‖2≤‖g​\(t\)‖2\\\|\{\\mathbb\{H\}\}\\\{g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\}\\\|\_\{2\}\\leq\\\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{2\}, the regularized gradient possesses reduced moment energy\. The regularized form of the algorithm becomes

δp​\(t\)=μ​ς​\(t\)1p,\\displaystyle\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu\\,\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\frac\{1\}\{p\}\},\(8\)v​\(t\)=ℍ​\{g​\(t\)\},\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\{\\mathbb\{H\}\}\\\{g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\},\(9\)g¯​\(t\)=v​\(t\)𝔼​\{\|g​\(t\)\|2\}12,\\displaystyle\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\frac\{v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{2\}\\\}^\{\\frac\{1\}\{2\}\}\},\(10\)g¯​\(t\)←g¯​\(t\)𝔼​\{\|g¯​\(t\)\|p\}1p,\\displaystyle\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leftarrow\\frac\{\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\|\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\},\(11\)Δ​\(t\+1\)=−δp​\(t\)​g¯​\(t\)\.\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(12\)Further, letG¯​\(t\)\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}denote a full\-rankn×mn\\times mmatrix composed of normalized gradient componentsg¯​\(t\)\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}associated with a single layer or parameter groupll, where1<n≤m1<n\\leq m\. Denote the update step in matrix form asΔl​\(t\+1\)=−δp​\(t\)​G~​\(t\)\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\. Now, suppose that, in order to strengthen enforcement of the trust\-region constraint𝔼​\{\|Δ​\(t\+1\)\|\}≤μ\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert\\\}\\leq\\mu, we additionally require a similar bound on the layer’s spectral norm,∥Δl​\(t\+1\)∥2≤μ\\lVert\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rVert\_\{2\}\\leq\\mu\. The algorithm then becomes of the form

δp​\(t\)=μ​ς​\(t\)1p,\\displaystyle\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu\\,\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\frac\{1\}\{p\}\},\(13\)v​\(t\)=ℍ​\{g​\(t\)\},\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\{\\mathbb\{H\}\}\\\{g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\},\(14\)g¯​\(t\)=v​\(t\)𝔼​\{\|g​\(t\)\|2\}12,\\displaystyle\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\frac\{v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{2\}\\\}^\{\\frac\{1\}\{2\}\}\},\(15\)g¯​\(t\)←g¯​\(t\)𝔼​\{\|g¯​\(t\)\|p\}1p,\\displaystyle\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leftarrow\\frac\{\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\|\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\},\(16\)G~​\(t\)=\(G¯​\(t\)​G¯​\(t\)⊺\)−12​G¯​\(t\),\\displaystyle\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\bigl\(\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\intercal\}\\bigr\)^\{\-\\frac\{1\}\{2\}\}\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\(17\)Δl​\(t\+1\)=−δp​\(t\)​G~​\(t\)\.\\displaystyle\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(18\)
The principal contributions of this paper are:

1. \(i\.\)App\-moment trust\-region formulation of the stochastic gradient update step, leading to a generalized family of normalization mechanisms for2≤p≤42\\leq p\\leq 4\.
2. \(ii\.\)A unified interpretation of adaptive moment estimation, learning\-rate scheduling, spectral lowpass filtering as momentum, and spectral normalization as complementary trust\-region mechanisms acting on different properties of the update process\.
3. \(iii\.\)A trust\-region\-preserving spectral lowpass regularization form of the update step\.
4. \(iv\.\)A matrix\-operator formulation that additionally enforces an operator\-level spectral trust\-region constraint while preserving the underlying moment\-based trust\-region guarantees\.

### 1\.3Related Work

TheGmakemechanism draws connections between several research directions in stochastic gradient optimization, including classic trust\-region optimization, adaptive moment estimation, momentum, and matrix orthogonalization\.

#### Adaptive moment estimation\.

Adaptive learning\-rate algorithms such asRMSPropandAdamnormalize gradient components via second\-moment estimates and has become the most popular optimizer in deep learning\(Kingma and Ba,[2015](https://arxiv.org/html/2608.04026#bib.bib324); Bottouet al\.,[2018](https://arxiv.org/html/2608.04026#bib.bib75)\)\. In contrast,Gmakeis derived from explicit moment\-constrained trust\-region considerations\. The resulting learning rule extends second\-moment normalization to a family of p\-moment normalization mechanisms forp∈\[2,4\]p\\in\[2,4\]\.

#### Trust\-region, normalized\-gradient methods, and learning\-rate schedules\.

Trust\-region methods regulate update magnitudes by constraining the optimization step to lie within a prescribed neighborhood in which a local model is considered reliable\(Parikh and Boyd,[2014](https://arxiv.org/html/2608.04026#bib.bib490); Connet al\.,[2000](https://arxiv.org/html/2608.04026#bib.bib115)\)\. Related ideas also appear in normalized\-gradient methods, where update directions are rescaled to control step magnitudes\. Learning\-rate schedules\(Bergsmaet al\.,[2024](https://arxiv.org/html/2608.04026#bib.bib53); Geet al\.,[2018](https://arxiv.org/html/2608.04026#bib.bib212)\)are also widely used to progressively reduce update magnitudes throughout training, thereby influencing the effective size of the optimization region explored by the algorithm\. The formulation in this paper differs in that the trust region is characterized through moment constraints on the update itself, yielding a learning\-rate mechanism directly linked to the statistical properties of the underlying gradient process\. Furthermore, common learning\-rate schedules emerge naturally in this framework as solutions to a trust\-region variational control problem rather than as externally specified heuristics\.

#### Momentum and filtering\.

Momentum methods such as Heavy\-Ball and Nesterov acceleration have long been used to improve the stochastic gradient optimization process\(Polyak,[2020](https://arxiv.org/html/2608.04026#bib.bib502); Sutskeveret al\.,[2013](https://arxiv.org/html/2608.04026#bib.bib634); Polyak,[1969](https://arxiv.org/html/2608.04026#bib.bib504)\)\. More recently, momentum mechanisms have also been interpreted from a lowpass signal\-processing perspective\(Somefun,[2026](https://arxiv.org/html/2608.04026#bib.bib618); Somefunet al\.,[2024](https://arxiv.org/html/2608.04026#bib.bib616)\), where they introduce a smoothing effect on gradient sequences and attenuate rapidly varying gradient components\(Liet al\.,[2025](https://arxiv.org/html/2608.04026#bib.bib379)\)\. This viewpoint provides useful intuition for understanding the noise\-reduction properties of momentum\-based optimization methods\.Gmakeincorporates a dissipative linear iteration\-invariant operator that preferentially attenuates high\-frequency gradient fluctuations\. Unlike previous filtering interpretations, the operator is introduced here as a means of strengthening satisfaction of an existing moment\-based trust\-region constraint\.

#### Matrix\-aware and spectral\-norm\-constrained updates\.

Recent optimizer developments have emphasized the matrix structure of neural\-network parameters, leading to orthogonalized and spectral\-normalized update rules\(Bernstein and Newhouse,[2024](https://arxiv.org/html/2608.04026#bib.bib56)\)\. Many such methods admit interpretations in terms of alternative matrix norms or operator\-norm constraints\(Largeet al\.,[2024](https://arxiv.org/html/2608.04026#bib.bib352)\)\. In contrast, spectral constraints are introduced here as a secondary mechanism for progressively tightening an existing moment\-based trust\-region framework rather than as the primary update rule\. The matrix\-operator form ofGmakeis related to this literature in that it applies spectral normalization to a matrix\-valued layer or parameter group composed ofpp\-moment\-normalized gradient components\.

### 1\.4Position of this Work

Gmakeunifies moment\-constrained trust\-region optimization, learning\-rate scheduling, moment estimation, lowpass spectral regularization, and spectral\-norm trust\-region control within a common framework\. The central idea is to characterize update magnitudes through explicitpp\-moment trust\-region constraints, leading to a family of learning\-rate mechanisms for2≤p≤42\\leq p\\leq 4governed by moment statistics\. In the special casep=4p=4, this yields a learning\-rate rule involving a normalized kurtosis estimator\.

The resulting trust\-region framework is further strengthened through two complementary mechanisms\. First, a linear iteration\-invariant operator provides lowpass spectral regularization while preserving the underlyingpp\-moment trust\-region guarantees\. Second, a matrix\-form of the algorithm enforces a spectral trust\-region constraint on a matrix\-shaped layer or parameter group composed ofpp\-moment\-normalized gradient components, thereby further strengthening the underlying maximum step\-size trust\-region constraint\. Together, these mechanisms progressively tighten enforcement of a common trust\-region radius\.

### 1\.5Organization of the Paper

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2608.04026#S2)derives the basic form ofGmake, establishes its moment\-constrained properties, and connection to learning\-rate schedules\. Section[3](https://arxiv.org/html/2608.04026#S3)introduces a trust\-region\-preserving spectral regularization based on a dissipative lowpass operator\. Then, Section[4](https://arxiv.org/html/2608.04026#S4)develops a matrix\-operator form that additionally enforces spectral trust\-region constraint\. Section[5](https://arxiv.org/html/2608.04026#S5)notes practical implementation strategies for the required statistical estimators, and Section[6](https://arxiv.org/html/2608.04026#S6)presents numerical experiments and discussions comparing theGmakeforms\. Limitations and gaps of this work are discussed in Section[7](https://arxiv.org/html/2608.04026#S7)\. Finally, the paper is concluded in Section[8](https://arxiv.org/html/2608.04026#S8)\.

## 2Gmake

The basic form of the stochastic gradient update in \([1](https://arxiv.org/html/2608.04026#S1.E1)\) can be defined as

Δ​\(t\+1\)=w​\(t\+1\)−w​\(t\)=−α​\(t\)​g​\(t\),\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\-w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\-\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\(19\)whereα​\(t\)\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}is a non\-negative real\-valued learning rate function of iterationtt\. Our goal is to design a learning\-rate mechanismα​\(t\)\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}that ensures theΔ​\(t\+1\)\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}satisfies the trust\-region problem involving itspp\-th moment𝔼​\{\|Δ​\(t\+1\)\|p\}≤μp\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}\\leq\\mu^\{p\}, so that𝔼​\{\|Δ​\(t\+1\)\|\}≤μ\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert\\\}\\leq\\mu\.

### 2\.1Learning\-rate mechanism

Recall thatδpp​\(t\)=𝔼​\{\|Δ​\(t\+1\)\|p\}\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}\. Substituting the update step into the trust\-region constraint yields

𝔼​\{\|Δ​\(t\+1\)\|p\}=𝔼​\{\|−α​\(t\)​g​\(t\)\|p\}=αp​\(t\)​𝔼​\{\|g​\(t\)\|p\}=δpp​\(t\)\.\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}=\\mathbb\{E\}\\\{\\lvert\-\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}=\\alpha^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}=\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(20\)Solving forα​\(t\)\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}gives the learning\-rate mechanism

α​\(t\)=δp​\(t\)𝔼​\{\|g​\(t\)\|p\}1p\.\\displaystyle\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\frac\{\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\}\.\(21\)Note that ifg¯​\(t\)=g​\(t\)/𝔼​\{\|g​\(t\)\|2\}12\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}/\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{2\}\\\}^\{\\frac\{1\}\{2\}\}, then it follows that𝔼​\{\|g​\(t\)\|p\}1p=𝔼​\{\|g¯​\(t\)\|p\}1p​𝔼​\{\|g​\(t\)\|2\}12\{\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\}=\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\\,\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{2\}\\\}^\{\\frac\{1\}\{2\}\}\. The learning rate \([21](https://arxiv.org/html/2608.04026#S2.E21)\) can be equivalently expressed as

α​\(t\)=δp​\(t\)𝔼​\{\|g¯​\(t\)\|p\}1p​𝔼​\{\|g​\(t\)\|2\}12,\\displaystyle\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\frac\{\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\\,\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{2\}\\\}^\{\\frac\{1\}\{2\}\}\},\(22\)and the update step \([20](https://arxiv.org/html/2608.04026#S2.E20)\) becomes

Δ​\(t\+1\)=−δp​\(t\)​g¯​\(t\)𝔼​\{\|g¯​\(t\)\|p\}1p\.\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\frac\{\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\}\.\(23\)In particular, forp=4p=4, the learning rate function has a meaningful interpretation of being composed of terms related to the second\-moment and normalized kurtosis ofg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\. The update step is equivalently

Δ​\(t\+1\)=−δ4​\(t\)​g¯​\(t\)𝔼​\{\|g¯​\(t\)\|4\}14\.\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{4\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\frac\{\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{4\}\\\}^\{\\frac\{1\}\{4\}\}\}\.\(24\)
Next, initially denoteg~​\(t\)=g¯​\(t\)/𝔼​\{\|g¯​\(t\)\|p\}1p\\tilde\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\{\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}/\{\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\}, then the update step \([22](https://arxiv.org/html/2608.04026#S2.E22)\) can be expressed asΔ​\(t\+1\)=−δp​\(t\)​g~​\(t\)\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\tilde\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, which enforces𝔼​\{\|g~​\(t\)\|p\}=1\\mathbb\{E\}\\\{\\lvert\\tilde\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}=1, and therefore𝔼​\{\|Δ​\(t\+1\)\|p\}≤μp\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}\\leq\\mu^\{p\}, sinceδpp​\(t\)≤μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu^\{p\}\. As a result, for1<p1<p, Lyapunov’s inequality implies

𝔼​\{\|Δ​\(t\+1\)\|\}≤𝔼​\{\|Δ​\(t\+1\)\|p\}1p≤μ\.\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert\\\}\\leq\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\\leq\\mu\.\(25\)
The update stepΔ​\(t\+1\)=−δp​\(t\)​g~​\(t\)\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\tilde\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}then directly satisfies both thepp\-th moment and expected maximum magnitude trust\-region constraints\. The resulting update algorithm sequentially follows the outlined equations \([3](https://arxiv.org/html/2608.04026#S1.E3)\)–\([6](https://arxiv.org/html/2608.04026#S1.E6)\)\.

In the vectorized case, consider a parameter group containingn′n^\{\\prime\}parameters, and letΔv​\(t\+1\)\\Delta^\{v\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}denote the corresponding update vector\. Since thepp\-moment normalization acts independently on each parameter, the trust\-region constraint \([25](https://arxiv.org/html/2608.04026#S2.E25)\) implies that, for any1≤r≤p1\\leq r\\leq p,

𝔼​‖Δv​\(t\+1\)‖rr\\displaystyle\\mathbb\{E\}\\\|\\Delta^\{v\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{r\}^\{r\}=∑i=1n′𝔼​\{\|Δiv​\(t\+1\)\|r\}≤∑i=1n′μr=n′​μr\.\\displaystyle=\\sum\_\{i=1\}^\{n^\{\\prime\}\}\\mathbb\{E\}\\\!\\left\\\{\\left\|\\Delta^\{v\}\_\{i\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\right\|^\{r\}\\right\\\}\\leq\\sum\_\{i=1\}^\{n^\{\\prime\}\}\\mu^\{r\}=n^\{\\prime\}\\mu^\{r\}\.\(26\)In particular,

𝔼​‖Δv​\(t\+1\)‖pp≤n′​μp\.\\displaystyle\\mathbb\{E\}\\\|\\Delta^\{v\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{p\}^\{p\}\\leq n^\{\\prime\}\\mu^\{p\}\.\(27\)Since0<μ<10<\\mu<1andr≥1r\\geq 1, it also follows thatμr≤μ\\mu^\{r\}\\leq\\mu, yielding a looser but sometimes convenient bound

𝔼​‖Δv​\(t\+1\)‖rr≤n′​μ\.\\displaystyle\\mathbb\{E\}\\\|\\Delta^\{v\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{r\}^\{r\}\\leq n^\{\\prime\}\\mu\.\(28\)Therefore, thepp\-th moment\-based trust\-region constraint directly controls the expected magnitude of the vectorized update in everyℓr\\ell\_\{r\}norm with1≤r≤p1\\leq r\\leq p, and in particular implies𝔼​‖Δv​\(t\+1\)‖pp≤n′​μp\\mathbb\{E\}\\\|\\Delta^\{v\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{p\}^\{p\}\\leq n^\{\\prime\}\\mu^\{p\}\.

Note that𝔼​\{\|g¯​\(t\)\|p\}1p≥1\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}\\geq 1for allp≥2p\\geq 2\. For convenience, we can re\-write the underlying learning\-rate mechanism \([22](https://arxiv.org/html/2608.04026#S2.E22)\) as

α​\(t\)=δp​\(t\)κp​\(t\)​λ2​\(t\)≤μλ2​\(t\),\\displaystyle\\alpha\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\frac\{\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\\leq\\frac\{\\mu\}\{\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\},\(29\)whereκp​\(t\)=𝔼​\{\|g¯​\(t\)\|p\}1p\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}, andλ2​\(t\)=𝔼​\{\|g​\(t\)\|2\}12\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{2\}\\\}^\{\\frac\{1\}\{2\}\}\. The overall algorithm in \([3](https://arxiv.org/html/2608.04026#S1.E3)\)–\([6](https://arxiv.org/html/2608.04026#S1.E6)\) can be interpreted as first usingλ2​\(t\)\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}to normalize the variance scale ofg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, and then applyingκp​\(t\)\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}to yield the fully normalizedg¯​\(t\)\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, whosepp\-th moment satisfies the trust\-region constraint\.

In the special case,p=2p=2, ideallyκ2​\(t\)≊1\\kappa\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\approxeq 1, then \([29](https://arxiv.org/html/2608.04026#S2.E29)\) reduces to the same mechanism used inRMSPropandAdam, whereg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}is normalized solely byλ2​\(t\)\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\. Consequently, forp=2p=2, this algorithm applies an additional normalization byκ2​\(t\)\\kappa\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}to compensate for scaling errors that remain after theλ2​\(t\)\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}normalization\. As a result, this approach can be interpreted as a principled refinement of the normalization mechanism employed byRMSPropandAdam\.

Forp≥2p\\geq 2, when the normalization factorκp​\(t\)\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}remains close to its nominal minimum value of one, the resulting learning\-rate mechanism becomes increasingly similar to thep=2p=2case\. Consequently, the benefits of higher\-order moment control may be limited on low\-variance or statistically well\-behaved datasets or gradient processes\. Conversely, when higher\-order gradient fluctuations become more pronounced andκp​\(t\)\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}departs further from one, the additional normalization acts as a stronger trust\-region control mechanism\.

### 2\.2Learning\-rate Schedules as Trust\-region shaping functions

The trust\-region problem requires thatδpp​\(t\)≤μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu^\{p\}for alltt, and thatδpp​\(t\)→0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0ast→τt\\to\\tau\. This can be interpreted as progressively tightening the trust\-region constraints over the learning window to enforce asymptotic stability of the update step\.

In order to obtain the smoothest possible variation of the trust\-region radius across iterations over the learning window or horizont∈\{0,1,…,τ\}t\\in\\\{0,1,\\ldots,\\tau\\\}, we can define a total variational energy functional minimization problem directly onδpp​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},

min0≤δpp​\(t\)≤μp⁡𝒟​\(δp,τ\)=∑t=1τ\[δpp​\(t\)−δpp​\(t−1\)\]2,\\displaystyle\\min\_\{0\\leq\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.3014pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.3014pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.3014pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{2\.71246pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu^\{p\}\}\\\>\{\\mathcal\{D\}\}\(\\delta\_\{p\},\\tau\)=\\sum\_\{t=1\}^\{\\tau\}\\bigl\[\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\\bigr\]^\{2\},\(30\)subject to fixed boundary values: the terminal boundary valueδpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0, together with at least one specified initial boundary value satisfyingδpp​\(0\)≤μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}\\leq\\mu^\{p\}\. The energy minimization problem \([30](https://arxiv.org/html/2608.04026#S2.E30)\) represents the canonical first\-order smoothness functional that penalize rapid variations inδpp​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, the trust\-region radius between successive iterations\. Equivalently, \([30](https://arxiv.org/html/2608.04026#S2.E30)\) encouragesδpp​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}to remain as close as possible toδpp​\(t−1\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}throughout learning\. Consequently,δp​\(t\)\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}varies no faster than necessary while remaining consistent with the asymptotic stability requirement, thereby producing the smoothest trust\-region variation consistent with the boundary values\.

Furthermore, \([30](https://arxiv.org/html/2608.04026#S2.E30)\) is strictly convex inδpp​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, and therefore has a unique minimizer\. Differentiating with respect to any interior indexi∈\{1,…,τ−1\}i\\in\\\{1,\\ldots,\\tau\-1\\\}, and settingd​𝒟/d​δpp​\(i\)=0\{d\{\\mathcal\{D\}\}\}/\{d\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\)$\}\}\}\}=0results in the second\-order homogeneous linear difference equation with constant coefficients

δpp​\(i\+1\)−2​δpp​\(i\)\+δpp​\(i−1\)=0\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\+1\)$\}\}\}\-2\\,\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\)$\}\}\}\+\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\-1\)$\}\}\}=0\.\(31\)LetΔ​δpp​\(i\+1\)=δpp​\(i\+1\)−δpp​\(i\)\\Delta\\,\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\+1\)$\}\}\}=\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\+1\)$\}\}\}\-\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\)$\}\}\}, then \([31](https://arxiv.org/html/2608.04026#S2.E31)\) corresponds to the second\-order finite difference operatorΔ​\(Δ​δpp​\(i\+1\)\)=Δ2​δpp​\(i\+1\)=0\\Delta\\bigl\(\\Delta\\,\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\+1\)$\}\}\}\\bigr\)=\\Delta^\{2\}\\,\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(i\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(i\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(i\+1\)$\}\}\}=0\. The corresponding characteristic equation isz2−2​z\+1=\(z−1\)2=0z^\{2\}\-2\\,z\+1=\(z\-1\)^\{2\}=0which has11as a root with multiplicity 2\. Therefore, the general solution to \([31](https://arxiv.org/html/2608.04026#S2.E31)\) must be a polynomial intt, of at most degree11\(Elaydi,[2005](https://arxiv.org/html/2608.04026#bib.bib170); Lueker,[1980](https://arxiv.org/html/2608.04026#bib.bib400)\),

δpp​\(t\)=a0\+a1​t\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=a\_\{0\}\+a\_\{1\}\\,t\.\(32\)The complete solution can then be obtained by plugging in at leastb≥2b\\geq 2fixed boundary values: the fixed terminal value, and at least one fixed initial value, leading tob−1b\-1sub\-interval\(s\) of the learning window\.

### 2\.32\-point boundary value

Consider the initial and terminal boundary values:δpp​\(0\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=\\mu^\{p\},δpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0over the single sub\-interval0≤t≤τ0\\leq t\\leq\\tauSolving \([32](https://arxiv.org/html/2608.04026#S2.E32)\), subject toδpp​\(0\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=\\mu^\{p\},δpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0givesa0=μpa\_\{0\}=\\mu^\{p\}, anda1=−μp/τa\_\{1\}=\-\\mu^\{p\}/\\tau\. Therefore, the complete solution to \([31](https://arxiv.org/html/2608.04026#S2.E31)\) subject to the 2\-point boundary value is

δpp​\(t\)=μp​\(1−tτ\),0≤t≤τ,\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu^\{p\}\\,\\Bigl\(1\-\\frac\{t\}\{\\tau\}\\Bigr\),\\quad 0\\leq t\\leq\\tau,\(33\)which is known as the linear decay schedule\. We can simplify this to a normalized functional form\. Letu=t/τu=t/\\tau, define an input functionx​\(u\)≔u∈\[0,1\]x\(u\)\\coloneq u\\in\[0,1\], then

δpp​\(t\)=μp​\(1−x​\(u\)\),0≤u≤1\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu^\{p\}\\,\\bigl\(1\-x\(u\)\\bigr\),\\quad 0\\leq u\\leq 1\.\(34\)

### 2\.43\-point boundary value

Consider the three fixed boundary values:δpp​\(0\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=0,δpp​\(k\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k\)$\}\}\}=\\mu^\{p\},δpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0, where0≤k<τ0\\leq k<\\tau\. This corresponds to two sub\-intervals\. On the first interval over0≤t≤k0\\leq t\\leq k, the solution is obtained by solving \([32](https://arxiv.org/html/2608.04026#S2.E32)\) subject toδpp​\(0\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=0,δpp​\(k\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k\)$\}\}\}=\\mu^\{p\}\. On the second interval overk≤t≤τk\\leq t\\leq\\tau, the solution is obtained by solving \([32](https://arxiv.org/html/2608.04026#S2.E32)\) subject toδpp​\(k\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k\)$\}\}\}=\\mu^\{p\},δpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0\. Combining the piecewise solutions, the complete solution to \([31](https://arxiv.org/html/2608.04026#S2.E31)\) subject to the 3\-point boundary value is

δpp​\(t\)=\{μp​tk,0≤t≤k,k\>0μp​τ−tτ−kk≤t≤τ,k≥0\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\begin\{cases\}\\mu^\{p\}\\,\\frac\{t\}\{k\},&0\\leq t\\leq k,\\quad k\>0\\\\ \\mu^\{p\}\\,\\frac\{\\tau\-t\}\{\\tau\-k\}&k\\leq t\\leq\\tau,\\quad k\\geq 0\.\\end\{cases\}\(35\)Simplifying further, defineu=t/τ∈\[0,1\]u=t/\\tau\\in\[0,1\], a normalized rise\-timem=k/τ∈\[0,1\)m=k/\\tau\\in\[0,1\), and the reparameterized input function

x​\(u;m\)=\{u,m=0,max⁡\{m−um,u−m1−m\},0<m<1,\\displaystyle x\(u;m\)=\\begin\{cases\}u,&m=0,\\\\ \\max\\bigl\\\{\\frac\{m\-u\}\{m\},\\,\\frac\{u\-m\}\{1\-m\}\\bigr\\\},&0<m<1,\\end\{cases\}\(36\)then equivalently, \([35](https://arxiv.org/html/2608.04026#S2.E35)\) is

δpp​\(t\)=μp​\(1−x​\(u;m\)\),0≤u≤1,0≤m<1\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu^\{p\}\\,\\bigl\(1\-x\(u;m\)\\bigr\),\\quad 0\\leq u\\leq 1,\\quad 0\\leq m<1\.\(37\)For its first sub\-interval, the hat\-shaped \(or triangular\-like\) mapping \([35](https://arxiv.org/html/2608.04026#S2.E35)\) corresponds to a warmup schedule that reachesμp\\mu^\{p\}at timemm, and then a decaying schedule to zero in the second sub\-interval\. Importantly, observe that both \([34](https://arxiv.org/html/2608.04026#S2.E34)\) and \([37](https://arxiv.org/html/2608.04026#S2.E37)\) share the same underlying functional form\. Whenm=0m=0, it follows thatx​\(u;m\)=x​\(u;0\)=ux\(u;m\)=x\(u;0\)=u, therefore \([34](https://arxiv.org/html/2608.04026#S2.E34)\) is a special case of \([37](https://arxiv.org/html/2608.04026#S2.E37)\) whenm=0m=0\.

### 2\.54\-point boundary value

Consider the four fixed boundary values:δpp​\(0\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=0,δpp​\(k\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k\)$\}\}\}=\\mu^\{p\},δpp​\(k′\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k^\{\\prime\}\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k^\{\\prime\}\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k^\{\\prime\}\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k^\{\\prime\}\)$\}\}\}=\\mu^\{p\},δpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0, where0≤k≤k′<τ0\\leq k\\leq k^\{\\prime\}<\\tau\. This corresponds to three sub\-intervals\. On the first interval over0≤t≤k0\\leq t\\leq k, the solution is obtained by solving \([32](https://arxiv.org/html/2608.04026#S2.E32)\) subject toδpp​\(0\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=0,δpp​\(k\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k\)$\}\}\}=\\mu^\{p\}\. On the second interval overk≤t≤k′k\\leq t\\leq k^\{\\prime\}, the solution is obtained by solving \([32](https://arxiv.org/html/2608.04026#S2.E32)\) subject toδpp​\(k\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k\)$\}\}\}=\\mu^\{p\},δpp​\(k′\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k^\{\\prime\}\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k^\{\\prime\}\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k^\{\\prime\}\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k^\{\\prime\}\)$\}\}\}=\\mu^\{p\}\. Finally, for the third interval overk′≤t≤τk^\{\\prime\}\\leq t\\leq\\tau, the solution is obtained by solving \([32](https://arxiv.org/html/2608.04026#S2.E32)\) subject toδpp​\(k′\)=μp\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k^\{\\prime\}\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(k^\{\\prime\}\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(k^\{\\prime\}\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(k^\{\\prime\}\)$\}\}\}=\\mu^\{p\},δpp​\(τ\)=0\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(\\tau\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(\\tau\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(\\tau\)$\}\}\}=0\. Combining the three piecewise solutions, the complete solution to \([31](https://arxiv.org/html/2608.04026#S2.E31)\) subject to the 4\-point boundary value, when simplified is

δpp​\(t\)=\{μp​tk,0≤t≤k,k\>0μpk≤t≤k′,k,k′≥0,μp​τ−tτ−k′k′≤t≤τ,k′≥0\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\begin\{cases\}\\mu^\{p\}\\,\\frac\{t\}\{k\},&0\\leq t\\leq k,\\quad k\>0\\\\ \\mu^\{p\}\\,&k\\leq t\\leq k^\{\\prime\},\\quad k,\\,k^\{\\prime\}\\geq 0,\\\\ \\mu^\{p\}\\,\\frac\{\\tau\-t\}\{\\tau\-k^\{\\prime\}\}&k^\{\\prime\}\\leq t\\leq\\tau,\\quad k^\{\\prime\}\\geq 0\.\\end\{cases\}\(38\)Defineu=t/τ∈\[0,1\]u=t/\\tau\\in\[0,1\], and letm=k/τ∈\[0,1\)m=k/\\tau\\in\[0,1\)indicate the rise time and subsequently, the start of a plateau of widthε=\(k′−k\)/τ∈\[0,1\)\\varepsilon=\(k^\{\\prime\}\-k\)/\\tau\\in\[0,1\), so thatk=m​τk=m\\tau, andk′=\(m\+ε\)​τk^\{\\prime\}=\(m\+\\varepsilon\)\\tau\. Then the reparameterized input functionx​\(r;m,ε\)∈\[0,1\]x\(r;m,\\varepsilon\)\\in\[0,1\]for \([38](https://arxiv.org/html/2608.04026#S2.E38)\) is

x​\(u;m\)≔\{max⁡\{0,u−ε1−ε\},m=0,max⁡\{m−um,0,u−\(m\+ε\)1−\(m\+ε\)\},0<m<1,\\displaystyle x\(u;m\)\\coloneq\\begin\{cases\}\\max\\bigl\\\{0,\\tfrac\{u\-\\varepsilon\}\{1\-\\varepsilon\}\\bigr\\\},&m=0,\\\\ \\max\\bigl\\\{\\tfrac\{m\-u\}\{m\},\\,0,\\,\\tfrac\{u\-\(m\+\\varepsilon\)\}\{1\-\(m\+\\varepsilon\)\}\\bigr\\\},&0<m<1,\\end\{cases\}\(39\)and equivalently \([38](https://arxiv.org/html/2608.04026#S2.E38)\) can be expressed as

δpp​\(t\)=μp​\(1−x​\(u;m,ε\)\),0≤u≤1,0≤m<1\.\\displaystyle\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu^\{p\}\\,\\bigl\(1\-x\(u;m,\\varepsilon\)\\bigr\),\\quad 0\\leq u\\leq 1,\\quad 0\\leq m<1\.\(40\)The resulting function \([38](https://arxiv.org/html/2608.04026#S2.E38)\) then corresponds in the first sub\-interval to a warmup schedule that rises toμp\\mu^\{p\}, and then a plateau in the second sub\-interval, followed by a decaying schedule to zero in the third sub\-interval\. This leads a trapezoidal shape, and exactly a warmup\-stable\-decay schedule\.

Form=0,ε=0m=0,\\varepsilon=0, it follows thatx​\(u;m,ε\)=x​\(u;0,0\)=ux\(u;m,\\varepsilon\)=x\(u;0,0\)=u, therefore \([34](https://arxiv.org/html/2608.04026#S2.E34)\) is a special case of \([40](https://arxiv.org/html/2608.04026#S2.E40)\)\. Similarly, whenm≠0,ε=0m\\neq 0,\\varepsilon=0, it follows thatx​\(u;m,ε\)=x​\(u;m,0\)=x​\(u;m\)x\(u;m,\\varepsilon\)=x\(u;m,0\)=x\(u;m\), therefore \([37](https://arxiv.org/html/2608.04026#S2.E37)\) is a special case of \([40](https://arxiv.org/html/2608.04026#S2.E40)\)\. Importantly, both \([34](https://arxiv.org/html/2608.04026#S2.E34)\), \([37](https://arxiv.org/html/2608.04026#S2.E37)\) and \([40](https://arxiv.org/html/2608.04026#S2.E40)\) share the same underlying functional form, with \([40](https://arxiv.org/html/2608.04026#S2.E40)\) having a more general input reparameterization function\.

Finally, denoteδpp​\(t\)=μp​ς​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu^\{p\}\\,\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, whereς​\(t\)=1−x​\(u;m,ε\)\\varsigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=1\-x\(u;m,\\varepsilon\), then

δp​\(t\)=μ​ς1p​\(t\)\\displaystyle\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu\\,\\varsigma^\{\\frac\{1\}\{p\}\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\(41\)is the unique minimizer of the first\-order total variational energy functional, which satisfiesδp​\(t\)≤μ\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu,δp​\(t\)→0\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0ast→τt\\to\\tausubject to2≤b≤42\\leq b\\leq 4fixed boundary points\.

![Refer to caption](https://arxiv.org/html/2608.04026v1/wins_lins_1.png)

![Refer to caption](https://arxiv.org/html/2608.04026v1/wins_lins_1_v2.png)

Figure 1:Different reparameterizations of the boundary\-value problem onδpp​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, withμ=1\\mu=1lead to the shapes of common learning\-rate schedules, sharing the same linear mapping1−x​\(u;m,ε\)1\-x\(u;m,\\varepsilon\)\.Remarks:The preceding analysis shows that many commonly used learning\-rate schedules arise naturally as solutions to a trust\-region variational problem\. By treatingδpp​\(t\)\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}as a varying trust\-region radius and minimizing its total variation subject to fixed boundary values, we obtain schedules that evolve as smoothly as possible, while still satisfying the asymptotic requirementδ​\(t\)→0\\delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0ast→τt\\to\\tau\.

The resultingpp\-th root linear decay, warmup\-decay, and warmup\-stable\-decay schedules are illustrated in Figure[1](https://arxiv.org/html/2608.04026#S2.F1)\. They correspond respectively to 2\-point, 3\-point, and 4\-point boundary\-value problems, yet all share a common variational structure\. Consequently, learning\-rate schedules may be interpreted as vanishing trust\-region functions that progressively tighten the admissible trust\-region of the update step over the learning horizon\. Under this interpretation, schedule design is no longer an independent heuristic, but rather a principled consequence of solving a first\-order variational optimization problem subject to trust\-region boundary constraints\. More generally, alternative variational functionals would produce different classes of schedules\(Somefun,[2026](https://arxiv.org/html/2608.04026#bib.bib618)\), each reflecting the structure of the underlying variational principle\. Sinceδp​\(t\)\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}directly controls the allowable update magnitude, learning\-rate scheduling serves as a complementary mechanism through which the trust\-region constraints are gradually tightened over the learning horizon\.

### 2\.6Underlying Taylor\-series Model

The role of thepp\-moment trust\-region constraint can also be viewed from the perspective of a Taylor\-series local approximation model\. Let the update step \([20](https://arxiv.org/html/2608.04026#S2.E20)\) generated by \([22](https://arxiv.org/html/2608.04026#S2.E22)\) satisfy thepp\-moment trust\-region constraint

𝔼​\{\|Δ​\(t\+1\)\|p\}=δpp​\(t\)≤μp,\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}=\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu^\{p\},\(42\)whereδp​\(t\)→0\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0ast→τt\\to\\tau\. Forp\>r\>0p\>r\>0, Lyapunov’s inequality implies

𝔼​\{\|Δ​\(t\+1\)\|r\}1r≤𝔼​\{\|Δ​\(t\+1\)\|p\}1p=δp​\(t\)\.\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{r\}\\\}^\{\\frac\{1\}\{r\}\}\\leq\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}^\{\\frac\{1\}\{p\}\}=\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(43\)Consequently, all lower moments up to orderppare simultaneously controlled,

𝔼​\{\|Δ​\(t\+1\)\|r\}≤δpr​\(t\),r≤p\.\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{r\}\\\}\\leq\\delta\_\{p\}^\{r\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\qquad r\\leq p\.\(44\)
In particular, sinceΔ​\(t\+1\)=−δp​\(t\)κp​\(t\)​g¯​\(t\)\\displaystyle\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\frac\{\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\\,\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},𝔼​\{\|g¯​\(t\)\|2\}=1\\mathbb\{E\}\\\{\\lvert\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{2\}\\\}=1andκp​\(t\)≥1\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\geq 1, it follows that

𝔼​\{\|Δ​\(t\+1\)\|2\}=δp2​\(t\)κp2​\(t\)≤δp2​\(t\)\.\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{2\}\\\}=\\frac\{\\delta\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\kappa\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\\leq\\delta\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(45\)Therefore, app\-moment trust\-region constraint would control every moment entering a Taylor\-series expansion up to degreepp\. Consider the updatew​\(t\+1\)=w​\(t\)\+Δ​\(t\+1\)w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\. Iff​\(t\)f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}has a Lipschitz continuousg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}with constantc2\>0c\_\{2\}\>0, then by Taylor’s theorem, the second\-order Taylor model associated with the expected loss decrease with respect tow​\(t\)w\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}is

𝔼​\{Δ​f​\(t\+1\)\}=𝔼​\{g​\(t\)​Δ​\(t\+1\)\}\+12​𝔼​\{h​\(t\)​Δ2​\(t\+1\)\}\+O​\(𝔼​\{\|Δ​\(t\+1\)\|3\}\),\\displaystyle\\mathbb\{E\}\\\{\\Delta f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\}=\\mathbb\{E\}\\\{g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\}\+\\frac\{1\}\{2\}\\,\\mathbb\{E\}\\\{h\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\Delta^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\}\+O\\\!\\left\(\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{3\}\\\}\\right\),\(46\)whereΔ​f​\(t\+1\)=f​\(t\+1\)−f​\(t\)\\Delta f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\-f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}and\|h​\(t\)\|≤c2\\lvert h\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert\\leq c\_\{2\}, and \([46](https://arxiv.org/html/2608.04026#S2.E46)\) follows directly from Taylor’s theorem with remainder\. Since Assumption 1 implies that the second derivative is bounded, there exists a constantC\>0C\>0such that the third\-order remainder satisfies\|R​\(3\)\|≤C​\|Δ​\(t\+1\)\|3\|R\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(3\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(3\)$\}\}\}\|\\leq C\|\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\|^\{3\}\. Consequently,\|E​\{R​\(3\)\}\|≤E​\{\|R​\(3\)\|\}≤C​E​\{\|Δ​\(t\+1\)\|3\}\|E\\\{R\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(3\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(3\)$\}\}\}\\\}\|\\leq E\\\{\|R\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(3\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(3\)$\}\}\}\|\\\}\\leq C\\,E\\\{\|\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\|^\{3\}\\\}, and thereforeE​\{R​\(3\)\}=O​\(E​\{\|Δ​\(t\+1\)\|3\}\)E\\\{R\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(3\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(3\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(3\)$\}\}\}\\\}=O\(E\\\{\|\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\|^\{3\}\\\}\)\. Applying the previously established moment bounds onΔ​\(t\+1\)\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}, by using \([44](https://arxiv.org/html/2608.04026#S2.E44)\) and \([45](https://arxiv.org/html/2608.04026#S2.E45)\) in \([46](https://arxiv.org/html/2608.04026#S2.E46)\) yields

𝔼​\{Δ​f​\(t\+1\)\}≤−δp​\(t\)κp​\(t\)​λ2​\(t\)\+c22​δp2​\(t\)κp2​\(t\)\+O​\(δp3​\(t\)\)\.\\displaystyle\\mathbb\{E\}\\\{\\Delta f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\}\\leq\-\\frac\{\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\\,\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\\frac\{c\_\{2\}\}\{2\}\\,\\frac\{\\delta\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\kappa\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\+O\(\\delta\_\{p\}^\{3\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\)\.\(47\)Observe that the second\-order Taylor model containsΔ​\(t\+1\)\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\},Δ2​\(t\+1\)\\Delta^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}, and a leading neglected remainder of orderΔ3​\(t\+1\)\\Delta^\{3\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\. To simultaneously control the first\-order term, second\-order term, and cubic remainder, it is sufficient thatp≥3p\\geq 3\. The choicep=4p=4is particularly attractive because it is the smallest moment order that simultaneously controls the variance and tail heaviness behavior of the update step process\. Although larger valuesp\>4p\>4provide progressively stronger higher\-order moment control, they do not fundamentally improve the underlying second\-order Taylor model\. Moreover, the benefits of controlling moments beyond order four often diminish faster than the difficulty of reliably estimating them in practice\(Ramachandranet al\.,[2025](https://arxiv.org/html/2608.04026#bib.bib526); Jondeau and Rockinger,[2006](https://arxiv.org/html/2608.04026#bib.bib303)\)\. Combined with the expected maximum\-magnitude trust\-region constraint, the choice ofp=4p=4therefore provides a practical level of update control without requiring very high\-order moment estimates\.

More importantly, the distinction between a fixed and a vanishing trust\-region constraint can also be noted\. Ifδp​\(t\)=μ\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\mu, we have𝔼​\{\|Δ​\(t\+1\)\|p\}≤μp\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}\\leq\\mu^\{p\}, then all lower moments remain bounded, and the Taylor approximation error remains controlled but generally non\-vanishing\. In contrast, the vanishing trust\-region scheduleδp​\(t\)→0\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0implies

𝔼​\{\|Δ​\(t\+1\)\|r\}→0,r≤p,\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{r\}\\\}\\to 0,\\qquad r\\leq p,\(48\)causing every lower\-order moment of the update step to vanish\. As a result,O​\(δp2​\(t\)\)/O​\(δp​\(t\)\)→0\{O\(\\delta\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\)\}/\{O\(\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\)\}\\to 0,O​\(δp3​\(t\)\)/O​\(δp​\(t\)\)→0\{O\(\\delta\_\{p\}^\{3\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\)\}/\{O\(\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\)\}\\to 0, so that both the second\-order correction and higher\-order Taylor remainder become asymptotically negligible relative to the leading first\-order term\. Consequently,

𝔼​\{Δ​f​\(t\+1\)\}≤−δp​\(t\)κp​\(t\)​λ2​\(t\)\+O​\(δp2​\(t\)\),\\displaystyle\\mathbb\{E\}\\\{\\Delta f\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\}\\leq\-\\frac\{\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\{\\kappa\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\}\\,\\lambda\_\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+O\(\\delta\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\),\(49\)and the local Taylor model becomes progressively more accurate as learning proceeds\.

Remarks:Therefore, the role of the vanishingpp\-moment trust\-region constraint extends beyond merely controlling update magnitudes\. By driving all lower moments of the update step to vanish, it simultaneously contracts the trust region and enforces asymptotic validity of the local Taylor\-series model used to describe the expected decrease in the loss\. As learning proceeds andδp​\(t\)→0\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\to 0, the expected loss decrease becomes increasingly well described by the first\-order term, while higher\-order terms become asymptotically negligible\.

## 3Spectral regularization

Denotev​\(t\)=ℍ​\{g​\(t\)\}v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\{\\mathbb\{H\}\}\\\{g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\}as the spectral regularization of theg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}sequence\. A linear operatorℍ\{\\mathbb\{H\}\}satisfying the spectral properties defined in \([7](https://arxiv.org/html/2608.04026#S1.E7)\) is non\-expansive in itsℓ2\\ell\_\{2\}norm, therefore∥v​\(t\)∥∞≤∥ℍ∥∞​∥g​\(t\)∥∞\\lVert v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rVert\_\{\\infty\}\\leq\\lVert\{\\mathbb\{H\}\}\\rVert\_\{\\infty\}\\,\\lVert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rVert\_\{\\infty\}, so it cannot increase its input signal energy\. Prior to the spectral regularization ofg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, and using Assumption 2, if𝔼​\{\|Δ​\(t\+1\)\|p\}=αp​\(t\)​𝔼​\{\|g​\(t\)\|p\}≤αp​\(t\)​∥g​\(t\)∥∞p≤δpp​\(t\)\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}=\\alpha^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\mathbb\{E\}\\\{\\lvert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}\\leq\\alpha^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\lVert g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rVert^\{p\}\_\{\\infty\}\\leq\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}forp≥2p\\geq 2, then after the regularization,

𝔼​\{\|Δ​\(t\+1\)\|p\}=αp​\(t\)​𝔼​\{\|v​\(t\)\|p\}≤αp​\(t\)​∥v​\(t\)∥∞p≤∥ℍ∥∞p​δpp​\(t\)\.\\displaystyle\\mathbb\{E\}\\\{\\lvert\\Delta\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\rvert^\{p\}\\\}=\\alpha^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\mathbb\{E\}\\\{\\lvert v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rvert^\{p\}\\\}\\leq\\alpha^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\lVert v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\rVert^\{p\}\_\{\\infty\}\\leq\\lVert\{\\mathbb\{H\}\}\\rVert^\{p\}\_\{\\infty\}\\,\\delta\_\{p\}^\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(50\)Therefore, all trust\-region guarantees inherited from the underlyingpp\-moment mechanism are preserved and may become strictly tighter after regularization\. Importantly, as a filter,ℍ\{\\mathbb\{H\}\}does not introduce an additional trust\-region constraint\. Rather, it acts as a lowpass filter removing high\-frequency gradient fluctuations before the learning\-rate mechanism is applied\. The trust\-region constraints continue to determine the allowable update magnitude, while the filter improves the quality of the gradient signal used to generate that update\.

### 3\.1Trust\-Region\-Preserving Spectral Regularization

Letℍ\{\\mathbb\{H\}\}be a first\-order, linear iteration\-invariant filter characterized by a pole locationβ∈ℝ\\beta\\in\{\\mathbb\{R\}\}and zero locationγ∈ℝ\\gamma\\in\{\\mathbb\{R\}\}, with transfer function

ℍ​\(z\)=η​1−γ​z−11−β​z−1,η=1−β1−γ,\\displaystyle\{\\mathbb\{H\}\}\(z\)=\\eta\\frac\{1\-\\gamma z^\{\-1\}\}\{1\-\\beta z^\{\-1\}\},\\qquad\\eta=\\frac\{1\-\\beta\}\{1\-\\gamma\},\(51\)wherez=ej​ωz=e^\{j\\omega\}over frequenciesω∈\[−π,π\]\\omega\\in\[\-\\pi,\\pi\], andℍ​\(1\)=1\{\\mathbb\{H\}\}\(1\)=1\. The squared magnitude, frequency response is

\|ℍ​\(ej​ω\)\|2=\(1−β\)2\(1−γ\)2​1\+γ2−2​γ​cos⁡ω1\+β2−2​β​cos⁡ω\.\\displaystyle\\lvert\{\\mathbb\{H\}\}\(e^\{j\\omega\}\)\\rvert^\{2\}=\\frac\{\(1\-\\beta\)^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\frac\{1\+\\gamma^\{2\}\-2\\gamma\\cos\\omega\}\{1\+\\beta^\{2\}\-2\\beta\\cos\\omega\}\.\(52\)To show that the system’s worst\-case gain, itsℍ∞\{\\mathbb\{H\}\}\_\{\\infty\}norm defined as∥ℍ∥∞=maxω∥ℍ\(ej​ω\)∥2=maxω\|ℍ\(ej​ω\)\|≤1,∀ω∈\[−π,π\]\\lVert\{\\mathbb\{H\}\}\\rVert\_\{\\infty\}=\\max\_\{\\omega\}\\,\\lVert\{\\mathbb\{H\}\}\(e^\{j\\omega\}\)\\rVert\_\{2\}=\\max\_\{\\omega\}\\,\\lvert\{\\mathbb\{H\}\}\(e^\{j\\omega\}\)\\rvert\\leq 1,\\,\\forall\\,\\omega\\in\[\-\\pi,\\pi\], it suffices to show conditions for which the denominator of \([52](https://arxiv.org/html/2608.04026#S3.E52)\) is always greater than or equal to its numerator\. Rearranging terms, this implies

Δ​\(ω\)\\displaystyle\\Delta\(\\omega\)=\(1−γ\)2​\(1\+β2−2​β​cos⁡ω\)−\(1−β\)2​\(1\+γ2−2​γ​cos⁡ω\)≥0\\displaystyle=\(1\-\\gamma\)^\{2\}\(1\+\\beta^\{2\}\-2\\beta\\cos\\omega\)\-\(1\-\\beta\)^\{2\}\(1\+\\gamma^\{2\}\-2\\gamma\\cos\\omega\)\\geq 0=2​\(β−γ\)​\(1−β​γ\)​\(1−cos⁡ω\)≥0\.\\displaystyle=2\(\\beta\-\\gamma\)\(1\-\\beta\\gamma\)\(1\-\\cos\\omega\)\\geq 0\.\(53\)Note that in \([53](https://arxiv.org/html/2608.04026#S3.E53)\), the following holds:1−cos⁡ω≥01\-\\cos\\omega\\geq 0is non\-negative for allω\\omega; factorβ−γ≥0\\beta\-\\gamma\\geq 0if\|γ\|<\|β\|\\lvert\\gamma\\rvert<\\lvert\\beta\\rvert; and factor1−β​γ\>01\-\\beta\\gamma\>0if\|β\|<1\|\\beta\|<1and\|γ\|<1\|\\gamma\|<1\. Therefore, a sufficient condition for the system’s∥ℍ∥∞≤1\\lVert\{\\mathbb\{H\}\}\\rVert\_\{\\infty\}\\leq 1is

0≤β<1,\|γ\|<\|β\|\.\\displaystyle 0\\leq\\beta<1,\\qquad\\lvert\\gamma\\rvert<\\lvert\\beta\\rvert\.\(54\)
We will refer to this effect as a lowpass spectral regularization ofg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, providing strong attenuation of high\-frequency spectral components, while having a spectral norm less than one at all frequencies\.

### 3\.2Variance reduction factor

A useful measure of the output variance is the systemℍ2\{\\mathbb\{H\}\}\_\{2\}norm, its average energy over all frequencies,

∥ℍ∥22=12​π​∫−ππ∥ℍ​\(ej​ω\)∥F2​𝑑ω=12​π​∫−ππ\|ℍ​\(ej​ω\)\|2​𝑑ω=1−β1\+β​1\+γ2−2​β​γ\(1−γ\)2\\displaystyle\\lVert\{\\mathbb\{H\}\}\\rVert\_\{2\}^\{2\}=\\frac\{1\}\{2\\pi\}\\int\_\{\-\\pi\}^\{\\pi\}\\,\\lVert\{\\mathbb\{H\}\}\(e^\{j\\omega\}\)\\rVert\_\{F\}^\{2\}\\,d\\omega=\\frac\{1\}\{2\\pi\}\\int\_\{\-\\pi\}^\{\\pi\}\\,\\lvert\{\\mathbb\{H\}\}\(e^\{j\\omega\}\)\\rvert^\{2\}\\,d\\omega=\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{1\+\\gamma^\{2\}\-2\\beta\\gamma\}\{\(1\-\\gamma\)^\{2\}\}\(55\)In particular, for a white noise input signal, this quantifies the residual noise variance after filtering\(Orfanidis,[1995](https://arxiv.org/html/2608.04026#bib.bib476)\)\. Minimizing \([55](https://arxiv.org/html/2608.04026#S3.E55)\) is equivalent to maximizing variance reduction\. Given0≤β<10\\leq\\beta<1, \([55](https://arxiv.org/html/2608.04026#S3.E55)\) can be used to placeγ\\gamma\. Differentiating with respect toγ\\gamma,dd​γ​‖ℍ‖22\\frac\{d\}\{d\\gamma\}\\\|\{\\mathbb\{H\}\}\\\|\_\{2\}^\{2\}is strictly positive, and increasing over the admissible range−β<γ<β\-\\beta<\\gamma<\\beta\. Consequently, the minimum variance placement for the zero locationγ\\gammais

γMVR=−β\.\\displaystyle\\boxed\{\\gamma\_\{\\rm MVR\}=\-\\beta\}\.\(56\)
Substitutingγ=−β\\gamma=\-\\betaintoη=1−β1−γ\\eta=\\frac\{1\-\\beta\}\{1\-\\gamma\}, yieldsηMVR=1−β1\+β\\eta\_\{\\rm MVR\}=\\frac\{1\-\\beta\}\{1\+\\beta\}\. Asβ→1\\beta\\to 1,ηMVR=o​\(1−β\)\\eta\_\{\\rm MVR\}=o\(1\-\\beta\)\. Note that ifη=1−β\\eta=1\-\\beta, thenγ=0\\gamma=0\. Either way, maximum variance reduction is obtained at the cost of an increasingly smallη\\eta\. The normalization gainη\\etagoverns how the filter, realized in \([61](https://arxiv.org/html/2608.04026#S3.E61)\) responds to changes in its input signal\. In this case, the input signalg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}carries information about the local optimization landscape\. Asη→1\\eta\\to 1, minimal variance reduction, but more responsiveness to changes in the optimization landscape\. However asη→0\\eta\\to 0, maximal variance reduction, more frequency components in the input signal are attenuated more strongly, smoothing effect is increased, leading to less responsiveness to meaningful changes ing​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\. Consequently, in this setting, maximizing variance reduction alone is not an efficient design\. A useful filter should simultaneously reduce variance while preserving the ability to track changes in its input\.

### 3\.3Variance reduction per unit gain

A more efficient design is to evaluate the variance measure relative to its responsiveness measure, defined as

J​\(γ\)=∥ℍ∥22η=1\+γ2−2​β​γ\(1\+β\)​\(1−γ\)\.\\displaystyle J\(\\gamma\)=\\frac\{\\lVert\{\\mathbb\{H\}\}\\rVert\_\{2\}^\{2\}\}\{\\eta\}=\\frac\{1\+\\gamma^\{2\}\-2\\beta\\gamma\}\{\(1\+\\beta\)\(1\-\\gamma\)\}\.\(57\)Minimizing \([57](https://arxiv.org/html/2608.04026#S3.E57)\) is equivalent to maximizing the filter’s variance reduction per unit gain\. Differentiating with respect toγ\\gammayields\(1−γ\)2=2​\(1−β\)\(1\-\\gamma\)^\{2\}=2\(1\-\\beta\), whose feasible root isγinterior=1−2​\(1−β\)\\gamma\_\{\\rm interior\}=1\-\\sqrt\{2\(1\-\\beta\)\}\. Subject to−β≤γ≤β\-\\beta\\leq\\gamma\\leq\\beta, the optimal solution becomes

γVRG=max\{−β,1−2​\(1−β\)\}\.\\displaystyle\\boxed\{\\gamma\_\{\\rm VRG\}=\\max\\left\\\{\-\\beta,\\;1\-\\sqrt\{2\(1\-\\beta\)\}\\right\\\}\.\}\(58\)The transition occurs when−β=1−2​\(1−β\)\-\\beta=1\-\\sqrt\{2\(1\-\\beta\)\}, which yields a thresholdβ=5−2≈0\.236\\beta=\\sqrt\{5\}\-2\\approx 0\.236related to the golden ratio\. Therefore, for small0≤β<5−20\\leq\\beta<\\sqrt\{5\}\-2, the variance\-per\-unit\-gain optimumγVRG=−β\\gamma\_\{\\rm VRG\}=\-\\betacoincides with the maximum variance reduction choice\. Otherwise, asβ→1\\beta\\to 1, for5−2≤β<1\\sqrt\{5\}\-2\\leq\\beta<1, the variance\-per\-unit\-gain optimum increasingly favors preservation of responsiveness,γVRG=1−2​\(1−β\)\\gamma\_\{\\rm VRG\}=1\-\\sqrt\{2\(1\-\\beta\)\}\.

### 3\.4Canonical Filter Realization

A direct difference equation realization of the first\-order filter \([51](https://arxiv.org/html/2608.04026#S3.E51)\) is

v​\(t\)=β​v​\(t−1\)\+η​\(g​\(t\)−γ​g​\(t−1\)\)\.\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\beta\\,v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\\eta\\,\\big\(g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\\gamma\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\\big\)\.\(59\)In general, \([59](https://arxiv.org/html/2608.04026#S3.E59)\) admits the controllable canonical realization\(Oppenheim and Schafer,[2010](https://arxiv.org/html/2608.04026#bib.bib469); Smith,[2007](https://arxiv.org/html/2608.04026#bib.bib610)\),

q​\(t\)=β​q​\(t−1\)\+g​\(t\),v​\(t\)=η​\(q​\(t\)−γ​q​\(t−1\)\),\\displaystyle\\begin\{aligned\} q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}&=\\beta\\,q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\\\ v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}&=\\eta\\bigl\(q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\\gamma q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\\bigr\),\\end\{aligned\}\(60\)whereq​\(t\)q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}is the state generated during realization of the filter\. Additional refinements, such as dividing the filter output by1−βt1\-\\beta^\{t\}, compensate for the transient\-response bias of the state during the initial iterations\.

To make the lowpass regularization ofg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}explicit, defineq~​\(t\)=\(1−β\)​q​\(t\)\\tilde\{q\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\(1\-\\beta\)\\,q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\. Sinceη=1−β1−γ\\eta=\\tfrac\{1\-\\beta\}\{1\-\\gamma\}and1−η=β−γ1−γ1\-\\eta=\\tfrac\{\\beta\-\\gamma\}\{1\-\\gamma\}, it follows thatη​\(β−γ\)​q​\(t−1\)=\(1−η\)​q~​\(t−1\)\\eta\(\\beta\-\\gamma\)q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}=\(1\-\\eta\)\\tilde\{q\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\. Substituting into \([60](https://arxiv.org/html/2608.04026#S3.E60)\) yields

q~​\(t\)=β​q~​\(t−1\)\+\(1−β\)​g​\(t\),v​\(t\)=η​g​\(t\)\+\(1−η\)​q~​\(t−1\),⇔e​\(t\)=β​e​\(t−1\)\+\(g​\(t−1\)−g​\(t\)\),v​\(t\)=g​\(t\)\+\(1−η\)​e​\(t\),\\displaystyle\\begin\{aligned\} \\tilde\{q\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}&=\\beta\\,\\tilde\{q\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\(1\-\\beta\)\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\\\ v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}&=\\eta\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\(1\-\\eta\)\\,\\tilde\{q\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\},\\end\{aligned\}\\qquad\\Leftrightarrow\\qquad\\begin\{aligned\} e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}&=\\beta\\,e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\\bigl\(g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\-g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\bigr\),\\\\ v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}&=g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\(1\-\\eta\)\\,e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\end\{aligned\}\(61\)where the right\-hand\-side representation follows from the change of variablese​\(t\)=q~​\(t−1\)−g​\(t\)e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\tilde\{q\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\-g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.

### 3\.5Beyond Variance Reduction

From \([61](https://arxiv.org/html/2608.04026#S3.E61)\), recalle​\(t\)=β​e​\(t−1\)\+\(g​\(t−1\)−g​\(t\)\)e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\beta\\,e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\(g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\-g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\)\. Using the unit\-delay operatorz−1z^\{\-1\}, we have\(g​\(t\)−g​\(t−1\)\)=\(1−z−1\)​g​\(t\)\(g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\)=\(1\-z^\{\-1\}\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\. Thee​\(t\)e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}state recursion with zero initializatione​\(0\)=0e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=0can be expressed as

e​\(t\)=−∑k=0t−1βk​z−k​\(1−z−1\)​g​\(t\),\\displaystyle e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\-\\sum\_\{k=0\}^\{t\-1\}\\beta^\{k\}z^\{\-k\}\(1\-z^\{\-1\}\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\(62\)and hencev​\(t\)=g​\(t\)\+\(1−η\)​e​\(t\)v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\(1\-\\eta\)\\,e\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}is

v​\(t\)=g​\(t\)−\(1−η\)​∑k=0t−1βk​z−k​\(1−z−1\)​g​\(t\)\.\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\(1\-\\eta\)\\sum\_\{k=0\}^\{t\-1\}\\beta^\{k\}z^\{\-k\}\(1\-z^\{\-1\}\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(63\)Equation \([62](https://arxiv.org/html/2608.04026#S3.E62)\) shows that the filter state is an exponentially weighted series of delayed first\-order finite differences\. Hence, \([63](https://arxiv.org/html/2608.04026#S3.E63)\) can be interpreted asg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}being regularized through a correction term formed from an exponentially weighted series of delayed first\-order finite differences\. Each termz−k​\(1−z−1\)​g​\(t\)z^\{\-k\}\(1\-z^\{\-1\}\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, is generated by delayed applications of the first\-order finite\-difference operator\(1−z−1\)\(1\-z^\{\-1\}\)ong​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}, wherez−k​g​\(t\)=g​\(t−k\)z^\{\-k\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-k\)$\}\}\}, andz−k​\(1−z−1\)​g​\(t\)=\(1−z−1\)​g​\(t−k\)z^\{\-k\}\(1\-z^\{\-1\}\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\(1\-z^\{\-1\}\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-k\)$\}\}\}\. Consequently, gradient componentsg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}that vary rapidly across iterations incur larger correction terms, whereas nearly constant or slowly varying components produce smaller corrections\. Therefore, beyond typical noise variance reduction, the filter performs a trust\-region\-preserving smoothness regularization of the gradient sequence by selectively penalizing large iteration\-to\-iteration changes in each gradient component\.

Using a local finite\-difference expansion of the delayed sequenceg​\(t−k\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-k\)$\}\}\}about indextt,

g​\(t−k\)=g​\(t\)−k​Δ​g​\(t\)\+k​k−12​Δ2​g​\(t\)−⋯\\displaystyle g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-k\)$\}\}\}=g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-k\\,\\Delta g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+k\\tfrac\{k\-1\}\{2\}\\,\\Delta^\{2\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\\cdots\(64\)The expansion \([64](https://arxiv.org/html/2608.04026#S3.E64)\) involves finite differencesΔi​g​\(t\)=\(1−z−1\)i​g​\(t\)\\Delta^\{i\}\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\(1\-z^\{\-1\}\)^\{i\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}of increasing orderii\. Consequently, applying\(1−z−1\)\(1\-z^\{\-1\}\)to both sides of \([64](https://arxiv.org/html/2608.04026#S3.E64)\), and neglecting the third\-order term and above, gives\(1−z−1\)​g​\(t−k\)\(1\-z^\{\-1\}\)\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-k\)$\}\}\}, the weighted term in \([63](https://arxiv.org/html/2608.04026#S3.E63)\) as approximately

\(1−z−1\)​g​\(t−k\)≈Δ​g​\(t\)−k​Δ2​g​\(t\),\\displaystyle\(1\-z^\{\-1\}\)\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-k\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-k\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-k\)$\}\}\}\\approx\\Delta g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-k\\,\\Delta^\{2\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\(65\)and then \([63](https://arxiv.org/html/2608.04026#S3.E63)\) becomes

v​\(t\)≈g​\(t\)−\(1−η\)​\(∑k=0t−1βk\)​Δ​g​\(t\)\+\(1−η\)​\(∑k=0t−1k​βk\)​Δ2​g​\(t\)\.\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\approx g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\(1\-\\eta\)\\Biggl\(\\sum\_\{k=0\}^\{t\-1\}\\beta^\{k\}\\Biggr\)\\Delta g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\(1\-\\eta\)\\Biggl\(\\sum\_\{k=0\}^\{t\-1\}k\\,\\beta^\{k\}\\Biggr\)\\Delta^\{2\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(66\)Forttsufficiently large relative to\(1−β\)−1\(1\-\\beta\)^\{\-1\}, the finite exponentially\-weighted sums may be further approximated by their infinite\-series limits,

v​\(t\)≈g​\(t\)−\(1−η\)​\(∑k=0∞βk\)​Δ​g​\(t\)\+\(1−η\)​\(∑k=0∞k​βk\)​Δ2​g​\(t\)\.\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\approx g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\(1\-\\eta\)\\Biggl\(\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}\\Biggr\)\\Delta g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\(1\-\\eta\)\\Biggl\(\\sum\_\{k=0\}^\{\\infty\}k\\,\\beta^\{k\}\\Biggr\)\\Delta^\{2\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(67\)Then, substituting

∑k=0∞βk=11−β,∑k=0∞k​βk=β\(1−β\)2,\\displaystyle\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}=\\frac\{1\}\{1\-\\beta\},\\qquad\\sum\_\{k=0\}^\{\\infty\}k\\beta^\{k\}=\\frac\{\\beta\}\{\(1\-\\beta\)^\{2\}\},\(68\)it follows that

v​\(t\)≈g​\(t\)−1−η1−β​\(Δ​g​\(t\)−β1−β​Δ2​g​\(t\)\)\.\\displaystyle v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\approx g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\\frac\{1\-\\eta\}\{1\-\\beta\}\\Bigl\(\\Delta g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\-\\frac\{\\beta\}\{1\-\\beta\}\\,\\Delta^\{2\}g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\Bigr\)\.\(69\)Consequently, the approximation in \([69](https://arxiv.org/html/2608.04026#S3.E69)\) suggests that the first\-order filter implicitly performs a local prediction of theg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}trajectory\. The filter output combines the current gradient with weighted estimates of its local trend and curvature, represented by the first\-order and second\-order finite differences\. From this viewpoint, the filter acts not only as a variance\-reduction mechanism, but also as a gradient\-shaping operator that partially extrapolates the local evolution ofg​\(t\)g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}while suppressing high\-frequency fluctuations\.

### 3\.6Heavy\-ball and Nesterov momentum

Although, our design choice \([58](https://arxiv.org/html/2608.04026#S3.E58)\), makes the linear operator act, by default, as an efficient lowpass filter, the same filter admits other closely related configurations\. More generally, by appropriate choices ofγ\\gamma, the linear operator can be used to recover both Heavy\-ball and Nesterov momentum\. Algebraically, analyzing \([60](https://arxiv.org/html/2608.04026#S3.E60)\), Heavy\-ball momentum can be recovered by selectingγHB=0\\gamma\_\{\\rm HB\}=0,

v​\(t\)=β​v​\(t−1\)\+\(1−β\)​g​\(t\),\\displaystyle\\begin\{aligned\} &v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\beta\\,v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\(1\-\\beta\)\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\end\{aligned\}\(70\)while Nesterov momentum corresponds toγNAG=β1\+β\\gamma\_\{\\rm NAG\}=\\frac\{\\beta\}\{1\+\\beta\},

q​\(t\)=β​q​\(t−1\)\+\(1−β\)​g​\(t\)v​\(t\)=β​q​\(t\)\+\(1−β\)​g​\(t\)\.\\displaystyle\\begin\{aligned\} &q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\beta\\,q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\(1\-\\beta\)\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\\ &v\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\beta\\,q\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\+\(1\-\\beta\)\\,g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\\end\{aligned\}\(71\)Given0≤β<10\\leq\\beta<1, four special operating points of the filter are therefore

γMVR=−β,γHB=0,γNAG=β1\+β,γVRG=max\{−β,1−2​\(1−β\)\}\.\\displaystyle\\boxed\{\\gamma\_\{\\rm MVR\}=\-\\beta,\\qquad\\gamma\_\{\\rm HB\}=0,\\qquad\\gamma\_\{\\rm NAG\}=\\frac\{\\beta\}\{1\+\\beta\},\\qquad\\gamma\_\{\\rm VRG\}=\\max\\\{\-\\beta,1\-\\sqrt\{2\(1\-\\beta\)\}\\\}\.\}\(72\)
Reducing∥ℍ∥22\\lVert\{\\mathbb\{H\}\}\\rVert\_\{2\}^\{2\}to satisfy the moment\-based trust\-region constraints, leads to a tradeoff between two competing effects\. Maximum variance reduction causes a simultaneous maximum reduction in the filter’s normalization gain, leading to a smaller effective maximum step\-size and less responsive dynamics\. The efficient solution balancing this tradeoff is a filter design with maximal variance reduction per unit gain\.

![Refer to caption](https://arxiv.org/html/2608.04026v1/vrg_vs_beta_plt.png)Figure 2:Variance reduction per unit gain\. Forβ\>5−2\\beta\>\\sqrt\{5\}\-2,γMVR\\gamma\_\{\\rm MVR\}clearly becomes inefficient\. Observe thatγNAG\\gamma\_\{\\rm NAG\}closely matchesγVRG\\gamma\_\{\\rm VRG\}asβ≫0\.5\\beta\\gg 0\.5, especiallyβ≈0\.8−0\.9\\beta\\approx 0\.8\-0\.9, whereasγHB\\gamma\_\{\\rm HB\}closely matchesγVRG\\gamma\_\{\\rm VRG\}forβ≈0\.4−0\.6\\beta\\approx 0\.4\-0\.6\. At the cross\-over pointβ≈0\.71\\beta\\approx 0\.71,γNAG\\gamma\_\{\\rm NAG\}overtakesγHB\\gamma\_\{\\rm HB\}as an efficient design\.Heavy\-ball and Nesterov momentum are specific operating choices ofγ\\gammawithin the same family\. In particular, for smallerβ\\beta, Heavy\-ball achieves more variance reduction per unit gain than Nesterov’s momentum\. In contrast, for largerβ→1\\beta\\to 1, Nesterov’s momentum achieves more variance reduction per unit gain\. Additionally, this lens provides a unified spectral regularization interpretation of classic momentum methods through the pole\-zero locations of a first\-order lowpass filter, which correspond to tradeoffs between maximal variance reduction and responsiveness of the filtering dynamics to the local optimization landscape\. The trust\-region preserving regularized algorithm is outlined in \([8](https://arxiv.org/html/2608.04026#S1.E8)\)–\([12](https://arxiv.org/html/2608.04026#S1.E12)\), with \([9](https://arxiv.org/html/2608.04026#S1.E9)\) being the only addition to the basic algorithm\.

## 4Matrix\-operator form

In Section[2](https://arxiv.org/html/2608.04026#S2), a vectorized parameter group was considered\. In contrast, now letn′=n​mn^\{\\prime\}=nm, and letG~\(t\):=G¯\(t\)\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}:=\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}denote a full\-rankn×mn\\times mmatrix composed of normalized gradient componentsg¯​\(t\)\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}associated with a single layer or parameter groupll, where1<n≤m1<n\\leq m\. In matrix form, denote the update step in \([8](https://arxiv.org/html/2608.04026#S1.E8)\)–\([12](https://arxiv.org/html/2608.04026#S1.E12)\) as

Δl​\(t\+1\)=−δp​\(t\)​G~​\(t\)\.\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.Thepp\-moment normalization acts on each individuali​jijentry of the matrix, where1≤i≤n1\\leq i\\leq nand1≤j≤m1\\leq j\\leq m, and therefore enforces𝔼​\{\|Δi​jl​\(t\+1\)\|r\}≤μr\\mathbb\{E\}\\\{\|\{\\Delta^\{l\}\_\{ij\}\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\|^\{r\}\\\}\\leq\\mu^\{r\},1≤r≤p1\\leq r\\leq p, which implies𝔼​\{\|G~i​j​\(t\)\|r\}≤1\\mathbb\{E\}\\\{\|\\tilde\{G\}\_\{ij\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{r\}\\\}\\leq 1\. Viewing the update as a matrix, the corresponding operator gain satisfies, forp≥2p\\geq 2,

𝔼​‖Δl​\(t\+1\)‖22≤𝔼​‖Δl​\(t\+1\)‖F2=δp2​\(t\)​∑i​jm​n𝔼​\{\|G~i​j​\(t\)\|2\}≤μ2​m​n\.\\displaystyle\\mathbb\{E\}\\\|\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{2\}^\{2\}\\leq\\mathbb\{E\}\\\|\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{F\}^\{2\}=\\delta\_\{p\}^\{2\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\sum\_\{ij\}^\{mn\}\\mathbb\{E\}\\\{\|\\tilde\{G\}\_\{ij\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{2\}\\\}\\leq\\mu^\{2\}mn\.\(73\)Therefore, although the entrywisepp\-moment trust\-region controls the magnitude of the vectorized update, it does not directly control the operator gain of the corresponding matrix update group\. In particular, the bound \([73](https://arxiv.org/html/2608.04026#S4.E73)\) exceeds the desired trust\-region radiusμ2\\mu^\{2\}\.

An algebraic consequence is that if𝔼​‖Δl​\(t\+1\)‖2≤μ,\\mathbb\{E\}\\\|\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{2\}\\leq\\mu,then𝔼​\|Δi​jl​\(t\+1\)\|≤𝔼​‖Δl​\(t\+1\)‖2≤μ\.\\mathbb\{E\}\|\{\\Delta^\{l\}\_\{ij\}\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\|\\leq\\mathbb\{E\}\\\|\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{2\}\\leq\\mu\.Therefore, a tighter form of trust\-region control is obtained by constraining the entire matrix update group at the spectral level according to

‖Δl​\(t\+1\)‖2≤μ′,\\displaystyle\\\|\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{2\}\\leq\\mu^\{\\prime\},\(74\)whereμ≤μ′<μ​m​n\\mu\\leq\\mu^\{\\prime\}<\\mu\\sqrt\{mn\}\. In this range, the special caseμ′=μ\\mu^\{\\prime\}=\\mucorresponds to the strongest trust\-region constraint relative to \([73](https://arxiv.org/html/2608.04026#S4.E73)\)\. An effective way to control the spectrum of a matrix is through an orthogonalization step\(Higham,[2008](https://arxiv.org/html/2608.04026#bib.bib257)\), such as,

G~​\(t\)=\(G¯​\(t\)​G¯​\(t\)⊺\)−12​G¯​\(t\)\.\\displaystyle\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\left\(\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\intercal\}\\right\)^\{\-\\frac\{1\}\{2\}\}\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\.\(75\)By the singular\-value decompositionG¯​\(t\)=U​\(t\)​Σ​\(t\)​V​\(t\)⊺,\\bar\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=U\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\Sigma\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}V\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\intercal\},it follows thatG~​\(t\)=U​\(t\)​V​\(t\)⊺,\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=U\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}V\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\intercal\},and therefore‖G~​\(t\)‖2=1\.\\\|\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{2\}=1\.The layer or matrix group update stepΔl​\(t\+1\)=−δp​\(t\)​G~​\(t\)\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}=\-\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\,\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}then satisfies

‖Δl​\(t\+1\)‖2=δp​\(t\)​‖G~​\(t\)‖2=δp​\(t\)≤μ\.\\displaystyle\\\|\\Delta^\{l\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\\\|\_\{2\}=\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{2\}=\\delta\_\{p\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\leq\\mu\.\(76\)SinceG~​\(t\)=U​\(t\)​V​\(t\)⊺\\tilde\{G\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=U\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}V\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}^\{\\intercal\}, eachi​jijentry, satisfies\|G~i​j​\(t\)\|=\|ui⊺​\(t\)​vj​\(t\)\|≤‖ui​\(t\)‖2​‖vj​\(t\)‖2≤1,\|\\tilde\{G\}\_\{ij\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|=\|u\_\{i\}^\{\\intercal\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}v\_\{j\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|\\leq\\\|u\_\{i\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{2\}\\\|v\_\{j\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\\\|\_\{2\}\\leq 1,and therefore\|Δi​jl​\(t\+1\)\|≤μ\|\\Delta^\{l\}\_\{ij\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\+1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\+1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\+1\)$\}\}\}\|\\leq\\muis also satisfied\.

This matrix\-operator form of the regularized algorithm is outlined in \([13](https://arxiv.org/html/2608.04026#S1.E13)\)–\([18](https://arxiv.org/html/2608.04026#S1.E18)\), with \([17](https://arxiv.org/html/2608.04026#S1.E17)\) introduced\. Moment normalization does not directly control the operator gain \(largest singular value\) of the matrix group, while spectral normalization does not regulate the statistical variability of its individual entries\. Therefore, the precedingpp\-moment normalization and the subsequent spectral normalization are complementary trust\-region mechanisms that operate on different properties of the update matrix group\.

By directly controlling the operator gain of the entire matrix update group to be less than the maximum trust\-region constantμ\\mu, spectral normalization strengthens the underlying trust\-region framework\.

## 5Practical realizations

This section discusses practical realizations of the statistical expectations appearing inGmake, together with a practical realization of the principal matrix inverse square\-root required by the matrix\-operator form\.

Matrix inverse square\-root\. For computing the principal matrix inverse square\-root in \([17](https://arxiv.org/html/2608.04026#S1.E17)\), we adopt the efficient polynomial recursion ofLakić \([1998](https://arxiv.org/html/2608.04026#bib.bib345)\)\.

Linear statistical estimators\.Gmakerequires online estimates of𝔼​\{\|g​\(t\)\|2\}≥0\\mathbb\{E\}\\\{\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{2\}\\\}\\geq 0and𝔼​\{\|g¯​\(t\)\|p\}≥1\\mathbb\{E\}\\\{\|\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\}\\\}\\geq 1\. Since these expectations are unavailable*a priori*, they must be replaced by recursive estimators\. Here, we adopt classical estimators from the stochastic approximation literature whose statistical properties are already well established\(Zoubiret al\.,[2018](https://arxiv.org/html/2608.04026#bib.bib733); Lehmann and Casella,[2005](https://arxiv.org/html/2608.04026#bib.bib364); James and Stein,[1961](https://arxiv.org/html/2608.04026#bib.bib294)\)\. A common practical estimator is the exponentially weighted moving average \(EMA\)

m​\(t\)=ρ​m​\(t−1\)\+\(1−ρ\)​x​\(t\),0<ρ<1,\\displaystyle m\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\rho\\,m\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\-1\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\-1\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\-1\)$\}\}\}\+\(1\-\\rho\)\\,x\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\qquad 0<\\rho<1,\(77\)which may be used to estimate both quantities by selectingx​\(t\)=\|g​\(t\)\|2x\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\|g\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{2\},m​\(0\)=0m\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=0, orx​\(t\)=\|g¯​\(t\)\|px\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\|\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\},m​\(0\)=1m\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(0\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(0\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(0\)$\}\}\}=1, respectively\. For the normalized moment𝔼​\{\|g¯​\(t\)\|p\}\\mathbb\{E\}\\\{\|\\bar\{g\}\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}\|^\{p\}\\\}, sometimes a possible nominal or steady\-state value can often be close to unity\. In such situations, a linear shrinkage estimator \(LSE\)

m​\(t\)=ρ​m0\+\(1−ρ\)​x​\(t\),0<ρ<1,\\displaystyle m\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\}=\\rho\\,m\_\{0\}\+\(1\-\\rho\)\\,x\\mathchoice\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\scriptsize$\(t\)$\}\}\}\{\\text\{\\raisebox\{0\.43057pt\}\{\\tiny$\(t\)$ \}\}\}\{\\text\{\\raisebox\{3\.87495pt\}\{\\tiny$\(t\)$\}\}\},\\qquad 0<\\rho<1,\(78\)with prior valuem0=1m\_\{0\}=1, may also provide an alternative realization\(James and Stein,[1961](https://arxiv.org/html/2608.04026#bib.bib294)\)\. When the prior is accurate, the LSE can reduce estimator variance\(Ledoit and Wolf,[2004](https://arxiv.org/html/2608.04026#bib.bib358)\)while avoiding the additional memory state required by a second EMA\.

For both realizations, values ofρ\\rhovery close to unity typically provide improved averaging\(Haykin,[2014](https://arxiv.org/html/2608.04026#bib.bib240); Ljung and Söderström,[1983](https://arxiv.org/html/2608.04026#bib.bib394)\)\. Both estimators are also subject to initial transient bias from the true underlying mean value\(Boxet al\.,[2015](https://arxiv.org/html/2608.04026#bib.bib80); Goodwin and Sin,[1984](https://arxiv.org/html/2608.04026#bib.bib215)\)\. The typical bias correction may be applied by dividing the estimate by1−ρt1\-\\rho^\{t\}\.

## 6Numerical Experiments

The algorithms defined by \([3](https://arxiv.org/html/2608.04026#S1.E3)\)–\([6](https://arxiv.org/html/2608.04026#S1.E6)\), \([8](https://arxiv.org/html/2608.04026#S1.E8)\)–\([12](https://arxiv.org/html/2608.04026#S1.E12)\), and \([13](https://arxiv.org/html/2608.04026#S1.E13)\)–\([18](https://arxiv.org/html/2608.04026#S1.E18)\) will be referred to as the basic, spectrally filtered, and matrix\-operator forms ofGmake, respectively\. These correspond to progressively stronger realizations of the underlying trust\-region framework\. As proof of concept, we compare the three forms for bothp=2p=2andp=4p=4\. All experiments, were repeated three times, and the average training loss and validation loss curves are reported\. The GPT2\-124M model processes 8192 tokens per iteration

### 6\.1FineWeb\-Edu

![Refer to caption](https://arxiv.org/html/2608.04026v1/tr_t2_GMAKE_GPT124M_FWEB50M__cmps_asgm_runs=3_blksz=128_bsize=64.png)

![Refer to caption](https://arxiv.org/html/2608.04026v1/tt_t2_GMAKE_GPT124M_FWEB50M__cmps_asgm_runs=3_blksz=128_bsize=64.png)

Figure 3:Training \(left\) and validation \(right\) loss for the basic \(B\), spectrally filtered \(F\), and matrix\-operator \(M\) forms ofGmakewithp=2p=2andp=4p=4on GPT2\-124M trained on FineWeb\-Edu\.Figure[3](https://arxiv.org/html/2608.04026#S6.F3)compares the basic \(B\), filtered \(F\), and matrix\-operator \(M\) forms ofGmakefor bothp=2p=2andp=4p=4on GPT2\-124M trained on a 50 million token subset of FineWeb\-Edu\. In all experiments, the same maximum trust\-region radiusμ=5×10−4\\mu=5\\times 10^\{\-4\}, filter parameterβ=0\.9\\beta=0\.9, averaging coefficientρ=0\.999\\rho=0\.999, and linear decay schedule were used\.

Several observations can be made\. First, the filtered realizations substantially improve both training and validation performance relative to their corresponding basic forms\. This is consistent with the analysis of Section 3, where spectral lowpass regularization \(momentum\) acts as a trust\-region\-preserving mechanism that improves the quality of the gradient signal before update generation\.

Second, within the basic realization, the fourth\-moment form consistently outperforms the corresponding second\-moment form\. This suggests that, when a relatively large trust\-region radius is permitted, the additional higher\-order moment normalization provides beneficial update control beyond that obtained from second\-moment normalization alone\.

Third, the matrix\-operator realizations remain competitive with their corresponding filtered forms despite satisfying a stronger operator\-level trust\-region constraint\. Since all realizations use the same maximum trust\-region radiusμ\\mu, the matrix\-operator form provides the strongest trust\-region guarantees among the three realizations while maintaining comparable optimization performance\.

### 6\.2TinyStories

![Refer to caption](https://arxiv.org/html/2608.04026v1/tr_t2_GMAKE_GPT124M_TSTORIES__cmps_asgm_runs=3_blksz=128_bsize=64.png)

![Refer to caption](https://arxiv.org/html/2608.04026v1/tt_t2_GMAKE_GPT124M_TSTORIES__cmps_asgm_runs=3_blksz=128_bsize=64.png)

Figure 4:Training \(left\) and validation \(right\) loss for the basic \(B\), spectrally filtered \(F\), and matrix\-operator \(M\) forms ofGmakewithp=2p=2andp=4p=4on GPT2\-124M trained on TinyStories\.The results on TinyStories largely mirror those observed on FineWeb\-Edu\. The filtered realizations again outperform their corresponding basic forms, supporting the view that spectral lowpass regularization provides a useful strengthening of the underlying trust\-region framework\.

The fourth\-moment realization again achieves lower losses than the second\-moment realization in the basic form\. However, after the introduction of filtering and matrix\-operator normalization, the performance gap between the second\- and fourth\-moment realizations becomes considerably smaller\.

Across both datasets, the filtered and matrix\-operator training\-loss trajectories are nearly indistinguishable, while their corresponding validation losses remain similarly close\. These observations suggest that progressively stronger trust\-region controls can be imposed without materially degrading optimization performance\.

### 6\.3Effect of Strengthening the Trust\-Region

![Refer to caption](https://arxiv.org/html/2608.04026v1/tt_tl_GMAKE_GPT124M_FWEB50M__cmps_asgm_runs=3_blksz=128_bsize=64.png)

![Refer to caption](https://arxiv.org/html/2608.04026v1/tt_tl_GMAKE_GPT124M_TSTORIES__cmps_asgm_runs=3_blksz=128_bsize=64.png)

Figure 5:Validation loss on GPT2\-124M trained on FineWeb\-Edu \(left\) and TinyStories \(right\)\. Solid lines correspond to the basicGmakerealization with trust\-region radiusμ=5×10−4\\mu=5\\times 10^\{\-4\}, while dash\-dotted lines correspond toμ=3×10−4\\mu=3\\times 10^\{\-4\}\.RMSPropis included only forμ=3×10−4\\mu=3\\times 10^\{\-4\}because training withμ=5×10−4\\mu=5\\times 10^\{\-4\}was unstable\. The larger trust\-region radius highlights the advantage of the fourth\-moment realization over the second\-moment realization, while reducing the trust\-region radius substantially narrows the performance gap between the two forms, with the second\-moment realization becoming slightly better\.Figure[5](https://arxiv.org/html/2608.04026#S6.F5)investigates the effect of reducing the maximum trust\-region radius fromμ=5×10−4\\mu=5\\times 10^\{\-4\}toμ=3×10−4\\mu=3\\times 10^\{\-4\}in the basic realization\.

A notable observation is that the performance advantage of the fourth\-moment realization becomes substantially smaller as the trust\-region radius is reduced\. Forμ=5×10−4\\mu=5\\times 10^\{\-4\}, the fourth\-moment realization consistently outperforms the second\-moment realization on both datasets\. However, at the smaller trust\-region radius, the performance gap narrows considerably and, in some cases, the second\-moment realization achieves slightly lower validation loss\.

This behavior is consistent with the trust\-region interpretation developed in Section 2\. The higher\-order normalization mechanism becomes most useful when larger update magnitudes are permitted\. As the trust\-region radius is reduced, update magnitudes are already more tightly constrained, leaving less opportunity for the higher\-order moment normalization to provide additional benefit\.

For reference,RMSPropwas also evaluated atμ=3×10−4\\mu=3\\times 10^\{\-4\}\. Training withμ=5×10−4\\mu=5\\times 10^\{\-4\}was found to be substantially less stable and is therefore omitted from the comparison\. At the smaller trust\-region radius, the performance ofRMSPropand the second\-moment realization become comparable\.

### 6\.4Trust\-Region Insights

The preceding experiments suggest several observations that are best understood through the trust\-region interpretation developed in Sections[2](https://arxiv.org/html/2608.04026#S2)\-[4](https://arxiv.org/html/2608.04026#S4)of this paper\.

First, for the larger trust\-region radiusμ=5×10−4\\mu=5\\times 10^\{\-4\}, the fourth\-moment realization consistently outperforms the corresponding second\-moment realization in the basic form on both FineWeb\-Edu and TinyStories\. This behaviour is consistent with the interpretation that the normalized fourth\-moment estimator provides additional update control beyond second\-moment normalization alone when relatively large update magnitudes are permitted\.

Second, the introduction of progressively stronger trust\-region controls appears to reduce the performance gap between the second\-moment and fourth\-moment realizations\. This effect is observed after the introduction of spectral lowpass regularization, matrix\-operator normalization, and also when the maximum trust\-region radius is reduced fromμ=5×10−4\\mu=5\\times 10^\{\-4\}toμ=3×10−4\\mu=3\\times 10^\{\-4\}in the basic realization\.

Figure[5](https://arxiv.org/html/2608.04026#S6.F5)provides additional evidence for this observation\. As the maximum trust\-region radius is reduced, the advantage of the fourth\-moment realization becomes substantially smaller, with the second\-moment and fourth\-moment realizations achieving comparable performance on both datasets\. This suggests that the benefits of higher\-order moment normalization are most pronounced when the admissible trust region is relatively large\.

Taken together, these results indicate that the relative advantage of the fourth\-moment realization may be inversely related to the strength of the surrounding trust\-region controls\. When the trust\-region constraints are weak, the additional higher\-order normalization provides a more meaningful correction to the update process\. As progressively stronger trust\-region controls are introduced through a smaller trust\-region radius, spectral filtering, or operator\-level normalization, the second\-moment realization appears sufficient to capture a larger fraction of the achievable performance gains\.

Finally,RMSPropwas observed to train less stably than itsGmakeequivalent at the larger trust\-region radiusμ=5×10−4\\mu=5\\times 10^\{\-4\}, and therefore is reported only forμ=3×10−4\\mu=3\\times 10^\{\-4\}\. This observation is consistent with the trust\-region interpretation developed in this paper, where the additional normalization mechanisms ofGmakepermit stable optimization under larger admissible update magnitudes\.

## 7Limitations, Gaps and Future Work

Although this trust\-region framework establishes theoretical connections between moment estimation, learning\-rate scheduling, spectral filtering, and matrix\-operator normalization, several questions remain open\.

A notable empirical observation is that the performance advantage of the fourth\-moment realization is most pronounced when the trust\-region constraints are relatively weak\. In the basic realization, operated with a larger trust\-region radius, the fourth\-moment form initially outperformed its second\-moment counterpart\. However, this advantage diminished substantially as stronger trust\-region controls were introduced, whether through spectral filtering, matrix\-operator normalization, or a smaller maximum trust\-region radius\. This trend suggests that part of the benefit provided by fourth\-moment normalization may overlap with the stabilization and regularization effects already supplied by these stronger trust\-region mechanisms\.

The precise cause of this behavior remains unclear\. One possible explanation is that these mechanisms partially address the same update fluctuations that the higher\-order moment normalization is designed to regulate, thereby reducing its incremental benefit\. Alternatively, the observation may indicate that spectral lowpass regularization and spectral trust\-region control naturally strengthen second\-moment mechanisms more than higher\-order moment mechanisms\.

Nonetheless, this observation may expose a limitation of the current framework\. While spectral lowpass regularization appears to provide a natural strengthening of the second\-moment trust\-region mechanism, a corresponding regularization principle specifically aimed at higher\-order moment control has not yet been identified\. It therefore remains unclear whether an analogous mechanism exists and, if so, whether it could recover or further enhance the benefits of thep=4p=4realization\. More broadly, a systematic investigation of the performance and stability characteristics across the range0<p≤40<p\\leq 4, including the regimep<2p<2, may provide deeper insight into the role of moment order in trust\-region control and accelerated learning\.

Moreover, the numerical experiments were intended primarily as proof\-of\-concept demonstrations rather than a comprehensive empirical evaluation\. For consistency across experiments, all realizations were evaluated using the same values ofμ\\mu,β\\beta, andρ\\rho, and only two representative choices ofppwere examined\. While the analysis focused on the range2≤p≤42\\leq p\\leq 4for convenience in relating the framework to second\-moment quantities, the theoretical development extends more broadly, and practical computation remains feasible for0<p≤40<p\\leq 4\. An important direction for future work is therefore a systematic investigation of the sensitivity of the basic, filtered, and matrix\-operator formulations to these hyperparameters\. Such studies could also explore alternative normalization orderings, adaptive selections ofpp, and spectral\-level trust\-region constructions across a wider range of learning problems\.

## 8Conclusion

This paper developed a trust\-region framework for the moment estimation mechanism in stochastic gradient optimization\. The derived family of learning\-rate mechanisms generalizes second\-moment methods throughpp\-th moment trust\-region constraints\. Within this framework,RMSPropandAdamcan be interpreted as similar to the second\-moment realizations of the basic and spectrally filteredGmakeformulations, respectively\.

TheGmakeframework further reveals that learning\-rate scheduling, momentum, and spectral normalization can be interpreted as complementary trust\-region mechanisms acting on different properties of the update process\. By extending the basic realization through spectral filtering and matrix\-operator normalization, progressively stronger trust\-region controls are obtained within a common theoretical framework\. The experimental results further suggest the existence of a trust\-region hierarchy, in which the incremental benefit of higher\-order moment normalization decreases as progressively stronger trust\-region controls are imposed on the update process\.

Taken together, these developments provide a unified trust\-region interpretation of several mechanisms commonly used in optimizing deep neural networks via the stochastic gradient algorithm\.

## References

- S\. Bergsma, N\. S\. Dey, G\. Gosal, G\. Gray, D\. Soboleva, and J\. Hestness \(2024\)Straight to zero: Why linearly decaying the learning rate to zero works best for LLMs\.InThe Thirteenth International Conference on Learning Representations,Singapore\.External Links:[Link](https://openreview.net/forum?id=hrOlBgHsMI)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px2.p1.1)\.
- J\. Bernstein and L\. Newhouse \(2024\)Old Optimizer, New Norm: An Anthology\.External Links:[Link](https://arxiv.org/abs/2409.20325v1)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px4.p1.1)\.
- L\. Bottou, F\. E\. Curtis, and J\. Nocedal \(2018\)Optimization methods for large\-scale machine learning\.SIAM Review60\(2\),pp\. 223–311\.Cited by:[§1\.1](https://arxiv.org/html/2608.04026#S1.SS1.p3.4),[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px1.p1.1)\.
- G\. E\. P\. Box, G\. M\. Jenkins, G\. C\. Reinsel, and G\. M\. Ljung \(2015\)Time series analysis: Forecasting and control\.5 edition,Wiley\.Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p4.2)\.
- A\. R\. Conn, N\. I\. M\. Gould, and P\. L\. Toint \(2000\)Trust Region Methods\.MOS\-SIAM Series on Optimization,Society for Industrial and Applied Mathematics\.External Links:[Document](https://dx.doi.org/10.1137/1.9780898719857),ISBN 978\-0\-89871\-460\-9Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px2.p1.1)\.
- S\. Elaydi \(2005\)An Introduction to Difference Equations\.Springer,New York\.External Links:ISBN 978\-0\-387\-27602\-1Cited by:[§2\.2](https://arxiv.org/html/2608.04026#S2.SS2.p3.9)\.
- R\. Ge, S\. M\. Kakade, R\. Kidambi, and P\. Netrapalli \(2018\)Rethinking learning rate schedules for stochastic optimization\.InInternational Conference on Learning Representations,New Orleans, LA\.External Links:[Link](https://openreview.net/forum?id=HJePy3RcF7)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px2.p1.1)\.
- G\. C\. Goodwin and K\. S\. Sin \(1984\)Adaptive Filtering Prediction and Control\.Dover Publications,Englewood Cliffs, N\.J\.External Links:ISBN 978\-0\-486\-46932\-4Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p4.2)\.
- S\. Haykin \(2014\)Adaptive Filter Theory\.5th, intern\. edition,Pearson,Upper Saddle River, NJ\.External Links:ISBN 978\-0\-13\-267145\-3Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p4.2)\.
- N\. J\. Higham \(2008\)Functions of Matrices: Theory and Computation\.Society for Industrial and Applied Mathematics\.External Links:[Link](https://arxiv.org/html/2608.04026v1/10.1137/1.9780898717778),ISBN 978\-0\-89871\-646\-7Cited by:[§4](https://arxiv.org/html/2608.04026#S4.p2.4)\.
- W\. James and C\. Stein \(1961\)Estimation with Quadratic Loss\.InProceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics,Vol\.4\.1,pp\. 361–380\.Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p3.2),[§5](https://arxiv.org/html/2608.04026#S5.p3.8)\.
- E\. Jondeau and M\. Rockinger \(2006\)Optimal Portfolio Allocation under Higher Moments\.European Financial Management12\(1\),pp\. 29–55\.External Links:ISSN 1468\-036X,[Document](https://dx.doi.org/10.1111/j.1354-7798.2006.00309.x)Cited by:[§2\.6](https://arxiv.org/html/2608.04026#S2.SS6.p2.24)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.In3rd International Conference on Learning Representations,San Diego, CA, USA,\.Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.04026#S1.p1.1)\.
- S\. Lakić \(1998\)On the Computation of the Matrix k\-th Root\.Journal of Applied Mathematics and Mechanics, ZAMM78\(3\),pp\. 167–172\.External Links:ISSN 1521\-4001Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p2.1)\.
- T\. Large, Y\. Liu, M\. Huh, H\. Bahng, P\. Isola, and J\. Bernstein \(2024\)Scalable Optimization in the Modular Norm\.Advances in Neural Information Processing Systems37,pp\. 73501–73548\.External Links:[Document](https://dx.doi.org/10.52202/079017-2338)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px4.p1.1)\.
- O\. Ledoit and M\. Wolf \(2004\)A well\-conditioned estimator for large\-dimensional covariance matrices\.Journal of Multivariate Analysis88\(2\),pp\. 365–411\.External Links:ISSN 0047\-259X,[Document](https://dx.doi.org/10.1016/S0047-259X%2803%2900096-4)Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p3.8)\.
- E\. L\. Lehmann and G\. Casella \(2005\)Theory of Point Estimation\.2 edition,Springer,New York, NY Berlin Heidelberg\.External Links:ISBN 978\-0\-387\-98502\-2Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p3.2)\.
- X\. Li, J\. Luo, Z\. Zheng, H\. Wang, L\. Luo, L\. Wen, L\. Wu, and S\. Xu \(2025\)On the Performance Analysis of Momentum Method: A Frequency Domain Perspective\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tznvtmSEiN)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px3.p1.1)\.
- L\. Ljung and T\. Söderström \(1983\)Theory and practice of recursive identification\.MIT Press\.Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p4.2)\.
- G\. S\. Lueker \(1980\)Some Techniques for Solving Recurrences\.ACM Comput\. Surv\.12\(4\),pp\. 419–436\.External Links:ISSN 0360\-0300,[Document](https://dx.doi.org/10.1145/356827.356832)Cited by:[§2\.2](https://arxiv.org/html/2608.04026#S2.SS2.p3.9)\.
- A\. V\. Oppenheim and R\. W\. Schafer \(2010\)Discrete\-Time Signal Processing\.3rd edition,Pearson,Upper Saddle River, NJ\.External Links:ISBN 978\-0\-13\-198842\-2Cited by:[§3\.4](https://arxiv.org/html/2608.04026#S3.SS4.p1.4)\.
- S\. J\. Orfanidis \(1995\)Introduction to Signal Processing\.2 edition,Prentice\-Hall, Inc\.,NJ, USA\.External Links:ISBN 978\-0\-13\-209172\-5Cited by:[§3\.2](https://arxiv.org/html/2608.04026#S3.SS2.p1.7)\.
- N\. Parikh and S\. Boyd \(2014\)Proximal Algorithms\.Foundations and Trends in Optimization1\(3\),pp\. 127–239\.External Links:ISSN 2167\-3888,[Document](https://dx.doi.org/10.1561/2400000003)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px2.p1.1)\.
- B\. T\. Polyak \(1969\)The conjugate gradient method in extremal problems\.USSR Computational Mathematics and Mathematical Physics9\(4\),pp\. 94–112\.External Links:ISSN 0041\-5553,[Document](https://dx.doi.org/10.1016/0041-5553%2869%2990035-4)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px3.p1.1)\.
- B\. T\. Polyak \(2020\)Accelerated gradient methods: history and properties\.In7th International Conference on Control and Optimization with Industrial Applications,Vol\.1,Baku, Azerbaijan,pp\. 23–25\.Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px3.p1.1)\.
- S\. N\. Ramachandran, M\. K\. Lal, and S\. Sra \(2025\)Cross\-fluctuation phase transitions reveal sampling dynamics in diffusion models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=b4X6cz1F9l)Cited by:[§2\.6](https://arxiv.org/html/2608.04026#S2.SS6.p2.24)\.
- J\. O\. Smith \(2007\)Pole Zero Analysis I\.InIntroduction to Digital Filters with Audio Applications,External Links:[Link](https://ccrma.stanford.edu/%5Ctextasciitilde%20jos/fp/Pole%5C_Zero%5C_Analysis%5C_I.html),ISBN 978\-0\-9745607\-1\-7Cited by:[§3\.4](https://arxiv.org/html/2608.04026#S3.SS4.p1.4)\.
- O\. A\. Somefun, S\. Lee, and V\. J\. Mathews \(2024\)AUTOSGM: A Unified Lowpass Regularization Framework for Accelerated Learning\.Proceedings of IEEE International Conference on Acoustics, SPeech and Signal Processing\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP48485.2024.10448203)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px3.p1.1)\.
- O\. Somefun \(2026\)Fundamental signal processing elements in accelerated learning\.Ph\.D\. Thesis,Oregon State University,Corvallis, OR\.External Links:[Link](https://ir.library.oregonstate.edu/downloads/76537923z)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px3.p1.1),[§2\.5](https://arxiv.org/html/2608.04026#S2.SS5.p5.2)\.
- I\. Sutskever, J\. Martens, G\. Dahl, and G\. Hinton \(2013\)On the importance of initialization and momentum in deep learning\.InProceedings of the 30th International Conference on Machine Learning,Atlanta, GA, USA,pp\. 1139–1147\.External Links:ISSN 1938\-7228,[Link](https://proceedings.mlr.press/v28/sutskever13.html)Cited by:[§1\.3](https://arxiv.org/html/2608.04026#S1.SS3.SSS0.Px3.p1.1)\.
- A\. M\. Zoubir, V\. Koivunen, E\. Ollila, and M\. Muma \(2018\)Robust Statistics for Signal Processing\.1 edition,Cambridge University Press\.External Links:[Document](https://dx.doi.org/10.1017/9781139084291)Cited by:[§5](https://arxiv.org/html/2608.04026#S5.p3.2)\.