Anytime Training with Schedule-Free Spectral Optimization

# Anytime Training with Schedule-Free Spectral Optimization
Source: [https://arxiv.org/html/2605.23061](https://arxiv.org/html/2605.23061)
Anuj Apte Pranav Deshpande Niraj Kumar Shouvanik Chakrabarti Junhyung Lyle Kim Global Technology Applied Research, JPMorganChase New York, NY 10001, USA

\(May 2026\)

###### Abstract

Standard neural network training relies on learning\-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re\-tuning as data availability changes\. Schedule\-Free \(SF\) methods address this by removing explicit schedules, yet SF\-AdamW, the current state\-of\-the\-art anytime optimizer, consistently underperforms well\-tuned AdamW baselines\. We propose SF\-NorMuon, a schedule\-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF\-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across11–8×8\\timesChinchilla horizons\. On the theoretical side, we prove a stationarity guarantee for schedule\-free spectral dynamics and identify weight decay at the fast iterate as essential for long\-horizon stability\. SF\-NorMuon enables practitioners to obtain high\-quality checkpoints at any point during training without committing to a horizon in advance\. By closing the performance gap with tuned baselines, SF\-NorMuon makes horizon\-free optimization more practical, taking a step towards truly open\-ended, continual learning\.

## 1Introduction

In the standard machine learning paradigm, after curating a dataset, a model is trained and then evaluated on a held\-out, previously unseen portion of the data\[[28](https://arxiv.org/html/2605.23061#bib.bib17),[23](https://arxiv.org/html/2605.23061#bib.bib16),[7](https://arxiv.org/html/2605.23061#bib.bib18)\]\. However, models are increasingly being deployed in scenarios where large amounts of data comparable to or even exceeding the training data are generated during the deployment period\. Examples of this include large language models \(LLMs\)\[[9](https://arxiv.org/html/2605.23061#bib.bib21),[75](https://arxiv.org/html/2605.23061#bib.bib22),[82](https://arxiv.org/html/2605.23061#bib.bib20),[56](https://arxiv.org/html/2605.23061#bib.bib19)\]interacting with billions of users, and Vision\-Language\-Action \(VLA\) models controlling robots in industries\[[8](https://arxiv.org/html/2605.23061#bib.bib23),[54](https://arxiv.org/html/2605.23061#bib.bib24),[40](https://arxiv.org/html/2605.23061#bib.bib25)\]\. As a result, judiciously leveraging the a priori unknown amount of data available at deployment time is crucial for improving model capabilities through continual learning\[[58](https://arxiv.org/html/2605.23061#bib.bib26),[78](https://arxiv.org/html/2605.23061#bib.bib27)\]\. However, beyond addressing catastrophic forgetting\[[42](https://arxiv.org/html/2605.23061#bib.bib28),[19](https://arxiv.org/html/2605.23061#bib.bib29)\], there is also an optimization obstacle to overcome\[[18](https://arxiv.org/html/2605.23061#bib.bib7),[55](https://arxiv.org/html/2605.23061#bib.bib8),[16](https://arxiv.org/html/2605.23061#bib.bib9),[37](https://arxiv.org/html/2605.23061#bib.bib5)\]\.

Most existing training methods are not designed to be anytime, as they depend on learning rate schedules tied to a fixed training horizon and require extensive hyperparameter tuning under a predetermined compute budget\[[52](https://arxiv.org/html/2605.23061#bib.bib30),[29](https://arxiv.org/html/2605.23061#bib.bib32),[30](https://arxiv.org/html/2605.23061#bib.bib31)\]\. For example, the standard cosine decay schedule has a strong path dependence\. Consider training an LLaMA\-2\-style transformer with cosine decay as shown in[fig\.˜1](https://arxiv.org/html/2605.23061#S1.F1)\. The black dashed lines highlight the performance discrepancy: at the same token count, a run tuned for a shorter horizon significantly outperforms one tuned for a longer horizon, despite seeing identical data \(e\.g\.: 2×\\timesvs 4×\\timesChinchilla at 31B tokens for 772M model\)\. This occurs because cosine decay ties the learning rate to the training horizon\. At any intermediate point, runs with longer horizons have barely begun decaying, while shorter\-horizon runs have already annealed to near zero, resulting in the big discrepancy in performances\. Ideally, we want the model to learn as much as possible from a given set of training data regardless of how much data awaits in the future\.

![Refer to caption](https://arxiv.org/html/2605.23061v1/x1.png)Figure 1:Comparison ofSF\-NorMuon\(this work\),SF\-AdamW, andtunedAdamWbaselines for LLaMA\-2\-style transformers trained on FineWeb\-100B\.Left:125M model with AdamW learning rate tuned per horizon with cosine schedule\.Right:772M model with a single optimized AdamW configuration across horizons\. Dashed lines highlight that learning rate schedule for a long horizon is sub\-optimal for a smaller token budget\. The grid of hyperparameters are summarized in[table˜4](https://arxiv.org/html/2605.23061#A4.T4), and validation losses for different learning rates are illustrated in[fig\.˜6](https://arxiv.org/html/2605.23061#A4.F6)\.SF\-NorMuonconsistently out performsSF\-AdamW, and matchestunedAdamWbaselines\.To that end,Schedule\-Free\(SF\) methods\[[16](https://arxiv.org/html/2605.23061#bib.bib9)\]were recently proposed, removing the need for an explicit learning\-rate schedule\. The SF framework maintains three iterate sequences\. In its most basic form, SF\-SGD iterates as follows:

yt\\displaystyle y\_\{t\}=β​xt\+\(1−β\)​zt,\\displaystyle=\\beta x\_\{t\}\+\(1\-\\beta\)z\_\{t\},\(1\)zt\+1\\displaystyle z\_\{t\+1\}=zt−η​∇ℒ​\(yt;ξt\),\\displaystyle=z\_\{t\}\-\\eta\\nabla\\mathcal\{L\}\(y\_\{t\};\\xi\_\{t\}\),\(2\)xt\+1\\displaystyle x\_\{t\+1\}=\(1−ct\+1\)​xt\+ct\+1​zt\+1,\\displaystyle=\(1\-c\_\{t\+1\}\)x\_\{t\}\+c\_\{t\+1\}z\_\{t\+1\},\(3\)wherez1=x1z\_\{1\}=x\_\{1\}andct\+1=1/\(t\+1\)c\_\{t\+1\}=1/\(t\+1\)\. Theyysequence is where the gradient is evaluated at interpolated points, in spirit of Nesterov’s accelerated method\[[57](https://arxiv.org/html/2605.23061#bib.bib2),[45](https://arxiv.org/html/2605.23061#bib.bib1)\], thezzsequence is thefast iteratetaking the \(stochastic\) gradient direction with a constant step sizeη\\etathroughout the training, and thexxsequence is theevaluationsequence where validation loss is computed\. Rewriting the update, we obtain

xt\+1=xt−η​ct\+1​∇ℒ​\(yt;ξt\)\+ct\+1​\(zt\+1−xt\)\.x\_\{t\+1\}=x\_\{t\}\-\\eta c\_\{t\+1\}\\nabla\\mathcal\{L\}\(y\_\{t\};\\xi\_\{t\}\)\+c\_\{t\+1\}\(z\_\{t\+1\}\-x\_\{t\}\)\.\(4\)Thus, even thoughzzmoves at a constant rate, the effective learning rate forxxisηeff=η/\(t\+1\)\\eta\_\{\\textrm\{eff\}\}=\\eta/\(t\+1\), which decays over the duration of the training\. Note that sincexxis a linear combination ofyyandxxit can be computed on the fly, and so it does not need additional memory overhead\. An alternative but closely related approach for horizon\-free training is simply training with a constant learning rate, and returning an some moving average111For instance, withβ=0\\beta=0in \([1](https://arxiv.org/html/2605.23061#S1.E1)\), SF\-SGD recovers constant SGD with uniform averaging\.of the weights\[[27](https://arxiv.org/html/2605.23061#bib.bib35),[48](https://arxiv.org/html/2605.23061#bib.bib33),[55](https://arxiv.org/html/2605.23061#bib.bib8),[31](https://arxiv.org/html/2605.23061#bib.bib6)\]\. Notably, SF\-AdamW, the practical variant of SF\-SGD, won the first prize in the self\-tuning track of theAlgoPerfchallenge\[[38](https://arxiv.org/html/2605.23061#bib.bib34)\], demonstrating a new level of effectiveness for anytime training\. For completeness,[appendix˜B](https://arxiv.org/html/2605.23061#A2)reviews SF\-SGD and SF\-AdamW, and summarizes the convergence guarantee for SF\-SGD\.

Despite this success, as observed in\[[26](https://arxiv.org/html/2605.23061#bib.bib3),[72](https://arxiv.org/html/2605.23061#bib.bib37),[67](https://arxiv.org/html/2605.23061#bib.bib36)\], SF\-AdamW is unable to match the performance of well\-tuned AdamW\[[41](https://arxiv.org/html/2605.23061#bib.bib38),[53](https://arxiv.org/html/2605.23061#bib.bib39)\], the primary workhorse for optimizing neural networks\. This is also confirmed in[fig\.˜1](https://arxiv.org/html/2605.23061#S1.F1), where SF\-AdamW \(blue\) consistently lies above the well\-tuned AdamW baselines \(red stars, tuning grid in[table˜4](https://arxiv.org/html/2605.23061#A4.T4)\) across all Chinchilla horizons\. Since the schedule\-free averaging mechanism in \([1](https://arxiv.org/html/2605.23061#S1.E1)\)–\([3](https://arxiv.org/html/2605.23061#S1.E3)\) is agnostic to the base update rule applied at \([2](https://arxiv.org/html/2605.23061#S1.E2)\), a natural question is whether a different choice of base optimizer can close this gap\.

In modern neural networks, the vast majority of learnable parameters reside in weight matrices that act linearly on incoming activations\. However, AdamW treats weight matrices as flat vectors, discarding how they act on activations\. The spectral norm provides a natural measure of the effect a weight matrix update has on the output activation, motivating recent interest in spectral optimization methods such as Muon\[[34](https://arxiv.org/html/2605.23061#bib.bib46),[33](https://arxiv.org/html/2605.23061#bib.bib47)\]and its variants\[[70](https://arxiv.org/html/2605.23061#bib.bib59),[49](https://arxiv.org/html/2605.23061#bib.bib58),[2](https://arxiv.org/html/2605.23061#bib.bib48),[1](https://arxiv.org/html/2605.23061#bib.bib80),[4](https://arxiv.org/html/2605.23061#bib.bib64),[63](https://arxiv.org/html/2605.23061#bib.bib81),[3](https://arxiv.org/html/2605.23061#bib.bib82),[51](https://arxiv.org/html/2605.23061#bib.bib83),[39](https://arxiv.org/html/2605.23061#bib.bib84),[43](https://arxiv.org/html/2605.23061#bib.bib88)\]\. Muon performs steepest descent under the spectral norm via the polar decomposition of the gradient computed by Newton\-Schulz iterations, and has been successfully scaled to trillions of tokens\[[50](https://arxiv.org/html/2605.23061#bib.bib42),[74](https://arxiv.org/html/2605.23061#bib.bib43),[22](https://arxiv.org/html/2605.23061#bib.bib45),[71](https://arxiv.org/html/2605.23061#bib.bib44)\]\. Neural network loss surfaces are characterized by many nearly\-flat directions corresponding to small Hessian eigenvalues\[[64](https://arxiv.org/html/2605.23061#bib.bib52),[21](https://arxiv.org/html/2605.23061#bib.bib53)\], and thus taking uniformly sized steps in all spectral directions via the polar decomposition enables Muon to extract more information from the training data per step\[[14](https://arxiv.org/html/2605.23061#bib.bib51)\]\. This motivates the question:

Does optimization under the spectral norm improve upon existing schedule\-free methods, and can it close the gap with well\-tuned, horizon\-dependent AdamW baselines?

Apart from the choice of base geometry, schedule\-free methods exhibit a stability challenge that becomes critical for long horizon training\. As shown in[fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2), the validation loss increases for long runs in absence of weight decay\. This is in sharp contrast to the convex optimization theory underlying schedule\-free methods, where convergence holds without regularization\[[16](https://arxiv.org/html/2605.23061#bib.bib9),[66](https://arxiv.org/html/2605.23061#bib.bib4)\]\. Moreover, weight decay atyyas originally proposed in\[[16](https://arxiv.org/html/2605.23061#bib.bib9)\], is inadequate for the spectral case, where the instability is significantly amplified \(right panel of[fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2)\)\. Therefore, in schedule\-free methods weight decay plays a qualitatively different role: it is not merely a regularizer that improves generalization\[[44](https://arxiv.org/html/2605.23061#bib.bib63),[13](https://arxiv.org/html/2605.23061#bib.bib40),[61](https://arxiv.org/html/2605.23061#bib.bib41)\], but is*necessary*for long\-horizon stability\.

Combining these two insights, spectral geometry to speed up training and weight decay at the fast iterate to ensure stability, our contributions are as follows:

- •We propose SF\-SpectralSGD \(with momentum\), a schedule\-free spectral method for matrix optimization, and prove a stationarity guarantee matching theO~​\(T−1/4\)\\widetilde\{O\}\(T^\{\-1/4\}\)rate achieved by recent work on convergence of Muon\[[10](https://arxiv.org/html/2605.23061#bib.bib54),[69](https://arxiv.org/html/2605.23061#bib.bib55),[65](https://arxiv.org/html/2605.23061#bib.bib56),[47](https://arxiv.org/html/2605.23061#bib.bib57)\]\([theorem˜2\.1](https://arxiv.org/html/2605.23061#S2.Thmtheorem1)and[corollary˜2\.1](https://arxiv.org/html/2605.23061#S2.Thmcorollary1)\)\.
- •For practical deep learning, we incorporate the per\-neuron normalization of NorMuon\[[49](https://arxiv.org/html/2605.23061#bib.bib58),[70](https://arxiv.org/html/2605.23061#bib.bib59)\]and obtain SF\-NorMuon, as summarized in[algorithm˜1](https://arxiv.org/html/2605.23061#alg1)\. SF\-NorMuon substantially improves over the state\-of\-the\-art anytime optimizer SF\-AdamW, and matches tuned AdamW across training horizons \(c\.f\.,[fig\.˜1](https://arxiv.org/html/2605.23061#S1.F1),[table˜1](https://arxiv.org/html/2605.23061#S4.T1)\)\. In particular, including explicit momentum buffer and per\-neuron normalization is crucial in achieving good practical performance \(c\.f\.,[fig\.˜3](https://arxiv.org/html/2605.23061#S2.F3)\)\.
- •We identify the key role of weight decay in schedule\-free optimization\. Unlike in standard optimizers, the fast iterateztz\_\{t\}in \([2](https://arxiv.org/html/2605.23061#S1.E2)\) continues to move at constant learning rate throughout training, so weight decay is a necessity for long\-horizon stability\. For spectral optimization, the aggressive polar updates amplify the instability \([fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2)\), but weight decay directly applied toztz\_\{t\}leads to stable training\. We prove boundedness of all iterates \([lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1)\), and derive a closed\-form steady\-state characterization \([lemma˜3\.2](https://arxiv.org/html/2605.23061#S3.Thmlemma2)\) that closely matches empirical training dynamics \([fig\.˜4](https://arxiv.org/html/2605.23061#S3.F4)\)\.

## 2Schedule\-Free Spectral Optimization

![Refer to caption](https://arxiv.org/html/2605.23061v1/x2.png)Figure 2:Weight decay strategies for schedule\-free optimizers\.Left:SF\-AdamW with no decay \(orange\) diverges; decay atYY\(blue\) exhibits the best performance up to∼\\sim30B tokens \(c\.f\.,[table˜1](https://arxiv.org/html/2605.23061#S4.T1)\), but eventually diverges\. Decay atZZ\(green\) yield stable training, but is suboptimal in the early phase\.Right:SF\-NorMuon with no decay \(orange\) diverges faster; decay atYY\(blue\) is also unstable\. Decay atZZ\(green\) maintains stable and performs the best throughout, motivating the analysis in[section˜3](https://arxiv.org/html/2605.23061#S3)\.The fundamental design choice in a schedule\-free optimizer is the geometry used for the fast update\. When inputs and outputs are measured in the Euclidean norm∥⋅∥2\\\|\\cdot\\\|\_\{2\}, the induced operator norm on a matrixA∈ℝm×nA\\in\\mathbb\{R\}^\{m\\times n\}is

‖A‖2→2:=sup‖u‖2⩽1‖A​u‖2=σmax​\(A\)=‖A‖op\.\\\|A\\\|\_\{2\\to 2\}\\;:=\\;\\sup\_\{\\\|u\\\|\_\{2\}\\leqslant 1\}\\\|Au\\\|\_\{2\}\\;=\\;\\sigma\_\{\\max\}\(A\)\\;=\\;\\\|A\\\|\_\{\\mathrm\{op\}\}\.\(5\)whereσmax​\(A\)\\sigma\_\{\\max\}\(A\)denotes the largest singular value ofAA, which is precisely the spectral norm\. Geometrically, it measures the maximum factor by whichAAcan stretch any unit vector, making it the natural measure of how much a weight perturbation can alter the network’s activations\. The analysis for other induced norms, and their corresponding optimizers is presented in[appendix˜A](https://arxiv.org/html/2605.23061#A1)\.

For matrix\-valued weights acting as linear maps on activations, the spectral norm is the natural metric, since it equals the operator norm governing the largest change an update induces on activations\. The steepest\-descent direction under a spectral\-norm constraint solves\[[6](https://arxiv.org/html/2605.23061#bib.bib49),[60](https://arxiv.org/html/2605.23061#bib.bib76)\]

argmin‖Δ​W‖op⩽η​⟨∇f​\(W\),Δ​W⟩F\.\\textrm\{argmin\}\_\{\\\|\\Delta W\\\|\_\{\\mathrm\{op\}\}\\leqslant\\eta\}\\langle\\nabla f\(W\),\\Delta W\\rangle\_\{F\}\.\(6\)Let∇f​\(W\)=U​Σ​V⊤\\nabla f\(W\)=U\\Sigma V^\{\\top\}be the SVD withU∈ℝm×rU\\in\\mathbb\{R\}^\{m\\times r\},V∈ℝn×rV\\in\\mathbb\{R\}^\{n\\times r\}orthonormal andΣ=diag⁡\(σ1,…,σr\)\\Sigma=\\operatorname\{diag\}\(\\sigma\_\{1\},\\ldots,\\sigma\_\{r\}\)\. DecomposingΔ​W=U​Q​V⊤\+R\\Delta W=UQV^\{\\top\}\+RwhereRRis orthogonal to the column/row space of the gradient, onlyQQcontributes:

⟨∇f​\(W\),Δ​W⟩F=tr​\(Σ⊤​Q\)=∑i=1rσi​Qi​i\.\\langle\\nabla f\(W\),\\Delta W\\rangle\_\{F\}=\\mathrm\{tr\}\(\\Sigma^\{\\top\}Q\)=\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}Q\_\{ii\}\.\(7\)Since‖Q‖op⩽η\\\|Q\\\|\_\{\\mathrm\{op\}\}\\leqslant\\etaimplies\|Qi​i\|⩽η\|Q\_\{ii\}\|\\leqslant\\eta, the minimum is achieved atQ⋆=−η​IrQ^\{\\star\}=\-\\eta I\_\{r\}\. Thus, the polar factor is the steepest\-descent direction under spectral norm:

Δ​W⋆=−η​U​V⊤=−η​polar​\(∇f​\(W\)\)\.\\Delta W^\{\\star\}=\-\\eta\\,UV^\{\\top\}=\-\\eta\\,\\mathrm\{polar\}\(\\nabla f\(W\)\)\.\(8\)
Unlike Adam\[[41](https://arxiv.org/html/2605.23061#bib.bib38)\]or Lion\[[11](https://arxiv.org/html/2605.23061#bib.bib60)\]which act entry\-wise and distort the gradient’s singular\-vector structure, the polar factorU​V⊤UV^\{\\top\}preserves the left and right singular subspaces\. The polar update extracts learning signal from all singular directions of the gradient and not just those with the largest entries, and by making all the singular values unity, it makes uniform progress along all singular directions\. This uniform treatment is more aggressive than coordinate\-wise updates, so we include an explicit momentum bufferMtM\_\{t\}to smooth the gradient before taking its polar factor\. We build on this spectral\-norm geometry to design a schedule\-free optimizer that respects how weight matrices act on activations, while obtaining implicit learning\-rate decay through online averaging\.

### 2\.1Schedule\-Free Spectral Descent with Momentum

For parametersβ,μ∈\[0,1\)\\beta,\\mu\\in\[0,1\)and step\-sizeη\>0\\eta\>0, update rules for SF\-Spectral Descent with Momentum are given by

Yt\\displaystyle Y\_\{t\}=β​Xt\+\(1−β\)​Zt,\\displaystyle=\\beta X\_\{t\}\+\(1\-\\beta\)Z\_\{t\},Gt\\displaystyle G\_\{t\}=∇f​\(Yt;ξt\),\\displaystyle=\\nabla f\(Y\_\{t\};\\xi\_\{t\}\),Mt\\displaystyle M\_\{t\}=μ​Mt−1\+\(1−μ\)​Gt,\\displaystyle=\\mu M\_\{t\-1\}\+\(1\-\\mu\)G\_\{t\},Pt\\displaystyle P\_\{t\}=polar​\(Mt\),\\displaystyle=\\mathrm\{polar\}\(M\_\{t\}\),Zt\+1\\displaystyle Z\_\{t\+1\}=Zt−η​Pt,\\displaystyle=Z\_\{t\}\-\\eta P\_\{t\},Xt\+1\\displaystyle X\_\{t\+1\}=\(1−ct\+1\)​Xt\+ct\+1​Zt\+1,\\displaystyle=\(1\-c\_\{t\+1\}\)X\_\{t\}\+c\_\{t\+1\}Z\_\{t\+1\},\(9\)withct\+1=1/\(t\+1\)c\_\{t\+1\}=1/\(t\+1\),X1=Z1X\_\{1\}=Z\_\{1\}andM1=G1M\_\{1\}=G\_\{1\}\. HereZtZ\_\{t\}is the fast sequence with polar update,XtX\_\{t\}is the returned schedule\-free average,YtY\_\{t\}is the gradient\-evaluation point, andMtM\_\{t\}is an explicit momentum buffer, which is crucial in practical performance \(c\.f\.,[fig\.˜3](https://arxiv.org/html/2605.23061#S2.F3)\)\.

Assume the existence ofW⋆∈ℝm×nW^\{\\star\}\\in\\mathbb\{R\}^\{m\\times n\}such thatf​\(W⋆\)=f⋆f\(W^\{\\star\}\)=f^\{\\star\}and∇f​\(W⋆\)=0\\nabla f\(W^\{\\star\}\)=0, and define

r:=min⁡\{m,n\},Δ:=f​\(Y1\)−f⋆\.r:=\\min\\\{m,n\\\},\\qquad\\Delta:=f\(Y\_\{1\}\)\-f^\{\\star\}\.\(10\)
Our convergence analysis relies on the following standard assumptions:

###### Assumption 2\.1\(Frobenius Lipschitz smoothness\)\.

There is a constantLF\>0L\_\{F\}\>0such that for allU,V∈ℝm×nU,V\\in\\mathbb\{R\}^\{m\\times n\},

‖∇f​\(U\)−∇f​\(V\)‖F⩽LF​‖U−V‖F\.\\left\\lVert\\nabla f\(U\)\-\\nabla f\(V\)\\right\\rVert\_\{F\}\\leqslant L\_\{F\}\\left\\lVert U\-V\\right\\rVert\_\{F\}\.\(11\)

###### Assumption 2\.2\(Unbiased gradient and bounded\-variance noise\)\.

Letℱt\\mathcal\{F\}\_\{t\}be the sigma\-field generated by all randomness up to the construction ofYtY\_\{t\}\. We assume, for some batch sizeB⩾1B\\geqslant 1andσ2⩾0\\sigma^\{2\}\\geqslant 0,

𝔼​\[Gt∣ℱt\]=∇f​\(Yt\),𝔼​\[‖Gt−∇f​\(Yt\)‖F2∣ℱt\]⩽σ2/B\.\\mathbb\{E\}\\\!\\bigl\[G\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\\bigr\]=\\nabla f\(Y\_\{t\}\),\\qquad\\mathbb\{E\}\\\!\\left\[\\left\\lVert G\_\{t\}\-\\nabla f\(Y\_\{t\}\)\\right\\rVert\_\{F\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leqslant\\sigma^\{2\}/B\.\(12\)

###### Assumption 2\.3\(Bounded diameter\)\.

There exists a constantDF\>0D\_\{F\}\>0such that

‖W⋆‖F⩽DF/2,‖Zt‖F⩽DF/2for all​t\.\\\|W^\{\\star\}\\\|\_\{F\}\\leqslant D\_\{F\}/2,\\qquad\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant D\_\{F\}/2\\qquad\\text\{for all \}t\.\(13\)

Note that bounded iterate assumption is typically employed in the convergence analysis of spectral methods \(e\.g\., see\[[25](https://arxiv.org/html/2605.23061#bib.bib50), Theorem 7\]and\[[4](https://arxiv.org/html/2605.23061#bib.bib64), Theorem 1\]\), and if necessary it can be enforced algorithmically by considering a projected variant that keep the fast iteratesZtZ\_\{t\}inside the corresponding norm ball\. Further, with weight decay atZZ,[˜2\.3](https://arxiv.org/html/2605.23061#S2.Thmassumption3)can be lifted, as can be seen in[lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1)\.

Under these assumptions, we obtain the following convergence guarantee:

###### Theorem 2\.1\(Stationarity of SF\-Spectral Descent with Momentum\)\.

Under[˜2\.1](https://arxiv.org/html/2605.23061#S2.Thmassumption1)–[2\.3](https://arxiv.org/html/2605.23061#S2.Thmassumption3), for everyT⩾1T\\geqslant 1, SF\-Sepctral Descent with Momentum in \([9](https://arxiv.org/html/2605.23061#S2.E9)\) satisfies the following:

1T​∑t=1T𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}Δ\+2​LF​DF2​\(log⁡\(e​T\)\+π2/6\)\(1−β\)​T​η\+2​r​σ\(1−β\)​B​\(1−μ1\+μ\+1\(1−μ\)​T\)\\displaystyle\\frac\{\\Delta\+2L\_\{F\}D\_\{F\}^\{2\}\\\!\\left\(\\log\(eT\)\+\\pi^\{2\}/6\\right\)\}\{\(1\-\\beta\)T\\eta\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\beta\)\\sqrt\{B\}\}\\left\(\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\+\\frac\{1\}\{\(1\-\\mu\)T\}\\right\)\+2​r​LF​DF​log⁡\(e​T\)\(1−β\)​T​\(1\+μ1−μ\)\+LF​r​η\(1−β\)​\(1\+3​μ1−μ\)\.\\displaystyle\\quad\+\\frac\{2\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{\(1\-\\beta\)T\}\\left\(\\frac\{1\+\\mu\}\{1\-\\mu\}\\right\)\+\\frac\{L\_\{F\}r\\,\\eta\}\{\(1\-\\beta\)\}\\left\(\\frac\{1\+3\\mu\}\{1\-\\mu\}\\right\)\.\(14\)

The first term is the descent term, the next captures the stochastic noise, the third term is the price of schedule\-free drift, and the final term reflects the constant spectral step and momentum tracking\. The explicit momentum stabilizes the aggressive polar steps at the cost of a tracking error\. By optimizing the terms that appear on the right hand side of the theorem above, we obtain the following corollary for anyβ∈\[0,1\)\\beta\\in\[0,1\), both with and without noise\.

###### Corollary 2\.1\(Optimized convergence rates for SF\-Spectral Descent\)\.

Under[˜2\.1](https://arxiv.org/html/2605.23061#S2.Thmassumption1)–[2\.3](https://arxiv.org/html/2605.23061#S2.Thmassumption3), for any fixedβ∈\[0,1\)\\beta\\in\[0,1\)there exist choicesμT∈\[0,1\)\\mu\_\{T\}\\in\[0,1\)andηT\>0\\eta\_\{T\}\>0such that

1T​∑t=1T𝔼​‖∇f​\(Yt\)‖∗=O~​\(T−1/4\)​\(σ\>0\);and​1T​∑t=1T‖∇f​\(Yt\)‖∗=O~​\(T−1/2\)​\(σ=0\)\.\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=\\widetilde\{O\}\(T^\{\-1/4\}\)~~\(\\sigma\>0\);~~\\text\{and\}~~\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=\\widetilde\{O\}\(T^\{\-1/2\}\)~~\(\\sigma=0\)\.\(15\)

These convergence rates match the ones derived in recent works on convergence of Muon\[[10](https://arxiv.org/html/2605.23061#bib.bib54),[69](https://arxiv.org/html/2605.23061#bib.bib55),[65](https://arxiv.org/html/2605.23061#bib.bib56),[47](https://arxiv.org/html/2605.23061#bib.bib57)\]\. An analogous result can also be proven under Lipschitz smoothness measured in the spectral norm rather than the Frobenius norm\. Since the Frobenius\-smooth version is more directly comparable to existing matrix\-optimization methods, we state it here and defer the spectral\-smooth variant, together with the full proofs of both results, to the[appendix˜C](https://arxiv.org/html/2605.23061#A3)\.

### 2\.2Schedule\-Free NorMuon for Neural Network Training

Algorithm 1Schedule\-Free NorMuon \(SF\-NorMuon\)1:Input:base learning rate

η\\eta, interpolation parameter

β1\\beta\_\{1\}, variance parameter

β2\\beta\_\{2\}, momentum parameter

μ\\mu, perturbation parameter

ε\\varepsilon, weight decay

λ\\lambda, warmup steps

TwarmupT\_\{\\text\{warmup\}\}
2:for

t=1t=1to

TTdo

3:

Yt←\(1−β1\)​Zt\+β1​XtY\_\{t\}\\leftarrow\(1\-\\beta\_\{1\}\)Z\_\{t\}\+\\beta\_\{1\}X\_\{t\}⊳\\trianglerightInterpolate weights

4:

Gt←∇𝐖ℒ​\(Yt,ζt\)G\_\{t\}\\leftarrow\\nabla\_\{\\mathbf\{W\}\}\\mathcal\{L\}\(Y\_\{t\},\\zeta\_\{t\}\)⊳\\trianglerightCompute gradient at interpolation point

5:

Mt←μ​Mt−1\+\(1−μ\)​GtM\_\{t\}\\leftarrow\\mu M\_\{t\-1\}\+\(1\-\\mu\)G\_\{t\}⊳\\trianglerightExplicit momentum buffer

6:

7:

Pt←polar​\(Mt\)P\_\{t\}\\leftarrow\\text\{polar\}\(M\_\{t\}\)⊳\\trianglerightPolar factor of momentum

8:

vt←β2​vt−1\+\(1−β2\)​meancols⁡\(Pt⊙Pt\)v\_\{t\}\\leftarrow\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\operatorname\{mean\}\_\{\\text\{cols\}\}\(P\_\{t\}\\odot P\_\{t\}\)⊳\\trianglerightRow\-wise second moment

9:

Vt←ExpandRows​\(vt\)V\_\{t\}\\leftarrow\\mathrm\{ExpandRows\}\(v\_\{t\}\)⊳\\trianglerightBroadcast to matrix

10:

P^t←Pt⊘\(Vt\+ε\)\\widehat\{P\}\_\{t\}\\leftarrow P\_\{t\}\\oslash\(\\sqrt\{V\_\{t\}\}\+\\varepsilon\)⊳\\trianglerightAdaptive row\-wise normalization

11:

12:

ηt←η​min⁡\(1,t/Twarmup\)\\eta\_\{t\}\\leftarrow\\eta\\min\(1,t/T\_\{\\text\{warmup\}\}\)⊳\\trianglerightLearning rate with warmup

13:

η^t←0\.2​ηt​m​n/‖P^t‖F\\hat\{\\eta\}\_\{t\}\\leftarrow 0\.2\\eta\_\{t\}\\sqrt\{mn\}/\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}⊳\\trianglerightMatch Adam RMS scaling

14:

st←st−1\+ηt2s\_\{t\}\\leftarrow s\_\{t\-1\}\+\\eta\_\{t\}^\{2\}
15:

ct\+1←ηt2/stc\_\{t\+1\}\\leftarrow\\eta\_\{t\}^\{2\}/s\_\{t\}⊳\\trianglerightAveraging coefficient

16:

17:

Zt\+1←Zt−η​λ​Zt−η^t​P^tZ\_\{t\+1\}\\leftarrow Z\_\{t\}\-\\eta\\lambda Z\_\{t\}\-\\hat\{\\eta\}\_\{t\}\\widehat\{P\}\_\{t\}⊳\\trianglerightUpdate with weight decay atZtZ\_\{t\}

18:

Xt\+1←\(1−ct\+1\)​Xt\+ct\+1​Zt\+1X\_\{t\+1\}\\leftarrow\(1\-c\_\{t\+1\}\)X\_\{t\}\+c\_\{t\+1\}Z\_\{t\+1\}⊳\\trianglerightUpdate averaged iterate

19:endfor

20:Return:

XTX\_\{T\}

To make schedule\-free spectral descent practical for neural network training, we incorporate three refinements below\. The resulting algorithm is SF\-NorMuon, which we summarize in[algorithm˜1](https://arxiv.org/html/2605.23061#alg1)\. A PyTorch implementation of this algorithm is provided in[appendix˜F](https://arxiv.org/html/2605.23061#A6)\.

Learning rate warm up and scaling\.We warm up the learning rate linearly, which stabilizes early training\[[24](https://arxiv.org/html/2605.23061#bib.bib61)\]and enables larger learning rates\[[35](https://arxiv.org/html/2605.23061#bib.bib62)\]\. After computing the adaptively normalized directionP^t\\widehat\{P\}\_\{t\}, the algorithm scales the base learning rate byη^t=0\.2​ηt​max⁡\(m,n\)/‖P^t‖F\\hat\{\\eta\}\_\{t\}=0\.2\\eta\_\{t\}\\sqrt\{\\max\(m,n\)\}/\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}\. This scaling serves two purposes\[[50](https://arxiv.org/html/2605.23061#bib.bib42)\]: \(i\) It normalizes the effective step size to be comparable to Adam’s RMS\-normalized updates, enablingη\\etaandλ\\lambdato be shared with SF\-AdamW; and \(ii\) it provides additional stability by accounting for the overall magnitude of the normalized update direction\.

Weight decay at the fast iterateZZ\.Weight decay is applied toZtZ\_\{t\}, rather thanYtY\_\{t\}\(as originally proposed in\[[15](https://arxiv.org/html/2605.23061#bib.bib12)\]for SF\-AdamW\)\. This choice treats weight decay as a modification to the base optimizer dynamics rather than as anℓ2\\ell\_\{2\}regularizer in the loss, which would correspond to decay atYtY\_\{t\}\. The scaled weight decay termηt​λ​Zt\\eta\_\{t\}\\lambda Z\_\{t\}is subtracted fromZtZ\_\{t\}in the base update\. Weight decay atZtZ\_\{t\}is necessary for long\-horizon training with schedule\-free methods, as we explain in[section˜3](https://arxiv.org/html/2605.23061#S3)\.

Row\-wise Normalization\.We incorporate row\-wise adaptive normalization\[[49](https://arxiv.org/html/2605.23061#bib.bib58),[70](https://arxiv.org/html/2605.23061#bib.bib59)\]\. The polar factor can produce orthogonalized updates with high variance in step sizes across individual neurons, as different rows of a weight matrix may have vastly different update magnitudes\. To accomodate, SF\-NorMuon maintains a running averagevt∈ℝmv\_\{t\}\\in\\mathbb\{R\}^\{m\}of the squared row norms of the polar factorPtP\_\{t\}, capturing the typical magnitude of updates for each neuron and enabling per\-neuron step size adaptation\. The exponential moving average with parameterβ2\\beta\_\{2\}provides stability while remaining responsive to changes in gradient statistics, analogous to the second\-moment estimation in Adam, but operating at the neuron level rather than element\-wise\.

![Refer to caption](https://arxiv.org/html/2605.23061v1/x3.png)Figure 3:Importance of explicit momentum and row\-wise normalization for SF\-NorMuon\.Left:Ablating explicit momentum \(μ=0\\mu=0\) significantly degrades performance, with final loss increasing from 3\.14 to 3\.28\. This validates the importance of smoothing the gradient before computing the polar factor, as discussed in[section˜2](https://arxiv.org/html/2605.23061#S2)\.Right:Ablating row\-wise normalization leads to a smaller gap, with SF\-NorMuon reaching the same loss approximately 12% faster\.In[fig\.˜3](https://arxiv.org/html/2605.23061#S2.F3), we ablate the two key aspects of SF\-NorMuon: \(i\) the explicit momentum buffer \(by settingμ=0\\mu=0\), which reduces to direct polar decomposition of the gradient, and \(ii\) the row\-wise adaptive normalization \(by removing thevtv\_\{t\}computation and normalization step\), which reduces to uniform spectral steps\. These ablations help isolate the contribution of each component to the overall performance\. We find that removing the row\-wise adaptive normalization slows down convergence by around12%12\\%at 8x Chinchilla ratio, but removing momentum leads to a significant drop in performance\. Note that the original SF\-SGD or SF\-AdamW does not have an explicit momentum buffer\. Therefore, the above ablation study illustrates how direct application of schedule\-free dynamics to spectral optimzer \(e\.g\., replacing∇ℒ​\(yt;ξt\)\\nabla\\mathcal\{L\}\(y\_\{t\};\\xi\_\{t\}\)in \([2](https://arxiv.org/html/2605.23061#S1.E2)\) with polar\(∇ℒ​\(yt;ξt\)\)\(\\nabla\\mathcal\{L\}\(y\_\{t\};\\xi\_\{t\}\)\)\) would perform\.

## 3Necessity of Weight Decay in Long Horizon Training

Modern language models are trained for very long horizons\[[68](https://arxiv.org/html/2605.23061#bib.bib85),[46](https://arxiv.org/html/2605.23061#bib.bib86),[20](https://arxiv.org/html/2605.23061#bib.bib87)\], and the training tokens for Mixture\-of\-Experts models can exceed 30 times the Chinchilla ratio to the number of active parameters\[[22](https://arxiv.org/html/2605.23061#bib.bib45),[71](https://arxiv.org/html/2605.23061#bib.bib44),[74](https://arxiv.org/html/2605.23061#bib.bib43)\]\. Thus, to assess long\-horizon behavior of schedule\-free methods, we train the 125M LLaMa\-2\-style model for up to 48×\\timesChinchilla with the default hyper\-parameters\.

Interestingly, for SF methods, the validation loss atXXdiverges without weight decayingspecifically atZZ\. This is in sharp contrast to convex optimization theory, where convergence holds without regularization\[[16](https://arxiv.org/html/2605.23061#bib.bib9),[66](https://arxiv.org/html/2605.23061#bib.bib4)\]\. In particular, for SF\-AdamW in[fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2)\(left panel\), decay atYYperforms the best up to around 30B tokens, but eventually leads to divergence\. Note that the original paper\[[16](https://arxiv.org/html/2605.23061#bib.bib9)\]recommended decaying atYY, which is sensible from the perspective ofℓ2\\ell\_\{2\}regularization\. In contrast, for SF\-NorMuon in[fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2)\(right panel\), decaying atZZis not only stable \(see also[lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1)\) but also maintains the best validation loss throughout the long training exceeding 50B tokens\.

To better understand this phenomenon, we consider a quasi steady\-state analysis, following\[[77](https://arxiv.org/html/2605.23061#bib.bib71),[17](https://arxiv.org/html/2605.23061#bib.bib70)\]\. Recall from \([4](https://arxiv.org/html/2605.23061#S1.E4)\) that the schedule\-free method can be rewritten as

Xt\+1=Xt−ct\+1​η​Ut\+ct\+1​\(Zt\+1−Xt\),X\_\{t\+1\}=X\_\{t\}\-c\_\{t\+1\}\\eta U\_\{t\}\+c\_\{t\+1\}\(Z\_\{t\+1\}\-X\_\{t\}\),\(16\)wherect\+1=1/\(t\+1\)c\_\{t\+1\}=1/\(t\+1\), andUtU\_\{t\}is either gradient divided by second moment buffer for SF\-AdamW or polar transform of momentum buffer for SF\-NorMuon, respectively\. As can be seen, the evaluation sequenceXtX\_\{t\}evolves with an*effective*learning rateηteff=ct\+1​η=η/\(t\+1\)\\eta\_\{t\}^\{\\mathrm\{eff\}\}=c\_\{t\+1\}\\eta=\\eta/\(t\+1\)that decays to zero ast→∞t\\to\\infty, which makes the divergence surprising\.

The culprit is in the fastZtZ\_\{t\}sequence: it evolves with constant learning rateη\\etathroughout training, and in fact, we observe in practice that validation loss atZtZ\_\{t\}often diverges\. SinceXtX\_\{t\}averages all past\{Zs\}s⩽t\\\{Z\_\{s\}\\\}\_\{s\\leqslant t\},ZtZ\_\{t\}eventually contaminates theXtX\_\{t\}sequence\.

For SF\-NorMuon, applying weight decay directly atZtZ\_\{t\}addresses this issue, while maintaining good performance\. To make this precise, we prove that weight decay atZtZ\_\{t\}ensures all iterates to remain bounded, and characterize the precise steady\-state norm to whichZtZ\_\{t\}converges\.

###### Lemma 3\.1\.

Consider the updateZt\+1←Zt−η​λ​Zt−η^t​P^tZ\_\{t\+1\}\\leftarrow Z\_\{t\}\-\\eta\\lambda Z\_\{t\}\-\\hat\{\\eta\}\_\{t\}\\widehat\{P\}\_\{t\}and assume0<η​λ<10<\\eta\\lambda<1withη^=0\.2​η​m​n/‖P^t‖F\\hat\{\\eta\}=0\.2\\eta\\sqrt\{mn\}/\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}\. Then, we have for alltt,

‖Zt‖F⩽‖Z0‖F​\(1−η​λ\)t\+0\.2​m​nλ⟹‖Zt‖F⩽0\.2​m​nλ​as​t→∞\.\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant\\\|Z\_\{0\}\\\|\_\{F\}\(1\-\\eta\\lambda\)^\{t\}\+\\frac\{0\.2\\sqrt\{mn\}\}\{\\lambda\}\\implies\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant\\frac\{0\.2\\sqrt\{mn\}\}\{\\lambda\}~\\textrm\{as\}~t\\to\\infty\.\(17\)Furthermore, sinceXtX\_\{t\}andYtY\_\{t\}are convex combinations of\{Zs\}s⩽t\\\{Z\_\{s\}\\\}\_\{s\\leqslant t\}, they satisfy the same bound\.

In[fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2), we see that weight decay atYtY\_\{t\}eventually leads to divergence, for both SF\-NorMuon and SF\-AdamW\. We make the following remark regarding this observation:

![Refer to caption](https://arxiv.org/html/2605.23061v1/x4.png)Figure 4:Quasi steady\-state analysis for SF\-NorMuon with decay at𝐙\\mathbf\{Z\}\(averaged over layers\)\.Left:The ratio\|ρt\+1−ρt\|/ρt\|\\rho\_\{t\+1\}\-\\rho\_\{t\}\|/\\rho\_\{t\}remains below 1% after warmup, validating the quasi steady\-state hypothesis \([˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmhypothesis1)\)\.Right:Alignmentαt\\alpha\_\{t\}\(purple\) and RMS\(Z\)\(Z\)\(red\) over training\. The theoretical predictionλ​ρt=0\.1​\[−αt\+αt2\+2​η​λ\]\\lambda\\rho\_\{t\}=0\.1\[\-\\alpha\_\{t\}\+\\sqrt\{\\alpha\_\{t\}^\{2\}\+2\\eta\\lambda\}\]from[lemma˜3\.2](https://arxiv.org/html/2605.23061#S3.Thmlemma2)\(blue dashed\) closely tracks the observed values\. This training run corresponds toη=0\.01,λ=0\.05\\eta=0\.01,\\lambda=0\.05\.Having established iterate boundedness in[lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1), we now study how the norm ofZZevolves during the training\. For this purpose, the analysis greatly simplifies if we make a quasi steady\-state hypothesis following\[[77](https://arxiv.org/html/2605.23061#bib.bib71),[17](https://arxiv.org/html/2605.23061#bib.bib70)\], which states that change in norm ofZZbetween two iterations is much smaller than the norm itself\.

###### Hypothesis 3\.1\(Quasi Steady\-State\)\.

There exists an iteration countτ\\tauand a constantε⩾0\\varepsilon\\geqslant 0such that for allt⩾τt\\geqslant\\tau,

\|‖Zt\+1‖F−‖Zt‖F\|⩽ε​‖Zt‖F\.\\left\|\\\|Z\_\{t\+1\}\\\|\_\{F\}\-\\\|Z\_\{t\}\\\|\_\{F\}\\right\|\\leqslant\\varepsilon\\\|Z\_\{t\}\\\|\_\{F\}~\.\(20\)

To enable uniform comparison across layers of different shapes, we define the RMS normρt:=‖Zt‖F/m​n\\rho\_\{t\}:=\\\|Z\_\{t\}\\\|\_\{F\}/\\sqrt\{mn\}for anm×nm\\times nweight matrix, and the alignmentαt:=⟨P^t,Zt⟩/\(‖P^t‖F​‖Zt‖F\)∈\[−1,1\]\\alpha\_\{t\}:=\\langle\\widehat\{P\}\_\{t\},Z\_\{t\}\\rangle/\(\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}\\\|Z\_\{t\}\\\|\_\{F\}\)\\in\[\-1,1\], which is the cosine of the angle between the preconditioned update and the current iterate\. The left panel of[fig\.˜4](https://arxiv.org/html/2605.23061#S3.F4)shows that\|ρt\+1−ρt\|/ρt\|\\rho\_\{t\+1\}\-\\rho\_\{t\}\|/\\rho\_\{t\}remains small after warmup, validating the hypothesis\. Based on this, we derive the steady\-state value ofρt\\rho\_\{t\}:

###### Lemma 3\.2\(Quasi Steady\-State ofZZ\)\.

Consider the updateZt\+1←Zt−η​λ​Zt−η^t​P^tZ\_\{t\+1\}\\leftarrow Z\_\{t\}\-\\eta\\lambda Z\_\{t\}\-\\hat\{\\eta\}\_\{t\}\\widehat\{P\}\_\{t\}and assume0<η​λ≪10<\\eta\\lambda\\ll 1withη^=0\.2​η​m​n/‖P^t‖F\\hat\{\\eta\}=0\.2\\eta\\sqrt\{mn\}/\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}\. Then, the steady\-state satisfies

λ​ρt=0\.1​\[−αt\+αt2\+2​η​λ\]\.\\lambda\\rho\_\{t\}=0\.1\\Bigl\[\-\\alpha\_\{t\}\+\\sqrt\{\\alpha\_\{t\}^\{2\}\+2\\eta\\lambda\}\\Bigr\]\.\(21\)

After the end of the warm\-up period, the alignmentαt\\alpha\_\{t\}approaches a small and negative value that persists for the rest of the training period as shown by the purple curve in right panel of[fig\.˜4](https://arxiv.org/html/2605.23061#S3.F4)\. Based on the[lemma˜3\.2](https://arxiv.org/html/2605.23061#S3.Thmlemma2), we expect that the RMS norm ofZZ\(depicted in red\) will asymptote to a constant value, which is also confirmed by the red curve sitting on top of the theoretical prediction shown in blue\. Crucially, since theZZnorm reaches a fixed value, the fast iterate remains controlled throughout training, and stabilizes other sequences as shown in[lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1)\.

## 4Experiments on Training Language Models

Table 1:Validation loss atXXacross Chinchilla horizons\.The best loss at each horizon is indicated withbold\.*Speedup*is the percentage of steps saved by SF\-NorMuon to reach the final loss of SF\-AdamW\. SF methods use hyperparameters tuned at2×2\\timesChinchilla for both models\. AdamW uses cosine schedule tuned per\-horizon \(c\.f\.,[table˜4](https://arxiv.org/html/2605.23061#A4.T4)\) for 125M model, and the configuration for8×8\\timeswas reused for the 772M model\. SF\-AdamW uses weight decay atYYas it performs the best in these horizons \(c\.f\.,[fig\.˜2](https://arxiv.org/html/2605.23061#S2.F2)\) and is the originally proposed option in\[[16](https://arxiv.org/html/2605.23061#bib.bib9)\]\.We validate SF\-NorMuon on auto\-regressive language modeling by training decoder only transformers at two scales: a 125M parameter model and a 772M parameter model\. Both architectures follow the LLaMA\-2 design\[[76](https://arxiv.org/html/2605.23061#bib.bib65)\], with tied input\-output embeddings, rotary positional embeddings \(RoPE\)\[[73](https://arxiv.org/html/2605.23061#bib.bib74)\], RMSNorm for pre\-normalization\[[80](https://arxiv.org/html/2605.23061#bib.bib66)\], and squared ReLU activations in the MLP blocks\[[81](https://arxiv.org/html/2605.23061#bib.bib69)\]\. Additional architecture details are provided in[appendix˜D](https://arxiv.org/html/2605.23061#A4)\. Our training code is based on the NanoGPT repository\[[36](https://arxiv.org/html/2605.23061#bib.bib77),[32](https://arxiv.org/html/2605.23061#bib.bib78)\]\.

Dataset and training setup\.We train on the FineWeb\-100B dataset\[[59](https://arxiv.org/html/2605.23061#bib.bib67)\]tokenized with the GPT\-2 tokenizer\[[62](https://arxiv.org/html/2605.23061#bib.bib68)\]\. All experiments use batch size of 512 sequences and sequence length of 1024 tokens, yielding approximately 524K tokens per gradient step\. We adopt the Chinchilla scaling\[[29](https://arxiv.org/html/2605.23061#bib.bib32)\], whereN×N\\timesChinchilla denotes training on∼20​N\\sim 20Ntokens per model parameter\. For our 125M model, this corresponds to 2\.5B tokens at1×1\\times\(5,000 steps\), scaling up to 21B tokens at8×8\\times\(40,000 steps\)\.

Hyperparameter tuning\.We start with the 150M model\. For schedule\-free optimizers, we use 2,000 warm up steps and tune hyper\-parameters at2×2\\timesChinchilla, and reuse these for all settings\. The best hyperparameters for SF\-AdamW areη=0\.01\\eta=0\.01,\(β1,β2\)=\(0\.95,0\.99\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.95,0\.99\), andλ=0\.05\\lambda=0\.05\. For SF\-NorMuon, we reuse the same learning rate and weight decay parameter, following the0\.2​ηt​m​n/‖Pt^‖F0\.2\\eta\_\{t\}\\sqrt\{mn\}/\\\|\\widehat\{P\_\{t\}\}\\\|\_\{F\}rescaling factor from\[[74](https://arxiv.org/html/2605.23061#bib.bib43)\]\. The other hyper\-parameters were set to\(β1,β2\)=\(0\.9,0\.95\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.95\), momentumμ=0\.8\\mu=0\.8\. For the non\-matrix parameters \(embeddings, layer norms\), both schedule\-free methods run SF\-AdamW\. For the AdamW baseline, we tune the learning rateseparately at each horizon, using 2,500 warm up steps followed by cosine decay to zero; comparison of these runs can be found in[fig\.˜6](https://arxiv.org/html/2605.23061#A4.F6)in[appendix˜D](https://arxiv.org/html/2605.23061#A4)\. Other hyperparameters were tuned at2×2\\timesChinchilla\. This represents a well\-tuned baseline with per\-horizon optimization, using optimizedη∈\{0\.004,0\.008\}\\eta\\in\\\{0\.004,0\.008\\\}, andβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, andλ=0\.1\\lambda=0\.1\. Additional hyper\-parameter tuning details and learning rate sweeps are provided in[table˜4](https://arxiv.org/html/2605.23061#A4.T4)in[appendix˜D](https://arxiv.org/html/2605.23061#A4)\.

For the 772M parameter model, we reuse the optimal8×8\\timesChinchilla hyper\-parameters from AdamW and the2×2\\times\-tuned hyper\-parameters for schedule\-free methods\. This approach is motivated by the observation that losses near the optimal learning rates exhibit only small variation, and recent work has shown similar optimal learning rates across small and large Llama\-2 model scales \(see Table 7 and Table 35 in\[[67](https://arxiv.org/html/2605.23061#bib.bib36)\]\)\. We omit the8×8\\timesChinchilla runs for the large model due to compute limitations\.

Results\.[table˜1](https://arxiv.org/html/2605.23061#S4.T1)reports validation losses \(evaluated atXX\) across training horizons\. Note that the SF\-NorMuon uses exactly the same learning rate and weight decay as SF\-AdamW\. For both models, SF\-NorMuon with weight decay atZZconsistently outperforms SF\-AdamW across all horizons\. Crucially, SF\-NorMuon matches the per\-horizon tuned AdamW baselines \(except at8×8\\timesChinchilla where it lags marginally\), while SF\-AdamW lags behind consistency\. This demonstrates that SF\-NorMuon delivers competitive performance with AdamW at any stopping point, without a horizon\-specific learning rate schedule\. For the 772M model, the speedup advantage of SF\-NorMuon over SF\-AdamW is even more pronounced: 50% at1×1\\timesand 52% at2×2\\timesChinchilla\.

![Refer to caption](https://arxiv.org/html/2605.23061v1/x5.png)Figure 5:Learning rate sweep comparison between SF\-AdamW and SF\-NorMuon\.Improvement across learning rates\.A key practical advantage of SF\-NorMuon is robustness to the choice of learning rate\.[fig\.˜5](https://arxiv.org/html/2605.23061#S4.F5)shows a learning\-rate sweep at8×8\\timesChinchilla for both schedule\-free methods: SF\-NorMuon achieves lower validation loss than SF\-AdamW across all learning rates tested\. Importantly, SF\-NorMuon’s performance degrades gracefully away from the optimum, confirming that the spectral geometry provides a favorable optimization landscape even without careful per\-run tuning\.

Other findings\.We present the full convergence proofs under both Frobenius and spectral smoothness in[appendix˜C](https://arxiv.org/html/2605.23061#A3)\. Additional material includes: a self\-contained derivation of spectral and sign update rules from first principles \([appendix˜A](https://arxiv.org/html/2605.23061#A1)\), a review of SF\-SGD and SF\-AdamW \([appendix˜B](https://arxiv.org/html/2605.23061#A2)\), detailed model architectures and hyperparameter search spaces \([appendix˜D](https://arxiv.org/html/2605.23061#A4)\), an ablation study with a comparison against tuned NorMuon across all horizons \([appendix˜E](https://arxiv.org/html/2605.23061#A5)\), and a PyTorch implementation of SF\-NorMuon in[algorithm˜1](https://arxiv.org/html/2605.23061#alg1)\([appendix˜F](https://arxiv.org/html/2605.23061#A6)\)\.

## 5Conclusion

In this work, we proposed SF\-NorMuon, a schedule\-free spectral optimizer that matches per\-horizon tuned AdamW across Chinchilla horizons without requiring schedule or horizon specification in advance\. We proved aO~​\(T−1/4\)\\widetilde\{O\}\(T^\{\-1/4\}\)stationarity guarantee for the underlying schedule\-free spectral dynamics for smooth non\-convex functions and identified weight decay at the fast iterate is a necessity for long\-horizon stability, which is a departure from the convex optimization theory\. A limitation of this work is the diversity of model architectures and datasets; extending the validation of SF\-NorMuon to more diverse settings, and considering larger scales and multi\-stage continual learning pipelines are therefore a natural direction for future work\.

## Acknowledgments

We thank Anthony Ashmore for insightful discussions on the importance of learning rate scheduling\. We thank Aaron Defazio for valuable guidance on schedule\-free methods\. We are grateful to Pragna Subrahmanya for assistance with computing infrastructure\. We thank Rob Otter for the executive support of the work, and our colleagues at the Global Technology Applied Research center of JPMorganChase for support\.

## Author Contributions

A\.A\. and J\.L\.K\. conceived the algorithm\. A\.A\. developed the codebase, conducted the language modeling experiments, and established the convergence and steady\-state theory\. P\.D\. performed additional experiments\. J\.L\.K\. carried out exploratory experiments, shaped the overall research direction and oversaw the presentation of the material\. S\.C\. and N\.K\. provided feedback and contributed to discussions throughout the project\. All authors contributed to writing the manuscript\.

## Disclaimer

This paper was prepared for informational purposes with contributions from the Global Technology Applied Research center of JPMorgan Chase & Co\. This paper is not a product of the Research Department of JPMorgan Chase & Co\. or its affiliates\. Neither JPMorgan Chase & Co\. nor any of its affiliates makes any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect to the completeness, accuracy, or reliability of the information contained herein and the potential legal, compliance, tax, or accounting effects thereof\. This document is not intended as investment research or investment advice, or as a recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction\.

## References

- \[1\]K\. Ahn, N\. Amsel, and J\. Langford\(2025\)Dion2: a simple method to shrink matrix in muon\.External Links:2512\.16928,[Link](https://arxiv.org/abs/2512.16928)Cited by:[1st item](https://arxiv.org/html/2605.23061#A6.I1.i1.p1.2),[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[2\]K\. Ahn, B\. Xu, N\. Abreu, and J\. Langford\(2025\)Dion: distributed orthonormalized updates\.arXiv preprint: 2504\.05295\.Cited by:[1st item](https://arxiv.org/html/2605.23061#A6.I1.i1.p1.2),[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[3\]N\. Amsel, D\. Persson, C\. Musco, and R\. M\. Gower\(2026\)The polar express: optimal matrix sign methods and their application to the muon algorithm\.External Links:2505\.16932,[Link](https://arxiv.org/abs/2505.16932)Cited by:[1st item](https://arxiv.org/html/2605.23061#A6.I1.i1.p1.2),[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[4\]K\. An, Y\. Liu, R\. Pan, Y\. Ren, S\. Ma, D\. Goldfarb, and T\. Zhang\(2025\)ASGO: adaptive structured gradient optimization\.External Links:2503\.20762,[Link](https://arxiv.org/abs/2503.20762)Cited by:[Appendix C](https://arxiv.org/html/2605.23061#A3.SS0.SSS0.Px3.p3.3),[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.23061#S2.SS1.p4.2)\.
- \[5\]J\. Ansel, E\. Yang, H\. He, N\. Gimelshein, A\. Jain, M\. Voznesensky, B\. Bao, P\. Bell, D\. Berard, E\. Burovski, G\. Chauhan, A\. Chourdia, W\. Constable, A\. Desmaison, Z\. DeVito, E\. Ellison, W\. Feng, J\. Gong, M\. Gschwind, B\. Hirsh, S\. Huang, K\. Kalambarkar, L\. Kirsch, M\. Lazos, M\. Lezcano, Y\. Liang, J\. Liang, Y\. Lu, C\. K\. Luk, B\. Maher, Y\. Pan, C\. Puhrsch, M\. Reso, M\. Saroufim, M\. Y\. Siraichi, H\. Suk, S\. Zhang, M\. Suo, P\. Tillet, X\. Zhao, E\. Wang, K\. Zhou, R\. Zou, X\. Wang, A\. Mathews, W\. Wen, G\. Chanan, P\. Wu, and S\. Chintala\(2024\-04\)PyTorch 2: faster machine learning through dynamic python bytecode transformation and graph compilation\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,ASPLOS ’24,pp\. 929–947\.External Links:[Link](http://dx.doi.org/10.1145/3620665.3640366),[Document](https://dx.doi.org/10.1145/3620665.3640366)Cited by:[Appendix D](https://arxiv.org/html/2605.23061#A4.p1.1)\.
- \[6\]J\. Bernstein and L\. Newhouse\(2024\)Old optimizer, new norm: an anthology\.External Links:2409\.20325,[Link](https://arxiv.org/abs/2409.20325)Cited by:[Appendix A](https://arxiv.org/html/2605.23061#A1.p1.3),[§2](https://arxiv.org/html/2605.23061#S2.p2.11)\.
- \[7\]C\. M\. Bishop and H\. Bishop\(2024\)Deep learning: foundations and concepts\.Springer International Publishing\.External Links:ISBN 9783031454684,[Link](http://dx.doi.org/10.1007/978-3-031-45468-4),[Document](https://dx.doi.org/10.1007/978-3-031-45468-4)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[8\]A\. Brohan, N\. Brown, J\. Carbajal, Y\. Chebotar, X\. Chen, K\. Choromanski, T\. Ding, D\. Driess, A\. Dubey, C\. Finn, P\. Florence, C\. Fu, M\. G\. Arenas, K\. Gopalakrishnan, K\. Han, K\. Hausman, A\. Herzog, J\. Hsu, B\. Ichter, A\. Irpan, N\. Joshi, R\. Julian, D\. Kalashnikov, Y\. Kuang, I\. Leal, L\. Lee, T\. E\. Lee, S\. Levine, Y\. Lu, H\. Michalewski, I\. Mordatch, K\. Pertsch, K\. Rao, K\. Reymann, M\. Ryoo, G\. Salazar, P\. Sanketi, P\. Sermanet, J\. Singh, A\. Singh, R\. Soricut, H\. Tran, V\. Vanhoucke, Q\. Vuong, A\. Wahid, S\. Welker, P\. Wohlhart, J\. Wu, F\. Xia, T\. Xiao, P\. Xu, S\. Xu, T\. Yu, and B\. Zitkovich\(2023\)RT\-2: vision\-language\-action models transfer web knowledge to robotic control\.External Links:arXiv:2307\.15818Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[9\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[10\]D\. Chang, Y\. Liu, and G\. Yuan\(2026\)On the convergence of muon and beyond\.External Links:2509\.15816,[Link](https://arxiv.org/abs/2509.15816)Cited by:[1st item](https://arxiv.org/html/2605.23061#S1.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2605.23061#S2.SS1.p7.1)\.
- \[11\]X\. Chen, C\. Liang, D\. Huang, E\. Real, K\. Wang, H\. Pham, X\. Dong, T\. Luong, C\. Hsieh, Y\. Lu,et al\.\(2023\)Symbolic discovery of optimization algorithms\.Advances in neural information processing systems36,pp\. 49205–49233\.Cited by:[§2](https://arxiv.org/html/2605.23061#S2.p3.2)\.
- \[12\]J\. Choquette, W\. Gandhi, O\. Giroux, N\. Stam, and R\. Krashinsky\(2021\-03\)NVIDIA a100 tensor core gpu: performance and innovation\.IEEE Micro41\(2\),pp\. 29–35\.External Links:ISSN 1937\-4143,[Link](http://dx.doi.org/10.1109/MM.2021.3061394),[Document](https://dx.doi.org/10.1109/mm.2021.3061394)Cited by:[Appendix D](https://arxiv.org/html/2605.23061#A4.p1.1)\.
- \[13\]F\. D’Angelo, M\. Andriushchenko, A\. Varre, and N\. Flammarion\(2024\)Why do we need weight decay in modern deep learning?\.External Links:2310\.04415,[Link](https://arxiv.org/abs/2310.04415)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p7.1)\.
- \[14\]D\. Davis and D\. Drusvyatskiy\(2026\)When do spectral gradient updates help in deep learning?\.External Links:2512\.04299,[Link](https://arxiv.org/abs/2512.04299)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[15\]A\. Defazio and K\. Mishchenko\(2023\)Learning\-rate\-free learning by d\-adaptation\.InInternational Conference on Machine Learning,pp\. 7449–7479\.Cited by:[§2\.2](https://arxiv.org/html/2605.23061#S2.SS2.p3.8)\.
- \[16\]A\. Defazio, X\. Yang, H\. Mehta, K\. Mishchenko, A\. Khaled, and A\. Cutkosky\(2024\)The road less scheduled\.Advances in Neural Information Processing Systems37,pp\. 9974–10007\.Cited by:[Appendix B](https://arxiv.org/html/2605.23061#A2.p1.3),[Appendix F](https://arxiv.org/html/2605.23061#A6.p1.1),[§1](https://arxiv.org/html/2605.23061#S1.p1.1),[§1](https://arxiv.org/html/2605.23061#S1.p3.13),[§1](https://arxiv.org/html/2605.23061#S1.p7.1),[§3](https://arxiv.org/html/2605.23061#S3.p2.6),[Table 1](https://arxiv.org/html/2605.23061#S4.T1),[Table 1](https://arxiv.org/html/2605.23061#S4.T1.8.4.3)\.
- \[17\]A\. Defazio\(2025\)Why gradients rapidly increase near the end of training\.External Links:2506\.02285,[Link](https://arxiv.org/abs/2506.02285)Cited by:[§3](https://arxiv.org/html/2605.23061#S3.p3.6),[§3](https://arxiv.org/html/2605.23061#S3.p7.2)\.
- \[18\]T\. Degris, K\. Javed, A\. Sharifnassab, Y\. Liu, and R\. Sutton\(2024\)Step\-size optimization for continual learning\.arXiv preprint arXiv:2401\.17401\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[19\]M\. Delange, R\. Aljundi, M\. Masana, S\. Parisot, X\. Jia, A\. Leonardis, G\. Slabaugh, and T\. Tuytelaars\(2021\)A continual learning survey: defying forgetting in classification tasks\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.External Links:ISSN 1939\-3539,[Link](http://dx.doi.org/10.1109/TPAMI.2021.3057446),[Document](https://dx.doi.org/10.1109/tpami.2021.3057446)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[20\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch Transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§3](https://arxiv.org/html/2605.23061#S3.p1.1)\.
- \[21\]B\. Ghorbani, S\. Krishnan, and Y\. Xiao\(2019\)An investigation into neural net optimization via hessian eigenvalue density\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 2232–2241\.External Links:[Link](https://proceedings.mlr.press/v97/ghorbani19b.html)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[22\]GLM\-5\-Team\(2026\)GLM\-5: from vibe coding to agentic engineering\.External Links:2602\.15763,[Link](https://arxiv.org/abs/2602.15763)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§3](https://arxiv.org/html/2605.23061#S3.p1.1)\.
- \[23\]I\. Goodfellow, Y\. Bengio, and A\. Courville\(2016\)Deep learning\.MIT Press\.Note:[http://www\.deeplearningbook\.org](http://www.deeplearningbook.org/)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[24\]P\. Goyal, P\. Dollár, R\. Girshick, P\. Noordhuis, L\. Wesolowski, A\. Kyrola, A\. Tulloch, Y\. Jia, and K\. He\(2017\)Accurate, large minibatch sgd: training imagenet in 1 hour\.arXiv preprint arXiv:1706\.02677\.Cited by:[§2\.2](https://arxiv.org/html/2605.23061#S2.SS2.p2.4)\.
- \[25\]V\. Gupta, T\. Koren, and Y\. Singer\(2018\)Shampoo: preconditioned stochastic tensor optimization\.External Links:1802\.09568,[Link](https://arxiv.org/abs/1802.09568)Cited by:[Appendix C](https://arxiv.org/html/2605.23061#A3.SS0.SSS0.Px3.p3.3),[§2\.1](https://arxiv.org/html/2605.23061#S2.SS1.p4.2)\.
- \[26\]A\. Hägele, E\. Bakouch, A\. Kosson, L\. B\. Allal, L\. Von Werra, and M\. Jaggi\(2024\)Scaling laws and compute\-optimal training beyond fixed training durations\.Advances in Neural Information Processing Systems37,pp\. 76232–76264\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p4.1)\.
- \[27\]A\. Hägele, E\. Bakouch, A\. Kosson, L\. B\. Allal, L\. V\. Werra, and M\. Jaggi\(2024\)Scaling laws and compute\-optimal training beyond fixed training durations\.External Links:2405\.18392,[Link](https://arxiv.org/abs/2405.18392)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p3.12)\.
- \[28\]T\. Hastie, R\. Tibshirani, and J\. Friedman\(2009\)The elements of statistical learning\.Springer New York\.External Links:ISBN 9780387848587,ISSN 2197\-568X,[Link](http://dx.doi.org/10.1007/978-0-387-84858-7),[Document](https://dx.doi.org/10.1007/978-0-387-84858-7)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[29\]J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre\(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[Appendix D](https://arxiv.org/html/2605.23061#A4.SS0.SSS0.Px2.p1.7),[§1](https://arxiv.org/html/2605.23061#S1.p2.2),[§4](https://arxiv.org/html/2605.23061#S4.p2.4)\.
- \[30\]S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui, X\. Long, Z\. Zheng, Y\. Fang, Y\. Huang, W\. Zhao, X\. Zhang, Z\. L\. Thai, K\. Zhang, C\. Wang, Y\. Yao, C\. Zhao, J\. Zhou, J\. Cai, Z\. Zhai, N\. Ding, C\. Jia, G\. Zeng, D\. Li, Z\. Liu, and M\. Sun\(2024\)MiniCPM: unveiling the potential of small language models with scalable training strategies\.External Links:2404\.06395,[Link](https://arxiv.org/abs/2404.06395)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p2.2)\.
- \[31\]P\. Izmailov, D\. Podoprikhin, T\. Garipov, D\. Vetrov, and A\. G\. Wilson\(2018\)Averaging weights leads to wider optima and better generalization\.arXiv preprint arXiv:1803\.05407\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p3.12)\.
- \[32\]K\. Jordan, J\. Bernstein, B\. Rappazzo, @fernbear\.bsky\.social, B\. Vlado, Y\. Jiacheng, F\. Cesista, B\. Koszarsky, and @Grad62304977\(2024\)Modded\-nanogpt: speedrunning the nanogpt baseline\.External Links:[Link](https://github.com/KellerJordan/modded-nanogpt)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p1.1)\.
- \[33\]K\. Jordanet al\.\(2024\)Muon \(github repository\): an optimizer for hidden layers in neural networks\.Note:GitHub repository, master branchExternal Links:[Link](https://github.com/KellerJordan/Muon)Cited by:[Appendix F](https://arxiv.org/html/2605.23061#A6.p1.1),[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[34\]K\. Jordanet al\.\(2024\)Muon: an optimizer for hidden layers in neural networks\.Note:[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)Accessed: 2026\-01\-25Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[35\]D\. S\. Kalra and M\. Barkeshli\(2024\)Why warmup the learning rate? underlying mechanisms and improvements\.Advances in Neural Information Processing Systems37,pp\. 111760–111801\.Cited by:[§2\.2](https://arxiv.org/html/2605.23061#S2.SS2.p2.4)\.
- \[36\]A\. Karpathy\(2022\)NanoGPT\.GitHub\.Note:[https://github\.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p1.1)\.
- \[37\]P\. Kasimbeg, V\. Roulet, N\. Agarwal, S\. Medapati, F\. Pedregosa, A\. Agarwala, and G\. E\. Dahl\(2025\)How far away are truly hyperparameter\-free learning algorithms?\.arXiv preprint arXiv:2505\.24005\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[38\]P\. Kasimbeg, F\. Schneider, R\. Eschenhagen, J\. Bae, C\. S\. Sastry, M\. Saroufim, F\. Boyuan, L\. Wright, E\. Z\. Yang, Z\. Nado, S\. Medapati, P\. Hennig, M\. Rabbat, and G\. E\. Dahl\(2025\)Accelerating neural network training: an analysis of the AlgoPerf competition\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CtM5xjRSfm)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p3.12)\.
- \[39\]A\. Khaled, K\. Ozkara, T\. Yu, M\. Hong, and Y\. Park\(2025\)MuonBP: faster muon via block\-periodic orthogonalization\.External Links:2510\.16981,[Link](https://arxiv.org/abs/2510.16981)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[40\]M\. J\. Kim, K\. Pertsch, S\. Karamcheti, T\. Xiao, A\. Balakrishna, S\. Nair, R\. Rafailov, E\. Foster, G\. Lam, P\. Sanketi, Q\. Vuong, T\. Kollar, B\. Burchfiel, R\. Tedrake, D\. Sadigh, S\. Levine, P\. Liang, and C\. Finn\(2024\)OpenVLA: an open\-source vision\-language\-action model\.External Links:arXiv:2406\.09246Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[41\]D\. P\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1412.6980)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p4.1),[§2](https://arxiv.org/html/2605.23061#S2.p3.2)\.
- \[42\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska, D\. Hassabis, C\. Clopath, D\. Kumaran, and R\. Hadsell\(2017\-03\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.External Links:ISSN 1091\-6490,[Link](http://dx.doi.org/10.1073/pnas.1611835114),[Document](https://dx.doi.org/10.1073/pnas.1611835114)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[43\]A\. Kravatskiy, I\. Kozyrev, N\. Kozlov, A\. Vinogradov, D\. Merkulov, and I\. Oseledets\(2025\)The ky fan norms and beyond: dual norms and combinations for matrix optimization\.External Links:2512\.09678,[Link](https://arxiv.org/abs/2512.09678)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[44\]A\. Krogh and J\. A\. Hertz\(1991\)A simple weight decay can improve generalization\.InProceedings of the 5th International Conference on Neural Information Processing Systems,NIPS’91,San Francisco, CA, USA,pp\. 950–957\.External Links:ISBN 1558602224Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p7.1)\.
- \[45\]G\. Lan\(2012\)An optimal method for stochastic composite optimization\.Mathematical Programming133\(1\),pp\. 365–397\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p3.6)\.
- \[46\]D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen\(2021\)GShard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2605.23061#S3.p1.1)\.
- \[47\]J\. Li and M\. Hong\(2025\)A note on the convergence of muon\.External Links:2502\.02900,[Link](https://arxiv.org/abs/2502.02900)Cited by:[1st item](https://arxiv.org/html/2605.23061#S1.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2605.23061#S2.SS1.p7.1)\.
- \[48\]Y\. Li, Y\. Ma, S\. Yan, C\. Zhang, J\. Liu, J\. Lu, Z\. Xu, M\. Chen, M\. Wang, S\. Zhan, J\. Ma, X\. Lai, D\. Liu, Y\. Luo, X\. Bin, H\. Ren, M\. Han, W\. Hao, B\. Yi, L\. Liu, B\. Ma, X\. Jia, X\. Zhou, S\. Qiao, L\. Xiang, and Y\. Wu\(2025\)Model merging in pre\-training of large language models\.External Links:2505\.12082,[Link](https://arxiv.org/abs/2505.12082)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p3.12)\.
- \[49\]Z\. Li, L\. Liu, C\. Liang, W\. Chen, and T\. Zhao\(2025\)NorMuon: making muon more efficient and scalable\.External Links:2510\.05491,[Link](https://arxiv.org/abs/2510.05491)Cited by:[§E\.1](https://arxiv.org/html/2605.23061#A5.SS1.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2605.23061#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§2\.2](https://arxiv.org/html/2605.23061#S2.SS2.p4.3)\.
- \[50\]J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. Yang\(2025\)Muon is scalable for llm training\.External Links:2502\.16982,[Link](https://arxiv.org/abs/2502.16982)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§2\.2](https://arxiv.org/html/2605.23061#S2.SS2.p2.4)\.
- \[51\]Z\. Liu, R\. Zhang, Z\. Wang, Y\. Zhao, Y\. Su, Z\. Yang, and Z\. Zhang\(2026\)Muon2: boosting muon via adaptive second\-moment preconditioning\.External Links:2604\.09967,[Link](https://arxiv.org/abs/2604.09967)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[52\]I\. Loshchilov and F\. Hutter\(2017\)SGDR: stochastic gradient descent with warm restarts\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Skq89Scxx)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p2.2)\.
- \[53\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p4.1)\.
- \[54\]Y\. Ma, Z\. Song, Y\. Zhuang, J\. Hao, and I\. King\(2024\)A survey on vision\-language\-action models for embodied ai\.External Links:arXiv:2405\.14093Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[55\]A\. Meterez, P\. A\. Nair, D\. Morwani, C\. Pehlevan, and S\. Kakade\(2026\)Anytime pretraining: horizon\-free learning\-rate schedules with weight averaging\.arXiv preprint arXiv:2602\.03702\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1),[§1](https://arxiv.org/html/2605.23061#S1.p3.12)\.
- \[56\]S\. Minaee, T\. Mikolov, N\. Nikzad, M\. Chenaghlu, R\. Socher, X\. Amatriain, and J\. Gao\(2024\)Large language models: a survey\.External Links:arXiv:2402\.06196Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[57\]Y\. Nesterov\(1983\)A method for solving the convex programming problem with convergence rate o \(1/k2\)\.InDokl akad nauk Sssr,Vol\.269,pp\. 543\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p3.6)\.
- \[58\]G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. Wermter\(2019\-05\)Continual lifelong learning with neural networks: a review\.Neural Networks113,pp\. 54–71\.External Links:ISSN 0893\-6080,[Link](http://dx.doi.org/10.1016/j.neunet.2019.01.012),[Document](https://dx.doi.org/10.1016/j.neunet.2019.01.012)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[59\]G\. Penedo, H\. Kydlíček, L\. B\. allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. V\. Werra, and T\. Wolf\(2024\)The fineweb datasets: decanting the web for the finest text data at scale\.External Links:2406\.17557,[Link](https://arxiv.org/abs/2406.17557)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p2.4)\.
- \[60\]T\. Pethick, W\. Xie, K\. Antonakopoulos, Z\. Zhu, A\. Silveti\-Falls, and V\. Cevher\(2025\)Training deep learning models with norm\-constrained lmos\.External Links:2502\.07529,[Link](https://arxiv.org/abs/2502.07529)Cited by:[§2](https://arxiv.org/html/2605.23061#S2.p2.11)\.
- \[61\]S\. Qiu, Z\. Chen, H\. Phan, Q\. Lei, and A\. G\. Wilson\(2026\)Hyperparameter transfer enables consistent gains of matrix\-preconditioned optimizers across scales\.External Links:2512\.05620,[Link](https://arxiv.org/abs/2512.05620)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p7.1)\.
- \[62\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p2.4)\.
- \[63\]L\. Ren, Y\. Liu, Y\. Shen, and W\. Chen\(2026\)Rethinking language model scaling under transferable hypersphere optimization\.External Links:2603\.28743,[Link](https://arxiv.org/abs/2603.28743)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[64\]L\. Sagun, U\. Evci, V\. U\. Guney, Y\. Dauphin, and L\. Bottou\(2017\)Empirical analysis of the hessian of over\-parametrized neural networks\.External Links:arXiv:1706\.04454Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1)\.
- \[65\]N\. Sato, H\. Naganuma, and H\. Iiduka\(2025\)Convergence bound and critical batch size of muon optimizer\.External Links:2507\.01598,[Link](https://arxiv.org/abs/2507.01598)Cited by:[1st item](https://arxiv.org/html/2605.23061#S1.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2605.23061#S2.SS1.p7.1)\.
- \[66\]F\. Schaipp, A\. Hägele, A\. Taylor, U\. Simsekli, and F\. Bach\(2025\)The surprising agreement between convex optimization theory and learning\-rate scheduling for large model training\.arXiv preprint arXiv:2501\.18965\.Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p7.1),[§3](https://arxiv.org/html/2605.23061#S3.p2.6)\.
- \[67\]A\. Semenov, M\. Pagliardini, and M\. Jaggi\(2026\)Benchmarking optimizers for large language model pretraining\.External Links:[Link](https://openreview.net/forum?id=Jw7khYzYzl)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p4.1),[§4](https://arxiv.org/html/2605.23061#S4.p4.3)\.
- \[68\]N\. Shazeer, A\. Mirhoseini, K\. Mahdavi, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2605.23061#S3.p1.1)\.
- \[69\]W\. Shen, R\. Huang, M\. Huang, C\. Shen, and J\. Zhang\(2026\)On the convergence analysis of muon\.External Links:2505\.23737,[Link](https://arxiv.org/abs/2505.23737)Cited by:[1st item](https://arxiv.org/html/2605.23061#S1.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2605.23061#S2.SS1.p7.1)\.
- \[70\]C\. Si, D\. Zhang, and W\. Shen\(2025\)AdaMuon: adaptive muon optimizer\.External Links:2507\.11005,[Link](https://arxiv.org/abs/2507.11005)Cited by:[2nd item](https://arxiv.org/html/2605.23061#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§2\.2](https://arxiv.org/html/2605.23061#S2.SS2.p4.3)\.
- \[71\]V\. Singh, L\. Krauss, S\. Jaghouar, M\. Sirovatka, C\. Goddard, F\. Obied, J\. M\. Ong, J\. Straube, Fern, A\. Harley, C\. Stewart, C\. Kealty, M\. Panahi, S\. Kirsten, A\. Deshpande, A\. Vij, A\. Bresnu, P\. Veldurthi, R\. Ravishankar, H\. Bishnoi, D\. Team, A\. A\. Team, P\. I\. Team, M\. McQuade, J\. Hagemann, and L\. Atkins\(2026\)Arcee trinity large technical report\.External Links:2602\.17004,[Link](https://arxiv.org/abs/2602.17004)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§3](https://arxiv.org/html/2605.23061#S3.p1.1)\.
- \[72\]M\. Song, B\. Baek, K\. Ahn, and C\. Yun\(2025\)Through the river: understanding the benefit of schedule\-free methods for language model training\.InHigh\-dimensional Learning Dynamics 2025,External Links:[Link](https://openreview.net/forum?id=b5HYeRzG9M)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p4.1)\.
- \[73\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\(2023\)RoFormer: enhanced transformer with rotary position embedding\.External Links:2104\.09864,[Link](https://arxiv.org/abs/2104.09864)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p1.1)\.
- \[74\]K\. Team\(2026\)Kimi k2\.5: visual agentic intelligence\.External Links:2602\.02276,[Link](https://arxiv.org/abs/2602.02276)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p5.1),[§3](https://arxiv.org/html/2605.23061#S3.p1.1),[§4](https://arxiv.org/html/2605.23061#S4.p3.12)\.
- \[75\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample\(2023\)LLaMA: open and efficient foundation language models\.External Links:2302\.13971,[Link](https://arxiv.org/abs/2302.13971)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[76\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom\(2023\)Llama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p1.1)\.
- \[77\]T\. van Laarhoven\(2017\)L2 regularization versus batch and weight normalization\.External Links:1706\.05350,[Link](https://arxiv.org/abs/1706.05350)Cited by:[§3](https://arxiv.org/html/2605.23061#S3.p3.6),[§3](https://arxiv.org/html/2605.23061#S3.p7.2)\.
- \[78\]L\. Wang, X\. Zhang, H\. Su, and J\. Zhu\(2024\)A comprehensive survey of continual learning: theory, method and application\.External Links:2302\.00487,[Link](https://arxiv.org/abs/2302.00487)Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.
- \[79\]K\. You, M\. Long, J\. Wang, and M\. I\. Jordan\(2019\)How does learning rate decay help modern neural networks?\.External Links:1908\.01878,[Link](https://arxiv.org/abs/1908.01878)Cited by:[§E\.2](https://arxiv.org/html/2605.23061#A5.SS2.SSS0.Px2.p1.1)\.
- \[80\]B\. Zhang and R\. Sennrich\(2019\)Root mean square layer normalization\.External Links:1910\.07467,[Link](https://arxiv.org/abs/1910.07467)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p1.1)\.
- \[81\]Z\. Zhang, Y\. Song, G\. Yu, X\. Han, Y\. Lin, C\. Xiao, C\. Song, Z\. Liu, Z\. Mi, and M\. Sun\(2024\)ReLU2wins: discovering efficient activation functions for sparse llms\.External Links:2402\.03804,[Link](https://arxiv.org/abs/2402.03804)Cited by:[§4](https://arxiv.org/html/2605.23061#S4.p1.1)\.
- \[82\]W\. X\. Zhao, K\. Zhou, J\. Li, T\. Tang, X\. Wang, Y\. Hou, Y\. Min, B\. Zhang, J\. Zhang, Z\. Dong, Y\. Du, C\. Yang, Y\. Chen, Z\. Chen, J\. Jiang, R\. Ren, Y\. Li, X\. Tang, Z\. Liu, P\. Liu, J\. Nie, and J\. Wen\(2023\)A survey of large language models\.External Links:arXiv:2303\.18223Cited by:[§1](https://arxiv.org/html/2605.23061#S1.p1.1)\.

## Technical Appendices and Supplementary Material for “Anytime Training with Schedule\-Free Spectral Optimization”

In this appendix, we provide more thorough preliminaries and background information that supplement the main text\. We also state the formal version of the informal theorems from the main text, as well as their proofs\. This appendix is organized as follows:

- •In[appendix˜A](https://arxiv.org/html/2605.23061#A1), we review metrized deep learning, and derive optimizers from norms on weight matrices\.
- •In[appendix˜B](https://arxiv.org/html/2605.23061#A2), we review the Schedule\-free method\. In particular, we discuss SF\-SGD and its convergence bound, and the algorithm for SF\-AdamW\.
- •In[appendix˜C](https://arxiv.org/html/2605.23061#A3)proves in the detail the convergence of SF\-Spectral Descent, for noisy Lipschitz smooth functions in both the Frobenius and spectral norm\. In addition, we prove the lemmas used in the steady\-state analysis\.
- •In[appendix˜D](https://arxiv.org/html/2605.23061#A4), we review the architecture of the language models used in our experiments and present the hyper\-parameter selection procedure\.
- •In[appendix˜E](https://arxiv.org/html/2605.23061#A5), studies the effect of ablating momentum term for SF\-NorMuon \(settingμ=0\\mu=0\), and ablating the row\-normalization\. Furthermore, we present a detailed comparison against NorMuon with cosine decay of learning rate\.
- •In[appendix˜F](https://arxiv.org/html/2605.23061#A6), we present a PyTorch implementation of SF\-NorMuon algorithm\.

## Appendix AOptimizers from Weight\-Space Matrix Geometry

For the reader’s convenience, we collect here the basic derivations of the update rules discussed in the main text\. The starting point is always the same: given a current iterateWtW\_\{t\}and gradientGt:=∇f​\(Wt\)G\_\{t\}:=\\nabla f\(W\_\{t\}\), we choose an updateΔ​Wt\\Delta W\_\{t\}by minimizing the first\-order model of the loss subject to a constraint, or equivalently a quadratic penalty, in the geometry of interest\. Our treatment follows the work of Bernstein and Newhouse\[[6](https://arxiv.org/html/2605.23061#bib.bib49)\]\.

### A\.1Steepest descent in a prescribed geometry

Let∥⋅∥𝒳\\\|\\cdot\\\|\_\{\\mathcal\{X\}\}be any norm on the space of matrices\. The corresponding norm\-constrained steepest\-descent step is

Δ​Wt∈arg⁡min‖Δ​W‖𝒳⩽η⁡⟨Gt,Δ​W⟩\.\\Delta W\_\{t\}\\in\\arg\\min\_\{\\\|\\Delta W\\\|\_\{\\mathcal\{X\}\}\\leqslant\\eta\}\\langle G\_\{t\},\\Delta W\\rangle\.\(22\)Since the objective is linear, the solution lies on the boundary of the ball and points in an extremal descent direction for the chosen norm\. Writing

‖Gt‖𝒳,∗:=sup‖U‖𝒳⩽1⟨Gt,U⟩\\\|G\_\{t\}\\\|\_\{\\mathcal\{X\},\*\}:=\\sup\_\{\\\|U\\\|\_\{\\mathcal\{X\}\}\\leqslant 1\}\\langle G\_\{t\},U\\rangle\(23\)for the dual norm, \([22](https://arxiv.org/html/2605.23061#A1.E22)\) is equivalent to

Δ​Wt=−η​Ut,Ut∈arg⁡max‖U‖𝒳⩽1⁡⟨Gt,U⟩,\\Delta W\_\{t\}=\-\\eta\\,U\_\{t\},\\qquad U\_\{t\}\\in\\arg\\max\_\{\\\|U\\\|\_\{\\mathcal\{X\}\}\\leqslant 1\}\\langle G\_\{t\},U\\rangle,\(24\)and the optimal value is−η​‖Gt‖𝒳,∗\-\\eta\\\|G\_\{t\}\\\|\_\{\\mathcal\{X\},\*\}\.

In practice one often uses the equivalent regularized local model

Δ​Wt∈arg⁡minΔ​W⁡\{⟨Gt,Δ​W⟩\+12​η​‖Δ​W‖𝒳,t2\},\\Delta W\_\{t\}\\in\\arg\\min\_\{\\Delta W\}\\left\\\{\\langle G\_\{t\},\\Delta W\\rangle\+\\frac\{1\}\{2\\eta\}\\\|\\Delta W\\\|\_\{\\mathcal\{X\},t\}^\{2\}\\right\\\},\(25\)where∥⋅∥𝒳,t\\\|\\cdot\\\|\_\{\\mathcal\{X\},t\}may depend on time through a momentum estimate or a preconditioner\. The constrained form \([22](https://arxiv.org/html/2605.23061#A1.E22)\) produces normalized directions such as sign and polar updates, while the regularized form \([25](https://arxiv.org/html/2605.23061#A1.E25)\) produces the preconditioned\-gradient forms used by practical optimizers\.

### A\.2The sign geometry:p=1p=1,q=∞q=\\infty

Take the matrix norm

‖A‖ℓ1→ℓ∞=sup‖u‖1⩽1‖A​u‖∞\.\\\|A\\\|\_\{\\ell\_\{1\}\\to\\ell\_\{\\infty\}\}=\\sup\_\{\\\|u\\\|\_\{1\}\\leqslant 1\}\\\|Au\\\|\_\{\\infty\}\.\(26\)A direct calculation shows that this is simply the entrywise max norm:

‖A‖ℓ1→ℓ∞=maxi,j⁡\|Ai​j\|\.\\\|A\\\|\_\{\\ell\_\{1\}\\to\\ell\_\{\\infty\}\}=\\max\_\{i,j\}\|A\_\{ij\}\|\.\(27\)Indeed, for each rowii,

\|\(A​u\)i\|=\|∑jAi​j​uj\|⩽maxj⁡\|Ai​j\|​∑j\|uj\|⩽maxj⁡\|Ai​j\|,\|\(Au\)\_\{i\}\|=\\left\|\\sum\_\{j\}A\_\{ij\}u\_\{j\}\\right\|\\leqslant\\max\_\{j\}\|A\_\{ij\}\|\\sum\_\{j\}\|u\_\{j\}\|\\leqslant\\max\_\{j\}\|A\_\{ij\}\|,\(28\)so‖A​u‖∞⩽maxi,j⁡\|Ai​j\|\\\|Au\\\|\_\{\\infty\}\\leqslant\\max\_\{i,j\}\|A\_\{ij\}\|\. Equality is achieved by choosinguuto be a signed coordinate vector supported on an entry attaining the maximum\.

Substituting \([27](https://arxiv.org/html/2605.23061#A1.E27)\) into \([22](https://arxiv.org/html/2605.23061#A1.E22)\) gives

minmaxi,j⁡\|Δ​Wi​j\|⩽η​∑i,j\(Gt\)i​j​\(Δ​W\)i​j\.\\min\_\{\\max\_\{i,j\}\|\\Delta W\_\{ij\}\|\\leqslant\\eta\}\\sum\_\{i,j\}\(G\_\{t\}\)\_\{ij\}\(\\Delta W\)\_\{ij\}\.\(29\)This decouples coordinatewise, and for each entry the minimizer is

\(Δ​Wt\)i​j=−η​sign​\(\(Gt\)i​j\)\.\(\\Delta W\_\{t\}\)\_\{ij\}=\-\\eta\\,\\mathrm\{sign\}\\bigl\(\(G\_\{t\}\)\_\{ij\}\\bigr\)\.\(30\)Hence the steepest\-descent rule is

Δ​Wt=−η​sign​\(Gt\),\\Delta W\_\{t\}=\-\\eta\\,\\mathrm\{sign\}\(G\_\{t\}\),\(31\)which is the base update underlying signSGD:

Wt\+1=Wt−η​sign​\(Gt\)\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\mathrm\{sign\}\(G\_\{t\}\)\.\(32\)

### A\.3Momentum in the sign geometry

To overcome the noise from the computing the gradient on a mini\-batch a standard practical modification is to replace the raw gradient by a smoothed momentum estimate\. Let

Mt=β​Mt−1\+\(1−β\)​Gt,0⩽β<1\.M\_\{t\}=\\beta M\_\{t\-1\}\+\(1\-\\beta\)G\_\{t\},\\qquad 0\\leqslant\\beta<1\.\(33\)Applying the same steepest\-descent rule toMtM\_\{t\}instead ofGtG\_\{t\}yields

Wt\+1=Wt−η​sign​\(Mt\),W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\mathrm\{sign\}\(M\_\{t\}\),\(34\)which is the update for signSGD with momentum\.

##### Lion\.

Lion uses a sign update applied to a momentum direction that directly incorporates current gradient as well\. In the standard implementation, one first forms

Ct=β1​Mt−1\+\(1−β1\)​Gt,C\_\{t\}=\\beta\_\{1\}M\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)G\_\{t\},\(35\)then updates

Wt\+1=Wt−η​sign​\(Ct\),W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\mathrm\{sign\}\(C\_\{t\}\),\(36\)and finally refreshes the state

Mt=β2​Mt−1\+\(1−β2\)​Gt\.M\_\{t\}=\\beta\_\{2\}M\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)G\_\{t\}\.\(37\)Thus Lion can be viewed as a two\-timescale version of the basic sign steepest\-descent rule\.

##### Adam withβ2=0\\beta\_\{2\}=0\.

Adam augments the gradient with both a momentum buffer and a variance buffer:

mt=β1​mt−1\+\(1−β1\)​Gt,vt=β2​vt−1\+\(1−β2\)​Gt⊙2,m\_\{t\}=\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)G\_\{t\},\\qquad v\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)G\_\{t\}^\{\\odot 2\},\(38\)whereGt⊙2G\_\{t\}^\{\\odot 2\}denotes the entrywise square\. Its coordinatewise update is

Wt\+1=Wt−η​mtvt\+ε\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\frac\{m\_\{t\}\}\{\\sqrt\{v\_\{t\}\}\+\\varepsilon\}\.\(39\)Whenβ2=0\\beta\_\{2\}=0, the variance buffer collapses to the current squared gradient,

vt=Gt⊙2,v\_\{t\}=G\_\{t\}^\{\\odot 2\},\(40\)and so

Wt\+1=Wt−η​sign​\(Mt\)\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\mathrm\{sign\}\(M\_\{t\}\)\.\(41\)Thus, Adam withβ2=0\\beta\_\{2\}=0keeps the momentum buffer but normalizes it coordinatewise by the current gradient magnitude\. This leads us back to signSGD with momentum\.

### A\.4The spectral geometry:p=q=2p=q=2

Now take the matrix norm

‖A‖ℓ2→ℓ2=‖A‖op,\\\|A\\\|\_\{\\ell\_\{2\}\\to\\ell\_\{2\}\}=\\\|A\\\|\_\{\\mathrm\{op\}\},\(42\)the spectral norm\. The norm\-constrained step becomes

Δ​Wt∈arg⁡min‖Δ​W‖op⩽η⁡⟨Gt,Δ​W⟩\.\\Delta W\_\{t\}\\in\\arg\\min\_\{\\\|\\Delta W\\\|\_\{\\mathrm\{op\}\}\\leqslant\\eta\}\\langle G\_\{t\},\\Delta W\\rangle\.\(43\)Let the singular value decomposition ofGtG\_\{t\}be

Gt=Ut​Σt​Vt⊤\.G\_\{t\}=U\_\{t\}\\Sigma\_\{t\}V\_\{t\}^\{\\top\}\.\(44\)By von Neumann’s trace inequality,

⟨Gt,U⟩⩽‖Gt‖∗​‖U‖opfor all​U,\\langle G\_\{t\},U\\rangle\\leqslant\\\|G\_\{t\}\\\|\_\{\*\}\\\|U\\\|\_\{\\mathrm\{op\}\}\\qquad\\text\{for all \}U,\(45\)so over the unit spectral\-norm ball the maximum of⟨Gt,U⟩\\langle G\_\{t\},U\\rangleis‖Gt‖∗\\\|G\_\{t\}\\\|\_\{\*\}, attained by

U=Ut​Vt⊤\.U=U\_\{t\}V\_\{t\}^\{\\top\}\.\(46\)This matrix is the polar factor ofGtG\_\{t\}, which we denote bypolar​\(Gt\)\\mathrm\{polar\}\(G\_\{t\}\)\. Therefore

Δ​Wt=−η​polar​\(Gt\),\\Delta W\_\{t\}=\-\\eta\\,\\mathrm\{polar\}\(G\_\{t\}\),\(47\)and the corresponding update for spectral descent is

Wt\+1=Wt−η​polar​\(Gt\)\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\mathrm\{polar\}\(G\_\{t\}\)\.\(48\)

### A\.5Momentum in the spectral geometry

##### Muon\.

Exactly as in the sign case, we may replaceGtG\_\{t\}by a momentum\-smoothed matrix

Mt=β​Mt−1\+\(1−β\)​Gt\.M\_\{t\}=\\beta M\_\{t\-1\}\+\(1\-\\beta\)G\_\{t\}\.\(49\)Applying spectral steepest descent toMtM\_\{t\}gives the update for Muon

Wt\+1=Wt−η​polar​\(Mt\)\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,\\mathrm\{polar\}\(M\_\{t\}\)\.\(50\)In practical implementations, this polar factor is not computed by an exact SVD but is approximated by a small number of Newton\-Schulz iterations\. This minimizes the computational overhead as they can be implemented using only matrix multiplications \(GEMMs\)\.

##### Shampoo\.

Shampoo maintains left and right second\-moment buffers for each matrix parameter and updates by preconditioning with their inverse quarter\-powers

Mt=β1​Mt−1\+\(1−β1\)​Gt,Lt=β2​Lt−1\+\(1−β2\)​Mt​Mt⊤,Rt=β2​Rt−1\+\(1−β2\)​Mt⊤​Mt,M\_\{t\}=\\beta\_\{1\}M\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)G\_\{t\},\\qquad L\_\{t\}=\\beta\_\{2\}L\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)M\_\{t\}M\_\{t\}^\{\\top\},\\qquad R\_\{t\}=\\beta\_\{2\}R\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)M\_\{t\}^\{\\top\}M\_\{t\},\(51\)followed by the update

Wt\+1=Wt−η​Lt−1/4​Mt​Rt−1/4\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,L\_\{t\}^\{\-1/4\}M\_\{t\}R\_\{t\}^\{\-1/4\}\.\(52\)In the limiting caseβ2=0\\beta\_\{2\}=0, these buffers collapse to the instantaneous matrices

Lt=Mt​Mt⊤,Rt=Mt⊤​Mt\.L\_\{t\}=M\_\{t\}M\_\{t\}^\{\\top\},\\qquad R\_\{t\}=M\_\{t\}^\{\\top\}M\_\{t\}\.\(53\)IfMt=U​Σ​V⊤M\_\{t\}=U\\Sigma V^\{\\top\}is the singular value decomposition, then

Lt−1/4​Mt​Rt−1/4=U​Σ−1/2​U⊤​\(U​Σ​V⊤\)​V​Σ−1/2​V⊤=U​V⊤=polar​\(Mt\),L\_\{t\}^\{\-1/4\}M\_\{t\}R\_\{t\}^\{\-1/4\}=U\\Sigma^\{\-1/2\}U^\{\\top\}\\,\(U\\Sigma V^\{\\top\}\)\\,V\\Sigma^\{\-1/2\}V^\{\\top\}=UV^\{\\top\}=\\mathrm\{polar\}\(M\_\{t\}\),\(54\)where the identity is exact in the idealized full\-rank case \(and otherwise understood with pseudo\-inverses on the support ofMtM\_\{t\}\)\. Hence, when the second\-moment buffers are collapsed in this way, Shampoo reduces to the same polar transform of the momentum matrix that underlies spectral descent and Muon\.

## Appendix BReview of SF\-SGD and SF\-AdamW

Schedule\-free \(SF\) methods were introduced to remove the need for a prescribed training horizon in learning\-rate scheduling\. The basic idea is to keep a*fast*sequenceztz\_\{t\}, on which the base optimizer runs at a nearly constant learning rate, while returning an*averaged*sequencextx\_\{t\}whose effective learning rate decays automatically through online averaging\. Gradients are evaluated at an interpolation pointyty\_\{t\}between these two states\. In this way, schedule\-free optimization replaces explicit time\-based annealing by implicit annealing through online weight averaging\[[16](https://arxiv.org/html/2605.23061#bib.bib9)\]\.

##### SF\-SGD\.

In its simplest form, Schedule\-Free SGD maintains three sequences,

yt=β​xt\+\(1−β\)​zt,gt=∇ℒ​\(yt;ξt\),zt\+1=zt−η​gt,y\_\{t\}=\\beta x\_\{t\}\+\(1\-\\beta\)z\_\{t\},\\qquad g\_\{t\}=\\nabla\\mathcal\{L\}\(y\_\{t\};\\xi\_\{t\}\),\\qquad z\_\{t\+1\}=z\_\{t\}\-\\eta g\_\{t\},\(55\)together with the online average

xt\+1=\(1−ct\+1\)​xt\+ct\+1​zt\+1,ct\+1=1t\+1\.x\_\{t\+1\}=\(1\-c\_\{t\+1\}\)x\_\{t\}\+c\_\{t\+1\}z\_\{t\+1\},\\qquad c\_\{t\+1\}=\\frac\{1\}\{t\+1\}\.\(56\)Hereztz\_\{t\}is the base iterate,xtx\_\{t\}is the returned iterate, andyty\_\{t\}is the point at which the stochastic gradient is computed\. The parameterβ∈\[0,1\]\\beta\\in\[0,1\]interpolates between Polyak\-Ruppert averaging \(β=0\\beta=0\) and primal averaging \(β=1\\beta=1\)\.

##### Implicit decay of the effective learning rate\.

The averaging update can be rewritten as

xt\+1−xt=ct\+1​\(zt\+1−xt\)=ct\+1​\(zt−xt\)−ct\+1​η​gt\.x\_\{t\+1\}\-x\_\{t\}=c\_\{t\+1\}\(z\_\{t\+1\}\-x\_\{t\}\)=c\_\{t\+1\}\(z\_\{t\}\-x\_\{t\}\)\-c\_\{t\+1\}\\eta g\_\{t\}\.\(57\)Using

yt=β​xt\+\(1−β\)​zt⟹zt−xt=yt−xt1−β,y\_\{t\}=\\beta x\_\{t\}\+\(1\-\\beta\)z\_\{t\}\\qquad\\Longrightarrow\\qquad z\_\{t\}\-x\_\{t\}=\\frac\{y\_\{t\}\-x\_\{t\}\}\{1\-\\beta\},\(58\)we obtain

xt\+1−xt=ct\+11−β​\(yt−xt\)−ct\+1​η​gt\.x\_\{t\+1\}\-x\_\{t\}=\\frac\{c\_\{t\+1\}\}\{1\-\\beta\}\(y\_\{t\}\-x\_\{t\}\)\-c\_\{t\+1\}\\eta g\_\{t\}\.\(59\)Thus the returned iteratextx\_\{t\}moves as though it were taking a gradient step with*effective*learning rate

ηteff≈ct\+1​η=ηt\+1\.\\eta\_\{t\}^\{\\mathrm\{eff\}\}\\approx c\_\{t\+1\}\\eta=\\frac\{\\eta\}\{t\+1\}\.\(60\)Thus, the effective learning rate decays like1/t1/teven though the base update onztz\_\{t\}uses a constant step size\. This is the basic mechanism by which schedule\-free methods replace an explicit schedule\. In the convex Lipschitz setting, this implicit learning rate decay is sufficient to retain the optimal worst\-case convergence rate ofO​\(T−1/2\)O\(T^\{\-1/2\}\)\. For a convexGG\-Lipschitz function withD=‖x1−x⋆‖D=\\\|x\_\{1\}\-x\_\{\\star\}\\\|, the choice ofη=D/\(G​T\)\\eta=D/\(G\\sqrt\{T\}\)for anyβ∈\[0,1\]\\beta\\in\[0,1\]leads to

𝔼​\[F​\(xT\)−F​\(x⋆\)\]⩽D​GT\.\\mathbb\{E\}\\bigl\[F\(x\_\{T\}\)\-F\(x\_\{\\star\}\)\\bigr\]\\leqslant\\frac\{DG\}\{\\sqrt\{T\}\}\.\(61\)

##### SF\-AdamW\.

For deep\-learning applications, the raw gradient is replaced by a preconditioned update together with warmup and decoupled weight decay\.

Algorithm 2Schedule\-Free AdamW1:Input:base learning rate

η\\eta, interpolation parameter

β1\\beta\_\{1\}, variance parameter

β2\\beta\_\{2\}, weight decay

λ\\lambda, warmup steps

TwarmupT\_\{\\mathrm\{warmup\}\}, numerical constant

ε\\varepsilon, decay location

y~t∈\{yt,zt\}\\tilde\{y\}\_\{t\}\\in\\\{y\_\{t\},z\_\{t\}\\\}
2:for

t=1,2,…,Tt=1,2,\\dots,Tdo

3:

yt←\(1−β1\)​zt\+β1​xty\_\{t\}\\leftarrow\(1\-\\beta\_\{1\}\)z\_\{t\}\+\\beta\_\{1\}x\_\{t\}⊳\\trianglerightMomentum via interpolation

4:

gt←∇ℒ​\(yt;ξt\)g\_\{t\}\\leftarrow\\nabla\\mathcal\{L\}\(y\_\{t\};\\xi\_\{t\}\)⊳\\trianglerightGradient is evaluated atyty\_\{t\}

5:

vt←β2​vt−1\+\(1−β2\)​gt⊙2v\_\{t\}\\leftarrow\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}^\{\\odot 2\}
6:

ηt←η​1−β2t​min⁡\(1,t/Twarmup\)\\eta\_\{t\}\\leftarrow\\eta\\sqrt\{1\-\\beta\_\{2\}^\{t\}\}\\,\\min\\\!\\left\(1,t/T\_\{\\mathrm\{warmup\}\}\\right\)⊳\\trianglerightWarmup and Adam bias\-correction

7:

zt\+1←zt−ηt​gt/\(vt\+ε\)−ηt​λ​y~tz\_\{t\+1\}\\leftarrow z\_\{t\}\-\\eta\_\{t\}\\,g\_\{t\}/\(\\sqrt\{v\_\{t\}\}\+\\varepsilon\)\-\\eta\_\{t\}\\lambda\\tilde\{y\}\_\{t\}
8:

st←st−1\+ηt2s\_\{t\}\\leftarrow s\_\{t\-1\}\+\\eta\_\{t\}^\{2\}
9:

ct\+1←ηt2/stc\_\{t\+1\}\\leftarrow\\eta\_\{t\}^\{2\}/s\_\{t\}
10:

xt\+1←\(1−ct\+1\)​xt\+ct\+1​zt\+1x\_\{t\+1\}\\leftarrow\(1\-c\_\{t\+1\}\)x\_\{t\}\+c\_\{t\+1\}z\_\{t\+1\}⊳\\trianglerightUpdate weighted iterate average

11:endfor

12:return

xTx\_\{T\}

The choicey~t=yt\\tilde\{y\}\_\{t\}=y\_\{t\}corresponds to applying weight decay at the gradient location, which matches the interpretation of decay as anℓ2\\ell\_\{2\}\-regularizer in the loss, whiley~t=zt\\tilde\{y\}\_\{t\}=z\_\{t\}applies decay at the fast iterate\. A useful practical feature is that SF\-AdamW does not require more memory than AdamW\. One storesztz\_\{t\}andyty\_\{t\}in memory, and reconstructs

xt=yt−\(1−β1\)​ztβ1x\_\{t\}=\\frac\{y\_\{t\}\-\(1\-\\beta\_\{1\}\)z\_\{t\}\}\{\\beta\_\{1\}\}\(62\)when evaluation is needed\. Note that unlike classical momentum, which introduces a separate momentum buffer, schedule\-free momentum is implemented through the interpolationyt=\(1−β\)​zt\+β​xty\_\{t\}=\(1\-\\beta\)z\_\{t\}\+\\beta x\_\{t\}in weight space\.

## Appendix CConvergence of Schedule\-Free Spectral optimization

We first state the iterations of the schedule\-free spectral optimization and the assumptions of Lipschitz smoothness and bounded diameter\.

##### Iterates\.

For parameters0⩽β<10\\leqslant\\beta<1,0⩽μ<10\\leqslant\\mu<1, stepsizeη\>0\\eta\>0, and batch sizeB⩾1B\\geqslant 1, consider

Yt\\displaystyle Y\_\{t\}=β​Xt\+\(1−β\)​Zt,\\displaystyle=\\beta X\_\{t\}\+\(1\-\\beta\)Z\_\{t\},\(63\)Gt\\displaystyle G\_\{t\}=1B​∑i=1B∇f​\(Yt;ξt,i\),\\displaystyle=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\nabla f\(Y\_\{t\};\\xi\_\{t,i\}\),\(64\)M0\\displaystyle M\_\{0\}=G0,Mt=μ​Mt−1\+\(1−μ\)​Gt\(t⩾1\),\\displaystyle=G\_\{0\},\\qquad M\_\{t\}=\\mu M\_\{t\-1\}\+\(1\-\\mu\)G\_\{t\}\\quad\(t\\geqslant 1\),\(65\)Pt\\displaystyle P\_\{t\}=polar​\(Mt\),\\displaystyle=\\mathrm\{polar\}\(M\_\{t\}\),\(66\)Zt\+1\\displaystyle Z\_\{t\+1\}=Zt−η​Pt,\\displaystyle=Z\_\{t\}\-\\eta P\_\{t\},\(67\)Xt\+1\\displaystyle X\_\{t\+1\}=tt\+1​Xt\+1t\+1​Zt\+1,X0=Z0\.\\displaystyle=\\frac\{t\}\{t\+1\}X\_\{t\}\+\\frac\{1\}\{t\+1\}Z\_\{t\+1\},\\qquad X\_\{0\}=Z\_\{0\}\.\(68\)
Assume there existsW⋆∈ℝm×nW^\{\\star\}\\in\\mathbb\{R\}^\{m\\times n\}such that

f​\(W⋆\)=f⋆,∇f​\(W⋆\)=0,f\(W^\{\\star\}\)=f^\{\\star\},\\qquad\\nabla f\(W^\{\\star\}\)=0,
and let

r:=min⁡\{m,n\},Δ:=f​\(Y0\)−f⋆\.r:=\\min\\\{m,n\\\},\\qquad\\Delta:=f\(Y\_\{0\}\)\-f^\{\\star\}\.
Throughout, the stochastic oracle is unbiased with bounded Frobenius variance:

𝔼\[Gt∣ℱt\]=∇f\(Yt\)=:gt,𝔼\[∥Gt−gt∥F2∣ℱt\]⩽σ2B\.\\mathbb\{E\}\[G\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=\\nabla f\(Y\_\{t\}\)=:g\_\{t\},\\qquad\\mathbb\{E\}\\\!\\left\[\\\|G\_\{t\}\-g\_\{t\}\\\|\_\{F\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leqslant\\frac\{\\sigma^\{2\}\}\{B\}\.\(69\)

##### Smoothness assumptions\.

For the spectral proof, we assume there exists a constantLSL\_\{S\}such that

‖∇f​\(A\)−∇f​\(B\)‖∗⩽LS​‖A−B‖opfor all​A,B∈ℝm×n\.\\\|\\nabla f\(A\)\-\\nabla f\(B\)\\\|\_\{\*\}\\leqslant L\_\{S\}\\\|A\-B\\\|\_\{\\mathrm\{op\}\}\\qquad\\text\{for all \}A,B\\in\\mathbb\{R\}^\{m\\times n\}\.\(70\)
For the Frobenius proof, we assume there exists a constantLFL\_\{F\}such that

‖∇f​\(A\)−∇f​\(B\)‖F⩽LF​‖A−B‖Ffor all​A,B∈ℝm×n\.\\\|\\nabla f\(A\)\-\\nabla f\(B\)\\\|\_\{F\}\\leqslant L\_\{F\}\\\|A\-B\\\|\_\{F\}\\qquad\\text\{for all \}A,B\\in\\mathbb\{R\}^\{m\\times n\}\.\(71\)

##### Bounded diameter assumptions\.

For the spectral proof, we assume there exists a constantDS\>0D\_\{S\}\>0such that

‖W⋆‖op⩽DS/2,‖Zt‖op⩽DS/2for all​t,\\\|W^\{\\star\}\\\|\_\{\\mathrm\{op\}\}\\leqslant D\_\{S\}/2,\\qquad\\\|Z\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\leqslant D\_\{S\}/2\\qquad\\text\{for all \}t,\(72\)while for the Frobenius proof, we assume there exists a constantDF\>0D\_\{F\}\>0such that

‖W⋆‖F⩽DF/2,‖Zt‖F⩽DF/2for all​t\.\\\|W^\{\\star\}\\\|\_\{F\}\\leqslant D\_\{F\}/2,\\qquad\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant D\_\{F\}/2\\qquad\\text\{for all \}t\.\(73\)
Consequently,‖Xt‖op,‖Yt‖op⩽DS/2\\\|X\_\{t\}\\\|\_\{\\mathrm\{op\}\},\\\|Y\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\leqslant D\_\{S\}/2for alltt, and hence

‖Yt−W⋆‖op⩽DS,‖Zt−W⋆‖op⩽DSfor all​t,\\\|Y\_\{t\}\-W^\{\\star\}\\\|\_\{\\mathrm\{op\}\}\\leqslant D\_\{S\},\\qquad\\\|Z\_\{t\}\-W^\{\\star\}\\\|\_\{\\mathrm\{op\}\}\\leqslant D\_\{S\}\\qquad\\text\{for all \}t,and a similar statement holds for the Frobenius norm\.

These are standard bounded\-iterate assumptions \(see Theorem 7 in\[[25](https://arxiv.org/html/2605.23061#bib.bib50)\]and Theorem 1 in\[[4](https://arxiv.org/html/2605.23061#bib.bib64)\]\), and they can be enforced algorithmically by considering a projected variant that keep the fast iteratesZtZ\_\{t\}inside the corresponding norm ball\. The iteratesX,YX,Yare unweighted and weighted averages of current and previousZZiterates and hence remain within the ball\.

### C\.1Reusable identities and bounds

###### Lemma C\.1\(Iterate identities\)\.

Define

St:=Zt−Yt\.S\_\{t\}:=Z\_\{t\}\-Y\_\{t\}\.ThenSt=β​\(Zt−Xt\)S\_\{t\}=\\beta\(Z\_\{t\}\-X\_\{t\}\), and for everyt⩾0t\\geqslant 0,

St\+1\\displaystyle S\_\{t\+1\}=tt\+1​\(St−β​η​Pt\),\\displaystyle=\\frac\{t\}\{t\+1\}\\bigl\(S\_\{t\}\-\\beta\\eta P\_\{t\}\\bigr\),\(74\)Yt\+1−Yt\\displaystyle Y\_\{t\+1\}\-Y\_\{t\}=Stt\+1−η​\(1−β\+βt\+1\)​Pt\.\\displaystyle=\\frac\{S\_\{t\}\}\{t\+1\}\-\\eta\\left\(1\-\\beta\+\\frac\{\\beta\}\{t\+1\}\\right\)P\_\{t\}\.\(75\)Consequently, if∥⋅∥⋄\\\|\\cdot\\\|\_\{\\diamond\}is any norm for which‖Yt−W⋆‖⋄⩽D⋄\\\|Y\_\{t\}\-W^\{\\star\}\\\|\_\{\\diamond\}\\leqslant D\_\{\\diamond\}and‖Zt−W⋆‖⋄⩽D⋄\\\|Z\_\{t\}\-W^\{\\star\}\\\|\_\{\\diamond\}\\leqslant D\_\{\\diamond\}, then

‖St‖⋄⩽2​D⋄,‖Yt\+1−Yt‖⋄⩽2​D⋄t\+1\+η​‖Pt‖⋄\.\\\|S\_\{t\}\\\|\_\{\\diamond\}\\leqslant 2D\_\{\\diamond\},\\qquad\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{\\diamond\}\\leqslant\\frac\{2D\_\{\\diamond\}\}\{t\+1\}\+\\eta\\\|P\_\{t\}\\\|\_\{\\diamond\}\.\(76\)In particular,

‖Yt\+1−Yt‖op\\displaystyle\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{\\mathrm\{op\}\}⩽2​DSt\+1\+η,\\displaystyle\\leqslant\\frac\{2D\_\{S\}\}\{t\+1\}\+\\eta,\(77\)‖Yt\+1−Yt‖F\\displaystyle\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{F\}⩽2​DFt\+1\+η​r\.\\displaystyle\\leqslant\\frac\{2D\_\{F\}\}\{t\+1\}\+\\eta\\sqrt\{r\}\.\(78\)

###### Proof\.

SinceYt=β​Xt\+\(1−β\)​ZtY\_\{t\}=\\beta X\_\{t\}\+\(1\-\\beta\)Z\_\{t\}, we have

St=Zt−Yt=β​\(Zt−Xt\)\.S\_\{t\}=Z\_\{t\}\-Y\_\{t\}=\\beta\(Z\_\{t\}\-X\_\{t\}\)\.Using \([67](https://arxiv.org/html/2605.23061#A3.E67)\) and \([68](https://arxiv.org/html/2605.23061#A3.E68)\),

St\+1\\displaystyle S\_\{t\+1\}=Zt\+1−Yt\+1\\displaystyle=Z\_\{t\+1\}\-Y\_\{t\+1\}=β​\(Zt\+1−Xt\+1\)\\displaystyle=\\beta\(Z\_\{t\+1\}\-X\_\{t\+1\}\)=β​\(Zt\+1−tt\+1​Xt−1t\+1​Zt\+1\)\\displaystyle=\\beta\\left\(Z\_\{t\+1\}\-\\frac\{t\}\{t\+1\}X\_\{t\}\-\\frac\{1\}\{t\+1\}Z\_\{t\+1\}\\right\)=tt\+1​β​\(Zt\+1−Xt\)\\displaystyle=\\frac\{t\}\{t\+1\}\\beta\(Z\_\{t\+1\}\-X\_\{t\}\)=tt\+1​β​\(\(Zt−η​Pt\)−Xt\)\\displaystyle=\\frac\{t\}\{t\+1\}\\beta\\bigl\(\(Z\_\{t\}\-\\eta P\_\{t\}\)\-X\_\{t\}\\bigr\)=tt\+1​\(St−β​η​Pt\),\\displaystyle=\\frac\{t\}\{t\+1\}\\bigl\(S\_\{t\}\-\\beta\\eta P\_\{t\}\\bigr\),which proves \([74](https://arxiv.org/html/2605.23061#A3.E74)\)\. Then

Yt\+1−Yt\\displaystyle Y\_\{t\+1\}\-Y\_\{t\}=\(Zt\+1−St\+1\)−\(Zt−St\)\\displaystyle=\(Z\_\{t\+1\}\-S\_\{t\+1\}\)\-\(Z\_\{t\}\-S\_\{t\}\)=−η​Pt−St\+1\+St\\displaystyle=\-\\eta P\_\{t\}\-S\_\{t\+1\}\+S\_\{t\}=Stt\+1−η​\(1−β\+βt\+1\)​Pt,\\displaystyle=\\frac\{S\_\{t\}\}\{t\+1\}\-\\eta\\left\(1\-\\beta\+\\frac\{\\beta\}\{t\+1\}\\right\)P\_\{t\},which is \([75](https://arxiv.org/html/2605.23061#A3.E75)\)\. The norm bounds follow from

‖St‖⋄=‖Zt−Yt‖⋄⩽‖Zt−W⋆‖⋄\+‖Yt−W⋆‖⋄⩽2​D⋄,\\\|S\_\{t\}\\\|\_\{\\diamond\}=\\\|Z\_\{t\}\-Y\_\{t\}\\\|\_\{\\diamond\}\\leqslant\\\|Z\_\{t\}\-W^\{\\star\}\\\|\_\{\\diamond\}\+\\\|Y\_\{t\}\-W^\{\\star\}\\\|\_\{\\diamond\}\\leqslant 2D\_\{\\diamond\},together with‖Pt‖op⩽1\\\|P\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\leqslant 1and‖Pt‖F⩽r\\\|P\_\{t\}\\\|\_\{F\}\\leqslant\\sqrt\{r\}\. ∎

###### Lemma C\.2\(Polar alignment\)\.

For everyt⩾0t\\geqslant 0,

⟨gt,Pt⟩⩾‖gt‖∗−2​‖gt−Mt‖∗\.\\langle g\_\{t\},P\_\{t\}\\rangle\\geqslant\\\|g\_\{t\}\\\|\_\{\*\}\-2\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\.\(79\)

###### Proof\.

SincePt=polar​\(Mt\)P\_\{t\}=\\mathrm\{polar\}\(M\_\{t\}\),

⟨Mt,Pt⟩=‖Mt‖∗\.\\langle M\_\{t\},P\_\{t\}\\rangle=\\\|M\_\{t\}\\\|\_\{\*\}\.Also,

⟨gt−Mt,Pt⟩⩾−‖gt−Mt‖∗​‖Pt‖op⩾−‖gt−Mt‖∗\.\\langle g\_\{t\}\-M\_\{t\},P\_\{t\}\\rangle\\geqslant\-\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\\\|P\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\geqslant\-\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\.Therefore

⟨gt,Pt⟩=⟨Mt,Pt⟩\+⟨gt−Mt,Pt⟩⩾‖Mt‖∗−‖gt−Mt‖∗\.\\langle g\_\{t\},P\_\{t\}\\rangle=\\langle M\_\{t\},P\_\{t\}\\rangle\+\\langle g\_\{t\}\-M\_\{t\},P\_\{t\}\\rangle\\geqslant\\\|M\_\{t\}\\\|\_\{\*\}\-\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\.Finally,

‖Mt‖∗⩾‖gt‖∗−‖gt−Mt‖∗,\\\|M\_\{t\}\\\|\_\{\*\}\\geqslant\\\|g\_\{t\}\\\|\_\{\*\}\-\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\},which yields \([79](https://arxiv.org/html/2605.23061#A3.E79)\)\. ∎

###### Lemma C\.3\(Stochastic EMA error\)\.

Let

C0=g0,Ct=μ​Ct−1\+\(1−μ\)​gt,Et:=Ct−Mt\.C\_\{0\}=g\_\{0\},\\qquad C\_\{t\}=\\mu C\_\{t\-1\}\+\(1\-\\mu\)g\_\{t\},\\qquad E\_\{t\}:=C\_\{t\}\-M\_\{t\}\.Then

Et=μ​Et−1\+\(1−μ\)​\(gt−Gt\),E\_\{t\}=\\mu E\_\{t\-1\}\+\(1\-\\mu\)\(g\_\{t\}\-G\_\{t\}\),\(80\)and

1T​∑t=0T−1𝔼​‖Et‖∗⩽r​\(σ\(1−μ\)​T​B\+1−μ1\+μ​σB\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|E\_\{t\}\\\|\_\{\*\}\\leqslant\\sqrt\{r\}\\left\(\\frac\{\\sigma\}\{\(1\-\\mu\)T\\sqrt\{B\}\}\+\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\\frac\{\\sigma\}\{\\sqrt\{B\}\}\\right\)\.\(81\)

###### Proof\.

The recursion \([80](https://arxiv.org/html/2605.23061#A3.E80)\) is immediate from the definitions\. Using \([69](https://arxiv.org/html/2605.23061#A3.E69)\),

𝔼​\[‖Et‖F2∣ℱt\]⩽μ2​‖Et−1‖F2\+\(1−μ\)2​σ2B\.\\mathbb\{E\}\\\!\\left\[\\\|E\_\{t\}\\\|\_\{F\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leqslant\\mu^\{2\}\\\|E\_\{t\-1\}\\\|\_\{F\}^\{2\}\+\(1\-\\mu\)^\{2\}\\frac\{\\sigma^\{2\}\}\{B\}\.Solving this recursion and applying Jensen gives

𝔼​‖Et‖F⩽μt​σB\+1−μ1\+μ​σB\.\\mathbb\{E\}\\\|E\_\{t\}\\\|\_\{F\}\\leqslant\\mu^\{t\}\\frac\{\\sigma\}\{\\sqrt\{B\}\}\+\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\\frac\{\\sigma\}\{\\sqrt\{B\}\}\.Since‖A‖∗⩽r​‖A‖F\\\|A\\\|\_\{\*\}\\leqslant\\sqrt\{r\}\\,\\\|A\\\|\_\{F\}, averaging overt=0,…,T−1t=0,\\dots,T\-1yields \([81](https://arxiv.org/html/2605.23061#A3.E81)\)\. ∎

###### Lemma C\.4\(Tracking bound under spectral smoothness\)\.

Assume \([70](https://arxiv.org/html/2605.23061#A3.E70)\) and \([72](https://arxiv.org/html/2605.23061#A3.E72)\)\. Then

1T​∑t=0T−1𝔼​‖gt−Mt‖∗⩽μ​LS​η1−μ\+2​μ​LS​DS​log⁡\(e​T\)\(1−μ\)​T\+r​\(σ\(1−μ\)​T​B\+1−μ1\+μ​σB\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\\leqslant\\frac\{\\mu L\_\{S\}\\eta\}\{1\-\\mu\}\+\\frac\{2\\mu L\_\{S\}D\_\{S\}\\log\(eT\)\}\{\(1\-\\mu\)T\}\+\\sqrt\{r\}\\left\(\\frac\{\\sigma\}\{\(1\-\\mu\)T\\sqrt\{B\}\}\+\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\\frac\{\\sigma\}\{\\sqrt\{B\}\}\\right\)\.\(82\)

###### Proof\.

Write

gt−Mt=\(gt−Ct\)\+\(Ct−Mt\),At:=‖gt−Ct‖∗\.g\_\{t\}\-M\_\{t\}=\(g\_\{t\}\-C\_\{t\}\)\+\(C\_\{t\}\-M\_\{t\}\),\\qquad A\_\{t\}:=\\\|g\_\{t\}\-C\_\{t\}\\\|\_\{\*\}\.Then

At\\displaystyle A\_\{t\}=‖gt−μ​Ct−1−\(1−μ\)​gt‖∗\\displaystyle=\\\|g\_\{t\}\-\\mu C\_\{t\-1\}\-\(1\-\\mu\)g\_\{t\}\\\|\_\{\*\}=μ​‖gt−Ct−1‖∗\\displaystyle=\\mu\\\|g\_\{t\}\-C\_\{t\-1\}\\\|\_\{\*\}⩽μ​‖gt−gt−1‖∗\+μ​At−1\\displaystyle\\leqslant\\mu\\\|g\_\{t\}\-g\_\{t\-1\}\\\|\_\{\*\}\+\\mu A\_\{t\-1\}⩽μ​LS​‖Yt−Yt−1‖op\+μ​At−1\\displaystyle\\leqslant\\mu L\_\{S\}\\\|Y\_\{t\}\-Y\_\{t\-1\}\\\|\_\{\\mathrm\{op\}\}\+\\mu A\_\{t\-1\}⩽μ​LS​\(2​DSt\+η\)\+μ​At−1,\\displaystyle\\leqslant\\mu L\_\{S\}\\left\(\\frac\{2D\_\{S\}\}\{t\}\+\\eta\\right\)\+\\mu A\_\{t\-1\},where we used \([77](https://arxiv.org/html/2605.23061#A3.E77)\)\. Iterating and averaging gives

1T​∑t=0T−1At⩽μ​LS​η1−μ\+2​μ​LS​DS​log⁡\(e​T\)\(1−μ\)​T\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}A\_\{t\}\\leqslant\\frac\{\\mu L\_\{S\}\\eta\}\{1\-\\mu\}\+\\frac\{2\\mu L\_\{S\}D\_\{S\}\\log\(eT\)\}\{\(1\-\\mu\)T\}\.Combining this with Lemma[C\.3](https://arxiv.org/html/2605.23061#A3.Thmlemma3)proves \([82](https://arxiv.org/html/2605.23061#A3.E82)\)\. ∎

###### Lemma C\.5\(Tracking bound under Frobenius smoothness\)\.

Assume \([71](https://arxiv.org/html/2605.23061#A3.E71)\) and \([73](https://arxiv.org/html/2605.23061#A3.E73)\)\. Then

1T​∑t=0T−1𝔼​‖gt−Mt‖∗⩽μ​r​LF​η1−μ\+2​μ​r​LF​DF​log⁡\(e​T\)\(1−μ\)​T\+r​\(σ\(1−μ\)​T​B\+1−μ1\+μ​σB\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\\leqslant\\frac\{\\mu rL\_\{F\}\\eta\}\{1\-\\mu\}\+\\frac\{2\\mu\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{\(1\-\\mu\)T\}\+\\sqrt\{r\}\\left\(\\frac\{\\sigma\}\{\(1\-\\mu\)T\\sqrt\{B\}\}\+\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\\frac\{\\sigma\}\{\\sqrt\{B\}\}\\right\)\.\(83\)

###### Proof\.

Again write

gt−Mt=\(gt−Ct\)\+\(Ct−Mt\),At:=‖gt−Ct‖∗\.g\_\{t\}\-M\_\{t\}=\(g\_\{t\}\-C\_\{t\}\)\+\(C\_\{t\}\-M\_\{t\}\),\\qquad A\_\{t\}:=\\\|g\_\{t\}\-C\_\{t\}\\\|\_\{\*\}\.Then

At\\displaystyle A\_\{t\}⩽μ​‖gt−gt−1‖∗\+μ​At−1\\displaystyle\\leqslant\\mu\\\|g\_\{t\}\-g\_\{t\-1\}\\\|\_\{\*\}\+\\mu A\_\{t\-1\}⩽μ​r​‖gt−gt−1‖F\+μ​At−1\\displaystyle\\leqslant\\mu\\sqrt\{r\}\\,\\\|g\_\{t\}\-g\_\{t\-1\}\\\|\_\{F\}\+\\mu A\_\{t\-1\}⩽μ​r​LF​‖Yt−Yt−1‖F\+μ​At−1\\displaystyle\\leqslant\\mu\\sqrt\{r\}\\,L\_\{F\}\\\|Y\_\{t\}\-Y\_\{t\-1\}\\\|\_\{F\}\+\\mu A\_\{t\-1\}⩽μ​r​LF​\(2​DFt\+η​r\)\+μ​At−1,\\displaystyle\\leqslant\\mu\\sqrt\{r\}\\,L\_\{F\}\\left\(\\frac\{2D\_\{F\}\}\{t\}\+\\eta\\sqrt\{r\}\\right\)\+\\mu A\_\{t\-1\},where we used \([78](https://arxiv.org/html/2605.23061#A3.E78)\)\. Iterating and averaging gives

1T​∑t=0T−1At⩽μ​r​LF​η1−μ\+2​μ​r​LF​DF​log⁡\(e​T\)\(1−μ\)​T\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}A\_\{t\}\\leqslant\\frac\{\\mu rL\_\{F\}\\eta\}\{1\-\\mu\}\+\\frac\{2\\mu\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{\(1\-\\mu\)T\}\.Combining with Lemma[C\.3](https://arxiv.org/html/2605.23061#A3.Thmlemma3)proves \([83](https://arxiv.org/html/2605.23061#A3.E83)\)\. ∎

### C\.2Spectral\-smooth analysis

###### Theorem C\.1\(Stationarity at theYY\-iterates under spectral smoothness\)\.

Assume \([69](https://arxiv.org/html/2605.23061#A3.E69)\), \([70](https://arxiv.org/html/2605.23061#A3.E70)\), and \([72](https://arxiv.org/html/2605.23061#A3.E72)\)\. Then for everyT⩾1T\\geqslant 1,

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}Δ\+2​LS​DS2​\(log⁡\(e​T\)\+π26\)\(1−β\)​T​η\+LS​η2​\(1−β\)\+2​μ​LS​η\(1−β\)​\(1−μ\)\\displaystyle\\frac\{\\Delta\+2L\_\{S\}D\_\{S\}^\{2\}\\\!\\left\(\\log\(eT\)\+\\frac\{\\pi^\{2\}\}\{6\}\\right\)\}\{\(1\-\\beta\)T\\eta\}\+\\frac\{L\_\{S\}\\eta\}\{2\(1\-\\beta\)\}\+\\frac\{2\\mu L\_\{S\}\\eta\}\{\(1\-\\beta\)\(1\-\\mu\)\}\+2​LS​DS​log⁡\(e​T\)\(1−β\)​T​\(1\+2​μ1−μ\)\+2​r​σ\(1−β\)​B​1−μ1\+μ\\displaystyle\\quad\+\\frac\{2L\_\{S\}D\_\{S\}\\log\(eT\)\}\{\(1\-\\beta\)T\}\\left\(1\+\\frac\{2\\mu\}\{1\-\\mu\}\\right\)\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\beta\)\\sqrt\{B\}\}\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\+2​r​σ\(1−β\)​\(1−μ\)​T​B\.\\displaystyle\\quad\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\beta\)\(1\-\\mu\)T\\sqrt\{B\}\}\.\(84\)

###### Proof\.

By \([70](https://arxiv.org/html/2605.23061#A3.E70)\),

f​\(Yt\+1\)⩽f​\(Yt\)\+⟨gt,Yt\+1−Yt⟩\+LS2​‖Yt\+1−Yt‖op2\.f\(Y\_\{t\+1\}\)\\leqslant f\(Y\_\{t\}\)\+\\langle g\_\{t\},Y\_\{t\+1\}\-Y\_\{t\}\\rangle\+\\frac\{L\_\{S\}\}\{2\}\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{\\mathrm\{op\}\}^\{2\}\.Using Lemma[C\.1](https://arxiv.org/html/2605.23061#A3.Thmlemma1),

Yt\+1−Yt=Stt\+1−η​\(1−β\+βt\+1\)​Pt,Y\_\{t\+1\}\-Y\_\{t\}=\\frac\{S\_\{t\}\}\{t\+1\}\-\\eta\\left\(1\-\\beta\+\\frac\{\\beta\}\{t\+1\}\\right\)P\_\{t\},hence

f​\(Yt\+1\)⩽\\displaystyle f\(Y\_\{t\+1\}\)\\leqslant\{\}f​\(Yt\)\+1t\+1​⟨gt,St⟩−η​\(1−β\+βt\+1\)​⟨gt,Pt⟩\+LS2​‖Yt\+1−Yt‖op2\.\\displaystyle f\(Y\_\{t\}\)\+\\frac\{1\}\{t\+1\}\\langle g\_\{t\},S\_\{t\}\\rangle\-\\eta\\left\(1\-\\beta\+\\frac\{\\beta\}\{t\+1\}\\right\)\\langle g\_\{t\},P\_\{t\}\\rangle\+\\frac\{L\_\{S\}\}\{2\}\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{\\mathrm\{op\}\}^\{2\}\.Now‖gt‖∗⩽LS​DS\\\|g\_\{t\}\\\|\_\{\*\}\\leqslant L\_\{S\}D\_\{S\},‖St‖op⩽2​DS\\\|S\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\leqslant 2D\_\{S\},‖Yt\+1−Yt‖op⩽2​DS/\(t\+1\)\+η\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\leqslant 2D\_\{S\}/\(t\+1\)\+\\eta, and Lemma[C\.2](https://arxiv.org/html/2605.23061#A3.Thmlemma2)gives

⟨gt,Pt⟩⩾‖gt‖∗−2​‖gt−Mt‖∗\.\\langle g\_\{t\},P\_\{t\}\\rangle\\geqslant\\\|g\_\{t\}\\\|\_\{\*\}\-2\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\.Therefore

f​\(Yt\+1\)⩽f​\(Yt\)−η​\(1−β\)​‖gt‖∗\+2​η​‖gt−Mt‖∗\+2​LS​DS2t\+1\+LS2​\(2​DSt\+1\+η\)2\.f\(Y\_\{t\+1\}\)\\leqslant f\(Y\_\{t\}\)\-\\eta\(1\-\\beta\)\\\|g\_\{t\}\\\|\_\{\*\}\+2\\eta\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\+\\frac\{2L\_\{S\}D\_\{S\}^\{2\}\}\{t\+1\}\+\\frac\{L\_\{S\}\}\{2\}\\left\(\\frac\{2D\_\{S\}\}\{t\+1\}\+\\eta\\right\)^\{2\}\.\(85\)Summing \([85](https://arxiv.org/html/2605.23061#A3.E85)\) fromt=0t=0toT−1T\-1, taking expectations, and usingf​\(YT\)⩾f⋆f\(Y\_\{T\}\)\\geqslant f^\{\\star\}, we obtain

η​\(1−β\)​∑t=0T−1𝔼​‖gt‖∗⩽\\displaystyle\\eta\(1\-\\beta\)\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|g\_\{t\}\\\|\_\{\*\}\\leqslant\{\}Δ\+2​η​∑t=0T−1𝔼​‖gt−Mt‖∗\+2​LS​DS2​∑t=0T−11t\+1\\displaystyle\\Delta\+2\\eta\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\+2L\_\{S\}D\_\{S\}^\{2\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{t\+1\}\+LS2​∑t=0T−1\(2​DSt\+1\+η\)2\.\\displaystyle\\quad\+\\frac\{L\_\{S\}\}\{2\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\frac\{2D\_\{S\}\}\{t\+1\}\+\\eta\\right\)^\{2\}\.Using

∑t=0T−11t\+1⩽log⁡\(e​T\),∑t=0T−11\(t\+1\)2⩽π26,\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{t\+1\}\\leqslant\\log\(eT\),\\qquad\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{\(t\+1\)^\{2\}\}\\leqslant\\frac\{\\pi^\{2\}\}\{6\},and substituting Lemma[C\.4](https://arxiv.org/html/2605.23061#A3.Thmlemma4)yields \([84](https://arxiv.org/html/2605.23061#A3.E84)\)\. ∎

### C\.3Frobenius\-smooth analysis

###### Theorem C\.2\(Stationarity at theYY\-iterates under Frobenius smoothness\)\.

Assume \([69](https://arxiv.org/html/2605.23061#A3.E69)\), \([71](https://arxiv.org/html/2605.23061#A3.E71)\), and \([73](https://arxiv.org/html/2605.23061#A3.E73)\)\. Then for everyT⩾1T\\geqslant 1,

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}Δ\+2​LF​DF2​\(log⁡\(e​T\)\+π26\)\(1−β\)​T​η\+LF​r​η2​\(1−β\)\+2​μ​LF​r​η\(1−β\)​\(1−μ\)\\displaystyle\\frac\{\\Delta\+2L\_\{F\}D\_\{F\}^\{2\}\\\!\\left\(\\log\(eT\)\+\\frac\{\\pi^\{2\}\}\{6\}\\right\)\}\{\(1\-\\beta\)T\\eta\}\+\\frac\{L\_\{F\}r\\,\\eta\}\{2\(1\-\\beta\)\}\+\\frac\{2\\mu L\_\{F\}r\\,\\eta\}\{\(1\-\\beta\)\(1\-\\mu\)\}\+2​r​LF​DF​log⁡\(e​T\)\(1−β\)​T​\(1\+2​μ1−μ\)\+2​r​σ\(1−β\)​B​1−μ1\+μ\\displaystyle\\quad\+\\frac\{2\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{\(1\-\\beta\)T\}\\left\(1\+\\frac\{2\\mu\}\{1\-\\mu\}\\right\)\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\beta\)\\sqrt\{B\}\}\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\+2​r​σ\(1−β\)​\(1−μ\)​T​B\.\\displaystyle\\quad\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\beta\)\(1\-\\mu\)T\\sqrt\{B\}\}\.\(86\)

###### Proof\.

By \([71](https://arxiv.org/html/2605.23061#A3.E71)\),

f​\(Yt\+1\)⩽f​\(Yt\)\+⟨gt,Yt\+1−Yt⟩\+LF2​‖Yt\+1−Yt‖F2\.f\(Y\_\{t\+1\}\)\\leqslant f\(Y\_\{t\}\)\+\\langle g\_\{t\},Y\_\{t\+1\}\-Y\_\{t\}\\rangle\+\\frac\{L\_\{F\}\}\{2\}\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{F\}^\{2\}\.Using Lemma[C\.1](https://arxiv.org/html/2605.23061#A3.Thmlemma1),

Yt\+1−Yt=Stt\+1−η​\(1−β\+βt\+1\)​Pt\.Y\_\{t\+1\}\-Y\_\{t\}=\\frac\{S\_\{t\}\}\{t\+1\}\-\\eta\\left\(1\-\\beta\+\\frac\{\\beta\}\{t\+1\}\\right\)P\_\{t\}\.Hence

f​\(Yt\+1\)⩽\\displaystyle f\(Y\_\{t\+1\}\)\\leqslant\{\}f​\(Yt\)\+1t\+1​⟨gt,St⟩−η​\(1−β\+βt\+1\)​⟨gt,Pt⟩\+LF2​‖Yt\+1−Yt‖F2\.\\displaystyle f\(Y\_\{t\}\)\+\\frac\{1\}\{t\+1\}\\langle g\_\{t\},S\_\{t\}\\rangle\-\\eta\\left\(1\-\\beta\+\\frac\{\\beta\}\{t\+1\}\\right\)\\langle g\_\{t\},P\_\{t\}\\rangle\+\\frac\{L\_\{F\}\}\{2\}\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{F\}^\{2\}\.Since‖gt‖F⩽LF​DF\\\|g\_\{t\}\\\|\_\{F\}\\leqslant L\_\{F\}D\_\{F\},‖St‖F⩽2​DF\\\|S\_\{t\}\\\|\_\{F\}\\leqslant 2D\_\{F\},‖Yt\+1−Yt‖F⩽2​DF/\(t\+1\)\+η​r\\\|Y\_\{t\+1\}\-Y\_\{t\}\\\|\_\{F\}\\leqslant 2D\_\{F\}/\(t\+1\)\+\\eta\\sqrt\{r\}, and Lemma[C\.2](https://arxiv.org/html/2605.23061#A3.Thmlemma2)gives

⟨gt,Pt⟩⩾‖gt‖∗−2​‖gt−Mt‖∗,\\langle g\_\{t\},P\_\{t\}\\rangle\\geqslant\\\|g\_\{t\}\\\|\_\{\*\}\-2\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\},we obtain

f​\(Yt\+1\)⩽f​\(Yt\)−η​\(1−β\)​‖gt‖∗\+2​η​‖gt−Mt‖∗\+2​LF​DF2t\+1\+LF2​\(2​DFt\+1\+η​r\)2\.f\(Y\_\{t\+1\}\)\\leqslant f\(Y\_\{t\}\)\-\\eta\(1\-\\beta\)\\\|g\_\{t\}\\\|\_\{\*\}\+2\\eta\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\+\\frac\{2L\_\{F\}D\_\{F\}^\{2\}\}\{t\+1\}\+\\frac\{L\_\{F\}\}\{2\}\\left\(\\frac\{2D\_\{F\}\}\{t\+1\}\+\\eta\\sqrt\{r\}\\right\)^\{2\}\.\(87\)Summing \([87](https://arxiv.org/html/2605.23061#A3.E87)\) fromt=0t=0toT−1T\-1, taking expectations, and usingf​\(YT\)⩾f⋆f\(Y\_\{T\}\)\\geqslant f^\{\\star\}, we get

η​\(1−β\)​∑t=0T−1𝔼​‖gt‖∗⩽\\displaystyle\\eta\(1\-\\beta\)\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|g\_\{t\}\\\|\_\{\*\}\\leqslant\{\}Δ\+2​η​∑t=0T−1𝔼​‖gt−Mt‖∗\+2​LF​DF2​∑t=0T−11t\+1\\displaystyle\\Delta\+2\\eta\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|g\_\{t\}\-M\_\{t\}\\\|\_\{\*\}\+2L\_\{F\}D\_\{F\}^\{2\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{t\+1\}\+LF2​∑t=0T−1\(2​DFt\+1\+η​r\)2\.\\displaystyle\\quad\+\\frac\{L\_\{F\}\}\{2\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\frac\{2D\_\{F\}\}\{t\+1\}\+\\eta\\sqrt\{r\}\\right\)^\{2\}\.Using

∑t=0T−11t\+1⩽log⁡\(e​T\),∑t=0T−11\(t\+1\)2⩽π26,\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{t\+1\}\\leqslant\\log\(eT\),\\qquad\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{\(t\+1\)^\{2\}\}\\leqslant\\frac\{\\pi^\{2\}\}\{6\},and substituting Lemma[C\.5](https://arxiv.org/html/2605.23061#A3.Thmlemma5)yields \([86](https://arxiv.org/html/2605.23061#A3.E86)\)\. ∎

### C\.4Corollaries with optimized hyperparameters

###### Corollary C\.1\(Tuned rates under spectral smoothness\)\.

Assume the hypotheses of Theorem[C\.1](https://arxiv.org/html/2605.23061#A3.Thmtheorem1), and fixβ∈\[0,1\)\\beta\\in\[0,1\)\.

1. \(i\)Noisy case\.There exist choicesμT∈\[0,1\)\\mu\_\{T\}\\in\[0,1\)andηT\>0\\eta\_\{T\}\>0such that 1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(\(log⁡TT\)1/4\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\left\(\\frac\{\\log T\}\{T\}\\right\)^\{1/4\}\\right\)\.\(88\)
2. \(ii\)Noiseless case\.Ifσ=0\\sigma=0, then there exist choicesμT∈\[0,1\)\\mu\_\{T\}\\in\[0,1\)andηT\>0\\eta\_\{T\}\>0such that 1T​∑t=0T−1‖∇f​\(Yt\)‖∗=O​\(log⁡TT\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\sqrt\{\\frac\{\\log T\}\{T\}\}\\right\)\.\(89\)
3. \(iii\)Log\-free specialization whenβ=0\\beta=0\.Ifβ=0\\beta=0, then the schedule\-free drift disappears, and the rates improve to 1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(T−1/4\)in the noisy case,\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(T^\{\-1/4\}\\right\)\\qquad\\text\{in the noisy case,\}\(90\)and 1T​∑t=0T−1‖∇f​\(Yt\)‖∗=O​\(T−1/2\)in the noiseless case\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(T^\{\-1/2\}\\right\)\\qquad\\text\{in the noiseless case\.\}\(91\)

###### Proof\.

Let

AT,S:=Δ\+2​LS​DS2​\(log⁡\(e​T\)\+π26\)\.A\_\{T,S\}:=\\Delta\+2L\_\{S\}D\_\{S\}^\{2\}\\\!\\left\(\\log\(eT\)\+\\frac\{\\pi^\{2\}\}\{6\}\\right\)\.ThenAT,S=O​\(log⁡T\)A\_\{T,S\}=O\(\\log T\)\.

For part \(i\), write

α:=1−μ∈\(0,1\]\.\\alpha:=1\-\\mu\\in\(0,1\]\.Then Theorem[C\.1](https://arxiv.org/html/2605.23061#A3.Thmtheorem1)becomes

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}11−β\[AT,ST​η\+LS​\(4−3​α\)2​αη\+2​LS​DS​log⁡\(e​T\)T2−αα\\displaystyle\\frac\{1\}\{1\-\\beta\}\\Biggl\[\\frac\{A\_\{T,S\}\}\{T\\eta\}\+\\frac\{L\_\{S\}\(4\-3\\alpha\)\}\{2\\alpha\}\\eta\+\\frac\{2L\_\{S\}D\_\{S\}\\log\(eT\)\}\{T\}\\frac\{2\-\\alpha\}\{\\alpha\}\+2​r​σBα2−α\+2​r​σα​T​B\]\.\\displaystyle\\hskip 92\.47145pt\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\}\{2\-\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha T\\sqrt\{B\}\}\\Biggr\]\.\(92\)For fixedα\\alpha, the exact optimizer inη\\etais

ηT,S⋆​\(α\)=2​AT,S​αLS​T​\(4−3​α\)\.\\eta\_\{T,S\}^\{\\star\}\(\\alpha\)=\\sqrt\{\\frac\{2A\_\{T,S\}\\alpha\}\{L\_\{S\}T\(4\-3\\alpha\)\}\}\.\(93\)Substituting \([93](https://arxiv.org/html/2605.23061#A3.E93)\) into \([92](https://arxiv.org/html/2605.23061#A3.E92)\) gives

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}11−β\[2​AT,S​LST⋅4−3​αα\+2​LS​DS​log⁡\(e​T\)T2−αα\\displaystyle\\frac\{1\}\{1\-\\beta\}\\Biggl\[\\sqrt\{\\frac\{2A\_\{T,S\}L\_\{S\}\}\{T\}\\cdot\\frac\{4\-3\\alpha\}\{\\alpha\}\}\+\\frac\{2L\_\{S\}D\_\{S\}\\log\(eT\)\}\{T\}\\frac\{2\-\\alpha\}\{\\alpha\}\+2​r​σBα2−α\+2​r​σα​T​B\]\.\\displaystyle\\hskip 92\.47145pt\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\}\{2\-\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha T\\sqrt\{B\}\}\\Biggr\]\.\(94\)
Choose

αT,S:=min⁡\{1,2​AT,S​LS​Br​σ2​T\},μT,S:=1−αT,S,ηT,S:=ηT,S⋆​\(αT,S\)\.\\alpha\_\{T,S\}:=\\min\\left\\\{1,\\;2\\sqrt\{\\frac\{A\_\{T,S\}L\_\{S\}B\}\{r\\sigma^\{2\}T\}\}\\right\\\},\\qquad\\mu\_\{T,S\}:=1\-\\alpha\_\{T,S\},\\qquad\\eta\_\{T,S\}:=\\eta\_\{T,S\}^\{\\star\}\(\\alpha\_\{T,S\}\)\.\(95\)This is the leading\-order optimal choice obtained by balancing the first and third terms in \([94](https://arxiv.org/html/2605.23061#A3.E94)\)\. With this choice,

2​AT,S​LST⋅4−3​αT,SαT,S=O​\(\(AT,ST\)1/4\),\\sqrt\{\\frac\{2A\_\{T,S\}L\_\{S\}\}\{T\}\\cdot\\frac\{4\-3\\alpha\_\{T,S\}\}\{\\alpha\_\{T,S\}\}\}=O\\\!\\left\(\\left\(\\frac\{A\_\{T,S\}\}\{T\}\\right\)^\{1/4\}\\right\),and

2​r​σB​αT,S2−αT,S=O​\(\(AT,ST\)1/4\)\.\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\_\{T,S\}\}\{2\-\\alpha\_\{T,S\}\}\}=O\\\!\\left\(\\left\(\\frac\{A\_\{T,S\}\}\{T\}\\right\)^\{1/4\}\\right\)\.The remaining terms are lower order:

2​LS​DS​log⁡\(e​T\)T​2−αT,SαT,S=O​\(log⁡TT\),2​r​σαT,S​T​B=O​\(1T​log⁡T\)\.\\frac\{2L\_\{S\}D\_\{S\}\\log\(eT\)\}\{T\}\\frac\{2\-\\alpha\_\{T,S\}\}\{\\alpha\_\{T,S\}\}=O\\\!\\left\(\\sqrt\{\\frac\{\\log T\}\{T\}\}\\right\),\\qquad\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha\_\{T,S\}T\\sqrt\{B\}\}=O\\\!\\left\(\\frac\{1\}\{\\sqrt\{T\\log T\}\}\\right\)\.Hence

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(\(AT,ST\)1/4\)=O​\(\(log⁡TT\)1/4\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\left\(\\frac\{A\_\{T,S\}\}\{T\}\\right\)^\{1/4\}\\right\)=O\\\!\\left\(\\left\(\\frac\{\\log T\}\{T\}\\right\)^\{1/4\}\\right\),which proves part \(i\)\.

For part \(ii\), whenσ=0\\sigma=0, the stochastic terms vanish from \([92](https://arxiv.org/html/2605.23061#A3.E92)\)\. The remaining bound is increasing inμ\\mu, hence minimized by

μT,S=0equivalentlyαT,S=1\.\\mu\_\{T,S\}=0\\qquad\\text\{equivalently\}\\qquad\\alpha\_\{T,S\}=1\.Then the exact optimizer inη\\etais

ηT,S=2​AT,SLS​T\.\\eta\_\{T,S\}=\\sqrt\{\\frac\{2A\_\{T,S\}\}\{L\_\{S\}T\}\}\.\(96\)SubstitutingμT,S=0\\mu\_\{T,S\}=0and \([96](https://arxiv.org/html/2605.23061#A3.E96)\) into Theorem[C\.1](https://arxiv.org/html/2605.23061#A3.Thmtheorem1)gives

1T​∑t=0T−1‖∇f​\(Yt\)‖∗⩽11−β​\[2​LS​AT,ST\+2​LS​DS​log⁡\(e​T\)T\]\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\frac\{1\}\{1\-\\beta\}\\left\[\\sqrt\{\\frac\{2L\_\{S\}A\_\{T,S\}\}\{T\}\}\+\\frac\{2L\_\{S\}D\_\{S\}\\log\(eT\)\}\{T\}\\right\]\.SinceAT,S=O​\(log⁡T\)A\_\{T,S\}=O\(\\log T\), this implies

1T​∑t=0T−1‖∇f​\(Yt\)‖∗=O​\(log⁡TT\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\sqrt\{\\frac\{\\log T\}\{T\}\}\\right\),which proves part \(ii\)\.

For part \(iii\), supposeβ=0\\beta=0\. ThenYt=ZtY\_\{t\}=Z\_\{t\}, hence

St=Zt−Yt=0for all​t,S\_\{t\}=Z\_\{t\}\-Y\_\{t\}=0\\qquad\\text\{for all \}t,and therefore

Yt\+1−Yt=−η​Pt\.Y\_\{t\+1\}\-Y\_\{t\}=\-\\eta P\_\{t\}\.Thus the harmonic\-drift term disappears from both the descent and tracking arguments\. Repeating the proof of Theorem[C\.1](https://arxiv.org/html/2605.23061#A3.Thmtheorem1)withSt≡0S\_\{t\}\\equiv 0gives

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}ΔT​η\+LS​η2\+2​μ​LS​η1−μ\+2​r​σB​1−μ1\+μ\+2​r​σ\(1−μ\)​T​B\.\\displaystyle\\frac\{\\Delta\}\{T\\eta\}\+\\frac\{L\_\{S\}\\eta\}\{2\}\+\\frac\{2\\mu L\_\{S\}\\eta\}\{1\-\\mu\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\mu\)T\\sqrt\{B\}\}\.\(97\)Writingα:=1−μ\\alpha:=1\-\\mu, this becomes

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽ΔT​η\+LS​\(4−3​α\)2​α​η\+2​r​σB​α2−α\+2​r​σα​T​B\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\frac\{\\Delta\}\{T\\eta\}\+\\frac\{L\_\{S\}\(4\-3\\alpha\)\}\{2\\alpha\}\\eta\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\}\{2\-\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha T\\sqrt\{B\}\}\.For fixedα\\alpha, the exact optimizer inη\\etais

η⋆=2​Δ​αLS​T​\(4−3​α\)\.\\eta^\{\\star\}=\\sqrt\{\\frac\{2\\Delta\\,\\alpha\}\{L\_\{S\}T\(4\-3\\alpha\)\}\}\.Substituting this back and balancing the first two leading terms gives

αT≍min⁡\{1,2​Δ​LS​Br​σ2​T\},μT:=1−αT,ηT:=2​Δ​αTLS​T​\(4−3​αT\)\.\\alpha\_\{T\}\\asymp\\min\\left\\\{1,\\;2\\sqrt\{\\frac\{\\Delta L\_\{S\}B\}\{r\\sigma^\{2\}T\}\}\\right\\\},\\qquad\\mu\_\{T\}:=1\-\\alpha\_\{T\},\\qquad\\eta\_\{T\}:=\\sqrt\{\\frac\{2\\Delta\\,\\alpha\_\{T\}\}\{L\_\{S\}T\(4\-3\\alpha\_\{T\}\)\}\}\.With this choice,

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(T−1/4\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\(T^\{\-1/4\}\),which proves the noisy claim in \([90](https://arxiv.org/html/2605.23061#A3.E90)\)\.

If in additionσ=0\\sigma=0, then \([97](https://arxiv.org/html/2605.23061#A3.E97)\) reduces to

1T​∑t=0T−1‖∇f​\(Yt\)‖∗⩽ΔT​η\+LS​η2\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\frac\{\\Delta\}\{T\\eta\}\+\\frac\{L\_\{S\}\\eta\}\{2\}\.Its exact optimizer is

μT=0,ηT=2​ΔLS​T\.\\mu\_\{T\}=0,\\qquad\\eta\_\{T\}=\\sqrt\{\\frac\{2\\Delta\}\{L\_\{S\}T\}\}\.Substituting yields

1T​∑t=0T−1‖∇f​\(Yt\)‖∗⩽2​LS​ΔT=O​\(T−1/2\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\sqrt\{\\frac\{2L\_\{S\}\\Delta\}\{T\}\}=O\(T^\{\-1/2\}\),which proves \([91](https://arxiv.org/html/2605.23061#A3.E91)\)\. ∎

###### Corollary C\.2\(Tuned rates under Frobenius smoothness\)\.

Assume the hypotheses of Theorem[C\.2](https://arxiv.org/html/2605.23061#A3.Thmtheorem2), and fixβ∈\[0,1\)\\beta\\in\[0,1\)\.

1. \(i\)Noisy case\.There exist choicesμT∈\[0,1\)\\mu\_\{T\}\\in\[0,1\)andηT\>0\\eta\_\{T\}\>0such that 1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(\(log⁡TT\)1/4\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\left\(\\frac\{\\log T\}\{T\}\\right\)^\{1/4\}\\right\)\.\(98\)
2. \(ii\)Noiseless case\.Ifσ=0\\sigma=0, then there exist choicesμT∈\[0,1\)\\mu\_\{T\}\\in\[0,1\)andηT\>0\\eta\_\{T\}\>0such that 1T​∑t=0T−1‖∇f​\(Yt\)‖∗=O​\(log⁡TT\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\sqrt\{\\frac\{\\log T\}\{T\}\}\\right\)\.\(99\)
3. \(iii\)Log\-free specialization whenβ=0\\beta=0\.Ifβ=0\\beta=0, then the schedule\-free drift disappears, and the rates improve to 1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(T−1/4\)in the noisy case,\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(T^\{\-1/4\}\\right\)\\qquad\\text\{in the noisy case,\}\(100\)and 1T​∑t=0T−1‖∇f​\(Yt\)‖∗=O​\(T−1/2\)in the noiseless case\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(T^\{\-1/2\}\\right\)\\qquad\\text\{in the noiseless case\.\}\(101\)

###### Proof\.

Let

AT,F:=Δ\+2​LF​DF2​\(log⁡\(e​T\)\+π26\)\.A\_\{T,F\}:=\\Delta\+2L\_\{F\}D\_\{F\}^\{2\}\\\!\\left\(\\log\(eT\)\+\\frac\{\\pi^\{2\}\}\{6\}\\right\)\.ThenAT,F=O​\(log⁡T\)A\_\{T,F\}=O\(\\log T\)\.

For part \(i\), write

α:=1−μ∈\(0,1\]\.\\alpha:=1\-\\mu\\in\(0,1\]\.Then Theorem[C\.2](https://arxiv.org/html/2605.23061#A3.Thmtheorem2)becomes

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}11−β\[AT,FT​η\+LF​r​\(4−3​α\)2​αη\+2​r​LF​DF​log⁡\(e​T\)T2−αα\\displaystyle\\frac\{1\}\{1\-\\beta\}\\Biggl\[\\frac\{A\_\{T,F\}\}\{T\\eta\}\+\\frac\{L\_\{F\}r\(4\-3\\alpha\)\}\{2\\alpha\}\\eta\+\\frac\{2\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{T\}\\frac\{2\-\\alpha\}\{\\alpha\}\+2​r​σBα2−α\+2​r​σα​T​B\]\.\\displaystyle\\hskip 92\.47145pt\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\}\{2\-\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha T\\sqrt\{B\}\}\\Biggr\]\.\(102\)For fixedα\\alpha, the exact optimizer inη\\etais

ηT,F⋆​\(α\)=2​AT,F​αLF​r​T​\(4−3​α\)\.\\eta\_\{T,F\}^\{\\star\}\(\\alpha\)=\\sqrt\{\\frac\{2A\_\{T,F\}\\alpha\}\{L\_\{F\}r\\,T\(4\-3\\alpha\)\}\}\.\(103\)Substituting \([103](https://arxiv.org/html/2605.23061#A3.E103)\) into \([102](https://arxiv.org/html/2605.23061#A3.E102)\) gives

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}11−β\[2​AT,F​LF​rT⋅4−3​αα\+2​r​LF​DF​log⁡\(e​T\)T2−αα\\displaystyle\\frac\{1\}\{1\-\\beta\}\\Biggl\[\\sqrt\{\\frac\{2A\_\{T,F\}L\_\{F\}r\}\{T\}\\cdot\\frac\{4\-3\\alpha\}\{\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{T\}\\frac\{2\-\\alpha\}\{\\alpha\}\+2​r​σBα2−α\+2​r​σα​T​B\]\.\\displaystyle\\hskip 92\.47145pt\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\}\{2\-\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha T\\sqrt\{B\}\}\\Biggr\]\.\(104\)
Choose

αT,F:=min⁡\{1,2​AT,F​LF​Bσ2​T\},μT,F:=1−αT,F,ηT,F:=ηT,F⋆​\(αT,F\)\.\\alpha\_\{T,F\}:=\\min\\left\\\{1,\\;2\\sqrt\{\\frac\{A\_\{T,F\}L\_\{F\}B\}\{\\sigma^\{2\}T\}\}\\right\\\},\\qquad\\mu\_\{T,F\}:=1\-\\alpha\_\{T,F\},\\qquad\\eta\_\{T,F\}:=\\eta\_\{T,F\}^\{\\star\}\(\\alpha\_\{T,F\}\)\.\(105\)This is the leading\-order optimal choice obtained by balancing the first and third terms in \([104](https://arxiv.org/html/2605.23061#A3.E104)\)\. With this choice,

2​AT,F​LF​rT⋅4−3​αT,FαT,F=O​\(\(AT,FT\)1/4\),\\sqrt\{\\frac\{2A\_\{T,F\}L\_\{F\}r\}\{T\}\\cdot\\frac\{4\-3\\alpha\_\{T,F\}\}\{\\alpha\_\{T,F\}\}\}=O\\\!\\left\(\\left\(\\frac\{A\_\{T,F\}\}\{T\}\\right\)^\{1/4\}\\right\),and

2​r​σB​αT,F2−αT,F=O​\(\(AT,FT\)1/4\)\.\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\_\{T,F\}\}\{2\-\\alpha\_\{T,F\}\}\}=O\\\!\\left\(\\left\(\\frac\{A\_\{T,F\}\}\{T\}\\right\)^\{1/4\}\\right\)\.The remaining terms are lower order:

2​r​LF​DF​log⁡\(e​T\)T​2−αT,FαT,F=O​\(log⁡TT\),2​r​σαT,F​T​B=O​\(1T​log⁡T\)\.\\frac\{2\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{T\}\\frac\{2\-\\alpha\_\{T,F\}\}\{\\alpha\_\{T,F\}\}=O\\\!\\left\(\\sqrt\{\\frac\{\\log T\}\{T\}\}\\right\),\\qquad\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha\_\{T,F\}T\\sqrt\{B\}\}=O\\\!\\left\(\\frac\{1\}\{\\sqrt\{T\\log T\}\}\\right\)\.Hence

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(\(AT,FT\)1/4\)=O​\(\(log⁡TT\)1/4\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\left\(\\frac\{A\_\{T,F\}\}\{T\}\\right\)^\{1/4\}\\right\)=O\\\!\\left\(\\left\(\\frac\{\\log T\}\{T\}\\right\)^\{1/4\}\\right\),which proves part \(i\)\.

For part \(ii\), whenσ=0\\sigma=0, the stochastic terms vanish from \([102](https://arxiv.org/html/2605.23061#A3.E102)\)\. The remaining bound is increasing inμ\\mu, hence minimized by

μT,F=0equivalentlyαT,F=1\.\\mu\_\{T,F\}=0\\qquad\\text\{equivalently\}\\qquad\\alpha\_\{T,F\}=1\.Then the exact optimizer inη\\etais

ηT,F=2​AT,FLF​r​T\.\\eta\_\{T,F\}=\\sqrt\{\\frac\{2A\_\{T,F\}\}\{L\_\{F\}r\\,T\}\}\.\(106\)SubstitutingμT,F=0\\mu\_\{T,F\}=0and \([106](https://arxiv.org/html/2605.23061#A3.E106)\) into Theorem[C\.2](https://arxiv.org/html/2605.23061#A3.Thmtheorem2)gives

1T​∑t=0T−1‖∇f​\(Yt\)‖∗⩽11−β​\[2​LF​r​AT,FT\+2​r​LF​DF​log⁡\(e​T\)T\]\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\frac\{1\}\{1\-\\beta\}\\left\[\\sqrt\{\\frac\{2L\_\{F\}r\\,A\_\{T,F\}\}\{T\}\}\+\\frac\{2\\sqrt\{r\}\\,L\_\{F\}D\_\{F\}\\log\(eT\)\}\{T\}\\right\]\.SinceAT,F=O​\(log⁡T\)A\_\{T,F\}=O\(\\log T\), this implies

1T​∑t=0T−1‖∇f​\(Yt\)‖∗=O​\(log⁡TT\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\\\!\\left\(\\sqrt\{\\frac\{\\log T\}\{T\}\}\\right\),which proves part \(ii\)\.

For part \(iii\), supposeβ=0\\beta=0\. ThenYt=ZtY\_\{t\}=Z\_\{t\}, so

St=0andYt\+1−Yt=−η​Pt\.S\_\{t\}=0\\qquad\\text\{and\}\\qquad Y\_\{t\+1\}\-Y\_\{t\}=\-\\eta P\_\{t\}\.Thus the harmonic\-drift term disappears from both the descent and tracking arguments\. Repeating the proof of Theorem[C\.2](https://arxiv.org/html/2605.23061#A3.Thmtheorem2)withSt≡0S\_\{t\}\\equiv 0gives

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\{\}ΔT​η\+LF​r​η2\+2​μ​LF​r​η1−μ\+2​r​σB​1−μ1\+μ\+2​r​σ\(1−μ\)​T​B\.\\displaystyle\\frac\{\\Delta\}\{T\\eta\}\+\\frac\{L\_\{F\}r\\,\\eta\}\{2\}\+\\frac\{2\\mu L\_\{F\}r\\,\\eta\}\{1\-\\mu\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{1\-\\mu\}\{1\+\\mu\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\(1\-\\mu\)T\\sqrt\{B\}\}\.\(107\)Writingα:=1−μ\\alpha:=1\-\\mu, this becomes

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗⩽ΔT​η\+LF​r​\(4−3​α\)2​α​η\+2​r​σB​α2−α\+2​r​σα​T​B\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\frac\{\\Delta\}\{T\\eta\}\+\\frac\{L\_\{F\}r\(4\-3\\alpha\)\}\{2\\alpha\}\\eta\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\sqrt\{B\}\}\\sqrt\{\\frac\{\\alpha\}\{2\-\\alpha\}\}\+\\frac\{2\\sqrt\{r\}\\,\\sigma\}\{\\alpha T\\sqrt\{B\}\}\.For fixedα\\alpha, the exact optimizer inη\\etais

η⋆=2​Δ​αLF​r​T​\(4−3​α\)\.\\eta^\{\\star\}=\\sqrt\{\\frac\{2\\Delta\\,\\alpha\}\{L\_\{F\}r\\,T\(4\-3\\alpha\)\}\}\.Substituting this back and balancing the first two leading terms gives

αT≍min⁡\{1,2​Δ​LF​Bσ2​T\},μT:=1−αT,ηT:=2​Δ​αTLF​r​T​\(4−3​αT\)\.\\alpha\_\{T\}\\asymp\\min\\left\\\{1,\\;2\\sqrt\{\\frac\{\\Delta L\_\{F\}B\}\{\\sigma^\{2\}T\}\}\\right\\\},\\qquad\\mu\_\{T\}:=1\-\\alpha\_\{T\},\\qquad\\eta\_\{T\}:=\\sqrt\{\\frac\{2\\Delta\\,\\alpha\_\{T\}\}\{L\_\{F\}r\\,T\(4\-3\\alpha\_\{T\}\)\}\}\.With this choice,

1T​∑t=0T−1𝔼​‖∇f​\(Yt\)‖∗=O​\(T−1/4\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}=O\(T^\{\-1/4\}\),which proves \([100](https://arxiv.org/html/2605.23061#A3.E100)\)\.

If in additionσ=0\\sigma=0, then \([107](https://arxiv.org/html/2605.23061#A3.E107)\) reduces to

1T​∑t=0T−1‖∇f​\(Yt\)‖∗⩽ΔT​η\+LF​r​η2\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\frac\{\\Delta\}\{T\\eta\}\+\\frac\{L\_\{F\}r\\,\\eta\}\{2\}\.Its exact optimizer is

μT=0,ηT=2​ΔLF​r​T\.\\mu\_\{T\}=0,\\qquad\\eta\_\{T\}=\\sqrt\{\\frac\{2\\Delta\}\{L\_\{F\}r\\,T\}\}\.Substituting yields

1T​∑t=0T−1‖∇f​\(Yt\)‖∗⩽2​LF​r​ΔT=O​\(T−1/2\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\\|\\nabla f\(Y\_\{t\}\)\\\|\_\{\*\}\\leqslant\\sqrt\{\\frac\{2L\_\{F\}r\\,\\Delta\}\{T\}\}=O\(T^\{\-1/2\}\),which proves \([101](https://arxiv.org/html/2605.23061#A3.E101)\)\. ∎

### C\.5Steady\-State Analysis for Weight Decay atZZ

In this subsection, we prove the boundedness and steady\-state characterization lemmas stated in[section˜3](https://arxiv.org/html/2605.23061#S3)\. For convenience, we restate the lemmas here\.

#### C\.5\.1Proof of[lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1)

Lemma[3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1)\(Restated\)\.Consider the updateZt\+1←Zt−η​λ​Zt−η^t​P^tZ\_\{t\+1\}\\leftarrow Z\_\{t\}\-\\eta\\lambda Z\_\{t\}\-\\hat\{\\eta\}\_\{t\}\\widehat\{P\}\_\{t\}and assume0<η​λ<10<\\eta\\lambda<1withη^=0\.2​η​m​n/‖P^t‖F\\hat\{\\eta\}=0\.2\\eta\\sqrt\{mn\}/\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}\. Then for alltt, we have

‖Zt‖F⩽‖Z0‖F​\(1−η​λ\)t\+0\.2​m​nλ⟹‖Zt‖F⩽0\.2​m​nλ​as​t→∞\.\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant\\\|Z\_\{0\}\\\|\_\{F\}\(1\-\\eta\\lambda\)^\{t\}\+\\frac\{0\.2\\sqrt\{mn\}\}\{\\lambda\}\\implies\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant\\frac\{0\.2\\sqrt\{mn\}\}\{\\lambda\}~\\textrm\{as\}~t\\to\\infty\.Furthermore, sinceXtX\_\{t\}andYtY\_\{t\}are convex combinations of\{Zs\}s⩽t\\\{Z\_\{s\}\\\}\_\{s\\leqslant t\}, they satisfy the same bound\.

###### Proof\.

Triangle inequality gives

‖Zt\+1‖F⩽\(1−η​λ\)​‖Zt‖F\+η^t​‖P^t‖F=\(1−η​λ\)​‖Zt‖F\+0\.2​η​m​n\.\\\|Z\_\{t\+1\}\\\|\_\{F\}\\leqslant\(1\-\\eta\\lambda\)\\\|Z\_\{t\}\\\|\_\{F\}\+\\hat\{\\eta\}\_\{t\}\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}=\(1\-\\eta\\lambda\)\\\|Z\_\{t\}\\\|\_\{F\}\+0\.2\\eta\\sqrt\{mn\}\.Unrolling and bounding the geometric sum:

‖Zt‖F⩽\(1−η​λ\)t​‖Z0‖F\+0\.2​η​m​n​∑k=0t−1\(1−η​λ\)k⩽\(1−η​λ\)t​‖Z0‖F\+0\.2​m​nλ\.\\\|Z\_\{t\}\\\|\_\{F\}\\leqslant\(1\-\\eta\\lambda\)^\{t\}\\\|Z\_\{0\}\\\|\_\{F\}\+0\.2\\eta\\sqrt\{mn\}\\sum\_\{k=0\}^\{t\-1\}\(1\-\\eta\\lambda\)^\{k\}\\leqslant\(1\-\\eta\\lambda\)^\{t\}\\\|Z\_\{0\}\\\|\_\{F\}\+\\frac\{0\.2\\sqrt\{mn\}\}\{\\lambda\}\.∎

#### C\.5\.2Proof of[lemma˜3\.2](https://arxiv.org/html/2605.23061#S3.Thmlemma2)

Lemma[3\.2](https://arxiv.org/html/2605.23061#S3.Thmlemma2)\(Quasi\-Steady\-State ofZZ, Restated\)\.Under the same conditions as[lemma˜3\.1](https://arxiv.org/html/2605.23061#S3.Thmlemma1), forη​λ≪1\\eta\\lambda\\ll 1the steady\-state satisfies

λ​ρt=0\.1​\[−αt\+αt2\+2​η​λ\],\\lambda\\rho\_\{t\}=0\.1\\Bigl\[\-\\alpha\_\{t\}\+\\sqrt\{\\alpha\_\{t\}^\{2\}\+2\\eta\\lambda\}\\Bigr\],whereρt:=‖Zt‖F/m​n\\rho\_\{t\}:=\\\|Z\_\{t\}\\\|\_\{F\}/\\sqrt\{mn\}is the RMS norm andαt:=⟨P^t,Zt⟩/\(‖P^t‖F​‖Zt‖F\)\\alpha\_\{t\}:=\\langle\\widehat\{P\}\_\{t\},Z\_\{t\}\\rangle/\(\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}\\\|Z\_\{t\}\\\|\_\{F\}\)is the alignment\.

###### Proof\.

Squaring the updateZt\+1=\(1−η​λ\)​Zt−η^t​P^tZ\_\{t\+1\}=\(1\-\\eta\\lambda\)Z\_\{t\}\-\\hat\{\\eta\}\_\{t\}\\widehat\{P\}\_\{t\}gives

‖Zt\+1‖F2\\displaystyle\\\|Z\_\{t\+1\}\\\|\_\{F\}^\{2\}=\(1−η​λ\)2​‖Zt‖F2−2​\(1−η​λ\)​η^t​⟨Zt,P^t⟩\+η^t2​‖P^t‖F2\.\\displaystyle=\(1\-\\eta\\lambda\)^\{2\}\\\|Z\_\{t\}\\\|\_\{F\}^\{2\}\-2\(1\-\\eta\\lambda\)\\hat\{\\eta\}\_\{t\}\\langle Z\_\{t\},\\widehat\{P\}\_\{t\}\\rangle\+\\hat\{\\eta\}\_\{t\}^\{2\}\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}^\{2\}\.\(108\)The NorMuon learning rateη^t=0\.2​η​m​n/‖P^t‖F\\hat\{\\eta\}\_\{t\}=0\.2\\eta\\sqrt\{mn\}/\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}ensures thatη^t​‖P^t‖F=0\.2​η​m​n\\hat\{\\eta\}\_\{t\}\\\|\\widehat\{P\}\_\{t\}\\\|\_\{F\}=0\.2\\eta\\sqrt\{mn\}is constant\. Substituting into \([108](https://arxiv.org/html/2605.23061#A3.E108)\):

‖Zt\+1‖F2=\(1−η​λ\)2​‖Zt‖F2−0\.4​η​\(1−η​λ\)​αt​m​n​‖Zt‖F\+0\.04​η2​m​n\.\\\|Z\_\{t\+1\}\\\|\_\{F\}^\{2\}=\(1\-\\eta\\lambda\)^\{2\}\\\|Z\_\{t\}\\\|\_\{F\}^\{2\}\-0\.4\\eta\(1\-\\eta\\lambda\)\\alpha\_\{t\}\\sqrt\{mn\}\\\|Z\_\{t\}\\\|\_\{F\}\+0\.04\\eta^\{2\}mn\.Dividing bym​nmn:

ρt\+12\\displaystyle\\rho\_\{t\+1\}^\{2\}=\(1−η​λ\)2​ρt2−0\.4​η​\(1−η​λ\)​αt​ρt\+0\.04​η2\.\\displaystyle=\(1\-\\eta\\lambda\)^\{2\}\\rho\_\{t\}^\{2\}\-0\.4\\eta\(1\-\\eta\\lambda\)\\alpha\_\{t\}\\rho\_\{t\}\+0\.04\\eta^\{2\}\.\(109\)In steady state,ρt\+1=ρt=:ρt\\rho\_\{t\+1\}=\\rho\_\{t\}=:\\rho\_\{t\}\. Rearranging \([109](https://arxiv.org/html/2605.23061#A3.E109)\):

ρt2​\[1−\(1−η​λ\)2\]=−0\.4​η​\(1−η​λ\)​αt​ρt\+0\.04​η2\.\\rho\_\{t\}^\{2\}\\bigl\[1\-\(1\-\\eta\\lambda\)^\{2\}\\bigr\]=\-0\.4\\eta\(1\-\\eta\\lambda\)\\alpha\_\{t\}\\rho\_\{t\}\+0\.04\\eta^\{2\}\.Forη​λ≪1\\eta\\lambda\\ll 1, we have1−\(1−η​λ\)2≈2​η​λ1\-\(1\-\\eta\\lambda\)^\{2\}\\approx 2\\eta\\lambdaand1−η​λ≈11\-\\eta\\lambda\\approx 1, giving the quadraticλ​ρt2\+0\.2​αt​ρt−0\.02​η=0\\lambda\\rho\_\{t\}^\{2\}\+0\.2\\alpha\_\{t\}\\rho\_\{t\}\-0\.02\\eta=0\. Taking the positive root:

λ​ρt=0\.1​\[−αt\+αt2\+2​η​λ\]\.\\lambda\\rho\_\{t\}=0\.1\\Bigl\[\-\\alpha\_\{t\}\+\\sqrt\{\\alpha\_\{t\}^\{2\}\+2\\eta\\lambda\}\\Bigr\]\.∎

## Appendix DArchitecture and Hyperparameter Details

This appendix provides complete details of the model architectures and hyperparameter sweeps used in our experiments\. All experiments were conducted on 8×\\timesNVIDIA A100\-40GB GPUs\[[12](https://arxiv.org/html/2605.23061#bib.bib72)\]using PyTorch 2\[[5](https://arxiv.org/html/2605.23061#bib.bib73)\]\.

##### Model Architecture

Table 2:Small and large Model architecture details\.We use a Llama\-style GPT architecture with rotary positional embeddings \(RoPE\), RMSNorm, and QK\-normalization in attention\. The MLP uses a squared ReLU activation\. Weight tying is applied between the token embedding and output head\.

##### Training Regimes

Table 3:Training budgets in steps and tokens for each model and regime\.We follow Chinchilla\-optimal training budgets\[[29](https://arxiv.org/html/2605.23061#bib.bib32)\], scaling the number of training steps proportionally with model size\. For the small model, we evaluate four training horizons denoted1×1\\times,2×2\\times,4×4\\times, and8×8\\timesChinchilla\-optimal\. For the large model, we train at1×1\\times,2×2\\times, and4×4\\timesChinchilla\-optimal\. Table[3](https://arxiv.org/html/2605.23061#A4.T3)summarizes the training budgets\.

##### Hyperparameter Search

We conduct extensive hyperparameter sweeps for each optimizer using the 125M model\. In particular, for AdamW we tuned the all hyper parameters except learning rate at2×2\\timesChinchilla but thelearning rate was tuned for each horizon, while for SF\-AdamW we tuned at2×2\\timesChinchilla and used the same configuration for runs up to8×8\\timesChinchilla\. Using the learning rate adjustment described in Section[2](https://arxiv.org/html/2605.23061#S2), SF\-NorMuon was able to use an identical learning rate and weight decay as SF\-AdamW\. Table[4](https://arxiv.org/html/2605.23061#A4.T4)summarizes the search space\.

Table 4:Hyperparameter search space for each optimizer\. Values inboldindicate the best configuration found\. The learning rate in bold is for AdamW corresponds to the2×2\\timestraining duration\.OptimizerHyperparameterValues SearchedAdamWLearning rate\{0\.002, 0\.004, 0\.006,0\.008, 0\.01\}β1\\beta\_\{1\}\{0\.9, 0\.95, 0\.99, 0\.995\}β2\\beta\_\{2\}\{0\.9,0\.95, 0\.99, 0\.995\}Weight decayλ\\lambda\{0, 0\.05,0\.1\}Warmup steps\{1000, 1500, 2000,2500\}SchedulerCosine decay to 0SF\-AdamWLearning rate\{0\.005,0\.008, 0\.01, 0\.012, 0\.015\}β1\\beta\_\{1\}\{0\.9,0\.95, 0\.99, 0\.995\}β2\\beta\_\{2\}\{0\.9, 0\.95,0\.99, 0\.995\}Weight decayλ\\lambda\{0,0\.05, 0\.1\}Warmup steps\{1500,2000, 2500\}Decay location\{𝐲\\mathbf\{y\},zz\}SF\-NorMuonLearning rate\{0\.005,0\.008, 0\.01, 0\.012, 0\.015\}β1\\beta\_\{1\}\{0\.9, 0\.95, 0\.99, 0\.995\}β2\\beta\_\{2\}\{0\.9,0\.95, 0\.99, 0\.995\}Momentumμ\\mu\{0\.8, 0\.9, 0\.95, 0\.99\}Weight decayλ\\lambda\{0,0\.05, 0\.1\}Warmup steps\{1500,2000, 2500\}Decay location\{yy,𝐳\\mathbf\{z\}\}Table[5](https://arxiv.org/html/2605.23061#A4.T5)reports the best hyperparameter configuration for each optimizer on the 125M model at8×8\\timesChinchilla\-optimal training\. These configurations were used for all experiments conducted for the large model\.

Table 5:Best hyperparameter configurations for each optimizer on the small model \(learning rate corresponds to8×8\\timesChinchilla for AdamW\)\.
##### Remarks\.

We make the following observations from our hyperparameter sweeps:

- •Momentum parameters:For SF\-AdamW,\(β1,β2\)=\(0\.95,0\.99\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.95,0\.99\)consistently outperforms the SF\-AdamW default of\(0\.9,0\.95\)\(0\.9,0\.95\)\. For SF\-NorMuon, we use separate beta values for 1D and 2D parameters, with the NorMuon\-specific momentumμ=0\.8\\mu=0\.8providing the best results\.
- •Decay location:For SF\-AdamW, applying weight decay at theyyiterate \(gradient evaluation point\) works best\. For SF\-NorMuon, applying decay at thezziterate \(base sequence\) achieves the lowest validation loss\.

##### Learning Rate Sweep Results

Figures[6](https://arxiv.org/html/2605.23061#A4.F6)–[5](https://arxiv.org/html/2605.23061#S4.F5)show the validation loss curves across different learning rates for each optimizer\. For all experiments, other hyperparameters are fixed at their optimal values from Table[5](https://arxiv.org/html/2605.23061#A4.T5)\.

![Refer to caption](https://arxiv.org/html/2605.23061v1/x6.png)Figure 6:Learning rate sweep for AdamW with cosine scheduling on the 125M model across four training horizons:1×1\\times\(5K steps\),2×2\\times\(10K steps\),4×4\\times\(20K steps\), and8×8\\times\(40K steps\) Chinchilla\-optimal\. The optimal learning rate isη=0\.008\\eta=0\.008for1×1\\times–4×4\\timesChinchilla, andη=0\.004\\eta=0\.004for8×8\\timesChinchilla, indicating that longer training benefits from smaller learning rates\.

## Appendix EAblation of SF\-NorMuon and Comparison with NorMuon

We conduct ablation experiments to validate the key design choices in SF\-NorMuon, and compare schedule\-free methods against NorMuon across multiple training horizons\.

### E\.1Ablation of SF\-NorMuon Components

In[fig\.˜3](https://arxiv.org/html/2605.23061#S2.F3), we conduct ablation experiments to validate the two key modifications that distinguish SF\-NorMuon from schedule\-free spectral descent: the explicit momentum buffer and the row\-wise adaptive normalization\. All experiments use the 125M parameter model trained for 8×\\timesChinchilla tokens \(≈\\approx21B tokens\) with decay atZZ\.

##### Explicit momentum\.

As discussed in §[2](https://arxiv.org/html/2605.23061#S2), the polar update treats all singular directions uniformly, which can be more aggressive than coordinate\-wise methods\. We therefore introduce an explicit momentum bufferMt=μ​Mt−1\+\(1−μ\)​GtM\_\{t\}=\\mu M\_\{t\-1\}\+\(1\-\\mu\)G\_\{t\}to smooth the gradient before computing its polar factor\. The left panel of[fig\.˜3](https://arxiv.org/html/2605.23061#S2.F3)compares SF\-NorMuon \(μ=0\.8\\mu=0\.8\) against the ablation with no explicit momentum \(μ=0\\mu=0\)\. Removing momentum increases the final validation loss from 3\.14 to 3\.28, a rather large degradation\. This confirms that explicit momentum is essential for stable, effective training with spectral updates\.

##### Row\-wise normalization\.

The polar factor can produce updates with high variance across neurons, as different rows of a weight matrix may have vastly different typical magnitudes\. Row\-wise adaptive normalization addresses this by maintaining a running average of squared row norms and normalizing accordingly\. The right panel of[fig\.˜3](https://arxiv.org/html/2605.23061#S2.F3)compares SF\-NorMuon against the variant without row normalization on a tighter y\-axis scale to reveal the small but consistent improvement\. SF\-NorMuon achieves a final loss of 3\.136 compared to 3\.142 for SF\-Muon, and reaches the same loss approximately 12% faster in terms of tokens processed\. This speedup is quite similar to the speedup achieved by NorMuon relative to Muon\[[49](https://arxiv.org/html/2605.23061#bib.bib58)\]\.

### E\.2Comparison with NorMuon

We compare schedule\-free methods \(SF\-NorMuon, SF\-AdamW\) against their scheduled counterparts \(NorMuon, AdamW\) across multiple training horizons \(1×\\times, 2×\\times, 4×\\times, 8×\\timesChinchilla\)\. All scheduled runs use a cosine learning rate schedule tuned for each horizon\.[table˜6](https://arxiv.org/html/2605.23061#A5.T6)reports the final validation loss for each method at each horizon, and[fig\.˜7](https://arxiv.org/html/2605.23061#A5.F7)visualizes the training curves\.

Table 6:Final validation loss at each training horizon \(125M model\)\. Scheduled methods \(NorMuon, AdamW\) use cosine decay tuned for each horizon; schedule\-free methods \(SF\-NorMuon, SF\-AdamW\) use the same hyperparameters throughout and can be evaluated at any checkpoint\.##### NorMuon vs AdamW\.

Across all horizons, NorMuon consistently outperforms AdamW\. This confirms the benefits of spectral\-norm geometry for matrix\-valued parameters as discussed in §[2](https://arxiv.org/html/2605.23061#S2), because the polar update makes uniform progress across all singular directions of the gradient rather than being dominated by large coordinate\-wise entries\.

![Refer to caption](https://arxiv.org/html/2605.23061v1/x7.png)Figure 7:Optimizer comparison on the 125M model\. Scheduled methods \(NorMuon, AdamW\) are shown with separate curves for each horizon \(1×\\times, 2×\\times, 4×\\times, 8×\\timesChinchilla\), while schedule\-free methods \(SF\-NorMuon, SF\-AdamW\) use a single run evaluated at any point\. SF\-NorMuon closely tracks scheduled NorMuon throughout training, demonstrating that the benefits of spectral\-norm optimization transfer effectively to the schedule\-free setting\.
##### SF\-NorMuon is competitive with scheduled NorMuon\.

Despite being an anytime optimizer that does not know the training horizon in advance, SF\-NorMuon closely tracks the performance of scheduled NorMuon\. At each horizon, the gap between SF\-NorMuon and NorMuon is only∼0\.02−0\.03\\sim 0\.02\-0\.03nats\. This is notable because scheduled methods benefit from the learning rate cool down at the end of training horizon\[[79](https://arxiv.org/html/2605.23061#bib.bib75)\], yet SF\-NorMuon remains competitive through implicit learning rate decay via online averaging\. In contrast, SF\-AdamW under performs both AdamW and also SF\-NorMuon substantially\.

## Appendix FReference PyTorch Implementation

Below we provide a self\-contained PyTorch implementation of SF\-NorMuon, closely following[algorithm˜1](https://arxiv.org/html/2605.23061#alg1)\. Our implementation is based on the code from\[[16](https://arxiv.org/html/2605.23061#bib.bib9),[33](https://arxiv.org/html/2605.23061#bib.bib47)\]\.

##### Implementation notes\.

- •Thezeropower\_via\_newtonschulz5function computes an approximation to the polar factorP=polar​\(M\)P=\\mathrm\{polar\}\(M\)using 5 iterations of the Newton\-Schulz method with coefficients\(a,b,c\)=\(3\.4445,−4\.7750,2\.0315\)\(a,b,c\)=\(3\.4445,\-4\.7750,2\.0315\)\. The function handles both tall and wide matrices by transposing when necessary\. There are other alternatives methods to accomplish the polar transformation which either use randomized methods\[[2](https://arxiv.org/html/2605.23061#bib.bib48),[1](https://arxiv.org/html/2605.23061#bib.bib80)\]or differ in the choice of coefficients\[[3](https://arxiv.org/html/2605.23061#bib.bib82)\]\.
- •In addition to the live weightsp\(the gradient evaluation sequenceYtY\_\{t\}\), the optimizer maintains three state variables per parameter:z\(the fast iterateZtZ\_\{t\}\),v\(the row\-wise second moment estimatevtv\_\{t\}\), andmom\(the momentum bufferMtM\_\{t\}\)\.
- •Theeval\(\)method computesXtX\_\{t\}on the fly usingZtZ\_\{t\}andYtY\_\{t\}\. Thetrain\(\)method restorespto theYtY\_\{t\}iterate for further training\.
- •Weight decay is applied to the fast iterateZtZ\_\{t\}, consistent with our analysis in[section˜3](https://arxiv.org/html/2605.23061#S3), and the learning rate scaling factoreta\_scale\(default 0\.2\) normalizes the effective step size to be comparable to Adam’s RMS\-normalized updates\.

1importmath,torch

2

3@torch\.compile

4defzeropower\_via\_newtonschulz5\(G,steps=5,eps=1e\-7\):

5assertlen\(G\.shape\)==2

6a,b,c=\(3\.4445,\-4\.7750,2\.0315\)

7X=G\.bfloat16\(\)

8X/=\(X\.norm\(\)\+eps\)

9ifG\.size\(0\)\>G\.size\(1\):

10X=X\.T

11for\_inrange\(steps\):

12A=X@X\.T

13B=A@X

14X=a\*X\+b\*B\+c\*A@B

15returnX\.T

16for\_inrange\(steps\):

17A=X@X\.T

18B=A@X

19X=a\*X\+b\*B\+c\*A@B

20returnX

21

22classNorMuonScheduleFree\(torch\.optim\.Optimizer\):

23def\_\_init\_\_\(self,params,lr=0\.005,betas=\(0\.9,0\.95\),momentum=0\.8,

24eps=1e\-8,weight\_decay=0\.1,warmup\_steps=2000,eta\_scale=0\.2\):

25defaults=dict\(lr=lr,betas=betas,momentum=momentum,eps=eps,

26weight\_decay=weight\_decay,warmup\_steps=warmup\_steps,

27eta\_scale=eta\_scale,k=0,train\_mode=False,weight\_sum=0\.0\)

28super\(\)\.\_\_init\_\_\(params,defaults\)

29

30@torch\.no\_grad\(\)

31defeval\(self\):

32forgroupinself\.param\_groups:

33ifgroup\["train\_mode"\]:

34beta=group\["betas"\]\[0\]

35forpingroup\["params"\]:

36state=self\.state\[p\]

37if"z"instate:p\.lerp\_\(end=state\["z"\],weight=1\.0\-1\.0/beta\)

38group\["train\_mode"\]=False

39

40@torch\.no\_grad\(\)

41deftrain\(self\):

42forgroupinself\.param\_groups:

43ifnotgroup\["train\_mode"\]:

44beta=group\["betas"\]\[0\]

45forpingroup\["params"\]:

46state=self\.state\[p\]

47if"z"instate:p\.lerp\_\(end=state\["z"\],weight=1\.0\-beta\)

48group\["train\_mode"\]=True

49

50@torch\.no\_grad\(\)

51defstep\(self,closure=None\):

52loss=closure\(\)ifclosureelseNone

53forgroupinself\.param\_groups:

54beta,beta2=group\["betas"\]

55mu,eps=group\["momentum"\],group\["eps"\]

56eta\_scale,decay=group\["eta\_scale"\],group\["weight\_decay"\]

57k,warmup\_steps=group\["k"\],group\["warmup\_steps"\]

58sched=\(k\+1\)/warmup\_stepsifk<warmup\_stepselse1\.0

59lr=group\["lr"\]\*sched

60weight=lr\*lr

61weight\_sum=group\["weight\_sum"\]=group\["weight\_sum"\]\+weight

62ckp1=weight/weight\_sum

63forpingroup\["params"\]:

64ifp\.gradisNone:continue

65grad,state=p\.grad,self\.state\[p\]

66if"z"notinstate:

67state\["z"\]=p\.clone\(\)

68state\["v"\]=torch\.zeros\(p\.shape\[0\],device=p\.device,dtype=torch\.float32\)

69state\["mom"\]=torch\.zeros\_like\(p\)

70z,v,mom=state\["z"\],state\["v"\],state\["mom"\]

71mom\.mul\_\(mu\)\.add\_\(grad,alpha=1\.0\-mu\)

72P=zeropower\_via\_newtonschulz5\(mom\)\.to\(p\.dtype\)

73row\_ms=\(P\*P\)\.mean\(dim=1\)\.float\(\)

74v\.mul\_\(beta2\)\.add\_\(row\_ms,alpha=1\.0\-beta2\)

75Phat=P/\(v\.sqrt\(\)\+eps\)\.to\(P\.dtype\)\.unsqueeze\(1\)

76m,n=p\.shape

77eta\_hat=eta\_scale\*lr\*math\.sqrt\(m\*n\)/max\(1e\-12,Phat\.float\(\)\.norm\(\)\)

78x\_t=\(p\-\(1\.0\-beta\)\*z\)/beta

79ifdecay\!=0\.0:z\.sub\_\(z,alpha=lr\*decay\)

80z\.sub\_\(Phat,alpha=eta\_hat\)

81x\_tp1=\(1\.0\-ckp1\)\*x\_t\+ckp1\*z

82p\.copy\_\(\(1\.0\-beta\)\*z\+beta\*x\_tp1\)

83group\["k"\]=k\+1

84returnloss