Spectral Scaling Laws of Muon

# Spectral Scaling Laws of Muon
Source: [https://arxiv.org/html/2606.04058](https://arxiv.org/html/2606.04058)
###### Abstract

Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open\-source state\-of\-the\-art models adopting Muon\. To keep these updates tractable, Muon performs the orthonormalization with the Newton–Schulz \(NS\) iteration\. Since NS is only approximate, directions with small singular values fail to be orthonormalized\. In Muon, NS is applied to the momentum matrix at every step, yet little is known about how the singular value spectrum of these momentum matrices behaves during training, or how that behavior changes with model size\. We present the first systematic study of this question\. Tracking singular value quantiles of the momentum buffer across layers in models ranging from 77M to 2\.8B parameters, we observe a consistent picture: after a short burn\-in, the quantiles stabilize at a value determined by the layer type and model size\. These stabilization values follow remarkably clean power laws in model size, with layer\-dependent exponents\. Layers up to mid\-late depth scale very mildly with model sizeMM\(aroundM−0\.25M^\{\-0\.25\}\), so the standard 5\-step NS configuration used at academic scale will continue to orthonormalize them at much larger scales\. Some of the late layers, however, scale much more aggressively \(up toM−0\.96M^\{\-0\.96\}\) and will fall into the NS failure regime at frontier scale unless one uses more NS iterations or better\-tuned coefficients\. NS iterations are computationally expensive at scale; our laws give practitioners a principled, layer\-aware recipe for choosing the minimum NS configuration that still orthonormalizes the directions that matter — avoiding unnecessary computation without sacrificing update quality\.

## 1Introduction

Pre\-training large language models \(LLMs\) is a costly process that consumes millions of GPU hours, making the choice of optimizer a central design decision: even modest gains in optimizer efficiency translate into substantial savings\. AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2606.04058#bib.bib31); Kingma and Ba,[2015](https://arxiv.org/html/2606.04058#bib.bib30)\)has long been the standard optimizer for training LLMs\(DeepSeek\-AI,[2024](https://arxiv.org/html/2606.04058#bib.bib28); Llama Team,[2024](https://arxiv.org/html/2606.04058#bib.bib29); Team OLMoet al\.,[2025](https://arxiv.org/html/2606.04058#bib.bib27)\)\. More recently, orthonormalized\-update optimizers such as Muon\(Jordanet al\.,[2024b](https://arxiv.org/html/2606.04058#bib.bib33); Bernstein and Newhouse,[2025](https://arxiv.org/html/2606.04058#bib.bib25)\)have begun to take its place, providing more stable training and better hyperparameter transfer\(Pethicket al\.,[2025](https://arxiv.org/html/2606.04058#bib.bib24)\)across scales\. At larger scale,Liuet al\.\([2025](https://arxiv.org/html/2606.04058#bib.bib23)\)show that Muon achieves twice the compute efficiency of AdamW\. Notably, the recent state\-of\-the\-art models Kimi\-K2, GLM\-5, and DeepSeek\-V4\(Kimi Teamet al\.,[2026](https://arxiv.org/html/2606.04058#bib.bib21); GLM\-5,[2026](https://arxiv.org/html/2606.04058#bib.bib22); DeepSeek\-AI,[2026](https://arxiv.org/html/2606.04058#bib.bib26)\)were all trained with Muon\.

Muon performs approximate orthonormalization of the momentum matrices using the Newton–Schulz \(NS\) iteration\(Higham,[2008](https://arxiv.org/html/2606.04058#bib.bib20); Kovarik,[1970](https://arxiv.org/html/2606.04058#bib.bib19); Björck and Bowie,[1971](https://arxiv.org/html/2606.04058#bib.bib18)\), which repeatedly applies an odd polynomial to push each singular value toward11\. Since NS is only approximate, directions whose singular values are too small fail to be properly orthonormalized \(see[Figure 2](https://arxiv.org/html/2606.04058#S2.F2)\)\. Whether a given NS configuration is accurate enough therefore depends on where the momentum singular values actually reside during training: if they are large, even a cheap NS configuration orthonormalizes them correctly; if they are small, more accurate one is required\. The academic community typically uses the55\-polynomial NS coefficients introduced byCesistaet al\.\([2025](https://arxiv.org/html/2606.04058#bib.bib16)\), which were popularized by the NanoGPT speedrun\(Jordanet al\.,[2024a](https://arxiv.org/html/2606.04058#bib.bib34)\)\. The recent frontier\-scale DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.04058#bib.bib26)\)uses a more accurate composition of1010polynomials\. In both cases the same configuration is applied uniformly across all layers\. Since each NS step carries a non\-trivial cost at scale\(Essential AI,[2025](https://arxiv.org/html/2606.04058#bib.bib17); Ahnet al\.,[2025b](https://arxiv.org/html/2606.04058#bib.bib32)\), a natural question is whether55polynomials already suffice at scale, or whether1010are needed — and crucially, whether the answer is the same for every layer\.

To answer this, we conduct the first systematic study of how the singular values of Muon’s momentum matrices evolve during training, tracking quantiles at multiple depths in GPT\-2\-style models ranging from 77M to 2\.8B parameters\. A consistent picture emerges across all layers and model sizes: after a short burn\-in period, the singular value quantiles stabilize around a value that depends on the layer type and decreases with model size\. Fitting power laws to these stabilization values reveals a remarkably clean log\-log linear relationship with layer\-dependent exponents \(see[Figure 1](https://arxiv.org/html/2606.04058#S1.F1)\)\. This lets us extrapolate, at each depth, how accurate NS must be to orthonormalize enough directions to preserve update quality at frontier scale\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x1.png)Figure 1:Scaling laws for the stabilization values of the singular value quantiles of normalized momentum matrices of different depth layers and model sizes\.The exponents vary substantially across depth\. Layers up to mid\-late depth scale very mildly with model size \(aroundM−0\.25M^\{\-0\.25\}\); the NS approximation used in the NanoGPT experiments remains accurate enough for them at much larger scales\. Some of the late layers, however, scale much more aggressively \(up toM−0\.96M^\{\-0\.96\}\) and will fall into the NS failure regime at frontier scale unless one uses more NS iterations or better\-tuned coefficients\.

Together, these findings let practitioners choose, layer by layer, the cheapest NS configuration that still orthonormalizes the directions that matter at any target scale\. Our main contributions are:

- •The first systematic study of the singular value spectrum of Muon’s momentum buffer across layers and model sizes \(77M–2\.8B\)\.
- •Spectral power laws relating stabilization values to model size, with layer\-dependent exponents\.
- •A practical recipe for selecting layer\-specific NS configurations at frontier scale, derived directly from the fitted laws\.

### 1\.1Related work

Several works address the cost of running Muon at scale when weight matrices are sharded across devices\.Ahnet al\.\([2025b](https://arxiv.org/html/2606.04058#bib.bib32),[a](https://arxiv.org/html/2606.04058#bib.bib6)\)propose Dion, a distributed optimizer that achieves communication\-efficient orthonormalized updates via low\-rank approximations, whileZhaoet al\.\([2026](https://arxiv.org/html/2606.04058#bib.bib8)\)instead apply NS independently on each shard with periodic global synchronization for training stability\.

A parallel line of work explores matrix\-preconditioned optimizers for deep learning, including Shampoo\(Guptaet al\.,[2018](https://arxiv.org/html/2606.04058#bib.bib15)\), SOAP\(Vyaset al\.,[2024](https://arxiv.org/html/2606.04058#bib.bib14)\), and COSMOS\(Chenet al\.,[2026](https://arxiv.org/html/2606.04058#bib.bib9)\)\.Anilet al\.\([2020](https://arxiv.org/html/2606.04058#bib.bib13)\)andShiet al\.\([2023](https://arxiv.org/html/2606.04058#bib.bib12)\)make Shampoo practical at scale, though it requires heuristics such as learning rate grafting\(Agarwalet al\.,[2020](https://arxiv.org/html/2606.04058#bib.bib11)\)to match Adam in practice and, to our knowledge, has not yet been adopted at frontier scale\.Eschenhagenet al\.\([2025](https://arxiv.org/html/2606.04058#bib.bib10)\)mitigates some of these heuristics by adaptively updating the preconditioner\. Closer to Muon,Liet al\.\([2025](https://arxiv.org/html/2606.04058#bib.bib7)\)augments orthonormalized updates with Adam\-style second moments, adding adaptive per\-coordinate scaling on top of Muon’s spectral\-norm step\.Wenet al\.\([2025](https://arxiv.org/html/2606.04058#bib.bib5)\)benchmark many of these optimizers across model sizes and data\-to\-model ratios\.

Scaling laws were pioneered byKaplanet al\.\([2020](https://arxiv.org/html/2606.04058#bib.bib3)\), who showed that language model loss follows clean power laws in parameters, training tokens, and compute\.Hoffmannet al\.\([2022](https://arxiv.org/html/2606.04058#bib.bib4)\)refined these relationships into compute\-optimal token\-to\-parameter ratios, establishing that prior large models were significantly undertrained\. A complementary direction asks what optimizer hyperparameters scale predictably with model size\.Yanget al\.\([2022](https://arxiv.org/html/2606.04058#bib.bib2)\)show that optimal learning rates transfer zero\-shot across scales under theμ\\muP parameterization, demonstrating that optimizer hyperparameters obey their own scaling structure\. We contribute to these lines of work: we show that, after a short burn\-in, the singular value quantiles of Muon’s momentum buffers stabilize at values that follow power laws in model size, with layer\-dependent exponents\.

## 2Background: Muon and Newton\-Schulz

This section establishes the background for the rest of the paper\. We first describe the Muon optimizer \([subsection 2\.1](https://arxiv.org/html/2606.04058#S2.SS1)\) and review the Newton\-Schulz iteration it uses for approximate orthonormalization \([subsection 2\.2](https://arxiv.org/html/2606.04058#S2.SS2)\), highlighting its key limitation: directions with sufficiently small singular values fail to be orthonormalized\. We then run a controlled experiment \([subsection 2\.3](https://arxiv.org/html/2606.04058#S2.SS3)\) to determine which fraction of singular directions must be orthonormalized for Muon to retain its benefits, which fixes the quantile range we track for the rest of the paper\.

### 2\.1The Muon Optimizer

Muon\(Jordanet al\.,[2024b](https://arxiv.org/html/2606.04058#bib.bib33); Bernstein and Newhouse,[2025](https://arxiv.org/html/2606.04058#bib.bib25)\)replaces the raw gradient update on 2D weight matrices with an*orthonormalized*update\.At each step Muon maintains a momentum bufferMtM\_\{t\}and applies a Newton\-Schulz \(NS\) iteration to approximately orthonormalize it before stepping the parameters \(See Algorithm[1](https://arxiv.org/html/2606.04058#alg1)\)\.

Algorithm 1Muon Optimizer1:Learning rate

η\\eta, momentum coefficient

μ\\mu, initial parameters

Θ0\\Theta\_\{0\}
2:Initialize momentum buffer

M0←0M\_\{0\}\\leftarrow 0
3:for

t=0,1,2,…t=0,1,2,\\ldotsdo

4:Compute gradient

Gt=∇Θℒ​\(Θt\)G\_\{t\}=\\nabla\_\{\\Theta\}\\mathcal\{L\}\(\\Theta\_\{t\}\)
5:Update momentum:

Mt\+1←μ⋅Mt\+GtM\_\{t\+1\}\\leftarrow\\mu\\cdot M\_\{t\}\+G\_\{t\}
6:Orthonormalize:

Ot\+1←NS​\(Mt\+1\)O\_\{t\+1\}\\leftarrow\\mathrm\{NS\}\(M\_\{t\+1\}\)
7:Update parameters:

Θt\+1←Θt−η⋅Ot\+1\\Theta\_\{t\+1\}\\leftarrow\\Theta\_\{t\}\-\\eta\\cdot O\_\{t\+1\}
8:endfor

HereNS​\(⋅\)\\mathrm\{NS\}\(\\cdot\)denotes the Newton\-Schulz iteration described in[subsection 2\.2](https://arxiv.org/html/2606.04058#S2.SS2)\. Muon is motivated by steepest descent under the spectral norm\(Bernstein and Newhouse,[2025](https://arxiv.org/html/2606.04058#bib.bib25)\), and has been shown to double the compute efficiency compared to AdamW on language model training tasks in scale\(Liuet al\.,[2025](https://arxiv.org/html/2606.04058#bib.bib23)\)\.

### 2\.2Newton\-Schulz Iteration for Approximate Orthonormalization

We now focus our attention on the problem of approximate orthonormalization\. Exact orthonormalization is expensive for large matrices, so practical implementations use fast iterative approximations\.

The standard approximation in this line of work is the Newton\-Schulz \(NS\) iteration\. LetAAbe a momentum matrix to be orthonormalized\. NS first normalizes

A~0=A‖A‖F,\\widetilde\{A\}\_\{0\}=\\frac\{A\}\{\\\|A\\\|\_\{F\}\},to transform all singular values to\[0,1\]\[0,1\]interval and then applies a sequence of odd\-degree polynomialsp0,…,pn−1p\_\{0\},\\dots,p\_\{n\-1\}\. In practice, eachpkp\_\{k\}is taken to be a degree\-5 odd polynomial of the form

A~k\+1=pk​\(A~k\)=ak​A~k\+bk​\(A~k​A~k⊤\)​A~k\+ck​\(A~k​A~k⊤\)2​A~k,k=0,1,…,n−1,\\displaystyle\\widetilde\{A\}\_\{k\+1\}=p\_\{k\}\(\\widetilde\{A\}\_\{k\}\)=a\_\{k\}\\widetilde\{A\}\_\{k\}\+b\_\{k\}\\left\(\\widetilde\{A\}\_\{k\}\\widetilde\{A\}\_\{k\}^\{\\top\}\\right\)\\widetilde\{A\}\_\{k\}\+c\_\{k\}\\left\(\\widetilde\{A\}\_\{k\}\\widetilde\{A\}\_\{k\}^\{\\top\}\\right\)^\{2\}\\widetilde\{A\}\_\{k\},\\quad k=0,1,\\ldots,n\-1,\(1\)Recall that for any matrix with SVDA=U​S​V⊤A=USV^\{\\top\}, an odd polynomial satisfies

p​\(A\)=U​p​\(S\)​V⊤,p\(A\)=U\\,p\(S\)\\,V^\{\\top\},wherep​\(S\)p\(S\)appliesppelementwise to the diagonal ofSS\. This means the singular vectors are*exactly preserved*at every step, and only the singular values are modified\. Unrolling[Equation 1](https://arxiv.org/html/2606.04058#S2.E1)nntimes, the final result is

A~n=U​\(pn∘⋯∘p1\)​\(S\)⏟=⁣:f​\(S\)​V⊤,\\widetilde\{A\}\_\{n\}=U\\,\\underbrace\{\(p\_\{n\}\\circ\\cdots\\circ p\_\{1\}\)\(S\)\}\_\{=:\\,f\(S\)\}\\,V^\{\\top\},Thus, the NS procedure reduces to a one\-dimensional problem: find a scalar compositionf=pn∘⋯∘p1f=p\_\{n\}\\circ\\cdots\\circ p\_\{1\}such thatf​\(σ\)≈1f\(\\sigma\)\\approx 1for every singular valueσ∈\[0,1\]\\sigma\\in\[0,1\]\. In other words,ffshould approximate the sign function on\(0,1\]\(0,1\], pushing every singular value toward11regardless of where it starts\. When this condition holds,A~n≈U​V⊤\\widetilde\{A\}\_\{n\}\\approx UV^\{\\top\}, recovering the orthonormal factor in the polar decomposition ofAA\. However, since eachpip\_\{i\}is an odd polynomial,f​\(0\)=0f\(0\)=0for any choice of polynomials, soffcannot approximate the sign function in a neighborhood of zero—a fundamental limitation of the NS family\.

As a concrete example, consider the canonical polynomial used for NS and in the introduction of Muon\(Jordanet al\.,[2024b](https://arxiv.org/html/2606.04058#bib.bib33)\):

p​\(x\)=2​x−1\.5​x3\+0\.5​x5,p\(x\)=2x\-1\.5x^\{3\}\+0\.5x^\{5\},appliedn=5n=5times, i\.e\.f=p∘5f=p^\{\\circ 5\}\.[Figure 2](https://arxiv.org/html/2606.04058#S2.F2)plotsf​\(σ\)f\(\\sigma\)as a function ofσ∈\[0,1\]\\sigma\\in\[0,1\]\. One can see that the approximation is accurate forσ\>0\.05\\sigma\>0\.05, pushing those values close to11\. However, the composition is approximately linear near the origin: forσ≤0\.003\\sigma\\leq 0\.003one can verify numerically thatf​\(σ\)≤0\.1f\(\\sigma\)\\leq 0\.1\. In other words, any direction whose singular value falls below roughly0\.0030\.003will remain essentially*unorthonormalized*after five NS steps — its effective contribution to the update is suppressed by a factor of10×10\\timesor more relative to a direction with large singular value\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x2.png)Figure 2:The NS mapf​\(σ\)=p∘5​\(σ\)f\(\\sigma\)=p^\{\\circ 5\}\(\\sigma\)for the canonical polynomialp​\(x\)=2​x−1\.5​x3\+0\.5​x5p\(x\)=2x\-1\.5x^\{3\}\+0\.5x^\{5\}\. The left plot shows the full rangeσ∈\[0,1\]\\sigma\\in\[0,1\]\. The right plot shows a zoom\-in view of the regionσ∈\[0,0\.05\]\\sigma\\in\[0,0\.05\]\.In practice, different implementations use different NS configurations\. The NanoGPT speedrun\(Jordanet al\.,[2024a](https://arxiv.org/html/2606.04058#bib.bib34)\)uses optimized 5\-step polynomials \(see[Figure 8](https://arxiv.org/html/2606.04058#A1.F8)\), while DeepSeek\-V4\(DeepSeek\-AI,[2026](https://arxiv.org/html/2606.04058#bib.bib26)\)employs a more accurate 10\-step composition \(see[Figure 9](https://arxiv.org/html/2606.04058#A1.F9)\)\. Since each NS step carries a non\-trivial cost at scale\(Essential AI,[2025](https://arxiv.org/html/2606.04058#bib.bib17); Ahnet al\.,[2025b](https://arxiv.org/html/2606.04058#bib.bib32)\), a natural question is whether the additional steps are necessary to maintain update quality\. To answer this, one must understand how the singular values of the momentum matrices actually behave during training — which we study systematically in[section 3](https://arxiv.org/html/2606.04058#S3)\.

To understand which quantiles of the momentum spectrum are practically relevant, we run a controlled experiment with rank\-pporthonormal updates — using only the topppfraction of singular directions — and measure how closely they track full Muon’s performance\. This determines which quantiles to focus on in the sections that follow\.

### 2\.3How much orthonormalization is needed?

We now investigate how many singular directions must be orthonormalized to retain the benefits of Muon\. To this end, we introduce*rank\-pporthonormal updates*: given the SVDM=U​S​V⊤∈ℝm×nM=USV^\{\\top\}\\in\\mathbb\{R\}^\{m\\times n\}of the momentum matrix, the update direction is formed using only the top\-kksingular vectors,

O=U:,1:k​V:,1:k⊤,k=⌊min⁡\(m,n\)⋅p⌋,O=U\_\{:,\\,1:k\}\\,V\_\{:,\\,1:k\}^\{\\top\},\\qquad k=\\left\\lfloor\\min\(m,n\)\\cdot p\\right\\rfloor,wherep∈\{0\.1,0\.25,0\.5,0\.9\}p\\in\\\{0\.1,\\,0\.25,\\,0\.5,\\,0\.9\\\}denotes the fraction of singular directions retained\. We pretrain GPT\-2\-style models with 77M, 160M, and 354M parameters \([Table 1](https://arxiv.org/html/2606.04058#S3.T1)\) across this range ofppvalues, each for a Chinchilla\-optimal number of tokens\(Hoffmannet al\.,[2022](https://arxiv.org/html/2606.04058#bib.bib4)\)\. For more details see[subsection A\.1](https://arxiv.org/html/2606.04058#A1.SS1)\.

[Figure 3](https://arxiv.org/html/2606.04058#S2.F3)reveals a monotonic degradation asppdecreases:p=0\.9p=0\.9is essentially indistinguishable from full Muon, and evenp=0\.5p=0\.5incurs only a minor performance gap\. Quantifying these gaps more precisely \([Figure 11](https://arxiv.org/html/2606.04058#A2.F11),[Figure 11](https://arxiv.org/html/2606.04058#A2.F11)\),p=0\.25p=0\.25updates are around10​–​20%10\\text\{\-\-\}20\\%less token\-efficient than full Muon which is a gap that may be acceptable in practice\.p=0\.1p=0\.1, in contrast, is around50%50\\%less efficient and impractical\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x3.png)Figure 3:Pre\-training models of sizes 77M, 160M, and 354M parameters with rank\-pporthonormal updates\.Ahnet al\.\([2025b](https://arxiv.org/html/2606.04058#bib.bib32)\)run a similar low\-rank ablation for the Dion optimizer \(their Figure 2\) and observe that the performance gap relative to full Dion narrows with model scale\. Their setting differs from ours in one important way: Dion uses*error feedback*, accumulating the residual of the low\-rank approximation back into the momentum buffer, which compensates for the discarded information\. Our rank\-ppupdates have no such compensation, so we do not expect the same narrowing trend\.

A potential concern is that the validation curves in[Figure 3](https://arxiv.org/html/2606.04058#S2.F3)run parallel after the initial phase, suggesting the gap may stem from Muon’s faster convergence early in training rather than from a fundamental advantage of full orthonormalization\. To check this, we pre\-train 77M and 160M models with full Muon for 125 and 250 steps respectively — well into the regime where its advantage overp=0\.1p=0\.1has already opened up — and then switch top=0\.1p=0\.1updates\. As shown in[Figure 4](https://arxiv.org/html/2606.04058#S2.F4), the gap relative to full Muon remains large in both cases, confirming that the gap is not an artifact of early\-training dynamics\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x4.png)Figure 4:We compare training with low\-rankr=0\.1r=0\.1updates from scratch against first running full Muon for 125 \(or 250\) steps and then switching to low\-rank updates\. In all cases the gap relative to full Muon remains large, indicating that the performance difference is not simply due to Muon’s faster convergence at the start of training\.##### Takeaway\.

Orthonormalizing roughly the top half of singular directions is enough to recover \(or nearly recover\) full Muon, but orthonormalizing only the top10%10\\%is not\. To understand which NS approximations are needed to orthonormalize this range of directions, we now turn to the singular value spectrum of the momentum matrices\.

## 3Spectral Dynamics of the Momentum Buffer

In this section we track the quantiles of normalized singular values of the momentum matrices and observe how they evolve during the training\. Before proceeding we provide necessary notation and setup\.

##### Notation\.

LetA∈ℝm×nA\\in\\mathbb\{R\}^\{m\\times n\}be a matrix with singular values sorted in descending orderσ1​\(A\)≥σ2​\(A\)≥⋯≥σr​\(A\)\\sigma\_\{1\}\(A\)\\geq\\sigma\_\{2\}\(A\)\\geq\\cdots\\geq\\sigma\_\{r\}\(A\), wherer=min⁡\(m,n\)r=\\min\(m,n\)\. Forq∈\(0,1\]q\\in\(0,1\]we define the*qq\-quantile singular value*

σq​\(A\):=σ⌈q⋅r⌉​\(A\),\\sigma\_\{q\}\(A\):=\\sigma\_\{\\lceil q\\cdot r\\rceil\}\(A\),so thatσ0\.5​\(A\)\\sigma\_\{0\.5\}\(A\)is the median \(roughly half of singular values are larger\) andσ1\.0​\(A\)=σr​\(A\)\\sigma\_\{1\.0\}\(A\)=\\sigma\_\{r\}\(A\)is the smallest\. Note that under this conventionσ0\.1​\(A\)\\sigma\_\{0\.1\}\(A\)is a*large*singular value \(only∼10%\\sim\\\!10\\%are larger\) andσ0\.9​\(A\)\\sigma\_\{0\.9\}\(A\)is a*small*one\. We track these quantiles forA=M\(t\)/‖M\(t\)‖FA=M^\{\(t\)\}/\\\|M^\{\(t\)\}\\\|\_\{F\}, the Frobenius\-normalized momentum matrix of a given layer at training steptt\. This is exactly the input that NS sees asA~0\\widetilde\{A\}\_\{0\}\([subsection 2\.2](https://arxiv.org/html/2606.04058#S2.SS2)\), so the tracked quantiles are directly comparable to NS’s failure threshold\. We use them to understand how the singular value spectrum evolves over training and to quantify the fraction of directions that a given NS configuration fails to orthonormalize\.

##### Setup\.

We pretrain a suite of GPT\-2\-style language models ranging from 77M to 2\.8B parameters with Muon; configurations are detailed in[Table 1](https://arxiv.org/html/2606.04058#S3.T1)\. Each model is trained for the Chinchilla\-optimal number of tokens\(Hoffmannet al\.,[2022](https://arxiv.org/html/2606.04058#bib.bib4)\)\.

Table 1:Model configurations used in our experiments\.Since models vary in depth across configurations, we select four*relative depth checkpoints*to ensure comparability across model sizes\. Concretely, for a model withNNtransformer layers we monitor layers⌊N4⌋,⌊2​N4⌋,⌊3​N4⌋​N,\\left\\lfloor\\frac\{N\}\{4\}\\right\\rfloor,\\left\\lfloor\\frac\{2N\}\{4\}\\right\\rfloor,\\left\\lfloor\\frac\{3N\}\{4\}\\right\\rfloor N,corresponding to the*mid\-early*,*mid*,*mid\-late*, and*final*layers of the network\. Within each selected layer we track all six momentum matrices in the block — the four attention projectionsQQ,KK,VV,OOand the two MLP projections\. This gives4×6=244\\times 6=24momentum buffers per model\. For each, we record the singular value quantilesσq​\(M\(t\)\)\\sigma\_\{q\}\(M^\{\(t\)\}\)forq∈\{0\.1,0\.25,0\.5,0\.75,0\.9\}q\\in\\\{0\.1,0\.25,0\.5,0\.75,0\.9\\\}at every training steptt\.

### 3\.1Stabilization of Singular Value Quantiles

We now investigate how the tracked quantiles evolve during training\.[Figure 5](https://arxiv.org/html/2606.04058#S3.F5)plotsσ0\.5​\(M\(t\)\)\\sigma\_\{0\.5\}\(M^\{\(t\)\}\)for three layer types across different model sizes over the first15001500training steps\. A consistent phenomenon emerges across model sizes and layer types: after a short transient phase, the quantiles stabilize at values that persist for the remainder of training\. The same pattern holds for all tracked layer types and quantiles \([Figure 12](https://arxiv.org/html/2606.04058#A2.F12),[Figure 13](https://arxiv.org/html/2606.04058#A2.F13)\)\.

Notably, the shape of the transient differs by matrix type\. ForQQandKKmatrices, the quantiles exhibit a sharp decrease followed by a recovery before stabilizing, whereas forVV,OO, and MLP matrices the quantiles increase monotonically from the start before stabilizing\. We also observe that the stabilized values decrease monotonically with model size, suggesting that as model size increases a fixed NS configuration will fail to orthonormalize an increasing fraction of directions — a hypothesis we make quantitative in[section 4](https://arxiv.org/html/2606.04058#S4)\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x5.png)Figure 5:Quantile evolution for the 50% quantile of the normalized singular values for 3 fixed layer types and model sizes\.
### 3\.2Stabilization of the Full Spectrum

Since all tracked quantiles stabilize, the full spectrum stabilizes as well\.[Figure 6](https://arxiv.org/html/2606.04058#S3.F6)shows the distribution of normalized singular values for the selected momentum matrices of the 2\.8B model at step14501450\. The top row plots the full spectrum, while the bottom row shows the same distribution with the leading singular value removed to reveal the bulk\. Two consistent features emerge across all layer types \(see[Figure 14](https://arxiv.org/html/2606.04058#A2.F14)for the bulk of every tracked matrix\): \(i\) each spectrum is dominated by a single outlier singular value, often an order of magnitude or more larger than the rest of the distribution; and \(ii\) once this outlier is removed, the remaining singular values are concentrated near zero, with the count decaying roughly exponentially as the singular value grows\. The scale of the bulk varies markedly across layer types — a variation we exploit in[section 4](https://arxiv.org/html/2606.04058#S4)when we fit layer\-dependent scaling exponents\. This heavy concentration near zero is precisely what places the late layers at risk of NS failure at scale\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x6.png)Figure 6:We plot the distribution of normalized singular values for the selected momentum matrices of the 2\.8B model at step 1500\. The top row plots the full spectrum, while the bottom row shows the same distribution with the leading singular value removed to reveal the bulk\.
### 3\.3Quantile Dynamics under Rank\-ppUpdates

Recall that in[subsection 2\.3](https://arxiv.org/html/2606.04058#S2.SS3)we studied the effect of rank\-pporthonormal updates on validation loss\. Here we complement that analysis by examining how the50%50\\%quantile of the normalized momentum matrices evolves under these updates\. As shown in[Figure 7](https://arxiv.org/html/2606.04058#S3.F7)\(for the 354M model\), the trajectories forp=0\.9p=0\.9andp=0\.5p=0\.5— the regimes that closely match Muon’s validation loss — closely track Muon’s quantile as well\. Atp=0\.25p=0\.25the quantile begins to deviate downward, and atp=0\.1p=0\.1the deviation grows further\. The same pattern holds across most tracked layers \([Figure 15](https://arxiv.org/html/2606.04058#A2.F15)\)\.

This yields a clean correspondence: rank\-ppupdates that closely track Muon’s performance also closely track its singular value dynamics, while those with degraded performance exhibit deviating, typically smaller quantile trajectories\.

This correspondence has a direct consequence for NS\. As long as NS still orthonormalizes at least the top50%50\\%of directions, its induced quantile dynamics fall under thep≥0\.5p\\geq 0\.5branch above and closely track full Muon\. In this regime the scaling laws we derive in[section 4](https://arxiv.org/html/2606.04058#S4)— fit to full\-Muon stabilization values — are*self\-consistent*: choosing an NS configuration so that the predicted50%50\\%\-quantile sits above its failure threshold will indeed orthonormalize that fraction at the target scale\. For NS configurations that orthonormalize only the top25%25\\%\(or fewer\) of directions at scale, the laws*may*underestimate how many directions NS misses, since thep=0\.25p=0\.25andp=0\.1p=0\.1trajectories sit below full Muon’s\. Quantifying this effect at scale would require fitting separate laws to rank\-ppruns, which need a per\-step SVD and are far more expensive than running Muon itself; we leave this to future work\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x7.png)Figure 7:Quantile dynamics for the 50% quantile of the normalized momentum matrices for the 354M model under rank\-pporthonormal updates\. The trajectories forp=0\.9p=0\.9andp=0\.5p=0\.5closely track Muon’s quantile, while atp=0\.25p=0\.25andp=0\.1p=0\.1the deviation grows bigger\.

## 4Spectral Scaling Laws

As discussed in[section 3](https://arxiv.org/html/2606.04058#S3), the singular value quantiles of the momentum matrices stabilize after a short burn\-in phase, with stabilization values that decrease with model size\. To predict what fraction of directions a given NS configuration will orthonormalize at scale, we need to understand how these stabilization values scale with model size\. We study this scaling for quantilesq∈\{0\.1,0\.25,0\.5,0\.75,0\.9\}q\\in\\\{0\.1,0\.25,0\.5,0\.75,0\.9\\\}\.

For each model size and layer type, we estimate the stabilization value by averaging the corresponding quantile over training steps13001300–15001500, and plot it against model size on a log\-log scale\.[Figure 1](https://arxiv.org/html/2606.04058#S1.F1)shows the scaling laws for six representative layer types; we observe a remarkably clean power law in model size\. As shown in[Figure 16](https://arxiv.org/html/2606.04058#A2.F16), the same pattern holds across all six tracked layer types: for each, all five quantiles share the same scaling exponent — and that exponent depends on the layer type\.

The exponents vary substantially across depth\. The mid\-early, mid, and mid\-late layers scale very mildly with model size, with exponents around−0\.25\-0\.25— meaning that increasing model size by a factor of3232decreases the stabilization value by only roughly a factor of22\. The final layers, in contrast, scale down far more aggressively: the final MLP projection matrix, for instance, has an exponent of−0\.96\-0\.96, so its stabilization value decreases nearly linearly with model size\. Thus, we have a wide range of scaling exponents across layers, making a uniform NS configuration suboptimal at scale\.

### 4\.1Case Study: Extrapolating to Frontier Scale

To illustrate how the fitted laws are used in practice, consider a300300B\-scale training run \(a∼100×\\sim\\\!100\\timesjump from our largest fitted scale of 2\.8B\)\. We compare two contrasting layer types: the mid\-lateQQprojection and the finalOOprojection\. Suppose we want to orthonormalize at least50%50\\%of the directions in each; then the relevant quantile isq=0\.5q=0\.5\.

##### Mid\-lateQQ\.

From[Figure 1](https://arxiv.org/html/2606.04058#S1.F1), theq=0\.5q=0\.5stabilization value at 2\.8B is around5⋅10−35\\cdot 10^\{\-3\}\. The fitted exponent for this layer type is−0\.27\-0\.27, so the law predicts a value at 300B of

5⋅10−3⋅100−0\.27≈1\.4⋅10−3\.5\\cdot 10^\{\-3\}\\cdot 100^\{\-0\.27\}\\;\\approx\\;1\.4\\cdot 10^\{\-3\}\.This sits above the NanoGPT 5\-step failure regime \([Figure 8](https://arxiv.org/html/2606.04058#A1.F8)\), so the standard 5\-step NS configuration will continue to sufficiently orthonormalize this layer correctly at 300B\.

##### FinalOO\.

For the finalOOprojection, theq=0\.5q=0\.5value at 2\.8B is around10−310^\{\-3\}, with a fitted exponent of−0\.66\-0\.66\. The law predicts a value at 300B of

10−3⋅100−0\.66≈5⋅10−5,10^\{\-3\}\\cdot 100^\{\-0\.66\}\\;\\approx\\;5\\cdot 10^\{\-5\},which falls inside the NanoGPT failure regime \([Figure 8](https://arxiv.org/html/2606.04058#A1.F8)\)\. For this layer one would need a more accurate NS configuration — e\.g\., the 10\-step composition used by DeepSeek\-V4 \([Figure 9](https://arxiv.org/html/2606.04058#A1.F9)\)\.

## 5Conclusion

We presented the first systematic study of how Muon’s momentum spectrum evolves during training and scales with model size\. Across models from 77M to 2\.8B parameters and layers at all relative depths, we identified a consistent picture: after a short burn\-in, every quantile of the momentum spectrum stabilizes at a value determined by the layer type and model size, and these stabilization values follow clean power laws in model size with layer\-dependent exponents\.

The exponents differ markedly across layers — ranging from roughly−0\.25\-0\.25for mid\-early through mid\-late layers down to−0\.96\-0\.96for the final MLP projection\. This wide range is the central finding of our paper and has a direct practical consequence: a uniform NS configuration applied across all layers is unavoidably suboptimal at scale, since the layers that need the most accurate orthonormalization are precisely those whose singular values shrink fastest with model size\. Our case study illustrates this concretely: extrapolating from the 2\.8B model, a300300B\-scale training run can continue to use the55\-step NanoGPT NS coefficients for the majority of its layers, but some of the final layers will fall into the NS failure regime unless a more accurate configuration — such as the1010\-step composition used by DeepSeek\-V4 — is applied to those layers\.

Together, these results turn a previously opaque design choice — how accurate must NS be? — into a quantitative, layer\-aware decision that can be made directly from our scaling laws\. We see several natural extensions\. First, the stabilization phenomenon and the particular exponents we measure may be specific to GPT\-2\-style language models trained with Muon\. Studying analogous scaling laws for other architectures \(e\.g\., Mixture\-of\-Experts models\) and for other optimizers that rely on iterative matrix\-function approximations — most notably Shampoo\(Guptaet al\.,[2018](https://arxiv.org/html/2606.04058#bib.bib15)\)and its descendants\(Vyaset al\.,[2024](https://arxiv.org/html/2606.04058#bib.bib14); Eschenhagenet al\.,[2025](https://arxiv.org/html/2606.04058#bib.bib10)\)— is a natural next step\. Second, designing NS coefficients specifically tuned to the empirical singular value distribution of each layer is a promising avenue for further reducing the cost of orthonormalization at frontier scale\. We leave both directions to future work\.

## References

- N\. Agarwal, R\. Anil, E\. Hazan, T\. Koren, and C\. Zhang \(2020\)Disentangling adaptive gradient methods from learning rates\.arXiv preprint arXiv:2002\.11803\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1)\.
- K\. Ahn, N\. Amsel, and J\. Langford \(2025a\)Dion2: a simple method to shrink matrix in muon\.arXiv preprint arXiv:2512\.16928\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p1.1)\.
- K\. Ahn, B\. Xu, N\. Abreu, Y\. Fan, G\. Magakyan, P\. Sharma, Z\. Zhan, and J\. Langford \(2025b\)Dion: distributed orthonormalized updates\.arXiv preprint arXiv:2504\.05295\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p1.1),[§1](https://arxiv.org/html/2606.04058#S1.p2.5),[§2\.2](https://arxiv.org/html/2606.04058#S2.SS2.p4.1),[§2\.3](https://arxiv.org/html/2606.04058#S2.SS3.p3.1)\.
- R\. Anil, V\. Gupta, T\. Koren, K\. Regan, and Y\. Singer \(2020\)Scalable second order optimization for deep learning\.arXiv preprint arXiv:2002\.09018\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1)\.
- J\. Bernstein and L\. Newhouse \(2025\)Modular duality in deep learning\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.04058#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.04058#S2.SS1.p2.1)\.
- Å\. Björck and C\. Bowie \(1971\)An iterative algorithm for computing the best estimate of an orthogonal matrix\.SIAM Journal on Numerical Analysis\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p2.5)\.
- F\. L\. Cesista, Y\. Jiacheng, and K\. Jordan \(2025\)Squeezing 1\-2% efficiency gains out of muon by optimizing the newton\-schulz coefficients\.External Links:[Link](https://leloykun.github.io/ponder/muon-opt-coeffs/)Cited by:[§A\.2\.1](https://arxiv.org/html/2606.04058#A1.SS2.SSS1.p1.1),[§1](https://arxiv.org/html/2606.04058#S1.p2.5)\.
- L\. Chen, Z\. Zhao, T\. Chen, H\. Ye, B\. Chen, Y\. Cheng, and Y\. Chen \(2026\)COSMOS: a hybrid adaptive optimizer for efficient training of large language models\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4\-pro\.Note:Hugging Face model card and technical reportDeepSeek\-V4 Preview releaseCited by:[§A\.2\.2](https://arxiv.org/html/2606.04058#A1.SS2.SSS2.p1.1),[§1](https://arxiv.org/html/2606.04058#S1.p1.1),[§1](https://arxiv.org/html/2606.04058#S1.p2.5),[§2\.2](https://arxiv.org/html/2606.04058#S2.SS2.p4.1)\.
- R\. Eschenhagen, A\. Defazio, S\. Lee, R\. E\. Turner, and H\. M\. Shi \(2025\)Purifying shampoo: investigating shampoo’s heuristics by decomposing its preconditioner\.InAdvances in Neural Information Processing Systems,Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1),[§5](https://arxiv.org/html/2606.04058#S5.p3.1)\.
- Essential AI \(2025\)Layer sharding for large\-scale training with muon\.External Links:[Link](https://www.essential.ai/research/infra)Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p2.5),[§2\.2](https://arxiv.org/html/2606.04058#S2.SS2.p4.1)\.
- GLM\-5 \(2026\)GLM\-5: pushing the frontier of open\-source large language models\.arXiv preprint arXiv:2602\.15763\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- V\. Gupta, T\. Koren, and Y\. Singer \(2018\)Shampoo: preconditioned stochastic tensor optimization\.InInternational Conference on Machine Learning,Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1),[§5](https://arxiv.org/html/2606.04058#S5.p3.1)\.
- N\. J\. Higham \(2008\)Functions of matrices: theory and computation\.Society for Industrial and Applied Mathematics\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p2.5)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[§A\.1](https://arxiv.org/html/2606.04058#A1.SS1.p2.8),[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p3.1),[§2\.3](https://arxiv.org/html/2606.04058#S2.SS3.p1.5),[§3](https://arxiv.org/html/2606.04058#S3.SS0.SSS0.Px2.p1.1)\.
- K\. Jordan, J\. Bernstein, B\. Rappazzo, @fernbear\.bsky\.social, B\. Vlado, Y\. Jiacheng, F\. Cesista, B\. Koszarsky, and @Grad62304977 \(2024a\)Modded\-nanogpt: speedrunning the nanogpt baseline\.External Links:[Link](https://github.com/KellerJordan/modded-nanogpt)Cited by:[§A\.1](https://arxiv.org/html/2606.04058#A1.SS1.p1.5),[§A\.2\.1](https://arxiv.org/html/2606.04058#A1.SS2.SSS1.p1.1),[§1](https://arxiv.org/html/2606.04058#S1.p2.5),[§2\.2](https://arxiv.org/html/2606.04058#S2.SS2.p4.1)\.
- K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024b\)Muon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.04058#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2606.04058#S2.SS2.p3.11)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p3.1)\.
- Kimi Team, T\. Bai, Y\. Bai, Y\. Bao, S\. H\. Cai, Y\. Cao,et al\.\(2026\)Kimi k2\.5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.International Conference on Learning Representations\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- Z\. Kovarik \(1970\)Some iterative methods for improving orthonormality\.SIAM Journal on Numerical Analysis\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p2.5)\.
- Z\. Li, L\. Chen, T\. Chen, B\. Chen, H\. Ye, and Y\. Chen \(2025\)NorMuon: making muon more efficient and scalable\.arXiv preprint arXiv:2510\.05491\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1)\.
- J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin,et al\.\(2025\)Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.04058#S2.SS1.p2.1)\.
- Llama Team \(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.International Conference on Learning Representations\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf \(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,Cited by:[§A\.1](https://arxiv.org/html/2606.04058#A1.SS1.p2.8)\.
- T\. Pethick, W\. Xie, K\. Antonakopoulos, Z\. Zhu, A\. Silveti\-Falls, and V\. Cevher \(2025\)Training deep learning models with norm\-constrained lmos\.InInternational Conference on Machine Learning,Cited by:[§A\.1](https://arxiv.org/html/2606.04058#A1.SS1.p1.5),[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- H\. M\. Shi, T\. Lee, S\. Padmanabhan, V\. Xu, K\. Han, J\. Xie, C\. Zhao, S\. Nie, Y\. Gong, Y\. Zhang,et al\.\(2023\)A distributed data\-parallel PyTorch implementation of the distributed shampoo optimizer for training neural networks at\-scale\.arXiv preprint arXiv:2309\.06497\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1)\.
- Team OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo,et al\.\(2025\)2 olmo 2 furious\.arXiv preprint arXiv:2501\.00656\.Cited by:[§1](https://arxiv.org/html/2606.04058#S1.p1.1)\.
- N\. Vyas, D\. Morwani, R\. Zhao, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. Kakade \(2024\)SOAP: improving and stabilizing shampoo using adam\.InAdvances in Neural Information Processing Systems,Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1),[§5](https://arxiv.org/html/2606.04058#S5.p3.1)\.
- K\. Wen, D\. Hall, T\. Ma, and P\. Liang \(2025\)Fantastic pretraining optimizers and where to find them\.arXiv preprint arXiv:2509\.02046\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p2.1)\.
- G\. Yang, E\. J\. Hu, I\. Babuschkin, S\. Sidor, X\. Liu, D\. Farhi, N\. Ryder, J\. Pachocki, W\. Chen, and J\. Gao \(2022\)Tensor programs V: tuning large neural networks via zero\-shot hyperparameter transfer\.arXiv preprint arXiv:2203\.03466\.Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p3.1)\.
- R\. Zhao, H\. Liu, D\. Morwani, and A\. Jadbabaie \(2026\)MuonBP: faster muon via block\-periodic orthogonalization\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2606.04058#S1.SS1.p1.1)\.

## Appendix AAppendix

### A\.1Details on pre\-training

We used the modded\-nanogpt codebase\[Jordanet al\.,[2024a](https://arxiv.org/html/2606.04058#bib.bib34)\]for all experiments\. All matrix\-valued parameters are trained with Muon, while non\-matrix parameters \(embeddings, LM head, and biases\) are trained with AdamW with\(β1,β2\)=\(0\.9,0\.95\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.95\)and learning rate0\.0020\.002\. For both optimizers we use a weight decay of0\.010\.01throughout\. For clarity, initialization scaling is omitted from Algorithm[1](https://arxiv.org/html/2606.04058#alg1); in practice, we scale matrix parameters bydout/din\\sqrt\{d\_\{\\text\{out\}\}/d\_\{\\text\{in\}\}\}and LM head parameters by1/din1/\\sqrt\{d\_\{\\text\{in\}\}\}, which promotes learning rate transfer across scales\[Pethicket al\.,[2025](https://arxiv.org/html/2606.04058#bib.bib24)\]\.

For the experiments in[subsection 2\.3](https://arxiv.org/html/2606.04058#S2.SS3), we tuned the learning rate for 77M models over the grid\{0\.01,0\.02,0\.03,0\.04,0\.05\}\\\{0\.01,0\.02,0\.03,0\.04,0\.05\\\}\. We observed little sensitivity and found0\.030\.03to be optimal across all rank\-ppconfigurations; we adopted it for the 160M and 354M models as well\. For the experiments in[section 3](https://arxiv.org/html/2606.04058#S3), we observed no meaningful difference between learning rates0\.010\.01,0\.020\.02, and0\.030\.03at the 160M scale; we therefore fixed the learning rate to0\.010\.01across all model sizes, leveraging Muon’s known learning\-rate transfer property\. We used a constant learning rate followed by linear decay over the final 10% of training\. All models are trained on the FineWeb dataset\[Penedoet al\.,[2024](https://arxiv.org/html/2606.04058#bib.bib1)\]for a Chinchilla\-optimal token budget\[Hoffmannet al\.,[2022](https://arxiv.org/html/2606.04058#bib.bib4)\]of20×20\\timesthe number of model parameters\. All experiments were run on L40 or H200 GPUs, with larger models requiring 2 GPUs\.

### A\.2On the Newton Schultz Approximation

#### A\.2\.1NanoGPT NS Coefficients

Here we present the NS polynomials used and popularized by the NanoGPT speedrun\[Jordanet al\.,[2024a](https://arxiv.org/html/2606.04058#bib.bib34), Cesistaet al\.,[2025](https://arxiv.org/html/2606.04058#bib.bib16)\]\.

b1​\(x\)\\displaystyle b\_\{1\}\(x\)=4\.0848​x−6\.8946​x3\+2\.9270​x5\\displaystyle=0848x\-8946x^\{3\}\+9270x^\{5\}\(2\)b2​\(x\)\\displaystyle b\_\{2\}\(x\)=3\.9505​x−6\.3029​x3\+2\.6377​x5\\displaystyle=9505x\-3029x^\{3\}\+6377x^\{5\}b3​\(x\)\\displaystyle b\_\{3\}\(x\)=3\.7418​x−5\.5913​x3\+2\.3037​x5\\displaystyle=7418x\-5913x^\{3\}\+3037x^\{5\}b4​\(x\)\\displaystyle b\_\{4\}\(x\)=2\.8769​x−3\.1427​x3\+1\.2046​x5\\displaystyle=8769x\-1427x^\{3\}\+2046x^\{5\}b5​\(x\)\\displaystyle b\_\{5\}\(x\)=2\.8366​x−3\.0525​x3\+1\.2012​x5\\displaystyle=8366x\-0525x^\{3\}\+2012x^\{5\}
The full NS map is thenf=b5∘b4∘b3∘b2∘b1f=b\_\{5\}\\circ b\_\{4\}\\circ b\_\{3\}\\circ b\_\{2\}\\circ b\_\{1\}\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x8.png)Figure 8:The NS mapf​\(σ\)=b5∘b4∘b3∘b2∘b1​\(σ\)f\(\\sigma\)=b\_\{5\}\\circ b\_\{4\}\\circ b\_\{3\}\\circ b\_\{2\}\\circ b\_\{1\}\(\\sigma\)forbib\_\{i\}in[Equation 2](https://arxiv.org/html/2606.04058#A1.E2)\.
#### A\.2\.2DeepSeek\-V4 NS Coefficients

\[DeepSeek\-AI,[2026](https://arxiv.org/html/2606.04058#bib.bib26)\]uses the following NS coefficients:

a​\(x\)=2​x−1\.5​x3\+0\.5​x5\\displaystyle a\(x\)=2x\-5x^\{3\}\+5x^\{5\}\(3\)c​\(x\)=3\.4445​x−4\.7750​x3\+2\.0315​x5\\displaystyle c\(x\)=4445x\-7750x^\{3\}\+0315x^\{5\}
The full NS map is thenf=a∘2∘c∘8f=a^\{\\circ 2\}\\circ c^\{\\circ 8\}\. While this approximation is very good \(see[Figure 9](https://arxiv.org/html/2606.04058#A1.F9)\), is uses 10 steps and hence is computationally more expensive\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x9.png)Figure 9:The NS mapf​\(σ\)=a∘2∘c∘8​\(σ\)f\(\\sigma\)=a^\{\\circ 2\}\\circ c^\{\\circ 8\}\(\\sigma\)foraaandccin[Equation 3](https://arxiv.org/html/2606.04058#A1.E3)\.

## Appendix BAppendix B

![Refer to caption](https://arxiv.org/html/2606.04058v1/x10.png)Figure 10:We observe that Muon needs around\(80−90\)%\(80\-90\)\\%of the iterations to match the final loss of the rankp=0\.25p=0\.25run\.
![Refer to caption](https://arxiv.org/html/2606.04058v1/x11.png)Figure 11:We observe that Muon needs around\(50−55\)%\(50\-55\)\\%of the iterations to match the final loss of the rankp=0\.1p=0\.1run\.

![Refer to caption](https://arxiv.org/html/2606.04058v1/x12.png)Figure 12:Quantile evolution for the 25% quantile for all layer types and model sizes\.![Refer to caption](https://arxiv.org/html/2606.04058v1/x13.png)Figure 13:Quantile evolution for the 50% quantile for all layer types and model sizes\.![Refer to caption](https://arxiv.org/html/2606.04058v1/x14.png)Figure 14:Normalized singular value spectra of the 2\.8B model at step 1450, with the dominant singular value removed, shown for every tracked weight matrix\.![Refer to caption](https://arxiv.org/html/2606.04058v1/x15.png)Figure 15:Quantile dynamics for the 50% quantile of the normalized momentum matrices for the 354M model under rank\-pporthonormal updates\. The trajectories forp=0\.9p=0\.9andp=0\.5p=0\.5closely track Muon’s quantile, while atp=0\.25p=0\.25andp=0\.1p=0\.1the deviation grows bigger\.![Refer to caption](https://arxiv.org/html/2606.04058v1/x16.png)Figure 16:Scaling laws for all tracked quantiles and layer types across model sizes\.