How Much Orthogonalization Does Muon Need?

# How Much Orthogonalization Does Muon Need?
Source: [https://arxiv.org/html/2606.00371](https://arxiv.org/html/2606.00371)
###### Abstract

Muon optimizers improve neural\-network training by replacing ill\-conditioned momentum updates with approximately semi\-orthogonal updates\. This motivates a practical question: how much orthogonalization does Muon actually require? We study this question using a relaxed cubic Newton–Schulz schedule derived directly for Muon’s low precision singular value band\. The resulting five\-step cubic construction uses ten dominant matrix multiplications, compared with fifteen for five quintic Newton–Schulz iterations\. The cubic schedule is not intended as a more accurate polar solver; instead, it is a principled low\-cost variant that lets us probe the relation between polar accuracy, spectral shaping, and training quality\. Across synthetic diagnostics, NanoGPT ablations, and training experiments on hybrid MoE/Mamba models, we find that training quality is not governed monotonically by polar\-decomposition accuracy: truncated Polar Express, Muon\-Jordan, cubic Newton–Schulz, and an explicit FP32 SVD polar factor can reach nearly indistinguishable final loss on GPT\-2 Small, and cubic5 matches the Muon\-Jordan quintic update within about10−310^\{\-3\}validation loss on hybrid MoE/Mamba models with one billion to four billion parameters\. These results support cubic5 as a practical low\-cost Muon orthogonalization variant, with empirical evidence of training\-quality parity in the settings tested\.

## 1Introduction

### 1\.1The Muon Optimizer

Muon is a newly proposed optimizer for matrix\-valued hidden layer parameters in neural networks\[[9](https://arxiv.org/html/2606.00371#bib.bib1)\]\. It can be viewed as stochastic gradient descent with momentum followed by an orthogonalization step\. LetWt∈ℝm×nW\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}be a weight matrix,GtG\_\{t\}its stochastic gradient, andMtM\_\{t\}a momentum buffer\. A simplified Muon update is

Mt\\displaystyle M\_\{t\}=β​Mt−1\+\(1−β\)​Gt,\\displaystyle=\\beta M\_\{t\-1\}\+\(1\-\\beta\)G\_\{t\},\(1\)Wt\+1\\displaystyle W\_\{t\+1\}=Wt−η​polar⁡\(Mt\)\.\\displaystyle=W\_\{t\}\-\\eta\\,\\operatorname\{polar\}\(M\_\{t\}\)\.\(2\)IfMt=U​Σ​V⊤M\_\{t\}=U\\Sigma V^\{\\top\}is a singular value decomposition, then

polar⁡\(Mt\)=U​V⊤\.\\operatorname\{polar\}\(M\_\{t\}\)=UV^\{\\top\}\.\(3\)Thus exact Muon would preserve the singular vectors of the momentum update but replace all nonzero singular values by one\. Jordan et al\. motivate this operation empirically by observing that transformer momentum updates are often nearly low\-rank: a few directions dominate the update, while many “rare” directions have small singular values but may still matter for learning\[[9](https://arxiv.org/html/2606.00371#bib.bib1)\]\. Orthogonalization can therefore be understood as a spectral reshaping operation that amplifies these suppressed directions\.

This perspective is different from the standard numerical linear algebra objective of computing a highly accurate polar factor\. In neural network training, the update must remain useful after interaction with momentum, learning rate schedules, normalization, weight decay, stochastic gradients, and low precision arithmetic\. An exact SVD polar factor is the natural mathematical reference point, but it need not optimize training loss\. The central question of this work is therefore not merely whether a Newton–Schulz iteration converges to the polar factor, but how accurately Muon must orthogonalize its updates to preserve training quality\.

The key observation behind these iterations is that odd matrix polynomials act diagonally on singular values\. IfX=U​Σ​V⊤X=U\\Sigma V^\{\\top\}and

X\+=a​X\+b​\(X​X⊤\)​X\+c​\(X​X⊤\)2​X,X^\{\+\}=aX\+b\(XX^\{\\top\}\)X\+c\(XX^\{\\top\}\)^\{2\}X,\(4\)then

X\+=U​\(a​Σ\+b​Σ3\+c​Σ5\)​V⊤\.X^\{\+\}=U\\left\(a\\Sigma\+b\\Sigma^\{3\}\+c\\Sigma^\{5\}\\right\)V^\{\\top\}\.\(5\)The matrix problem can therefore be studied through scalar polynomial maps on the singular values\. Classical polar decomposition\[[6](https://arxiv.org/html/2606.00371#bib.bib9),[5](https://arxiv.org/html/2606.00371#bib.bib10),[2](https://arxiv.org/html/2606.00371#bib.bib6),[11](https://arxiv.org/html/2606.00371#bib.bib5),[7](https://arxiv.org/html/2606.00371#bib.bib8),[13](https://arxiv.org/html/2606.00371#bib.bib11),[12](https://arxiv.org/html/2606.00371#bib.bib12)\]algorithms and recently proposed hybrid polar decomposition method\[[8](https://arxiv.org/html/2606.00371#bib.bib13)\]optimize these polynomials for high accuracy convergence to one\. Muon, however, is used in a deep learning regime where low precision approximate directions can be sufficient\. This creates room for polynomial schedules that are less accurate as polar solvers but cheaper or differently biased as spectral transforms\.

### 1\.2Polynomial Polar Decompositions for Muon

The original Muon implementation uses five iterations of a fixed quintic Newton–Schulz polynomial,

p​\(x\)=3\.4445​x−4\.7750​x3\+2\.0315​x5,p\(x\)=3\.4445x\-4\.7750x^\{3\}\+2\.0315x^\{5\},\(6\)run in bfloat16 after Frobenius normalization\[[9](https://arxiv.org/html/2606.00371#bib.bib1)\]\. We refer to this baseline as Muon\-Jordan\. Its coefficients were chosen for the relaxed Muon objective: after repeated composition, singular values are allowed to lie in a band around one rather than converging to machine precision\. Liu et al\.\[[10](https://arxiv.org/html/2606.00371#bib.bib2)\]further demonstrated that Muon can be scaled to larger Mixture\-of\-Experts \(MoE\) models containing 16 billion parameters\.

Polar Express revisits this polynomial\-design problem from the perspective of matrix sign and polar decomposition methods\[[1](https://arxiv.org/html/2606.00371#bib.bib3)\]\. It uses an adaptive sequence of quintic polynomials chosen by a minimax criterion\. This improves the quality of the polar approximation and, when inserted into Muon, can improve validation loss across learning rate sweeps\. Polar Express keeps the same basic computational structure as standard quintic Newton–Schulz: each iteration applies a degree\-five odd polynomial using three dominant matrix multiplications\.

A complementary direction is to reduce the cost of the same polynomial iteration by restructuring the computation\. Gram Newton–Schulz observes that many Muon matrices are rectangular and that standard Newton–Schulz repeatedly forms symmetric Gram matrices\[[15](https://arxiv.org/html/2606.00371#bib.bib4)\]\. By moving more work to the smaller Gram matrix and using hardware\-aware symmetric kernels, Gram Newton–Schulz reduces the runtime of the orthogonalization step while preserving the underlying polynomial iteration\. This is orthogonal to coefficient design: one can improve the polynomial, the kernel, or both\.

### 1\.3Contributions

This work uses a relaxed cubic construction to probe a broader question: how much orthogonalization does Muon need, and what kind of spectral shaping matters for training? We make the following contributions\.

- •We derive an adaptive cubic Newton–Schulz schedule whose coefficients are chosen from the current worst\-case lower singular value bound and a relaxed target band\[0\.7,1\.3\]\[0\.7,1\.3\]\.
- •We identify a practical five\-step schedule using the bfloat16 effective lower boundl0=7×10−3l\_\{0\}=7\\times 10^\{\-3\}\. This schedule uses 10 dominant matrix multiplications, compared with 15 for five quintic iterations\.
- •We compare cubic5, truncated Polar Express, truncated Muon\-Jordan, and an explicit FP32 SVD polar factor inside NanoGPT training\. This separates polar\-decomposition accuracy from optimizer quality: exact SVD does not improve over the strongest approximate Newton–Schulz updates in our GPT\-2 Small runs\.
- •We characterize a cost\-quality frontier\. Cubic5 is close to Polar Express and SVD at the default GPT\-2 Small setting, and it remains within approximately10−310^\{\-3\}validation loss of the Muon\-Jordan quintic update on hybrid MoE/Mamba models with 1B to 4B parameters\. These results support cubic5 as a viable low\-cost Muon orthogonalization variant in the evaluated regimes, not as a uniformly better update rule\.
- •We validate the scalar construction on synthetic matrices and show by microbenchmarking that reducing the polynomial degree lowers the cost of the orthogonalization subroutine\.

Our claims are intentionally calibrated\. Cubic5 is not presented as a more accurate polar decomposition algorithm than Polar Express, nor as a uniformly better Muon update\. Rather, it is a practical low\-cost Muon orthogonalization variant whose empirical behavior also helps probe how polynomial\-induced spectral transforms affect training, beyond classical polar convergence alone\.

## 2Relaxed Cubic Newton–Schulz for Muon Optimizer

### 2\.1Relaxed singular value Targets

Classical polar decomposition aims to map every nonzero singular value to one\. Muon relaxes this requirement\. The original Muon coefficient search was guided by the observation that training can tolerate singular values in a band such as\[0\.7,1\.3\]\[0\.7,1\.3\]\[[9](https://arxiv.org/html/2606.00371#bib.bib1)\]\. This changes the polynomial\-design problem: the relaxed band is a useful design target, not a guarantee that training quality is determined solely by whether every singular value lands in that interval\.

We work with the normalized iterateX0=X/‖X‖FX\_\{0\}=X/\\\|X\\\|\_\{F\}, so all singular values are at most one\. In exact arithmetic, the smallest nonzero singular value may be arbitrarily small\. Since Muon is used in a neural network training setting with bfloat16 precision, a lower bound far below the relative precision is not practically meaningful\. We therefore use

l0=7×10−3<ϵbf16=2−7=0\.0078125l\_\{0\}=7\\times 10^\{\-3\}<\\epsilon\_\{\\text\{bf16\}\}=2^\{\-7\}=0\.0078125\(7\)as an effective lower bound\. This value produces a schedule that reaches the relaxed lower target0\.70\.7in five cubic iterations\. Muon\-Jordan and Polar Express use a more conservative lower boundl0=10−3l\_\{0\}=10^\{\-3\}\. If we use the same lower bound, the schedule requires seven cubic iterations\.

### 2\.2Cubic Coefficient Derivation

We derive the cubic coefficients using the same approach as Chen–Chow’s scaled cubic Newton–Schulz method\[[3](https://arxiv.org/html/2606.00371#bib.bib7)\]\. At iterationtt, suppose all protected singular values lie in an interval\[lt,rt\]\[l\_\{t\},r\_\{t\}\]\. We choose an odd cubic polynomial

ft​\(x\)=at​x\+bt​x3,bt<0,f\_\{t\}\(x\)=a\_\{t\}x\+b\_\{t\}x^\{3\},\\qquad b\_\{t\}<0,\(8\)with peak valueu=1\.3u=1\.3\. Sincebt<0b\_\{t\}<0, the polynomial increases until its critical pointktk\_\{t\}and then decreases\. We impose three conditions:

ft′​\(kt\)\\displaystyle f\_\{t\}^\{\\prime\}\(k\_\{t\}\)=0,\\displaystyle=0,\(9\)ft​\(kt\)\\displaystyle f\_\{t\}\(k\_\{t\}\)=u,\\displaystyle=u,\(10\)ft​\(lt\)\\displaystyle f\_\{t\}\(l\_\{t\}\)=ft​\(rt\)\.\\displaystyle=f\_\{t\}\(r\_\{t\}\)\.\(11\)The first two conditions set the shape oncektk\_\{t\}is known\. Fromft′​\(x\)=at\+3​bt​x2f\_\{t\}^\{\\prime\}\(x\)=a\_\{t\}\+3b\_\{t\}x^\{2\},

at=−3​bt​kt2\.a\_\{t\}=\-3b\_\{t\}k\_\{t\}^\{2\}\.\(12\)Combining this withft​\(kt\)=uf\_\{t\}\(k\_\{t\}\)=ugives

bt=−u2​kt3,at=3​u2​kt\.b\_\{t\}=\-\\frac\{u\}\{2k\_\{t\}^\{3\}\},\\qquad a\_\{t\}=\\frac\{3u\}\{2k\_\{t\}\}\.\(13\)The endpoint condition determines the peak location\. Substituting the above form intoft​\(lt\)=ft​\(rt\)f\_\{t\}\(l\_\{t\}\)=f\_\{t\}\(r\_\{t\}\)yields

3​lt​kt2−lt3=3​rt​kt2−rt3,3l\_\{t\}k\_\{t\}^\{2\}\-l\_\{t\}^\{3\}=3r\_\{t\}k\_\{t\}^\{2\}\-r\_\{t\}^\{3\},\(14\)and hence

kt2=rt2\+rt​lt\+lt23\.k\_\{t\}^\{2\}=\\frac\{r\_\{t\}^\{2\}\+r\_\{t\}l\_\{t\}\+l\_\{t\}^\{2\}\}\{3\}\.\(15\)Equivalently, if

αt=1kt=3rt2\+rt​lt\+lt2,\\alpha\_\{t\}=\\frac\{1\}\{k\_\{t\}\}=\\sqrt\{\\frac\{3\}\{r\_\{t\}^\{2\}\+r\_\{t\}l\_\{t\}\+l\_\{t\}^\{2\}\}\},\(16\)then

ft​\(x\)=u2​\(3​αt​x−αt3​x3\)\.f\_\{t\}\(x\)=\\frac\{u\}\{2\}\\left\(3\\alpha\_\{t\}x\-\\alpha\_\{t\}^\{3\}x^\{3\}\\right\)\.\(17\)
The endpoint equality is important\. Because the cubic has a single interior maximum on the protected interval, the minimum over\[lt,rt\]\[l\_\{t\},r\_\{t\}\]occurs at one of the endpoints\. Enforcingft​\(lt\)=ft​\(rt\)f\_\{t\}\(l\_\{t\}\)=f\_\{t\}\(r\_\{t\}\)balances the two endpoints and maximizes the new worst\-case lower bound under the chosen peak constraint\.

### 2\.3Adaptive Worst\-Case Schedule

The coefficients are recomputed at every step from the current lower bound\. We initialize

l0=7×10−3,r0=1,l\_\{0\}=7\\times 10^\{\-3\},\\qquad r\_\{0\}=1,\(18\)and after the first step protect the relaxed upper range by setting

rt=u=1\.3,t≥1\.r\_\{t\}=u=1\.3,\\qquad t\\geq 1\.\(19\)The worst\-case lower bound evolves as

lt\+1=ft​\(lt\)\.l\_\{t\+1\}=f\_\{t\}\(l\_\{t\}\)\.\(20\)Forl0=7×10−3l\_\{0\}=7\\times 10^\{\-3\}, this construction reacheslt≥0\.7l\_\{t\}\\geq 0\.7after five steps\. The resulting cubic5 coefficients are

The corresponding matrix iteration is

Xt\+1=at​Xt\+bt​\(Xt​Xt⊤\)​Xt,X\_\{t\+1\}=a\_\{t\}X\_\{t\}\+b\_\{t\}\(X\_\{t\}X\_\{t\}^\{\\top\}\)X\_\{t\},\(21\)with the usual transpose trick for tall matrices so that the smaller Gram matrix is formed\. This preserves singular vectors and appliesftf\_\{t\}to each singular value\.

![Refer to caption](https://arxiv.org/html/2606.00371v1/figures/relaxed_cubic_ns.png)Figure 1:Adaptive relaxed cubic Newton–Schulz schedule for the effective bfloat16 lower boundl0=7×10−3l\_\{0\}=7\\times 10^\{\-3\}\. The coefficients are recomputed from the current worst\-case lower bound, and the scalar lower\-bound trajectory reaches the relaxed target band in five cubic iterations\.
### 2\.4FLOP and Matrix\-Multiplication Count

We count only dominant matrix multiplications and ignore scalar multiplications, matrix additions, and normalization\. A cubic step

Xt\+1=at​Xt\+bt​\(Xt​Xt⊤\)​XtX\_\{t\+1\}=a\_\{t\}X\_\{t\}\+b\_\{t\}\(X\_\{t\}X\_\{t\}^\{\\top\}\)X\_\{t\}\(22\)requires two dominant multiplications: one to form the Gram matrix and one to multiply it back intoXtX\_\{t\}\. Therefore five cubic steps require ten dominant matrix multiplications\.

A quintic Newton–Schulz step of the form

Xt\+1=at​Xt\+bt​\(Xt​Xt⊤\)​Xt\+ct​\(Xt​Xt⊤\)2​XtX\_\{t\+1\}=a\_\{t\}X\_\{t\}\+b\_\{t\}\(X\_\{t\}X\_\{t\}^\{\\top\}\)X\_\{t\}\+c\_\{t\}\(X\_\{t\}X\_\{t\}^\{\\top\}\)^\{2\}X\_\{t\}\(23\)requires three dominant multiplications: one for the Gram matrix, one for the Gram\-square or polynomial\-in\-Gram term, and one to multiply the result byXtX\_\{t\}\. Five quintic steps, including Muon\-Jordan and Polar Express, therefore require fifteen dominant multiplications\.

For anm×nm\\times nmatrix withm≥nm\\geq n, each cubic step has leading cost approximately

2​m​n2\+2​m​n2=4​m​n22mn^\{2\}\+2mn^\{2\}=4mn^\{2\}\(24\)floating point operations when the smallern×nn\\times nGram matrix is used\. A quintic step adds ann×nn\\times nmultiplication, yielding roughly

4​m​n2\+2​n3\.4mn^\{2\}\+2n^\{3\}\.\(25\)For highly rectangular matrices, the rectangular multiplications dominate and the cubic schedule saves roughly one third of the dominant multiplications\. For near\-square matrices, then3n^\{3\}Gram\-square term is also substantial, so the benefit remains meaningful\. Actual wall\-clock speedups depend on batching, kernel fusion, symmetric\-kernel use, matrix aspect ratio, and whether the orthogonalization subroutine is a large fraction of the optimizer step\[[15](https://arxiv.org/html/2606.00371#bib.bib4)\]\.

## 3Numerical Results

The experiments are organized around two questions\. First, does the relaxed cubic derivation produce the expected spectral transform and subroutine cost reduction? Second, how closely does training quality track polar accuracy once the orthogonalization routine is inserted into Muon? This second question is central to the optimizer setting: an update can be numerically less accurate as a polar factor while still being similarly useful for training\.

### 3\.1Singular Value Diagnostics and Orthogonalization Microbenchmark

The first experiment checks whether the scalar construction transfers to matrix polynomials in low precision\. We construct an FP32 square matrixM=U​diag⁡\(s\)​V⊤M=U\\operatorname\{diag\}\(s\)V^\{\\top\}with controlled singular valuess=logspace⁡\(10−5,1\)s=\\operatorname\{logspace\}\(10^\{\-5\},1\), cast it to bfloat16, run the matrix polynomial, and compute output singular values in FP32\. The diagnostic compares cubic5 in bfloat16, cubic5 in FP32, Polar Express in bfloat16, and Muon\-Jordan in bfloat16\.

![Refer to caption](https://arxiv.org/html/2606.00371v1/x1.png)Figure 2:Synthetic singular value diagnostic\. Left: scalar polynomial transfer on a wide range of input singular values\. Right: output singular value spectrum for matrices whose input singular values lie in the protected interval\[7×10−3,1\]\[7\\times 10^\{\-3\},1\]\.The result supports the intended interpretation of the bfloat16 lower bound\. The method does not claim that singular values below7×10−37\\times 10^\{\-3\}are guaranteed to enter the relaxed band\. Instead, singular values at or above this effective lower bound are mapped stably into the target region, while much smaller singular values do not produce numerical blow\-up or contaminate the output spectrum\. The FP32 and bfloat16 cubic curves are close on the protected range, which indicates that the scalar derivation remains predictive for the matrix implementation\.

We also benchmark the orthogonalization subroutine on representative matrix shapes on an NVIDIA H200 GPU\. The microbenchmark shows that the cubic5 routine reduces the cost of the orthogonalization step relative to Polar Express\. This is consistent with the dominant multiplication count: both methods use the same normalization and general Newton–Schulz structure, but cubic5 avoids the Gram\-square multiplication in each iteration\. We treat this as a subroutine\-level result, not as a claim of proportional end\-to\-end training speedup\.

Table 1:Orthogonalization\-subroutine microbenchmark\. Matrix shapes follow the Kimi K2 and Llama 3 70B examples used in the Gram Newton–Schulz discussion\. Speedup is the median runtime ratio of Polar Express over relaxed cubic5 for the same synthetic gradient matrix\.
### 3\.2Orthogonalization Accuracy vs\. Training Quality

The central training ablation compares four ways of producing a Muon update on GPT\-2 Small: the relaxed cubic schedule truncated to one through five steps, Polar Express truncated to one through five steps, Muon\-Jordan truncated to one through five steps, and an explicit FP32 SVD polar factor\. The SVD baseline computes the polar factor most directly, but it is much too expensive to be a practical Muon routine\. Its purpose here is conceptual: it distinguishes exact polar orthogonalization from the update that achieves the best validation loss\. Figure[3](https://arxiv.org/html/2606.00371#S3.F3)shows the result for random seed0\. We repeated the same ablation with seeds4242and3355033633550336and observed the same qualitative pattern\.

![Refer to caption](https://arxiv.org/html/2606.00371v1/x2.png)Figure 3:Final validation loss versus orthogonalization steps on GPT\-2 Small\. Cubic5, Polar Express, and Muon\-Jordan all approach the same quality region as more steps are added, while the explicit FP32 SVD polar factor does not improve over the strongest approximate Newton–Schulz updates\.This comparison is informative\. Cubic5 improves monotonically from final loss3\.38053\.3805with one step to3\.28113\.2811with five steps\. Polar Express also improves with additional steps, reaching3\.27983\.2798at five steps\. Muon\-Jordan improves rapidly through four steps, from3\.37293\.3729to3\.28093\.2809, and then slightly increases to3\.28223\.2822at five steps\. The gap among the strongest variants is very small: five\-step cubic5 reaches3\.28113\.2811, four\-step Polar Express reaches3\.28243\.2824, five\-step Polar Express reaches3\.27983\.2798, four\-step Muon\-Jordan reaches3\.28093\.2809, and the explicit FP32 SVD polar factor also reaches3\.28093\.2809\. These runs differ by less than0\.0030\.003final validation loss, while their numerical interpretation and matrix\-multiplication costs are quite different\.

This ablation suggests that optimizer quality is not determined solely by polar decomposition accuracy\. If polar accuracy were the governing factor, the SVD baseline would be expected to dominate the approximate polynomial methods\. Instead, training quality appears to depend on the effective spectral transform applied to the momentum update\. Polar Express shapes the spectrum more effectively than cubic5 at very small step counts, but by five cubic steps the cheaper cubic5 schedule reaches the same quality region\. The non\-monotone five\-step Muon\-Jordan result further indicates that adding another approximate orthogonalization step need not monotonically improve training loss\. We therefore view cubic5 as a principled point on a cost\-quality frontier rather than as a method that dominates truncated quintic or exact SVD baselines\.

### 3\.3Nemotron\-3\-Nano Hybrid MoE/Mamba Scaling Experiments

The NanoGPT ablation in Figure[3](https://arxiv.org/html/2606.00371#S3.F3)isolates the effect of the orthogonalization routine in a controlled small\-model setting\. We next test whether the same relaxed cubic update remains viable when training larger Nemotron\-3\-Nano models\[[14](https://arxiv.org/html/2606.00371#bib.bib14)\]with Megatron\-LM\. We use three hybrid MoE/Mamba configurations: 1B\-A315M, 2B\-A500M, and 4B\-A770M \(with 1B/2B/4B total parameters and 315M/500M/770M active parameters, respectively\)\. Their core architectural hyperparameters are listed in Table[2](https://arxiv.org/html/2606.00371#S3.T2)\. All three models use grouped query attention with 2 query groups, 128 experts, top\-6 routing, a sigmoid router score, and sequence\-level auxiliary load\-balancing loss with coefficient10−410^\{\-4\}\. We use 88B/140B/215B token subsets of the full Nemotron\-3 dataset\[[14](https://arxiv.org/html/2606.00371#bib.bib14)\]for the training\. Training uses sequence length 8192, bfloat16 arithmetic, global batch size 768, Muon optimizer, and a warmup\-stable\-decay learning\-rate schedule\[[4](https://arxiv.org/html/2606.00371#bib.bib15)\]with a minus\-square\-root decay style for the final WSD decay\. The token budgets and learning\-rate ranges are summarized in Table[3](https://arxiv.org/html/2606.00371#S3.T3)\.

Table 2:Core architectural hyperparameters for the Nemotron\-3\-Nano hybrid MoE/Mamba scaling experiments\. All three models use 128 experts and top\-6 routing\. The FFN column denotes the per\-expert hidden size, and the shared\-expert column denotes the shared expert intermediate size\.Table 3:Training token budgets and learning\-rate ranges for the Nemotron\-3\-Nano model scaling experiments\.Table 4:Final validation metrics for Nemotron\-3\-Nano models trained with Megatron\-LM\. The Muon\-Jordan baseline uses the fixed quintic five\-step Newton–Schulz polynomial, while cubic5 uses the relaxed cubic coefficients\.Δ\\Deltais cubic5 minus Muon\-Jordan; negative values favor cubic5\.![Refer to caption](https://arxiv.org/html/2606.00371v1/x3.png)Figure 4:Training loss comparison for the Nemotron\-3\-Nano\-4B\-A770M model\. The inset shows the relative difference between cubic5 and Muon\-Jordan using the same trained\-token axis\.The final validation losses are nearly indistinguishable at all three scales\. For the 1B and 2B models, cubic5 is slightly lower than the Muon\-Jordan quintic update on both validation loss and PPL loss\. For the 4B model, cubic5 is slightly higher, by about10−310^\{\-3\}validation loss and0\.0040\.004PPL loss\. The corresponding test\-set metrics show the same qualitative pattern: cubic5 is lower by0\.0007190\.000719and0\.0000960\.000096test loss on the 1B and 2B models respectively, and higher by0\.0009360\.000936test loss on the 4B model\.

The 4B training curve in Figure[4](https://arxiv.org/html/2606.00371#S3.F4)provides a trajectory\-level view of the largest scaling experiment\. The relative difference is largest \(about 2%\) at the beginning of training, but it decays rapidly and then fluctuates around zero for the remaining token budget\. This pattern is consistent with a transient difference in the early optimization dynamics rather than a persistent shift in the training trajectory\. After the initial phase, cubic5 and Muon\-Jordan enter the same loss regime\. The late\-stage behavior is therefore consistent with the final validation metrics: cubic5 does not improve over Muon\-Jordan on the 4B model, but it preserves essentially the same training trajectory while using a lower\-degree orthogonalization polynomial\. The 1B and 2B model training curves are very similar to the 4B model and we omit them for brevity\.

Taken together, these experiments provide the main training\-scale evidence in this draft\. Cubic5 does not uniformly improve over Muon\-Jordan, but it reaches comparable validation metrics in these scaling experiments while using a lower\-degree orthogonalization polynomial\. This supports cubic5 as a viable low\-cost Muon orthogonalization variant in regimes where the orthogonalization subroutine is a relevant part of optimizer cost\. It also reinforces the smaller ablation’s message that, once the spectral transform is in a suitable regime, training quality can be relatively insensitive to the exact polynomial used for approximate orthogonalization\.

## 4Discussion and Limitations

The main conclusion is that Muon’s orthogonalization requirement is an optimizer requirement, not simply a polar\-decomposition accuracy requirement\. The relaxed cubic schedule was derived from a worst\-case singular value target, but the training experiments show that this target is neither a necessary nor a sufficient description of optimizer quality\. Four\-step Polar Express and four\-step Muon\-Jordan can work well even though their scalar behavior differs from both exact SVD and the full five\-step schedules, and an explicit FP32 SVD polar factor does not improve over the strongest approximate Newton–Schulz updates on GPT\-2 Small\. This suggests that Muon benefits from a broader spectral reshaping effect: ill\-conditioned momentum updates are made more isotropic, but the exact singular value map that best serves training depends on the training phase, learning rate, and real update spectra\.

The evidence therefore positions cubic5 as a principled low\-cost point on the Muon orthogonalization frontier\. Its advantage is computational: it removes one dominant matrix multiplication from each Newton–Schulz step by using a cubic instead of a quintic polynomial, giving a five\-step routine with ten dominant matrix multiplications\. The Nemotron\-3\-Nano experiments suggest that cubic5 can match the Muon\-Jordan quintic update closely on larger hybrid MoE/Mamba models, but the differences are small enough that they should be interpreted as equivalence\-level evidence rather than as a quality advantage\. This motivates further evaluation of cubic5 as a low\-cost Muon orthogonalization variant, without claiming universal dominance over Muon\-Jordan or Polar Express\.

The SVD comparison also changes how future Muon polynomial work should be evaluated\. Exact polar decomposition is the natural mathematical endpoint, but it is not a reliable proxy for training quality\. More informative diagnostics should measure how a candidate update compares to strong Muon baselines on the spectra that actually occur during training: update cosine similarity, Frobenius relative error to Polar Express or SVD, amplification of bulk versus tail singular directions, and how these quantities evolve during the early phase of training\. Such measurements would connect scalar polynomial design to optimizer dynamics more directly than asking only whether all singular values enter a fixed relaxed band\.

Finally, the current evidence separates subroutine speed from end\-to\-end training speed\. Reducing the Newton–Schulz multiplication count lowers the cost of the orthogonalization routine, but the full optimizer step includes communication, parameter updates, data movement, and non\-Muon parameters\. The Nemotron\-3\-Nano experiments in Table[4](https://arxiv.org/html/2606.00371#S3.T4)primarily establish training\-quality parity; they should not yet be read as an end\-to\-end throughput study\. End\-to\-end speedups will depend on model architecture, matrix aspect ratios, hardware kernels, and parallelism\. Future work should combine relaxed polynomial schedules with hardware\-aware implementations such as Gram Newton–Schulz and measure full training wall\-clock time at larger scales\.

## Acknowledgements

The author sincerely thanks Wenkai Shao, Mikail Khona, Hao Wu, and Jin Zhou for their valuable discussions and assistance with the experiments\.

## References

- \[1\]N\. Amsel, D\. Persson, C\. Musco, and R\. M\. Gower\(2026\)The polar express: optimal matrix sign methods and their application to the muon algorithm\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yRtgZ1K8hO)Cited by:[§1\.2](https://arxiv.org/html/2606.00371#S1.SS2.p2.1)\.
- \[2\]R\. Byers and H\. Xu\(2008\-01\)A new scaling for Newton’s iteration for the polar decomposition and its backward stability\.SIAM Journal on Matrix Analysis and Applications30\(2\),pp\. 822–843\.External Links:[Document](https://dx.doi.org/10.1137/070699895)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[3\]J\. Chen and E\. Chow\(2014\)A stable scaling of Newton\-Schulz for improving the sign function computation of a Hermitian matrix\(Website\)Note:Preprint ANL/MCS\-P5059\-0114, Argonne National LaboratoryCited by:[§2\.2](https://arxiv.org/html/2606.00371#S2.SS2.p1.2)\.
- \[4\]A\. Hägele, E\. Bakouch, A\. Kosson, L\. B\. Allal, L\. V\. Werra, and M\. Jaggi\(2024\)Scaling laws and compute\-optimal training beyond fixed training durations\.External Links:2405\.18392,[Link](https://arxiv.org/abs/2405.18392)Cited by:[§3\.3](https://arxiv.org/html/2606.00371#S3.SS3.p1.1)\.
- \[5\]N\. J\. Higham and R\. S\. Schreiber\(1990\)Fast polar decomposition of an arbitrary matrix\.SIAM Journal on Scientific and Statistical Computing11\(4\),pp\. 648–655\.External Links:[Document](https://dx.doi.org/10.1137/0911038)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[6\]N\. J\. Higham\(1986\)Computing the polar decomposition\-with applications\.SIAM Journal on Scientific and Statistical Computing7\(4\),pp\. 1160–1174\.External Links:[Document](https://dx.doi.org/10.1137/0907079)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[7\]N\. J\. Higham\(2008\)Functions of matrices: theory and computation\.SIAM,Philadelphia, PA, USA\.Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[8\]H\. Huang\(2024\-07\)New parallel algorithms for large\-scale matrix computations\.Ph\.D\. Thesis,Georgia Institute of Technology\.Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[9\]K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein\(2024\)Muon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p1.3),[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p1.5),[§1\.2](https://arxiv.org/html/2606.00371#S1.SS2.p1.2),[§2\.1](https://arxiv.org/html/2606.00371#S2.SS1.p1.1)\.
- \[10\]J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. Yang\(2025\)Muon is scalable for LLM training\.External Links:2502\.16982,[Link](https://arxiv.org/abs/2502.16982)Cited by:[§1\.2](https://arxiv.org/html/2606.00371#S1.SS2.p1.2)\.
- \[11\]Y\. Nakatsukasa, Z\. Bai, and F\. Gygi\(2010\-09\)Optimizing Halley’s iteration for computing the matrix polar decomposition\.SIAM Journal on Matrix Analysis and Applications31\(5\),pp\. 2700–2720\.External Links:[Document](https://dx.doi.org/10.1137/090774999)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[12\]Y\. Nakatsukasa and R\. W\. Freund\(2016\-01\)Computing fundamental matrix decompositions accurately via the matrix sign function in two iterations: the power of Zolotarev’s functions\.SIAM Review58\(3\),pp\. 461–493\.External Links:[Document](https://dx.doi.org/10.1137/140990334)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[13\]Y\. Nakatsukasa and N\. J\. Higham\(2013\-01\)Stable and efficient spectral divide and conquer algorithms for the symmetric eigenvalue decomposition and the svd\.SIAM Journal on Scientific Computing35\(3\),pp\. A1325–A1349\.External Links:[Document](https://dx.doi.org/10.1137/120876605)Cited by:[§1\.1](https://arxiv.org/html/2606.00371#S1.SS1.p3.3)\.
- \[14\]NVIDIA, A\. Blakeman, A\. Basant, A\. Khattar, A\. Renduchintala, A\. Bercovich, A\. Ficek, A\. Bjorlin, A\. Taghibakhshi, A\. S\. Deshmukh, A\. S\. Mahabaleshwarkar, A\. Tao, A\. Shors, A\. Aithal, A\. Poojary, A\. Dattagupta, B\. Buddharaju, B\. Chen, B\. Ginsburg, B\. Wang, B\. Norick, B\. Butterfield, B\. Catanzaro, C\. del Mundo, C\. Dong, C\. Harvey, C\. Parisien, D\. Su, D\. Korzekwa, D\. Yin, D\. Gitman, D\. Mosallanezhad, D\. Narayanan, D\. Fridman, D\. Rekesh, D\. Ma, D\. Pykhtar, D\. Ahn, D\. Riach, D\. Stosic, E\. Long, E\. Segal, E\. Evans, E\. Chung, E\. Galinkin, E\. Bakhturina, E\. Dobrowolska, F\. Jia, F\. Liu, G\. Prasad, G\. Shen, G\. Liu, G\. Chen, H\. Qian, H\. Ngo, H\. Liu, H\. Li, I\. Gitman, I\. Karmanov, I\. Moshkov, I\. Golan, J\. Kautz, J\. P\. Scowcroft, J\. Casper, J\. Seppanen, J\. Lu, J\. Sewall, J\. Zeng, J\. You, J\. Zhang, J\. Zhang, J\. Huang, J\. Xue, J\. Huang, J\. Conway, J\. Kamalu, J\. Barker, J\. Cohen, J\. Jennings, J\. Parmar, K\. Sapra, K\. Briski, K\. Chumachenko, K\. Luna, K\. Santhanam, K\. Kong, K\. Sivamani, K\. Pawelec, K\. Anik, K\. Li, L\. McAfee, L\. Derczynski, L\. Pavao, L\. Vega, L\. Voegtle, M\. Bala, M\. R\. de Melo, M\. N\. Sreedhar, M\. Chochowski, M\. Kliegl, M\. Stepniewska\-Dziubinska, M\. Le, M\. Novikov, M\. Samadi, M\. Andersch, M\. Evans, M\. Martinez, M\. Chrzanowski, M\. Ranzinger, M\. Blaz, M\. Smelyanskiy, M\. Fawzy, M\. Shoeybi, M\. Patwary, N\. Lee, N\. Tajbakhsh, N\. Xu, O\. Rybakov, O\. Kuchaiev, O\. Delalleau, O\. Nitski, P\. Chadha, P\. Shamis, P\. Micikevicius, P\. Molchanov, P\. Dykas, P\. Fischer, P\. Aquilanti, P\. Bialecki, P\. Varshney, P\. Gundecha, P\. Tredak, R\. Karimi, R\. Kandu, R\. El\-Yaniv, R\. Joshi, R\. Waleffe, R\. Zhang, S\. Kavanaugh, S\. Jain, S\. Kriman, S\. Lym, S\. Satheesh, S\. Muralidharan, S\. Narenthiran, S\. Anandaraj, S\. Bak, S\. Kashirsky, S\. Han, S\. Acharya, S\. Ghosh, S\. T\. Sreenivas, S\. Clay, S\. Thomas, S\. Prabhumoye, S\. Pachori, S\. Toshniwal, S\. Prayaga, S\. Jain, S\. Das, S\. Kierat, S\. Majumdar, S\. Han, S\. Singhal, S\. Niverty, S\. Alborghetti, S\. Panguluri, S\. Bhendigeri, S\. N\. Akter, S\. Migacz, T\. Shiri, T\. Kong, T\. Roman, T\. Ronen, T\. Saar, T\. Konuk, T\. Rintamaki, T\. Poon, U\. De, V\. Noroozi, V\. Singh, V\. Korthikanti, V\. Kurin, W\. U\. Ahmad, W\. Du, W\. Ping, W\. Dai, W\. Byeon, X\. Ren, Y\. Xu, Y\. Choi, Y\. Zhang, Y\. Lin, Y\. Suhara, Z\. Yu, Z\. Li, Z\. Li, Z\. Zhu, Z\. Yang, and Z\. Chen\(2025\)Nemotron\-H: a family of accurate and efficient hybrid mamba\-transformer models\.External Links:2504\.03624,[Link](https://arxiv.org/abs/2504.03624)Cited by:[§3\.3](https://arxiv.org/html/2606.00371#S3.SS3.p1.1)\.
- \[15\]J\. Zhang, N\. Amsel, B\. Chen, and T\. Dao\(2026\)Gram Newton\-Schulz\.External Links:[Link](https://dao-ailab.github.io/blog/2026/gram-newton-schulz/)Cited by:[§1\.2](https://arxiv.org/html/2606.00371#S1.SS2.p3.1),[§2\.4](https://arxiv.org/html/2606.00371#S2.SS4.p3.5)\.