Scaling Muon for Diffusion Transformers

# Scaling Muon for Diffusion Transformers
Source: [https://arxiv.org/html/2608.20818](https://arxiv.org/html/2608.20818)
Chenghao Li1,2, Xiao Han2, Xinxin Huang2, Wei Liu2, Boyang Li2, Bing Xiao2,Heran Zhang2, Juanma Perez Rua2, Ke Xu2, Kangning Liu2, Linjun Kuang2, Na Li2,Tan Wang2, Tian Xie2, Wei Peng2, Yang Pei2, Yifan Xu2, Yuanhao Zhai2,Yuwei Lin2, Zhe Wang2, Zihao He2, Daniel Li2, Junbiao Tang2, Ziyang Jiang2, Dake Chen21University of Southern California2Meta

###### Abstract

The matrix\-aware optimizer Muon improves large model training by balancing updates across singular directions, yet its scaling behavior and end\-to\-end efficiency on large Diffusion Transformers \(DiTs\) remain unclear\. We first establish Muon’s scaling behavior on DiTs from 1\.3B to 15B parameters, showing that its optimization and generative quality advantages over AdamW persist across model scales\. However, at scale, the 5\-step Newton–Schulz iteration \(NS5\) performed at every optimization step, together with full\-momentum materialization, introduces substantial computation and communication overhead that can offset Muon’s step\-efficiency advantage\. We introduce*Periodic Row\-wise Muon*, which performs a full NS5 spectral update once everyKKsteps and applies a low compute and communication cost row\-wise constrained update based on the current momentum at the remaining steps\. We further co\-design a distributed implementation that operates directly on sharded momentum during non\-refresh steps and accelerates spectral refreshes through bucketed all\-gather and communication–computation overlap\. Across all scales, Muon improves the best observed generative quality over AdamW by 12\.9–19\.1%\. Compared with vanilla Muon, Periodic Row\-wise Muon remains within 0\.5% in best generative quality on the 1\.3B–4B models and improves it by 4\.5% at 9B\. It reduces optimizer time by 46\.9–54\.3%, end\-to\-end step time by 15\.7–24\.3%, and logical communication volume by 66\.7%, while reaching its respective best generative quality with 33\.7–64\.8% less active training time\. These results show that Periodic Row\-wise Muon preserves Muon’s generative quality advantage while translating it into end\-to\-end training efficiency for large DiTs\.

## 1Introduction

The continued scaling of Diffusion Transformers \(DiTs\)\([26](https://arxiv.org/html/2608.20818#bib.bib21);[11](https://arxiv.org/html/2608.20818#bib.bib16);[8](https://arxiv.org/html/2608.20818#bib.bib22)\)has improved model capacity and generative quality, while making training efficiency a central concern\. Prior work has characterized predictable improvements with increased training compute\([36](https://arxiv.org/html/2608.20818#bib.bib18);[22](https://arxiv.org/html/2608.20818#bib.bib17)\), but has typically treated the optimizer as fixed\. At billion\-parameter scale, however, optimizer efficiency can no longer be judged solely by the number of steps required to reach a lower loss\. Fewer optimization steps must translate into lower GPU\-hours and shorter wall\-clock time under distributed training\. Evaluating optimizer scalability therefore requires considering both its optimization advantage and its realized compute and communication cost under distributed training\.

Muon\([15](https://arxiv.org/html/2608.20818#bib.bib4)\)presents a promising alternative for improving large\-scale training efficiency\. Unlike AdamW\([24](https://arxiv.org/html/2608.20818#bib.bib23)\), which applies coordinate\-wise adaptive updates, Muon treats two\-dimensional weight tensors as matrices and applies a finite\-step Newton–Schulz \(NS\) transformation to their momentum, producing update directions with global spectral structure\. Recent work has shown that, when properly calibrated, Muon can match AdamW with substantially fewer training FLOPs in large language model pretraining and post\-training\([23](https://arxiv.org/html/2608.20818#bib.bib5)\)\. Evidence from smaller diffusion models likewise suggests improved optimization despite a higher per\-step compute and communication cost, while showing that loss, generative quality, and runtime may rank optimizers differently\([30](https://arxiv.org/html/2608.20818#bib.bib19)\)\. It therefore remains unclear whether Muon’s optimization advantage persists as DiTs scale to ten\-billion parameters and whether its distributed overhead can be reduced sufficiently to improve end\-to\-end training efficiency\.

Answering these questions requires considering optimization and compute and communication costs jointly\. Under the sharded execution path studied in this work, Muon’s NS5 transformation introduces additional matrix multiplications, full\-momentum communication, synchronization, and temporary materialization\([23](https://arxiv.org/html/2608.20818#bib.bib5)\)\. These costs motivate reducing the frequency and the exposed overhead of full spectral transformations, while preserving Muon’s optimization advantage\.

We first establish that Muon’s optimization advantage persists at scale by training DiTs from 1\.3B to 15B parameters on the GPIC dataset\([6](https://arxiv.org/html/2608.20818#bib.bib24)\)\. Across all four scales, Muon achieves lower validation loss and improves the best observed FD\-DINO over AdamW by 12\.9–19\.1%\. However, its additional computation and communication make vanilla Muon substantially more expensive per step\. These results demonstrate Muon’s algorithmic scalability on large DiTs while exposing a systems bottleneck to realizing its optimization advantage efficiently\.

To address this bottleneck, we revisit whether global spectral geometry must be imposed at every optimization step and introduce*Periodic Row\-wise Muon*, which performs a full NS5 update once everyKKsteps, and applies a row\-wise constrained normalization operator \(RowNorm\)\([20](https://arxiv.org/html/2608.20818#bib.bib1)\)at the remaining steps\. The method alternates between two complementary matrix geometries\. Periodic spectral \(refresh\) steps provide global coupling across rows and singular directions, whereas low compute and communication cost row\-wise constrained \(non\-refresh\) steps provide local scale control and maintain stable optimization behavior between consecutive spectral refreshes\.

We co\-design the distributed execution with the periodic update\. On non\-refresh steps, RowNorm operates directly on sharded momentum and for matrix shapes whose normalization rows span ranks, only their norm statistics are all\-reduced instead of the full matrix\. On refresh steps, we pipeline bucketed momentum all\-gathers for upcoming matrices with NS computation on already available buckets to reduce exposed communication latency\. This design removes full\-momentum materialization on non\-refresh steps and reduces the exposed overhead of the remaining spectral refreshes\.

Across the 1\.3B–4B models, Periodic Row\-wise Muon’s best observed FD\-DINO remains within 0\.5% of vanilla Muon, while outperforming it by approximately 4\.5% at 9B and 2\.7% at 15B\. Relative to vanilla Muon, it reduces optimizer time by 46\.9–54\.3%, end\-to\-end step time by 15\.7–24\.3%, and logical optimizer\-communication volume by 66\.7% across all scales\.

Our main contributions are as follows:\(1\)we characterize Muon on 1\.3B–15B DiTs, showing that its advantages over AdamW in validation loss and generative quality persist across scale while identifying NS5 computation and full\-momentum communication as its main systems bottlenecks;\(2\)we introduce Periodic Row\-wise Muon, which replaces most NS5 transformations with low compute and communication cost RowNorm updates from the current momentum; and\(3\)we develop a distributed implementation that operates on sharded momentum during non\-refresh steps and pipelines bucketed all\-gather with NS5 computation during refresh steps, substantially reducing systems overhead while retaining generative quality comparable to vanilla Muon\.

## 2Muon for DiTs

### 2\.1Scaling Muon for DiTs Training

Given a clean data samplexx, a diffusion timestepaa, and noiseϵ\\epsilon, letxax\_\{a\}denote the noisy input constructed according to the prescribed noise schedule\. We train a DiT to predict the corresponding velocity targetv⋆v^\{\\star\}by minimizing

ℒ⁡\(θ\)=𝔼x,a,ϵ​\[‖vθ​\(xa,a\)−v⋆​\(xa,a\)‖22\]\.\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{x,a,\\epsilon\}\\left\[\\left\\lVert v\_\{\\theta\}\(x\_\{a\},a\)\-v^\{\\star\}\(x\_\{a\},a\)\\right\\rVert\_\{2\}^\{2\}\\right\]\.\(1\)We omit additional conditioning variables, such as text embeddings, for notational simplicity\. The AdamW baseline applies AdamW to all trainable parameters\. Muon instead applies matrix\-aware updates to two\-dimensional hidden weight matrices, while biases, normalization parameters, and other non\-matrix parameters continue to be updated by AdamW\.

Consider a weight matrix at training stepttdenotesWt∈ℝm×nW\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}with gradientGt=∇Wtℒ​\(θt\)G\_\{t\}=\\nabla\_\{W\_\{t\}\}\\mathcal\{L\}\(\\theta\_\{t\}\)\. Muon applies a finite spectral transformation consisting ofJNS=5J\_\{\\mathrm\{NS\}\}=5Newton–Schulz iterationsΦ5\\Phi\_\{5\}to its momentumMtM\_\{t\}, and decoupled weight decay update:

𝒫t=Φ5​\(Mt\),Wt\+1=\(1−ηt​λ\)​Wt−ηt​s​\(Wt\)​𝒫t,\\mathcal\{P\}\_\{t\}=\\Phi\_\{5\}\(M\_\{t\}\),\\qquad W\_\{t\+1\}=\(1\-\\eta\_\{t\}\\lambda\)W\_\{t\}\-\\eta\_\{t\}\\,s\(W\_\{t\}\)\\mathcal\{P\}\_\{t\},\(2\)whereηt\\eta\_\{t\}is learning rate,λ\\lambdais weight decay coefficient, ands⁡\(Wt\)s\(W\_\{t\}\)is shape\-dependent update scale\.

The finite mapΦ5\\Phi\_\{5\}is inspired by the matrix polar factor and reshapes the singular values of the momentum to produce a globally coupled spectral direction𝒫t\\mathcal\{P\}\_\{t\}\. Importantly, NS5 is not equivalent to computing an exact SVD polar decomposition, nor does it generally yield an exactly orthogonal matrix after55iterations\. Our algorithmic definitions, theoretical analysis, and experiments therefore use the implemented finite mapΦ5\\Phi\_\{5\}, rather than assuming convergence to the exact polar factor\.

We characterize Muon’s scalability for large scale DiTs training along two dimensions: algorithmic and systems scalability\. Algorithmic scalability answers whether Muon maintains stable optimization behavior as model size increases and whether its advantages over AdamW in validation loss and generative quality persist across model scales\. Systems scalability answers whether these algorithmic gains translate into realized efficiency under large\-scale distributed training\.

### 2\.2Computation Complexity

The most direct computational difference between Muon and AdamW arises from updating two\-dimensional matrices\. ConsiderW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\},r=min⁡\(m,n\)r=\\min\(m,n\),c=max⁡\(m,n\)c=\\max\(m,n\)\. Because AdamW maintains and applies element\-wise statistics, its arithmetic complexity for this matrix isCAdamW​\(m,n\)=Θ⁡\(m​n\)=Θ⁡\(r​c\)C\_\{\\mathrm\{AdamW\}\}\(m,n\)=\\Theta\(mn\)=\\Theta\(rc\)\. In contrast, a finite step Newton–Schulz transformation repeatedly performs matrix multiplications\. Even choosing the smaller Gram\-matrix orientation yieldsCNS​\(m,n\)=Θ⁡\(JNS​r2​c\)C\_\{\\mathrm\{NS\}\}\(m,n\)=\\Theta\\\!\\left\(J\_\{\\mathrm\{NS\}\}r^\{2\}c\\right\)\. Thus, NS5 incurs an additional factor ofΘ⁡\(JNS​r\)\\Theta\(J\_\{\\mathrm\{NS\}\}r\)in arithmetic per matrix\. Ignoring implementation constants and hardware throughput, their arithmetic cost ratio for one matrix isΘ⁡\(JNS​r\)\\Theta\(J\_\{\\mathrm\{NS\}\}r\)\.

### 2\.3Distributed Cost under Large Scale Distributed Training

Local arithmetic alone does not characterize Muon’s cost at scale, which also depends on how its full\-matrix spectral transformation is mapped onto sharded optimizer states\. In our execution path, each two\-dimensional parameter and its momentum are sharded along dimension zero, so each rank stores a subset of rows and the corresponding momentum shard\. AdamW is coordinate\-separable because each rank can update its local parameters using only its local first\- and second\-moment shards, without materializing the full matrix\. Muon is not shard\-separable under this execution path because Newton–Schulz iterations couple rows and singular directions through Gram matrices and matrix multiplications\. In general, a rank cannot recover its portion of the spectral update from its local momentum shard alone\. Our vanilla implementation therefore all\-gathers the momentum shards to every rank, executes NS5 on the full matrix at every rank, retains the output slice corresponding to the local parameter shard, and then discards the remaining output and releases the temporary full\-matrix buffers\([23](https://arxiv.org/html/2608.20818#bib.bib5)\)\.

Muon’s systems cost consequently includes the GEMMs and associated matrix operations of replicated NS5 execution, together with full\-momentum communication and temporary full\-matrix materialization\. Suppose the sharding group containsppranks and each momentum element occupiesbbbytes\. Ignoring padding and protocol overhead, all\-gathering anm×nm\\times nmomentum matrix requires each rank to logically receive approximately

VAG​\(m,n\)≈b​m​n​p−1p​bytes\.V\_\{\\mathrm\{AG\}\}\(m,n\)\\approx bmn\\frac\{p\-1\}\{p\}\\ \\text\{bytes\}\.\(3\)This quantity excludes parameter communication already required by forward and backward propagation\. Logical communication volume alone, however, does not determine communication time, which also depends on the collective count, bucket sizes, launch latency, the fraction of inter\-node traffic, network topology and contention, and the extent of communication–computation overlap\. In particular, many small buckets may incur substantial launch latency without changing the total communication volume\.

Together, repeated spectral computation and full\-momentum communication make Muon substantially more expensive per step and may offset its optimization advantage\. This motivates reducing the frequency of full spectral transformations while retaining effective updates\.

## 3Periodic Row\-wise Muon

We introduce*Periodic Row\-wise Muon*, which performs an NS5 spectral update periodically and applies RowNorm to the current momentum at the remaining steps\. Rather than treating RowNorm as a numerical approximation to NS5, our method alternates between two matrix updates induced by distinct constraint geometries\. We then exploit the locality of RowNorm to reduce momentum collective communication and full\-matrix materialization\.

### 3\.1Complementary Spectral and Row\-wise Geometries

For a two\-dimensional momentumMt∈ℝm×nM\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}, letM~t=𝒪⁡\(Mt\)\\widetilde\{M\}\_\{t\}=\\mathcal\{O\}\(M\_\{t\}\)denoteMtM\_\{t\}ifm≤nm\\leq nandMt⊤M\_\{t\}^\{\\top\}otherwise, so thatM~t∈ℝr×c\\widetilde\{M\}\_\{t\}\\in\\mathbb\{R\}^\{r\\times c\}, wherer=min⁡\(m,n\)≤c=max⁡\(m,n\)r=\\min\(m,n\)\\leq c=\\max\(m,n\)\. The resulting direction is mapped back through𝒪−1\\mathcal\{O\}^\{\-1\}\.

For a full\-row\-rank matrixX∈ℝr×cX\\in\\mathbb\{R\}^\{r\\times c\}, the ideal polar factor𝒫⁡\(X\)\\mathcal\{P\}\(X\)is

𝒫\(X\)=\(XX⊤\)−1/2X∈arg​max‖U‖2≤1⟨X,U⟩,⟨A,B⟩=tr\(A⊤B\)\.\\mathcal\{P\}\(X\)=\(XX^\{\\top\}\)^\{\-1/2\}X\\in\\operatorname\*\{arg\\,max\}\_\{\\left\\lVert U\\right\\rVert\_\{2\}\\leq 1\}\\left\\langle X,U\\right\\rangle,\\qquad\\left\\langle A,B\\right\\rangle=\\operatorname\{tr\}\(A^\{\\top\}B\)\.\(4\)The spectral\-norm constraint globally couples the rows and singular directions ofXX\. Practical Muon uses a finite NS5 map motivated by this spectral geometry rather than computing the exact polar factor\.

RowNorm, previously studied in symmetry\-compatible optimizer design\([20](https://arxiv.org/html/2608.20818#bib.bib1)\), instead normalizes each row independently,

ℛϵ\(X\)i:=Xi:max\(∥Xi:∥2,ϵ\),i=1,…,r\.\\mathcal\{R\}\_\{\\epsilon\}\(X\)\_\{i:\}=\\frac\{X\_\{i:\}\}\{\\max\\\!\\left\(\\lVert X\_\{i:\}\\rVert\_\{2\},\\epsilon\\right\)\},\\qquad i=1,\\ldots,r\.\(5\)For nonzero rows, RowNorm maximizes⟨X,U⟩\\left\\langle X,U\\right\\ranglesubject to‖U‖2,∞:=maxi∥Ui:∥2≤1\\left\\lVert U\\right\\rVert\_\{2,\\infty\}:=\\max\_\{i\}\\lVert U\_\{i:\}\\rVert\_\{2\}\\leq 1\. Thus, the ideal polar direction imposes a globally coupled spectral constraint, whereas RowNorm preserves each momentum row’s direction while controlling its magnitude independently\. The complete variational characterizations and their relation are provided in Appendices[A\.2](https://arxiv.org/html/2608.20818#A1.SS2)and[A\.3](https://arxiv.org/html/2608.20818#A1.SS3)\.

Periodic spectral correction is further motivated by the local stability of the ideal polar direction\. For full\-row\-rank matricesAAandBB,

‖𝒫⁡\(A\)−𝒫⁡\(B\)‖F≤2​‖A−B‖Fσmin​\(A\)\+σmin​\(B\)\.\\left\\lVert\\mathcal\{P\}\(A\)\-\\mathcal\{P\}\(B\)\\right\\rVert\_\{\\mathrm\{F\}\}\\leq\\frac\{2\\left\\lVert A\-B\\right\\rVert\_\{\\mathrm\{F\}\}\}\{\\sigma\_\{\\min\}\(A\)\+\\sigma\_\{\\min\}\(B\)\}\.\(6\)Hence, moderate momentum changes imply moderate changes in the ideal polar direction as long as the matrices remain away from rank degeneracy\. This motivates periodically reimposing global spectral structure and using a lower compute and communication cost structured direction between refreshes\. The proof and limitations of this exact\-polar argument are discussed in Appendix[A\.4](https://arxiv.org/html/2608.20818#A1.SS4)\.

### 3\.2Periodic Row\-wise Muon

Building on the preceding analysis, we propose*Periodic Row\-wise Muon*\. Given a periodK≥1K\\geq 1and a RowNorm multiplierγ\>0\\gamma\>0, define the refresh indicatorρt=𝕀\[tmodK=0\]\\rho\_\{t\}=\\mathbb\{I\}\\\!\\left\[t\\bmod K=0\\right\]\. At every step, we use the current momentumMtM\_\{t\}to compute the parameter update

D~t=\{NS5⁡\(M~t\),ρt=1,γ​ℛϵ​\(M~t\),ρt=0,Wt\+1=\(1−ηt​λ\)​Wt−ηt​s​\(Wt\)​𝒪−1​\(D~t\)\.\\widetilde\{D\}\_\{t\}=\\begin\{cases\}\\operatorname\{NS5\}\(\\widetilde\{M\}\_\{t\}\),&\\rho\_\{t\}=1,\\\\\[3\.0pt\] \\gamma\\,\\mathcal\{R\}\_\{\\epsilon\}\(\\widetilde\{M\}\_\{t\}\),&\\rho\_\{t\}=0,\\end\{cases\}\\qquad W\_\{t\+1\}=\(1\-\\eta\_\{t\}\\lambda\)W\_\{t\}\-\\eta\_\{t\}s\(W\_\{t\}\)\\mathcal\{O\}^\{\-1\}\(\\widetilde\{D\}\_\{t\}\)\.\(7\)
The periodKKcontrols the frequency of spectral updates, whereasγ\\gammacalibrates the effective step size of the RowNorm branch relative to the spectral branch\. This calibration is necessary because the two normalization maps correspond to different feasible sets\. For a full\-row\-rank matrix with nonzero rows, the exact polar factor and the unregularized RowNorm direction both have Frobenius normr\\sqrt\{r\}\. However, equal Frobenius norms do not imply equal stable update amplitudes\. NS5 couples all rows under spectral\-norm geometry, whereas RowNorm updates rows independently underℓ2,∞\\ell\_\{2,\\infty\}geometry\. Their alignment with the gradient and the local curvature encountered along their directions can therefore differ\. Directly reusing the Muon learning rate is equivalent to settingγ=1\\gamma=1, which implicitly assumes that the two branches have the same stable step\-size range without theoretical justification\.

We write the effective learning rate of an off\-refresh step asηtRN=γ​ηt\\eta\_\{t\}^\{\\mathrm\{RN\}\}=\\gamma\\eta\_\{t\}\. Here,ηt\\eta\_\{t\}retains the global learning rate schedule of vanilla muon, whileγ\\gammaonly specifies the relative scale between the two update geometries\. Both normalization maps remove the global scale of the input momentum, ands⁡\(Wt\)s\(W\_\{t\}\)already accounts for shape\-dependent scaling\. The remaining calibration is therefore a dimensionless, branch\-specific quantity\. We use a fixedγ\\gammato preserve a constant ratio between the RowNorm and Muon learning rate schedules, rather than introducing a separate time varying schedule for RowNorm\. Complete pseudocode is provided in Appendix[C](https://arxiv.org/html/2608.20818#A3)\.

Under standard conditional\-alignment and bounded\-second\-moment assumptions, Appendix[A\.5](https://arxiv.org/html/2608.20818#A1.SS5)establishes a finite\-horizon descent bound for the periodic update\. Appendix[A\.6](https://arxiv.org/html/2608.20818#A1.SS6)further gives the blockwise dependence onγ\\gamma, including its interaction with the remaining optimizer direction\.

### 3\.3Distributed Execution

The two branches of Periodic Row\-wise Muon have different communication requirements\. A refresh step must still materialize the complete momentum matrix to execute NS5\. To reduce the exposed latency of this global path, we introduce a bucketed all\-gather pipeline with communication–computation overlap\. Specifically, we partition the Muon matrices into communication buckets\. Once the momentum of the current bucket has been all\-gathered, we immediately reconstruct its full matrices and execute NS5 while asynchronously all\-gathering the next bucket\. Communication for the next bucket is thereby overlapped with NS5 computation on the current bucket, and only a bounded number of momentum matrices need to be materialized at any time\.

On non\-refresh steps, RowNorm operates directly on the momentum shards\. Ifm≤nm\\leq n, the orientation is unchanged and every RowNorm row is local, requiring no optimizer\-specific collective\. Ifm\>nm\>n, then𝒪⁡\(M\)=M⊤\\mathcal\{O\}\(M\)=M^\{\\top\}, so each oriented row spans ranks\. If rankppowns the original rows indexed by𝒮p\\mathcal\{S\}\_\{p\}, it computes

qj\(p\)=∑i∈𝒮pMi​j2,νj=max⁡\{\(∑pqj\(p\)\)1/2,ϵ\}\.q\_\{j\}^\{\(p\)\}=\\sum\_\{i\\in\\mathcal\{S\}\_\{p\}\}M\_\{ij\}^\{2\},\\qquad\\nu\_\{j\}=\\max\\left\\\{\\left\(\\sum\_\{p\}q\_\{j\}^\{\(p\)\}\\right\)^\{1/2\},\\epsilon\\right\\\}\.\(8\)The denominators therefore require a sum all\-reduce of onlynnscalars, rather than an all\-gather of allm​nmnmomentum entries\. Appendix[B\.2](https://arxiv.org/html/2608.20818#A2.SS2)proves equivalence to dense RowNorm\.

In the implementation, we first compute the local statistics for all tall matrices, pack them, and launch bucketed asynchronous all\-reduces\. While these collectives are in flight, we compute the local statistics and RowNorm updates for the remaining matrices\. Once the all\-reduces complete, we use the global statistics to finish the tall\-matrix updates\. This ordering hides the statistics communication for tall matrices behind local computation on the remaining matrices\. Complete pseudocode is provide in Appendix[C](https://arxiv.org/html/2608.20818#A3)

For an oriented matrix inℝr×c\\mathbb\{R\}^\{r\\times c\},r≤cr\\leq c, the dominant computation of NS5 isCNS5=Θ⁡\(5​r2​c\)C\_\{\\mathrm\{NS5\}\}=\\Theta\(5r^\{2\}c\), whereas RowNorm requires onlyCRN=Θ⁡\(r​c\)C\_\{\\mathrm\{RN\}\}=\\Theta\(rc\)\. LetVAGV\_\{\\mathrm\{AG\}\}andVRNV\_\{\\mathrm\{RN\}\}denote the total per\-rank logical optimizer payloads on refresh and non\-refresh steps\. The average matrix\-processing computation over one periodC¯​\(K\)\\overline\{C\}\(K\)and period\-averaged payloadV⁡\(K\)V\(K\)satisfy

C¯​\(K\)=1K​CNS5\+K−1K​CRN,V⁡\(K\)VAG=1K\+K−1K​VRNVAG\.\\overline\{C\}\(K\)=\\frac\{1\}\{K\}C\_\{\\mathrm\{NS5\}\}\+\\frac\{K\-1\}\{K\}C\_\{\\mathrm\{RN\}\},\\qquad\\frac\{V\(K\)\}\{V\_\{\\mathrm\{AG\}\}\}=\\frac\{1\}\{K\}\+\\frac\{K\-1\}\{K\}\\frac\{V\_\{\\mathrm\{RN\}\}\}\{V\_\{\\mathrm\{AG\}\}\}\.\(9\)For complete\-row matricesVRN=0V\_\{\\mathrm\{RN\}\}=0, while for a tall matrixVRN/VAG=O⁡\(1/m\)V\_\{\\mathrm\{RN\}\}/V\_\{\\mathrm\{AG\}\}=O\(1/m\)under the logical payload model\. For example, the total optimizer\-specific payload forK=3K=3approaches1/31/3of vanilla Muon when the norm\-statistics payload is small\. Detailed arithmetic, correctness, communication, and overlap models are given in Appendix[B](https://arxiv.org/html/2608.20818#A2)\.

## 4Experiments

We systematically compare AdamW, vanilla Muon, and Periodic Row\-wise Muon across DiT models ranging from approximately 1\.3B to 15B parameters\. Beyond the main comparison, we include all experimental results in Appendix[E](https://arxiv.org/html/2608.20818#A5)\.

### 4\.1Experimental Setup

Dataset and models\.We train text\-to\-image Diffusion Transformers from scratch on GPIC\-Full\([6](https://arxiv.org/html/2608.20818#bib.bib24)\)\. GPIC contains 100M training image–text pairs collected from Flickr and Wikimedia, captioned with Qwen3\-VL\-4B\-Instruct\([4](https://arxiv.org/html/2608.20818#bib.bib25)\)\. We train four MMDiT\([26](https://arxiv.org/html/2608.20818#bib.bib21)\)configurations at512×512512\\times 512resolution, containing approximately 1\.3B, 4B, 9B, and 15B parameters\. All models share the same overall architecture and conditioning modules, and differ only in hidden dimension, depth, and number of attention heads\.

Optimization and training\.Unless otherwise specified, Periodic Row\-wise Muon usesK=3K=3andγ=0\.15\\gamma=0\.15throughout and their selection and analysis are provided in Appendix[E\.1](https://arxiv.org/html/2608.20818#A5.SS1)\. At each model scale, all three optimizers are trained for 60,000 steps with the same global batch size of 4,096, and therefore process the same number of training examples\. All experiments use 32 nodes with 256 NVIDIA H100 GPUs in total and FSDP2 in Pytorch\. We normalize all time quantities by the mean step time of the 1\.3B AdamW run, which is defined as one unit\.

Evaluation\.We select a fixed set of 50,000 prompts from the GPIC test set and generate one512×512512\\times 512image per prompt\. All methods use the same sampling configuration with a fixed classifier\-free guidance scale of 5\.0\. We report FD\-DINOv2\([32](https://arxiv.org/html/2608.20818#bib.bib26)\), FID\([14](https://arxiv.org/html/2608.20818#bib.bib27)\), Maximum Mean Discrepancy \(MMD\), Precision, Recall, Density, Coverage\([18](https://arxiv.org/html/2608.20818#bib.bib28);[29](https://arxiv.org/html/2608.20818#bib.bib29);[25](https://arxiv.org/html/2608.20818#bib.bib30)\), HPSv2\.1\([35](https://arxiv.org/html/2608.20818#bib.bib31)\), and GenEval2\([16](https://arxiv.org/html/2608.20818#bib.bib32)\)\. Completed experimental settings are provided in Appendix[D](https://arxiv.org/html/2608.20818#A4)\.

\(a\)Validation loss versus training progress\.\(b\)FD\-DINO throughout training and the best observed FD\-DINO across model scales\.\(c\)Validation loss versus wall\-clock time\.
Figure 1:Scaling behavior of Muon for DiTs training\. Muon consistently improves step efficiency and generation quality, while its higher per\-step overhead can offset this advantage in wall\-clock time\.
### 4\.2Scaling Muon for Diffusion Transformers

We first examine whether Muon retains its optimization advantage as Diffusion Transformers scale from 1\.3B to 15B parameters\. Figure[1](https://arxiv.org/html/2608.20818#S4.F1)\(a\) compares the validation loss trajectories of AdamW and Muon over 60k optimization steps\. Across all four model scales, Muon consistently achieves lower validation loss than AdamW throughout the main training regime\. The achievable validation loss also decreases as model size increases, showing that Muon’s optimization advantage remains stable when scaling to substantially larger DiTs\.

This advantage also extends to generation quality\. Figure[1](https://arxiv.org/html/2608.20818#S4.F1)\(b\) shows that, across the 1\.3B–15B models, Muon improves the best observed FD\-DINO over AdamW by 12\.9–19\.1%, with the relative ordering across model scales remaining largely consistent throughout training\. Together with the validation loss results, this demonstrates that Muon’s optimization advantage persists across scale in both optimization progress and generation quality\. Complete checkpoint evaluations are provided in Appendix[E\.4](https://arxiv.org/html/2608.20818#A5.SS4)\.

The step\-wise advantage of Muon, however, does not directly translate into wall\-clock time efficiency\. Figure[1](https://arxiv.org/html/2608.20818#S4.F1)\(c\) replots the validation loss trajectories against normalized active training time\. Although Muon ultimately reaches a lower validation loss, its larger per\-step compute and communication cost shifts its trajectory to the right in wall\-clock space\. Within the highlighted training regime, AdamW reaches the same intermediate loss levels earlier than Muon across all 4 model scales\. Thus, the optimization advantage observed per training step is partially or fully offset by Muon’s optimizer overhead under a fixed wall\-clock time budget\. This empirical gap motivates reducing Muon’s per\-step overhead while preserving its generation quality advantage\.

Figure 2:Generative quality across model scales\. We report FD\-DINO improvement over AdamW at both the final checkpoint and the best checkpoint observed during training\.Figure 3:Generative quality frontier under optional checkpoint selection\.
### 4\.3Periodic Row\-wise Muon Generation Quality and Training Efficiency

We next evaluate whether Periodic Row\-wise Muon preserves vanilla Muon’s generation quality while reducing training compute and communication cost\. Figure[2](https://arxiv.org/html/2608.20818#S4.F2)shows that it improves final\-checkpoint FD\-DINO over AdamW by 11\.1–17\.8% across scales and slightly outperforms vanilla Muon at 1\.3B\. Its best\-observed FD\-DINO remains within 0\.5% of vanilla Muon at 1\.3B and 4B, while improving it by approximately 4\.5% at 9B and 2\.7% at 15B\. Table[1](https://arxiv.org/html/2608.20818#S4.T1)shows a similarly competitive broader generation\-quality profile: at 9B, Periodic Row\-wise Muon increases GenEval2 AM from 51\.77 to 57\.33 and GM from 11\.35 to 15\.97, while at 15B its primary fidelity metrics remain close to vanilla Muon and Coverage and Density improve\.

Given their comparable best generation quality, Figure[3](https://arxiv.org/html/2608.20818#S4.F3)compares the best achieved FD\-DINO against normalized active training time\. Periodic Row\-wise Muon reaches its respective best result with 33\.7%, 36\.1%, 64\.8%, and 57\.4% less active time at 1\.3B, 4B, 9B, and 15B, respectively, with the largest gains at scale\. A complementary comparison with AdamW in active\-time efficiency and final generation quality is provided in Appendix[E\.3](https://arxiv.org/html/2608.20818#A5.SS3)\.

Table 1:Final 60k\-step comparison\. Bold denotes the best result among optimizers at the same model size\.ModelOptimizerDiscrepancyFidelity & DiversityAlignmentCompositionalityFD\-DINO↓\\downarrowFID↓\\downarrowMMD\-DINO↓\\downarrowPrecision↑\\uparrowRecall↑\\uparrowCoverage↑\\uparrowDensity↑\\uparrowHPSv2↑\\uparrowGenEval2 AM↑\\uparrowGenEval2 GM↑\\uparrow1\.3BAdamW53\.9353\.936\.716\.710\.03120\.03120\.94050\.94050\.88740\.88740\.94010\.94010\.96650\.966520\.5320\.5338\.6638\.667\.107\.10Muon46\.0846\.086\.546\.540\.0268\\mathbf\{0\.0268\}0\.94870\.94870\.9071\\mathbf\{0\.9071\}0\.9492\\mathbf\{0\.9492\}0\.9918\\mathbf\{0\.9918\}20\.84\\mathbf\{20\.84\}44\.7244\.729\.389\.38Periodic Row\-wise Muon45\.65\\mathbf\{45\.65\}6\.29\\mathbf\{6\.29\}0\.02700\.02700\.9517\\mathbf\{0\.9517\}0\.90650\.90650\.94850\.94850\.97560\.975620\.3620\.3645\.89\\mathbf\{45\.89\}9\.75\\mathbf\{9\.75\}4BAdamW51\.3951\.397\.357\.350\.02710\.02710\.93710\.93710\.89950\.89950\.93880\.93880\.9898\\mathbf\{0\.9898\}21\.40\\mathbf\{21\.40\}45\.7045\.709\.139\.13Muon41\.62\\mathbf\{41\.62\}6\.08\\mathbf\{6\.08\}0\.02610\.02610\.93850\.93850\.9136\\mathbf\{0\.9136\}0\.94310\.94310\.96390\.963920\.6620\.6646\.5846\.589\.069\.06Periodic Row\-wise Muon42\.2542\.256\.356\.350\.0218\\mathbf\{0\.0218\}0\.9413\\mathbf\{0\.9413\}0\.90570\.90570\.9516\\mathbf\{0\.9516\}0\.98370\.983721\.3421\.3449\.38\\mathbf\{49\.38\}10\.58\\mathbf\{10\.58\}9BAdamW41\.2641\.266\.196\.190\.02160\.02160\.94550\.94550\.90450\.90450\.94950\.94950\.9873\\mathbf\{0\.9873\}21\.3921\.3945\.9345\.939\.669\.66Muon33\.97\\mathbf\{33\.97\}5\.61\\mathbf\{5\.61\}0\.01990\.01990\.9516\\mathbf\{0\.9516\}0\.9269\\mathbf\{0\.9269\}0\.9531\\mathbf\{0\.9531\}0\.96010\.960122\.60\\mathbf\{22\.60\}51\.7751\.7711\.3511\.35Periodic Row\-wise Muon36\.7036\.706\.136\.130\.0192\\mathbf\{0\.0192\}0\.94940\.94940\.91340\.91340\.95110\.95110\.97930\.979322\.4422\.4457\.33\\mathbf\{57\.33\}15\.97\\mathbf\{15\.97\}15BAdamW40\.2340\.236\.246\.240\.02170\.02170\.94480\.94480\.90950\.90950\.94730\.94730\.96380\.963821\.9921\.9947\.3447\.349\.189\.18Muon33\.51\\mathbf\{33\.51\}5\.57\\mathbf\{5\.57\}0\.0191\\mathbf\{0\.0191\}0\.94450\.94450\.9304\\mathbf\{0\.9304\}0\.95010\.95010\.93670\.936722\.37\\mathbf\{22\.37\}57\.93\\mathbf\{57\.93\}14\.55\\mathbf\{14\.55\}Periodic Row\-wise Muon33\.9933\.995\.635\.630\.01970\.01970\.9482\\mathbf\{0\.9482\}0\.92480\.92480\.9554\\mathbf\{0\.9554\}0\.9673\\mathbf\{0\.9673\}21\.3521\.3552\.2652\.2611\.9711\.97

### 4\.4System Efficiency

We next analyze the systems\-level source of the wall\-clock time improvement in Table[2](https://arxiv.org/html/2608.20818#S4.T2)\. Across the 4 model scales, Periodic Row\-wise Muon reduces end\-to\-end step time by 15\.7–24\.3% relative to vanilla Muon, while reducing optimizer time by 46\.9–54\.3%\. On the largest 15B model, optimizer time decreases by 54\.3%, resulting in a 23\.4% reduction in total step time\.

The communication measurements exhibit the expected reduction from periodic spectral refreshes\. WithK=3K=3, the average number of optimizer specific bucketed all\-gathers is reduced by around 66\.6%\. The corresponding logical communication volume decreases by 66\.7% at every scale\. The sharded RowNorm path introduces only approximately 0\.7 small all\-reduces per step\.

Figure[4](https://arxiv.org/html/2608.20818#S4.F4)shows the execution breakdown for the 15B model\. Periodic Row\-wise Muon replaces two of every three NS5 phases and their full\-momentum all\-gathers with short RowNorm updates, while overlapping refresh communication with NS5 computation and norm\-statistics communication with local computation\. This reduces both optimizer computation and exposed communication\.

Table 2:System\-efficiency comparison on 32 H100 nodes\.ModelMethodComputationCommunicationStep time↓\\downarrowOpt\. time↓\\downarrowOpt\. share↓\\downarrowAG/step↓\\downarrowAR/step↓\\downarrowComm\. volume↓\\downarrowGiB/rank/step1\.3BMuon1\.8730\.83544\.6%10\.00\.02\.38Periodic Row\-wise Muon1\.419\(−24\.3%\)\(\-24\.3\\%\)0\.443\(−46\.9%\)\(\-46\.9\\%\)31\.3%3\.30\.70\.79\(−66\.7%\)\(\-66\.7\\%\)4BMuon2\.3521\.00942\.9%32\.00\.07\.58Periodic Row\-wise Muon1\.878\(−20\.2%\)\(\-20\.2\\%\)0\.536\(−46\.9%\)\(\-46\.9\\%\)28\.6%10\.70\.72\.53\(−66\.7%\)\(\-66\.7\\%\)9BMuon4\.1531\.68940\.7%80\.00\.017\.30Periodic Row\-wise Muon3\.500\(−15\.7%\)\(\-15\.7\\%\)0\.868\(−48\.6%\)\(\-48\.6\\%\)24\.8%26\.70\.75\.77\(−66\.7%\)\(\-66\.7\\%\)15BMuon6\.5272\.88444\.2%119\.00\.028\.62Periodic Row\-wise Muon5\.002\(−23\.4%\)\(\-23\.4\\%\)1\.317\(−54\.3%\)\(\-54\.3\\%\)26\.3%39\.70\.79\.54\(−66\.7%\)\(\-66\.7\\%\)

![Refer to caption](https://arxiv.org/html/2608.20818v1/profiler_trace_15B_muon_periodic_xlarge_font_wrapped_overlap_no_model_label.png)Figure 4:Simplified profiler traces for the 15B model\.
### 4\.5Ablation Studies

Algorithmic ablation\.Table[3](https://arxiv.org/html/2608.20818#S4.T3)\(a\) isolates the roles of periodic spectral refresh, RowNorm geometry, and branch\-specific scale calibration\. We compare vanilla Muon, RowNorm at every step, Periodic Row\-wise Muon withγ=1\\gamma=1, a scalar\-controlled baseline that appliesγ\\gammato both NS5 and RowNorm, and the complete Periodic Row\-wise Muon withK=3K=3andγ=0\.15\\gamma=0\.15on 1\.3B scale\. Overall, the complete method provides the best generation quality\.

Table 3:Ablation studies\. \(a\) Algorithm\-level ablation of Periodic Row\-wise Muon on the 1\.3B model\. \(b\) System\-level ablation of the distributed implementation on the 15B model\.\(a\)VariantNS5 ScheduleOff\-refresh Updateγ\\gammaBest FD\-DINO↓\\downarrowVanilla MuonEvery step––41\.733RowNorm every stepNoneRowNorm0\.1551\.322Periodic Row\-wise Muon \(γ=1\\gamma=1\)K=3K=3RowNorm1\.044\.294Scalar\-controlled \(γ\\gammafor both NS5 and RowNorm\)K=3K=3Scalar\-controlled0\.1544\.713Periodic Row\-wise Muon \(γ=0\.15\\gamma=0\.15\)K=3K=3RowNorm0\.1541\.913

\(b\)VariantShardedBucketedComm\.–Comp\.Step Time↓\\downarrowOpt\. Time↓\\downarrowComm\. Volume↓\\downarrowRowNormAll\-gatherOverlapNaive Periodic\(K=3,γ=0\.15\)\(K=3,\\gamma=0\.15\)×\\times×\\times×\\times5\.6852\.04628\.62\+ Sharded RowNorm✓\\checkmark×\\times×\\times5\.1961\.4989\.54\\mathbf\{9\.54\}\+ Bucketed All\-gather✓\\checkmark✓\\checkmark×\\times5\.0281\.3499\.54\\mathbf\{9\.54\}Full System✓\\checkmark✓\\checkmark✓\\checkmark5\.002\\mathbf\{5\.002\}1\.317\\mathbf\{1\.317\}9\.54\\mathbf\{9\.54\}

System ablation\.Table[3](https://arxiv.org/html/2608.20818#S4.T3)\(b\) incrementally adds the system optimizations to a 15B naive periodic baseline, which uses the same update rule but materializes full momentum every step without bucketing or overlap\. Sharded RowNorm reduces communication volume by66\.7%66\.7\\%, while bucketed all\-gather and communication–computation overlap further reduce exposed refresh latency\. Together, these optimizations reduce optimizer and end\-to\-end step time by35\.6%35\.6\\%and12\.0%12\.0\\%, respectively\.

## 5Related Work

Unlike AdamW’s element\-wise adaptive updates\([24](https://arxiv.org/html/2608.20818#bib.bib23)\), matrix\-aware optimizers exploit the matrix structure of model parameters\. Shampoo and SOAP construct structured preconditioners\([13](https://arxiv.org/html/2608.20818#bib.bib2);[34](https://arxiv.org/html/2608.20818#bib.bib3)\), whereas Muon applies a finite Newton–Schulz transformation to momentum matrices, producing updates with global spectral structure\([15](https://arxiv.org/html/2608.20818#bib.bib4)\)\. Muon has subsequently been shown to scale to large language models and improve the compute–quality trade\-off over AdamW\([23](https://arxiv.org/html/2608.20818#bib.bib5);[31](https://arxiv.org/html/2608.20818#bib.bib6)\)\. In parallel, prior work has established predictable DiT scaling across model size, data, compute, and training hyperparameters\([11](https://arxiv.org/html/2608.20818#bib.bib16);[22](https://arxiv.org/html/2608.20818#bib.bib17);[36](https://arxiv.org/html/2608.20818#bib.bib18)\), while recent studies evaluate matrix\-aware optimization in diffusion training and adapt momentum orthogonalization to diffusion\-specific parameter structures\([30](https://arxiv.org/html/2608.20818#bib.bib19);[7](https://arxiv.org/html/2608.20818#bib.bib20)\)\. Together, these directions motivate evaluating optimizer scalability through optimization progress, algorithmic compute, generative quality, and realized distributed efficiency\.

Recent work reduces Muon’s spectral transformation cost, which is particularly important for large DiTs\. One direction accelerates orthogonalization through GPU\-friendly polynomial iterations, Gram\-matrix formulations, or optimized Newton–Schulz polynomials\([3](https://arxiv.org/html/2608.20818#bib.bib7);[37](https://arxiv.org/html/2608.20818#bib.bib8);[12](https://arxiv.org/html/2608.20818#bib.bib9)\); another reduces its frequency, scope, or input size through block\-periodic updates, temporal reuse, alternating spectral and sign\-based updates, tiled transformations, or row/column subsampling\([17](https://arxiv.org/html/2608.20818#bib.bib10);[10](https://arxiv.org/html/2608.20818#bib.bib11);[5](https://arxiv.org/html/2608.20818#bib.bib12);[33](https://arxiv.org/html/2608.20818#bib.bib13);[1](https://arxiv.org/html/2608.20818#bib.bib36)\)\. Other variants post\-process every step NS orthogonalization with row\- or neuron\-wise normalization and optional neuron\-wise second\-moment statistics\([21](https://arxiv.org/html/2608.20818#bib.bib37);[38](https://arxiv.org/html/2608.20818#bib.bib38)\)\. Distributed orthonormalized optimizers and specialized Muon runtimes further reduce communication, redundant computation, and exposed optimizer latency under model sharding\([2](https://arxiv.org/html/2608.20818#bib.bib14);[9](https://arxiv.org/html/2608.20818#bib.bib15)\)\. Our approach is complementary to faster spectral kernels, it reduces spectral refresh frequency and directly applies RowNorm to the current momentum as a low compute and communication cost dense substitute between refreshes\. RowNorm also admits recently studied symmetry properties\([20](https://arxiv.org/html/2608.20818#bib.bib1)\), and its locality allows us to avoid full\-momentum materialization on non\-refresh steps in large\-scale distributed training\.

## 6Conclusion and Limitations

We show that Muon’s optimization and generative quality advantages over AdamW persist as DiTs scale from 1\.3B to 15B parameters, but its every step NS5 computation and full momentum communication introduce substantial distributed overhead\. Periodic Row\-wise Muon addresses this bottleneck by combining periodic spectral refreshes with low compute and communication cost RowNorm updates and a sharded distributed implementation\. Across all scales, it preserves generation quality broadly comparable to vanilla Muon while reducing optimizer time by 46\.9–54\.3%\. However, our evaluation is limited to one DiT family, dataset, resolution, and 32\-node H100 configuration, and does not cover other training regimes\. We also use fixed globalKKandγ\\gamma, leaving layer\-wise and adaptive schedules unexplored\. Finally, refresh steps still retain full momentum communication and materialization, while the realized speedup may vary with hardware topology and parallelism strategy\. We will leave these limitations as future works\.

## References

- Ahnet al\.\(2025\)K\. Ahn, N\. Amsel, and J\. LangfordDion2: a simple method to shrink matrix in muon\.arXiv preprint arXiv:2512\.16928\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Ahn and Xu \(2025\)K\. Ahn and B\. XuDion: a communication\-efficient optimizer for large models\.arXiv e\-prints,pp\. arXiv–2504\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Amselet al\.\(2026\)N\. Amsel, D\. Persson, C\. Musco, and R\. M\. GowerThe polar express: optimal matrix sign methods and their application to the muon algorithm\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 138323–138360\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§D\.1](https://arxiv.org/html/2608.20818#A4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p1.1)\.
- Bolatovet al\.\(2026\)A\. Bolatov, A\. Riabinin, N\. Kornilov, A\. Veprikov, S\. Horváth, M\. Takáč, and A\. BeznosikovLionMuon: alternating spectral and sign descent for efficient training\.arXiv preprint arXiv:2605\.19811\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Chandrasegaranet al\.\(2026\)K\. Chandrasegaran, K\. Sargent, S\. Agarwal, M\. Jang, M\. Poli, J\. C\. Niebles, J\. Johnson, J\. Wu, and L\. Fei\-FeiGPIC: a giant permissive image corpus for visual generation\.arXiv preprint arXiv:2605\.30341\.Cited by:[§D\.1](https://arxiv.org/html/2608.20818#A4.SS1.p1.1),[§1](https://arxiv.org/html/2608.20818#S1.p4.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p1.1)\.
- Chenet al\.\(2026a\)C\. Chen, P\. Sun, and K\. YuanCMuon: accelerating and stabilizing diffusion transformer training via chunked momentum orthogonalization\.arXiv preprint arXiv:2608\.02502\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Chenet al\.\(2024\)J\. Chen, J\. Yu, C\. Ge, L\. Yao, E\. Xie, Z\. Wang, J\. Kwok, P\. Luo, H\. Lu, and Z\. LiPixart\-α\\alpha: fast training of diffusion transformer for photorealistic text\-to\-image synthesis\.InInternational conference on learning representations,Vol\.2024,pp\. 57611–57640\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p1.1)\.
- Chenet al\.\(2026b\)V\. Chen, S\. Liu, R\. Cheng, D\. Yang, S\. Li, R\. Yu, L\. Liang, H\. Su, R\. Gan, H\. Wang,et al\.DMuon: efficient distributed muon training with near\-adam overhead\.arXiv preprint arXiv:2606\.27153\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Devet al\.\(2026\)B\. Dev, S\. Bohara, M\. Takáč, and S\. HorváthCacheMuon: using temporal preconditioning to approximate polar factor\.arXiv preprint arXiv:2606\.16371\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Esseret al\.\(2024\)P\. Esser, S\. Kulal, A\. Blattmann, R\. Entezari, J\. Müller, H\. Saini, Y\. Levi, D\. Lorenz, A\. Sauer, F\. Boesel,et al\.Scaling rectified flow transformers for high\-resolution image synthesis\.InForty\-first international conference on machine learning,Cited by:[§D\.2](https://arxiv.org/html/2608.20818#A4.SS2.p1.1),[§1](https://arxiv.org/html/2608.20818#S1.p1.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Grishinaet al\.\(2025\)E\. Grishina, M\. Smirnov, and M\. RakhubaAccelerating newton\-schulz iteration for orthogonalization via chebyshev\-type polynomials\.arXiv preprint arXiv:2506\.10935\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Guptaet al\.\(2018\)V\. Gupta, T\. Koren, and Y\. SingerShampoo: preconditioned stochastic tensor optimization\.InInternational Conference on Machine Learning,pp\. 1842–1850\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Heuselet al\.\(2017\)M\. Heusel, H\. Ramsauer, T\. Unterthiner, B\. Nessler, and S\. HochreiterGANs trained by a two time\-scale update rule converge to a local nash equilibrium\.Advances in neural information processing systems30\.Cited by:[2nd item](https://arxiv.org/html/2608.20818#A4.I1.i2.p1.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks, 2024\.URL https://kellerjordan\. github\. io/posts/muon6\(3\),pp\. 4\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p2.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Kamathet al\.\(2025\)A\. Kamath, K\. Chang, R\. Krishna, L\. Zettlemoyer, Y\. Hu, and M\. GhazvininejadGeneval 2: addressing benchmark drift in text\-to\-image evaluation\.arXiv preprint arXiv:2512\.16853\.Cited by:[7th item](https://arxiv.org/html/2608.20818#A4.I1.i7.p1.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Khaledet al\.\(2026\)A\. Khaled, K\. Ozkara, T\. Yu, M\. Hong, and Y\. ParkMuonBP: faster muon via block\-periodic orthogonalization\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 17389–17416\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Kynkäänniemiet al\.\(2019\)T\. Kynkäänniemi, T\. Karras, S\. Laine, J\. Lehtinen, and T\. AilaImproved precision and recall metric for assessing generative models\.Advances in neural information processing systems32\.Cited by:[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Labset al\.\(2025\)B\. F\. Labs, S\. Batifol, A\. Blattmann, F\. Boesel, S\. Consul, C\. Diagne, T\. Dockhorn, J\. English, Z\. English, P\. Esser,et al\.Flux\. 1 kontext: flow matching for in\-context image generation and editing in latent space\.arXiv preprint arXiv:2506\.15742\.Cited by:[§D\.2](https://arxiv.org/html/2608.20818#A4.SS2.p1.1)\.
- Lau and Su \(2026\)T\. T\. Lau and W\. SuSymmetry\-compatible principle for optimizer design: embeddings, lm heads, swiglu mlps, and moe routers\.arXiv preprint arXiv:2605\.18106\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p5.1),[§3\.1](https://arxiv.org/html/2608.20818#S3.SS1.p3.1),[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Liet al\.\(2025\)Z\. Li, L\. Liu, C\. Liang, W\. Chen, and T\. ZhaoNorMuon: making muon more efficient and scalable\.arXiv preprint arXiv:2510\.05491\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Lianget al\.\(2026\)Z\. Liang, H\. He, C\. Yang, and B\. DaiScaling laws for diffusion transformers\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 84488–84521\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p1.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p2.1),[§1](https://arxiv.org/html/2608.20818#S1.p3.1),[§2\.3](https://arxiv.org/html/2608.20818#S2.SS3.p1.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p2.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Naeemet al\.\(2020\)M\. F\. Naeem, S\. J\. Oh, Y\. Uh, Y\. Choi, and J\. YooReliable fidelity and diversity metrics for generative models\.InInternational conference on machine learning,pp\. 7176–7185\.Cited by:[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Peebles and Xie \(2023\)W\. Peebles and S\. XieScalable diffusion models with transformers\.In2023 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 4172–4182\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§D\.2](https://arxiv.org/html/2608.20818#A4.SS2.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§D\.2](https://arxiv.org/html/2608.20818#A4.SS2.p1.1)\.
- Sajjadiet al\.\(2018\)M\. S\. Sajjadi, O\. Bachem, M\. Lucic, O\. Bousquet, and S\. GellyAssessing generative models via precision and recall\.Advances in neural information processing systems31\.Cited by:[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Schaipp \(2025\)F\. SchaippOptimization benchmark for diffusion models on dynamical systems\.arXiv preprint arXiv:2510\.19376\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p2.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Shahet al\.\(2025\)I\. Shah, A\. M\. Polloreno, K\. Stratos, P\. Monk, A\. Chaluvaraju, A\. Hojel, A\. Ma, A\. Thomas, A\. Tanwer, D\. J\. Shah,et al\.Practical efficiency of muon for pretraining\.arXiv preprint arXiv:2505\.02222\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Steinet al\.\(2023\)G\. Stein, J\. Cresswell, R\. Hosseinzadeh, Y\. Sui, B\. Ross, V\. Villecroze, Z\. Liu, A\. L\. Caterini, E\. Taylor, and G\. Loaiza\-GanemExposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models\.Advances in Neural Information Processing Systems36,pp\. 3732–3784\.Cited by:[1st item](https://arxiv.org/html/2608.20818#A4.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Tanget al\.\(2026\)Z\. Tang, T\. Xu, Y\. Saad, and Y\. XiHierarchical Muon: tiled newton\-schulz updates for efficient muon optimization\.arXiv preprint arXiv:2606\.27216\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Vyaset al\.\(2025\)N\. Vyas, D\. Morwani, R\. Zhao, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. KakadeSOAP: improving and stabilizing shampoo using adam for language modeling\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 93423–93444\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Wuet al\.\(2023\)X\. Wu, Y\. Hao, K\. Sun, Y\. Chen, F\. Zhu, R\. Zhao, and H\. LiHuman preference score v2: a solid benchmark for evaluating human preferences of text\-to\-image synthesis\.arXiv preprint arXiv:2306\.09341\.Cited by:[6th item](https://arxiv.org/html/2608.20818#A4.I1.i6.p1.1),[§4\.1](https://arxiv.org/html/2608.20818#S4.SS1.p3.1)\.
- Yinet al\.\(2025\)Y\. Yin, Y\. Zhao, M\. Zheng, K\. Lin, J\. Ou, R\. Chen, V\. S\. Huang, J\. Wang, X\. Tao, P\. Wan,et al\.Towards precise scaling laws for video diffusion transformers\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 18155–18165\.Cited by:[§1](https://arxiv.org/html/2608.20818#S1.p1.1),[§5](https://arxiv.org/html/2608.20818#S5.p1.1)\.
- Zhanget al\.\(2026a\)J\. Zhang, N\. Amsel, B\. Chen, and T\. DaoGram newton\-schulz\.External Links:[Link](https://dao-ailab.github.io/blog/2026/gram-newton-schulz/)Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.
- Zhanget al\.\(2026b\)R\. Zhang, Y\. Zhao, Z\. Liu, Z\. Wang, Y\. Su, L\. Tan, and Z\. ZhangMUON\+: towards more effective muon via one additional normalization step for llm pre\-training\.arXiv preprint arXiv:2602\.21545\.Cited by:[§5](https://arxiv.org/html/2608.20818#S5.p2.1)\.

## Appendix

## Appendix AMathematical Analysis

This appendix gives the complete statements and proofs supporting the geometric interpretation and the optimization\-feasibility discussion in the main text\. The results deliberately distinguish the exact polar factor from the finite Newton–Schulz map used in the implementation\. They also distinguish a conditional descent guarantee from an unconditional convergence theorem for the proposed optimizer\.

### A\.1Notation and update convention

For matrices of the same shape, let

⟨A,B⟩≔tr⁡\(A⊤​B\),‖A‖F2≔⟨A,A⟩,\\langle A,B\\rangle\\coloneqq\\operatorname\{tr\}\(A^\{\\top\}B\),\\quad\\\|A\\\|\_\{\\mathrm\{F\}\}^\{2\}\\coloneqq\\langle A,A\\rangle,\(10\)and let‖A‖2\\\|A\\\|\_\{2\}denote the spectral norm\. ForX∈ℝr×cX\\in\\mathbb\{R\}^\{r\\times c\}, define the row\-wise maximum norm

∥X∥2,∞:=max1≤i≤r∥Xi:∥2\.\\\|X\\\|\_\{2,\\infty\}:=\\max\_\{1\\leq i\\leq r\}\\\|X\_\{i:\}\\\|\_\{2\}\.\(11\)The practical RowNorm map is

ℛε\(X\)i::=Xi:max\{∥Xi:∥2,ε\},ε\>0\.\\operatorname\{\\mathcal\{R\}\}\_\{\\varepsilon\}\(X\)\_\{i:\}:=\\frac\{X\_\{i:\}\}\{\\max\\\{\\\|X\_\{i:\}\\\|\_\{2\},\\varepsilon\\\}\},\\qquad\\varepsilon\>0\.\(12\)When every row is nonzero,ℛ⁡\(X\)\\operatorname\{\\mathcal\{R\}\}\(X\)denotes the unregularized map obtained by setting the denominator to∥Xi:∥2\\\|X\_\{i:\}\\\|\_\{2\}\.

For the optimization result,θt\\theta\_\{t\}denotes the vector containing all trainable parameters, and we write the implemented update as

θt\+1=θt−ηt​Ht\.\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\_\{t\}H\_\{t\}\.\(13\)HereHtH\_\{t\}is the complete effective direction\. It includes the selected NS5 or RowNorm matrix blocks, Muon’s shape factors, the RowNorm multiplier, all AdamW\-updated blocks, any parameter\-group learning rate ratios relative toηt\\eta\_\{t\}, and decoupled weight decay\. This convention is important because the RowNorm multiplier scales only the RowNorm blocks, not the complete optimizer direction\.

### A\.2RowNorm as a row\-wise variational direction

###### Lemma 1\(Row\-wise variational characterization\)\.

LetX∈ℝr×cX\\in\\mathbb\{R\}^\{r\\times c\}, and suppose that every row ofXXis nonzero\. Define

∥U∥2,∞:=max1≤i≤r∥Ui,:∥2\.\\\|U\\\|\_\{2,\\infty\}:=\\max\_\{1\\leq i\\leq r\}\\\|U\_\{i,:\}\\\|\_\{2\}\.Then

R⁡\(X\)=arg​maxU∈ℝr×c⁡\{⟨X,U⟩:‖U‖2,∞≤1\},R\(X\)=\\operatorname\*\{arg\\,max\}\_\{U\\in\\mathbb\{R\}^\{r\\times c\}\}\\left\\\{\\langle X,U\\rangle:\\\|U\\\|\_\{2,\\infty\}\\leq 1\\right\\\},\(14\)where the maximizer is unique\.

###### Proof\.

The constraint‖U‖2,∞≤1\\\|U\\\|\_\{2,\\infty\}\\leq 1is equivalent to∥Ui:∥2≤1\\\|U\_\{i:\}\\\|\_\{2\}\\leq 1for every row\. Hence, by Cauchy–Schwarz,

⟨X,U⟩=∑i=1r⟨Xi:,Ui:⟩≤∑i=1r∥Xi:∥2∥Ui:∥2≤∑i=1r∥Xi:∥2\.\\langle X,U\\rangle=\\sum\_\{i=1\}^\{r\}\\langle X\_\{i:\},U\_\{i:\}\\rangle\\leq\\sum\_\{i=1\}^\{r\}\\\|X\_\{i:\}\\\|\_\{2\}\\\|U\_\{i:\}\\\|\_\{2\}\\leq\\sum\_\{i=1\}^\{r\}\\\|X\_\{i:\}\\\|\_\{2\}\.\(15\)Equality is attained byUi:=Xi:/∥Xi:∥2U\_\{i:\}=X\_\{i:\}/\\\|X\_\{i:\}\\\|\_\{2\}for everyii, which is exactlyU=ℛ⁡\(X\)U=\\operatorname\{\\mathcal\{R\}\}\(X\)\. Because everyXi:X\_\{i:\}is nonzero, equality in the two inequalities requires this choice row by row, proving uniqueness\. ∎

Lemma[1](https://arxiv.org/html/2608.20818#Thmlemma1)characterizes RowNorm in its own geometry\. In particular, it neither treats RowNorm as an approximation to NS5 nor asserts that the two updates follow the same optimization trajectory\.

### A\.3Polar and Row\-wise Variational Geometries

We use the orientation convention from Section[3\.1](https://arxiv.org/html/2608.20818#S3.SS1): a matrix is transposed when necessary so thatX∈ℝr×cX\\in\\mathbb\{R\}^\{r\\times c\}withr≤cr\\leq c\. IfXXhas full row rank, its row\-polar factor is

𝒫\(X\):=\(XX⊤\)−1/2X,𝒫\(X\)𝒫\(X\)⊤=Ir\.\\mathcal\{P\}\(X\):=\(XX^\{\\top\}\)^\{\-1/2\}X,\\qquad\\mathcal\{P\}\(X\)\\mathcal\{P\}\(X\)^\{\\top\}=I\_\{r\}\.\(16\)
###### Proposition 1\(Spectral variational characterization\)\.

LetX∈ℝr×cX\\in\\mathbb\{R\}^\{r\\times c\}, wherer≤cr\\leq c, and suppose thatrank⁡\(X\)=r\\operatorname\{rank\}\(X\)=r\. Then

𝒫⁡\(X\)=arg⁡maxU∈ℝr×c​⟨X,U⟩subject to‖U‖2≤1\.\\mathcal\{P\}\(X\)=\\underset\{U\\in\\mathbb\{R\}^\{r\\times c\}\}\{\\arg\\max\}\\;\\langle X,U\\rangle\\quad\\text\{subject to\}\\quad\\\|U\\\|\_\{2\}\\leq 1\.\(17\)The maximizer is unique, and the optimal value is the nuclear norm‖X‖∗\\\|X\\\|\_\{\*\}\.

###### Proof\.

LetX=A​Σ​B⊤X=A\\Sigma B^\{\\top\}be a thin singular value decomposition, whereA∈ℝr×rA\\in\\mathbb\{R\}^\{r\\times r\}is orthogonal,B∈ℝc×rB\\in\\mathbb\{R\}^\{c\\times r\}has orthonormal columns, andΣ=diag⁡\(σ1,…,σr\)\\Sigma=\\operatorname\{diag\}\(\\sigma\_\{1\},\\ldots,\\sigma\_\{r\}\)withσi\>0\\sigma\_\{i\}\>0\. ExtendBBto a square orthogonal matrixV=\[B​B⟂\]∈ℝc×cV=\[B\\;B\_\{\\perp\}\]\\in\\mathbb\{R\}^\{c\\times c\}\. For any feasibleUU, setZ=A⊤​U​VZ=A^\{\\top\}UV\. Orthogonal invariance gives‖Z‖2≤1\\\|Z\\\|\_\{2\}\\leq 1, and

⟨X,U⟩=tr⁡\(Σ​A⊤​U​B\)=∑i=1rσi​Zi​i≤∑i=1rσi=‖X‖∗\.\\langle X,U\\rangle=\\operatorname\{tr\}\(\\Sigma A^\{\\top\}UB\)=\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}Z\_\{ii\}\\leq\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}=\\\|X\\\|\_\{\*\}\.\(18\)The upper bound is attained byU=A​B⊤=𝒫⁡\(X\)U=AB^\{\\top\}=\\mathcal\{P\}\(X\)\. Because everyσi\\sigma\_\{i\}is positive, equality requiresZi​i=1Z\_\{ii\}=1for everyi≤ri\\leq r\. A contraction whose firstrrdiagonal entries are all one must haveZ=\[Ir​0\]Z=\[I\_\{r\}\\;0\]; otherwise at least one row would have Euclidean norm greater than one\. ThusU=A​B⊤U=AB^\{\\top\}is the unique maximizer\. ∎

The two feasible sets obey

\{U:‖U‖2≤1\}⊆\{U:‖U‖2,∞≤1\}⊆\{U:‖U‖2≤r\}\.\\\{U:\\\|U\\\|\_\{2\}\\leq 1\\\}\\subseteq\\\{U:\\\|U\\\|\_\{2,\\infty\}\\leq 1\\\}\\subseteq\\\{U:\\\|U\\\|\_\{2\}\\leq\\sqrt\{r\}\\\}\.\(19\)Indeed, every row norm is at most the spectral norm, while‖U‖2≤‖U‖F≤r​‖U‖2,∞\\\|U\\\|\_\{2\}\\leq\\\|U\\\|\_\{\\mathrm\{F\}\}\\leq\\sqrt\{r\}\\\|U\\\|\_\{2,\\infty\}\. Thus the polar direction solves a globally coupled spectral\-norm problem, whereas RowNorm solves a product of independent row\-wise problems\. Finite NS5 is the implemented spectral map motivated by the former geometry; the proposition does not identify finite NS5 with the exact polar factor\.

### A\.4Conditional stability of the ideal polar direction

###### Proposition 2\(Perturbation of the row\-polar factor\)\.

LetA,B∈ℝr×cA,B\\in\\mathbb\{R\}^\{r\\times c\}, wherer≤cr\\leq c, and suppose that both matrices have full row rank\. Then

‖𝒫⁡\(A\)−𝒫⁡\(B\)‖F≤2​‖A−B‖Fσmin​\(A\)\+σmin​\(B\)\.\\\|\\mathcal\{P\}\(A\)\-\\mathcal\{P\}\(B\)\\\|\_\{\\mathrm\{F\}\}\\leq\\frac\{2\\\|A\-B\\\|\_\{\\mathrm\{F\}\}\}\{\\sigma\_\{\\min\}\(A\)\+\\sigma\_\{\\min\}\(B\)\}\.\(20\)Consequently, on any region satisfyingσmin​\(X\)≥σ0\>0\\sigma\_\{\\min\}\(X\)\\geq\\sigma\_\{0\}\>0, the polar map is1/σ01/\\sigma\_\{0\}\-Lipschitz in Frobenius norm\.

###### Proof\.

Write the row\-polar decompositions as

A=HA​PA,B=HB​PB,A=H\_\{A\}P\_\{A\},\\qquad B=H\_\{B\}P\_\{B\},\(21\)whereHA=\(A​A⊤\)1/2H\_\{A\}=\(AA^\{\\top\}\)^\{1/2\},HB=\(B​B⊤\)1/2H\_\{B\}=\(BB^\{\\top\}\)^\{1/2\},PA=𝒫⁡\(A\)P\_\{A\}=\\mathcal\{P\}\(A\), andPB=𝒫⁡\(B\)P\_\{B\}=\\mathcal\{P\}\(B\)\. BothPA​PA⊤=IrP\_\{A\}P\_\{A\}^\{\\top\}=I\_\{r\}andPB​PB⊤=IrP\_\{B\}P\_\{B\}^\{\\top\}=I\_\{r\}\. LetC=PA​PB⊤C=P\_\{A\}P\_\{B\}^\{\\top\},sym⁡\(C\):=\(C\+C⊤\)/2\\operatorname\{sym\}\(C\):=\(C\+C^\{\\top\}\)/2\. Since‖C‖2≤1\\\|C\\\|\_\{2\}\\leq 1,Ir−sym⁡\(C\)I\_\{r\}\-\\operatorname\{sym\}\(C\)is positive semidefinite\. Direct expansion gives

⟨A−B,PA−PB⟩\\displaystyle\\langle A\-B,P\_\{A\}\-P\_\{B\}\\rangle=tr⁡\(HA​\(Ir−C⊤\)\)\+tr⁡\(HB​\(Ir−C\)\)\\displaystyle=\\operatorname\{tr\}\\\!\\left\(H\_\{A\}\(I\_\{r\}\-C^\{\\top\}\)\\right\)\+\\operatorname\{tr\}\\\!\\left\(H\_\{B\}\(I\_\{r\}\-C\)\\right\)≥\(σmin​\(A\)\+σmin​\(B\)\)​\(r−tr⁡\(C\)\)\.\\displaystyle\\geq\\bigl\(\\sigma\_\{\\min\}\(A\)\+\\sigma\_\{\\min\}\(B\)\\bigr\)\\bigl\(r\-\\operatorname\{tr\}\(C\)\\bigr\)\.\(22\)The skew\-symmetric parts vanish inside the traces becauseHAH\_\{A\}andHBH\_\{B\}are symmetric\. Moreover,

r−tr⁡\(C\)=12​‖PA−PB‖F2\.r\-\\operatorname\{tr\}\(C\)=\\frac\{1\}\{2\}\\\|P\_\{A\}\-P\_\{B\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(23\)Combining this identity with Cauchy–Schwarz yields

‖A−B‖F​‖PA−PB‖F≥σmin​\(A\)\+σmin​\(B\)2​‖PA−PB‖F2\.\\\|A\-B\\\|\_\{\\mathrm\{F\}\}\\\|P\_\{A\}\-P\_\{B\}\\\|\_\{\\mathrm\{F\}\}\\geq\\frac\{\\sigma\_\{\\min\}\(A\)\+\\sigma\_\{\\min\}\(B\)\}\{2\}\\\|P\_\{A\}\-P\_\{B\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(24\)IfPA=PBP\_\{A\}=P\_\{B\}, the result is immediate; otherwise division by‖PA−PB‖F\\\|P\_\{A\}\-P\_\{B\}\\\|\_\{\\mathrm\{F\}\}proves equation[20](https://arxiv.org/html/2608.20818#A1.E20)\. ∎

Proposition[2](https://arxiv.org/html/2608.20818#Thmproposition2)provides a conditional motivation for periodic spectral correction: moderate momentum drift implies moderate drift of the ideal polar direction only when the relevant matrices remain away from rank degeneracy\. The proposition concerns the exact polar factor and does not establish the same perturbation bound for the finite NS5 map used in the implementation\. We therefore use it as geometric motivation rather than as a guarantee that finite NS5 remains unchanged between refreshes\.

### A\.5Conditional finite\-horizon descent

Letℱt\\mathcal\{F\}\_\{t\}denote the information available before the stochastic direction at stepttis realized\. The following result applies directly to the complete effective direction in equation[13](https://arxiv.org/html/2608.20818#A1.E13); it does not require the iterates to track those of vanilla Muon\.

###### Theorem 1\(Finite\-horizon bound under conditional alignment \)\.

Suppose thatf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}isLL\-smooth and bounded below byfinff\_\{\\inf\}\. Assume that, for eachtt, there are deterministic constantsat\>0a\_\{t\}\>0,bt≥0b\_\{t\}\\geq 0, andvt≥0v\_\{t\}\\geq 0such that

𝔼⁡\[⟨∇f​\(θt\),Ht⟩∣ℱt\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\langle\\nabla f\(\\theta\_\{t\}\),H\_\{t\}\\rangle\\mid\\mathcal\{F\}\_\{t\}\\right\]≥at​‖∇f​\(θt\)‖22,\\displaystyle\\geq a\_\{t\}\\\|\\nabla f\(\\theta\_\{t\}\)\\\|\_\{2\}^\{2\},\(25\)𝔼⁡\[‖Ht‖22∣ℱt\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\\|H\_\{t\}\\\|\_\{2\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\\right\]≤bt​‖∇f​\(θt\)‖22\+vt\.\\displaystyle\\leq b\_\{t\}\\\|\\nabla f\(\\theta\_\{t\}\)\\\|\_\{2\}^\{2\}\+v\_\{t\}\.\(26\)If

dt:=at−L​ηt​bt2\>0for​t=0,…,T−1,d\_\{t\}:=a\_\{t\}\-\\frac\{L\\eta\_\{t\}b\_\{t\}\}\{2\}\>0\\qquad\\text\{for \}t=0,\\ldots,T\-1,\(27\)then

∑t=0T−1ηt​dt​𝔼​‖∇f​\(θt\)‖22≤f⁡\(θ0\)−finf\+L2​∑t=0T−1ηt2​vt\.\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}d\_\{t\}\\,\\mathbb\{E\}\\\|\\nabla f\(\\theta\_\{t\}\)\\\|\_\{2\}^\{2\}\\leq f\(\\theta\_\{0\}\)\-f\_\{\\inf\}\+\\frac\{L\}\{2\}\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{2\}v\_\{t\}\.\(28\)This statement holds for any deterministic switching schedule, including a periodic schedule with one NS5 step followed byK−1K\-1RowNorm steps\.

###### Proof\.

ByLL\-smoothness and equation[13](https://arxiv.org/html/2608.20818#A1.E13),

f⁡\(θt\+1\)≤f⁡\(θt\)−ηt​⟨∇f​\(θt\),Ht⟩\+L​ηt22​‖Ht‖22\.f\(\\theta\_\{t\+1\}\)\\leq f\(\\theta\_\{t\}\)\-\\eta\_\{t\}\\langle\\nabla f\(\\theta\_\{t\}\),H\_\{t\}\\rangle\+\\frac\{L\\eta\_\{t\}^\{2\}\}\{2\}\\\|H\_\{t\}\\\|\_\{2\}^\{2\}\.\(29\)Taking conditional expectation and applying equation[25](https://arxiv.org/html/2608.20818#A1.E25)and equation[26](https://arxiv.org/html/2608.20818#A1.E26)gives

𝔼⁡\[f⁡\(θt\+1\)∣ℱt\]≤f⁡\(θt\)−ηt​dt​‖∇f​\(θt\)‖22\+L​ηt22​vt\.\\mathbb\{E\}\[f\(\\theta\_\{t\+1\}\)\\mid\\mathcal\{F\}\_\{t\}\]\\leq f\(\\theta\_\{t\}\)\-\\eta\_\{t\}d\_\{t\}\\\|\\nabla f\(\\theta\_\{t\}\)\\\|\_\{2\}^\{2\}\+\\frac\{L\\eta\_\{t\}^\{2\}\}\{2\}v\_\{t\}\.\(30\)Taking total expectation, summing overt=0,…,T−1t=0,\\ldots,T\-1, and usingf⁡\(θT\)≥finff\(\\theta\_\{T\}\)\\geq f\_\{\\inf\}proves equation[28](https://arxiv.org/html/2608.20818#A1.E28)\. ∎

For example, ifηt=η\\eta\_\{t\}=\\eta,at≥a\>0a\_\{t\}\\geq a\>0,0≤bt≤b0\\leq b\_\{t\}\\leq bfor someb\>0b\>0,vt≤vv\_\{t\}\\leq v, and0<η<2​a/\(L​b\)0<\\eta<2a/\(Lb\), then

1T​∑t=0T−1𝔼​‖∇f​\(θt\)‖22≤f⁡\(θ0\)−finfT​η​\(a−L​η​b/2\)\+L​η​v2​\(a−L​η​b/2\)\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla f\(\\theta\_\{t\}\)\\\|\_\{2\}^\{2\}\\leq\\frac\{f\(\\theta\_\{0\}\)\-f\_\{\\inf\}\}\{T\\eta\\left\(a\-L\\eta b/2\\right\)\}\+\\frac\{L\\eta v\}\{2\\left\(a\-L\\eta b/2\\right\)\}\.\(31\)This is a finite\-horizon stationary\-point bound with a noise\-dependent residual, not a claim of asymptotic convergence for the finite cosine schedule used in our experiments\.

The theorem is explicitly conditional\. RowNorm is positively aligned with its own input momentum whenever the momentum has a nonzero row:

⟨M,ℛ\(M\)⟩=∑i=1r∥Mi:∥2\>0\.\\langle M,\\operatorname\{\\mathcal\{R\}\}\(M\)\\rangle=\\sum\_\{i=1\}^\{r\}\\\|M\_\{i:\}\\\|\_\{2\}\>0\.\(32\)However, this identity alone does not imply positive alignment with the true objective gradient, because in general⟨∇f​\(θt\),ℛ⁡\(Mt\)⟩\\langle\\nabla f\(\\theta\_\{t\}\),\\operatorname\{\\mathcal\{R\}\}\(M\_\{t\}\)\\rangleand⟨Mt,ℛ⁡\(Mt\)⟩\\langle M\_\{t\},\\operatorname\{\\mathcal\{R\}\}\(M\_\{t\}\)\\rangleare different quantities\. We therefore use Theorem[1](https://arxiv.org/html/2608.20818#Thmtheorem1)only as a sufficient\-condition result and do not claim that its alignment assumption is verified throughout training\.

### A\.6The blockwise role of the RowNorm multiplier

Conditioned on a fixed optimization historyℱt\\mathcal\{F\}\_\{t\}and the current iterateθt\\theta\_\{t\}, decompose the complete effective direction on an off\-refresh step as

Ht​\(γ\)=Ht\(0\)\+γ​Ht\(RN\),H\_\{t\}\(\\gamma\)=H\_\{t\}^\{\(0\)\}\+\\gamma H\_\{t\}^\{\(\\mathrm\{RN\}\)\},\(33\)whereHt\(RN\)H\_\{t\}^\{\(\\mathrm\{RN\}\)\}contains the unscaled RowNorm directions in the blocks to which the method is applied, andHt\(0\)H\_\{t\}^\{\(0\)\}contains all remaining contributions\. Define the conditional scalar quantities

A0,t\\displaystyle A\_\{0,t\}:=𝔼⁡\[⟨∇f​\(θt\),Ht\(0\)⟩∣ℱt\],\\displaystyle:=\\mathbb\{E\}\[\\langle\\nabla f\(\\theta\_\{t\}\),H\_\{t\}^\{\(0\)\}\\rangle\\mid\\mathcal\{F\}\_\{t\}\],\(34\)ARN,t\\displaystyle A\_\{\\mathrm\{RN\},t\}:=𝔼⁡\[⟨∇f​\(θt\),Ht\(RN\)⟩∣ℱt\],\\displaystyle:=\\mathbb\{E\}\[\\langle\\nabla f\(\\theta\_\{t\}\),H\_\{t\}^\{\(\\mathrm\{RN\}\)\}\\rangle\\mid\\mathcal\{F\}\_\{t\}\],\(35\)B0,t\\displaystyle B\_\{0,t\}:=𝔼⁡\[‖Ht\(0\)‖22∣ℱt\],\\displaystyle:=\\mathbb\{E\}\[\\\|H\_\{t\}^\{\(0\)\}\\\|\_\{2\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\],\(36\)BRN,t\\displaystyle B\_\{\\mathrm\{RN\},t\}:=𝔼⁡\[‖Ht\(RN\)‖22∣ℱt\],\\displaystyle:=\\mathbb\{E\}\[\\\|H\_\{t\}^\{\(\\mathrm\{RN\}\)\}\\\|\_\{2\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\],\(37\)B×,t\\displaystyle B\_\{\\times,t\}:=𝔼⁡\[⟨Ht\(0\),Ht\(RN\)⟩∣ℱt\]\.\\displaystyle:=\\mathbb\{E\}\[\\langle H\_\{t\}^\{\(0\)\},H\_\{t\}^\{\(\\mathrm\{RN\}\)\}\\rangle\\mid\\mathcal\{F\}\_\{t\}\]\.\(38\)Then the two quantities entering the smoothness bound are exactly

𝔼⁡\[⟨∇f​\(θt\),Ht​\(γ\)⟩∣ℱt\]\\displaystyle\\mathbb\{E\}\[\\langle\\nabla f\(\\theta\_\{t\}\),H\_\{t\}\(\\gamma\)\\rangle\\mid\\mathcal\{F\}\_\{t\}\]=A0,t\+γ​ARN,t,\\displaystyle=A\_\{0,t\}\+\\gamma A\_\{\\mathrm\{RN\},t\},\(39\)𝔼⁡\[‖Ht​\(γ\)‖22∣ℱt\]\\displaystyle\\mathbb\{E\}\[\\\|H\_\{t\}\(\\gamma\)\\\|\_\{2\}^\{2\}\\mid\\mathcal\{F\}\_\{t\}\]=B0,t\+2​γ​B×,t\+γ2​BRN,t\.\\displaystyle=B\_\{0,t\}\+2\\gamma B\_\{\\times,t\}\+\\gamma^\{2\}B\_\{\\mathrm\{RN\},t\}\.\(40\)Consequently, the conditional one\-step upper bound is

𝔼⁡\[f⁡\(θt\+1\)∣ℱt\]≤f⁡\(θt\)\\displaystyle\\mathbb\{E\}\[f\(\\theta\_\{t\+1\}\)\\mid\\mathcal\{F\}\_\{t\}\]\\leq f\(\\theta\_\{t\}\)−ηt​\(A0,t\+γ​ARN,t\)\\displaystyle\-\\eta\_\{t\}\\bigl\(A\_\{0,t\}\+\\gamma A\_\{\\mathrm\{RN\},t\}\\bigr\)\+L​ηt22​\(B0,t\+2​γ​B×,t\+γ2​BRN,t\)\.\\displaystyle\+\\frac\{L\\eta\_\{t\}^\{2\}\}\{2\}\\bigl\(B\_\{0,t\}\+2\\gamma B\_\{\\times,t\}\+\\gamma^\{2\}B\_\{\\mathrm\{RN\},t\}\\bigr\)\.\(41\)Thusγ\\gammais admissible whenever the measured or assumed bounds make the net descent term in equation[41](https://arxiv.org/html/2608.20818#A1.E41)positive\. Equation[39](https://arxiv.org/html/2608.20818#A1.E39)and[40](https://arxiv.org/html/2608.20818#A1.E40)also show why scaling the RowNorm blocks does*not*multiply the alignment and second\-moment constants of the complete direction byγ\\gammaandγ2\\gamma^\{2\}, respectively\. The theory motivates branch\-specific amplitude calibration but does not determine the numerical value used in the experiments\. That value is selected by the development\-set ablation in Appendix[E\.1](https://arxiv.org/html/2608.20818#A5.SS1)\.

## Appendix BComplexity and Distributed Execution

This appendix refines the asymptotic discussion in the main text with an implementation matched arithmetic model, a correctness proof for sharded RowNorm, and explicit per\-rank communication formulas\.

### B\.1Per\-matrix arithmetic

Consider an oriented matrixX∈ℝr×cX\\in\\mathbb\{R\}^\{r\\times c\}withr≤cr\\leq c\. We count one multiplication followed by one addition as two floating\-point operations\. The implemented quintic Newton–Schulz step can be written as

Gj\\displaystyle G\_\{j\}=Xj​Xj⊤,\\displaystyle=X\_\{j\}X\_\{j\}^\{\\top\},\(42\)Bj\\displaystyle B\_\{j\}=bj​Gj\+cj​Gj2,\\displaystyle=b\_\{j\}G\_\{j\}\+c\_\{j\}G\_\{j\}^\{2\},\(43\)Xj\+1\\displaystyle X\_\{j\+1\}=ajXj\+BjXj,j=0,…,JNS−1\.\\displaystyle=a\_\{j\}X\_\{j\}\+B\_\{j\}X\_\{j\},\\qquad j=0,\\ldots,J\_\{\\mathrm\{NS\}\}\-1\.\(44\)The three GEMMs in one iteration cost, to leading order,

2​r2​c,2​r3,2​r2​c2r^\{2\}c,\\qquad 2r^\{3\},\\qquad 2r^\{2\}c\(45\)FLOPs, respectively\. Therefore

CNS​J​\(r,c\)=JNS​\(4​r2​c\+2​r3\)\+O⁡\(JNS​\(r2\+r​c\)\)\.C\_\{\\mathrm\{NS\}J\}\(r,c\)=J\_\{\\mathrm\{NS\}\}\\bigl\(4r^\{2\}c\+2r^\{3\}\\bigr\)\+O\\\!\\left\(J\_\{\\mathrm\{NS\}\}\(r^\{2\}\+rc\)\\right\)\.\(46\)For NS5,

CNS5​\(r,c\)=20​r2​c\+10​r3\+O⁡\(r2\+r​c\)\.C\_\{\\mathrm\{NS5\}\}\(r,c\)=20r^\{2\}c\+10r^\{3\}\+O\(r^\{2\}\+rc\)\.\(47\)The lower\-order term includes scalar matrix combinations and the input normalization\. Equation[47](https://arxiv.org/html/2608.20818#A2.E47)is a FLOP model for the recurrence in equation[44](https://arxiv.org/html/2608.20818#A2.E44); measured kernel time can differ because GEMM efficiency depends on shape, dtype, and hardware\.

RowNorm performs one sum\-of\-squares reduction per row, one clamped inverse norm per row, and one rescaling per element\. Its arithmetic cost is

CRN​\(r,c\)=Θ⁡\(r​c\)\.C\_\{\\mathrm\{RN\}\}\(r,c\)=\\Theta\(rc\)\.\(48\)We do not assign a hardware\-independent exact FLOP count to square root, reciprocal, or fused reduction operations\. The relevant separation is therefore

CNS5​\(r,c\)=Θ⁡\(r2​c\),CRN​\(r,c\)=Θ⁡\(r​c\)\.C\_\{\\mathrm\{NS5\}\}\(r,c\)=\\Theta\(r^\{2\}c\),\\qquad C\_\{\\mathrm\{RN\}\}\(r,c\)=\\Theta\(rc\)\.\(49\)
Letℐ\\mathcal\{I\}be the set of matrices governed by the periodic rule\. The period\-averaged matrix\-processing computational cost is

C¯mat​\(K\)=1K​∑i∈ℐCNS5​\(ri,ci\)\+K−1K​∑i∈ℐCRN​\(ri,ci\)\.\\overline\{C\}\_\{\\mathrm\{mat\}\}\(K\)=\\frac\{1\}\{K\}\\sum\_\{i\\in\\mathcal\{I\}\}C\_\{\\mathrm\{NS5\}\}\(r\_\{i\},c\_\{i\}\)\+\\frac\{K\-1\}\{K\}\\sum\_\{i\\in\\mathcal\{I\}\}C\_\{\\mathrm\{RN\}\}\(r\_\{i\},c\_\{i\}\)\.\(50\)IfCfixedC\_\{\\mathrm\{fixed\}\}denotes momentum maintenance, non\-Muon parameter updates, and all other optimizer work performed every step, then

C¯opt​\(K\)=Cfixed\+C¯mat​\(K\)\.\\overline\{C\}\_\{\\mathrm\{opt\}\}\(K\)=C\_\{\\mathrm\{fixed\}\}\+\\overline\{C\}\_\{\\mathrm\{mat\}\}\(K\)\.\(51\)The periodic schedule removes exactly the fraction1−1/K1\-1/Kof spectral refresh events, but it does not remove the same fraction of total optimizer FLOPs or wall\-clock time because the terms in equation[51](https://arxiv.org/html/2608.20818#A2.E51)remain\.

### B\.2Correctness of sharded RowNorm

Suppose that the original matrixM∈ℝm×nM\\in\\mathbb\{R\}^\{m\\times n\}is sharded along its first dimension acrossppranks\. Rankqqowns the row\-index setSqS\_\{q\}, and the setsS1,…,SpS\_\{1\},\\ldots,S\_\{p\}form a disjoint partition of\{1,…,m\}\\\{1,\\ldots,m\\\}\.

Ifm≤nm\\leq n, the orientation map leavesMMunchanged\. Each oriented row is therefore stored completely on one rank, and applying equation[12](https://arxiv.org/html/2608.20818#A1.E12)to each local row is exactly the corresponding slice of dense RowNorm\.

Ifm\>nm\>n, the orientation map givesX=M⊤∈ℝn×mX=M^\{\\top\}\\in\\mathbb\{R\}^\{n\\times m\}\. An oriented row is now split across ranks\. For each original columnj∈\{1,…,n\}j\\in\\\{1,\\ldots,n\\\}, rankqqcomputes

sj\(q\):=∑i∈SqMi​j2\.s\_\{j\}^\{\(q\)\}:=\\sum\_\{i\\in S\_\{q\}\}M\_\{ij\}^\{2\}\.\(52\)After a sum all\-reduce, every rank forms

νj:=max⁡\{∑q=1psj\(q\),ε\}\.\\nu\_\{j\}:=\\max\\left\\\{\\sqrt\{\\sum\_\{q=1\}^\{p\}s\_\{j\}^\{\(q\)\}\},\\varepsilon\\right\\\}\.\(53\)
Each rank then rescales its local entries as

M^i​j\(q\):=Mi​jνj,i∈Sq,j=1,…,n\.\\widehat\{M\}\_\{ij\}^\{\(q\)\}:=\\frac\{M\_\{ij\}\}\{\\nu\_\{j\}\},\\qquad i\\in S\_\{q\},\\quad j=1,\\ldots,n\.\(54\)
###### Proposition 3\(Exact\-arithmetic equivalence\)\.

Under the partition above, concatenating the local outputsM^\(1\),…,M^\(p\)\\widehat\{M\}^\{\(1\)\},\\ldots,\\widehat\{M\}^\{\(p\)\}gives exactly

𝒪−1​\(Rϵ​\(𝒪⁡\(M\)\)\)\.\\mathcal\{O\}^\{\-1\}\\bigl\(R\_\{\\epsilon\}\(\\mathcal\{O\}\(M\)\)\\bigr\)\.

###### Proof\.

The complete\-row case follows immediately because each local denominator is computed from all entries of its row\. In the split\-row case,

∑q=1psj\(q\)=∑q=1p∑i∈SqMi​j2=∑i=1mMi​j2=∥Xj:∥22\.\\sum\_\{q=1\}^\{p\}s\_\{j\}^\{\(q\)\}=\\sum\_\{q=1\}^\{p\}\\sum\_\{i\\in S\_\{q\}\}M\_\{ij\}^\{2\}=\\sum\_\{i=1\}^\{m\}M\_\{ij\}^\{2\}=\\\|X\_\{j:\}\\\|\_\{2\}^\{2\}\.\(55\)Henceνj=max\{∥Xj:∥2,ε\}\\nu\_\{j\}=\\max\\\{\\\|X\_\{j:\}\\\|\_\{2\},\\varepsilon\\\}\. Every local entry of the oriented row is divided by the same dense denominator, so concatenating the local pieces yieldsXj:/νjX\_\{j:\}/\\nu\_\{j\}for everyjj, which is exactlyℛε⁡\(X\)\\operatorname\{\\mathcal\{R\}\}\_\{\\varepsilon\}\(X\)\. Mapping back through𝒪−1\\mathcal\{O\}^\{\-1\}completes the proof\. ∎

The proposition is exact over real arithmetic\. In floating\-point arithmetic, the all\-reduce may change the summation order and therefore introduce only the usual reduction\-order roundoff differences\.

### B\.3Per\-rank communication volume

We report logical algorithmic payload and exclude headers, padding, collective launch latency, and parameter communication already required by forward and backward propagation\. LetbMb\_\{M\}be the number of bytes per communicated momentum element andbSb\_\{S\}the number of bytes per communicated norm statistic\. Under a bandwidth\-optimal ring model, define

αAG​\(p\):=p−1p,αAR​\(p\):=2​p−1p\.\\alpha\_\{\\mathrm\{AG\}\}\(p\):=\\frac\{p\-1\}\{p\},\\qquad\\alpha\_\{\\mathrm\{AR\}\}\(p\):=2\\frac\{p\-1\}\{p\}\.\(56\)The factor of two inαAR​\(p\)\\alpha\_\{\\mathrm\{AR\}\}\(p\)accounts for the reduce\-scatter and all\-gather phases of a ring all\-reduce\.

For a refresh ofM∈ℝm×nM\\in\\mathbb\{R\}^\{m\\times n\}, the per\-rank all\-gather volume is

Vrefresh​\(M\)=αAG​\(p\)​bM​m​n\.V\_\{\\mathrm\{refresh\}\}\(M\)=\\alpha\_\{\\mathrm\{AG\}\}\(p\)b\_\{M\}mn\.\(57\)Every rank executes NS5 on the gathered matrix and retains its local output slice, so this execution path requires no subsequent scatter of the update\.

On an off\-refresh step, complete\-row RowNorm requires no optimizer\-specific collective\. A tall matrixm\>nm\>nrequires an all\-reduce ofnnstatistics, giving

VRN​\(M\)=\{αAR​\(p\)​bS​n,m\>n,0,m≤n\.V\_\{\\mathrm\{RN\}\}\(M\)=\\begin\{cases\}\\alpha\_\{\\mathrm\{AR\}\}\(p\)b\_\{S\}n,&m\>n,\\\\ 0,&m\\leq n\.\\end\{cases\}\(58\)For one tall matrix, the ratio is therefore

VRN​\(M\)Vrefresh​\(M\)=2​bSbM​m\.\\frac\{V\_\{\\mathrm\{RN\}\}\(M\)\}\{V\_\{\\mathrm\{refresh\}\}\(M\)\}=\\frac\{2b\_\{S\}\}\{b\_\{M\}m\}\.\(59\)
For a collectionℐ\\mathcal\{I\}of periodically updated matrices, define

Sfull:=∑i∈ℐmini,Stall:=∑i∈ℐ:mi\>nini\.S\_\{\\mathrm\{full\}\}:=\\sum\_\{i\\in\\mathcal\{I\}\}m\_\{i\}n\_\{i\},\\qquad S\_\{\\mathrm\{tall\}\}:=\\sum\_\{i\\in\\mathcal\{I\}:m\_\{i\}\>n\_\{i\}\}n\_\{i\}\.\(60\)The period\-averaged per\-rank payload is

V¯rank​\(K\)\\displaystyle\\overline\{V\}\_\{\\mathrm\{rank\}\}\(K\)=1K​αAG​\(p\)​bM​Sfull\+K−1K​αAR​\(p\)​bS​Stall\.\\displaystyle=\\frac\{1\}\{K\}\\alpha\_\{\\mathrm\{AG\}\}\(p\)b\_\{M\}S\_\{\\mathrm\{full\}\}\+\\frac\{K\-1\}\{K\}\\alpha\_\{\\mathrm\{AR\}\}\(p\)b\_\{S\}S\_\{\\mathrm\{tall\}\}\.\(61\)Relative to vanilla Muon’s every\-step refresh payload,

V¯rank​\(K\)αAG​\(p\)​bM​Sfull=1K\+K−1K​2​bS​StallbM​Sfull\.\\frac\{\\overline\{V\}\_\{\\mathrm\{rank\}\}\(K\)\}\{\\alpha\_\{\\mathrm\{AG\}\}\(p\)b\_\{M\}S\_\{\\mathrm\{full\}\}\}=\\frac\{1\}\{K\}\+\\frac\{K\-1\}\{K\}\\frac\{2b\_\{S\}S\_\{\\mathrm\{tall\}\}\}\{b\_\{M\}S\_\{\\mathrm\{full\}\}\}\.\(62\)When the reduced statistics are small relative to the full matrices, the second term is negligible and the ratio approaches1/K1/K\. This is a statement about logical payload, not a claim that collective time or total step time decreases by the same factor\.

### B\.4Idealized overlap model

Bucketed execution changes exposed latency rather than the logical byte count in equation[57](https://arxiv.org/html/2608.20818#A2.E57)\. ConsiderBBordered refresh buckets\. Letaℓa\_\{\\ell\}be the all\-gather time for bucketℓ\\ell, and letcℓc\_\{\\ell\}be its reconstruction, NS5, and slicing time\. In an ideal two\-stage pipeline that overlaps the computation of bucketℓ\\ellwith the all\-gather of bucketℓ\+1\\ell\+1, the makespan is

Tpipe=a1\+∑ℓ=1B−1max⁡\{cℓ,aℓ\+1\}\+cB\.T\_\{\\mathrm\{pipe\}\}=a\_\{1\}\+\\sum\_\{\\ell=1\}^\{B\-1\}\\max\\\{c\_\{\\ell\},a\_\{\\ell\+1\}\\\}\+c\_\{B\}\.\(63\)The corresponding exposed communication time beyond bucket computation is

Tcomm,exposed=a1\+∑ℓ=1B−1max⁡\{0,aℓ\+1−cℓ\}\.T\_\{\\mathrm\{comm,exposed\}\}=a\_\{1\}\+\\sum\_\{\\ell=1\}^\{B\-1\}\\max\\\{0,a\_\{\\ell\+1\}\-c\_\{\\ell\}\\\}\.\(64\)These equations describe an ideal dependency graph\. Actual traces may include stream synchronization, launch overhead, contention, and imperfect kernel concurrency; end\-to\-end claims are therefore based on measured profiler windows spanning complete periods\.

### B\.5Wall\-clock break\-even condition

LetNbase​\(q\)N\_\{\\mathrm\{base\}\}\(q\)andNper​\(q\)N\_\{\\mathrm\{per\}\}\(q\)be the numbers of optimization steps required by vanilla Muon and Periodic Muon, respectively, to reach a prespecified quality thresholdqq\. Let their measured mean step times beτbase\\tau\_\{\\mathrm\{base\}\}andτper\\tau\_\{\\mathrm\{per\}\}, using the same number of GPUs\. Periodic Muon improves time to quality exactly when

Nper​\(q\)​τper<Nbase​\(q\)​τbase,N\_\{\\mathrm\{per\}\}\(q\)\\tau\_\{\\mathrm\{per\}\}<N\_\{\\mathrm\{base\}\}\(q\)\\tau\_\{\\mathrm\{base\}\},\(65\)or equivalently,

Nper​\(q\)Nbase​\(q\)<τbaseτper\.\\frac\{N\_\{\\mathrm\{per\}\}\(q\)\}\{N\_\{\\mathrm\{base\}\}\(q\)\}<\\frac\{\\tau\_\{\\mathrm\{base\}\}\}\{\\tau\_\{\\mathrm\{per\}\}\}\.\(66\)Thus a lower per\-step optimizer systems cost is insufficient by itself: the realized speedup must exceed any increase in the number of steps required to reach the same quality\. If the GPU counts differ, the analogous GPU\-hour comparison multiplies each side by its respective accelerator count\.

## Appendix CAlgorithms

In this section, we provide the Periodic Row\-wise Muon algorithm[1](https://arxiv.org/html/2608.20818#algorithm1)and the system\-level algorithm[2](https://arxiv.org/html/2608.20818#algorithm2)\.

Algorithm 1Periodic Row\-wise MuonInput:two\-dimensional Muon parameters𝒲M\\mathcal\{W\}\_\{\\mathrm\{M\}\}, remaining parameters𝒲A\\mathcal\{W\}\_\{\\mathrm\{A\}\}, periodKK, and RowNorm multiplierγ\\gamma\.

1fort=0,…,T−1t=0,\\ldots,T\-1do

2Compute stochastic gradients and update every momentumMtM\_\{t\}as in Section 2\.

3Setρt←𝕀\[tmodK=0\]\\rho\_\{t\}\\leftarrow\\mathbb\{I\}\[t\\bmod K=0\]\.

4for eachWt∈𝒲MW\_\{t\}\\in\\mathcal\{W\}\_\{\\mathrm\{M\}\}do

5M~t←𝒪⁡\(Mt\)\\widetilde\{M\}\_\{t\}\\leftarrow\\mathcal\{O\}\(M\_\{t\}\)\.

6ifρt=1\\rho\_\{t\}=1thenD~t←NS5⁡\(M~t\)\\widetilde\{D\}\_\{t\}\\leftarrow\\operatorname\{NS5\}\(\\widetilde\{M\}\_\{t\}\)\.

7elseD~t←γ​ℛϵ​\(M~t\)\\widetilde\{D\}\_\{t\}\\leftarrow\\gamma\\mathcal\{R\}\_\{\\epsilon\}\(\\widetilde\{M\}\_\{t\}\)\.

8Dt←s⁡\(Wt\)​𝒪−1​\(D~t\)D\_\{t\}\\leftarrow s\(W\_\{t\}\)\\mathcal\{O\}^\{\-1\}\(\\widetilde\{D\}\_\{t\}\)\.

9Wt\+1←\(1−ηt​λ\)​Wt−ηt​DtW\_\{t\+1\}\\leftarrow\(1\-\\eta\_\{t\}\\lambda\)W\_\{t\}\-\\eta\_\{t\}D\_\{t\}\.

10end for

11Update𝒲A\\mathcal\{W\}\_\{\\mathrm\{A\}\}with the same AdamW rule as the baseline\.

12end for

Algorithm 2Distributed Periodic MuonInput:momentum shards\{Mt\(p\)\}\\\{M\_\{t\}^\{\(p\)\}\\\}on rankpp, refresh flagρt\\rho\_\{t\}, and communication buckets\{ℬℓ\}\\\{\\mathcal\{B\}\_\{\\ell\}\\\}\.

1ifρt=1\\rho\_\{t\}=1then// NS5 refresh path

2Asynchronously all\-gather the first momentum bucket\.

3for eachbucketℬℓ\\mathcal\{B\}\_\{\\ell\}do

4Wait for the current bucket and launch the next asynchronous all\-gather\.

5Reconstruct full momenta, execute NS5, and slice the results back to local shards\.

6Release the full\-matrix buffers of the current bucket\.

7end for

8else// sharded RowNorm path

9Compute local column\-squared statistics for all tall matrices\.

10Pack the statistics and launch bucketed asynchronous sum all\-reduces\.

11While communication is in flight, compute and finish local RowNorm updates for the remaining matrices\.

12Wait for all\-reduce completion; take the square root and then applyclampmin⁡\(ϵ\)\\operatorname\{clamp\}\_\{\\min\}\(\\epsilon\)\.

13Use the global column norms to finish RowNorm updates for tall matrices\.

14end if

15Applyγ\\gammato the off\-refresh RowNorm outputs only, and fuse shape scaling with the local parameter\-shard updates\.

## Appendix DDetailed Experimental Setup

### D\.1GPIC Dataset

GPIC \(Giant Permissive Image Corpus\)\([6](https://arxiv.org/html/2608.20818#bib.bib24)\)is an open image–text dataset designed for large\-scale visual\-generation research and contains approximately 28 trillion pixels\. Its full split comprises 100M training examples, 200K validation examples, and 1M test examples\. We train on GPIC\-Full, use the validation set to monitor training, and perform generation evaluation on the held\-out test set\.

GPIC applies vision\-language\-model\-based quality and safety filtering and removes duplicate and near\-duplicate images using SSCD visual features\. After filtering and deduplication, approximately 101\.3M images remain and are divided into 100M training, 200K validation, and 1M test examples\. Qwen3\-VL\-4B\-Instruct\([4](https://arxiv.org/html/2608.20818#bib.bib25)\)generates captions at multiple levels of detail rather than relying on potentially noisy or missing web metadata\. The training, validation, and test splits retain similar source and caption distributions\.

### D\.2Model Configurations

We use a latent space, dual stream MMDiT architecture\([11](https://arxiv.org/html/2608.20818#bib.bib16)\)\. A frozen FLUX\.1\-schnell\([19](https://arxiv.org/html/2608.20818#bib.bib33)\)VAE encodes images into the latent space, while frozen CLIP\-L, CLIP\-G\([27](https://arxiv.org/html/2608.20818#bib.bib34)\), and T5\-XXL\([28](https://arxiv.org/html/2608.20818#bib.bib35)\)encoders provide text conditioning\. The MMDiT backbone includes joint image–text attention, RoPE positional encoding, QK normalization, SwiGLU MLPs, and adaptive normalization\. All four models use the same components and connectivity, and differ only in hidden dimension, depth, and number of attention heads\.

In the Muon and Periodic Row\-wise Muon runs, the corresponding Muon update rule is applied to two\-dimensional weight matrices within the Transformer blocks\. Biases, normalization parameters, scalar parameters, and parameters outside the Transformer blocks are updated by AdamW\.

### D\.3Optimizer and Training Hyperparameters

All methods use decoupled weight decay\. The AdamW baseline applies AdamW to all parameters\. Muon and Periodic Row\-wise Muon apply matrix\-valued updates to two\-dimensional hidden\-layer weights and maintain a separate AdamW parameter group for all remaining parameters\.

All Muon experiments use five Newton–Schulz iterations with fixed coefficients\(a,b,c\)=\(3\.4445,−4\.7750,2\.0315\)\(a,b,c\)=\(3\.4445,\-4\.7750,2\.0315\)\.

At a given model scale, the three optimizers use the same model, training length, global batch size, and learning\-rate schedule, and therefore process the same number of training examples\. Because their optimizer computation and communication costs differ, they are not constrained to use the same wall\-clock time or total compute\.

### D\.4Generation Evaluation

We evaluate a checkpoint every 5,000 training steps, covering the complete trajectory from 5,000 to 60,000 steps\. We select a fixed set of 50,000 prompts from the GPIC test set and generate 50,000 images at512×512512\\times 512resolution\. All models and optimizers use the same sampling configuration with a fixed classifier\-free guidance scale of 5\.0\.

Table 4:Model configurations\. Muon\-routed parameters denotes the fraction of trainable parameters updated by Muon in the Muon\-based runs\.ModelTotal params\.Trainable params\.Hidden dim\.BlocksHeadsMuon\-routed1\.3B1,255,586,0991,255,585,7951,216191998\.69%4B3,950,236,0993,950,235,7151,920243099\.20%9B9,374,837,3159,374,836,8032,560324099\.48%15B14,848,237,25114,848,236,6112,880404599\.60%Table 5:Optimizer hyperparameters\. “Aux\. AdamW” denotes the auxiliary AdamW parameter group used in the Muon\-based runs\. All methods use weight decay0\.010\.01and AdamW\-updated parameter groups useϵ=10−8\\epsilon=10^\{\-8\}\.Main learning rateMethod1\.3B4B9B15BAdamW3\.0×10−43\.0\{\\times\}10^\{\-4\}2\.0×10−42\.0\{\\times\}10^\{\-4\}1\.4×10−41\.4\{\\times\}10^\{\-4\}1\.2×10−41\.2\{\\times\}10^\{\-4\}Muon5\.0×10−35\.0\{\\times\}10^\{\-3\}3\.0×10−33\.0\{\\times\}10^\{\-3\}2\.4×10−32\.4\{\\times\}10^\{\-3\}2\.0×10−32\.0\{\\times\}10^\{\-3\}Periodic Row\-wise Muon5\.0×10−35\.0\{\\times\}10^\{\-3\}3\.0×10−33\.0\{\\times\}10^\{\-3\}2\.4×10−32\.4\{\\times\}10^\{\-3\}2\.0×10−32\.0\{\\times\}10^\{\-3\}
MethodMain momentumor betasAux\. AdamWOrthogonalization / scheduleAdamW\(0\.9,0\.95\)\(0\.9,0\.95\)––Muon0\.950\.95LR=1\.0×10−4=1\.0\{\\times\}10^\{\-4\};
β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\)NS5 at every stepPeriodic Row\-wise Muon0\.950\.95LR=1\.0×10−4=1\.0\{\\times\}10^\{\-4\};
β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\)NS5 refresh everyK=3K=3steps;
RowNorm otherwise \(γ=0\.15\\gamma=0\.15\)

Table 6:Training settings shared by all methods\.SettingValueImage resolution512×512512\\times 512Training length60,000 stepsLocal / global batch size16 / 4,096Learning\-rate schedule2k warmup, then cosine decayHardware32 nodes, 256 NVIDIA H100 GPUsDistributed trainingPytorch FSDP2 with activation checkpointingWe evaluate generated images using the following metrics:

- •FD\-DINOv2\([32](https://arxiv.org/html/2608.20818#bib.bib26)\)computes the Fréchet distance between generated and real\-image distributions in DINOv2 feature space\.
- •FID\([14](https://arxiv.org/html/2608.20818#bib.bib27)\)computes the Fréchet distance between generated and real\-image distributions in Inception feature space\.
- •Maximum Mean Discrepancy \(MMD\)provides a non\-parametric measure of the discrepancy between generated and real feature distributions\.
- •Precision and Densitymeasure how closely generated samples align with the real\-data manifold and primarily characterize sample fidelity\.
- •Recall and Coveragemeasure how well the generated distribution covers the real data distribution and characterize generation diversity\.
- •HPSv2\.1\([35](https://arxiv.org/html/2608.20818#bib.bib31)\)measures text–image alignment and consistency with human preferences\.
- •GenEval2\([16](https://arxiv.org/html/2608.20818#bib.bib32)\)evaluates the correctness of generated objects, attributes, relations, and compositional semantics\.

Unless otherwise specified, all main comparisons and ablations use the same generation and evaluation protocol\.

## Appendix EAdditional Experimental Results

This section provides additional experimental results supporting the main comparisons\. We first describe the selection of the refresh periodKKand the RowNorm scaling factorγ\\gammafor Periodic Row\-wise Muon\. We then provide the complete training loss trajectories, checkpoint\-level generation evaluations, and additional profiler traces\.

### E\.1Selection ofKKandγ\\gamma

Periodic Row\-wise Muon introduces two primary hyperparameters: the refresh periodKK, which controls the frequency of full NS5 spectral updates, and the RowNorm scaling factorγ\\gamma, which controls the relative update magnitude of the RowNorm branch on non\-refresh steps\. To avoid tuning these hyperparameters separately at each model scale, we selectKKandγ\\gammaonce in a smaller development setting and directly transfer the same configuration to all 1\.3B–15B models in the main experiments without additional scale\-specific tuning\.

Specifically, we perform the hyperparameter selection experiments using the 1\.3B model for 30k training steps on 2 H100 nodes, corresponding to 16 H100 GPUs in total, with a local batch size of 16\. We otherwise retain the same model, optimizer hyperparameters, learning rate schedule, data pipeline, and training recipe used in the main experiments\. We evaluateK∈\{2,3,4\}K\\in\\\{2,3,4\\\}andγ∈\{0\.10,0\.15,0\.25,0\.35\}\\gamma\\in\\\{0\.10,0\.15,0\.25,0\.35\\\}\. To reduce sensitivity to noise at any single checkpoint, we use the mean FD\-DINO over the 20k, 25k, and 30k checkpoints as the selection criterion\.

Figure[5](https://arxiv.org/html/2608.20818#A5.F5)summarizes the results\. ForK=3K=3cases,γ=0\.15\\gamma=0\.15achieves the lowest late\-stage mean FD\-DINO among the tested scales\. We therefore useγ=0\.15\\gamma=0\.15when comparing different refresh periods\. IncreasingKKfrom 2 to 3 reduces the number of NS5 refreshes by 33% while increasing the best mean FD\-DINO by only 2\.8%\. In contrast, increasingKKfrom 3 to 4 provides a further 25% reduction in NS5 refreshes but increases mean FD\-DINO by 8\.2%\. We therefore selectK=3K=3andγ=0\.15\\gamma=0\.15as the default configuration, which provides a favorable trade\-off between generation quality and spectral\-update systems cost\.

Importantly, this selection procedure does not retuneKKorγ\\gammaon the larger models\. The same configuration is transferred directly to all model scales in the main experiments, so the observed scaling behavior does not result from scale\-specific hyperparameter tuning\.

Figure 5:Selection ofKKandγ\\gamma\. \(a\) Late\-stage FD\-DINO, averaged over the 20k, 25k, and 30k checkpoints, across differentKKandγ\\gamma\. \(b\) The best mean FD\-DINO overγ\\gammaat each refresh period\. IncreasingKKfrom 2 to 3 reduces the number of NS5 refreshes by33%33\\%with only a2\.8%2\.8\\%increase in FD\-DINO, whereas increasingKKfrom 3 to 4 saves a further25%25\\%but increases FD\-DINO by8\.2%8\.2\\%\. We therefore selectK=3K=3andγ=0\.15\\gamma=0\.15\.
### E\.2Training\-Loss Trajectories Across Model Scales

Figure[6](https://arxiv.org/html/2608.20818#A5.F6)shows the complete training\-loss trajectories of AdamW, Muon, and Periodic Row\-wise Muon over 60k optimization steps for the 1\.3B, 4B, 9B, and 15B models\. These results complement the validation loss comparisons in the main text by providing a higher resolution view of the optimization dynamics throughout training\.

All three optimizers exhibit stable decreasing training loss across all model scales, with no evidence of training divergence\. Muon generally achieves lower training loss throughout the main training regime, while Periodic Row\-wise Muon closely tracks the Muon trajectory\. This indicates that replacing two out of every three NS5 updates with the substantially cheaper RowNorm update does not destabilize optimization\. As model size increases from 1\.3B to 15B, the overall training loss level also decreases, consistent with the validation loss scaling behavior observed in the main experiments\.

Because minibatch training loss exhibits substantial stochastic variation, we do not use individual training loss values as proxies for generation quality\. Generation performance is instead evaluated using FD\-DINO, FID, and the additional generation metrics reported in the main text and in Appendix[E\.4](https://arxiv.org/html/2608.20818#A5.SS4)\. The purpose of Figure[6](https://arxiv.org/html/2608.20818#A5.F6)is therefore to verify that Periodic Row\-wise Muon remains stable over the complete 60k\-step optimization trajectory rather than only at the sparse generation evaluation checkpoints\.

![Refer to caption](https://arxiv.org/html/2608.20818v1/all_training_curve.png)Figure 6:Training loss curves across model scales over 60,000 optimizer steps\.
### E\.3Active Time Efficiency over AdamW

Figure 7:Generation quality for AdamW and Periodic Row\-wise Muon\.Figure[7](https://arxiv.org/html/2608.20818#A5.F7)directly compares Periodic Row\-wise Muon with AdamW under active training time\. Across all model sizes, Periodic Row\-wise Muon achieves lower FD\-DINO throughout most of the overlapping time range, indicating better generation quality for a comparable training time budget\. It also consistently attains lower FD\-DINO at the end of training\. Thus, the advantage over AdamW holds in both training time efficiency and the final generation quality reached by the model\.

### E\.4Complete Checkpoint Evaluation Results

The main text reports the most important final checkpoint metrics and the best observed FD\-DINO to keep the presentation compact\. Here we provide the complete checkpoint\-level generation evaluations for all 4 model scales in Tables[7](https://arxiv.org/html/2608.20818#A5.T7)–[10](https://arxiv.org/html/2608.20818#A5.T10)\.

For each model, we evaluate checkpoints every 5k steps from 5k to 60k using the same evaluation protocol for AdamW, Muon, and Periodic Row\-wise Muon\. At every checkpoint, we report FD\-DINO, FID, MMD\-DINO, Precision, Recall, Coverage, Density, HPSv2, and both the arithmetic and geometric means of GenEval2\. These tables therefore complement the fidelity metrics emphasized in the main text with the complete evolution of diversity, alignment, and compositionality throughout training\.

The full trajectories show that both Muon and Periodic Row\-wise Muon achieve substantial improvements over AdamW at multiple stages of training, while the relative ordering of the two Muon variants exhibits some checkpoint\-level variation\. In particular, the checkpoint that optimizes one generation metric does not necessarily optimize the others\. We therefore report the complete metric profile at the final checkpoint in the main text and treat the best\-observed FD\-DINO separately, rather than constructing an artificial “best” model by selecting different checkpoints for different metrics\.

These trajectories also provide additional context for the generation quality frontiers in the main text\. Different optimizers enter their best quality regimes at different points in training, and Periodic Row\-wise Muon typically reaches a generation quality regime comparable to vanilla Muon with substantially less normalized active training time\.

### E\.5Additional Profiler Traces Across Model Scales

The main text uses the 15B model to illustrate the systems behavior of Periodic Row\-wise Muon\. Figure[8](https://arxiv.org/html/2608.20818#A5.F8)provides the corresponding profiler traces for the 1\.3B, 4B, and 9B models, allowing us to verify whether the same execution pattern persists across model scales\.

All profiler traces use the same global time normalization as the systems measurements in the main text, meaning that the mean step time of the 1\.3B AdamW run is defined as one time unit\. The traces separately visualize the non\-optimizer phase, optimizer\-specific CPU phases, GPU computation, and communication activity for vanilla Muon and Periodic Row\-wise Muon\.

A consistent execution pattern is observed at all three scales\. Vanilla Muon performs a full NS5 update at every optimization step, resulting in a long spectral computation phase together with the corresponding optimizer communication\. Periodic Row\-wise Muon instead retains only one refresh step perK=3K=3period, while the two intervening steps use the much shorter RowNorm path\. On refresh steps, bucketed all\-gather overlaps communication for upcoming matrices with NS5 computation on matrices that are already available\. On RowNorm steps, the much smaller communication of norm statistics is similarly overlapped with local computation\.

Over the three consecutive steps shown in Figure[8](https://arxiv.org/html/2608.20818#A5.F8), the normalized time decreases from 5\.59 for vanilla Muon to 4\.20 for Periodic Row\-wise Muon on the 1\.3B model, from 7\.26 to 5\.68 on the 4B model, and from 12\.47 to 10\.26 on the 9B model\. The 15B trace presented in the main text exhibits the same qualitative behavior\. The consistency of these profiler traces across model scales indicates that the systems gains do not arise from a single model shape, but from the combined effects of periodic spectral refresh, sharded RowNorm, and communication–computation overlap\.

Table 7:All checkpoint evaluation metrics for the 1\.3B model\.OptimizerStepFidelityDiversityAlignmentCompositionalityFD\-DINO↓\\downarrowFID↓\\downarrowMMD\-DINO↓\\downarrowPrecision↑\\uparrowRecall↑\\uparrowCoverage↑\\uparrowDensity↑\\uparrowHPSv2↑\\uparrowGenEval2 AM↑\\uparrowGenEval2 GM↑\\uparrowAdamW5k266\.40266\.409\.949\.940\.2580\.2580\.8610\.8610\.3610\.3610\.6570\.6570\.9000\.90015\.7915\.7917\.1917\.191\.351\.3510k125\.51125\.517\.577\.570\.0940\.0940\.9080\.9080\.6810\.6810\.8420\.8420\.9470\.94715\.9315\.9322\.2422\.242\.772\.7715k82\.9382\.936\.626\.620\.0510\.0510\.9150\.9150\.7890\.7890\.8800\.8800\.9760\.97618\.0918\.0928\.5328\.534\.004\.0020k86\.8486\.847\.947\.940\.0550\.0550\.9250\.9250\.8120\.8120\.8900\.8901\.1071\.10717\.7117\.7129\.8329\.834\.194\.1925k66\.3066\.306\.586\.580\.0370\.0370\.9290\.9290\.8490\.8490\.9210\.9210\.9990\.99917\.5517\.5530\.9630\.964\.754\.7530k60\.3460\.346\.896\.890\.0340\.0340\.9410\.9410\.8590\.8590\.9290\.9291\.0651\.06519\.5619\.5636\.9236\.926\.716\.7135k53\.4353\.436\.336\.330\.0290\.0290\.9380\.9380\.8740\.8740\.9340\.9341\.0191\.01919\.9219\.9239\.3739\.378\.618\.6140k52\.6052\.606\.306\.300\.0300\.0300\.9470\.9470\.8790\.8790\.9470\.9471\.0181\.01819\.8219\.8237\.6037\.607\.167\.1645k53\.9253\.926\.536\.530\.0300\.0300\.9430\.9430\.8810\.8810\.9400\.9400\.9980\.99819\.7219\.7237\.9737\.977\.577\.5750k51\.5651\.566\.686\.680\.0280\.0280\.9470\.9470\.8870\.8870\.9400\.9401\.0191\.01920\.6120\.6138\.1638\.168\.018\.0155k53\.9753\.976\.616\.610\.0310\.0310\.9410\.9410\.8860\.8860\.9410\.9411\.0181\.01820\.4620\.4637\.6437\.646\.016\.0160k53\.9353\.936\.716\.710\.0310\.0310\.9410\.9410\.8870\.8870\.9400\.9400\.9660\.96620\.5320\.5338\.6638\.667\.107\.10Muon5k152\.03152\.037\.687\.680\.1260\.1260\.8820\.8820\.6330\.6330\.7780\.7780\.8820\.88216\.6416\.6420\.2820\.282\.232\.2310k73\.3573\.356\.266\.260\.0400\.0400\.9210\.9210\.8140\.8140\.8890\.8890\.9390\.93918\.7918\.7930\.5730\.574\.824\.8215k51\.5651\.565\.735\.730\.0240\.0240\.9430\.9430\.8660\.8660\.9200\.9200\.9820\.98219\.6919\.6938\.3338\.337\.457\.4520k48\.0648\.066\.166\.160\.0210\.0210\.9480\.9480\.8700\.8700\.9300\.9300\.9930\.99319\.7919\.7943\.6343\.639\.649\.6425k46\.1246\.126\.506\.500\.0220\.0220\.9470\.9470\.8840\.8840\.9320\.9320\.9800\.98020\.4320\.4340\.0040\.007\.657\.6530k52\.8652\.866\.846\.840\.0290\.0290\.9420\.9420\.8850\.8850\.9380\.9380\.9750\.97520\.6620\.6645\.2045\.208\.058\.0535k46\.6546\.656\.266\.260\.0220\.0220\.9440\.9440\.8960\.8960\.9430\.9430\.9910\.99121\.1421\.1446\.5746\.5710\.7710\.7740k41\.7341\.736\.016\.010\.0210\.0210\.9490\.9490\.9010\.9010\.9470\.9470\.9840\.98420\.9120\.9146\.7346\.7311\.3611\.3645k48\.2648\.266\.566\.560\.0270\.0270\.9430\.9430\.8970\.8970\.9450\.9450\.9910\.99120\.2920\.2936\.2736\.275\.455\.4550k50\.4150\.416\.746\.740\.0300\.0300\.9450\.9450\.8920\.8920\.9540\.9540\.9940\.99420\.4120\.4145\.3245\.328\.988\.9855k47\.9247\.926\.656\.650\.0280\.0280\.9450\.9450\.9000\.9000\.9500\.9500\.9850\.98520\.8120\.8144\.9144\.919\.229\.2260k46\.0846\.086\.546\.540\.0270\.0270\.9490\.9490\.9070\.9070\.9490\.9490\.9920\.99220\.8420\.8444\.7244\.729\.389\.38Periodic Row\-wiseMuon5k172\.46172\.468\.238\.230\.1450\.1450\.8850\.8850\.5690\.5690\.7660\.7660\.9210\.92116\.4516\.4519\.4619\.461\.671\.6710k82\.2982\.296\.746\.740\.0490\.0490\.9230\.9230\.8080\.8080\.8800\.8800\.9180\.91817\.5717\.5728\.5028\.504\.804\.8015k58\.9658\.966\.066\.060\.0280\.0280\.9280\.9280\.8580\.8580\.9160\.9160\.9690\.96918\.4518\.4536\.4936\.496\.036\.0320k67\.2267\.226\.976\.970\.0390\.0390\.9390\.9390\.8560\.8560\.9160\.9161\.0001\.00018\.4618\.4634\.7434\.745\.835\.8325k47\.3747\.376\.186\.180\.0210\.0210\.9380\.9380\.8760\.8760\.9380\.9380\.9970\.99719\.3619\.3636\.9536\.957\.247\.2430k44\.7744\.776\.116\.110\.0210\.0210\.9400\.9400\.8910\.8910\.9320\.9320\.9360\.93621\.1021\.1042\.7342\.738\.928\.9235k41\.9141\.915\.535\.530\.0190\.0190\.9440\.9440\.9020\.9020\.9440\.9440\.9810\.98120\.7320\.7347\.7147\.7110\.5310\.5340k47\.2947\.296\.636\.630\.0280\.0280\.9460\.9460\.8870\.8870\.9380\.9380\.9710\.97121\.4021\.4045\.6845\.6811\.3611\.3645k49\.7749\.776\.376\.370\.0300\.0300\.9470\.9470\.8940\.8940\.9440\.9440\.9810\.98120\.1820\.1845\.9745\.9710\.2110\.2150k45\.0145\.016\.136\.130\.0260\.0260\.9510\.9510\.8930\.8930\.9490\.9490\.9960\.99620\.3120\.3147\.4047\.4011\.2511\.2555k45\.7545\.756\.406\.400\.0270\.0270\.9490\.9490\.8960\.8960\.9510\.9511\.0051\.00520\.6920\.6947\.9347\.9311\.7111\.7160k45\.6545\.656\.296\.290\.0270\.0270\.9520\.9520\.9060\.9060\.9490\.9490\.9760\.97620\.3620\.3645\.8945\.899\.759\.75

Table 8:All checkpoint evaluation metrics for the 4B model\.OptimizerStepFidelityDiversityAlignmentCompositionalityFD\-DINO↓\\downarrowFID↓\\downarrowMMD\-DINO↓\\downarrowPrecision↑\\uparrowRecall↑\\uparrowCoverage↑\\uparrowDensity↑\\uparrowHPSv2↑\\uparrowGenEval2 AM↑\\uparrowGenEval2 GM↑\\uparrowAdamW5k223\.73223\.739\.179\.170\.1990\.1990\.8700\.8700\.4530\.4530\.7000\.7000\.9010\.90115\.9015\.9018\.6218\.621\.781\.7810k101\.76101\.768\.428\.420\.0660\.0660\.9080\.9080\.7460\.7460\.8570\.8570\.9500\.95018\.1718\.1722\.0122\.012\.262\.2615k72\.1672\.167\.677\.670\.0360\.0360\.9220\.9220\.8340\.8340\.8850\.8850\.9580\.95818\.3118\.3134\.1834\.186\.516\.5120k51\.8751\.876\.876\.870\.0230\.0230\.9410\.9410\.8660\.8660\.9260\.9260\.9980\.99820\.7320\.7337\.0937\.095\.765\.7625k50\.9450\.946\.396\.390\.0230\.0230\.9380\.9380\.8790\.8790\.9300\.9301\.0071\.00719\.3619\.3632\.8532\.855\.715\.7130k48\.7148\.716\.486\.480\.0220\.0220\.9400\.9400\.8880\.8880\.9360\.9361\.0261\.02620\.2420\.2438\.0938\.097\.747\.7435k47\.2847\.286\.806\.800\.0240\.0240\.9460\.9460\.8880\.8880\.9440\.9441\.0141\.01421\.6021\.6040\.0140\.017\.507\.5040k56\.9756\.977\.907\.900\.0290\.0290\.9330\.9330\.8990\.8990\.9370\.9370\.9720\.97220\.6320\.6341\.7441\.748\.898\.8945k61\.6161\.618\.728\.720\.0330\.0330\.9440\.9440\.8970\.8970\.9350\.9351\.0201\.02021\.3121\.3145\.9045\.9010\.6810\.6850k58\.0458\.048\.288\.280\.0300\.0300\.9390\.9390\.8990\.8990\.9410\.9410\.9940\.99420\.7220\.7244\.2644\.269\.729\.7255k57\.3557\.358\.018\.010\.0300\.0300\.9370\.9370\.9020\.9020\.9460\.9460\.9820\.98220\.8020\.8044\.0344\.039\.709\.7060k51\.3951\.397\.357\.350\.0270\.0270\.9370\.9370\.8990\.8990\.9390\.9390\.9900\.99021\.4021\.4045\.7045\.709\.139\.13Muon5k149\.32149\.327\.827\.820\.1190\.1190\.8880\.8880\.6310\.6310\.8040\.8040\.8940\.89416\.7316\.7319\.5419\.542\.312\.3110k65\.8865\.886\.646\.640\.0360\.0360\.9290\.9290\.8330\.8330\.9040\.9040\.9510\.95118\.2718\.2727\.7327\.734\.124\.1215k56\.0856\.086\.506\.500\.0300\.0300\.9390\.9390\.8690\.8690\.9290\.9290\.9740\.97420\.3920\.3940\.9940\.997\.987\.9820k40\.1540\.156\.256\.250\.0150\.0150\.9410\.9410\.9030\.9030\.9350\.9350\.9520\.95220\.8120\.8141\.1441\.148\.468\.4625k39\.2739\.275\.325\.320\.0180\.0180\.9450\.9450\.9020\.9020\.9430\.9430\.9670\.96721\.0021\.0046\.2046\.2010\.4110\.4130k40\.5040\.505\.945\.940\.0180\.0180\.9410\.9410\.9040\.9040\.9380\.9380\.9590\.95920\.8120\.8146\.5346\.539\.529\.5235k41\.9641\.965\.665\.660\.0200\.0200\.9460\.9460\.9050\.9050\.9470\.9470\.9830\.98319\.9119\.9143\.4543\.457\.417\.4140k55\.8255\.826\.996\.990\.0330\.0330\.9350\.9350\.9100\.9100\.9360\.9360\.9480\.94818\.9418\.9439\.7739\.776\.366\.3645k44\.3944\.395\.865\.860\.0250\.0250\.9400\.9400\.9130\.9130\.9460\.9460\.9630\.96320\.7920\.7946\.7546\.758\.188\.1850k46\.2446\.246\.136\.130\.0270\.0270\.9460\.9460\.9140\.9140\.9460\.9460\.9670\.96721\.1321\.1346\.4246\.428\.838\.8355k42\.0942\.096\.146\.140\.0220\.0220\.9460\.9460\.9120\.9120\.9410\.9410\.9660\.96621\.1321\.1350\.4050\.4010\.1910\.1960k41\.6241\.626\.086\.080\.0240\.0240\.9390\.9390\.9140\.9140\.9430\.9430\.9640\.96420\.6620\.6646\.5846\.589\.069\.06Periodic Row\-wiseMuon5k159\.06159\.069\.049\.040\.1250\.1250\.8920\.8920\.5940\.5940\.7990\.7990\.9690\.96915\.6115\.6118\.6818\.682\.052\.0510k72\.5972\.597\.207\.200\.0400\.0400\.9290\.9290\.8200\.8200\.8880\.8880\.9760\.97618\.9818\.9828\.4228\.423\.853\.8515k52\.1952\.196\.356\.350\.0220\.0220\.9290\.9290\.8760\.8760\.9170\.9170\.9460\.94618\.2518\.2535\.5235\.526\.206\.2020k39\.4239\.425\.615\.610\.0150\.0150\.9490\.9490\.8990\.8990\.9480\.9480\.9840\.98421\.0521\.0542\.9842\.988\.168\.1625k44\.5844\.586\.276\.270\.0210\.0210\.9350\.9350\.9010\.9010\.9310\.9310\.9370\.93720\.8720\.8738\.6338\.637\.197\.1930k39\.8039\.806\.156\.150\.0180\.0180\.9510\.9510\.9090\.9090\.9430\.9430\.9570\.95720\.2320\.2339\.5339\.536\.646\.6435k41\.9141\.916\.326\.320\.0200\.0200\.9410\.9410\.9100\.9100\.9410\.9410\.9650\.96520\.8320\.8340\.8740\.876\.956\.9540k43\.9343\.936\.646\.640\.0200\.0200\.9380\.9380\.9090\.9090\.9420\.9420\.9780\.97820\.4720\.4743\.9043\.908\.098\.0945k62\.8962\.898\.688\.680\.0320\.0320\.9330\.9330\.8980\.8980\.9420\.9421\.0921\.09219\.5219\.5243\.2343\.238\.138\.1350k52\.3652\.367\.307\.300\.0300\.0300\.9360\.9360\.9160\.9160\.9420\.9420\.9390\.93920\.5420\.5449\.3249\.3210\.8610\.8655k43\.7043\.706\.426\.420\.0230\.0230\.9410\.9410\.9110\.9110\.9500\.9500\.9860\.98621\.0021\.0049\.4149\.4111\.4411\.4460k42\.2542\.256\.356\.350\.0220\.0220\.9410\.9410\.9060\.9060\.9520\.9520\.9840\.98421\.3421\.3449\.3849\.3810\.5810\.58

Table 9:All checkpoint evaluation metrics for the 9B model\.OptimizerStepFidelityDiversityAlignmentCompositionalityFD\-DINO↓\\downarrowFID↓\\downarrowMMD\-DINO↓\\downarrowPrecision↑\\uparrowRecall↑\\uparrowCoverage↑\\uparrowDensity↑\\uparrowHPSv2↑\\uparrowGenEval2 AM↑\\uparrowGenEval2 GM↑\\uparrowAdamW5k184\.83184\.838\.378\.370\.1500\.1500\.8830\.8830\.5310\.5310\.7570\.7570\.9430\.94316\.0816\.0819\.1819\.181\.751\.7510k78\.0578\.056\.056\.050\.0430\.0430\.9240\.9240\.7940\.7940\.8900\.8900\.9910\.99119\.3219\.3227\.0127\.013\.413\.4115k59\.3059\.307\.467\.460\.0280\.0280\.9400\.9400\.8570\.8570\.9160\.9160\.9660\.96620\.6520\.6535\.7635\.766\.146\.1420k43\.2043\.206\.276\.270\.0160\.0160\.9440\.9440\.8810\.8810\.9330\.9330\.9670\.96720\.5820\.5841\.2841\.288\.238\.2325k47\.6747\.676\.436\.430\.0190\.0190\.9440\.9440\.8790\.8790\.9400\.9401\.0201\.02020\.2120\.2140\.9640\.968\.468\.4630k44\.4244\.426\.466\.460\.0210\.0210\.9450\.9450\.8950\.8950\.9460\.9461\.0121\.01220\.3820\.3840\.9340\.938\.098\.0935k44\.7244\.726\.586\.580\.0220\.0220\.9420\.9420\.9020\.9020\.9420\.9420\.9790\.97921\.7021\.7047\.6747\.6710\.1610\.1640k42\.2142\.216\.446\.440\.0210\.0210\.9440\.9440\.9050\.9050\.9490\.9490\.9750\.97520\.8220\.8244\.3344\.339\.959\.9545k48\.4348\.437\.127\.120\.0250\.0250\.9350\.9350\.9020\.9020\.9410\.9410\.9700\.97021\.1821\.1844\.5144\.519\.409\.4050k45\.5445\.546\.776\.770\.0250\.0250\.9420\.9420\.9050\.9050\.9420\.9420\.9580\.95820\.5420\.5442\.4542\.458\.898\.8955k41\.0141\.016\.186\.180\.0210\.0210\.9460\.9460\.9110\.9110\.9420\.9420\.9570\.95721\.1821\.1845\.6245\.6210\.3310\.3360k41\.2641\.266\.196\.190\.0220\.0220\.9460\.9460\.9040\.9040\.9500\.9500\.9870\.98721\.3921\.3945\.9345\.939\.669\.66Muon5k117\.21117\.216\.486\.480\.0850\.0850\.9050\.9050\.7060\.7060\.8360\.8360\.9370\.93716\.8216\.8222\.7622\.762\.212\.2110k47\.6447\.645\.515\.510\.0190\.0190\.9420\.9420\.8700\.8700\.9240\.9240\.9810\.98120\.3920\.3937\.7037\.708\.338\.3315k40\.0040\.005\.775\.770\.0170\.0170\.9370\.9370\.8970\.8970\.9380\.9380\.9580\.95821\.9721\.9746\.5246\.5211\.8211\.8220k38\.5738\.576\.466\.460\.0180\.0180\.9470\.9470\.9030\.9030\.9420\.9420\.9520\.95221\.9321\.9353\.0053\.0014\.1714\.1725k38\.9738\.975\.955\.950\.0180\.0180\.9430\.9430\.9110\.9110\.9430\.9430\.9740\.97420\.4920\.4949\.8149\.8112\.0212\.0230k39\.0739\.076\.136\.130\.0210\.0210\.9450\.9450\.9110\.9110\.9450\.9450\.9520\.95221\.3821\.3850\.1650\.1610\.0410\.0435k39\.3039\.305\.995\.990\.0200\.0200\.9500\.9500\.9100\.9100\.9410\.9410\.9660\.96622\.2722\.2754\.6554\.6514\.1414\.1440k35\.9235\.925\.795\.790\.0190\.0190\.9440\.9440\.9150\.9150\.9440\.9440\.9510\.95122\.0322\.0347\.4947\.4910\.3010\.3045k36\.2936\.295\.785\.780\.0200\.0200\.9500\.9500\.9180\.9180\.9490\.9490\.9670\.96721\.9821\.9852\.2152\.2111\.9211\.9250k34\.5334\.535\.715\.710\.0200\.0200\.9480\.9480\.9220\.9220\.9470\.9470\.9470\.94722\.8722\.8753\.9853\.9812\.8612\.8655k34\.4134\.415\.645\.640\.0200\.0200\.9500\.9500\.9200\.9200\.9490\.9490\.9640\.96422\.3622\.3651\.5551\.5511\.3411\.3460k33\.9733\.975\.615\.610\.0200\.0200\.9520\.9520\.9270\.9270\.9530\.9530\.9600\.96022\.6022\.6051\.7751\.7711\.3511\.35Periodic Row\-wiseMuon5k143\.77143\.778\.718\.710\.1080\.1080\.9030\.9030\.6300\.6300\.8240\.8240\.9600\.96015\.8315\.8319\.3819\.382\.292\.2910k60\.2960\.296\.706\.700\.0280\.0280\.9230\.9230\.8500\.8500\.9140\.9140\.9460\.94618\.5418\.5429\.5329\.534\.364\.3615k48\.1748\.176\.256\.250\.0210\.0210\.9370\.9370\.8840\.8840\.9310\.9310\.9450\.94520\.4620\.4640\.0940\.098\.108\.1020k38\.1438\.145\.725\.720\.0140\.0140\.9470\.9470\.8960\.8960\.9350\.9350\.9460\.94622\.1222\.1245\.3945\.399\.829\.8225k32\.4432\.445\.635\.630\.0110\.0110\.9430\.9430\.9070\.9070\.9470\.9470\.9540\.95421\.8321\.8351\.2551\.2511\.7511\.7530k38\.8138\.816\.176\.170\.0170\.0170\.9500\.9500\.9040\.9040\.9450\.9450\.9700\.97020\.9520\.9549\.4649\.4610\.2810\.2835k34\.5734\.575\.635\.630\.0160\.0160\.9520\.9520\.9180\.9180\.9480\.9480\.9790\.97922\.0822\.0850\.7350\.7310\.8910\.8940k37\.4737\.475\.895\.890\.0190\.0190\.9500\.9500\.9110\.9110\.9510\.9510\.9640\.96422\.3422\.3454\.9754\.9713\.5013\.5045k36\.8636\.866\.096\.090\.0200\.0200\.9460\.9460\.9160\.9160\.9460\.9460\.9650\.96522\.8222\.8257\.3557\.3515\.3515\.3550k35\.3735\.375\.855\.850\.0190\.0190\.9520\.9520\.9120\.9120\.9520\.9520\.9720\.97222\.5722\.5756\.1656\.1613\.4413\.4455k38\.1938\.196\.396\.390\.0200\.0200\.9430\.9430\.9210\.9210\.9540\.9540\.9670\.96722\.4722\.4755\.5255\.5212\.1812\.1860k36\.7036\.706\.136\.130\.0190\.0190\.9490\.9490\.9130\.9130\.9510\.9510\.9790\.97922\.4422\.4457\.3357\.3315\.9715\.97

Table 10:All checkpoint evaluation metrics for the 15B model\.OptimizerStepFidelityDiversityAlignmentCompositionalityFD\-DINO↓\\downarrowFID↓\\downarrowMMD\-DINO↓\\downarrowPrecision↑\\uparrowRecall↑\\uparrowCoverage↑\\uparrowDensity↑\\uparrowHPSv2↑\\uparrowGenEval2 AM↑\\uparrowGenEval2 GM↑\\uparrowAdamW5k175\.31175\.318\.928\.920\.1360\.1360\.8790\.8790\.5610\.5610\.7530\.7530\.9030\.90316\.0316\.0316\.8116\.811\.501\.5010k74\.8474\.847\.707\.700\.0350\.0350\.9270\.9270\.8050\.8050\.8950\.8950\.9700\.97019\.0119\.0125\.7525\.753\.083\.0815k53\.2153\.216\.426\.420\.0230\.0230\.9390\.9390\.8560\.8560\.9200\.9200\.9700\.97020\.8220\.8232\.8632\.865\.355\.3520k56\.2156\.217\.037\.030\.0280\.0280\.9350\.9350\.8700\.8700\.9250\.9250\.9870\.98718\.6618\.6635\.5835\.585\.705\.7025k40\.7540\.755\.865\.860\.0170\.0170\.9430\.9430\.8940\.8940\.9420\.9420\.9850\.98521\.9921\.9940\.2740\.277\.447\.4430k39\.5939\.596\.156\.150\.0170\.0170\.9470\.9470\.8900\.8900\.9420\.9420\.9990\.99921\.9821\.9842\.0342\.037\.877\.8735k38\.4638\.465\.865\.860\.0170\.0170\.9470\.9470\.8940\.8940\.9490\.9491\.0071\.00721\.3021\.3045\.3145\.319\.439\.4340k41\.3341\.336\.376\.370\.0220\.0220\.9320\.9320\.9100\.9100\.9400\.9400\.9370\.93722\.1322\.1347\.1047\.1011\.1111\.1145k38\.1738\.176\.006\.000\.0180\.0180\.9490\.9490\.9060\.9060\.9520\.9521\.0061\.00621\.7221\.7246\.7746\.778\.968\.9650k40\.4840\.486\.256\.250\.0210\.0210\.9500\.9500\.9140\.9140\.9500\.9500\.9810\.98122\.1822\.1847\.4947\.498\.778\.7755k40\.1340\.136\.286\.280\.0210\.0210\.9510\.9510\.9050\.9050\.9540\.9540\.9750\.97522\.2122\.2147\.2147\.219\.559\.5560k40\.2340\.236\.246\.240\.0220\.0220\.9450\.9450\.9090\.9090\.9470\.9470\.9640\.96421\.9921\.9947\.3447\.349\.189\.18Muon5k119\.42119\.427\.837\.830\.0880\.0880\.9030\.9030\.6940\.6940\.8460\.8460\.9230\.92317\.1617\.1621\.9621\.962\.262\.2610k54\.4954\.496\.356\.350\.0250\.0250\.9360\.9360\.8650\.8650\.9280\.9280\.9650\.96520\.2320\.2333\.8933\.895\.725\.7215k37\.8937\.895\.825\.820\.0150\.0150\.9420\.9420\.8980\.8980\.9410\.9410\.9510\.95122\.1022\.1043\.3543\.358\.738\.7320k37\.2437\.245\.415\.410\.0160\.0160\.9440\.9440\.9090\.9090\.9400\.9400\.9470\.94721\.9621\.9646\.3846\.389\.819\.8125k33\.4833\.485\.155\.150\.0140\.0140\.9470\.9470\.9140\.9140\.9450\.9450\.9520\.95222\.9222\.9254\.3054\.3014\.2114\.2130k37\.1337\.135\.845\.840\.0180\.0180\.9420\.9420\.9150\.9150\.9470\.9470\.9370\.93720\.9320\.9352\.7252\.7211\.4411\.4435k35\.9235\.925\.925\.920\.0180\.0180\.9430\.9430\.9170\.9170\.9500\.9500\.9610\.96120\.8720\.8752\.6752\.6710\.6710\.6740k33\.5533\.555\.375\.370\.0180\.0180\.9520\.9520\.9240\.9240\.9540\.9540\.9520\.95222\.8822\.8854\.6654\.6612\.1512\.1545k33\.2533\.255\.195\.190\.0180\.0180\.9450\.9450\.9210\.9210\.9500\.9500\.9480\.94822\.5322\.5357\.5057\.5013\.7613\.7650k34\.5434\.545\.745\.740\.0200\.0200\.9410\.9410\.9250\.9250\.9530\.9530\.9340\.93422\.6722\.6758\.1858\.1814\.1114\.1155k33\.4733\.475\.625\.620\.0190\.0190\.9460\.9460\.9210\.9210\.9490\.9490\.9490\.94922\.4322\.4358\.5358\.5313\.8413\.8460k33\.5133\.515\.575\.570\.0190\.0190\.9450\.9450\.9300\.9300\.9500\.9500\.9370\.93722\.3722\.3757\.9357\.9314\.5514\.55Periodic Row\-wiseMuon5k137\.51137\.517\.687\.680\.1010\.1010\.8990\.8990\.6410\.6410\.8170\.8170\.9170\.91717\.4917\.4919\.1519\.152\.612\.6110k56\.1856\.186\.376\.370\.0220\.0220\.9360\.9360\.8510\.8510\.9180\.9180\.9710\.97119\.3719\.3726\.8726\.873\.443\.4415k45\.1445\.146\.646\.640\.0190\.0190\.9390\.9390\.8850\.8850\.9380\.9380\.9560\.95621\.8621\.8634\.7634\.765\.735\.7320k40\.1340\.136\.346\.340\.0170\.0170\.9410\.9410\.9010\.9010\.9390\.9390\.9630\.96321\.8121\.8139\.1239\.126\.936\.9325k32\.3532\.355\.615\.610\.0120\.0120\.9490\.9490\.9120\.9120\.9470\.9470\.9660\.96622\.9022\.9049\.6849\.6811\.7411\.7430k35\.1235\.125\.585\.580\.0160\.0160\.9490\.9490\.9070\.9070\.9530\.9530\.9940\.99420\.5920\.5947\.5547\.5510\.7010\.7035k36\.9136\.916\.196\.190\.0170\.0170\.9440\.9440\.9140\.9140\.9430\.9430\.9590\.95919\.9419\.9448\.7148\.719\.459\.4540k38\.7238\.726\.026\.020\.0210\.0210\.9430\.9430\.9160\.9160\.9450\.9450\.9420\.94220\.4320\.4345\.8945\.899\.199\.1945k34\.3734\.375\.605\.600\.0180\.0180\.9410\.9410\.9190\.9190\.9510\.9510\.9540\.95421\.0821\.0853\.1053\.1012\.0812\.0850k52\.2552\.257\.867\.860\.0390\.0390\.9440\.9440\.9160\.9160\.9410\.9410\.9230\.92319\.6419\.6444\.6444\.648\.988\.9855k40\.2540\.256\.386\.380\.0260\.0260\.9420\.9420\.9150\.9150\.9490\.9490\.9450\.94520\.8020\.8047\.8047\.8010\.1210\.1260k33\.9933\.995\.635\.630\.0200\.0200\.9480\.9480\.9250\.9250\.9550\.9550\.9670\.96721\.3521\.3552\.2652\.2611\.9711\.97

![Refer to caption](https://arxiv.org/html/2608.20818v1/profiler_trace_1p3B_muon_periodic_xlarge_font_wrapped_overlap_no_model_label.png)\(a\)1\.3B model\.
![Refer to caption](https://arxiv.org/html/2608.20818v1/profiler_trace_4B_muon_periodic_xlarge_font_wrapped_overlap_no_model_label.png)\(b\)4B model\.
![Refer to caption](https://arxiv.org/html/2608.20818v1/profiler_trace_9B_muon_periodic_xlarge_font_wrapped_overlap_no_model_label.png)\(c\)9B model\.

Figure 8:Simplified profiler traces for the 1\.3B, 4B, and 9B models\. The corresponding 15B result is shown in Figure[4](https://arxiv.org/html/2608.20818#S4.F4)\.