MUON-RT：复用教师模型的轨迹，实现用于大语言模型自蒸馏的无动量 Muon

# Muon-RT: Reusing the Teacher’s Trajectory for Momentum-Free Muon for LLM Self-Distillation
Source: [https://arxiv.org/html/2610.41104](https://arxiv.org/html/2610.41104)
Vinay Kumar VermaAffiliation:IIT KanpurAakansha MishraAffiliation:Amazon

###### Abstract

Self\-distillation methods for large language models with an exponential moving average \(EMA\) teacher already retain a history of student updates\. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory\. We propose Muon with Reused Teacher \(Muon\-RT\), which reuses the teacher–student displacement to eliminate momentum buffers for hidden weight matrices\. We show that this displacement encodes an exponentially weighted history of applied updates and smooths temporally uncorrelated noise under a local model\. Muon\-RT removes its radial component, normalizes it spectrally, and mixes it with the current gradient after Newton–Schulz \(NS\) orthogonalization\. Under explicit regularity and alignment assumptions, we establish linear contraction of an expected objective\-gap bound toward a residual error level\. With Self\-Distillation Fine\-Tuning \(SDFT\) on Qwen2\.5\-7B\-Instruct, Muon\-RT improves average accuracy over AdamW by 2\.6 points with 42% less persistent memory, and over momentum Muon by 5\.4 points with 27% less persistent memory\. With Self\-Distillation Policy Optimization \(SDPO\) on Olmo\-3\-7B\-Instruct, it improves average accuracy over AdamW by 2\.0 points\.

## 1Introduction

Figure 1:Muon\-RT with SDFT on Qwen2\.5\-7B\-Instruct\. \(a\) Average accuracy over 2 datasets against memory per GPU\. \(b\) Gain over AdamW in points\.Self\-distillation\([Shenfeld et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib4);[Hübotter et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib9)\)provides an effective approach to adapting large language models \(LLMs\) by using privileged information to elicit supervision from the model’s existing capabilities\. The student learns from its own generated responses by matching a teacher’s predictions conditioned on additional context\. This formulation supports skill acquisition and policy improvement while helping preserve previously learned capabilities\. Such retention is an advantage over conventional fine\-tuning, which can cause forgetting\([Biderman et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib19);[Zheng et al\., 2025a](https://arxiv.org/html/2610.41104#bib.bib25)\), a recognized challenge when updating LLMs\([Shi et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib38);[Zheng et al\., 2025b](https://arxiv.org/html/2610.41104#bib.bib39)\)\. Within this family, Self\-Distillation Fine\-Tuning \(SDFT\) uses demonstrations to guide the acquisition of new skills\([Shenfeld et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib4)\), while Self\-Distillation Policy Optimization \(SDPO\) extends the approach to on\-policy post\-training using feedback from successful attempts\([Hübotter et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib9)\)\. In this settings, the teacher is maintained as an exponential moving average \(EMA\) of student weights, following approaches developed in semi\-supervised and self\-supervised learning\([Tarvainen and Valpola, 2017](https://arxiv.org/html/2610.41104#bib.bib1);[Grill et al\., 2020](https://arxiv.org/html/2610.41104#bib.bib2);[Caron et al\., 2021](https://arxiv.org/html/2610.41104#bib.bib3)\)\. The teacher therefore evolves alongside the student, providing an averaged model whose predictions guide subsequent learning\.

Optimizing EMA\-based self\-distillation incurs substantial memory costs because training must retain student and teacher parameters alongside optimizer state\. Adam[Kingma and Ba \(2014\)](https://arxiv.org/html/2610.41104#bib.bib58)and AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.41104#bib.bib6)\)store two running moment estimates per parameter\. Muon\([Jordan et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib54)\)reduces this overhead by maintaining one momentum buffer per hidden weight matrix and approximately orthogonalizing update directions using Newton–Schulz iterations\. Muon and its variants have been used to train Kimi K2\([Bai et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib53)\)and GLM\-4\.5\([Team et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib63)\)\. Nevertheless, optimizing self\-distillation with Muon still requires three persistent states for each hidden matrix: student weights, teacher weights, and momentum\. AdamW requires four, including its two moment buffers\. For example Qwen2\.5\-7B\-Instruct, momentum Muon and AdamW require 103\.2 GB and 129\.0 GB of GPU memory for persistent states, respectively, compared with 75\.4 GB for Muon\-RT \(Figure[1](https://arxiv.org/html/2610.41104#S1.F1)\)\. Existing memory\-efficient optimizers address state storage through low\-rank projection, quantization, or shared statistics\([Zhao et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib7);[Li et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib15);[Zhang et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib13);[Zhu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib31)\)\. The teacher offers another opportunity: reusing history already maintained by self\-distillation to remove a separate optimizer buffer\. We therefore ask:

Can the teacher’s existing parameter trajectory replace Muon’s momentum buffer reducing persistent memory while improving task accuracy in self\-distillation?

Why could the teacher replace momentum?In SDFT and SDPO, the teacher is not fixed; its weights continuously follow the evolving student through an EMA\. Unlike momentum, which averages gradients, the teacher averages weights\. The momentum\-like information therefore lies in the*teacher–student displacement*, not the teacher itself\. As the student progresses, this displacement records an exponentially weighted history of applied updates \(Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1)\), smoothing noisy steps while preserving persistent directions \(Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2)\)\. Under gradient descent, this history is proportional to gradient momentum; under Muon, it captures previously orthogonalized updates\. As optimization settles, the displacement shrinks as the teacher catches up with the student\. Although this cannot recover AdamW’s second\-moment statistics, however, it can replace Muon’s momentum buffer\.

Even though this approach appears simple, reusing the teacher–student displacement in Muon is not straightforward\. First, Muon orthogonalizes its updates, so the displacement averages past optimizer outputs rather than raw gradients\. Using it directly as momentum would feed these outputs back into the optimizer\. Second, mixing the displacement with gradients before Newton–Schulz leaves its influence uncontrolled, as Newton–Schulz can amplify weak singular directions\. This distinction matters in self\-distillation, where the teacher and learning targets continuously evolve with the student\. While momentum averages gradients from outdated targets, the displacement captures the updates actually applied to the student, providing a different source of historical guidance\.

We therefore propose Muon with Reused Teacher \(Muon\-RT\), which replaces the dedicated momentum buffer with information from the existing EMA teacher\. For each hidden weight matrix, Muon\-RT orthogonalizes the current gradient without momentum and separately computes the teacher–student displacement\. It removes the component parallel to the student weights and scales the remainder by an estimate of its largest singular value\. The two directions are then combined after orthogonalization using a small fixed mixing coefficient\. This preserves current gradient information while adding historical guidance without changing the training objective\. Only one small vector per matrix is retained to warm\-start power iteration across successive optimization steps\. The method applies wherever a parameter\-aligned EMA teacher is already maintained, including the studied SDFT and SDPO settings and mean\-teacher training\([Tarvainen and Valpola, 2017](https://arxiv.org/html/2610.41104#bib.bib1)\)\. Independent teachers lacking this alignment fall outside its scope\. Our contributions are:

- •We establish that the teacher–student displacement stores a discounted history of applied updates, is proportional to gradient momentum under gradient descent, and reduces variance under a local model with a fixed direction and temporally uncorrelated noise\.
- •We introduce Muon\-RT, eliminating hidden\-matrix momentum buffers through normalized teacher–student displacement added after orthogonalization\. Under regularity, bounded objective drift, and alignment assumptions, we establish linear contraction of an expected objective\-gap bound toward a residual error level\.
- •With SDFT, Muon\-RT improves average accuracy over AdamW by 2\.6 points with 42% less persistent memory, and over momentum Muon by 5\.4 points with 27% less \(Figure[1](https://arxiv.org/html/2610.41104#S1.F1)\)\. With SDPO, it gains 2\.0 points over AdamW\. Muon\-RT ranks first among Muon variants across both hosts’ six settings\.

## 2Preliminaries

##### Self\-distillation with an EMA\-teacher:

LetSSandTTdenote the student and teacher LLMs, with policiesπS\\pi\_\{S\}andπT\\pi\_\{T\}parameterized byθS\\theta\_\{S\}andθT\\theta\_\{T\}, respectively\. Both share the same architecture and initialization\. For a promptxx, the student samples a completiony∼πS\(⋅∣x\)y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\. The teacher evaluates the same tokens using privileged contextcc: an expert demonstration in SDFT\([Shenfeld et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib4)\), or rollout feedback, potentially including a successful sibling response, in SDPO\([Hübotter et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib9)\)\. At token positionjj, their next\-token distributions are

pS,j=πS\(⋅∣x,y<j\),pT,j=πT\(⋅∣x,c,y<j\)\.p\_\{S,j\}=\\pi\_\{S\}\(\\cdot\\mid x,y\_\{<j\}\),\\quad p\_\{T,j\}=\\pi\_\{T\}\(\\cdot\\mid x,c,y\_\{<j\}\)\.\(1\)The student minimizes the token\-level distillation objective

ℒ⁡\(θS\)=𝔼x,y,c​\[∑j=1\|y\|𝒟⁡\(pS,j,sg⁡\[pT,j\]\)\],\\mathcal\{L\}\(\\theta\_\{S\}\)=\\mathbb\{E\}\_\{x,y,c\}\\left\[\\sum\_\{j=1\}^\{\|y\|\}\\mathcal\{D\}\\\!\\left\(p\_\{S,j\},\\operatorname\{sg\}\[p\_\{T,j\}\]\\right\)\\right\],\(2\)where the expectation uses student\-generated completions and their associated teacher contexts,sg\\operatorname\{sg\}stops teacher gradients, and𝒟\\mathcal\{D\}denotes the chosen divergence\. SDFT formulates reverse KL but favors forward KL empirically, while SDPO employs Jensen–Shannon or reverse KL\. To stabilize distillation, the EMA\-teacher is updated after each student optimization step:

θT←τ​θT\+\(1−τ\)​θS,τ∈\[0,1\)\.\\theta\_\{T\}\\leftarrow\\tau\\theta\_\{T\}\+\(1\-\\tau\)\\theta\_\{S\},\\qquad\\tau\\in\[0,1\)\.\(3\)Our method assumes only that the EMA\-teacher is explicitly maintained and parameter\-wise aligned with the student\.

##### Muon Optimizer:

Muon updates each student hidden weight matrix𝐖S∈ℝA×B\\mathbf\{W\}\_\{S\}\\in\\mathbb\{R\}^\{A\\times B\}, while embeddings, output heads, norms, and biases typically use AdamW\([Jordan et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib54);[Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.41104#bib.bib6);[Liu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib5)\)\. Given gradient𝐆=∇𝐖Sℒ​\(θS\)\\mathbf\{G\}=\\nabla\_\{\\mathbf\{W\}\_\{S\}\}\\mathcal\{L\}\(\\theta\_\{S\}\)and momentum coefficientμ\\mu, Muon maintains𝐌←μ​𝐌\+𝐆\\mathbf\{M\}\\leftarrow\\mu\\mathbf\{M\}\+\\mathbf\{G\}and orthogonalizes the Nesterov direction:

𝐎mom=NSK​\(𝐆\+μ​𝐌\)\.\\mathbf\{O\}^\{\\mathrm\{mom\}\}=\\mathrm\{NS\}\_\{K\}\(\\mathbf\{G\}\+\\mu\\mathbf\{M\}\)\.\(4\)Here,NSK\\mathrm\{NS\}\_\{K\}appliesKKquintic Newton–Schulz iterations from𝐗0=𝐙/\(‖𝐙‖F\+ε\)\\mathbf\{X\}\_\{0\}=\\mathbf\{Z\}/\(\\\|\\mathbf\{Z\}\\\|\_\{\\mathrm\{F\}\}\+\\varepsilon\), with𝐙=𝐆\+μ​𝐌\\mathbf\{Z\}=\\mathbf\{G\}\+\\mu\\mathbf\{M\}:

𝐗r\+1=\(κ1​𝐈\+κ2​𝐗r​𝐗r⊤\+κ3​\(𝐗r​𝐗r⊤\)2\)​𝐗r\.\\mathbf\{X\}\_\{r\+1\}=\\big\(\\kappa\_\{1\}\\mathbf\{I\}\+\\kappa\_\{2\}\\mathbf\{X\}\_\{r\}\\mathbf\{X\}\_\{r\}^\{\\top\}\+\\kappa\_\{3\}\(\\mathbf\{X\}\_\{r\}\\mathbf\{X\}\_\{r\}^\{\\top\}\)^\{2\}\\big\)\\mathbf\{X\}\_\{r\}\.\(5\)The fixed coefficients approximately equalize nonzero singular values, yielding an approximate polar factor\([Jordan et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib54)\)\. With decoupled weight decay, the update is

𝐖S←\(1−η​λ\)​𝐖S−η​cA,B​𝐎mom\.\\mathbf\{W\}\_\{S\}\\leftarrow\(1\-\\eta\\lambda\)\\mathbf\{W\}\_\{S\}\-\\eta c\_\{A,B\}\\mathbf\{O\}^\{\\mathrm\{mom\}\}\.\(6\)whereη\\etais the learning rate,λ\\lambdais weight decay, andcA,B=0\.2​max⁡\(A,B\)c\_\{A,B\}=0\.2\\sqrt\{\\max\(A,B\)\}approximately matches AdamW’s update RMS\([Liu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib5)\)\. Combined with the EMA\-teacher, Muon maintains 3 persistent states per hidden matrix: student weights𝐖S\\mathbf\{W\}\_\{S\}, teacher weights𝐖T\\mathbf\{W\}\_\{T\}, and momentum𝐌\\mathbf\{M\}\. We do not count the gradient𝐆\\mathbf\{G\}as a persistent state: it is recomputed at every step and freed after the update, and every optimizer needs it in the same way\. During a step, Muon therefore holds 4 matrix\-sized tensors, and between steps it keeps 3\.

## 3Momentum\-Free Muon Optimization

![Refer to caption](https://arxiv.org/html/2610.41104v1/method.png)Figure 2:Overview of Muon\-RT\. \(a\) Each hidden matrix maintains student weights𝐖S\\mathbf\{W\}\_\{S\}and EMA\-teacher weights𝐖T\\mathbf\{W\}\_\{T\}, eliminating Muon’s momentum buffer𝐌\\mathbf\{M\}\. \(b\) Newton–Schulz approximately orthogonalizes the gradient𝐆\\mathbf\{G\}into𝐎\\mathbf\{O\}\. The radial component of the teacher–student displacement𝐖T−𝐖S\\mathbf\{W\}\_\{T\}\-\\mathbf\{W\}\_\{S\}is removed to obtain𝐇\\mathbf\{H\}, which is spectrally normalized into𝐏\\mathbf\{P\}\. The update combines both directions as𝐔=\(1−α\)​𝐎\+α​𝐏\\mathbf\{U\}=\(1\-\\alpha\)\\mathbf\{O\}\+\\alpha\\mathbf\{P\}\. \(c\) The EMA\-teacher tracks the student’s parameter trajectory with a temporal lag\.We first characterize the information retained by the EMA\-teacher, derive the proposed update, analyze memory efficiency and sharded implementation, and present convergence guarantees\. Throughout, we consider a student hidden weight matrix𝐖S∈ℝA×B\\mathbf\{W\}\_\{S\}\\in\\mathbb\{R\}^\{A\\times B\}and its corresponding teacher matrix𝐖T\\mathbf\{W\}\_\{T\}\. Each matrix is processed independently\. Figure[2](https://arxiv.org/html/2610.41104#S3.F2)illustrates the proposed framework\.

### 3\.1The Teacher Stores Applied Updates

Muon keeps a momentum buffer as large as each hidden matrix, while the EMA\-teacher already stores a history of the same trajectory\. If this history can take the role of momentum, the buffer becomes unnecessary\. We therefore first characterize what the teacher stores and why it cannot simply replace momentum\. Let𝐔\(s\)\\mathbf\{U\}^\{\(s\)\}denote the applied update direction at stepss, with𝐖S\(s\+1\)=𝐖S\(s\)−η​cA,B​𝐔\(s\)\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}=\\mathbf\{W\}\_\{S\}^\{\(s\)\}\-\\eta c\_\{A,B\}\\mathbf\{U\}^\{\(s\)\}, assuming constant learning rateη\\etaand zero weight decay\. Unrolling the EMA update in Eq\. \([3](https://arxiv.org/html/2610.41104#S2.E3)\), initialized with𝐖T\(0\)=𝐖S\(0\)\\mathbf\{W\}\_\{T\}^\{\(0\)\}=\\mathbf\{W\}\_\{S\}^\{\(0\)\}, gives \(Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1)\)

𝐖T\(s\)−𝐖S\(s\)\\displaystyle\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}=τ​η​cA,B​∑j=0s−1τj​𝐔\(s−1−j\)\.\\displaystyle=\\tau\\eta c\_\{A,B\}\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\mathbf\{U\}^\{\(s\-1\-j\)\}\.\(7\)Thus, the teacher lags behind the student, with its displacement encoding a discounted history of applied update directions\. Unlike Muon’s momentum, which accumulates raw gradients, this history contains optimizer outputs that have already been orthogonalized under standard Muon\. Directly using this displacement as momentum would therefore mix historical updates with current gradients, potentially reinforcing stale directions as the self\-distillation target evolves\. Moreover, combining both signals before Newton–Schulz entangles their contributions through nonlinear orthogonalization, making their relative influence difficult to control\. Muon\-RT instead processes the two signals separately and combines them after normalization, explicitly controlling the historical contribution through a mixing weight\.

### 3\.2The Muon\-RT Update

##### Gradient branch\.

We eliminate Muon’s momentum buffer and orthogonalize the current student gradient𝐆=∇𝐖Sℒ​\(θS\)\\mathbf\{G\}=\\nabla\_\{\\mathbf\{W\}\_\{S\}\}\\mathcal\{L\}\(\\theta\_\{S\}\):

𝐎=NSK​\(𝐆\)\.\\mathbf\{O\}=\\mathrm\{NS\}\_\{K\}\(\\mathbf\{G\}\)\.\(8\)This corresponds to Muon withμ=0\\mu=0\. Since the EMA\-teacher and its distillation targets evolve during training, accumulated gradients may lag behind the current objective \(Section[4\.5](https://arxiv.org/html/2610.41104#S4.SS5)\)\.

##### Teacher branch\.

Let𝐃=𝐖T−𝐖S\\mathbf\{D\}=\\mathbf\{W\}\_\{T\}\-\\mathbf\{W\}\_\{S\}denote the teacher–student displacement\. We remove its component parallel to the student weights:

𝐇=𝐃−⟨𝐃,𝐖S⟩F‖𝐖S‖F2\+ε​𝐖S\.\\mathbf\{H\}=\\mathbf\{D\}\-\\frac\{\\langle\\mathbf\{D\},\\mathbf\{W\}\_\{S\}\\rangle\_\{\\mathrm\{F\}\}\}\{\\\|\\mathbf\{W\}\_\{S\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\varepsilon\}\\mathbf\{W\}\_\{S\}\.\(9\)This suppresses radial changes associated with parameter scaling, including weight decay, while retaining the remaining trajectory information\. We normalize the residual by its estimated largest singular value:

𝐏=𝐇max⁡\(σ^max​\(𝐇\),ε\)\.\\mathbf\{P\}=\\frac\{\\mathbf\{H\}\}\{\\max\(\\widehat\{\\sigma\}\_\{\\max\}\(\\mathbf\{H\}\),\\varepsilon\)\}\.\(10\)This brings its spectral norm approximately to one, comparable to the orthogonalized gradient𝐎\\mathbf\{O\}\. We estimateσmax\\sigma\_\{\\max\}usingKPIK\_\{\\mathrm\{PI\}\}power iterations, warm\-started from the previous right singular vector, retaining only one vector of lengthBBper matrix\.

##### Mixing after orthogonalization\.

We combine the orthogonalized gradient and normalized teacher directions:

𝐔\\displaystyle\\mathbf\{U\}=\(1−α\)​𝐎\+α​𝐏,\\displaystyle=\(1\-\\alpha\)\\mathbf\{O\}\+\\alpha\\mathbf\{P\},\(11\)𝐖S\\displaystyle\\mathbf\{W\}\_\{S\}←\(1−η​λ\)​𝐖S−η​cA,B​𝐔\.\\displaystyle\\leftarrow\(1\-\\eta\\lambda\)\\mathbf\{W\}\_\{S\}\-\\eta c\_\{A,B\}\\mathbf\{U\}\.\(12\)Here,α∈\[0,1\]\\alpha\\in\[0,1\]controls the teacher contribution, with‖α​𝐏‖2≤α\\\|\\alpha\\mathbf\{P\}\\\|\_\{2\}\\leq\\alphaunder exact spectral normalization and an approximate bound with power iteration\. Settingα=0\\alpha=0recovers momentum\-free Muon\. The EMA\-teacher is updated afterward using Eq\. \([3](https://arxiv.org/html/2610.41104#S2.E3)\)\.

Algorithm 1One optimization step of Muon\-RT1:Student

θS\\theta\_\{S\}, EMA\-teacher

θT\\theta\_\{T\}, loss

ℒ\\mathcal\{L\}, mixing weight

α\\alpha, EMA decay

τ\\tau, learning rate

η\\eta, weight decay

λ\\lambda
2:Compute student gradients

∇θSℒ\\nabla\_\{\\theta\_\{S\}\}\\mathcal\{L\}
3:foreach Muon\-optimized hidden matrix

ℓ\\elldo

4:

𝐆ℓ←∇𝐖S,ℓℒ\\mathbf\{G\}\_\{\\ell\}\\leftarrow\\nabla\_\{\\mathbf\{W\}\_\{S,\\ell\}\}\\mathcal\{L\}
5:

𝐎ℓ←NSK​\(𝐆ℓ\)\\mathbf\{O\}\_\{\\ell\}\\leftarrow\\mathrm\{NS\}\_\{K\}\(\\mathbf\{G\}\_\{\\ell\}\)
6:

𝐃ℓ←𝐖T,ℓ−𝐖S,ℓ\\mathbf\{D\}\_\{\\ell\}\\leftarrow\\mathbf\{W\}\_\{T,\\ell\}\-\\mathbf\{W\}\_\{S,\\ell\}
7:Compute

𝐇ℓ\\mathbf\{H\}\_\{\\ell\}using Eq\. \([9](https://arxiv.org/html/2610.41104#S3.E9)\)

8:Estimate

σ^max​\(𝐇ℓ\)\\widehat\{\\sigma\}\_\{\\max\}\(\\mathbf\{H\}\_\{\\ell\}\)by power iteration

9:Compute

𝐏ℓ\\mathbf\{P\}\_\{\\ell\}using Eq\. \([10](https://arxiv.org/html/2610.41104#S3.E10)\)

10:

𝐔ℓ←\(1−α\)​𝐎ℓ\+α​𝐏ℓ\\mathbf\{U\}\_\{\\ell\}\\leftarrow\(1\-\\alpha\)\\mathbf\{O\}\_\{\\ell\}\+\\alpha\\mathbf\{P\}\_\{\\ell\}
11:Update

𝐖S,ℓ\\mathbf\{W\}\_\{S,\\ell\}in place via Eq\. \([12](https://arxiv.org/html/2610.41104#S3.E12)\)

12:Release temporary matrix buffers

13:endfor

14:Update remaining student parameters with AdamW

15:

θT←τ​θT\+\(1−τ\)​θS\\theta\_\{T\}\\leftarrow\\tau\\theta\_\{T\}\+\(1\-\\tau\)\\theta\_\{S\}

### 3\.3Memory and Sharded Implementation

Muon\-RT reduces the persistent states of each Muon\-optimized hidden matrix from\{𝐖S,𝐖T,𝐌\}\\\{\\mathbf\{W\}\_\{S\},\\mathbf\{W\}\_\{T\},\\mathbf\{M\}\\\}to\{𝐖S,𝐖T\}\\\{\\mathbf\{W\}\_\{S\},\\mathbf\{W\}\_\{T\}\\\}, eliminating the momentum buffer\. Including gradient storage𝐆\\mathbf\{G\}, this reduces the matrix\-sized tensors from 4 to 3\. Importantly,𝐎\\mathbf\{O\},𝐃\\mathbf\{D\},𝐇\\mathbf\{H\},𝐏\\mathbf\{P\}, and𝐔\\mathbf\{U\}are temporary computational tensors, not additional parameters or persistent states\. As summarized in Algorithm[1](https://arxiv.org/html/2610.41104#alg1), they are computed one matrix at a time, with buffers reused or released after updating𝐖S\\mathbf\{W\}\_\{S\}in place\. Only a power\-iteration vector of lengthBBper matrix is retained for warm\-starting\. Thus, the temporary workspace scales with the processed matrix size rather than requiring additional model\-wide buffers\. We report measured persistent memory in Section[4\.4](https://arxiv.org/html/2610.41104#S4.SS4)and detailed accounting in Appendix[A\.6](https://arxiv.org/html/2610.41104#A1.SS6)\.

We use Fully Sharded Data Parallel \(FSDP\)\([Zhao et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib21)\), with matching sharding layouts for student and EMA\-teacher parameters\. The displacement𝐃=𝐖T−𝐖S\\mathbf\{D\}=\\mathbf\{W\}\_\{T\}\-\\mathbf\{W\}\_\{S\}is computed locally, while the Frobenius inner products for radial projection are all\-reduced across ranks\. Full matrices𝐆\\mathbf\{G\}and𝐇\\mathbf\{H\}are temporarily gathered for Newton–Schulz orthogonalization and power iteration, respectively\. Each rank then updates only its corresponding shard of𝐖S\\mathbf\{W\}\_\{S\}, followed by the sharded EMA update\. With consistent global reductions and power\-iteration initialization, the sharded implementation matches the unsharded computation up to numerical differences \(Appendix[A\.4](https://arxiv.org/html/2610.41104#A1.SS4)\)\.

### 3\.4Theoretical Analysis

We now give the guarantees behind Muon\-RT; Appendix[A\.2](https://arxiv.org/html/2610.41104#A1.SS2)contains the proofs and the full assumptions\. All results hold for any differentiable host loss with a parameter\-aligned EMA\-teacher\.

###### Lemma 1\(EMA trajectory\)\.

If𝐖S\(s\+1\)=𝐖S\(s\)−η​cA,B​𝐔\(s\)\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}=\\mathbf\{W\}\_\{S\}^\{\(s\)\}\-\\eta c\_\{A,B\}\\mathbf\{U\}^\{\(s\)\},𝐖T\(s\+1\)=τ​𝐖T\(s\)\+\(1−τ\)​𝐖S\(s\+1\)\\mathbf\{W\}\_\{T\}^\{\(s\+1\)\}=\\tau\\mathbf\{W\}\_\{T\}^\{\(s\)\}\+\(1\-\\tau\)\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}and𝐖T\(0\)=𝐖S\(0\)\\mathbf\{W\}\_\{T\}^\{\(0\)\}=\\mathbf\{W\}\_\{S\}^\{\(0\)\}, then Eq\. \([7](https://arxiv.org/html/2610.41104#S3.E7)\) holds for everys≥1s\\geq 1\.

###### Lemma 2\(Temporal averaging\)\.

Let𝐔¯\(s\)=\(1−τ\)​∑j<sτj​𝐔\(s−1−j\)\\overline\{\\mathbf\{U\}\}^\{\(s\)\}=\(1\-\\tau\)\\sum\_\{j<s\}\\tau^\{j\}\\mathbf\{U\}^\{\(s\-1\-j\)\}be the rescaled displacement\. If locally𝐔\(s\)=𝐝\+𝛏\(s\)\\mathbf\{U\}^\{\(s\)\}=\\mathbf\{d\}\+\\boldsymbol\{\\xi\}^\{\(s\)\}with a fixed direction𝐝\\mathbf\{d\}and zero\-mean, temporally uncorrelated noise of varianceσξ2\\sigma\_\{\\xi\}^\{2\}, then𝔼⁡\[𝐔¯\(s\)\]=\(1−τs\)​𝐝\\mathbb\{E\}\[\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\]=\(1\-\\tau^\{s\}\)\\mathbf\{d\}and the variance of𝐔¯\(s\)\\overline\{\\mathbf\{U\}\}^\{\(s\)\}is1−τ1\+τ​\(1−τ2​s\)​σξ2\\frac\{1\-\\tau\}\{1\+\\tau\}\(1\-\\tau^\{2s\}\)\\sigma\_\{\\xi\}^\{2\}\.

Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2)shows that the teacher holds a smoothed estimate of persistent motion, with the noise variance reduced by a factor of up to\(1−τ\)/\(1\+τ\)\(1\-\\tau\)/\(1\+\\tau\)\. The closerτ\\tauis to 1, the more stable the teacher direction is than any single step\.

###### Theorem 1\(Convergence of Muon\-RT\)\.

Under A1–A4 with constant stepη\\eta,λ=0\\lambda\{=\}0andcA,Bc\_\{A,B\}absorbed intoη\\eta, letγR=aR−L​η2​bR\\gamma\_\{R\}=a\_\{R\}\-\\frac\{L\\eta\}\{2\}b\_\{R\}\. IfγR\>0\\gamma\_\{R\}\>0and0<2​μPL​η​γR<10<2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}<1, then withqR=1−2​μPL​η​γRq\_\{R\}=1\-2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\},

𝔼⁡\[Δs\+1\]≤qR​𝔼​\[Δs\]\+L​η22​σR2\+2​δs\.\\mathbb\{E\}\[\\Delta\_\{s\+1\}\]\\leq q\_\{R\}\\,\\mathbb\{E\}\[\\Delta\_\{s\}\]\+\\tfrac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\+2\\delta\_\{s\}\.\(13\)Hence𝔼⁡\[Δs\]\\mathbb\{E\}\[\\Delta\_\{s\}\]converges linearly at rateqRq\_\{R\}to a neighborhood of sizeO⁡\(L​η​σR2μPL​γR\+δ¯μPL​η​γR\)O\\big\(\\frac\{L\\eta\\sigma\_\{R\}^\{2\}\}\{\\mu\_\{\\mathrm\{PL\}\}\\gamma\_\{R\}\}\+\\frac\{\\bar\{\\delta\}\}\{\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\}\\big\)whenδs≤δ¯\\delta\_\{s\}\\leq\\bar\{\\delta\}\.

Proposition and Corollary in the Appendix[A\.1](https://arxiv.org/html/2610.41104#A1.SS1)further shows where gain must come from and when the reuse of the teacher parameters helps\.

## 4Experiments

We evaluate Muon\-RT as a drop\-in optimizer for two self\-distillation hosts: fine\-tuning with SDFT and on\-policy post\-training with SDPO\. Our evaluation covers task accuracy, retention of prior capabilities, convergence speed and memory\. Finally, ablations isolate each design choice, and we test the theory of Section[3\.4](https://arxiv.org/html/2610.41104#S3.SS4)against empirical measurements\.

Table 1:The accuracy without continual learning \(W/O CL\) and with continual learning \(CL\) task for SDFT \(Qwen2\.5\-7B\)\. HS, HE, IFE, TQA, and Wino denote HellaSwag, HumanEval, IFEval, TruthfulQA, and WinoGrande; Avg\. is the mean across previous tasks\.†denotes results from SDFT, with averages recomputed\. Muon\-RT shows significant improvement without CL setting, however for the CL setting its highly competitive\.### 4\.1Experimental Setup

Tasks and data\.For fine\-tuning with self\-distillation, we follow SDFT\([Shenfeld et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib4)\)on Qwen2\.5\-7B\-Instruct\([Yang et al\., 2024a](https://arxiv.org/html/2610.41104#bib.bib43)\)with its two tasks: tool use from ToolAlpaca\([Tang et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib48)\)and science questions from SciKnowEval\([Feng et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib47)\)\. For on\-policy post\-training, we follow SDPO\([Hübotter et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib9)\)on Olmo\-3\-7B\-Instruct\([Ettinger et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib49)\)with the biology and materials subsets of SciKnowEval\. In both settings, we use the default splits and metrics of the original papers\.

Baselines\.For SDFT, we compare with SFT, SFT with re\-invoked prior knowledge, DFT\([Wu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib52)\)and SDFT with AdamW, as reported by[Shenfeld et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib4)\. We additionally run SDFT with Muon and with MuonClip, which adds the query–key logit clipping \(QK\-Clip\) of Kimi K2\([Bai et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib53)\)\. Both Muon variant useμ=0\\mu\{=\}0, their stronger setting; momentumμ=0\.95\\mu\{=\}0\.95is evaluated as an ablation \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\)\. For SDPO, we compare with Group Relative Policy Optimization \(GRPO;[Shao et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib50)\), on\-policy GRPO and SDPO with AdamW, as reported by[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9), and run SDPO with Muon, MuonClip and Muon\-RT\. The training and evaluation details are provide in the Appendix[A\.9](https://arxiv.org/html/2610.41104#A1.SS9)and Appendix[A\.5](https://arxiv.org/html/2610.41104#A1.SS5)lists all hyperparameter settings\.

### 4\.2Fine\-Tuning with Self\-Distillation

Task accuracy\.Table[1](https://arxiv.org/html/2610.41104#S4.T1)shows that Muon\-RT achieves the highest new\-task accuracy

Table 2:On\-policy post\-training with SDPO \(Olmo\-3\-7B\)\. Best avg@16 accuracy \(%\) within 1 and 5 hours\.†from SDPO\. Best in bold, second\-best underlined\.on both tasks: 74\.2% on tool use and 71\.8% on science, compared with 70\.6% and 70\.2% for SDFT with AdamW\. It improves on momentum\-free Muon by 5\.1 and 2\.8 points, respectively\. The two methods share the same 32\-bit teacher and differ only in whether the teacher branch is reused, so we attribute this gain to teacher reuse\. Muon\-RT also outperforms MuonClip without any logit clipping\. Across both host models, Muon\-RT ranks first among the Muon variants in all six task and budget settings \(Tables[1](https://arxiv.org/html/2610.41104#S4.T1)and[2](https://arxiv.org/html/2610.41104#S4.T2)\)\.

Prior capabilities\.Muon\-RT changes only the optimizer, so the question is whether its faster learning comes at the cost of prior capabilities\. Retention is set mainly by the host loss: the prior\-capability averages of all self\-distillation runs lie within 2\.4 points of one another, and those of Muon\-RT are 11\.9 and 6\.7 points above SFT \(Table[1](https://arxiv.org/html/2610.41104#S4.T1)\)\. On science, Muon\-RT has the best HumanEval score \(70\.7\) and a higher average than SDFT with AdamW \(65\.3 against 64\.5\)\. On tool use, where Muon\-RT gains the most on the new task, all Muon variants average 1\.1–2\.4 points below AdamW, mostly on IFEval\. Even so Muon\-RT retains 96% of the AdamW average \(63\.0 against 65\.4\)\. Muon\-RT thus improves new\-task accuracy through optimization alone while preserving the retention advantage of self\-distillation over SFT\.

Figure 3:Training dynamics with SDFT\. \(a, b\) Best accuracy reached up to each step\. \(c\) Self\-distillation KL at learning rate5×10−55\{\\times\}10^\{\-5\}\. \(d\) Persistent memory per GPU between steps, with a bfloat16 mixed\-precision student \(32\-bit master weights\) and a 32\-bit teacher for every optimizer\.Convergence\.Figure[3](https://arxiv.org/html/2610.41104#S4.F3)\(a\)–\(c\) shows that Muon\-RT learns faster \(25\-step averages for the KL\)\. On tool use, it reaches 74\.2% after 100 of 252 steps, whereas MuonClip needs 250 steps to reach 73\.2% and Muon does not exceed 69\.1%\. On science, it leads at 50 steps and ends highest\. At the same learning rate, its self\-distillation KL is lower than that of both baselines for most of training, as Corollary[1](https://arxiv.org/html/2610.41104#Thmcorollary1)predicts when the teacher branch raises the effective descent coefficient\.

### 4\.3On\-Policy Post\-Training

It is evident from Table[2](https://arxiv.org/html/2610.41104#S4.T2), Muon\-RT is the most accurate optimizer for SDPO on both datasets and budgets, 2\.0 points above AdamW on average\. Its advantage grows with training time: its margin over SDPO with AdamW is 0\.7–1\.1 points after 1 hour and widens to 2\.4–3\.7 points after 5 hours\. This trend is consistent with the gain arising from information accumulated over the optimization trajectory rather than from any single step update\. On Materials, Muon\-RT outperforms all baselines \(82\.8% after 5 hours\)\. On Biology it is second only to GRPO, which uses a different training signal\.

### 4\.4Memory and Compute

Figure 4:Analysis of the teacher branch\. \(a\) Measured and predicted teacher–student displacement during training\. \(b\) Cosine with the mean gradient direction under gradient noise on a Qwen2\.5\-7B\-Instruct weight matrix\. \(c\) Change of the update caused by the teacher direction\. \(d\) Gradient norm during training\.Figure 5:Projected persistent memory of self\-distillation versus model size, with all states in 32 bits and gradients included\.Figure[3](https://arxiv.org/html/2610.41104#S4.F3)\(d\) reports persistent memory per GPU, measured between steps when gradients are freed, with every optimizer using a 32\-bit EMA\-teacher\. Because Muon\-RT keeps no optimizer state for hidden matrices, it needs only 37\.7 GB, 13\.9 GB \(27%\) less than Muon with its standard momentumμ=0\.95\\mu\{=\}0\.95and 26\.8 GB \(42%\) less than AdamW\. Momentum\-free Muon and MuonClip need the same 37\.7 GB but are 0\.8–5\.1 points less accurate \(Table[1](https://arxiv.org/html/2610.41104#S4.T1)\): Muon\-RT therefore obtains the benefit of momentum without storing a momentum buffer\. At 10T parameters, Muon\-RT needs 121 TB, 25% less than Muon or MuonClip with momentum \(160 TB\) and 40% less than AdamW \(200 TB; Figure[5](https://arxiv.org/html/2610.41104#S4.F5), Appendix[A\.6](https://arxiv.org/html/2610.41104#A1.SS6)\)\. The additional cost is one power iteration and one extra gather per matrix\. The optimizer step takes 2\.3 s, about 2% of a training step dominated by generation\.

### 4\.5Ablations and Analysis

Table 3:Ablations with SDFT \(accuracy, %\)\.We ablate each component of Muon\-RT with SDFT on both tasks \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\): the optimizer state it replaces, the teacher weightα\\alpha, the construction of the teacher branch, and its combination with logit clipping\. All ablations follow the setting of Table[1](https://arxiv.org/html/2610.41104#S4.T1), withα=0\.1\\alpha\{=\}0\.1unless varied, and every block includes the full method as reference\. The Appendix[A\.11](https://arxiv.org/html/2610.41104#A1.SS11)provides the detailed discussion about the ablation provide in the Table[1](https://arxiv.org/html/2610.41104#S4.T1)\.

### 4\.6Empirical Checks of the Theory

Figure[4](https://arxiv.org/html/2610.41104#S4.F4)tests each theoretical statement with a matching measurement\. The details are discussed in the Appendix[A\.10](https://arxiv.org/html/2610.41104#A1.SS10)where we empirically justify each theoretical claims\.

## 5Related Work

Training on the model’s own samples against a teacher improves adaptation\([Agarwal et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib10);[Ko et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib40)\), with further advances in self\-distillation\([Xu et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib42);[Wen et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib41)\)\. Such approaches can reduce forgetting relative to SFT\([Yang et al\., 2024b](https://arxiv.org/html/2610.41104#bib.bib20);[Shenfeld et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib36);[Chen et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib37)\), which often compromises prior capabilities\([Biderman et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib19);[Zheng et al\., 2025a](https://arxiv.org/html/2610.41104#bib.bib25);[Shi et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib38)\)\. In particular, SDFT\([Shenfeld et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib4)\)and SDPO\([Hübotter et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib9)\)use privileged context with an EMA\-teacher\. These methods redesign the objective but retain standard optimizers, keeping 2 model\-sized histories without exploiting their interaction\. We preserve their objectives and modify only the optimizer, improving accuracy with both hosts\.

Muon\([Jordan et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib54)\)brings orthogonalized updates to LLMs\([Liu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib5);[Bai et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib53)\), supported by analyses of matrix updates\([Amsel et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib35);[Large et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib29)\)and of its convergence\([Li and Hong, 2025](https://arxiv.org/html/2610.41104#bib.bib55);[Shen et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib56)\)\. Other optimizers include SOAP\([Vyas et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib12)\), Sophia\([Liu et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib11)\)and Lion\([Chen et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib44)\)\. Memory\-efficient methods use low\-rank projections\([Zhao et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib7);[Hao et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib32);[Zhu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib31)\), quantized or shared states\([Li et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib15);[Dettmers et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib45);[Zhang et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib13)\), block updates\([Luo et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib28);[Pan et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib27)\), or fused updates\([Lv et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib46);[Malladi et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib16)\)\. Others introduce slow averages\([Pagliardini et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib14)\)or examine optimizer tuning\([Zhao et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib26)\)\. These approaches redesign the optimizer’s*own*state rather than reuse model copies already maintained during training\. Muon\-RT eliminates Muon’s momentum state, saving 27% of the measured memory at 7B and 25% of the projected memory at 10T while improving accuracy\.

EMA\-teachers provide stable learning targets\([Tarvainen and Valpola, 2017](https://arxiv.org/html/2610.41104#bib.bib1);[Grill et al\., 2020](https://arxiv.org/html/2610.41104#bib.bib2);[Caron et al\., 2021](https://arxiv.org/html/2610.41104#bib.bib3)\), extending to representation learning\([Oquab et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib34)\)and EMA scaling\([Busbridge et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib17);[Morales\-Brotons et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib18)\)\. Weight averaging also improves optimization and model selection\([Polyak and Juditsky, 1992](https://arxiv.org/html/2610.41104#bib.bib62);[Izmailov et al\., 2018](https://arxiv.org/html/2610.41104#bib.bib61);[Zhang et al\., 2019](https://arxiv.org/html/2610.41104#bib.bib60);[Jang et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib33);[Ramé et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib30);[Defazio et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib8)\), and momentum itself can be written as an average of iterates\([Defazio, 2020](https://arxiv.org/html/2610.41104#bib.bib57)\)or mixed with the current gradient\([Ma and Yarats, 2019](https://arxiv.org/html/2610.41104#bib.bib59)\)\. These approaches treat the average as a model, a target or a dedicated buffer, without characterizing the information encoded by its displacement from current weights\. We show that this gap stores applied updates, averages out noise, and provides a reusable optimized direction\.

## 6Conclusion

We asked whether the EMA\-teacher of self\-distillation can replace Muon’s momentum buffer\. We showed that the teacher–student displacement is a discounted sum of applied updates \(Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1)\) that averages out their noise \(Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2)\), while the orthogonalized gradient is already optimal for a single step \(Proposition[1](https://arxiv.org/html/2610.41104#Thmproposition1)\); the teacher thus adds temporal information rather than a better instantaneous direction\. On this basis, Muon\-RT drops the momentum buffer and adds the radial\-free, spectrally normalized displacement after orthogonalization with a bounded weight, with linear convergence under the PL condition\. Empirically, it reached the highest accuracy among all optimizers with both SDFT and SDPO, converged faster with SDFT, retained prior capabilities well above SFT and reduced memory\. The ablations show that gradient momentum hurts when the target moves, whereas the teacher’s history helps when it enters after orthogonalization and stays consistent with the student\. Any method that keeps a parameter\-aligned EMA model can lend it to the optimizer\.

## Limitations

Our experiments cover 7B\-parameter models from 2 families and 4 tasks; larger models and further domains are left for future work\. The convergence analysis relies on smoothness, PL and alignment assumptions that are standard for non\-convex optimization but whose constants are not estimated for LLMs\. Muon\-RT applies when a parameter\-aligned EMA\-teacher already exists, as in SDFT and SDPO; for methods without one, a teacher kept only to replace momentum would not save memory\.

## Ethics Statement

This work changes how a model is optimized, not what it is trained on, and it uses public datasets \(ToolAlpaca, SciKnowEval\) under their released licenses and splits\. Models trained on science questions may still produce wrong answers about chemical, biological or material properties and should not be used for laboratory or safety decisions without expert checks\. Reducing optimizer memory lowers the hardware needed for continual adaptation; we report our compute in Appendix[A\.5](https://arxiv.org/html/2610.41104#A1.SS5)\. AI assistants were used only for language and writing refinement; the authors checked all content and remain responsible for it\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stańczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2306.13649)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Amselet al\.\(2025\)N\. Amsel, D\. Persson, C\. Musco, and R\. GowerThe Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm\.arXiv preprint arXiv:2505\.16932\.External Links:[Link](https://arxiv.org/abs/2505.16932)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Baiet al\.\(2025\)K\. T\. Y\. Bai, Y\. Bao, Y\. Charles, C\. Chen, G\. Chen, H\. Chen, H\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen,et al\.Kimi K2: Open Agentic Intelligence\.arXiv preprint arXiv:2507\.20534\.External Links:[Link](https://arxiv.org/abs/2507.20534)Cited by:[§A\.6](https://arxiv.org/html/2610.41104#A1.SS6.p1.1),[§A\.8](https://arxiv.org/html/2610.41104#A1.SS8.p1.1),[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Bidermanet al\.\(2024\)D\. Biderman, J\. Portes, J\. J\. G\. Ortiz, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle, C\. Blakeney, and J\. P\. CunninghamLoRA Learns Less and Forgets Less\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2405.09673)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Busbridgeet al\.\(2023\)D\. Busbridge, J\. Ramapuram, P\. Ablin, T\. Likhomanenko, E\. G\. Dhekane, X\. Suau, and R\. WebbHow to Scale Your EMA\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2307.13813)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Caronet al\.\(2021\)M\. Caron, H\. Touvron, I\. Misra, H\. Jégou, J\. Mairal, P\. Bojanowski, and A\. JoulinEmerging properties in self\-supervised vision transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 9650–9660\.External Links:[Document](https://dx.doi.org/10.1109/ICCV48922.2021.00951),[Link](https://openaccess.thecvf.com/content/ICCV2021/html/Caron_Emerging_Properties_in_Self-Supervised_Vision_Transformers_ICCV_2021_paper.html)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Chenet al\.\(2025\)H\. Chen, N\. Razin, K\. R\. Narasimhan, and D\. ChenRetaining by Doing: The Role of On\-Policy Data in Mitigating Forgetting\.arXiv preprint arXiv:2510\.18874\.External Links:[Link](https://arxiv.org/abs/2510.18874)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Chenet al\.\(2023\)X\. Chen, C\. Liang, D\. Huang, E\. Real, K\. Wang, Y\. Liu, H\. Pham, X\. Dong, T\. Luong, C\. Hsieh, Y\. Lu, and Q\. V\. LeSymbolic Discovery of Optimization Algorithms\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2302.06675)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Defazioet al\.\(2024\)A\. Defazio, X\. Yang, H\. Mehta, K\. Mishchenko, A\. Khaled, and A\. CutkoskyThe road less scheduled\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-0320),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/136b9a13861308c8948cd308ccd02658-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Defazio \(2020\)A\. DefazioMomentum via primal averaging: theoretical insights and learning rate schedules for non\-convex optimization\.arXiv preprint arXiv:2010\.00406\.Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Dehghaniet al\.\(2023\)M\. Dehghani, J\. Djolonga, B\. Mustafa, P\. Padlewski, J\. Heek, J\. Gilmer, A\. Steiner, M\. Caron, R\. Geirhos, I\. M\. Alabdulmohsin, R\. Jenatton, L\. Beyer,et al\.Scaling Vision Transformers to 22 Billion Parameters\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2302.05442)Cited by:[§A\.8](https://arxiv.org/html/2610.41104#A1.SS8.p1.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: Efficient Finetuning of Quantized LLMs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2305.14314)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Ettingeret al\.\(2025\)A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. W\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. D\. Morrison, J\. Poznanski, K\. Lo,et al\.Olmo 3\.arXiv preprint arXiv:2512\.13961\.External Links:[Link](https://arxiv.org/abs/2512.13961)Cited by:[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p1.1)\.
- Fenget al\.\(2024\)K\. Feng, K\. Ding, W\. Wang, X\. Zhuang, Z\. Wang, M\. Qin, Y\. Zhao, J\. Yao, Q\. Zhang, and H\. ChenSciKnowEval: Evaluating Multi\-level Scientific Knowledge of Large Language Models\.arXiv preprint arXiv:2406\.09098\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.09098)Cited by:[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p1.1)\.
- Grillet al\.\(2020\)J\. Grill, F\. Strub, F\. Altché, C\. Tallec, P\. H\. Richemond, E\. Buchatskaya, C\. Doersch, B\. A\. Pires, Z\. D\. Guo, M\. G\. Azar, B\. Piot, K\. Kavukcuoglu, R\. Munos, and M\. ValkoBootstrap your own latent: a new approach to self\-supervised learning\.InAdvances in Neural Information Processing Systems,Vol\.33\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Haoet al\.\(2024\)Y\. Hao, Y\. Cao, and L\. MouFlora: Low\-Rank Adapters Are Secretly Gradient Compressors\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2402.03293)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Hübotteret al\.\(2026\)J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. Kleine Buening, C\. Guestrin, and A\. KrauseReinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.External Links:[Link](https://arxiv.org/abs/2601.20802)Cited by:[§A\.3](https://arxiv.org/html/2610.41104#A1.SS3.p2.1),[§A\.5](https://arxiv.org/html/2610.41104#A1.SS5.p1.1),[§A\.9](https://arxiv.org/html/2610.41104#A1.SS9.p1.1),[Table 5](https://arxiv.org/html/2610.41104#A1.T5),[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Izmailovet al\.\(2018\)P\. Izmailov, D\. Podoprikhin, T\. Garipov, D\. Vetrov, and A\. G\. WilsonAveraging weights leads to wider optima and better generalization\.InConference on Uncertainty in Artificial Intelligence,Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Janget al\.\(2024\)D\. Jang, S\. Yun, and D\. HanModel Stock: All we need is just a few fine\-tuned models\.InEuropean Conference on Computer Vision \(ECCV\),External Links:[Link](https://arxiv.org/abs/2403.19522)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.Note:[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px2.p1.3),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Karimiet al\.\(2016\)H\. Karimi, J\. Nutini, and M\. SchmidtLinear Convergence of Gradient and Proximal\-Gradient Methods Under the Polyak\-Łojasiewicz Condition\.InJoint European Conference on Machine Learning and Knowledge Discovery in Databases \(ECML PKDD\),External Links:[Document](https://dx.doi.org/10.1007/978-3-319-46128-1%5F50),[Link](https://arxiv.org/abs/1608.04636)Cited by:[§A\.1](https://arxiv.org/html/2610.41104#A1.SS1.p2.1)\.
- Kingma and Ba \(2014\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.CoRRabs/1412\.6980\.External Links:[Link](http://arxiv.org/)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1)\.
- Koet al\.\(2024\)J\. Ko, S\. Kim, T\. Chen, and S\. YunDistiLLM: Towards Streamlined Distillation for Large Language Models\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2402.03898)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Largeet al\.\(2024\)T\. M\. J\. Large, Y\. Liu, M\. Huh, H\. Bahng, P\. Isola, and J\. BernsteinScalable Optimization in the Modular Norm\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2405.14813)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Liet al\.\(2023\)B\. Li, J\. Chen, and J\. ZhuMemory Efficient Optimizers with 4\-bit States\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2309.01507)Cited by:[§A\.3](https://arxiv.org/html/2610.41104#A1.SS3.p1.1),[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Li and Hong \(2025\)J\. Li and M\. HongA note on the convergence of Muon\.arXiv preprint arXiv:2502\.02900\.Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Liuet al\.\(2024\)H\. Liu, Z\. Li, D\. L\. W\. Hall, P\. Liang, and T\. MaSophia: A Scalable Stochastic Second\-order Optimizer for Language Model Pre\-training\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2305.14342)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. YangMuon is scalable for llm training\.External Links:2502\.16982,[Link](https://arxiv.org/abs/2502.16982)Cited by:[§A\.3](https://arxiv.org/html/2610.41104#A1.SS3.p1.1),[§A\.4](https://arxiv.org/html/2610.41104#A1.SS4.p1.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px2.p1.4),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px2.p1.1)\.
- Luoet al\.\(2024\)Q\. Luo, H\. Yu, and X\. LiBAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Document](https://dx.doi.org/10.52202/079017-0786),[Link](https://arxiv.org/abs/2404.02827)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Lvet al\.\(2024\)K\. Lv, Y\. Yang, T\. Liu, Q\. Gao, Q\. Guo, and X\. QiuFull Parameter Fine\-tuning for Large Language Models with Limited Resources\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2306.09782)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Ma and Yarats \(2019\)J\. Ma and D\. YaratsQuasi\-hyperbolic momentum and Adam for deep learning\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Malladiet al\.\(2023\)S\. Malladi, T\. Gao, E\. Nichani, A\. Damian, J\. D\. Lee, D\. Chen, and S\. AroraFine\-Tuning Language Models with Just Forward Passes\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2305.17333)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Morales\-Brotonset al\.\(2024\)D\. Morales\-Brotons, T\. Vogels, and H\. HendrikxExponential Moving Average of Weights in Deep Learning: Dynamics and Benefits\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2411.18704)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Oquabet al\.\(2024\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas,et al\.DINOv2: Learning Robust Visual Features without Supervision\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2304.07193)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Pagliardiniet al\.\(2025\)M\. Pagliardini, P\. Ablin, and D\. GrangierThe AdEMAMix Optimizer: Better, Faster, Older\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2409.03137)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Panet al\.\(2024\)R\. Pan, X\. Liu, S\. Diao, R\. Pi, J\. Zhang, C\. Han, and T\. ZhangLISA: Layerwise Importance Sampling for Memory\-Efficient Large Language Model Fine\-Tuning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2403.17919)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Polyak and Juditsky \(1992\)B\. T\. Polyak and A\. B\. JuditskyAcceleration of stochastic approximation by averaging\.SIAM Journal on Control and Optimization30\(4\),pp\. 838–855\.Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Raméet al\.\(2024\)A\. Ramé, N\. Vieillard, L\. Hussenot, R\. Dadashi, G\. Cideron, O\. Bachem, and J\. FerretWARM: On the Benefits of Weight Averaged Reward Models\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2401.12187)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.\-K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p2.1)\.
- Shenet al\.\(2025\)W\. Shen, R\. Huang, M\. Huang, C\. Shen, and J\. ZhangOn the convergence analysis of Muon\.arXiv preprint arXiv:2505\.23737\.Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Shenfeldet al\.\(2026\)I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. AgrawalSelf\-distillation enables continual learning\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 111542–111558\.External Links:[Link](https://proceedings.mlr.press/v306/shenfeld26a.html)Cited by:[§A\.3](https://arxiv.org/html/2610.41104#A1.SS3.p2.1),[§A\.5](https://arxiv.org/html/2610.41104#A1.SS5.p1.1),[§A\.9](https://arxiv.org/html/2610.41104#A1.SS9.p1.1),[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§2](https://arxiv.org/html/2610.41104#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Shenfeldet al\.\(2025\)I\. Shenfeld, J\. Pari, and P\. AgrawalRL’s Razor: Why Online Reinforcement Learning Forgets Less\.arXiv preprint arXiv:2509\.04259\.External Links:[Link](https://arxiv.org/abs/2509.04259)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Shiet al\.\(2025\)H\. Shi, Z\. Xu, H\. Wang, W\. Qin, W\. Wang, Y\. Wang, Z\. Wang, and H\. WangContinual Learning of Large Language Models: A Comprehensive Survey\.ACM Computing Surveys\.External Links:[Document](https://dx.doi.org/10.1145/3735633),[Link](https://arxiv.org/abs/2404.16789)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Tanget al\.\(2023\)Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. SunToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases\.arXiv preprint arXiv:2306\.05301\.External Links:[Link](https://arxiv.org/abs/2306.05301)Cited by:[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p1.1)\.
- Tarvainen and Valpola \(2017\)A\. Tarvainen and H\. ValpolaMean teachers are better role models: weight\-averaged consistency targets improve semi\-supervised deep learning results\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/68053af2923e00204c3ca7c6a3150cf7-Abstract.html)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§1](https://arxiv.org/html/2610.41104#S1.p6.1),[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Teamet al\.\(2025\)5\. Team, A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang, K\. Wang, L\. Zhong, M\. Liu, R\. Lu, S\. Cao, X\. Zhang, X\. Huang, Y\. Wei, Y\. Cheng, Y\. An, Y\. Niu, Y\. Wen, Y\. Bai, Z\. Du, Z\. Wang, Z\. Zhu, B\. Zhang, B\. Wen, B\. Wu, B\. Xu, C\. Huang, C\. Zhao, C\. Cai, C\. Yu, C\. Li, C\. Ge, C\. Huang, C\. Zhang, C\. Xu, C\. Zhu, C\. Li, C\. Yin, D\. Lin, D\. Yang, D\. Jiang, D\. Ai, E\. Zhu, F\. Wang, G\. Pan, G\. Wang, H\. Sun, H\. Li, H\. Li, H\. Hu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Wang, H\. Yang, H\. Liu, H\. Zhao, H\. Liu, H\. Yan, H\. Liu, H\. Chen, J\. Li, J\. Zhao, J\. Ren, J\. Jiao, J\. Zhao, J\. Yan, J\. Wang, J\. Gui, J\. Zhao, J\. Liu, J\. Li, J\. Li, J\. Lu, J\. Wang, J\. Yuan, J\. Li, J\. Du, J\. Du, J\. Liu, J\. Zhi, J\. Gao, K\. Wang, L\. Yang, L\. Xu, L\. Fan, L\. Wu, L\. Ding, L\. Wang, M\. Zhang, M\. Li, M\. Xu, M\. Zhao, M\. Zhai, P\. Du, Q\. Dong, S\. Lei, S\. Tu, S\. Yang, S\. Lu, S\. Li, S\. Li, Shuang\-Li, S\. Yang, S\. Yi, T\. Yu, W\. Tian, W\. Wang, W\. Yu, W\. L\. Tam, W\. Liang, W\. Liu, X\. Wang, X\. Jia, X\. Gu, X\. Ling, X\. Wang, X\. Fan, X\. Pan, X\. Zhang, X\. Zhang, X\. Fu, X\. Zhang, Y\. Xu, Y\. Wu, Y\. Lu, Y\. Wang, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Wang, Y\. Li, Y\. Su, Y\. Geng, Y\. Zhu, Y\. Yang, Y\. Li, Y\. Wu, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Zhang, Z\. Liu, Z\. Yang, Z\. Zhou, Z\. Qiao, Z\. Feng, Z\. Liu, Z\. Zhang, Z\. Wang, Z\. Yao, Z\. Wang, Z\. Liu, Z\. Chai, Z\. Li, Z\. Zhao, W\. Chen, J\. Zhai, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. TangGLM\-4\.5: agentic, reasoning, and coding \(arc\) foundation models\.External Links:2508\.06471,[Link](https://arxiv.org/abs/2508.06471)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1)\.
- Vyaset al\.\(2024\)N\. Vyas, D\. Morwani, R\. Zhao, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. KakadeSOAP: Improving and Stabilizing Shampoo using Adam\.arXiv preprint arXiv:2409\.11321\.External Links:[Link](https://arxiv.org/abs/2409.11321)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Wenet al\.\(2023\)Y\. Wen, Z\. Li, W\. Du, and L\. Mouf\-Divergence Minimization for Sequence\-Level Knowledge Distillation\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2307.15190)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Wortsmanet al\.\(2024\)M\. Wortsman, P\. J\. Liu, L\. Xiao, K\. Everett, A\. Alemi, B\. Adlam, J\. D\. Co\-Reyes, I\. Gur, A\. Kumar, R\. Novak, J\. Pennington, J\. N\. Sohl\-Dickstein,et al\.Small\-scale proxies for large\-scale Transformer training instabilities\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2309.14322)Cited by:[§A\.8](https://arxiv.org/html/2610.41104#A1.SS8.p1.1)\.
- Wuet al\.\(2025\)Y\. Wu, Y\. Zhou, Z\. Zhou, Y\. Peng, X\. Ye, X\. Hu, W\. Zhu, L\. Qi, M\. Yang, and X\. YangOn the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification\.arXiv preprint arXiv:2508\.05629\.External Links:[Link](https://arxiv.org/abs/2508.05629)Cited by:[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p2.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient Streaming Language Models with Attention Sinks\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2309.17453)Cited by:[§A\.8](https://arxiv.org/html/2610.41104#A1.SS8.p1.1)\.
- Xuet al\.\(2024\)W\. Xu, R\. Han, Z\. Wang, L\. T\. Le, D\. Madeka, L\. Li, W\. Y\. Wang, R\. Agarwal, C\. Lee, and T\. PfisterSpeculative Knowledge Distillation: Bridging the Teacher\-Student Gap Through Interleaved Sampling\.arXiv preprint arXiv:2410\.11325\.External Links:[Link](https://arxiv.org/abs/2410.11325)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Yanget al\.\(2024a\)Q\. A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin,et al\.Qwen2\.5 Technical Report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1](https://arxiv.org/html/2610.41104#S4.SS1.p1.1)\.
- Yanget al\.\(2024b\)Z\. Yang, Q\. Liu, T\. Pang, H\. Wang, H\. Feng, M\. Zhu, and W\. ChenSelf\-Distillation Bridges Distribution Gap in Language Model Fine\-Tuning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2402.13669)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Zhanget al\.\(2019\)M\. R\. Zhang, J\. Lucas, G\. Hinton, and J\. BaLookahead optimizer: k steps forward, 1 step back\.InAdvances in Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p3.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, C\. Chen, Z\. Li, T\. Ding, C\. Wu, D\. P\. Kingma, Y\. Ye, Z\. Luo, and R\. SunAdam\-mini: Use Fewer Learning Rates To Gain More\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2406.16793)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Zhaoet al\.\(2024\)J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. TianGaLore: memory\-efficient LLM training by gradient low\-rank projection\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 61121–61143\.External Links:[Link](https://proceedings.mlr.press/v235/zhao24s.html)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Zhaoet al\.\(2025\)R\. Zhao, D\. Morwani, D\. Brandfonbrener, N\. Vyas, and S\. KakadeDeconstructing What Makes a Good Optimizer for Language Models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2407.07972)Cited by:[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.
- Zhaoet al\.\(2023\)Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer, A\. Desmaison, C\. Balioglu,et al\.PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel\.Proceedings of the VLDB Endowment\.External Links:[Link](https://arxiv.org/abs/2304.11277)Cited by:[§3\.3](https://arxiv.org/html/2610.41104#S3.SS3.p2.1)\.
- Zhenget al\.\(2025a\)J\. Zheng, X\. Cai, S\. Qiu, and Q\. MaSpurious Forgetting in Continual Learning of Language Models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2501.13453)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1),[§5](https://arxiv.org/html/2610.41104#S5.p1.1)\.
- Zhenget al\.\(2025b\)J\. Zheng, S\. Qiu, C\. Shi, and Q\. MaTowards Lifelong Learning of Large Language Models: A Survey\.ACM Computing Surveys\.External Links:[Document](https://dx.doi.org/10.1145/3716629),[Link](https://arxiv.org/abs/2406.06391)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p1.1)\.
- Zhuet al\.\(2025\)H\. Zhu, Z\. \(\. Zhang, W\. Cong, X\. Liu, S\. Park, V\. Chandra, B\. Long, D\. Z\. Pan, Z\. Wang, and J\. LeeAPOLLO: SGD\-like Memory, AdamW\-level Performance\.InConference on Machine Learning and Systems \(MLSys\),External Links:[Link](https://arxiv.org/abs/2412.05270)Cited by:[§1](https://arxiv.org/html/2610.41104#S1.p2.1),[§5](https://arxiv.org/html/2610.41104#S5.p2.1)\.

## Appendix AAppendix

### A\.1Theoretical Analysis

This section provides remaining theoretical analysis\. The below proposition shows where this gain must come from\.

###### Proposition 1\(No instantaneous alignment gain\)\.

Let𝐐\\mathbf\{Q\}be the polar factor of𝐆≠𝟎\\mathbf\{G\}\\neq\\mathbf\{0\}\. Then𝐐∈arg⁡max‖𝐗‖2≤1⁡⟨𝐆,𝐗⟩F\\mathbf\{Q\}\\in\\arg\\max\_\{\\\|\\mathbf\{X\}\\\|\_\{2\}\\leq 1\}\\langle\\mathbf\{G\},\\mathbf\{X\}\\rangle\_\{\\mathrm\{F\}\}, with maximum‖𝐆‖∗\\\|\\mathbf\{G\}\\\|\_\{\*\}\. Hence any teacher direction with‖𝐏‖2≤1\\\|\\mathbf\{P\}\\\|\_\{2\}\\leq 1satisfies⟨𝐆,𝐏⟩F≤⟨𝐆,𝐐⟩F\\langle\\mathbf\{G\},\\mathbf\{P\}\\rangle\_\{\\mathrm\{F\}\}\\leq\\langle\\mathbf\{G\},\\mathbf\{Q\}\\rangle\_\{\\mathrm\{F\}\}\.

So the Newton–Schulz direction is already optimal for the current gradient, and the teacher branch contributes what a single gradient cannot: temporal information\. To make this precise, writeFsF\_\{s\}for the expected host loss at stepss\(it moves with the teacher\),Fs⋆=infFsF\_\{s\}^\{\\star\}=\\inf F\_\{s\}andΔs=Fs​\(𝐖S\(s\)\)−Fs⋆\\Delta\_\{s\}=F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-F\_\{s\}^\{\\star\}\. We assume \(A1\)LL\-smoothness, \(A2\) the PL condition12​‖∇Fs‖F2≥μPL​\(Fs−Fs⋆\)\\frac\{1\}\{2\}\\\|\\nabla F\_\{s\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\geq\\mu\_\{\\mathrm\{PL\}\}\(F\_\{s\}\-F\_\{s\}^\{\\star\}\)\([Karimi et al\., 2016](https://arxiv.org/html/2610.41104#bib.bib51)\), \(A3\) bounded teacher driftsup𝐖\|Fs\+1−Fs\|≤δs\\sup\_\{\\mathbf\{W\}\}\|F\_\{s\+1\}\-F\_\{s\}\|\\leq\\delta\_\{s\}, and \(A4\) alignment𝔼⁡⟨𝐠\(s\),𝐔\(s\)⟩≥aR​‖𝐠\(s\)‖2\\mathbb\{E\}\\langle\\mathbf\{g\}^\{\(s\)\},\\mathbf\{U\}^\{\(s\)\}\\rangle\\geq a\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|^\{2\}with second moment𝔼​‖𝐔\(s\)‖2≤bR​‖𝐠\(s\)‖2\+σR2\\mathbb\{E\}\\\|\\mathbf\{U\}^\{\(s\)\}\\\|^\{2\}\\leq b\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|^\{2\}\+\\sigma\_\{R\}^\{2\}, where𝐠\(s\)=∇Fs​\(𝐖S\(s\)\)\\mathbf\{g\}^\{\(s\)\}=\\nabla F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\.

The below corollary further shows when reuse of the teacher parameters are faster:

###### Corollary 1\(When reuse is faster\)\.

Let a baseline optimizer \(e\.g\. momentum\-free Muon\) satisfy A1–A4 withγB\>0\\gamma\_\{B\}\>0at the sameη\\eta\. IfγR\>γB\\gamma\_\{R\}\>\\gamma\_\{B\}, the Muon\-RT bound contracts strictly faster \(qR<qBq\_\{R\}<q\_\{B\}\); if, in addition, its neighborhood term is smaller, it also reaches a lower error floor\.

Corollary[1](https://arxiv.org/html/2610.41104#Thmcorollary1)shows that reuse helps when the teacher branch raises the effective descent coefficient, i\.e\. adds alignment with the true gradient over time without inflating the update’s second moment\.

### A\.2Theoretical Analysis: Proofs and Convergence

This appendix proves the results stated in Section[3\.4](https://arxiv.org/html/2610.41104#S3.SS4)and gives the full assumptions behind them\. Notation follows Section[3](https://arxiv.org/html/2610.41104#S3)\.

#### A\.2\.1Proof of Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1)

###### Proof\.

Define the displacement

𝐃\(s\)=𝐖T\(s\)−𝐖S\(s\)\.\\mathbf\{D\}^\{\(s\)\}=\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}\.\(14\)Using the EMA update, Eq\. \([3](https://arxiv.org/html/2610.41104#S2.E3)\),

𝐃\(s\)\\displaystyle\\mathbf\{D\}^\{\(s\)\}=τ​𝐖T\(s−1\)\+\(1−τ\)​𝐖S\(s\)−𝐖S\(s\)\\displaystyle=\\tau\\mathbf\{W\}\_\{T\}^\{\(s\-1\)\}\+\(1\-\\tau\)\\mathbf\{W\}\_\{S\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}=τ⁡\(𝐖T\(s−1\)−𝐖S\(s\)\)\\displaystyle=\\tau\\left\(\\mathbf\{W\}\_\{T\}^\{\(s\-1\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\right\)=τ⁡\(𝐖T\(s−1\)−𝐖S\(s−1\)\)\\displaystyle=\\tau\\left\(\\mathbf\{W\}\_\{T\}^\{\(s\-1\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\-1\)\}\\right\)\+τ⁡\(𝐖S\(s−1\)−𝐖S\(s\)\)\\displaystyle\\quad\+\\tau\\left\(\\mathbf\{W\}\_\{S\}^\{\(s\-1\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\right\)=τ⁡\(𝐃\(s−1\)\+η​cA,B​𝐔\(s−1\)\),\\displaystyle=\\tau\\left\(\\mathbf\{D\}^\{\(s\-1\)\}\+\\eta c\_\{A,B\}\\,\\mathbf\{U\}^\{\(s\-1\)\}\\right\),\(15\)where the last equality follows from the student update of Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1)\. Since𝐃\(0\)=𝟎\\mathbf\{D\}^\{\(0\)\}=\\mathbf\{0\}, recursively expanding Eq\. \([15](https://arxiv.org/html/2610.41104#A1.E15)\) gives

𝐃\(s\)=τ​η​cA,B​∑j=0s−1τj​𝐔\(s−1−j\),\\mathbf\{D\}^\{\(s\)\}=\\tau\\eta c\_\{A,B\}\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\mathbf\{U\}^\{\(s\-1\-j\)\},\(16\)which is Eq\. \([7](https://arxiv.org/html/2610.41104#S3.E7)\)\. ∎

#### A\.2\.2Proof of Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2)

###### Proof\.

Substituting the local model𝐔\(s\)=𝐝\+𝝃\(s\)\\mathbf\{U\}^\{\(s\)\}=\\mathbf\{d\}\+\\boldsymbol\{\\xi\}^\{\(s\)\}of Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2)into the definition of𝐔¯\(s\)\\overline\{\\mathbf\{U\}\}^\{\(s\)\},

𝐔¯\(s\)\\displaystyle\\overline\{\\mathbf\{U\}\}^\{\(s\)\}=\(1−τ\)​∑j=0s−1τj​\(𝐝\+𝝃\(s−1−j\)\)\\displaystyle=\(1\-\\tau\)\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\left\(\\mathbf\{d\}\+\\boldsymbol\{\\xi\}^\{\(s\-1\-j\)\}\\right\)=\(1−τ\)​\(∑j=0s−1τj\)​𝐝\\displaystyle=\(1\-\\tau\)\\left\(\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\right\)\\mathbf\{d\}\+\(1−τ\)∑j=0s−1τj𝝃\(s−1−j\)\.\\displaystyle\\quad\+\(1\-\\tau\)\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\boldsymbol\{\\xi\}^\{\(s\-1\-j\)\}\.\(17\)Since

\(1−τ\)​∑j=0s−1τj=1−τs,\(1\-\\tau\)\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}=1\-\\tau^\{s\},\(18\)taking expectations gives the stated mean\. After subtracting the mean,

𝐔¯\(s\)−𝔼⁡\[𝐔¯\(s\)\]=\(1−τ\)​∑j=0s−1τj​𝝃\(s−1−j\)\.\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\-\\mathbb\{E\}\[\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\]=\(1\-\\tau\)\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\boldsymbol\{\\xi\}^\{\(s\-1\-j\)\}\.\(19\)Therefore,

𝔼⁡\[‖𝐔¯\(s\)−𝔼⁡\[𝐔¯\(s\)\]‖F2\]\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\-\\mathbb\{E\}\[\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\]\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\]=\(1−τ\)2​𝔼​\[‖∑j=0s−1τj​𝝃\(s−1−j\)‖F2\]\.\\displaystyle\\quad=\(1\-\\tau\)^\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\sum\_\{j=0\}^\{s\-1\}\\tau^\{j\}\\boldsymbol\{\\xi\}^\{\(s\-1\-j\)\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\]\.\(20\)All cross terms vanish by temporal uncorrelatedness, yielding

𝔼⁡\[‖𝐔¯\(s\)−𝔼⁡\[𝐔¯\(s\)\]‖F2\]\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\-\\mathbb\{E\}\[\\overline\{\\mathbf\{U\}\}^\{\(s\)\}\]\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\]=\(1−τ\)2​∑j=0s−1τ2​j​σξ2\\displaystyle\\quad=\(1\-\\tau\)^\{2\}\\sum\_\{j=0\}^\{s\-1\}\\tau^\{2j\}\\sigma\_\{\\xi\}^\{2\}=\(1−τ\)2​1−τ2​s1−τ2​σξ2\\displaystyle\\quad=\(1\-\\tau\)^\{2\}\\frac\{1\-\\tau^\{2s\}\}\{1\-\\tau^\{2\}\}\\sigma\_\{\\xi\}^\{2\}=1−τ1\+τ​\(1−τ2​s\)​σξ2,\\displaystyle\\quad=\\frac\{1\-\\tau\}\{1\+\\tau\}\\left\(1\-\\tau^\{2s\}\\right\)\\sigma\_\{\\xi\}^\{2\},\(21\)which proves the variance statement of Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2)\. Lettings→∞s\\rightarrow\\inftygives the stated limit\. ∎

#### A\.2\.3Instantaneous alignment is not sufficient to explain the gain

The following proposition is useful for interpreting the role of the EMA branch\.

###### Restatement\(Proposition[1](https://arxiv.org/html/2610.41104#Thmproposition1)\)\.

Let𝐆=𝐋G​𝚺​𝐕G⊤\\mathbf\{G\}=\\mathbf\{L\}\_\{G\}\\boldsymbol\{\\Sigma\}\\mathbf\{V\}\_\{G\}^\{\\top\}be a compact singular value decomposition of𝐆≠𝟎\\mathbf\{G\}\\neq\\mathbf\{0\}and define its exact polar factor

𝐐=𝐋G​𝐕G⊤\.\\mathbf\{Q\}=\\mathbf\{L\}\_\{G\}\\mathbf\{V\}\_\{G\}^\{\\top\}\.\(22\)Then

𝐐∈arg⁡max‖𝐗‖2≤1⁡⟨𝐆,𝐗⟩F,\\boxed\{\\mathbf\{Q\}\\in\\arg\\max\_\{\\\|\\mathbf\{X\}\\\|\_\{2\}\\leq 1\}\\left\\langle\\mathbf\{G\},\\mathbf\{X\}\\right\\rangle\_\{\\mathrm\{F\}\},\}\(23\)and

⟨𝐆,𝐐⟩F=‖𝐆‖∗,\\left\\langle\\mathbf\{G\},\\mathbf\{Q\}\\right\\rangle\_\{\\mathrm\{F\}\}=\\\|\\mathbf\{G\}\\\|\_\{\*\},\(24\)where∥⋅∥∗\\\|\\cdot\\\|\_\{\*\}denotes the nuclear norm\. Therefore, for any spectrally normalized teacher direction𝐏\\mathbf\{P\}satisfying‖𝐏‖2≤1\\\|\\mathbf\{P\}\\\|\_\{2\}\\leq 1,

⟨𝐆,𝐏⟩F≤⟨𝐆,𝐐⟩F\.\\left\\langle\\mathbf\{G\},\\mathbf\{P\}\\right\\rangle\_\{\\mathrm\{F\}\}\\leq\\left\\langle\\mathbf\{G\},\\mathbf\{Q\}\\right\\rangle\_\{\\mathrm\{F\}\}\.\(25\)

###### Proof\.

By the duality between the nuclear and spectral norms,

‖𝐆‖∗=max‖𝐗‖2≤1⁡⟨𝐆,𝐗⟩F\.\\\|\\mathbf\{G\}\\\|\_\{\*\}=\\max\_\{\\\|\\mathbf\{X\}\\\|\_\{2\}\\leq 1\}\\left\\langle\\mathbf\{G\},\\mathbf\{X\}\\right\\rangle\_\{\\mathrm\{F\}\}\.\(26\)For𝐐=𝐋G​𝐕G⊤\\mathbf\{Q\}=\\mathbf\{L\}\_\{G\}\\mathbf\{V\}\_\{G\}^\{\\top\},‖𝐐‖2=1\\\|\\mathbf\{Q\}\\\|\_\{2\}=1, and

⟨𝐆,𝐐⟩F\\displaystyle\\left\\langle\\mathbf\{G\},\\mathbf\{Q\}\\right\\rangle\_\{\\mathrm\{F\}\}=tr⁡\(𝐆⊤​𝐐\)\\displaystyle=\\operatorname\{tr\}\\left\(\\mathbf\{G\}^\{\\top\}\\mathbf\{Q\}\\right\)=tr⁡\(𝐕G​𝚺​𝐋G⊤​𝐋G​𝐕G⊤\)\\displaystyle=\\operatorname\{tr\}\\left\(\\mathbf\{V\}\_\{G\}\\boldsymbol\{\\Sigma\}\\mathbf\{L\}\_\{G\}^\{\\top\}\\mathbf\{L\}\_\{G\}\\mathbf\{V\}\_\{G\}^\{\\top\}\\right\)=tr⁡\(𝚺\)=‖𝐆‖∗\.\\displaystyle=\\operatorname\{tr\}\\left\(\\boldsymbol\{\\Sigma\}\\right\)=\\\|\\mathbf\{G\}\\\|\_\{\*\}\.\(27\)Thus𝐐\\mathbf\{Q\}attains the spectral\-norm\-constrained maximum, proving Eqs\. \([23](https://arxiv.org/html/2610.41104#A1.E23)\)–\([24](https://arxiv.org/html/2610.41104#A1.E24)\)\. Since‖𝐏‖2≤1\\\|\\mathbf\{P\}\\\|\_\{2\}\\leq 1, Eq\. \([25](https://arxiv.org/html/2610.41104#A1.E25)\) follows immediately\. ∎

#### A\.2\.4Finite\-time convergence of Muon\-RT

We now analyze the complete mixed direction of Eq\. \([11](https://arxiv.org/html/2610.41104#S3.E11)\), written𝐔R\(s\)\\mathbf\{U\}\_\{R\}^\{\(s\)\}to distinguish it from baseline directions,

𝐔R\(s\)=\(1−α\)​𝐎\(s\)\+α​𝐏\(s\),α∈\[0,1\),\\mathbf\{U\}\_\{R\}^\{\(s\)\}=\(1\-\\alpha\)\\mathbf\{O\}^\{\(s\)\}\+\\alpha\\mathbf\{P\}^\{\(s\)\},\\qquad\\alpha\\in\[0,1\),\(28\)where𝐎\(s\)\\mathbf\{O\}^\{\(s\)\}is the momentum\-free Muon branch and𝐏\(s\)\\mathbf\{P\}^\{\(s\)\}is the normalized EMA\-trajectory branch\.

The host objective may itself move because the teacher changes\. With𝐖\\mathbf\{W\}denoting the Muon\-eligible matrix under analysis, we therefore write the expected host objective as

Fs​\(𝐖\)=𝔼ℬ​\[ℒ⁡\(𝐖,𝐖T\(s\),ℬ\)\],F\_\{s\}\(\\mathbf\{W\}\)=\\mathbb\{E\}\_\{\\mathcal\{B\}\}\\left\[\\mathcal\{L\}\\left\(\\mathbf\{W\},\\mathbf\{W\}\_\{T\}^\{\(s\)\};\\mathcal\{B\}\\right\)\\right\],\(29\)with

Fs⋆=inf𝐖Fs​\(𝐖\),Δs=Fs​\(𝐖S\(s\)\)−Fs⋆\.F\_\{s\}^\{\\star\}=\\inf\_\{\\mathbf\{W\}\}F\_\{s\}\(\\mathbf\{W\}\),\\qquad\\Delta\_\{s\}=F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-F\_\{s\}^\{\\star\}\.\(30\)
We use the following standard local conditions over the region visited during training\.

1. A1\.Smoothness\.EveryFsF\_\{s\}isLL\-smooth with respect to the Frobenius norm: Fs​\(𝐘\)≤Fs​\(𝐗\)\+⟨∇Fs​\(𝐗\),𝐘−𝐗⟩F\+L2​‖𝐘−𝐗‖F2\.\\begin\{split\}F\_\{s\}\(\\mathbf\{Y\}\)\\leq\{\}&F\_\{s\}\(\\mathbf\{X\}\)\+\\left\\langle\\nabla F\_\{s\}\(\\mathbf\{X\}\),\\mathbf\{Y\}\-\\mathbf\{X\}\\right\\rangle\_\{\\mathrm\{F\}\}\\\\ &\+\\frac\{L\}\{2\}\\\|\\mathbf\{Y\}\-\\mathbf\{X\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\\end\{split\}\(31\)
2. A2\.PL condition\.There existsμPL\>0\\mu\_\{\\mathrm\{PL\}\}\>0such that 12​‖∇Fs​\(𝐖\)‖F2≥μPL​\(Fs​\(𝐖\)−Fs⋆\)\\frac\{1\}\{2\}\\left\\\|\\nabla F\_\{s\}\(\\mathbf\{W\}\)\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\geq\\mu\_\{\\mathrm\{PL\}\}\\left\(F\_\{s\}\(\\mathbf\{W\}\)\-F\_\{s\}^\{\\star\}\\right\)\(32\)for all iterates under consideration\.
3. A3\.Controlled teacher\-induced objective drift\.There existsδs≥0\\delta\_\{s\}\\geq 0such that sup𝐖\|Fs\+1​\(𝐖\)−Fs​\(𝐖\)\|≤δs\.\\sup\_\{\\mathbf\{W\}\}\\left\|F\_\{s\+1\}\(\\mathbf\{W\}\)\-F\_\{s\}\(\\mathbf\{W\}\)\\right\|\\leq\\delta\_\{s\}\.\(33\)
4. A4\.Descent alignment and second moment\.Let 𝐠\(s\)=∇Fs​\(𝐖S\(s\)\)\.\\mathbf\{g\}^\{\(s\)\}=\\nabla F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\.\(34\)Conditioned on the optimization historyℱs\\mathcal\{F\}\_\{s\}, the Muon\-RT direction satisfies 𝔼⁡\[⟨𝐠\(s\),𝐔R\(s\)⟩F\|ℱs\]≥aR​‖𝐠\(s\)‖F2,\\mathbb\{E\}\\left\[\\left\\langle\\mathbf\{g\}^\{\(s\)\},\\mathbf\{U\}\_\{R\}^\{\(s\)\}\\right\\rangle\_\{\\mathrm\{F\}\}\\,\\middle\|\\,\\mathcal\{F\}\_\{s\}\\right\]\\geq a\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\(35\)and 𝔼⁡\[‖𝐔R\(s\)‖F2\|ℱs\]≤bR​‖𝐠\(s\)‖F2\+σR2\.\\mathbb\{E\}\\left\[\\\|\\mathbf\{U\}\_\{R\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\,\\middle\|\\,\\mathcal\{F\}\_\{s\}\\right\]\\leq b\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\sigma\_\{R\}^\{2\}\.\(36\)

The constants in A4 have a direct interpretation\.aRa\_\{R\}measures expected alignment with the current true descent direction,bRb\_\{R\}measures how much update energy grows with the gradient, andσR2\\sigma\_\{R\}^\{2\}collects residual update energy that does not scale with the gradient\. Because𝐎\(s\)\\mathbf\{O\}^\{\(s\)\}and𝐏\(s\)\\mathbf\{P\}^\{\(s\)\}are normalized, this residual includes their gradient\-independent magnitude as well as stochastic noise, andaRa\_\{R\}is understood over a region in which‖𝐠\(s\)‖F\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}is bounded\. A conservative set of valid coefficients can be obtained from separate branch bounds\. In particular, if

𝔼⁡\[⟨𝐠\(s\),𝐎\(s\)⟩F∣ℱs\]≥aO​‖𝐠\(s\)‖F2,\\mathbb\{E\}\\left\[\\langle\\mathbf\{g\}^\{\(s\)\},\\mathbf\{O\}^\{\(s\)\}\\rangle\_\{\\mathrm\{F\}\}\\mid\\mathcal\{F\}\_\{s\}\\right\]\\geq a\_\{O\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\(37\)𝔼⁡\[⟨𝐠\(s\),𝐏\(s\)⟩F∣ℱs\]≥aP​‖𝐠\(s\)‖F2,\\mathbb\{E\}\\left\[\\langle\\mathbf\{g\}^\{\(s\)\},\\mathbf\{P\}^\{\(s\)\}\\rangle\_\{\\mathrm\{F\}\}\\mid\\mathcal\{F\}\_\{s\}\\right\]\\geq a\_\{P\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\(38\)and

𝔼⁡\[‖𝐎\(s\)‖F2∣ℱs\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\mathbf\{O\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\mid\\mathcal\{F\}\_\{s\}\\right\]≤bO​‖𝐠\(s\)‖F2\+σO2,\\displaystyle\\leq b\_\{O\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\sigma\_\{O\}^\{2\},\(39\)𝔼⁡\[‖𝐏\(s\)‖F2∣ℱs\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\mathbf\{P\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\mid\\mathcal\{F\}\_\{s\}\\right\]≤bP​‖𝐠\(s\)‖F2\+σP2,\\displaystyle\\leq b\_\{P\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\sigma\_\{P\}^\{2\},\(40\)whereaPa\_\{P\}may be negative, then linearity of the inner product and convexity of the squared norm imply

aR=\(1−α\)​aO\+α​aP,a\_\{R\}=\(1\-\\alpha\)a\_\{O\}\+\\alpha a\_\{P\},\(41\)bR=\(1−α\)​bO\+α​bP,σR2=\(1−α\)​σO2\+α​σP2\.\\begin\{split\}b\_\{R\}&=\(1\-\\alpha\)b\_\{O\}\+\\alpha b\_\{P\},\\\\ \\sigma\_\{R\}^\{2\}&=\(1\-\\alpha\)\\sigma\_\{O\}^\{2\}\+\\alpha\\sigma\_\{P\}^\{2\}\.\\end\{split\}\(42\)These are conservative because they do not exploit any favorable cancellation between𝐎\(s\)\\mathbf\{O\}^\{\(s\)\}and𝐏\(s\)\\mathbf\{P\}^\{\(s\)\}\.

###### Restatement\(Theorem[1](https://arxiv.org/html/2610.41104#Thmtheorem1)\)\.

Suppose A1–A4 hold and the Muon\-RT student update is

𝐖S\(s\+1\)=𝐖S\(s\)−η​𝐔R\(s\)\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}=\\mathbf\{W\}\_\{S\}^\{\(s\)\}\-\\eta\\mathbf\{U\}\_\{R\}^\{\(s\)\}\(43\)with constantη\>0\\eta\>0, i\.e\., Eq\. \([12](https://arxiv.org/html/2610.41104#S3.E12)\) withλ=0\\lambda=0andcA,Bc\_\{A,B\}absorbed intoη\\eta\. Define the effective descent coefficient

γR=aR−L​η2​bR\.\\boxed\{\\gamma\_\{R\}=a\_\{R\}\-\\frac\{L\\eta\}\{2\}b\_\{R\}\.\}\(44\)If

γR\>0and0<2​μPL​η​γR<1,\\gamma\_\{R\}\>0\\qquad\\text\{and\}\\qquad 0<2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}<1,\(45\)then

𝔼⁡\[Δs\+1\]≤qR​𝔼​\[Δs\]\+L​η22​σR2\+2​δs,\\boxed\{\\mathbb\{E\}\[\\Delta\_\{s\+1\}\]\\leq q\_\{R\}\\,\\mathbb\{E\}\[\\Delta\_\{s\}\]\+\\frac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\+2\\delta\_\{s\},\}\(46\)where

qR=1−2​μPL​η​γR∈\(0,1\)\.q\_\{R\}=1\-2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\\in\(0,1\)\.\(47\)Consequently, for anyn≥1n\\geq 1,

𝔼⁡\[Δn\]≤qRn​Δ0\+∑s=0n−1qRn−1−s\(L​η22σR2\+2δs\)\.\\boxed\{\\begin\{aligned\} \\mathbb\{E\}\[\\Delta\_\{n\}\]\\leq\{\}&q\_\{R\}^\{n\}\\Delta\_\{0\}\\\\ &\+\\sum\_\{s=0\}^\{n\-1\}q\_\{R\}^\{n\-1\-s\}\\left\(\\frac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\+2\\delta\_\{s\}\\right\)\.\\end\{aligned\}\}\(48\)If the objective drift is uniformly bounded byδs≤δ¯\\delta\_\{s\}\\leq\\bar\{\\delta\}, then

𝔼⁡\[Δn\]≤qRn​Δ0\+L​η22​σR2\+2​δ¯1−qR​\(1−qRn\)\.\\boxed\{\\begin\{aligned\} \\mathbb\{E\}\[\\Delta\_\{n\}\]\\leq\{\}&q\_\{R\}^\{n\}\\Delta\_\{0\}\\\\ &\+\\frac\{\\frac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\+2\\bar\{\\delta\}\}\{1\-q\_\{R\}\}\\left\(1\-q\_\{R\}^\{n\}\\right\)\.\\end\{aligned\}\}\(49\)In particular,

lim supn→∞𝔼⁡\[Δn\]≤L​η​σR24​μPL​γR\+δ¯μPL​η​γR\.\\boxed\{\\limsup\_\{n\\rightarrow\\infty\}\\mathbb\{E\}\[\\Delta\_\{n\}\]\\leq\\frac\{L\\eta\\sigma\_\{R\}^\{2\}\}\{4\\mu\_\{\\mathrm\{PL\}\}\\gamma\_\{R\}\}\+\\frac\{\\bar\{\\delta\}\}\{\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\}\.\}\(50\)For a fixed objective,δ¯=0\\bar\{\\delta\}=0\.

###### Proof\.

ByLL\-smoothness ofFsF\_\{s\}and Eq\. \([43](https://arxiv.org/html/2610.41104#A1.E43)\),

Fs​\(𝐖S\(s\+1\)\)≤\\displaystyle F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\\leq\{\}Fs​\(𝐖S\(s\)\)−η​⟨𝐠\(s\),𝐔R\(s\)⟩F\\displaystyle F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-\\eta\\left\\langle\\mathbf\{g\}^\{\(s\)\},\\mathbf\{U\}\_\{R\}^\{\(s\)\}\\right\\rangle\_\{\\mathrm\{F\}\}\+L​η22​‖𝐔R\(s\)‖F2\.\\displaystyle\+\\frac\{L\\eta^\{2\}\}\{2\}\\\|\\mathbf\{U\}\_\{R\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(51\)Conditioning onℱs\\mathcal\{F\}\_\{s\}and applying Eqs\. \([35](https://arxiv.org/html/2610.41104#A1.E35)\) and \([36](https://arxiv.org/html/2610.41104#A1.E36)\),

𝔼⁡\[Fs​\(𝐖S\(s\+1\)\)∣ℱs\]≤\\displaystyle\\mathbb\{E\}\\left\[F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\\mid\\mathcal\{F\}\_\{s\}\\right\]\\leqFs​\(𝐖S\(s\)\)−η​aR​‖𝐠\(s\)‖F2\\displaystyle\\quad F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-\\eta a\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+L​η22​\(bR​‖𝐠\(s\)‖F2\+σR2\)\\displaystyle\\quad\+\\frac\{L\\eta^\{2\}\}\{2\}\\left\(b\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\sigma\_\{R\}^\{2\}\\right\)=Fs​\(𝐖S\(s\)\)−η⁡\(aR−L​η2​bR\)​‖𝐠\(s\)‖F2\\displaystyle=F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-\\eta\\left\(a\_\{R\}\-\\frac\{L\\eta\}\{2\}b\_\{R\}\\right\)\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+L​η22​σR2\\displaystyle\\quad\+\\frac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}=Fs​\(𝐖S\(s\)\)−η​γR​‖𝐠\(s\)‖F2\+L​η22​σR2\.\\displaystyle=F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-\\eta\\gamma\_\{R\}\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\frac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\.\(52\)Using the PL inequality in Eq\. \([32](https://arxiv.org/html/2610.41104#A1.E32)\),

‖𝐠\(s\)‖F2≥2​μPL​\(Fs​\(𝐖S\(s\)\)−Fs⋆\)=2​μPL​Δs\.\\\|\\mathbf\{g\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\geq 2\\mu\_\{\\mathrm\{PL\}\}\\left\(F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\)\}\)\-F\_\{s\}^\{\\star\}\\right\)=2\\mu\_\{\\mathrm\{PL\}\}\\Delta\_\{s\}\.\(53\)Substituting this into Eq\. \([52](https://arxiv.org/html/2610.41104#A1.E52)\) gives

𝔼⁡\[Fs​\(𝐖S\(s\+1\)\)−Fs⋆∣ℱs\]≤qR​Δs\+L​η22​σR2\.\\begin\{split\}&\\mathbb\{E\}\\left\[F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\-F\_\{s\}^\{\\star\}\\mid\\mathcal\{F\}\_\{s\}\\right\]\\\\ &\\quad\\leq q\_\{R\}\\Delta\_\{s\}\+\\frac\{L\\eta^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\.\\end\{split\}\(54\)
It remains to account for the fact that the EMA\-teacher changes the host objective between steps\. From A3,

Fs\+1​\(𝐖S\(s\+1\)\)≤Fs​\(𝐖S\(s\+1\)\)\+δs\.F\_\{s\+1\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\\leq F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\+\\delta\_\{s\}\.\(55\)Moreover,

Fs\+1⋆=inf𝐖Fs\+1​\(𝐖\)≥inf𝐖\(Fs​\(𝐖\)−δs\)=Fs⋆−δs\.\\begin\{split\}F\_\{s\+1\}^\{\\star\}&=\\inf\_\{\\mathbf\{W\}\}F\_\{s\+1\}\(\\mathbf\{W\}\)\\\\ &\\geq\\inf\_\{\\mathbf\{W\}\}\\left\(F\_\{s\}\(\\mathbf\{W\}\)\-\\delta\_\{s\}\\right\)=F\_\{s\}^\{\\star\}\-\\delta\_\{s\}\.\\end\{split\}\(56\)Combining Eqs\. \([55](https://arxiv.org/html/2610.41104#A1.E55)\) and \([56](https://arxiv.org/html/2610.41104#A1.E56)\),

Δs\+1\\displaystyle\\Delta\_\{s\+1\}=Fs\+1​\(𝐖S\(s\+1\)\)−Fs\+1⋆\\displaystyle=F\_\{s\+1\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\-F\_\{s\+1\}^\{\\star\}≤Fs​\(𝐖S\(s\+1\)\)−Fs⋆\+2​δs\.\\displaystyle\\leq F\_\{s\}\(\\mathbf\{W\}\_\{S\}^\{\(s\+1\)\}\)\-F\_\{s\}^\{\\star\}\+2\\delta\_\{s\}\.\(57\)Taking total expectation in Eq\. \([54](https://arxiv.org/html/2610.41104#A1.E54)\) and applying Eq\. \([57](https://arxiv.org/html/2610.41104#A1.E57)\) yields Eq\. \([46](https://arxiv.org/html/2610.41104#A1.E46)\)\.

Repeatedly applying the recursion gives Eq\. \([48](https://arxiv.org/html/2610.41104#A1.E48)\)\. Whenδs≤δ¯\\delta\_\{s\}\\leq\\bar\{\\delta\}, the geometric\-series identity

∑s=0n−1qRn−1−s=1−qRn1−qR\\sum\_\{s=0\}^\{n\-1\}q\_\{R\}^\{n\-1\-s\}=\\frac\{1\-q\_\{R\}^\{n\}\}\{1\-q\_\{R\}\}\(58\)gives Eq\. \([49](https://arxiv.org/html/2610.41104#A1.E49)\)\. Finally, using

1−qR=2​μPL​η​γR1\-q\_\{R\}=2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\(59\)and takingn→∞n\\rightarrow\\inftyyields Eq\. \([50](https://arxiv.org/html/2610.41104#A1.E50)\)\. ∎

#### A\.2\.5Comparison with a baseline optimizer

The previous result immediately gives a sufficient condition for a faster convergence guarantee than AdamW, momentum Muon, or momentum\-free Muon, provided the corresponding baseline satisfies the same local regularity assumptions\.

###### Restatement\(Corollary[1](https://arxiv.org/html/2610.41104#Thmcorollary1)\)\.

Let a baseline optimizerBBsatisfy A1–A4 at the same step sizeη\\etawith coefficientsaB,bB,σB2a\_\{B\},b\_\{B\},\\sigma\_\{B\}^\{2\}and objective\-drift boundδ¯B\\bar\{\\delta\}\_\{B\}\. Define

γB=aB−L​η2​bB,qB=1−2​μPL​η​γB\.\\begin\{split\}\\gamma\_\{B\}&=a\_\{B\}\-\\frac\{L\\eta\}\{2\}b\_\{B\},\\\\ q\_\{B\}&=1\-2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{B\}\.\\end\{split\}\(60\)If

γR\>γB\>0,\\boxed\{\\gamma\_\{R\}\>\\gamma\_\{B\}\>0,\}\(61\)then

so the Muon\-RT bound contracts strictly faster per optimization step\.

Furthermore, define the asymptotic upper\-bound terms

ΦR=L​η​σR24​μPL​γR\+δ¯RμPL​η​γR,\\Phi\_\{R\}=\\frac\{L\\eta\\sigma\_\{R\}^\{2\}\}\{4\\mu\_\{\\mathrm\{PL\}\}\\gamma\_\{R\}\}\+\\frac\{\\bar\{\\delta\}\_\{R\}\}\{\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\},\(63\)and

ΦB=L​η​σB24​μPL​γB\+δ¯BμPL​η​γB\.\\Phi\_\{B\}=\\frac\{L\\eta\\sigma\_\{B\}^\{2\}\}\{4\\mu\_\{\\mathrm\{PL\}\}\\gamma\_\{B\}\}\+\\frac\{\\bar\{\\delta\}\_\{B\}\}\{\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{B\}\}\.\(64\)If, in addition,

ΦR<ΦB,\\boxed\{\\Phi\_\{R\}<\\Phi\_\{B\},\}\(65\)then the Muon\-RT convergence bound has both a faster transient contraction and a strictly smaller asymptotic optimization neighborhood\.

###### Proof\.

Equation \([61](https://arxiv.org/html/2610.41104#A1.E61)\) implies

2​μPL​η​γR\>2​μPL​η​γB,2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\>2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{B\},\(66\)and therefore

qR=1−2​μPL​η​γR<1−2​μPL​η​γB=qB\.\\begin\{split\}q\_\{R\}&=1\-2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\\\\ &<1\-2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{B\}=q\_\{B\}\.\\end\{split\}\(67\)Hence the geometric transientqRnq\_\{R\}^\{n\}decays more rapidly thanqBnq\_\{B\}^\{n\}\. The second statement follows directly from Theorem[1](https://arxiv.org/html/2610.41104#Thmtheorem1):ΦR\\Phi\_\{R\}andΦB\\Phi\_\{B\}are the respective asymptotic upper bounds obtained by taking the limit superior of each finite\-time recursion\. IfΦR<ΦB\\Phi\_\{R\}<\\Phi\_\{B\}, the Muon\-RT bound approaches a smaller optimization neighborhood\. ∎

For the fixed\-objective caseδ¯R=δ¯B=0\\bar\{\\delta\}\_\{R\}=\\bar\{\\delta\}\_\{B\}=0, Eq\. \([65](https://arxiv.org/html/2610.41104#A1.E65)\) reduces to

σR2γR<σB2γB\.\\boxed\{\\frac\{\\sigma\_\{R\}^\{2\}\}\{\\gamma\_\{R\}\}<\\frac\{\\sigma\_\{B\}^\{2\}\}\{\\gamma\_\{B\}\}\.\}\(68\)Thus the relevant quantity is not variance alone, but the ratio between residual update energy and effective descent\. This formalizes the optimization mechanism suggested by Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2): a temporally smoothed trajectory is useful when the reduction in update noise is large enough relative to any loss of instantaneous descent alignment\.

#### A\.2\.6Iteration complexity

Because the branches are normalized,σR2\>0\\sigma\_\{R\}^\{2\}\>0and the bound converges to the neighborhoodΦR\\Phi\_\{R\}rather than to zero\. From Eq\. \([49](https://arxiv.org/html/2610.41104#A1.E49)\),

𝔼⁡\[Δn\]≤qRn​Δ0\+ΦR\.\\mathbb\{E\}\[\\Delta\_\{n\}\]\\leq q\_\{R\}^\{n\}\\Delta\_\{0\}\+\\Phi\_\{R\}\.\(69\)To reach𝔼⁡\[Δn\]≤ΦR\+ϵtol\\mathbb\{E\}\[\\Delta\_\{n\}\]\\leq\\Phi\_\{R\}\+\\epsilon\_\{\\mathrm\{tol\}\}, it is sufficient that

nR​\(ϵtol\)≥log⁡\(Δ0/ϵtol\)−log⁡qR\.n\_\{R\}\(\\epsilon\_\{\\mathrm\{tol\}\}\)\\geq\\frac\{\\log\(\\Delta\_\{0\}/\\epsilon\_\{\\mathrm\{tol\}\}\)\}\{\-\\log q\_\{R\}\}\.\(70\)For small2​μPL​η​γR2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\},−log⁡qR≈2​μPL​η​γR\-\\log q\_\{R\}\\approx 2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}, giving

nR​\(ϵtol\)≈12​μPL​η​γR​log⁡Δ0ϵtol\.n\_\{R\}\(\\epsilon\_\{\\mathrm\{tol\}\}\)\\approx\\frac\{1\}\{2\\mu\_\{\\mathrm\{PL\}\}\\eta\\gamma\_\{R\}\}\\log\\frac\{\\Delta\_\{0\}\}\{\\epsilon\_\{\\mathrm\{tol\}\}\}\.\(71\)Accordingly, the local iteration\-speedup predicted relative to a baseline, each measured to withinϵtol\\epsilon\_\{\\mathrm\{tol\}\}of its own neighborhood, is approximately

nB​\(ϵtol\)nR​\(ϵtol\)≈γRγB\.\\boxed\{\\frac\{n\_\{B\}\(\\epsilon\_\{\\mathrm\{tol\}\}\)\}\{n\_\{R\}\(\\epsilon\_\{\\mathrm\{tol\}\}\)\}\\approx\\frac\{\\gamma\_\{R\}\}\{\\gamma\_\{B\}\}\.\}\(72\)Equation \([72](https://arxiv.org/html/2610.41104#A1.E72)\) concerns optimization steps rather than wall\-clock time; the latter additionally depends on the per\-step cost of each optimizer\.

### A\.3Teacher Precision

Muon\-RT reads the teacher–student displacement𝐖T\(s\)−𝐖S\(s\)\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}, so the teacher must actually follow the EMA of Eq\. \([3](https://arxiv.org/html/2610.41104#S2.E3)\)\. Withτ=0\.99\\tau\{=\}0\.99, each step moves the teacher by1%1\\%of the displacement, and the displacement is only0\.10\.1–0\.3%0\.3\\%of the weights \(Appendix[A\.7](https://arxiv.org/html/2610.41104#A1.SS7)\)\. The per\-step change is therefore about10−510^\{\-5\}of a weight’s magnitude, while bfloat16 can only represent relative changes larger than about4×10−34\\times 10^\{\-3\}\(8 significand bits\)\. In 16\-bit storage most EMA steps round to zero\. We measured this on a Qwen2\.5\-7B\-Instruct key projection matrix with 250 momentum\-free Muon steps \(Figure[6](https://arxiv.org/html/2610.41104#A1.F6)\(a\); setup in Appendix[A\.7](https://arxiv.org/html/2610.41104#A1.SS7)\): the student moved by 2\.15 in Frobenius norm, a 32\-bit teacher by 1\.37, and a bfloat16 teacher by only 0\.66, with 79% of its entries never changing\. A 16\-bit displacement would thus reflect rounding rather than the averaged trajectory that Lemmas[1](https://arxiv.org/html/2610.41104#Thmlemma1)and[2](https://arxiv.org/html/2610.41104#Thmlemma2)describe\. For the same reason, optimizer states such as Muon’s momentum and AdamW’s moments are usually kept in 32 bits\([Liu et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib5);[Li et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib15)\); since the teacher takes over the role of optimizer state, we store it in 32 bits as well, and we do so for every optimizer we train in Tables[1](https://arxiv.org/html/2610.41104#S4.T1)–[3](https://arxiv.org/html/2610.41104#S4.T3), so that comparisons share the same teacher\. The teacher’s forward pass still runs in bfloat16, so the self\-distillation loss is computed as in SDFT\. The 32\-bit teacher adds 2 bytes per parameter \(7\.6 GB per GPU with 2 GPUs\); the memory in Figure[3](https://arxiv.org/html/2610.41104#S4.F3)\(d\) includes this cost\.

Precision setting\.Our precision differs from the original SDFT setting, which keeps the student and the teacher in bfloat16\. To make sure that the gains of the Muon variants come neither from precision nor from run\-to\-run randomness, we train every optimizer in the same setting: a bfloat16 mixed\-precision student with 32\-bit master weights and a teacher stored in 32 bits, with forward passes in bfloat16\. In this setting, SDFT and SDPO with AdamW reproduce their reported accuracy within run\-to\-run variation, so the AdamW rows taken from[Shenfeld et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib4)and[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9)are comparable with our runs\. All memory values in the paper are for a 32\-bit teacher \(Appendix[A\.6](https://arxiv.org/html/2610.41104#A1.SS6)\)\.

### A\.4Implementation Details

We build on the released SDFT code \(TRL with colocated vLLM generation\) and replace the optimizer\. Hidden matrices are all two\-dimensional weights inside the transformer blocks; embeddings, the output head, normalization weights and query/key/value biases use AdamW with the same learning rate, following[Liu et al\. \(2025\)](https://arxiv.org/html/2610.41104#bib.bib5)\. Student and teacher are sharded with FSDP2 using the same wrapping policy, so each teacher shard matches a student shard\. For every hidden matrix, the inner products in Eq\. \([9](https://arxiv.org/html/2610.41104#S3.E9)\) and the norms are reduced across shards; the gradient \(in bfloat16\) and𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}are gathered, Newton–Schulz and power iteration run on the full matrix on every rank, and each rank keeps its slice of𝐔\(s\)\\mathbf\{U\}^\{\(s\)\}\. The power\-iteration vector of each matrix is stored across steps \(1 vector of lengthBB, 4\.5 MB in total for Qwen2\.5\-7B\); starting from a random vector, 8 iterations underestimateσmax\\sigma\_\{\\max\}by about 8%, and the warm start removes this error within 3 steps\. We verified on a small model with 2 GPUs that \(i\)α=0\\alpha\{=\}0is bit\-identical to Muon withμ=0\\mu\{=\}0, \(ii\) sharded and unsharded updates are identical, and \(iii\) for a single matrix the update matches an exact singular\-value reference to5×10−75\\times 10^\{\-7\}, with‖𝐏\(s\)‖2=1\\\|\\mathbf\{P\}^\{\(s\)\}\\\|\_\{2\}\{=\}1\.

### A\.5Hyperparameters and Compute

Table 4:SDFT training settings for all optimizers\.Table[4](https://arxiv.org/html/2610.41104#A1.T4)lists the SDFT settings\. Following[Shenfeld et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib4)and[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9), we ran every configuration with 3 random seeds\. A SDFT run takes 1\.1–1\.8 hours on 2 H100 GPUs, dominated by generation\. For SDPO \(Table[5](https://arxiv.org/html/2610.41104#A1.T5)\), the model runs without thinking and every setting is identical to the experiments of[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9)without feedback on Olmo\-3\-7B\-Instruct: model, data splits, rollout and validation settings, loss, teacher EMA rate, schedule, weight decay and gradient clipping\. Only the optimizer of the hidden matrices and its learning rate change\. As for SDFT, Muon, MuonClip and Muon\-RT are trained with learning rates5×10−55\{\\times\}10^\{\-5\}and3×10−53\{\\times\}10^\{\-5\}, and we report the better run; the AdamW rows are taken from[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9)\. Accuracy is the best avg@16 reached within 1 and 5 hours of wall\-clock training, excluding initialization and validation, as in[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9)\.

Table 5:SDPO training settings\. All entries above the line are those of[Hübotter et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib9)\.
### A\.6Memory Projection

Figure[5](https://arxiv.org/html/2610.41104#S4.F5)projects the persistent model state of self\-distillation for a model withN=1013N\{=\}10^\{13\}parameters\. Per parameter, all optimizers store the student, its gradient and the EMA\-teacher in 32 bits \(12 bytes\); every state uses the same precision, so no optimizer is favored\. AdamW adds 2 32\-bit moments \(8 bytes\)\. Muon and MuonClip, in their standard form withμ=0\.95\\mu\{=\}0\.95, add a 32\-bit momentum \(4 bytes\) on hidden matrices and AdamW states on the remaining parameters; the per\-head statistics of MuonClip are negligible\. Muon\-RT adds no state on hidden matrices and AdamW states on the rest\. We take hidden matrices as 99% of the parameters, since embeddings and output head take about 0\.2% of the parameters in a 1T model\([Bai et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib53)\)\. This gives 20, 16\.04, 16\.04 and 12\.08 bytes per parameter, i\.e\. 200, 160, 160 and 121 TB\. Activations and generation buffers are excluded\. Our 7B measurements on 2 GPUs, all with a 32\-bit teacher, are taken between steps, when gradients are freed; without the 4\-byte gradient, the same accounting gives savings of 27% against Muon and 43% against AdamW, close to the measured 27% and 42%\. Muon\-RT, momentum\-free Muon and momentum\-free MuonClip need 37\.7 GB per GPU\. A Muon run that allocates the 32\-bit momentum of 6\.5B hidden parameters needs 51\.6 GB, close to37\.7\+13\.1=50\.837\.7\+13\.1=50\.8GB\. Storing the teacher in 32 instead of 16 bits adds 7\.6 GB per GPU: Muon withμ=0\.95\\mu\{=\}0\.95and a 16\-bit teacher needs 43\.9 GB, and43\.9\+7\.6=51\.543\.9\+7\.6=51\.5GB is close to the measurement\. AdamW with a 16\-bit teacher needs 56\.9 GB, so with a 32\-bit teacher it needs56\.9\+7\.6=64\.556\.9\+7\.6=64\.5GB, the value in Figures[1](https://arxiv.org/html/2610.41104#S1.F1)and[3](https://arxiv.org/html/2610.41104#S4.F3)\(d\)\.

### A\.7Learning\-Rate Sensitivity and Training Diagnostics

Figure 6:\(a\) Distance from the initial weights for the student and for teachers in 2 precisions\. \(b\) Size of the teacher–student displacement\. \(c\) Alignment of the 2 branches\. \(d\) Best tool\-use accuracy versus learning rate\.Learning rate\.Figure[6](https://arxiv.org/html/2610.41104#A1.F6)\(d\) shows that Muon\-RT exceeds the best momentum\-free Muon run \(69\.1%\) for every learning rate from2×10−52\\times 10^\{\-5\}to6×10−56\\times 10^\{\-5\}\(70\.1–74\.2%\) and peaks at3×10−53\\times 10^\{\-5\}\. Only at10−510^\{\-5\}and below do 2 epochs leave too few effective steps\.

Teacher branch\.We log, for every hidden matrix and step,‖𝐇\(s\)‖F/‖𝐖S\(s\)‖F\\\|\\mathbf\{H\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}/\\\|\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\},σmax​\(𝐇\(s\)\)/‖𝐇\(s\)‖F\\sigma\_\{\\max\}\(\\mathbf\{H\}^\{\(s\)\}\)/\\\|\\mathbf\{H\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\},cos⁡\(𝐆\(s\),𝐇\(s\)\)\\cos\(\\mathbf\{G\}^\{\(s\)\},\\mathbf\{H\}^\{\(s\)\}\)andcos⁡\(𝐎\(s\),𝐏\(s\)\)\\cos\(\\mathbf\{O\}^\{\(s\)\},\\mathbf\{P\}^\{\(s\)\}\), averaged over matrices\. The displacement grows during warmup, peaks at 0\.2–0\.3% of the weights and shrinks as the learning rate decays \(Figure[6](https://arxiv.org/html/2610.41104#A1.F6)\(b\)\)\. Its largest singular value carries about 6% of its Frobenius norm on all runs, so𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}is spread over many directions\. After the first 20 steps, the 15\-step averages of both cosines stay below 0\.015 in magnitude on all runs \(Figure[6](https://arxiv.org/html/2610.41104#A1.F6)\(c\)\): the teacher branch is nearly orthogonal to the current gradient and adds directions it does not cover\.

Experiments on a Qwen2\.5\-7B\-Instruct weight matrix\.Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(b, c\), Figure[6](https://arxiv.org/html/2610.41104#A1.F6)\(a\) and the constants below use the key projection matrix of layer 10 of Qwen2\.5\-7B\-Instruct \(512×3584512\\times 3584\)\. The student takes momentum\-free Muon steps𝐖←𝐖−η​cA,B​NSK​\(𝐆\(s\)\)\\mathbf\{W\}\\leftarrow\\mathbf\{W\}\-\\eta c\_\{A,B\}\\mathrm\{NS\}\_\{K\}\(\\mathbf\{G\}^\{\(s\)\}\)withη=5×10−5\\eta\{=\}5\{\\times\}10^\{\-5\}and𝐆\(s\)=𝐝\+ν​𝐄\(s\)/‖𝐄\(s\)‖F\\mathbf\{G\}^\{\(s\)\}=\\mathbf\{d\}\+\\nu\\,\\mathbf\{E\}^\{\(s\)\}/\\\|\\mathbf\{E\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}, where𝐝\\mathbf\{d\}is a fixed random direction with‖𝐝‖F=1\\\|\\mathbf\{d\}\\\|\_\{\\mathrm\{F\}\}\{=\}1,𝐄\(s\)\\mathbf\{E\}^\{\(s\)\}is fresh Gaussian noise andν\\nuis the noise level; the teacher follows Eq\. \([3](https://arxiv.org/html/2610.41104#S2.E3)\) withτ=0\.99\\tau\{=\}0\.99\. For Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(b\), we run 150 steps for eachν\\nuand average, over the last 20 steps, the cosine of𝐎\(s\)\\mathbf\{O\}^\{\(s\)\}and of𝐏\(s\)\\mathbf\{P\}^\{\(s\)\}withNSK​\(𝐝\)\\mathrm\{NS\}\_\{K\}\(\\mathbf\{d\}\)\. For Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(c\), we take𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}after 150 steps withν=2\\nu\{=\}2and a gradient of low stable rank,𝐆=𝐘1​𝐘2⊤\\mathbf\{G\}=\\mathbf\{Y\}\_\{1\}\\mathbf\{Y\}\_\{2\}^\{\\top\}with𝐘1∈ℝA×m\\mathbf\{Y\}\_\{1\}\\in\\mathbb\{R\}^\{A\\times m\},𝐘2∈ℝB×m\\mathbf\{Y\}\_\{2\}\\in\\mathbb\{R\}^\{B\\times m\}Gaussian,m∈\{4,16,64\}m\\in\\\{4,16,64\\\}, plus 1% Gaussian noise\. We compare‖𝐔−𝐎‖F/‖𝐎‖F\\\|\\mathbf\{U\}\-\\mathbf\{O\}\\\|\_\{\\mathrm\{F\}\}/\\\|\\mathbf\{O\}\\\|\_\{\\mathrm\{F\}\}for Eq\. \([11](https://arxiv.org/html/2610.41104#S3.E11)\) and for the pre\-Newton–Schulz mix𝐔=NSK​\(\(1−α\)​𝐆/‖𝐆‖F\+α​𝐇\(s\)/‖𝐇\(s\)‖F\)\\mathbf\{U\}=\\mathrm\{NS\}\_\{K\}\\big\(\(1\-\\alpha\)\\mathbf\{G\}/\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{F\}\}\+\\alpha\\mathbf\{H\}^\{\(s\)\}/\\\|\\mathbf\{H\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}\\big\)\. For Figure[6](https://arxiv.org/html/2610.41104#A1.F6)\(a\), the teacher is either kept in 32 bits or rounded to bfloat16 after every EMA step\.

Constants of Theorem[1](https://arxiv.org/html/2610.41104#Thmtheorem1)\.With the same matrix and gradients, the student follows Muon\-RT \(α=0\.1\\alpha\{=\}0\.1\), so the teacher records Muon\-RT’s own trajectory\. Over steps 51–150 we averageaX=⟨𝐝,𝐗⟩Fa\_\{X\}=\\langle\\mathbf\{d\},\\mathbf\{X\}\\rangle\_\{\\mathrm\{F\}\}andbX=‖𝐗‖F2b\_\{X\}=\\\|\\mathbf\{X\}\\\|\_\{\\mathrm\{F\}\}^\{2\}for𝐗∈\{𝐎\(s\),𝐏\(s\),𝐔\(s\)\}\\mathbf\{X\}\\in\\\{\\mathbf\{O\}^\{\(s\)\},\\mathbf\{P\}^\{\(s\)\},\\mathbf\{U\}^\{\(s\)\}\\\}, where𝐝\\mathbf\{d\}is the mean gradient\. For noise levelsν∈\{0\.5,1,2,4,8,16\}\\nu\\in\\\{0\.5,1,2,4,8,16\\\},aP/aOa\_\{P\}/a\_\{O\}is 1\.10, 1\.37, 1\.97, 3\.20, 5\.03 and 6\.62, soaRa\_\{R\}exceedsaB=aOa\_\{B\}\{=\}a\_\{O\}by 1%–56%, whilebP<bOb\_\{P\}<b\_\{O\}andbR<bOb\_\{R\}<b\_\{O\}\(e\.g\. 366 against 447 atν=16\\nu\{=\}16\)\. HenceγR\>γB\\gamma\_\{R\}\>\\gamma\_\{B\}for everyL​ηL\\eta\. The cosine between𝐆\(s\)\\mathbf\{G\}^\{\(s\)\}and𝐏\(s\)\\mathbf\{P\}^\{\(s\)\}falls from 0\.87 to 0\.03 asν\\nugrows, whileaP/aOa\_\{P\}/a\_\{O\}rises, so a small single\-batch cosine does not indicate a weak teacher direction\.

Teacher–student displacement without normalization\.For Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(a\), the predicted ratio‖𝐖T\(s\)−𝐖S\(s\)‖F/‖𝐖S\(s\)‖F\\\|\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}/\\\|\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}isτ​ψs/rms⁡\(𝐖\)\\tau\\psi\_\{s\}/\\mathrm\{rms\}\(\\mathbf\{W\}\), whereψs\\psi\_\{s\}is∑jτj​us−1−j\\sum\_\{j\}\\tau^\{j\}u\_\{s\-1\-j\}\(aligned updates\) or\(∑jτ2​j​us−1−j2\)1/2\(\\sum\_\{j\}\\tau^\{2j\}u\_\{s\-1\-j\}^\{2\}\)^\{1/2\}\(uncorrelated updates\),usu\_\{s\}is the logged learning rate times the update root\-mean\-square, and1/rms⁡\(𝐖\)1/\\mathrm\{rms\}\(\\mathbf\{W\}\)is averaged over the 196 hidden matrices\. From step 20 on, the measured mean ratio is 1\.2–1\.7 times the uncorrelated value and 0\.12–0\.34 times the aligned value\.

Lemma 1 range\.For Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(a\), letus=ηs​ρsu\_\{s\}=\\eta\_\{s\}\\rho\_\{s\}with the logged learning rateηs\\eta\_\{s\}and update root\-mean\-squareρs\\rho\_\{s\}\. By Eq\. \([7](https://arxiv.org/html/2610.41104#S3.E7)\) and the triangle inequality,‖𝐖T\(s\)−𝐖S\(s\)‖F\\\|\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}is proportional to at most∑jτj​us−1−j\\sum\_\{j\}\\tau^\{j\}u\_\{s\-1\-j\}\(aligned updates\) and, for uncorrelated updates, to\(∑jτ2​j​us−1−j2\)1/2\(\\sum\_\{j\}\\tau^\{2j\}u\_\{s\-1\-j\}^\{2\}\)^\{1/2\}\. We normalize both and the measured‖𝐇\(s\)‖F/‖𝐖S\(s\)‖F\\\|\\mathbf\{H\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}/\\\|\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}by their maxima\.

### A\.8Attention Logits and QK\-Clip

Growing attention logits can destabilize training with Muon, which motivated QK\-Clip\([Bai et al\., 2025](https://arxiv.org/html/2610.41104#bib.bib53);[Wortsman et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib22);[Dehghani et al\., 2023](https://arxiv.org/html/2610.41104#bib.bib24)\)\. We measured the maximum pre\-softmax logit of every head \(after rotary embeddings\) on 24 evaluation prompts\. The base Qwen2\.5\-7B\-Instruct already reaches about 4,800, and 26–28 heads exceed the threshold of 100 used by[Bai et al\. \(2025\)](https://arxiv.org/html/2610.41104#bib.bib53); these are attention\-sink heads driven by the query and key biases\([Xiao et al\., 2024](https://arxiv.org/html/2610.41104#bib.bib23)\)\. After SDFT, the global maximum stays within±1%\\pm 1\\%of the base value for AdamW, Muon and Muon\-RT alike, the largest heads grow by at most 13–20%, and the norms of the query and key weights change by less than 0\.05%\. Clipping at 100 would therefore rescale about 27 heads that are large by design\. Our MuonClip baseline instead uses per\-head thresholds of1\.1×1\.1\\timesthe largest logit observed in the first 10 steps\. Because logits do not grow, QK\-Clip is not needed in this setting\. With Muon\-RT, it is also harmful \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\(d\)\)\. In the tool\-use run with clipping, 8 clips occur, each on 1 head, with factors down to 0\.56; right after the first clip \(step 20, factor 0\.90\), the mean‖𝐇\(s\)‖F/‖𝐖S\(s\)‖F\\\|\\mathbf\{H\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}/\\\|\\mathbf\{W\}\_\{S\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}over all hidden matrices rises 12% above the run without clipping, after tracking it within 2% before\. To isolate the effect, we take the query matrix of layer 10 of Qwen2\.5\-7B\-Instruct, scale the 128 rows of 1 head in the student but not in the teacher, and form𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}and𝐏\(s\)\\mathbf\{P\}^\{\(s\)\}as in Eqs\. \([9](https://arxiv.org/html/2610.41104#S3.E9)\)–\([10](https://arxiv.org/html/2610.41104#S3.E10)\)\. A factor of 0\.90 \(0\.56\) creates a gap of 1\.9% \(8\.0%\) of the matrix norm, against 0\.1–0\.3% from learning; radial removal changes it by under 1\.5%, 97% \(99%\) of‖𝐇\(s\)‖F2\\\|\\mathbf\{H\}^\{\(s\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}lies in the clipped head, and−𝐏\(s\)\-\\mathbf\{P\}^\{\(s\)\}restricted to that head has cosine−1\.00\-1\.00with its weights\. The update therefore shrinks the clipped head further at every step\. The offset decays byτ\\tauper step, so a 1\.9% gap falls below 0\.2% only after about 225 steps\. Scaling the same query rows of the teacher by the same factor at every clip keeps𝐖T\(s\)−𝐖S\(s\)\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}free of this offset; in Table[3](https://arxiv.org/html/2610.41104#S4.T3)\(d\), this variant reaches 67\.3 on tool use and 70\.1 on science\.

### A\.9Training and evaluation\.

Only the optimizer changes; each host method keeps its loss, batch size and EMA settings same\. For every optimizer, the student is trained in bfloat16 mixed precision \(bfloat16 compute, 32\-bit master weights\), and the EMA\-teacher is stored in 32 bits with its forward pass in bfloat16\. Appendix[A\.3](https://arxiv.org/html/2610.41104#A1.SS3)shows that this precision does not affect accuracy\. We train Muon, MuonClip and Muon\-RT \(α=0\.1\\alpha\{=\}0\.1\) with learning rates5×10−55\{\\times\}10^\{\-5\}and3×10−53\{\\times\}10^\{\-5\}and report the better run\. Following[Shenfeld et al\. \(2026\)](https://arxiv.org/html/2610.41104#bib.bib4), we report the best checkpoint under greedy decoding and measure prior capabilities on HellaSwag, HumanEval, IFEval, MMLU, TruthfulQA, and Winogrande\. For SDPO, we report the best avg@16 within 1 and 5 hours of training\([Hübotter et al\., 2026](https://arxiv.org/html/2610.41104#bib.bib9)\)\.

### A\.10Empirical Checks of the Theory

Figure[4](https://arxiv.org/html/2610.41104#S4.F4)tests each theoretical statement with a matching measurement\.

Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1): the teacher stores applied updates\.Eq\. \([7](https://arxiv.org/html/2610.41104#S3.E7)\) predicts the size of𝐖T\(s\)−𝐖S\(s\)\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}from the learning rate and the size of the applied updates alone: it lies between the sum of their norms \(fully aligned updates\) and the root of the sum of their squares \(uncorrelated updates\)\. The measured displacement follows this range closely, with correlation 0\.97 \(Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(a\)\)\. It peaks near step 107, after the learning rate peaks, exactly as an EMA of applied updates should\. Without normalization, the measured size lies inside this range at every step after step 20, at 1\.2–1\.7 times the uncorrelated bound, so the applied updates share a persistent component \(Appendix[A\.7](https://arxiv.org/html/2610.41104#A1.SS7)\)\.

Lemma[2](https://arxiv.org/html/2610.41104#Thmlemma2): the teacher averages out noise\.On a Qwen2\.5\-7B\-Instruct weight matrix trained with noisy gradients of known mean, the current orthogonalized gradient loses the mean direction as gradient noise grows \(cosine 0\.87 to 0\.11\), while the teacher branch keeps it \(0\.99 to 0\.75; Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(b\)\)\.

Proposition[1](https://arxiv.org/html/2610.41104#Thmproposition1): the gain is temporal\.During training, the 2 branches are nearly orthogonal, with\|cos⁡\(𝐎\(s\),𝐏\(s\)\)\|<0\.015\|\\cos\(\\mathbf\{O\}^\{\(s\)\},\\mathbf\{P\}^\{\(s\)\}\)\|<0\.015after the first steps \(Appendix[A\.7](https://arxiv.org/html/2610.41104#A1.SS7)\)\. The teacher branch thus adds directions that the current gradient does not contain, as the proposition requires\.

Post\-Newton–Schulz mixing bounds the teacher\.Mixed before Newton–Schulz, evenα=0\.02\\alpha\{=\}0\.02changes the update by more than its own norm, because Newton–Schulz lifts the teacher’s weak directions to unit singular value \(Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(c\)\)\. Mixed after Newton–Schulz, as in Muon\-RT, the change grows linearly withα\\alpha, as Eq\. \([11](https://arxiv.org/html/2610.41104#S3.E11)\) guarantees\.

Theorem[1](https://arxiv.org/html/2610.41104#Thmtheorem1)and Corollary[1](https://arxiv.org/html/2610.41104#Thmcorollary1): faster convergence\.Under the PL condition, a smaller gradient norm means a smaller optimality gap\. At the same learning rate, Muon\-RT has the lowest gradient norm from step 40 onward \(Figure[4](https://arxiv.org/html/2610.41104#S4.F4)\(d\)\) and the lowest self\-distillation KL for most of training \(Figure[3](https://arxiv.org/html/2610.41104#S4.F3)\(c\)\), and it reaches its best accuracy earlier \(Figure[3](https://arxiv.org/html/2610.41104#S4.F3)\(a\)\)\. These observations are consistent with the corollary\. We also check its condition directly by estimating the constants of A4 on the Qwen2\.5\-7B\-Instruct weight matrix with gradients of known mean \(Appendix[A\.7](https://arxiv.org/html/2610.41104#A1.SS7)\): at every noise level, the teacher branch is better aligned with the mean gradient than the orthogonalized gradient \(up to 6\.6 times\) and has a smaller second moment, soaR\>aBa\_\{R\}\>a\_\{B\}andbR<bBb\_\{R\}<b\_\{B\}, which givesγR\>γB\\gamma\_\{R\}\>\\gamma\_\{B\}for any step size\. The single\-batch cosine between gradient and teacher direction stays small \(0\.03 at the highest noise\), so the near\-zero cosines measured in training do not contradict this condition\.

### A\.11Ablations and Analysis

We ablate each component of Muon\-RT with SDFT on both tasks \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\): the optimizer state it replaces, the teacher weightα\\alpha, the construction of the teacher branch, and its combination with logit clipping\. All ablations follow the setting of Table[1](https://arxiv.org/html/2610.41104#S4.T1), withα=0\.1\\alpha\{=\}0\.1unless varied, and every block includes the full method as reference\.

Which history helps\.Gradient history and teacher history have opposite effects \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\(a\)\)\. Standard momentum reduces accuracy by 0\.7–6\.2 points for both Muon and MuonClip on both tasks, whereas the teacher branch raises the accuracy of momentum\-free Muon by 5\.1 and 2\.8 points\. Orthogonalization normalizes each update, but with momentum consecutive updates share a direction for about1/\(1−μ\)=201/\(1\-\\mu\)\{=\}20steps and add up instead of partly cancelling, so the weights move further at the same learning rate\. In addition, self\-distillation targets move with the teacher at every step, so an average of past gradients points partly toward targets that no longer apply\. The teacher branch, by contrast, records the updates that were actually applied \(Lemma[1](https://arxiv.org/html/2610.41104#Thmlemma1)\), and the bounded mix adds this history without letting it override the current gradient\.

Teacher weight\.α=0\.1\\alpha\{=\}0\.1is the best value on both tasks \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\(b\)\)\. Both smaller and larger values reduce accuracy, confirming that the teacher should guide, not replace, the gradient direction \(Eq\. \([11](https://arxiv.org/html/2610.41104#S3.E11)\)\)\. The same weight is optimal for both tasks and is used without modification for SDPO, soα\\alpharequires no per\-task tuning\.

Teacher branch\.Every modification to the teacher branch reduces accuracy \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\(c\)\), by 1\.7–2\.6 points for mixing before Newton–Schulz up to 5\.0 points for a cosine gate\. The gradient and displacement are nearly orthogonal \(\|cos\|≈0\.004\|\\cos\|\\approx 0\.004\), causing the cosine gate to effectively switch the teacher off\. On the tool\-use task, even the weaker variants outperforms the momentum\-free Muon \(70\.5–72\.5 Vs\. 69\.1\), indicating that the teacher’s history is beneficial in every form; however, across both tasks, the bounded post\-Newton–Schulz formulation extracts the most from it\.

QK\-Clip\.Logit clipping helps Muon \(69\.1 to 73\.2 on tool use\) but hurts Muon\-RT \(Table[3](https://arxiv.org/html/2610.41104#S4.T3)\(d\); 71\.1 to 66\.0 at the same learning rate\)\. QK\-Clip rescales a head’s query weights in the student only, so each clip introduces an offset into𝐖T\(s\)−𝐖S\(s\)\\mathbf\{W\}\_\{T\}^\{\(s\)\}\-\\mathbf\{W\}\_\{S\}^\{\(s\)\}that is not learning motion\. On a real query matrix, a single clip by a factor 0\.9 creates a gap 6–19 times larger than the learning displacement, this gap is retained by radial removal and accounts for 97% of𝐇\(s\)\\mathbf\{H\}^\{\(s\)\}\(Appendix[A\.8](https://arxiv.org/html/2610.41104#A1.SS8)\)\. As a result, the teacher branch continues shrinking the clipped head at full weightα\\alphafor over 200 steps; in training, the first clip raises the mean displacement over all matrices by 12% within a single step\. Applying the same clip to the teacher removes this offset and recovers 1\.3 points on tool use \(67\.3\), but both tasks remain below Muon\-RT without clipping \(67\.3 and 70\.1\): each clip also perturbs the teacher’s attention patterns and therefore the self\-distillation target\. In any case, clipping is unnecessary here, as logits stay within 1% of their initial maximum for every optimizer\.