Muon 何时以及如何进行预条件化？结构化梯度噪声的作用

# When and How Does Muon Precondition? The Role of Structured Gradient Noise
Source: [https://arxiv.org/html/2610.31728](https://arxiv.org/html/2610.31728)
Xiaoran Chen1,∗Binghui Li2,∗,†Liwei Wang2,3,5,†Pinyan Lu4,† 1School of EECS, Peking University2Center for Machine Learning Research, Peking University3Center for Data Science, Peking University4Key Laboratory of Interdisciplinary Research of Computation and Economics,Shanghai University of Finance and Economics5State Key Laboratory of General Artificial Intelligence, Peking University chenxiaoran@stu\.pku\.edu\.cn\{libinghui,wanglw\}@pku\.edu\.cn lu\.pinyan@mail\.shufe\.edu\.cn

###### Abstract

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood\. In this work, we study how gradient noise geometry shapes Muon’s preconditioning benefits\. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate\. This motivates us to investigate structured noise arising from mini\-batch sampling in associative memory with heterogeneous task frequencies\. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning\. Our analysis shows that Muon can soften the dependence on task frequency from linear to square\-root, helping balance the learning of frequent and rare tasks\. We further identify rotation\-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients\. Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast–slow momentum\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding authors\.### 1Introduction

Muon has emerged as a prominent alternative to Adam for training the matrix parameters of large language models \(LLMs\)\([Jordan et al\., 2024](https://arxiv.org/html/2610.31728#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.31728#bib.bib12)\)\. Its defining operation replaces a gradient or momentum matrix by an approximate polar factor, flattening its nonzero singular values before the parameter update\. This matrix\-level normalization offers a different way to allocate updates across directions from Adam’s coordinatewise rescaling\. Recent frontier\-model reports, including DeepSeek–V4\.1–Flash\([DeepSeek\-AI, 2026](https://arxiv.org/html/2610.31728#bib.bib18)\)and Kimi–K3\([Kimi Team and others, 2026](https://arxiv.org/html/2610.31728#bib.bib19)\), adopt Muon variants for matrix optimization, with head\-wise orthogonalization further adapting the update to attention structure\. These developments make understanding when and why Muon improves learning an increasingly practical theoretical question\.

Existing theory has established convergence guarantees for practical orthogonalization procedures\([Kim and Oh, 2026](https://arxiv.org/html/2610.31728#bib.bib3)\)and identified preconditioning benefits in structured optimization problems\([Ma et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib2)\)\. Full\-batch associative\-memory analyses further show how spectral normalization can balance learning across tasks with heterogeneous frequencies\([Wang et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib6);[Li et al\., 2026b](https://arxiv.org/html/2610.31728#bib.bib7)\)\. In LLM pretraining, however, Muon acts on stochastic mini\-batch gradients, and nonlinear normalization need not preserve the benefits predicted from population gradients\. In particular, writingmsgn\\operatorname\{msgn\}for the partial polar map, generally𝔼⁡\[msgn⁡\(Gt\)∣Wt\]≠msgn⁡\(𝔼⁡\[Gt∣Wt\]\)\\mathbb\{E\}\[\\operatorname\{msgn\}\(G\_\{t\}\)\\mid W\_\{t\}\]\\neq\\operatorname\{msgn\}\(\\mathbb\{E\}\[G\_\{t\}\\mid W\_\{t\}\]\)\. Recent work has begun to characterize stochastic spectral dynamics and geometry\-dependent noise scales\([Paquette et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib9);[Naganuma et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib4)\); the interaction between task occurrence, gradient reliability, and temporal averaging motivates the question we study here:

Q1:When does Muon retain a directional learning advantage under stochastic gradient noise, and how do noise structure, batch size, and momentum determine this advantage?

Isotropic noise limits preconditioning\.Standard stochastic optimization analyses often control gradient noise through a scalar variance bound\([Ghadimi and Lan, 2013](https://arxiv.org/html/2610.31728#bib.bib20)\), leaving its directional structure unspecified\. We use isotropic Gaussian noise as a diagnostic baseline\. In the fixed\-dimensional low\-SNR limit, we prove that Muon’s conditional mean update is, to first order, a scalar multiple of the population gradient, as confirmed numerically in Figure[1](https://arxiv.org/html/2610.31728#S3.F1)\. SGD with a matched learning rate therefore reproduces this first\-order mean drift: applying the partial polar map to each stochastic mini\-batch gradient does not, by itself, produce directional preconditioning in the first\-order mean response\. Explaining such gains in this regime requires examining noise structure\.

Sampling controls balancing\.We study associative memory with heterogeneous task frequenciespjp\_\{j\}\([Wang et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib6);[Li et al\., 2026b](https://arxiv.org/html/2610.31728#bib.bib7)\)\. Mini\-batch sampling can leave task directions entirely unobserved; without momentum, Muon’s nullspace preservation then prevents updates along those directions\. For two tasks, we prove an exact balancing thresholdBbalB\_\{\\rm bal\}: below it, margins grow at unequal linear rates; above it, their gap has uniformly bounded exponential moments\. For rare tasks,Bbal∼\(log⁡2\)/pminB\_\{\\rm bal\}\\sim\(\\log 2\)/p\_\{\\min\}\. In a higher\-rank hierarchy, local perturbation estimates identify the expected task countB​pjBp\_\{j\}as a controlling scale\. Experiments exhibit corresponding frequency\-dependent crossovers, with frequent groups approaching balanced growth while rare groups remain occurrence\-limited\.

Label noise limits reliability\.Label fluctuations persist at the population optimum, so frequent observation alone does not ensure a reliable gradient\. Exact gradient statistics and a scalar projected counting model distinguish regimes governed by task countsB​pjBp\_\{j\}and corruption countsB​pj​ραBp\_\{j\}\\rho\_\{\\alpha\}\. Near the noisy optimum, an isotropic covariance closure with a locally homogeneous frequency approximation predicts a Muon contraction coefficient proportional toB​p\\sqrt\{Bp\}, compared with SGD’s linear dependence on the local frequency scalepp\.

Orthogonal equivariance differs from coordinatewise sign updates\.We use SignSGD as a proxy for Adam’s coordinatewise adaptivity\([Balles and Hennig, 2018](https://arxiv.org/html/2610.31728#bib.bib14);[Kunstner et al\., 2023](https://arxiv.org/html/2610.31728#bib.bib15)\)\. Muon is orthogonally equivariant whereas SignSGD is basis dependent\. In a two\-task rotated sparse\-support construction, a batch containing only the frequent task leaves the rare\-task Muon margin unchanged but can decrease the corresponding SignSGD margin under certain conditions\.

Momentum trades averaging for tracking\.Momentum reduces sampling fluctuations, with frozen\-state variance\-equivalent batch sizeBeff=B⁡\(1\+β\)/\(1−β\)B\_\{\\rm eff\}=B\(1\+\\beta\)/\(1\-\\beta\), but longer memory introduces tracking lag\. Under local tracking assumptions, the sufficient windowη≪1−β≪B​pj\\eta\\ll 1\-\\beta\\ll Bp\_\{j\}motivates longer memory for rare, slowly varying directions and shorter memory where rapid tracking is needed\. This provides a sampling\-based perspective on curvature\-conditioned multiscale momentum\([Zhu et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib11)\): the local Hessian blockpj​Γjp\_\{j\}\\Gamma\_\{j\}links rare tasks to low\-curvature directions when label statistics are comparable\. It also connects to occurrence\-dependent forgetting in Activation\-Keyed Momentum\([Hong and Qu, 2026](https://arxiv.org/html/2610.31728#bib.bib13)\), supporting selective long\-memory averaging as a means to improve estimates in slowly varying directions\.

Our contributions are threefold:

- •A diagnostic for stochastic directional gains\.We prove a low\-SNR expansion under isotropic Gaussian noise, showing that Muon’s first\-order conditional mean response can be matched by learning\-rate\-rescaled SGD despite samplewise spectral normalization\.
- •A theory of structured sampling\.We establish an exact two\-task balancing threshold and derive local higher\-rank sampling estimates\. With stochastic labels, we separate exact gradient statistics, scalar projected counting laws, and a square\-root frequency prediction under isotropic covariance and locally homogeneous frequency approximations\. A momentum\-free Muon–SignSGD comparison isolates rotation\-induced negative transfer under sparse sampling\.
- •A statistical basis for heterogeneous momentum\.We characterize frozen\-state variance reduction and local tracking error under temporal averaging, explaining when longer memory can help rare directions and providing a sampling\-based interpretation of fast–slow momentum designs\. Controlled experiments test these mechanisms, and a preliminary pretraining comparison illustrates their practical relevance\.

### 2Related Work

##### Muon and adaptive optimization\.

Muon applies approximate polar normalization to matrix\-valued momentum\([Jordan et al\., 2024](https://arxiv.org/html/2610.31728#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.31728#bib.bib12)\)\. Existing work establishes convergence with Newton–Schulz iterations\([Kim and Oh, 2026](https://arxiv.org/html/2610.31728#bib.bib3)\), preconditioning benefits in structured matrix problems\([Ma et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib2)\), and generalization gains from balanced learning on imbalanced data\([Vasudeva et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib5)\)\.[Magakyan et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib17)further identify scaling laws for momentum spectra and their implications for orthogonalization accuracy\. For coordinatewise adaptivity,[Li et al\. \(2026a\)](https://arxiv.org/html/2610.31728#bib.bib21)analyze SignSGD in feature\-space linear regression, characterizing its loss dynamics and batch\-size benefits relative to SGD\. We complement these perspectives by studying how stochastic gradient structure governs Muon’s conditional mean response and distinguishes polar normalization from coordinatewise sign updates under sparse task sampling\.

##### Stochastic spectral dynamics\.

[Paquette et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib9)characterize batch\-dependent preconditioning regimes for Muon and SignSGD in high\-dimensional matrix least squares\.[Naganuma et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib4)derive non\-Euclidean gradient\-noise scales for adaptive batch selection, while[Eschenhagen et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib16)examine how stochasticity and temporal adaptation distinguish Shampoo from spectral descent\. Our analysis focuses on random task support and persistent label fluctuations: we identify an exact finite\-batch balancing threshold and separate task occurrence from gradient reliability\. The isotropic Gaussian result provides a complementary fixed\-dimensional diagnostic for the absence of first\-order directional preconditioning\.

##### Associative memory learning\.

[Wang et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib6)connect Muon’s advantage over Adam to balanced learning of rare associations\.[Li et al\. \(2026b\)](https://arxiv.org/html/2610.31728#bib.bib7)analyze hierarchical task frequencies, label noise, and scaling laws through implicit matrix preconditioning\. Beyond orthogonal embeddings,[Kim et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib8)derive recovery and capacity scaling with Gaussian embeddings, including critical batch\-size behavior\. Building on the orthogonal associative\-memory framework, we isolate finite\-batch sampling effects through exact two\-task dynamics and a higher\-rank frequency\-dependent regime map\. Our SignSGD comparison further identifies rotation\-induced negative transfer when tasks are absent from a batch\.

##### Heterogeneous momentum\.

Muon momentum has been studied as spectral filtering\([Li et al\., 2026c](https://arxiv.org/html/2610.31728#bib.bib10)\), while activation\-keyed and curvature\-conditioned methods adapt memory across directions\([Hong and Qu, 2026](https://arxiv.org/html/2610.31728#bib.bib13);[Zhu et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib11)\)\. We complement these designs with a sampling\-based account of variance reduction and tracking lag\.

### 3Preliminaries

#### 3\.1Notations\.

ForG∈ℝd×dG\\in\\mathbb\{R\}^\{d\\times d\}with SVDG=U​Σ​V⊤G=U\\Sigma V^\{\\top\}, define the partial polar factormsgn\(G\):=Udiag\(𝟏\{σi\>0\}\)V⊤\\operatorname\{msgn\}\(G\):=U\\operatorname\{diag\}\(\\mathbf\{1\}\\\{\\sigma\_\{i\}\>0\\\}\)V^\{\\top\}, which replaces each nonzero singular value by one and reduces toU​V⊤UV^\{\\top\}whenGGis full rank\. The map is positively scale invariant,msgn⁡\(a​G\)=msgn⁡\(G\)\\operatorname\{msgn\}\(aG\)=\\operatorname\{msgn\}\(G\)fora\>0a\>0, and orthogonally equivariant,msgn⁡\(Q​G​R⊤\)=Q​msgn⁡\(G\)​R⊤\\operatorname\{msgn\}\(QGR^\{\\top\}\)=Q\\operatorname\{msgn\}\(G\)R^\{\\top\}for orthogonalQ,RQ,R\. It also preserves the right nullspace:ker⁡\(msgn⁡\(G\)\)=ker⁡\(G\)\\ker\(\\operatorname\{msgn\}\(G\)\)=\\ker\(G\)\.

Given the stochastic gradientGtG\_\{t\}, we study the idealized Muon dynamicsMt=β​Mt−1\+\(1−β\)​GtM\_\{t\}=\\beta M\_\{t\-1\}\+\(1\-\\beta\)G\_\{t\}andWt\+1=Wt−η​msgn⁡\(Mt\)W\_\{t\+1\}=W\_\{t\}\-\\eta\\operatorname\{msgn\}\(M\_\{t\}\), whereη\>0\\eta\>0is the learning rate andβ∈\[0,1\)\\beta\\in\[0,1\)is the momentum coefficient;β=0\\beta=0recovers Muon without momentum\. Our theoretical analysis uses exact partial polar normalization in task coordinates; rectangular representations are introduced where needed\. Unless stated otherwise, mini\-batches are independent across iterations and weight decay is omitted\. The pretraining experiment uses the practical construction cited in Section[6](https://arxiv.org/html/2610.31728#S6)\.

#### 3\.2Warmup: Isotropic Noise Limits Preconditioning

Figure 1:Isotropic noise removes first\-order directional preconditioning at low SNR\.Left: for an anisotropicd=8d=8signal, the response gain along each singular direction, normalized bycdc\_\{d\}, collapses near one across a wide range of signal singular values\. Right: the full conditional\-mean matrix remains within a few percent of the scalar\-response prediction over the tested low\-SNR amplitudes\.Consider momentum\-free Muon\. Condition on the current iterateWWand writeG=G¯\+σ​ΞG=\\bar\{G\}\+\\sigma\\Xi, whereG¯=∇ℒ​\(W\)\\bar\{G\}=\\nabla\\mathcal\{L\}\(W\)andΞ∈ℝd×d\\Xi\\in\\mathbb\{R\}^\{d\\times d\}has i\.i\.d\. standard Gaussian entries\. Although each polar update has a flat nonzero spectrum, its mean need not retain a preconditioning benefit\.

###### Proposition 3\.1\(Isotropic Gaussian mean response\)\.

Fixddand letcd=d−2​𝔼​‖Ξ‖∗\>0c\_\{d\}=d^\{\-2\}\\mathbb\{E\}\\\|\\Xi\\\|\_\{\*\}\>0\. As‖G¯‖F/σ→0\\\|\\bar\{G\}\\\|\_\{F\}/\\sigma\\to 0,

𝔼⁡\[msgn⁡\(G¯\+σ​Ξ\)\]=cdσ​G¯\+Od​\(‖G¯‖F3σ3\),\\mathbb\{E\}\[\\operatorname\{msgn\}\(\\bar\{G\}\+\\sigma\\Xi\)\]=\\frac\{c\_\{d\}\}\{\\sigma\}\\bar\{G\}\+O\_\{d\}\\\!\\left\(\\frac\{\\\|\\bar\{G\}\\\|\_\{F\}^\{3\}\}\{\\sigma^\{3\}\}\\right\),with the remainder measured in Frobenius norm\.

At the same parameter state, Muon with learning rateηM\\eta\_\{\\rm M\}has first\-order conditional mean update−ηM​\(cd/σ\)​G¯\-\\eta\_\{\\rm M\}\(c\_\{d\}/\\sigma\)\\bar\{G\}, matching SGD withηS=ηM​cd/σ\\eta\_\{\\rm S\}=\\eta\_\{\\rm M\}c\_\{d\}/\\sigma\. Thus its effective first\-order preconditioner is scalar: all singular directions receive the same multiplicative gain, leaving their relative scaling unchanged\. This equivalence concerns conditional means, not update covariances or stochastic trajectories\. Figure[1](https://arxiv.org/html/2610.31728#S3.F1)confirms the common directional gain for an anisotropic signal\. Appendix[A](https://arxiv.org/html/2610.31728#A1)gives the proof via rotational symmetry\. In the fixed\-dimensional low\-SNR regime, directional preconditioning in the first\-order mean response therefore requires structure absent from this isotropic benchmark, motivating our analysis of structured sampling\.

### 4Structured Sampling in Associative Memory

#### 4\.1Associative memory and stochastic gradients

We study a linear associative memory\([Wang et al\., 2026](https://arxiv.org/html/2610.31728#bib.bib6);[Li et al\., 2026b](https://arxiv.org/html/2610.31728#bib.bib7)\)that maps query embeddings to answer logits through a trainable matrixWW\. There areK≥2K\\geq 2associations\(Ej,E~j\)\(E\_\{j\},\\widetilde\{E\}\_\{j\}\), sampled with probabilitiespj\>0p\_\{j\}\>0, where∑jpj=1\\sum\_\{j\}p\_\{j\}=1\. WriteP=diag⁡\(p1,…,pK\)P=\\operatorname\{diag\}\(p\_\{1\},\\ldots,p\_\{K\}\)\. LetE=\[E1,…,EK\]E=\[E\_\{1\},\\ldots,E\_\{K\}\]andE~=\[E~1,…,E~K\]\\widetilde\{E\}=\[\\widetilde\{E\}\_\{1\},\\ldots,\\widetilde\{E\}\_\{K\}\], withE⊤​E=E~⊤​E~=IKE^\{\\top\}E=\\widetilde\{E\}^\{\\top\}\\widetilde\{E\}=I\_\{K\}\. For queryEjE\_\{j\}, answeriihas logitE~i⊤​W​Ej=W¯i​j\\widetilde\{E\}\_\{i\}^\{\\top\}WE\_\{j\}=\\overline\{W\}\_\{ij\}, whereW¯=E~⊤​W​E\\overline\{W\}=\\widetilde\{E\}^\{\\top\}WE\. The prediction probabilities areSi​j=exp⁡\(W¯i​j\)/∑k=1Kexp⁡\(W¯k​j\)S\_\{ij\}=\\exp\(\\overline\{W\}\_\{ij\}\)/\\sum\_\{k=1\}^\{K\}\\exp\(\\overline\{W\}\_\{kj\}\), and the deterministic\-label population loss isℒ\(W\)=−∑j=1KpjlogSj​j\\mathcal\{L\}\(W\)=\-\\sum\_\{j=1\}^\{K\}p\_\{j\}\\log S\_\{jj\}\.

For a batch of sizeBB, letqj=Nj/Bq\_\{j\}=N\_\{j\}/Bbe the empirical task frequency andQ=diag⁡\(q\)Q=\\operatorname\{diag\}\(q\), whereB​q∼Multinomial⁡\(B,p\)Bq\\sim\\operatorname\{Multinomial\}\(B,p\)\. The negative stochastic gradient isE~​\(I−S\)​Q​E⊤\\widetilde\{E\}\(I\-S\)QE^\{\\top\}\(Appendix[B](https://arxiv.org/html/2610.31728#A2)\)\. Orthogonal equivariance of the partial polar map gives the exact task\-coordinate Muon update without momentum:

W¯t\+1=W¯t\+η​msgn⁡\(\(I−St\)​Qt\)\.\\overline\{W\}\_\{t\+1\}=\\overline\{W\}\_\{t\}\+\\eta\\,\\operatorname\{msgn\}\\bigl\(\(I\-S\_\{t\}\)Q\_\{t\}\\bigr\)\.\(1\)If taskjjis absent,qj=0q\_\{j\}=0, its gradient column in task coordinates is zero\. Nullspace preservation then makes the corresponding polar\-update column zero\. This absence guarantee applies to the momentum\-free update; historical momentum can remain nonzero when a task is absent from the current batch\.

#### 4\.2Two tasks: an exact finite\-batch balancing transition

WithK=2K=2, the stochastic gradient is rank one, which makes the finite\-batch sampling effect exactly solvable\. Letp1≥p2=:pmin\>0p\_\{1\}\\geq p\_\{2\}=:p\_\{\\min\}\>0and letΔ1,Δ2\\Delta\_\{1\},\\Delta\_\{2\}be the two target\-vs\.\-nontarget margins\.

###### Theorem 4\.1\(Two\-task finite\-batch balancing transition\)\.

AssumeΔ1​\(0\)=Δ2​\(0\)=0\\Delta\_\{1\}\(0\)=\\Delta\_\{2\}\(0\)=0and fixη\>0\\eta\>0\. LetBbal=log⁡2/\[−log⁡\(1−pmin\)\]B\_\{\\rm bal\}=\\log 2/\[\-\\log\(1\-p\_\{\\min\}\)\]\.

\(i\) Unequal growth\.IfB<BbalB<B\_\{\\rm bal\}, then\(Δ1​\(t\),Δ2​\(t\)\)/\(η​t\)→a\.s\.2​\(p1B,1−p1B\)\(\\Delta\_\{1\}\(t\),\\Delta\_\{2\}\(t\)\)/\(\\eta t\)\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\sqrt\{2\}\\,\(p\_\{1\}^\{B\},1\-p\_\{1\}^\{B\}\)\.

\(ii\) Controlled gap\.IfB\>BbalB\>B\_\{\\rm bal\}, thensupt𝔼​ea​\|Δ1​\(t\)−Δ2​\(t\)\|<∞\\sup\_\{t\}\\mathbb\{E\}e^\{a\|\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\)\|\}<\\inftyfor somea\>0a\>0, and\(Δ1​\(t\)−Δ2​\(t\)\)/t→a\.s\.0\(\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\)\)/t\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.

The threshold is controlled by the probabilityp1Bp\_\{1\}^\{B\}of a batch containing only the frequent task\. As its margin pulls far ahead, a pure task\-1 batch advances onlyΔ1\\Delta\_\{1\}, while every batch containing task 2 asymptotically favors the rare task\. The boundaryp1B=1/2p\_\{1\}^\{B\}=1/2separates outward from restoring tail drift\. Forpmin≪1p\_\{\\min\}\\ll 1,Bbal∼\(log⁡2\)/pminB\_\{\\rm bal\}\\sim\(\\log 2\)/p\_\{\\min\}: batches of order the inverse tail frequency suffice for equal growth to leading linear order\. The critical caseB=BbalB=B\_\{\\rm bal\}is not covered by the theorem\.

#### 4\.3Higher rank: frequency\-dependent dynamical regimes

We next ask how the two\-task results extend when many associations occur at different population frequencies\. To represent multiple frequency scales while retaining a tractable matrix structure, we group theK=M​CK=MCassociations intoMMblocks of sizeCC\. Blockμ\\muhas total probabilityp~μ\\widetilde\{p\}\_\{\\mu\}, distributed uniformly among itsCCtasks, so thatpj=p~μ/Cp\_\{j\}=\\widetilde\{p\}\_\{\\mu\}/Cforj∈μj\\in\\mu, with∑μ=1Mp~μ=1\\sum\_\{\\mu=1\}^\{M\}\\widetilde\{p\}\_\{\\mu\}=1\. This block model can be viewed as a coarse\-grained representation of a heterogeneous frequency distribution: different blocks correspond to different frequency levels, while tasks within the same block are statistically exchangeable\. We takeW¯0=0\\overline\{W\}\_\{0\}=0\. Full\-batch dynamics preserve this block symmetry, whereas mini\-batch fluctuations generally break it\. Fori≠ji\\neq jin the same block, the marginW¯j​j−W¯i​j\\overline\{W\}\_\{jj\}\-\\overline\{W\}\_\{ij\}compares the correct answer with an incorrect answer from that block\. The first\-order expansion is in within\-block count fluctuations at fixed block totals \(Appendix[D\.4](https://arxiv.org/html/2610.31728#A4.SS4)\)\.

###### Proposition 4\.2\(Sampling and within\-block margin growth\)\.

Consider momentum\-free Muon andi≠ji\\neq jin the same block\.

\(i\) Full batch\.FromW¯0=0\\overline\{W\}\_\{0\}=0, each marginW¯j​j−W¯i​j\\overline\{W\}\_\{jj\}\-\\overline\{W\}\_\{ij\}increases by exactlyη\\etaper step, independently of task frequency\.

\(ii\) Sparse sampling\.This margin is unchanged when taskjjis absent; the task appears with probability1−\(1−pj\)B∼B​pj1\-\(1\-p\_\{j\}\)^\{B\}\\sim Bp\_\{j\}asB​pj→0Bp\_\{j\}\\to 0\.

\(iii\) Well\-sampled blocks\.Under the local, well\-sampled assumptions of Appendix[D\.5](https://arxiv.org/html/2610.31728#A4.SS5), at a fixed block\-symmetric state the first\-order margin increment is

η\+Oℙ​\(ηB​pj\),B​pj→∞\.\\eta\+O\_\{\\mathbb\{P\}\}\\\!\\left\(\\frac\{\\eta\}\{\\sqrt\{Bp\_\{j\}\}\}\\right\),\\qquad Bp\_\{j\}\\to\\infty\.

Thus sparse sampling limits update opportunities, whereas well\-sampled blocks recover the full\-batch margin increment to first order\. The underlying contrast\-to\-contrast perturbation cancels, leaving only coupling to block\-average directions\. These local estimates do not establish uniform long\-time balancing of the stochastic hierarchy\. Appendix[D\.7](https://arxiv.org/html/2610.31728#A4.SS7)separately proves finite\-horizon convergence to the full\-batch flow under large\-batch and small\-step limits\.

Figure[2](https://arxiv.org/html/2610.31728#S4.F2)\(b\) shows corresponding trajectory\-level crossovers: measured late margin\-growth rates approximately align againstB​pjBp\_\{j\}and approach the full\-batch response as expected task counts increase\. Frequent and rare groups can therefore exhibit different empirical regimes at the same batch size\.

Figure 2:Mini\-batch sampling controls the transition from frequency\-limited to balanced growth\.Left: the two\-task gap\-growth transition occurs at the exact thresholdB/Bbal=1B/B\_\{\\rm bal\}=1\. Right: in the higher\-rank hierarchy, late margin\-growth rates from different frequency blocks collapse when plotted against the expected task countB​pjBp\_\{j\}and approach the full\-batch unit response as that count increases\. Error bars summarize 64 trajectories; the dotted curve is the task\-presence guide1−e−B​pj1\-e^\{\-Bp\_\{j\}\}\.Appendices[C](https://arxiv.org/html/2610.31728#A3)and[D](https://arxiv.org/html/2610.31728#A4)give the exact two\-task proof and the higher\-rank perturbation analysis underlying the approach to balanced dynamics\.

#### 4\.4Label noise: three stochastic regimes

We next make the observed answer stochastic, with0<α<10<\\alpha<1\. Conditional on taskjj, symmetric label noise leaves the target label unchanged with probabilitycα=1−α\+α/Kc\_\{\\alpha\}=1\-\\alpha\+\\alpha/Kand returns a nontarget label with aggregate probabilityρα=1−cα\\rho\_\{\\alpha\}=1\-c\_\{\\alpha\}\. The crucial difference from task sampling is that label fluctuations remain nonzero at the finite population optimum\.

Letdjd\_\{j\}be the conditional label distribution,δj=S​ej−dj\\delta\_\{j\}=Se\_\{j\}\-d\_\{j\}, andΓj=diag⁡\(dj\)−dj​dj⊤\\Gamma\_\{j\}=\\operatorname\{diag\}\(d\_\{j\}\)\-d\_\{j\}d\_\{j\}^\{\\top\}\. For the stochastic gradient columng^j\\widehat\{g\}\_\{j\}, defineSNRj2=‖𝔼​g^j‖22/tr⁡Cov⁡\(g^j\)\\mathrm\{SNR\}\_\{j\}^\{2\}=\\\|\\mathbb\{E\}\\widehat\{g\}\_\{j\}\\\|\_\{2\}^\{2\}/\\operatorname\{tr\}\\operatorname\{Cov\}\(\\widehat\{g\}\_\{j\}\)\.

###### Proposition 4\.3\(Task frequency and gradient reliability decouple under label noise\)\.

\(i\) Deterministic labels\.Forδj≠0\\delta\_\{j\}\\neq 0,SNRj2=B​pj/\(1−pj\)\\mathrm\{SNR\}\_\{j\}^\{2\}=Bp\_\{j\}/\(1\-p\_\{j\}\)\.

\(ii\) Nondegenerate label noise\.Asδj→0\\delta\_\{j\}\\to 0,SNRj2∼B​pj​‖δj‖22/tr⁡Γj→0\\mathrm\{SNR\}\_\{j\}^\{2\}\\sim Bp\_\{j\}\\\|\\delta\_\{j\}\\\|\_\{2\}^\{2\}/\\operatorname\{tr\}\\Gamma\_\{j\}\\to 0\.

With deterministic labels, the stochastic fluctuation of a task shrinks together with its residual, so the number of task observations is enough to characterize directional reliability\. Label noise breaks this one\-scale picture: near the population optimum the mean residual vanishes, while the random observed label continues to fluctuate\. A task can therefore be sampled many times \(B​pj≫1Bp\_\{j\}\\gg 1\) and still have a low\-SNR gradient\.

Label noise introduces a second sampling scale,B​pj​ραBp\_\{j\}\\rho\_\{\\alpha\}, the expected number of corrupted task\-jjobservations\. Whenρα\\rho\_\{\\alpha\}is small, it can be much smaller than the task countB​pjBp\_\{j\}, leaving a broad intermediate range in which the task appears regularly but corruptions are still rare\. WritemB,p​\(Δ\)m\_\{B,p\}\(\\Delta\)for the mean margin increment divided byη\\etain the scalar counting model of Appendix[E\.4](https://arxiv.org/html/2610.31728#A5.SS4)\.

###### Proposition 4\.4\(Projected regimes and local closure under label noise\)\.

Assumeρα<1/2\\rho\_\{\\alpha\}<1/2\.

\(i\) Projected counting regimes\.Withp=pjp=p\_\{j\},

mB,p​\(Δ\)\\displaystyle m\_\{B,p\}\(\\Delta\)=B​p​\(1−2​ρα\)\+O⁡\(\(B​p\)2\),\\displaystyle=Bp\(1\-2\\rho\_\{\\alpha\}\)\+O\(\(Bp\)^\{2\}\),B​p≪1,\\displaystyle Bp\\ll 1,limΔ→∞mB,p​\(Δ\)\\displaystyle\\lim\_\{\\Delta\\to\\infty\}m\_\{B,p\}\(\\Delta\)≈1,\\displaystyle\\approx 1,1≪B​p≪ρα−1,\\displaystyle 1\\ll Bp\\ll\\rho\_\{\\alpha\}^\{\-1\},limΔ→∞mB,p​\(Δ\)\\displaystyle\\lim\_\{\\Delta\\to\\infty\}m\_\{B,p\}\(\\Delta\)<0,\\displaystyle<0,B​p​ρα≫1\.\\displaystyle Bp\\rho\_\{\\alpha\}\\gg 1\.\(ii\) Local closure\.Near the noisy population optimum, the isotropic covariance closure with a locally homogeneous frequency scalepppredicts contraction coefficients proportional toB​p\\sqrt\{Bp\}for Muon andppfor SGD\.

The counting model separates occurrence\-limited, corruption\-sparse, and corruption\-resolved behavior; a broad intermediate window requires smallρα\\rho\_\{\\alpha\}\. Resolving corruptions reverses the projected large\-margin drift, motivating a separate local analysis near the noisy optimum\. The closure predicts weaker frequency dependence for Muon than for SGD\. Both analyses use auxiliary models; Figure[3](https://arxiv.org/html/2610.31728#S4.F3)tests the closure prediction in the complete categorical model\.

Figure 3:Corrupted observations introduce a second sampling scale\.Left: full\-matrix Muon changes from positive to negative large\-margin drift as the expected number of corrupted task\-jjobservations,B​pj​ραBp\_\{j\}\\rho\_\{\\alpha\}, crosses an order\-one scale; dashed curves show the exact projected counting formula, whose crossover approacheslog⁡2\\log 2in the Poisson limit\. Right: near the population optimum under label noise, the full\-categorical Muon response scales empirically aspj0\.494p\_\{j\}^\{0\.494\}, close to the square\-root prediction under the auxiliary covariance and frequency approximations, whereas the SGD coefficient scales linearly inpjp\_\{j\}\.Appendix[E](https://arxiv.org/html/2610.31728#A5)derives the complete categorical covariance and local Gaussian limit, then the projected counting dynamics and covariance closure used for the explicit regime formulas\.

#### 4\.5Sparse support: Muon versus SignSGD

Without momentum, sparse mini\-batch sampling also distinguishes polar normalization from coordinatewise sign normalization\. Muon is orthogonally equivariant and preserves the missing\-task nullspace; SignSGD is coordinate dependent, so a basis rotation can turn an update for an observed task into an update of an unobserved task\.

Consider momentum\-free updates in the two\-task model with query embeddingE=\[cos⁡θsin⁡θ−sin⁡θcos⁡θ\]E=\\bigl\[\\begin\{smallmatrix\}\\cos\\theta&\\sin\\theta\\\\ \-\\sin\\theta&\\cos\\theta\\end\{smallmatrix\}\\bigr\],0<θ<π/20<\\theta<\\pi/2, and letc~=‖E~​\(1,−1\)⊤‖1\\widetilde\{c\}=\\\|\\widetilde\{E\}\(1,\-1\)^\{\\top\}\\\|\_\{1\}\.

###### Proposition 4\.5\(Muon avoids rotation\-induced updates to a missing task\)\.

A batch with no task\-2 samples givesΔ2Muon​\(t\+1\)−Δ2Muon​\(t\)=0\\Delta\_\{2\}^\{\\rm Muon\}\(t\+1\)\-\\Delta\_\{2\}^\{\\rm Muon\}\(t\)=0, whereasΔ2Sign​\(t\+1\)−Δ2Sign​\(t\)=η​c~​\(cos⁡θ−sin⁡θ\)\\Delta\_\{2\}^\{\\rm Sign\}\(t\+1\)\-\\Delta\_\{2\}^\{\\rm Sign\}\(t\)=\\eta\\widetilde\{c\}\(\\cos\\theta\-\\sin\\theta\)\. Ifp1B\>\(1\+cot⁡θ\)/2p\_\{1\}^\{B\}\>\(1\+\\cot\\theta\)/2, the rare\-task SignSGD margin decays linearly almost surely\.

Forθ\>π/4\\theta\>\\pi/4, every frequent\-only batch therefore decreases the rare\-task SignSGD margin\. The contrast is especially clear atB=1B=1\. Muon updates only the sampled task: the rare margin still grows, but on its slow occurrence clock\. After rotation, the same stream of frequent\-task observations produces a negative component in the rare SignSGD margin\. Figure[4](https://arxiv.org/html/2610.31728#S4.F4)shows the long\-time consequence forp2=0\.05p\_\{2\}=0\.05andθ=1\.15\\theta=1\.15radians\.

Figure 4:Sparse support yields slow positive tail learning for Muon but negative transfer for SignSGD\.Both panels use the sameB=1B=1,p1=0\.95p\_\{1\}=0\.95,p2=0\.05p\_\{2\}=0\.05mini\-batch streams over 64 runs\. Left: Muon preserves support, so the rare\-task margin remains nonnegative and grows on its occurrence clock\. Right: atθ=1\.15\>π/4\\theta=1\.15\>\\pi/4, rotated SignSGD drives the rare\-task margin downward while the frequent task continues to improve\. The positive answer\-embedding scalarc~\\widetilde\{c\}is absorbed into the SignSGD learning rate\. Curves show medians and ribbons show interquartile ranges\.Appendix[F](https://arxiv.org/html/2610.31728#A6)gives the exact rotated recursion and the long\-time negative\-transfer proof\.

### 5Momentum: Averaging and Tracking

Momentum changes the sampling regime seen by the polar map by averaging stochastic gradients across optimization steps\.

###### Proposition 5\.1\(Averaging scale and tracking under momentum\)\.

\(i\) Averaging\.The stationary EMA task frequency has variance\-equivalent batch sizeBeff​\(β\)=B⁡\(1\+β\)/\(1−β\)B\_\{\\rm eff\}\(\\beta\)=B\(1\+\\beta\)/\(1\-\\beta\)\. Relative frequency fluctuations vanish asBeff​pj→∞B\_\{\\rm eff\}p\_\{j\}\\to\\infty\.

\(ii\) Tracking\.If‖St−St−r‖F≤LS​η​r\\\|S\_\{t\}\-S\_\{t\-r\}\\\|\_\{F\}\\leq L\_\{S\}\\eta r, the frozen\-state buffer error is at mostLS​η​β/\(1−β\)L\_\{S\}\\eta\\beta/\(1\-\\beta\)and vanishes whenη/\(1−β\)→0\\eta/\(1\-\\beta\)\\to 0\.

At a frozen state in the deterministic\-label model, the stationary EMAMt=\(1−β\)​∑r≥0βr​Gt−rM\_\{t\}=\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}G\_\{t\-r\}can be expressed using exponentially averaged task frequencies, reducing their sampling variance without changing the physical mini\-batch size\.

At a frozen state, replacingBBbyBeffB\_\{\\rm eff\}reproduces task\-frequency variances, but not task\-absence probabilities or the full batch distribution\. Forβ\\betaclose to one,Beff≈2​B/\(1−β\)B\_\{\\rm eff\}\\approx 2B/\(1\-\\beta\), so1−β≪B​pj1\-\\beta\\ll Bp\_\{j\}suppresses relative frequency fluctuations\. EMA updates are also temporally correlated: reduced instantaneous variance need not reduce long\-time diffusion \(Appendix[C\.5](https://arxiv.org/html/2610.31728#A3.SS5)\)\.

Under the local tracking assumptions, combining averaging with a small frozen\-state buffer error gives the sufficient windowη≪1−β≪B​pj\\eta\\ll 1\-\\beta\\ll Bp\_\{j\}\. Transferring a buffer\-error bound to a polar\-direction bound additionally requires local control of the nonzero singular values or of relative perturbations\. The window is therefore a local design guide, not a global convergence guarantee\. It motivates longer memory for rare, slowly varying directions and shorter memory where rapid tracking is needed\. Under stochastic labels, averaging also reduces persistent label fluctuations\.

Figure 5:Momentum trades variance reduction against tracking\.Left: different physical batch sizes and momentum coefficients nearly collapse when the two\-task gap bias is plotted against the variance\-equivalent task countBeff​pminB\_\{\\rm eff\}p\_\{\\min\}\. Right: withBBandβ\\betafixed and a common horizonτmax=50\\tau\_\{\\max\}=50, the late\-window mean\-gap error stays small whileη/\(1−β\)\\eta/\(1\-\\beta\)is small and grows once the EMA horizon becomes comparable to the state\-evolution time scale\.Appendix[D\.6](https://arxiv.org/html/2610.31728#A4.SS6)derives the EMA variance and tracking decomposition, and Appendix[G](https://arxiv.org/html/2610.31728#A7)gives a complementary scalar bias–variance calculation\.

### 6Fast–Slow Momentum in Language\-Model Pretraining

##### From theory to practice\.

Our analysis motivates longer memory for rare, slowly varying directions and shorter memory where rapid tracking is needed\. To examine this principle in language\-model pretraining, we use the curvature\-conditioned fast–slow momentum construction of[Zhu et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib11)\. It retains the fast Muon update and adds a slow\-momentum contribution in estimated flat directions, identified through the lower singular subspaces of the fast momentum\. This provides a practical counterpart to the averaging–tracking trade\-off in our theory, without requiring explicit task\-frequency estimates\.

Fast–slow Muon \(schematic\)Decay rates0≤βf<βs<10\\leq\\beta\_\{\\rm f\}<\\beta\_\{\\rm s\}<1; mixing weightλ\>0\\lambda\>0; step sizeηt\\eta\_\{t\}\. InitializeMf=Ms=0M^\{\\rm f\}=M^\{\\rm s\}=0\.1:foreach stepttdo2:Compute gradientGtG\_\{t\}3:Mf←βf​Mf\+GtM^\{\\rm f\}\\leftarrow\\beta\_\{\\rm f\}M^\{\\rm f\}\+G\_\{t\}4:Ms←βs​Ms\+GtM^\{\\rm s\}\\leftarrow\\beta\_\{\\rm s\}M^\{\\rm s\}\+G\_\{t\}5:Ht←βf​Mf\+GtH\_\{t\}\\leftarrow\\beta\_\{\\rm f\}M^\{\\rm f\}\+G\_\{t\}6:Estimate flat projector𝒬t\\mathcal\{Q\}\_\{t\}fromHtH\_\{t\}7:Dtf←msgn⁡\(Ht\)D^\{\\rm f\}\_\{t\}\\leftarrow\\operatorname\{msgn\}\(H\_\{t\}\)8:Dts←𝒬t​\(RowScale⁡\(Ms\)\)D^\{\\rm s\}\_\{t\}\\leftarrow\\mathcal\{Q\}\_\{t\}\(\\operatorname\{RowScale\}\(M^\{\\rm s\}\)\)9:Wt\+1←Wt−ηt​\(Dtf\+λ​Dts\)W\_\{t\+1\}\\leftarrow W\_\{t\}\-\\eta\_\{t\}\(D^\{\\rm f\}\_\{t\}\+\\lambda D^\{\\rm s\}\_\{t\}\)10:endfor

Figure 6:Fast–slow momentum: core update and pretraining results\.Left: schematic momentum mixing following[Zhu et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib11);RowScale\\operatorname\{RowScale\}normalizes rows, and parameter and update scaling follow the cited implementation\. Right: validation loss from steps 20k to 30k for a 0\.12B dense language model, with a final reduction of0\.02090\.0209\. Other training settings are fixed; curves are unsmoothed, with one run per method\.
##### Pretraining results\.

We compare standard Muon with this fast–slow variant on a 0\.12B\-parameter dense language model trained for 30,000 steps, keeping the remaining training configuration fixed\. Across all 138 shared checkpoints from step 2,600 to 30,000, the fast–slow variant has lower validation loss\. Figure[6](https://arxiv.org/html/2610.31728#S6.F6)shows the final 10,000 steps\. At step 30,000, validation loss decreases from3\.00973\.0097to2\.98882\.9888, a reduction of0\.02090\.0209\. These single\-run results illustrate the practical relevance of heterogeneous momentum; variability across seeds and the effects of individual components remain to be evaluated\. Appendix[H\.8](https://arxiv.org/html/2610.31728#A8.SS8)documents the trajectory comparison\.

### 7Conclusion

We study when Muon’s preconditioning benefits survive stochastic gradient noise\. Under isotropic low\-SNR noise, its first\-order conditional mean update matches learning\-rate\-rescaled SGD, showing that samplewise spectral normalization alone does not guarantee directional preconditioning\. Structured sampling yields richer behavior: we establish an exact two\-task balancing threshold and analyze higher\-rank sampling effects governed byB​pjBp\_\{j\}\. Label noise further separates task coverage from gradient reliability; an isotropic covariance closure with a locally homogeneous frequency approximation predicts square\-root frequency dependence near the noisy optimum\. A separate momentum\-free SignSGD comparison highlights Muon’s protection against rotation\-induced negative transfer under sparse sampling\.

Our momentum analysis connects frozen\-state variance reduction with local tracking error, motivating longer memory for rare, slowly varying directions\. This provides a sampling\-based interpretation of heterogeneous momentum designs, supported by a preliminary 0\.12B pretraining experiment in which fast–slow momentum lowers validation loss relative to Muon\. Extending the analysis to evolving, non\-orthogonal representations and validating these benefits across training scales remain important directions\.

#### AI use statement

In this work, we used generative AI tools to assist with technical aspects of some mathematical proofs and derivations, and to assist with translation of the manuscript\. We did not use generative AI tools to formulate the research questions, conceptual framework, or theorem statements\. Additionally, we used generative AI tools to generate and edit code for controlled synthetic numerical experiments and figure generation\. We reviewed all AI\-assisted work\. All AI\-assisted mathematical arguments were checked and verified by the authors; AI\-generated numerical code was inspected and tested for correctness, and all translated or edited text and generated figures were reviewed by the authors\. We take responsibility for the final content of this work, including text, claims, code\-derived results, and other artifacts produced with the aid of generative AI\.

### References

- L\. Balles and P\. HennigDissecting adam: the sign, magnitude and variance of stochastic gradients\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 404–413\.External Links:[Link](https://proceedings.mlr.press/v80/balles18a.html)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p7.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4\.1\-Flash: pushing the limits of KV cache compression\.Note:arXiv:2609\.19969External Links:[Link](https://arxiv.org/abs/2609.19969)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p1.1)\.
- Eschenhagenet al\.\(2026\)R\. Eschenhagen, A\. Cai, T\. Lee, and H\. M\. ShiClarifying Shampoo: adapting spectral descent to stochasticity and the parameter trajectory\.Note:arXiv:2602\.09314External Links:[Link](https://arxiv.org/abs/2602.09314)Cited by:[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px2.p1.1)\.
- Ghadimi and Lan \(2013\)S\. Ghadimi and G\. LanStochastic first\- and zeroth\-order methods for nonconvex stochastic programming\.Note:arXiv:1309\.5549External Links:[Link](https://arxiv.org/abs/1309.5549)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p4.1)\.
- Hong and Qu \(2026\)E\. Hong and G\. QuActivation\-Keyed Momentum: an anisotropic momentum update via the delta rule\.Note:arXiv:2608\.19491External Links:[Link](https://arxiv.org/abs/2608.19491)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p8.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px4.p1.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.Note:Online technical noteExternal Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p1.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Kim and Oh \(2026\)G\. Y\. Kim and M\. OhConvergence of Muon with Newton–Schulz\.InInternational Conference on Learning Representations,Note:arXiv:2601\.19156External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/4c0129f1f7a036af5ed2409f1f9f8131-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p2.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2026\)J\. Kim, E\. Nichani, D\. Wu, A\. Bietti, and J\. D\. LeeSharp capacity scaling of spectral optimizers in learning associative memory\.Note:arXiv:2603\.26554External Links:[Link](https://arxiv.org/abs/2603.26554)Cited by:[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px3.p1.1)\.
- Kimi Teamet al\.\(2026\)Kimi Teamet al\.Kimi K3: open frontier intelligence\.Note:arXiv:2607\.24653External Links:[Link](https://arxiv.org/abs/2607.24653)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p1.1)\.
- Kunstneret al\.\(2023\)F\. Kunstner, J\. Chen, J\. W\. Lavington, and M\. SchmidtNoise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=a65YK0cqH8g)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p7.1)\.
- Liet al\.\(2026a\)B\. Li, J\. Wang, J\. Wang, L\. Wang, Z\. Wang, and L\. WuScaling\-law analysis of SignSGD: from feature\-space linear regression to LLM pre\-training\.InSciForDL, 2nd edition,External Links:[Link](https://openreview.net/forum?id=KQfV97S12I)Cited by:[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026b\)B\. Li, K\. Wang, H\. Zhong, P\. Lu, and L\. WangMuon in associative memory learning: training dynamics and scaling laws\.InInternational Conference on Machine Learning,Note:arXiv:2602\.05725External Links:[Link](https://arxiv.org/abs/2602.05725)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p2.1),[§1](https://arxiv.org/html/2610.31728#S1.p5.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.31728#S4.SS1.p1.1)\.
- Liet al\.\(2026c\)X\. Li, Z\. Zhang, W\. Liu, and H\. BaoDenoise first, orthogonalize later: understanding momentum in Muon via spectral filtering\.Note:arXiv:2606\.03899External Links:[Link](https://arxiv.org/abs/2606.03899)Cited by:[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px4.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. YangMuon is scalable for LLM training\.Note:arXiv:2502\.16982External Links:[Link](https://arxiv.org/abs/2502.16982)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p1.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Maet al\.\(2026\)J\. Ma, Y\. Huang, Y\. Chi, and Y\. ChenPreconditioning benefits of spectral orthogonalization in Muon\.Note:arXiv:2601\.13474External Links:[Link](https://arxiv.org/abs/2601.13474)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p2.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Magakyanet al\.\(2026\)G\. Magakyan, P\. A\. Parrilo, and A\. OzdaglarSpectral scaling laws of Muon\.Note:arXiv:2606\.04058External Links:[Link](https://arxiv.org/abs/2606.04058)Cited by:[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Naganumaet al\.\(2026\)H\. Naganuma, S\. Gupta, Y\. Briki, I\. Mitliagkas, I\. Rish, P\. Raman, and H\. M\. ShiAdaptive batch sizes using non\-euclidean gradient noise scales for stochastic sign and spectral descent\.InInternational Conference on Machine Learning,Note:arXiv:2602\.03001External Links:[Link](https://arxiv.org/abs/2602.03001)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p2.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px2.p1.1)\.
- Paquetteet al\.\(2026\)E\. Paquette, N\. Marshall, L\. Benigni, G\. Wang, A\. Agarwala, and C\. PaquettePhases of Muon: when Muon eclipses SignSGD\.Note:arXiv:2605\.09552External Links:[Link](https://arxiv.org/abs/2605.09552)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p2.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px2.p1.1)\.
- Vasudevaet al\.\(2026\)B\. Vasudeva, P\. Deora, Y\. Zhao, V\. Sharan, and C\. ThrampoulidisHow Muon’s spectral design benefits generalization: a study on imbalanced data\.InInternational Conference on Learning Representations,Note:arXiv:2510\.22980External Links:[Link](https://openreview.net/forum?id=YzjS4jcfmS)Cited by:[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)S\. Wang, F\. Zhang, J\. Li, C\. Du, C\. Du, T\. Pang, Z\. Yang, M\. Hong, and V\. Y\. F\. TanMuon outperforms Adam in tail\-end associative memory learning\.InInternational Conference on Learning Representations,Note:arXiv:2509\.26030External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/3eec5006051d9544e717067de3220198-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.31728#S1.p2.1),[§1](https://arxiv.org/html/2610.31728#S1.p5.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.31728#S4.SS1.p1.1)\.
- Zhuet al\.\(2026\)S\. Zhu, Y\. Fang, M\. Wang, and K\. YuanCurvature\-conditioned multiscale momentum with sphere constraints for LLM pretraining\.Note:arXiv:2608\.28442External Links:[Link](https://arxiv.org/abs/2608.28442)Cited by:[§H\.8](https://arxiv.org/html/2610.31728#A8.SS8.p1.1),[§1](https://arxiv.org/html/2610.31728#S1.p8.1),[§2](https://arxiv.org/html/2610.31728#S2.SS0.SSS0.Px4.p1.1),[Figure 6](https://arxiv.org/html/2610.31728#S6.F6),[§6](https://arxiv.org/html/2610.31728#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix

Guide to the appendices\.Appendix A proves the isotropic Gaussian response result\. Appendix B records the associative\-memory setup used throughout the structured analysis\. Appendix C gives the exact two\-task dynamics and their large\-batch/momentum refinements\. Appendix D supports the higher\-rank regime map through sparse\-task gating, full\-batch contrast dynamics, well\-sampled perturbation analysis, momentum averaging, and a final technical continuum refinement\. Appendix E separates exact categorical label\-noise results from the projected counting model and the additional covariance closure\. Appendix F proves the Muon–SignSGD sparse\-support contrast\. Appendix G contains an optional scalar EMA bias–variance calculation, and Appendix H collects numerical protocols and supplementary diagnostics\.

### Appendix AIsotropic Gaussian Mean Response: Proof of Proposition[3\.1](https://arxiv.org/html/2610.31728#S3.Thmtheorem1)

Throughout this appendix, we condition on the current iterate and suppress the time index\. We write

G=G¯\+σ​Ξ,G¯=∇ℒ​\(W\),G=\\bar\{G\}\+\\sigma\\Xi,\\qquad\\bar\{G\}=\\nabla\\mathcal\{L\}\(W\),whereΞ∈ℝd×d\\Xi\\in\\mathbb\{R\}^\{d\\times d\}has i\.i\.d\.𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)entries andσ\>0\\sigma\>0\.

#### A\.1Gaussian\-smoothing proof

Define

F⁡\(H\):=𝔼⁡\[msgn⁡\(Ξ\+H\)\],H∈ℝd×d,F\(H\):=\\mathbb\{E\}\[\\operatorname\{msgn\}\(\\Xi\+H\)\],\\qquad H\\in\\mathbb\{R\}^\{d\\times d\},and write

⟨A,B⟩F:=tr⁡\(A⊤​B\)\.\\langle A,B\\rangle\_\{F\}:=\\operatorname\{tr\}\(A^\{\\top\}B\)\.
##### Smoothness by Gaussian convolution\.

Let

ϕ\(A\)=\(2π\)−d2/2exp\(−12∥A∥F2\)\\phi\(A\)=\(2\\pi\)^\{\-d^\{2\}/2\}\\exp\\\!\\left\(\-\\frac\{1\}\{2\}\\\|A\\\|\_\{F\}^\{2\}\\right\)be the standard Gaussian density onℝd×d\\mathbb\{R\}^\{d\\times d\}\. Changing variables yields

F⁡\(H\)\\displaystyle F\(H\)=∫msgn⁡\(A\)​ϕ​\(A−H\)​𝑑A\\displaystyle=\\int\\operatorname\{msgn\}\(A\)\\phi\(A\-H\)\\,\\mathrm\{d\}A=𝔼⁡\[msgn⁡\(Ξ\)​exp⁡\(⟨H,Ξ⟩F−12​‖H‖F2\)\]\.\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\operatorname\{msgn\}\(\\Xi\)\\exp\\\!\\left\(\\langle H,\\Xi\\rangle\_\{F\}\-\\frac\{1\}\{2\}\\\|H\\\|\_\{F\}^\{2\}\\right\)\\right\]\.For‖H‖F≤r\\\|H\\\|\_\{F\}\\leq r, derivatives of any fixed order of the exponential factor are bounded by a constant, depending only on the derivative order andrr, times a polynomial in‖Ξ‖F\\\|\\Xi\\\|\_\{F\}multiplied byer​‖Ξ‖Fe^\{r\\\|\\Xi\\\|\_\{F\}\}\. Since‖msgn⁡\(Ξ\)‖F≤d\\\|\\operatorname\{msgn\}\(\\Xi\)\\\|\_\{F\}\\leq\\sqrt\{d\}and

𝔼⁡\[\(1\+‖Ξ‖Fk\)​er​‖Ξ‖F\]<∞\\mathbb\{E\}\\\!\\left\[\(1\+\\\|\\Xi\\\|\_\{F\}^\{k\}\)e^\{r\\\|\\Xi\\\|\_\{F\}\}\\right\]<\\inftyfor every fixedk≥0k\\geq 0andr\>0r\>0, differentiation under the expectation is justified locally to every order\. The same domination also implies continuity of the resulting derivatives\. HenceFFis smooth, and

D​F​\(0\)​\[X\]=𝔼⁡\[msgn⁡\(Ξ\)​⟨X,Ξ⟩F\]DF\(0\)\[X\]=\\mathbb\{E\}\\\!\\left\[\\operatorname\{msgn\}\(\\Xi\)\\langle X,\\Xi\\rangle\_\{F\}\\right\]\(2\)for everyX∈ℝd×dX\\in\\mathbb\{R\}^\{d\\times d\}\. This argument differentiates the shifted Gaussian density, not the nonsmooth mapmsgn\\operatorname\{msgn\}\.

##### Orthogonal equivariance of the derivative\.

We use the following elementary lemma\.

###### Lemma A\.1\.

LetT:ℝd×d→ℝd×dT:\\mathbb\{R\}^\{d\\times d\}\\to\\mathbb\{R\}^\{d\\times d\}be linear and satisfy

T⁡\(U​X​V⊤\)=U​T​\(X\)​V⊤T\(UXV^\{\\top\}\)=UT\(X\)V^\{\\top\}for everyX∈ℝd×dX\\in\\mathbb\{R\}^\{d\\times d\}and everyU,V∈O⁡\(d\)U,V\\in O\(d\)\. ThenT⁡\(X\)=c​XT\(X\)=cXfor some scalarc∈ℝc\\in\\mathbb\{R\}and allXX\.

###### Proof\.

Fixi,ji,j, and letD1,D2D\_\{1\},D\_\{2\}be arbitrary diagonal sign matrices satisfying\(D1\)i​i=\(D2\)j​j=1\(D\_\{1\}\)\_\{ii\}=\(D\_\{2\}\)\_\{jj\}=1\. Then

D1​Ei​j​D2⊤=Ei​j\.D\_\{1\}E\_\{ij\}D\_\{2\}^\{\\top\}=E\_\{ij\}\.Hence equivariance gives

T⁡\(Ei​j\)=D1​T​\(Ei​j\)​D2⊤\.T\(E\_\{ij\}\)=D\_\{1\}T\(E\_\{ij\}\)D\_\{2\}^\{\\top\}\.Varying all remaining diagonal signs independently forces every entry ofT⁡\(Ei​j\)T\(E\_\{ij\}\)except entry\(i,j\)\(i,j\)to vanish\. ThusT⁡\(Ei​j\)=ci​j​Ei​jT\(E\_\{ij\}\)=c\_\{ij\}E\_\{ij\}\. Independent row and column permutations map anyEi​jE\_\{ij\}to any other matrix unit, so equivariance forces allci​jc\_\{ij\}to equal a common scalarcc\. Linearity completes the proof\. ∎

The identities

msgn⁡\(U​A​V⊤\)=U​msgn⁡\(A\)​V⊤,U⊤​Ξ​V=dΞ\\operatorname\{msgn\}\(UAV^\{\\top\}\)=U\\operatorname\{msgn\}\(A\)V^\{\\top\},\\qquad U^\{\\top\}\\Xi V\\stackrel\{\{\\scriptstyle\\mathrm\{d\}\}\}\{\{=\}\}\\Xihold for allU,V∈O⁡\(d\)U,V\\in O\(d\)\. Therefore

F⁡\(U​H​V⊤\)\\displaystyle F\(UHV^\{\\top\}\)=U​𝔼​\[msgn⁡\(U⊤​Ξ​V\+H\)\]​V⊤\\displaystyle=U\\,\\mathbb\{E\}\\\!\\left\[\\operatorname\{msgn\}\(U^\{\\top\}\\Xi V\+H\)\\right\]V^\{\\top\}=U​F​\(H\)​V⊤\.\\displaystyle=UF\(H\)V^\{\\top\}\.Differentiating at zero gives

D​F​\(0\)​\[U​X​V⊤\]=U​D​F​\(0\)​\[X\]​V⊤\.DF\(0\)\[UXV^\{\\top\}\]=UDF\(0\)\[X\]V^\{\\top\}\.By Lemma[A\.1](https://arxiv.org/html/2610.31728#A1.Thmtheorem1), there is a scalarcdc\_\{d\}such that

D​F​\(0\)​\[X\]=cd​X\.DF\(0\)\[X\]=c\_\{d\}X\.

##### Identification of the coefficient\.

Taking the trace of this linear operator in the orthonormal basis\{Ei​j\}i,j=1d\\\{E\_\{ij\}\\\}\_\{i,j=1\}^\{d\}and using equation[2](https://arxiv.org/html/2610.31728#A1.E2), we obtain

d2​cd\\displaystyle d^\{2\}c\_\{d\}=∑i,j⟨Ei​j,D​F​\(0\)​\[Ei​j\]⟩F\\displaystyle=\\sum\_\{i,j\}\\langle E\_\{ij\},DF\(0\)\[E\_\{ij\}\]\\rangle\_\{F\}=∑i,j𝔼⁡\[msgn⁡\(Ξ\)i​j​Ξi​j\]\\displaystyle=\\sum\_\{i,j\}\\mathbb\{E\}\[\\operatorname\{msgn\}\(\\Xi\)\_\{ij\}\\Xi\_\{ij\}\]=𝔼​⟨msgn⁡\(Ξ\),Ξ⟩F\.\\displaystyle=\\mathbb\{E\}\\langle\\operatorname\{msgn\}\(\\Xi\),\\Xi\\rangle\_\{F\}\.For an SVDΞ=U​Σ​V⊤\\Xi=U\\Sigma V^\{\\top\},

⟨msgn⁡\(Ξ\),Ξ⟩F=tr⁡\(Σ\)=‖Ξ‖∗\.\\langle\\operatorname\{msgn\}\(\\Xi\),\\Xi\\rangle\_\{F\}=\\operatorname\{tr\}\(\\Sigma\)=\\\|\\Xi\\\|\_\{\*\}\.Consequently,

cd=1d2​𝔼​‖Ξ‖∗\>0\.c\_\{d\}=\\frac\{1\}\{d^\{2\}\}\\mathbb\{E\}\\\|\\Xi\\\|\_\{\*\}\>0\.The expectation is finite because‖Ξ‖∗≤d​‖Ξ‖F\\\|\\Xi\\\|\_\{\*\}\\leq\\sqrt\{d\}\\,\\\|\\Xi\\\|\_\{F\}, and it is strictly positive becauseΞ\\Xiis nonzero almost surely\. Ford=1d=1,c1=𝔼​\|ξ\|=2/πc\_\{1\}=\\mathbb\{E\}\|\\xi\|=\\sqrt\{2/\\pi\}, recovering the SignSGD coefficient\.

##### Taylor theorem with remainder\.

Sincemsgn⁡\(−A\)=−msgn⁡\(A\)\\operatorname\{msgn\}\(\-A\)=\-\\operatorname\{msgn\}\(A\)andΞ=d−Ξ\\Xi\\stackrel\{\{\\scriptstyle\\mathrm\{d\}\}\}\{\{=\}\}\-\\Xi, we haveF⁡\(−H\)=−F⁡\(H\)F\(\-H\)=\-F\(H\)\. Consequently,

F⁡\(0\)=0,D2​F​\(0\)=0\.F\(0\)=0,\\qquad D^\{2\}F\(0\)=0\.BecauseFFis smooth and the dimension is fixed,

Md:=sup‖Z‖F≤1‖D3​F​\(Z\)‖op<∞,M\_\{d\}:=\\sup\_\{\\\|Z\\\|\_\{F\}\\leq 1\}\\\|D^\{3\}F\(Z\)\\\|\_\{\\mathrm\{op\}\}<\\infty,where∥⋅∥op\\\|\\cdot\\\|\_\{\\mathrm\{op\}\}denotes the norm of a trilinear map induced by the Frobenius norm on its arguments and output\. Taylor’s theorem with integral remainder gives

F⁡\(H\)−D​F​\(0\)​\[H\]=12​∫01\(1−s\)2​D3​F​\(s​H\)​\[H,H,H\]​𝑑s\.F\(H\)\-DF\(0\)\[H\]=\\frac\{1\}\{2\}\\int\_\{0\}^\{1\}\(1\-s\)^\{2\}D^\{3\}F\(sH\)\[H,H,H\]\\,\\mathrm\{d\}s\.Hence, for every‖H‖F≤1\\\|H\\\|\_\{F\}\\leq 1,

‖F⁡\(H\)−cd​H‖F≤Md6​‖H‖F3\.\\\|F\(H\)\-c\_\{d\}H\\\|\_\{F\}\\leq\\frac\{M\_\{d\}\}\{6\}\\\|H\\\|\_\{F\}^\{3\}\.Finally, sinceσ\>0\\sigma\>0and the matrix sign is invariant under positive rescaling,

msgn⁡\(G¯\+σ​Ξ\)=msgn⁡\(Ξ\+G¯σ\)\.\\operatorname\{msgn\}\(\\bar\{G\}\+\\sigma\\Xi\)=\\operatorname\{msgn\}\\\!\\left\(\\Xi\+\\frac\{\\bar\{G\}\}\{\\sigma\}\\right\)\.TakingH=G¯/σH=\\bar\{G\}/\\sigma, for‖G¯‖F/σ≤1\\\|\\bar\{G\}\\\|\_\{F\}/\\sigma\\leq 1we obtain

‖𝔼⁡\[msgn⁡\(G¯\+σ​Ξ\)\]−cdσ​G¯‖F≤Md6​‖G¯‖F3σ3\.\\left\\\|\\mathbb\{E\}\[\\operatorname\{msgn\}\(\\bar\{G\}\+\\sigma\\Xi\)\]\-\\frac\{c\_\{d\}\}\{\\sigma\}\\bar\{G\}\\right\\\|\_\{F\}\\leq\\frac\{M\_\{d\}\}\{6\}\\frac\{\\\|\\bar\{G\}\\\|\_\{F\}^\{3\}\}\{\\sigma^\{3\}\}\.This proves the claimedOd​\(‖G¯‖F3/σ3\)O\_\{d\}\(\\\|\\bar\{G\}\\\|\_\{F\}^\{3\}/\\sigma^\{3\}\)remainder as‖G¯‖F/σ→0\\\|\\bar\{G\}\\\|\_\{F\}/\\sigma\\to 0\.

### Appendix BAssociative\-Memory Setup and Gradient Derivation

#### B\.1Gradient of the Associative Memory Model

Recall that

W¯=E~⊤​W​E,Si​j=exp⁡\(W¯i​j\)∑k=1Kexp⁡\(W¯k​j\)\.\\overline\{W\}=\\widetilde\{E\}^\{\\top\}WE,\\qquad S\_\{ij\}=\\frac\{\\exp\(\\overline\{W\}\_\{ij\}\)\}\{\\sum\_\{k=1\}^\{K\}\\exp\(\\overline\{W\}\_\{kj\}\)\}\.For taskjj, the cross\-entropy loss is

ℓj​\(W\)=−log⁡Sj​j=−E~j⊤​W​Ej\+log⁡\(∑k=1KeE~k⊤​W​Ej\)\.\\ell\_\{j\}\(W\)=\-\\log S\_\{jj\}=\-\\widetilde\{E\}\_\{j\}^\{\\top\}WE\_\{j\}\+\\log\\\!\\left\(\\sum\_\{k=1\}^\{K\}e^\{\\widetilde\{E\}\_\{k\}^\{\\top\}WE\_\{j\}\}\\right\)\.Differentiating gives

∇Wℓj​\(W\)\\displaystyle\\nabla\_\{W\}\\ell\_\{j\}\(W\)=−E~j​Ej⊤\+∑k=1KSk​j​E~k​Ej⊤\\displaystyle=\-\\widetilde\{E\}\_\{j\}E\_\{j\}^\{\\top\}\+\\sum\_\{k=1\}^\{K\}S\_\{kj\}\\widetilde\{E\}\_\{k\}E\_\{j\}^\{\\top\}=−\(E~j−∑k=1KSk​j​E~k\)​Ej⊤\.\\displaystyle=\-\\left\(\\widetilde\{E\}\_\{j\}\-\\sum\_\{k=1\}^\{K\}S\_\{kj\}\\widetilde\{E\}\_\{k\}\\right\)E\_\{j\}^\{\\top\}\.Hence, withP=diag⁡\(p1,…,pK\)P=\\operatorname\{diag\}\(p\_\{1\},\\ldots,p\_\{K\}\),

∇ℒ​\(W\)\\displaystyle\\nabla\\mathcal\{L\}\(W\)=−∑j=1K∑k=1Kpj\(δk​j−Sk​j\)E~kEj⊤\\displaystyle=\-\\sum\_\{j=1\}^\{K\}\\sum\_\{k=1\}^\{K\}p\_\{j\}\(\\delta\_\{kj\}\-S\_\{kj\}\)\\widetilde\{E\}\_\{k\}E\_\{j\}^\{\\top\}=−E~​\(I−S\)​P​E⊤\.\\displaystyle=\-\\widetilde\{E\}\(I\-S\)PE^\{\\top\}\.Similarly, for a mini\-batch with empirical frequenciesq⁡\(t\)=\(q1​\(t\),…,qK​\(t\)\)q\(t\)=\(q\_\{1\}\(t\),\\ldots,q\_\{K\}\(t\)\)andQt=diag⁡\(q⁡\(t\)\)Q\_\{t\}=\\operatorname\{diag\}\(q\(t\)\),

∇ℒ​\(Wt,ℬt\)=−E~​\(I−St\)​Qt​E⊤\.\\nabla\\mathcal\{L\}\(W\_\{t\};\\mathcal\{B\}\_\{t\}\)=\-\\widetilde\{E\}\(I\-S\_\{t\}\)Q\_\{t\}E^\{\\top\}\.Since the partial polar factor is equivariant under left and right orthogonal transformations,

msgn⁡\(E~​A​E⊤\)=E~​msgn⁡\(A\)​E⊤,\\operatorname\{msgn\}\\\!\\left\(\\widetilde\{E\}AE^\{\\top\}\\right\)=\\widetilde\{E\}\\operatorname\{msgn\}\(A\)E^\{\\top\},the task\-coordinate Muon update is

W¯t\+1=W¯t\+η​msgn⁡\(\(I−St\)​Qt\)\.\\overline\{W\}\_\{t\+1\}=\\overline\{W\}\_\{t\}\+\\eta\\operatorname\{msgn\}\\\!\\left\(\(I\-S\_\{t\}\)Q\_\{t\}\\right\)\.

### Appendix CTwo\-Task Finite\-Batch Dynamics and Refinements

The two\-task model is retained as an exactly solvable specialization of the finite\-batch sampling mechanism\. Because its stochastic gradient is rank one, it should not be interpreted as evidence for genuinely higher\-rank spectral effects\. Throughout this appendix, we assumeK=2K=2andp1≥p2=pmin\>0p\_\{1\}\\geq p\_\{2\}=p\_\{\\min\}\>0\. Each batch consists ofBBindependent samples from\(p1,p2\)\(p\_\{1\},p\_\{2\}\), independently across iterations\. Subsections C\.1–C\.4 establish the exact finite\-BBstatements used in Theorem[4\.1](https://arxiv.org/html/2610.31728#S4.Thmtheorem1); the final subsection records large\-batch and momentum\-averaging refinements rather than a second phase\-transition claim\. Thus

B​q1​\(t\)∼Binom⁡\(B,p1\),q2​\(t\)=1−q1​\(t\)\.Bq\_\{1\}\(t\)\\sim\\operatorname\{Binom\}\(B,p\_\{1\}\),\\qquad q\_\{2\}\(t\)=1\-q\_\{1\}\(t\)\.
#### C\.1Exact Reduction to the Margin Dynamics

Define

Δ1​\(t\)=\(W¯t\)11−\(W¯t\)21,Δ2​\(t\)=\(W¯t\)22−\(W¯t\)12,\\Delta\_\{1\}\(t\)=\(\\overline\{W\}\_\{t\}\)\_\{11\}\-\(\\overline\{W\}\_\{t\}\)\_\{21\},\\qquad\\Delta\_\{2\}\(t\)=\(\\overline\{W\}\_\{t\}\)\_\{22\}\-\(\\overline\{W\}\_\{t\}\)\_\{12\},and

ui​\(t\)=11\+eΔi​\(t\)\.u\_\{i\}\(t\)=\\frac\{1\}\{1\+e^\{\\Delta\_\{i\}\(t\)\}\}\.For two classes,

1−S11​\(t\)=S21​\(t\)=u1​\(t\),1−S22​\(t\)=S12​\(t\)=u2​\(t\)\.1\-S\_\{11\}\(t\)=S\_\{21\}\(t\)=u\_\{1\}\(t\),\\qquad 1\-S\_\{22\}\(t\)=S\_\{12\}\(t\)=u\_\{2\}\(t\)\.Therefore

\(I−St\)​Qt\\displaystyle\(I\-S\_\{t\}\)Q\_\{t\}=\[q1​\(t\)​\(1−S11​\(t\)\)−q2​\(t\)​S12​\(t\)−q1​\(t\)​S21​\(t\)q2​\(t\)​\(1−S22​\(t\)\)\]\\displaystyle=\\begin\{bmatrix\}q\_\{1\}\(t\)\(1\-S\_\{11\}\(t\)\)&\-q\_\{2\}\(t\)S\_\{12\}\(t\)\\\\ \-q\_\{1\}\(t\)S\_\{21\}\(t\)&q\_\{2\}\(t\)\(1\-S\_\{22\}\(t\)\)\\end\{bmatrix\}=\[1−1\]​\[q1​\(t\)​u1​\(t\)−q2​\(t\)​u2​\(t\)\]\.\\displaystyle=\\begin\{bmatrix\}1\\\\ \-1\\end\{bmatrix\}\\begin\{bmatrix\}q\_\{1\}\(t\)u\_\{1\}\(t\)&\-q\_\{2\}\(t\)u\_\{2\}\(t\)\\end\{bmatrix\}\.This matrix has rank one whenever at least oneqi​\(t\)q\_\{i\}\(t\)is nonzero, which always holds\. For a nonzero rank\-one matrixa​b⊤ab^\{\\top\},

msgn⁡\(a​b⊤\)=a‖a‖2​b⊤‖b‖2\.\\operatorname\{msgn\}\(ab^\{\\top\}\)=\\frac\{a\}\{\\\|a\\\|\_\{2\}\}\\frac\{b^\{\\top\}\}\{\\\|b\\\|\_\{2\}\}\.Since‖\(1,−1\)⊤‖2=2\\\|\(1,\-1\)^\{\\top\}\\\|\_\{2\}=\\sqrt\{2\}, we obtain

msgn⁡\(\(I−St\)​Qt\)=\[1/2−1/2\]​\[q1​\(t\)​u1​\(t\)−q2​\(t\)​u2​\(t\)\]q12​\(t\)​u12​\(t\)\+q22​\(t\)​u22​\(t\)\.\\operatorname\{msgn\}\\\!\\left\(\(I\-S\_\{t\}\)Q\_\{t\}\\right\)=\\begin\{bmatrix\}1/\\sqrt\{2\}\\\\ \-1/\\sqrt\{2\}\\end\{bmatrix\}\\frac\{\\begin\{bmatrix\}q\_\{1\}\(t\)u\_\{1\}\(t\)&\-q\_\{2\}\(t\)u\_\{2\}\(t\)\\end\{bmatrix\}\}\{\\sqrt\{q\_\{1\}^\{2\}\(t\)u\_\{1\}^\{2\}\(t\)\+q\_\{2\}^\{2\}\(t\)u\_\{2\}^\{2\}\(t\)\}\}\.Taking the corresponding column differences gives

Δi​\(t\+1\)−Δi​\(t\)=2​η​qi​\(t\)​ui​\(t\)q12​\(t\)​u12​\(t\)\+q22​\(t\)​u22​\(t\)\.\\Delta\_\{i\}\(t\+1\)\-\\Delta\_\{i\}\(t\)=\\sqrt\{2\}\\eta\\frac\{q\_\{i\}\(t\)u\_\{i\}\(t\)\}\{\\sqrt\{q\_\{1\}^\{2\}\(t\)u\_\{1\}^\{2\}\(t\)\+q\_\{2\}^\{2\}\(t\)u\_\{2\}^\{2\}\(t\)\}\}\.\(3\)

#### C\.2Full\-batch Dynamics

Setqi​\(t\)=piq\_\{i\}\(t\)=p\_\{i\}and take the continuous\-time limitτ=η​t\\tau=\\eta t\. Then

Δ˙i=2​pi​uip12​u12\+p22​u22\.\\dot\{\\Delta\}\_\{i\}=\\sqrt\{2\}\\,\\frac\{p\_\{i\}u\_\{i\}\}\{\\sqrt\{p\_\{1\}^\{2\}u\_\{1\}^\{2\}\+p\_\{2\}^\{2\}u\_\{2\}^\{2\}\}\}\.Dividing the two equations gives

d​Δ1d​Δ2=p1​u1p2​u2=p1​\(1\+eΔ2\)p2​\(1\+eΔ1\)\.\\frac\{d\\Delta\_\{1\}\}\{d\\Delta\_\{2\}\}=\\frac\{p\_\{1\}u\_\{1\}\}\{p\_\{2\}u\_\{2\}\}=\\frac\{p\_\{1\}\(1\+e^\{\\Delta\_\{2\}\}\)\}\{p\_\{2\}\(1\+e^\{\\Delta\_\{1\}\}\)\}\.Hence

p2​\(1\+eΔ1\)​d​Δ1=p1​\(1\+eΔ2\)​d​Δ2\.p\_\{2\}\(1\+e^\{\\Delta\_\{1\}\}\)\\,d\\Delta\_\{1\}=p\_\{1\}\(1\+e^\{\\Delta\_\{2\}\}\)\\,d\\Delta\_\{2\}\.UsingΔ1​\(0\)=Δ2​\(0\)=0\\Delta\_\{1\}\(0\)=\\Delta\_\{2\}\(0\)=0,

p2​\(Δ1\+eΔ1−1\)=p1​\(Δ2\+eΔ2−1\)\.p\_\{2\}\(\\Delta\_\{1\}\+e^\{\\Delta\_\{1\}\}\-1\)=p\_\{1\}\(\\Delta\_\{2\}\+e^\{\\Delta\_\{2\}\}\-1\)\.\(4\)Moreover,

2≤dd​τ​\(Δ1\+Δ2\)=2​p1​u1\+p2​u2p12​u12\+p22​u22≤2\.\\sqrt\{2\}\\leq\\frac\{d\}\{d\\tau\}\(\\Delta\_\{1\}\+\\Delta\_\{2\}\)=\\sqrt\{2\}\\,\\frac\{p\_\{1\}u\_\{1\}\+p\_\{2\}u\_\{2\}\}\{\\sqrt\{p\_\{1\}^\{2\}u\_\{1\}^\{2\}\+p\_\{2\}^\{2\}u\_\{2\}^\{2\}\}\}\\leq 2\.ThusΔ1\+Δ2→∞\\Delta\_\{1\}\+\\Delta\_\{2\}\\to\\infty\. Therefore, equation[4](https://arxiv.org/html/2610.31728#A3.E4)implies that both margins diverge, and the exponential terms dominate:

p2​eΔ1∼p1​eΔ2\.p\_\{2\}e^\{\\Delta\_\{1\}\}\\sim p\_\{1\}e^\{\\Delta\_\{2\}\}\.Therefore

Δ1−Δ2⟶log⁡p1p2,p1​u1∼p2​u2\.\\Delta\_\{1\}\-\\Delta\_\{2\}\\longrightarrow\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\},\\qquad p\_\{1\}u\_\{1\}\\sim p\_\{2\}u\_\{2\}\.Substituting back into the ODE yields

Δ˙i⟶1,Δi​\(τ\)τ⟶1\.\\dot\{\\Delta\}\_\{i\}\\longrightarrow 1,\\qquad\\frac\{\\Delta\_\{i\}\(\\tau\)\}\{\\tau\}\\longrightarrow 1\.

#### C\.3Small Mini\-batches: Occurrence\-limited Imbalance

Let

At=\{q1\(t\)=1\},ℙ\(At\)=p1B\.A\_\{t\}=\\\{q\_\{1\}\(t\)=1\\\},\\qquad\\mathbb\{P\}\(A\_\{t\}\)=p\_\{1\}^\{B\}\.OnAtA\_\{t\},

Δ1​\(t\+1\)−Δ1​\(t\)=2​η,Δ2​\(t\+1\)−Δ2​\(t\)=0\.\\Delta\_\{1\}\(t\+1\)\-\\Delta\_\{1\}\(t\)=\\sqrt\{2\}\\eta,\\qquad\\Delta\_\{2\}\(t\+1\)\-\\Delta\_\{2\}\(t\)=0\.OnAtcA\_\{t\}^\{c\}, both increments are nonnegative and their difference is at least−2​η\-\\sqrt\{2\}\\eta\. Hence

D⁡\(t\+1\)−D⁡\(t\)≥2​η​\(2​𝟏At−1\),D⁡\(t\):=Δ1​\(t\)−Δ2​\(t\)\.D\(t\+1\)\-D\(t\)\\geq\\sqrt\{2\}\\eta\\bigl\(2\\mathbf\{1\}\_\{A\_\{t\}\}\-1\\bigr\),\\qquad D\(t\):=\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\)\.By the strong law of large numbers,

lim inft→∞D⁡\(t\)η​t≥2​\(2​p1B−1\)a\.s\.\\liminf\_\{t\\to\\infty\}\\frac\{D\(t\)\}\{\\eta t\}\\geq\\sqrt\{2\}\(2p\_\{1\}^\{B\}\-1\)\\qquad\\text\{a\.s\.\}ThusD⁡\(t\)→\+∞D\(t\)\\to\+\\inftywhenp1B\>1/2p\_\{1\}^\{B\}\>1/2\.

We can then identify the exact asymptotic rates\. Since

u1​\(t\)u2​\(t\)=1\+eΔ2​\(t\)1\+eΔ1​\(t\)⟶0,\\frac\{u\_\{1\}\(t\)\}\{u\_\{2\}\(t\)\}=\\frac\{1\+e^\{\\Delta\_\{2\}\(t\)\}\}\{1\+e^\{\\Delta\_\{1\}\(t\)\}\}\\longrightarrow 0,every batch containing task\-2 asymptotically allocates the normalized update to task\-2, while a pure task\-1 batch allocates it to task\-1\. Consequently,

Δ1​\(t\)η​t→a\.s\.2​p1B,Δ2​\(t\)η​t→a\.s\.2​\(1−p1B\)\.\\frac\{\\Delta\_\{1\}\(t\)\}\{\\eta t\}\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\sqrt\{2\}p\_\{1\}^\{B\},\\qquad\\frac\{\\Delta\_\{2\}\(t\)\}\{\\eta t\}\\xrightarrow\{\\mathrm\{a\.s\.\}\}\\sqrt\{2\}\(1\-p\_\{1\}^\{B\}\)\.The transitionp1B=1/2p\_\{1\}^\{B\}=1/2is equivalent to

Bbal=log⁡2−log⁡p1=log⁡2−log⁡\(1−pmin\)\.B\_\{\\mathrm\{bal\}\}=\\frac\{\\log 2\}\{\-\\log p\_\{1\}\}=\\frac\{\\log 2\}\{\-\\log\(1\-p\_\{\\min\}\)\}\.

#### C\.4Large Mini\-batches: Recovery of Balanced Growth

Assume

p1B<12\.p\_\{1\}^\{B\}<\\frac\{1\}\{2\}\.We first establish an exact stability result for the discrete\-time gap

D⁡\(t\):=Δ1​\(t\)−Δ2​\(t\)\.D\(t\):=\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\)\.
##### Restoring drift in the tails\.

From equation[3](https://arxiv.org/html/2610.31728#A3.E3),

D⁡\(t\+1\)−D⁡\(t\)=2​η​q1​\(t\)​u1​\(t\)−q2​\(t\)​u2​\(t\)q12​\(t\)​u12​\(t\)\+q22​\(t\)​u22​\(t\)\.D\(t\+1\)\-D\(t\)=\\sqrt\{2\}\\eta\\frac\{q\_\{1\}\(t\)u\_\{1\}\(t\)\-q\_\{2\}\(t\)u\_\{2\}\(t\)\}\{\\sqrt\{q\_\{1\}^\{2\}\(t\)u\_\{1\}^\{2\}\(t\)\+q\_\{2\}^\{2\}\(t\)u\_\{2\}^\{2\}\(t\)\}\}\.\(5\)In particular,

\|D⁡\(t\+1\)−D⁡\(t\)\|≤2​η\.\|D\(t\+1\)\-D\(t\)\|\\leq\\sqrt\{2\}\\eta\.
Since both margins start from zero and have nonnegative increments,

Δi​\(t\)≥0,∀t\.\\Delta\_\{i\}\(t\)\\geq 0,\\qquad\\forall t\.IfD⁡\(t\)≥0D\(t\)\\geq 0, then

u1​\(t\)u2​\(t\)\\displaystyle\\frac\{u\_\{1\}\(t\)\}\{u\_\{2\}\(t\)\}=1\+eΔ2​\(t\)1\+eΔ1​\(t\)\\displaystyle=\\frac\{1\+e^\{\\Delta\_\{2\}\(t\)\}\}\{1\+e^\{\\Delta\_\{1\}\(t\)\}\}=1\+eΔ2​\(t\)1\+eΔ2​\(t\)\+D​\(t\)\\displaystyle=\\frac\{1\+e^\{\\Delta\_\{2\}\(t\)\}\}\{1\+e^\{\\Delta\_\{2\}\(t\)\+D\(t\)\}\}≤2​e−D⁡\(t\)\.\\displaystyle\\leq 2e^\{\-D\(t\)\}\.Thusu1/u2→0u\_\{1\}/u\_\{2\}\\to 0uniformly over the common offset of the two margins asD→\+∞D\\to\+\\infty\.

Ifq2​\(t\)=0q\_\{2\}\(t\)=0, thenq1​\(t\)=1q\_\{1\}\(t\)=1andD⁡\(t\+1\)−D⁡\(t\)=2​ηD\(t\+1\)\-D\(t\)=\\sqrt\{2\}\\eta\.

Ifq2​\(t\)\>0q\_\{2\}\(t\)\>0, thenq2​\(t\)≥1/Bq\_\{2\}\(t\)\\geq 1/B\. Since the batch proportions take values in a finite set, equation[5](https://arxiv.org/html/2610.31728#A3.E5)and the bound above imply, uniformly over all batches withq2​\(t\)\>0q\_\{2\}\(t\)\>0,

D⁡\(t\+1\)−D⁡\(t\)⟶−2​ηas​D​\(t\)→\+∞\.D\(t\+1\)\-D\(t\)\\longrightarrow\-\\sqrt\{2\}\\eta\\qquad\\text\{as \}D\(t\)\\to\+\\infty\.Consequently,

𝔼⁡\[D⁡\(t\+1\)−D⁡\(t\)∣ℱt\]⟶2​η​\(2​p1B−1\)<0as​D​\(t\)→\+∞\.\\mathbb\{E\}\\\!\\left\[D\(t\+1\)\-D\(t\)\\mid\\mathcal\{F\}\_\{t\}\\right\]\\longrightarrow\\sqrt\{2\}\\eta\\left\(2p\_\{1\}^\{B\}\-1\\right\)<0\\qquad\\text\{as \}D\(t\)\\to\+\\infty\.\(6\)
Similarly, ifD⁡\(t\)≤0D\(t\)\\leq 0, then

u2​\(t\)u1​\(t\)≤2​eD⁡\(t\)=2​e−\|D⁡\(t\)\|\.\\frac\{u\_\{2\}\(t\)\}\{u\_\{1\}\(t\)\}\\leq 2e^\{D\(t\)\}=2e^\{\-\|D\(t\)\|\}\.Hence, asD⁡\(t\)→−∞D\(t\)\\to\-\\infty, every batch containing task\-1 satisfies

D⁡\(t\+1\)−D⁡\(t\)⟶2​η,D\(t\+1\)\-D\(t\)\\longrightarrow\\sqrt\{2\}\\eta,whereas a pure task\-2 batch satisfies

D⁡\(t\+1\)−D⁡\(t\)=−2​η\.D\(t\+1\)\-D\(t\)=\-\\sqrt\{2\}\\eta\.Therefore,

𝔼⁡\[D⁡\(t\+1\)−D⁡\(t\)∣ℱt\]⟶2​η​\(1−2​p2B\)\>0as​D​\(t\)→−∞,\\mathbb\{E\}\\\!\\left\[D\(t\+1\)\-D\(t\)\\mid\\mathcal\{F\}\_\{t\}\\right\]\\longrightarrow\\sqrt\{2\}\\eta\\left\(1\-2p\_\{2\}^\{B\}\\right\)\>0\\qquad\\text\{as \}D\(t\)\\to\-\\infty,\(7\)where we usedp2B≤p1B<1/2p\_\{2\}^\{B\}\\leq p\_\{1\}^\{B\}<1/2\.

It follows that there exist constantsR,δ\>0R,\\delta\>0such that

𝔼⁡\[D⁡\(t\+1\)−D⁡\(t\)∣ℱt\]≤−δwhen​D​\(t\)≥R,\\mathbb\{E\}\\\!\\left\[D\(t\+1\)\-D\(t\)\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leq\-\\delta\\qquad\\text\{when \}D\(t\)\\geq R,and

𝔼⁡\[D⁡\(t\+1\)−D⁡\(t\)∣ℱt\]≥δwhen​D​\(t\)≤−R\.\\mathbb\{E\}\\\!\\left\[D\(t\+1\)\-D\(t\)\\mid\\mathcal\{F\}\_\{t\}\\right\]\\geq\\delta\\qquad\\text\{when \}D\(t\)\\leq\-R\.

##### Exponential moment bound\.

Let

Xt:=D⁡\(t\+1\)−D⁡\(t\),L:=2​η,X\_\{t\}:=D\(t\+1\)\-D\(t\),\\qquad L:=\\sqrt\{2\}\\eta,so that\|Xt\|≤L\|X\_\{t\}\|\\leq L\. IncreasingRRif necessary, assumeR\>LR\>L\.

WhenD⁡\(t\)≥RD\(t\)\\geq R, the sign ofD⁡\(t\)D\(t\)cannot change in one step, and hence

\|D⁡\(t\+1\)\|=D⁡\(t\)\+Xt\.\|D\(t\+1\)\|=D\(t\)\+X\_\{t\}\.For anyθ\>0\\theta\>0,

eθ​Xt≤1\+θ​Xt\+θ2​L22​eθ​L\.e^\{\\theta X\_\{t\}\}\\leq 1\+\\theta X\_\{t\}\+\\frac\{\\theta^\{2\}L^\{2\}\}\{2\}e^\{\\theta L\}\.Therefore, for sufficiently smallθ\>0\\theta\>0,

𝔼⁡\[eθ​Xt∣ℱt\]≤1−θ​δ2when​D​\(t\)≥R\.\\mathbb\{E\}\\\!\\left\[e^\{\\theta X\_\{t\}\}\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leq 1\-\\frac\{\\theta\\delta\}\{2\}\\qquad\\text\{when \}D\(t\)\\geq R\.The same argument applied to−Xt\-X\_\{t\}gives

𝔼⁡\[e−θ​Xt∣ℱt\]≤1−θ​δ2when​D​\(t\)≤−R\.\\mathbb\{E\}\\\!\\left\[e^\{\-\\theta X\_\{t\}\}\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leq 1\-\\frac\{\\theta\\delta\}\{2\}\\qquad\\text\{when \}D\(t\)\\leq\-R\.
Define

V⁡\(D\):=eθ​\|D\|\.V\(D\):=e^\{\\theta\|D\|\}\.There then exist constantsρ<1\\rho<1andb0<∞b\_\{0\}<\\inftysuch that

𝔼\[V\(D\(t\+1\)\)∣ℱt\]≤ρV\(D\(t\)\)\+b0𝟏\{\|D\(t\)\|≤R\}\.\\mathbb\{E\}\\\!\\left\[V\(D\(t\+1\)\)\\mid\\mathcal\{F\}\_\{t\}\\right\]\\leq\\rho V\(D\(t\)\)\+b\_\{0\}\\mathbf\{1\}\_\{\\\{\|D\(t\)\|\\leq R\\\}\}\.Iterating this inequality yields

supt≥0𝔼​eθ​\|D⁡\(t\)\|<∞\.\\sup\_\{t\\geq 0\}\\mathbb\{E\}e^\{\\theta\|D\(t\)\|\}<\\infty\.\(8\)
Hence, for everyε\>0\\varepsilon\>0,

ℙ⁡\(\|D⁡\(t\)\|\>ε​t\)≤bθ​e−θ​ε​t,\\mathbb\{P\}\\\!\\left\(\|D\(t\)\|\>\\varepsilon t\\right\)\\leq b\_\{\\theta\}e^\{\-\\theta\\varepsilon t\},and therefore

∑t=1∞ℙ⁡\(\|D⁡\(t\)\|\>ε​t\)<∞\.\\sum\_\{t=1\}^\{\\infty\}\\mathbb\{P\}\\\!\\left\(\|D\(t\)\|\>\\varepsilon t\\right\)<\\infty\.By the Borel–Cantelli lemma,

D⁡\(t\)t⟶0a\.s\.\\frac\{D\(t\)\}\{t\}\\longrightarrow 0\\qquad\\text\{a\.s\.\}\(9\)This proves vanishing asymptotic gap growth wheneverp1B<1/2p\_\{1\}^\{B\}<1/2\.

##### Late\-time effective drift\.

We next refine the description of the gap in the well\-sampled regime\. The remainder of this subsection concerns a late\-time, small\-step approximation rather than an exact invariant law of the discrete process\.

First note that both margins diverge almost surely\. A pure task\-iibatch occurs independently with probabilitypiB\>0p\_\{i\}^\{B\}\>0, and on every such batch,

Δi​\(t\+1\)−Δi​\(t\)=2​η\.\\Delta\_\{i\}\(t\+1\)\-\\Delta\_\{i\}\(t\)=\\sqrt\{2\}\\eta\.The strong law therefore implies

lim inft→∞Δi​\(t\)η​t≥2​piB\>0a\.s\.\\liminf\_\{t\\to\\infty\}\\frac\{\\Delta\_\{i\}\(t\)\}\{\\eta t\}\\geq\\sqrt\{2\}p\_\{i\}^\{B\}\>0\\qquad\\text\{a\.s\.\}Consequently,

ui​\(t\)=e−Δi​\(t\)​\(1\+o⁡\(1\)\)\.u\_\{i\}\(t\)=e^\{\-\\Delta\_\{i\}\(t\)\}\\left\(1\+o\(1\)\\right\)\.
Define the centered gap

γ⁡\(t\):=D⁡\(t\)−log⁡p1p2\.\\gamma\(t\):=D\(t\)\-\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\}\.Then

p1​u1​\(t\)p2​u2​\(t\)=e−γ⁡\(t\)​\(1\+o⁡\(1\)\)\.\\frac\{p\_\{1\}u\_\{1\}\(t\)\}\{p\_\{2\}u\_\{2\}\(t\)\}=e^\{\-\\gamma\(t\)\}\\left\(1\+o\(1\)\\right\)\.
For

a=q1p1,b=q2p2,a=\\frac\{q\_\{1\}\}\{p\_\{1\}\},\\qquad b=\\frac\{q\_\{2\}\}\{p\_\{2\}\},define

φ⁡\(γ,a,b\):=2​a​e−γ−ba2​e−2​γ\+b2\.\\varphi\(\\gamma;a,b\):=\\sqrt\{2\}\\frac\{ae^\{\-\\gamma\}\-b\}\{\\sqrt\{a^\{2\}e^\{\-2\\gamma\}\+b^\{2\}\}\}\.\(10\)Then, uniformly forγ\\gammain compact sets,

γ⁡\(t\+1\)−γ⁡\(t\)η=φ⁡\(γ⁡\(t\),q1​\(t\)p1,q2​\(t\)p2\)\+o⁡\(1\)as​t→∞\.\\frac\{\\gamma\(t\+1\)\-\\gamma\(t\)\}\{\\eta\}=\\varphi\\\!\\left\(\\gamma\(t\);\\frac\{q\_\{1\}\(t\)\}\{p\_\{1\}\},\\frac\{q\_\{2\}\(t\)\}\{p\_\{2\}\}\\right\)\+o\(1\)\\qquad\\text\{as \}t\\to\\infty\.\(11\)
Define the corresponding late\-time mean drift

ψB​\(γ\):=𝔼⁡\[φ⁡\(γ,q1p1,q2p2\)\]\.\\psi\_\{B\}\(\\gamma\):=\\mathbb\{E\}\\\!\\left\[\\varphi\\\!\\left\(\\gamma;\\frac\{q\_\{1\}\}\{p\_\{1\}\},\\frac\{q\_\{2\}\}\{p\_\{2\}\}\\right\)\\right\]\.\(12\)
Fora,b\>0a,b\>0,

∂γφ⁡\(γ,a,b\)=−2​a​b​e−γ​\(a​e−γ\+b\)\(a2​e−2​γ\+b2\)3/2<0\.\\partial\_\{\\gamma\}\\varphi\(\\gamma;a,b\)=\-\\sqrt\{2\}\\frac\{abe^\{\-\\gamma\}\\left\(ae^\{\-\\gamma\}\+b\\right\)\}\{\\left\(a^\{2\}e^\{\-2\\gamma\}\+b^\{2\}\\right\)^\{3/2\}\}<0\.Ifb=0b=0, thenφ⁡\(γ,a,0\)=2\\varphi\(\\gamma;a,0\)=\\sqrt\{2\}, whereas ifa=0a=0,φ⁡\(γ,0,b\)=−2\\varphi\(\\gamma;0,b\)=\-\\sqrt\{2\}\. Thusφ\\varphiis nonincreasing inγ\\gammafor every batch and strictly decreasing for every mixed batch\. Since mixed batches occur with positive probability,ψB\\psi\_\{B\}is strictly decreasing\.

Moreover,

limγ→\+∞ψB​\(γ\)=2​\(2​p1B−1\)<0,\\lim\_\{\\gamma\\to\+\\infty\}\\psi\_\{B\}\(\\gamma\)=\\sqrt\{2\}\\left\(2p\_\{1\}^\{B\}\-1\\right\)<0,and

limγ→−∞ψB​\(γ\)=2​\(1−2​p2B\)\>0\.\\lim\_\{\\gamma\\to\-\\infty\}\\psi\_\{B\}\(\\gamma\)=\\sqrt\{2\}\\left\(1\-2p\_\{2\}^\{B\}\\right\)\>0\.HenceψB\\psi\_\{B\}has a unique zero, denoted by

Suppose now that

B​pmin→∞\.Bp\_\{\\min\}\\to\\infty\.Since

Var⁡\(qipi\)=1−piB​pi,\\operatorname\{Var\}\\\!\\left\(\\frac\{q\_\{i\}\}\{p\_\{i\}\}\\right\)=\\frac\{1\-p\_\{i\}\}\{Bp\_\{i\}\},we have

qipi→ℙ1\.\\frac\{q\_\{i\}\}\{p\_\{i\}\}\\xrightarrow\{\\mathbb\{P\}\}1\.Since\|φ\|≤2\|\\varphi\|\\leq\\sqrt\{2\}, convergence in probability together with uniform boundedness implies

ψB​\(γ\)⟶ψ∞​\(γ\):=2​e−γ−1e−2​γ\+1\\psi\_\{B\}\(\\gamma\)\\longrightarrow\\psi\_\{\\infty\}\(\\gamma\):=\\sqrt\{2\}\\frac\{e^\{\-\\gamma\}\-1\}\{\\sqrt\{e^\{\-2\\gamma\}\+1\}\}for every fixedγ\\gamma\.

The limiting drift is strictly decreasing and has the unique zeroγ=0\\gamma=0\. Fix anyδ\>0\\delta\>0\. Since

ψ∞​\(−δ\)\>0,ψ∞​\(δ\)<0,\\psi\_\{\\infty\}\(\-\\delta\)\>0,\\qquad\\psi\_\{\\infty\}\(\\delta\)<0,the same inequalities hold forψB\\psi\_\{B\}whenB​pminBp\_\{\\min\}is sufficiently large\. By monotonicity,

\|γB∗\|<δ\.\|\\gamma\_\{B\}^\{\*\}\|<\\delta\.Therefore,

γB∗⟶0as​B​pmin→∞\.\\gamma\_\{B\}^\{\*\}\\longrightarrow 0\\qquad\\text\{as \}Bp\_\{\\min\}\\to\\infty\.\(13\)
For fixedp1,p2\>0p\_\{1\},p\_\{2\}\>0, this convergence can be sharpened\. Define

h⁡\(q\):=φ⁡\(0,qp1,1−qp2\)\.h\(q\):=\\varphi\\\!\\left\(0;\\frac\{q\}\{p\_\{1\}\},\\frac\{1\-q\}\{p\_\{2\}\}\\right\)\.A direct calculation gives

h⁡\(p1\)=0,h′​\(p1\)=1p1​p2,h′′​\(p1\)=p1−p2p12​p22\.h\(p\_\{1\}\)=0,\\qquad h^\{\\prime\}\(p\_\{1\}\)=\\frac\{1\}\{p\_\{1\}p\_\{2\}\},\\qquad h^\{\\prime\\prime\}\(p\_\{1\}\)=\\frac\{p\_\{1\}\-p\_\{2\}\}\{p\_\{1\}^\{2\}p\_\{2\}^\{2\}\}\.Since

𝔼⁡\[q1−p1\]=0,Var⁡\(q1\)=p1​p2B,\\mathbb\{E\}\[q\_\{1\}\-p\_\{1\}\]=0,\\qquad\\operatorname\{Var\}\(q\_\{1\}\)=\\frac\{p\_\{1\}p\_\{2\}\}\{B\},a second\-order expansion gives

ψB​\(0\)=p1−p22​B​p1​p2\+o⁡\(1B\)\.\\psi\_\{B\}\(0\)=\\frac\{p\_\{1\}\-p\_\{2\}\}\{2Bp\_\{1\}p\_\{2\}\}\+o\\\!\\left\(\\frac\{1\}\{B\}\\right\)\.Furthermore,

ψB′​\(0\)=−1\+o⁡\(1\)\.\\psi\_\{B\}^\{\\prime\}\(0\)=\-1\+o\(1\)\.Expanding the zero ofψB\\psi\_\{B\}around zero therefore yields

γB∗=p1−p22​B​p1​p2\+o⁡\(1B\)\.\\gamma\_\{B\}^\{\*\}=\\frac\{p\_\{1\}\-p\_\{2\}\}\{2Bp\_\{1\}p\_\{2\}\}\+o\\\!\\left\(\\frac\{1\}\{B\}\\right\)\.\(14\)
Thus mini\-batch fluctuations induce a small positive shift of the effective gap whenp1\>p2p\_\{1\}\>p\_\{2\}, and this shift vanishes in the large\-batch limit\.

##### Local diffusion approximation\.

Let

λB:=−ψB′​\(γB∗\)\>0\\lambda\_\{B\}:=\-\\psi\_\{B\}^\{\\prime\}\(\\gamma\_\{B\}^\{\*\}\)\>0and

vB:=Var⁡\[φ⁡\(γB∗,q1p1,q2p2\)\]\.v\_\{B\}:=\\operatorname\{Var\}\\\!\\left\[\\varphi\\\!\\left\(\\gamma\_\{B\}^\{\*\};\\frac\{q\_\{1\}\}\{p\_\{1\}\},\\frac\{q\_\{2\}\}\{p\_\{2\}\}\\right\)\\right\]\.Writing

x⁡\(t\):=γ⁡\(t\)−γB∗,x\(t\):=\\gamma\(t\)\-\\gamma\_\{B\}^\{\*\},the local recursion suggested by equation[11](https://arxiv.org/html/2610.31728#A3.E11)is

x⁡\(t\+1\)−x⁡\(t\)≃−η​λB​x​\(t\)\+η​ξt,x\(t\+1\)\-x\(t\)\\simeq\-\\eta\\lambda\_\{B\}x\(t\)\+\\eta\\xi\_\{t\},where

𝔼⁡\[ξt\]≃0,Var⁡\(ξt\)≃vB\.\\mathbb\{E\}\[\\xi\_\{t\}\]\\simeq 0,\\qquad\\operatorname\{Var\}\(\\xi\_\{t\}\)\\simeq v\_\{B\}\.
In the well\-sampled regime,

λB=1\+o⁡\(1\)\.\\lambda\_\{B\}=1\+o\(1\)\.Moreover, using

h′​\(p1\)=1p1​p2h^\{\\prime\}\(p\_\{1\}\)=\\frac\{1\}\{p\_\{1\}p\_\{2\}\}and

Var⁡\(q1\)=p1​p2B,\\operatorname\{Var\}\(q\_\{1\}\)=\\frac\{p\_\{1\}p\_\{2\}\}\{B\},the delta method gives

vB=1\+o⁡\(1\)B​p1​p2\.v\_\{B\}=\\frac\{1\+o\(1\)\}\{Bp\_\{1\}p\_\{2\}\}\.\(15\)
On the rescaled time

the corresponding local diffusion approximation is

d​x≃−λB​x​d​τ\+η​vB​d​𝒲τ\.dx\\simeq\-\\lambda\_\{B\}x\\,d\\tau\+\\sqrt\{\\eta v\_\{B\}\}\\,d\\mathcal\{W\}\_\{\\tau\}\.\(16\)The associated Ornstein–Uhlenbeck stationary variance is

η​vB2​λB=η2​B​p1​p2​\(1\+o​\(1\)\)\.\\frac\{\\eta v\_\{B\}\}\{2\\lambda\_\{B\}\}=\\frac\{\\eta\}\{2Bp\_\{1\}p\_\{2\}\}\\left\(1\+o\(1\)\\right\)\.Thus, within the late\-time local diffusion approximation,

D=log⁡p1p2\+γB∗\+Oℙ​\(ηB​p1​p2\)\.D=\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\}\+\\gamma\_\{B\}^\{\*\}\+O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{\\eta\}\{Bp\_\{1\}p\_\{2\}\}\}\\right\)\.\(17\)Together with equation[13](https://arxiv.org/html/2610.31728#A3.E13), this recovers the full\-batch gap in the well\-sampled, small\-step limit\.

#### C\.5Momentum: Averaging Across Steps and Tracking

We now consider

gt=\(q1​\(t\)​u1​\(t\),q2​\(t\)​u2​\(t\)\)⊤,mt=β​mt−1\+\(1−β\)​gt,g\_\{t\}=\\left\(q\_\{1\}\(t\)u\_\{1\}\(t\),q\_\{2\}\(t\)u\_\{2\}\(t\)\\right\)^\{\\top\},\\qquad m\_\{t\}=\\beta m\_\{t\-1\}\+\(1\-\\beta\)g\_\{t\},with

0<β<1,ϵ:=1−β\.0<\\beta<1,\\qquad\\epsilon:=1\-\\beta\.The normalized margin update is

Δ⁡\(t\+1\)−Δ⁡\(t\)=2​η​mt‖mt‖2\.\\Delta\(t\+1\)\-\\Delta\(t\)=\\sqrt\{2\}\\eta\\frac\{m\_\{t\}\}\{\\\|m\_\{t\}\\\|\_\{2\}\}\.
We first freeze the margins and characterize how momentum averages task frequencies across iterations\. We then restore the slow evolution of the margins\.

##### Frozen margins and the EMA task frequency\.

Fixu1,u2\>0u\_\{1\},u\_\{2\}\>0\. After initialization transients, the stationary momentum admits the representation

mt=\(1−β\)​∑k=0∞βk​gt−k\.m\_\{t\}=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}g\_\{t\-k\}\.Define the exponentially averaged task\-1 frequency

Zt:=\(1−β\)​∑k=0∞βk​q1​\(t−k\)\.Z\_\{t\}:=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}q\_\{1\}\(t\-k\)\.\(18\)Sinceq2​\(t\)=1−q1​\(t\)q\_\{2\}\(t\)=1\-q\_\{1\}\(t\)and

\(1−β\)​∑k=0∞βk=1,\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}=1,we obtain the exact frozen\-margin representation

mt=\(u1​Ztu2​\(1−Zt\)\)\.m\_\{t\}=\\begin\{pmatrix\}u\_\{1\}Z\_\{t\}\\\\ u\_\{2\}\(1\-Z\_\{t\}\)\\end\{pmatrix\}\.\(19\)
The first two moments ofZtZ\_\{t\}are

𝔼⁡\[Zt\]\\displaystyle\\mathbb\{E\}\[Z\_\{t\}\]=p1,\\displaystyle=p\_\{1\},\(20\)Var⁡\(Zt\)\\displaystyle\\operatorname\{Var\}\(Z\_\{t\}\)=\(1−β\)2​∑k=0∞β2​k​p1​p2B\\displaystyle=\(1\-\\beta\)^\{2\}\\sum\_\{k=0\}^\{\\infty\}\\beta^\{2k\}\\frac\{p\_\{1\}p\_\{2\}\}\{B\}=1−β1\+β​p1​p2B\.\\displaystyle=\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{p\_\{1\}p\_\{2\}\}\{B\}\.\(21\)
It is therefore natural to define the variance\-equivalent batch size

Beff:=B​1\+β1−β,B\_\{\\mathrm\{eff\}\}:=B\\frac\{1\+\\beta\}\{1\-\\beta\},\(22\)for which

Var⁡\(Zt\)=p1​p2Beff\.\\operatorname\{Var\}\(Z\_\{t\}\)=\\frac\{p\_\{1\}p\_\{2\}\}\{B\_\{\\mathrm\{eff\}\}\}\.
Since both tasks have positive probability andβ\>0\\beta\>0,

0<Zt<1a\.s\.0<Z\_\{t\}<1\\qquad\\text\{a\.s\.\}under the stationary representation\. Thus EMA averaging assigns positive weight to both tasks almost surely even when individual tasks rarely appear in a mini\-batch\.

##### Momentum\-induced effective gap\.

Define again

γ=D−log⁡p1p2\.\\gamma=D\-\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\}\.At late time,

p1​u1p2​u2=e−γ​\(1\+o⁡\(1\)\)\.\\frac\{p\_\{1\}u\_\{1\}\}\{p\_\{2\}u\_\{2\}\}=e^\{\-\\gamma\}\\left\(1\+o\(1\)\\right\)\.Using equation[19](https://arxiv.org/html/2610.31728#A3.E19), the normalized frozen\-buffer gap increment is therefore asymptotically

η​φ​\(γ,Ztp1,1−Ztp2\),\\eta\\varphi\\\!\\left\(\\gamma;\\frac\{Z\_\{t\}\}\{p\_\{1\}\},\\frac\{1\-Z\_\{t\}\}\{p\_\{2\}\}\\right\),whereφ\\varphiis defined in equation[10](https://arxiv.org/html/2610.31728#A3.E10)\.

Define the frozen\-momentum mean drift

ψB,β​\(γ\):=𝔼⁡\[φ⁡\(γ,Zp1,1−Zp2\)\],\\psi\_\{B,\\beta\}\(\\gamma\):=\\mathbb\{E\}\\\!\\left\[\\varphi\\\!\\left\(\\gamma;\\frac\{Z\}\{p\_\{1\}\},\\frac\{1\-Z\}\{p\_\{2\}\}\\right\)\\right\],\(23\)whereZZhas the stationary marginal distribution defined by equation[18](https://arxiv.org/html/2610.31728#A3.E18)\.

Since0<Z<10<Z<1almost surely,

∂γφ⁡\(γ,Zp1,1−Zp2\)<0a\.s\.\\partial\_\{\\gamma\}\\varphi\\\!\\left\(\\gamma;\\frac\{Z\}\{p\_\{1\}\},\\frac\{1\-Z\}\{p\_\{2\}\}\\right\)<0\\qquad\\text\{a\.s\.\}HenceψB,β\\psi\_\{B,\\beta\}is strictly decreasing\. Moreover,

limγ→−∞ψB,β​\(γ\)=2,\\lim\_\{\\gamma\\to\-\\infty\}\\psi\_\{B,\\beta\}\(\\gamma\)=\\sqrt\{2\},whereas

limγ→\+∞ψB,β​\(γ\)=−2\.\\lim\_\{\\gamma\\to\+\\infty\}\\psi\_\{B,\\beta\}\(\\gamma\)=\-\\sqrt\{2\}\.Therefore there exists a uniqueγB,β∗\\gamma\_\{B,\\beta\}^\{\*\}such that

ψB,β​\(γB,β∗\)=0\.\\psi\_\{B,\\beta\}\(\\gamma\_\{B,\\beta\}^\{\*\}\)=0\.
Unlike the single\-batch mean drift, the stationary frozen\-momentum drift has no analogue of the thresholdp1B=1/2p\_\{1\}^\{B\}=1/2: for everyβ\>0\\beta\>0, both tasks receive positive weight almost surely in the stationary EMA, producing restoring drift in both tails\. This is the basic mechanism by which momentum suppresses task\-frequency fluctuations across steps\.

##### Recovery of the full\-batch gap\.

From equation[21](https://arxiv.org/html/2610.31728#A3.E21),

Var⁡\(Zp1\)=p2Beff​p1,\\operatorname\{Var\}\\\!\\left\(\\frac\{Z\}\{p\_\{1\}\}\\right\)=\\frac\{p\_\{2\}\}\{B\_\{\\mathrm\{eff\}\}p\_\{1\}\},and

Var⁡\(1−Zp2\)=p1Beff​p2\.\\operatorname\{Var\}\\\!\\left\(\\frac\{1\-Z\}\{p\_\{2\}\}\\right\)=\\frac\{p\_\{1\}\}\{B\_\{\\mathrm\{eff\}\}p\_\{2\}\}\.Hence, if

Beff​pmin⟶∞,B\_\{\\mathrm\{eff\}\}p\_\{\\min\}\\longrightarrow\\infty,\(24\)then

Zp1→ℙ1,1−Zp2→ℙ1\.\\frac\{Z\}\{p\_\{1\}\}\\xrightarrow\{\\mathbb\{P\}\}1,\\qquad\\frac\{1\-Z\}\{p\_\{2\}\}\\xrightarrow\{\\mathbb\{P\}\}1\.Since\|φ\|≤2\|\\varphi\|\\leq\\sqrt\{2\},

ψB,β​\(γ\)⟶ψ∞​\(γ\)=2​e−γ−1e−2​γ\+1\\psi\_\{B,\\beta\}\(\\gamma\)\\longrightarrow\\psi\_\{\\infty\}\(\\gamma\)=\\sqrt\{2\}\\frac\{e^\{\-\\gamma\}\-1\}\{\\sqrt\{e^\{\-2\\gamma\}\+1\}\}for every fixedγ\\gamma\.

As before, monotonicity and uniqueness of the zero ofψ∞\\psi\_\{\\infty\}imply

γB,β∗⟶0as​Beff​pmin→∞\.\\gamma\_\{B,\\beta\}^\{\*\}\\longrightarrow 0\\qquad\\text\{as \}B\_\{\\mathrm\{eff\}\}p\_\{\\min\}\\to\\infty\.\(25\)
Since

Beff=B​1\+β1−β,B\_\{\\mathrm\{eff\}\}=B\\frac\{1\+\\beta\}\{1\-\\beta\},condition \([24](https://arxiv.org/html/2610.31728#A3.E24)\) is equivalent, up to the bounded factor1\+β1\+\\beta, to

1−βB​pmin⟶0\.\\frac\{1\-\\beta\}\{Bp\_\{\\min\}\}\\longrightarrow 0\.
For fixedp1,p2\>0p\_\{1\},p\_\{2\}\>0, the finite\-sampling bias can be sharpened by a second\-order expansion\. Using the functionhhdefined in the previous subsection,

𝔼⁡\[h⁡\(Z\)\]=12​h′′​\(p1\)​Var⁡\(Z\)\+o⁡\(Beff−1\)\.\\mathbb\{E\}\[h\(Z\)\]=\\frac\{1\}\{2\}h^\{\\prime\\prime\}\(p\_\{1\}\)\\operatorname\{Var\}\(Z\)\+o\\\!\\left\(B\_\{\\mathrm\{eff\}\}^\{\-1\}\\right\)\.Therefore,

ψB,β​\(0\)=p1−p22​Beff​p1​p2\+o⁡\(Beff−1\)\.\\psi\_\{B,\\beta\}\(0\)=\\frac\{p\_\{1\}\-p\_\{2\}\}\{2B\_\{\\mathrm\{eff\}\}p\_\{1\}p\_\{2\}\}\+o\\\!\\left\(B\_\{\\mathrm\{eff\}\}^\{\-1\}\\right\)\.Moreover,

ψB,β′​\(0\)=−1\+o⁡\(1\)\.\\psi\_\{B,\\beta\}^\{\\prime\}\(0\)=\-1\+o\(1\)\.Expanding the unique zero ofψB,β\\psi\_\{B,\\beta\}around zero gives

γB,β∗=p1−p22​Beff​p1​p2\+o⁡\(Beff−1\)\.\\gamma\_\{B,\\beta\}^\{\*\}=\\frac\{p\_\{1\}\-p\_\{2\}\}\{2B\_\{\\mathrm\{eff\}\}p\_\{1\}p\_\{2\}\}\+o\\\!\\left\(B\_\{\\mathrm\{eff\}\}^\{\-1\}\\right\)\.\(26\)Equivalently,

γB,β∗=1−β1\+β​p1−p22​B​p1​p2\+o⁡\(Beff−1\)\.\\gamma\_\{B,\\beta\}^\{\*\}=\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{p\_\{1\}\-p\_\{2\}\}\{2Bp\_\{1\}p\_\{2\}\}\+o\\\!\\left\(B\_\{\\mathrm\{eff\}\}^\{\-1\}\\right\)\.Thus momentum suppresses the finite\-sampling displacement of the effective gap according toBeffB\_\{\\mathrm\{eff\}\}\.

##### Tracking slowly evolving margins\.

We now return to the actual dynamics, in whichui​\(t\)u\_\{i\}\(t\)evolve over time\. The EMA has a memory length of order

ϵ−1=\(1−β\)−1\\epsilon^\{\-1\}=\(1\-\\beta\)^\{\-1\}iterations\. Over one iteration, the margins move byO⁡\(η\)O\(\\eta\)\. In the late\-time balanced regime, the relative variation of the mean gradient over one iteration is therefore alsoO⁡\(η\)O\(\\eta\)\. Across one EMA window, this produces a relative tracking error of order

O⁡\(ηϵ\)\.O\\\!\\left\(\\frac\{\\eta\}\{\\epsilon\}\\right\)\.\(27\)
On the other hand, the frozen\-buffer fluctuation scale follows from equation[21](https://arxiv.org/html/2610.31728#A3.E21)\. Near balanced growth, where

p1​u1≍p2​u2,p\_\{1\}u\_\{1\}\\asymp p\_\{2\}u\_\{2\},we obtain

‖mt−𝔼⁡\[mt\]‖2‖𝔼⁡\[mt\]‖2=Oℙ​\(ϵB​pmin\)\.\\frac\{\\\|m\_\{t\}\-\\mathbb\{E\}\[m\_\{t\}\]\\\|\_\{2\}\}\{\\\|\\mathbb\{E\}\[m\_\{t\}\]\\\|\_\{2\}\}=O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{\\epsilon\}\{Bp\_\{\\min\}\}\}\\right\)\.\(28\)
Thus the two scale\-separation requirements are

ηϵ≪1,ϵB​pmin≪1\.\\frac\{\\eta\}\{\\epsilon\}\\ll 1,\\qquad\\frac\{\\epsilon\}\{Bp\_\{\\min\}\}\\ll 1\.\(29\)Equivalently,

η≪1−β≪B​pmin\.\\eta\\ll 1\-\\beta\\ll Bp\_\{\\min\}\.The first condition ensures that the EMA tracks the slowly evolving gradient direction, while the second ensures that momentum averaging provides sufficiently small relative task\-frequency fluctuations\.

##### Long\-time diffusion with momentum\.

Momentum reduces the instantaneous variance of the buffer but also introduces serial correlations\. We now examine their combined effect on the leading long\-time diffusion coefficient\.

Freeze the margins and write

g¯=\(p1​u1,p2​u2\)⊤,nt:=mt−g¯\.\\bar\{g\}=\\left\(p\_\{1\}u\_\{1\},p\_\{2\}u\_\{2\}\\right\)^\{\\top\},\\qquad n\_\{t\}:=m\_\{t\}\-\\bar\{g\}\.The fresh\-gradient covariance is

Σ=p1​p2B​\[u12−u1​u2−u1​u2u22\]\.\\Sigma=\\frac\{p\_\{1\}p\_\{2\}\}\{B\}\\begin\{bmatrix\}u\_\{1\}^\{2\}&\-u\_\{1\}u\_\{2\}\\\\ \-u\_\{1\}u\_\{2\}&u\_\{2\}^\{2\}\\end\{bmatrix\}\.The stationary EMA covariance satisfies

Cov⁡\(nt\)=1−β1\+β​Σ\.\\operatorname\{Cov\}\(n\_\{t\}\)=\\frac\{1\-\\beta\}\{1\+\\beta\}\\Sigma\.More generally,

Cov⁡\(nt,nt\+k\)=β\|k\|​1−β1\+β​Σ\.\\operatorname\{Cov\}\(n\_\{t\},n\_\{t\+k\}\)=\\beta^\{\|k\|\}\\frac\{1\-\\beta\}\{1\+\\beta\}\\Sigma\.Consequently,

∑k∈ℤCov⁡\(n0,nk\)\\displaystyle\\sum\_\{k\\in\\mathbb\{Z\}\}\\operatorname\{Cov\}\(n\_\{0\},n\_\{k\}\)=1−β1\+β​\(1\+2​∑k=1∞βk\)​Σ\\displaystyle=\\frac\{1\-\\beta\}\{1\+\\beta\}\\left\(1\+2\\sum\_\{k=1\}^\{\\infty\}\\beta^\{k\}\\right\)\\Sigma=Σ\.\\displaystyle=\\Sigma\.\(30\)Thus the reduction in instantaneous variance is exactly compensated, at the covariance level, by the increased correlation time\. To translate this into fluctuations of the gap, define

ℓ⁡\(m\):=2​m1−m2‖m‖2\.\\ell\(m\):=\\sqrt\{2\}\\frac\{m\_\{1\}\-m\_\{2\}\}\{\\\|m\\\|\_\{2\}\}\.Then

D⁡\(t\+1\)−D⁡\(t\)=η​ℓ​\(mt\)\.D\(t\+1\)\-D\(t\)=\\eta\\,\\ell\(m\_\{t\}\)\.Linearizing aroundg¯\\bar\{g\},

ℓ\(mt\)−ℓ\(g¯\)≃∇ℓ\(g¯\)⊤nt\.\\ell\(m\_\{t\}\)\-\\ell\(\\bar\{g\}\)\\simeq\\nabla\\ell\(\\bar\{g\}\)^\{\\top\}n\_\{t\}\.Hence, by equation[30](https://arxiv.org/html/2610.31728#A3.E30),

∑k∈ℤCov\(ℓ\(m0\),ℓ\(mk\)\)≃∇ℓ\(g¯\)⊤Σ∇ℓ\(g¯\)\.\\sum\_\{k\\in\\mathbb\{Z\}\}\\operatorname\{Cov\}\\\!\\left\(\\ell\(m\_\{0\}\),\\ell\(m\_\{k\}\)\\right\)\\simeq\\nabla\\ell\(\\bar\{g\}\)^\{\\top\}\\Sigma\\nabla\\ell\(\\bar\{g\}\)\.
At the full\-batch balanced point

p1​u1=p2​u2,p\_\{1\}u\_\{1\}=p\_\{2\}u\_\{2\},a direct calculation gives

∇ℓ\(g¯\)⊤Σ∇ℓ\(g¯\)=1B​p1​p2\.\\nabla\\ell\(\\bar\{g\}\)^\{\\top\}\\Sigma\\nabla\\ell\(\\bar\{g\}\)=\\frac\{1\}\{Bp\_\{1\}p\_\{2\}\}\.\(31\)SinceγB,β∗→0\\gamma\_\{B,\\beta\}^\{\*\}\\to 0in the well\-sampled regime, the same coefficient holds to leading order around the momentum\-induced equilibrium\.

Therefore, on the slow time

the local gap dynamics are approximated by

d​γ≃−\(γ−γB,β∗\)​d​τ\+ηB​p1​p2​d​𝒲τ,d\\gamma\\simeq\-\\left\(\\gamma\-\\gamma\_\{B,\\beta\}^\{\*\}\\right\)d\\tau\+\\sqrt\{\\frac\{\\eta\}\{Bp\_\{1\}p\_\{2\}\}\}\\,d\\mathcal\{W\}\_\{\\tau\},\(32\)up to corrections associated with finite effective sample size and tracking error\. In particular, the local stationary fluctuation scale is

Oℙ​\(ηB​p1​p2\),O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{\\eta\}\{Bp\_\{1\}p\_\{2\}\}\}\\right\),which matches the leading fluctuation scale of the momentum\-free well\-sampled regime\.

Combining the momentum\-induced equilibrium, the tracking error, and the local diffusion scale gives the late\-time approximation

D=log⁡p1p2\+γB,β∗\+O⁡\(η1−β\)\+Oℙ​\(ηB​p1​p2\)\.D=\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\}\+\\gamma\_\{B,\\beta\}^\{\*\}\+O\\\!\\left\(\\frac\{\\eta\}\{1\-\\beta\}\\right\)\+O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{\\eta\}\{Bp\_\{1\}p\_\{2\}\}\}\\right\)\.\(33\)
Finally, consider a scale\-separated regime satisfying

B​pmin1−β⟶∞,η1−β⟶0\.\\frac\{Bp\_\{\\min\}\}\{1\-\\beta\}\\longrightarrow\\infty,\\qquad\\frac\{\\eta\}\{1\-\\beta\}\\longrightarrow 0\.The first condition implies

Beff​pmin⟶∞,B\_\{\\mathrm\{eff\}\}p\_\{\\min\}\\longrightarrow\\infty,and hence, by equation[25](https://arxiv.org/html/2610.31728#A3.E25),

γB,β∗⟶0\.\\gamma\_\{B,\\beta\}^\{\*\}\\longrightarrow 0\.Moreover, sincep1≥1/2p\_\{1\}\\geq 1/2andp2=pminp\_\{2\}=p\_\{\\min\},

ηB​p1​p2=O⁡\(η1−β​1−βB​pmin\)⟶0\.\\frac\{\\eta\}\{Bp\_\{1\}p\_\{2\}\}=O\\\!\\left\(\\frac\{\\eta\}\{1\-\\beta\}\\frac\{1\-\\beta\}\{Bp\_\{\\min\}\}\\right\)\\longrightarrow 0\.Thus the late\-time local approximation gives

Δ1−Δ2=log⁡p1p2\+oℙ​\(1\),B​pmin1−β→∞,η1−β→0\.\\Delta\_\{1\}\-\\Delta\_\{2\}=\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\}\+o\_\{\\mathbb\{P\}\}\(1\),\\qquad\\frac\{Bp\_\{\\min\}\}\{1\-\\beta\}\\to\\infty,\\qquad\\frac\{\\eta\}\{1\-\\beta\}\\to 0\.\(34\)
Momentum therefore acts as cross\-step task aggregation\. Under frozen margins, it replaces the noisy within\-batch frequency by an exponentially averaged frequency with variance\-equivalent batch sizeBeff=B⁡\(1\+β\)/\(1−β\)B\_\{\\mathrm\{eff\}\}=B\(1\+\\beta\)/\(1\-\\beta\)\. When the EMA also tracks the slowly evolving margins, this variance reduction drives the gap toward the full\-batch value\. At the same time, the reduction in instantaneous variance is compensated by increased correlation across steps, leaving the leading local long\-time diffusion coefficient unchanged\.

### Appendix DHigher\-Rank Sampling Dynamics and Momentum Averaging

Throughout this section, theK=M​CK=MCtasks are partitioned intoMMblocks of sizeCC\. Tasks in blockμ\\muhave common population frequency

pj=p~μC,∑μ=1Mp~μ=1\.p\_\{j\}=\\frac\{\\tilde\{p\}\_\{\\mu\}\}\{C\},\\qquad\\sum\_\{\\mu=1\}^\{M\}\\tilde\{p\}\_\{\\mu\}=1\.The organization mirrors Proposition[4\.2](https://arxiv.org/html/2610.31728#S4.Thmtheorem2)\. Appendix D\.1 gives the exact occurrence\-gated clock in the frequency\-limited regime, and Appendix D\.3 gives the balanced full\-batch contrast response\. Appendices D\.4–D\.5 then establish how the well\-sampled response approaches that full\-batch behavior and quantify the residualOℙ\(\(Bpj\)−1/2\)O\_\{\\mathbb\{P\}\}\(\(Bp\_\{j\}\)^\{\-1/2\}\)coupling; the cancellation removes the direct contrast perturbation, leaving contrast\-to\-average coupling as the leading first\-order effect\. Appendix D\.6 treats momentum as averaging across steps with a separate tracking condition\. A finite\-horizon continuum theorem is retained in Appendix D\.7 as a technical refinement\.

#### D\.1Sparse regime: exact occurrence\-gated clock and block\-symmetrized reference

We begin with the part of the hierarchical dynamics that is exact for arbitrary batch size\. The partial polar factor has the same right nullspace as its input: ifG​x=0Gx=0, thenmsgn⁡\(G\)​x=0\\operatorname\{msgn\}\(G\)x=0\. Indeed, for an SVDG=U​Σ​V⊤G=U\\Sigma V^\{\\top\},G​x=0Gx=0impliesΣ​V⊤​x=0\\Sigma V^\{\\top\}x=0; the coordinates ofV⊤​xV^\{\\top\}xcorresponding to positive singular values therefore vanish, which also givesmsgn⁡\(G\)​x=0\\operatorname\{msgn\}\(G\)x=0\.

For taskjj, ifqj​\(t\)=0q\_\{j\}\(t\)=0then

\(I−St\)​Qt​ej=0,\(I\-S\_\{t\}\)Q\_\{t\}e\_\{j\}=0,so

\(W¯t\+1−W¯t\)​ej=0\.\(\\overline\{W\}\_\{t\+1\}\-\\overline\{W\}\_\{t\}\)e\_\{j\}=0\.\(35\)Since‖msgn⁡\(G\)‖op≤1\\\|\\operatorname\{msgn\}\(G\)\\\|\_\{\\mathrm\{op\}\}\\leq 1,

∥W¯Tej−W¯0ej∥2≤ηNj\(T\),Nj\(T\):=∑t<T𝟏\{qj\(t\)\>0\}\.\\\|\\overline\{W\}\_\{T\}e\_\{j\}\-\\overline\{W\}\_\{0\}e\_\{j\}\\\|\_\{2\}\\leq\\eta N\_\{j\}\(T\),\\qquad N\_\{j\}\(T\):=\\sum\_\{t<T\}\\mathbf\{1\}\\\{q\_\{j\}\(t\)\>0\\\}\.\(36\)For independent mini\-batches, the indicators are i\.i\.d\. Bernoulli with parameter

rj=1−\(1−pj\)B,r\_\{j\}=1\-\(1\-p\_\{j\}\)^\{B\},and henceNj​\(T\)/T→rjN\_\{j\}\(T\)/T\\to r\_\{j\}almost surely\. The expansionrj=B​pj\+O⁡\(\(B​pj\)2\)r\_\{j\}=Bp\_\{j\}\+O\(\(Bp\_\{j\}\)^\{2\}\)in the sparse regime gives the trajectory\-level frequency clock used in the main text\.

Next define the empirical block masses

πμ,t=∑j∈μqj​\(t\)\\pi\_\{\\mu,t\}=\\sum\_\{j\\in\\mu\}q\_\{j\}\(t\)and the block\-averaged count matrix

Qt∥=diag⁡\(π1,tC​IC,…,πM,tC​IC\)\.Q\_\{t\}^\{\\parallel\}=\\operatorname\{diag\}\\\!\\left\(\\frac\{\\pi\_\{1,t\}\}\{C\}I\_\{C\},\\ldots,\\frac\{\\pi\_\{M,t\}\}\{C\}I\_\{C\}\\right\)\.Starting fromW~0=0\\widetilde\{W\}\_\{0\}=0, consider

W~t\+1=W~t\+η​msgn⁡\(\(I−S~t\)​Qt∥\)\.\\widetilde\{W\}\_\{t\+1\}=\\widetilde\{W\}\_\{t\}\+\\eta\\operatorname\{msgn\}\(\(I\-\\widetilde\{S\}\_\{t\}\)Q\_\{t\}^\{\\parallel\}\)\.Both the initialization andQt∥Q\_\{t\}^\{\\parallel\}are invariant under permutations within each block, so the trajectory remains block symmetric\. The decomposition in the next subsection therefore applies at every step\. Wheneverπμ,t\>0\\pi\_\{\\mu,t\}\>0, the contrast singular value onUμU\_\{\\mu\}is positive and the partial polar factor equals the identity there; whenπμ,t=0\\pi\_\{\\mu,t\}=0, that contrast block is in the right nullspace\. Hence

Uμ⊤\(W~t\+1−W~t\)Uμ=η𝟏\{πμ,t\>0\}IC−1,U\_\{\\mu\}^\{\\top\}\(\\widetilde\{W\}\_\{t\+1\}\-\\widetilde\{W\}\_\{t\}\)U\_\{\\mu\}=\\eta\\mathbf\{1\}\\\{\\pi\_\{\\mu,t\}\>0\\\}I\_\{C\-1\},\(37\)Summing over steps gives

Uμ⊤W~TUμ=ηNμblk\(T\)IC−1,Nμblk\(T\):=∑t<T𝟏\{πμ,t\>0\}\.U\_\{\\mu\}^\{\\top\}\\widetilde\{W\}\_\{T\}U\_\{\\mu\}=\\eta N\_\{\\mu\}^\{\\mathrm\{blk\}\}\(T\)I\_\{C\-1\},\\qquad N\_\{\\mu\}^\{\\mathrm\{blk\}\}\(T\):=\\sum\_\{t<T\}\\mathbf\{1\}\\\{\\pi\_\{\\mu,t\}\>0\\\}\.\(38\)This sampled block\-symmetric process is an exact finite\-BBbenchmark\. Appendix D\.7 records the stronger finite\-horizon continuum refinement obtained after taking large expected task counts and a small step size\.

#### D\.2Block decomposition at a symmetric reference state

For each blockμ\\mu, let

Xμ∈ℝC×\(C−1\)X\_\{\\mu\}\\in\\mathbb\{R\}^\{C\\times\(C\-1\)\}have orthonormal columns spanning the zero\-sum subspace

\{x∈ℝC:𝟏C⊤​x=0\}\.\\left\\\{x\\in\\mathbb\{R\}^\{C\}:\\mathbf\{1\}\_\{C\}^\{\\top\}x=0\\right\\\}\.LetUμ∈ℝK×\(C−1\)U\_\{\\mu\}\\in\\mathbb\{R\}^\{K\\times\(C\-1\)\}denote the embedding ofXμX\_\{\\mu\}into blockμ\\mu\. Define the normalized block\-average vectors

\(yμ\)i=\{1/C,i∈μ,0,otherwise,\(y\_\{\\mu\}\)\_\{i\}=\\begin\{cases\}1/\\sqrt\{C\},&i\\in\\mu,\\\\ 0,&\\text\{otherwise\},\\end\{cases\}and set

Y=\(y1,…,yM\),𝒰=\(U1,…,UM,Y\)\.Y=\(y\_\{1\},\\ldots,y\_\{M\}\),\\qquad\\mathcal\{U\}=\(U\_\{1\},\\ldots,U\_\{M\},Y\)\.Then𝒰\\mathcal\{U\}is orthogonal\.

Consider a block\-symmetric softmax matrixS~\\widetilde\{S\}, and write

A:=I−S~\.A:=I\-\\widetilde\{S\}\.Block symmetry implies that each within\-block contrast subspace is invariant underAA\. Hence there exist scalarsaμa\_\{\\mu\}and a matrixL∈ℝM×ML\\in\\mathbb\{R\}^\{M\\times M\}such that

𝒰⊤​A​𝒰=\[diag⁡\(aμ​IC−1\)00L\]\.\\mathcal\{U\}^\{\\top\}A\\mathcal\{U\}=\\begin\{bmatrix\}\\operatorname\{diag\}\\\!\\left\(a\_\{\\mu\}I\_\{C\-1\}\\right\)&0\\\\ 0&L\\end\{bmatrix\}\.\(39\)Equivalently, ifdμd\_\{\\mu\}denotes the eigenvalue ofS~\\widetilde\{S\}on the within\-block zero\-sum subspace of blockμ\\mu, then

aμ=1−dμ\.a\_\{\\mu\}=1\-d\_\{\\mu\}\.Along the block\-symmetric full\-batch trajectory considered below,aμ\>0a\_\{\\mu\}\>0\.

Now let

Q∥=diag⁡\(r1​IC,…,rM​IC\),rμ\>0\.Q^\{\\parallel\}=\\operatorname\{diag\}\\left\(r\_\{1\}I\_\{C\},\\ldots,r\_\{M\}I\_\{C\}\\right\),\\qquad r\_\{\\mu\}\>0\.In the basis𝒰\\mathcal\{U\},

𝒰⊤​Q∥​𝒰=\[diag⁡\(rμ​IC−1\)00R\],R:=diag⁡\(r1,…,rM\)\.\\mathcal\{U\}^\{\\top\}Q^\{\\parallel\}\\mathcal\{U\}=\\begin\{bmatrix\}\\operatorname\{diag\}\\\!\\left\(r\_\{\\mu\}I\_\{C\-1\}\\right\)&0\\\\ 0&R\\end\{bmatrix\},\\qquad R:=\\operatorname\{diag\}\(r\_\{1\},\\ldots,r\_\{M\}\)\.Therefore the block\-symmetric reference gradient

G~:=A​Q∥\\widetilde\{G\}:=AQ^\{\\parallel\}takes the form

𝒰⊤​G~​𝒰=\[diag⁡\(σμ​IC−1\)00H\],σμ:=aμ​rμ,H:=L​R\.\\mathcal\{U\}^\{\\top\}\\widetilde\{G\}\\mathcal\{U\}=\\begin\{bmatrix\}\\operatorname\{diag\}\\\!\\left\(\\sigma\_\{\\mu\}I\_\{C\-1\}\\right\)&0\\\\ 0&H\\end\{bmatrix\},\\qquad\\sigma\_\{\\mu\}:=a\_\{\\mu\}r\_\{\\mu\},\\qquad H:=LR\.\(40\)
SinceS~\\widetilde\{S\}is a positive column\-stochastic matrix,A=I−S~A=I\-\\widetilde\{S\}has rankK−1K\-1\. Becauseaμ\>0a\_\{\\mu\}\>0, it follows from equation[39](https://arxiv.org/html/2610.31728#A4.E39)thatLL, and henceHH, has rankM−1M\-1\.

#### D\.3Full\-batch balanced contrast dynamics

Under full\-batch training,

Q=P=diag⁡\(p~1C​IC,…,p~MC​IC\)\.Q=P=\\operatorname\{diag\}\\left\(\\frac\{\\tilde\{p\}\_\{1\}\}\{C\}I\_\{C\},\\ldots,\\frac\{\\tilde\{p\}\_\{M\}\}\{C\}I\_\{C\}\\right\)\.Starting fromW¯0=0\\overline\{W\}\_\{0\}=0, the full\-batch dynamics preserve block symmetry\. Thus\(I−St\)​P\(I\-S\_\{t\}\)Phas the form equation[40](https://arxiv.org/html/2610.31728#A4.E40)at every iteration, with

rμ=p~μC\.r\_\{\\mu\}=\\frac\{\\tilde\{p\}\_\{\\mu\}\}\{C\}\.
Since every contrast singular valueσμ\\sigma\_\{\\mu\}is positive, the matrix sign acts as the identity on every within\-block contrast subspace:

msgn⁡\(\(I−St\)​P\)=∑μ=1MUμ​Uμ⊤\+Y​msgn⁡\(Ht\)​Y⊤\.\\operatorname\{msgn\}\\\!\\left\(\(I\-S\_\{t\}\)P\\right\)=\\sum\_\{\\mu=1\}^\{M\}U\_\{\\mu\}U\_\{\\mu\}^\{\\top\}\+Y\\operatorname\{msgn\}\(H\_\{t\}\)Y^\{\\top\}\.\(41\)Using

∑μ=1MUμ​Uμ⊤=IK−Y​Y⊤,\\sum\_\{\\mu=1\}^\{M\}U\_\{\\mu\}U\_\{\\mu\}^\{\\top\}=I\_\{K\}\-YY^\{\\top\},we may equivalently write

msgn⁡\(\(I−St\)​P\)=IK\+Y⁡\(msgn⁡\(Ht\)−IM\)​Y⊤\.\\operatorname\{msgn\}\\\!\\left\(\(I\-S\_\{t\}\)P\\right\)=I\_\{K\}\+Y\\left\(\\operatorname\{msgn\}\(H\_\{t\}\)\-I\_\{M\}\\right\)Y^\{\\top\}\.\(42\)Since

‖msgn⁡\(Ht\)−IM‖op≤2,\\left\\\|\\operatorname\{msgn\}\(H\_\{t\}\)\-I\_\{M\}\\right\\\|\_\{\\mathrm\{op\}\}\\leq 2,every entry of the correction term in equation[42](https://arxiv.org/html/2610.31728#A4.E42)isO⁡\(1/C\)O\(1/C\)\. Hence

msgn⁡\(\(I−St\)​P\)=IK\+Oentry​\(1C\)\.\\operatorname\{msgn\}\\\!\\left\(\(I\-S\_\{t\}\)P\\right\)=I\_\{K\}\+O\_\{\\mathrm\{entry\}\}\\\!\\left\(\\frac\{1\}\{C\}\\right\)\.
More strongly, the within\-block statement is exact:

Uμ⊤​msgn⁡\(\(I−St\)​P\)​Uν=δμ​ν​IC−1\.U\_\{\\mu\}^\{\\top\}\\operatorname\{msgn\}\\\!\\left\(\(I\-S\_\{t\}\)P\\right\)U\_\{\\nu\}=\\delta\_\{\\mu\\nu\}I\_\{C\-1\}\.\(43\)Therefore

Uμ⊤​\(W¯t\+1−W¯t\)​Uμ=η​IC−1\.U\_\{\\mu\}^\{\\top\}\\left\(\\overline\{W\}\_\{t\+1\}\-\\overline\{W\}\_\{t\}\\right\)U\_\{\\mu\}=\\eta I\_\{C\-1\}\.Thus full\-batch Muon exactly equalizes the update on all within\-block contrast directions; all remaining nontrivial structure is confined to theMM\-dimensional block\-average subspace\.

#### D\.4Mini\-batch perturbation decomposition around the symmetric reference

We analyze mini\-batch sampling fluctuations around a fixed block\-symmetric reference stateS~\\widetilde\{S\}\. The calculation is local and one\-step: it quantifies the instantaneous response of the polar map rather than the full departure ofStS\_\{t\}from the block\-symmetric manifold\.

Let

Q=diag⁡\(q1,…,qK\),Q=\\operatorname\{diag\}\(q\_\{1\},\\ldots,q\_\{K\}\),whereB​qBqis the empirical frequency vector of a mini\-batch of sizeBB\. Define the empirical block frequencies

πμ:=∑j∈μqj\.\\pi\_\{\\mu\}:=\\sum\_\{j\\in\\mu\}q\_\{j\}\.Decompose

Q=Q∥\+Q⟂,Q=Q^\{\\parallel\}\+Q^\{\\perp\},where

Q∥=diag⁡\(π1C​IC,…,πMC​IC\)\.Q^\{\\parallel\}=\\operatorname\{diag\}\\left\(\\frac\{\\pi\_\{1\}\}\{C\}I\_\{C\},\\ldots,\\frac\{\\pi\_\{M\}\}\{C\}I\_\{C\}\\right\)\.\(44\)Forj∈μj\\in\\mu, define the within\-block residual

\(ϵμ\)j:=qj−πμC\.\(\\epsilon\_\{\\mu\}\)\_\{j\}:=q\_\{j\}\-\\frac\{\\pi\_\{\\mu\}\}\{C\}\.Then

𝟏C⊤​ϵμ=0,\\mathbf\{1\}\_\{C\}^\{\\top\}\\epsilon\_\{\\mu\}=0,and the restriction ofQ⟂Q^\{\\perp\}to blockμ\\muis

diag⁡\(ϵμ\)\.\\operatorname\{diag\}\(\\epsilon\_\{\\mu\}\)\.
Conditioned onπμ\\pi\_\{\\mu\}, theB​πμB\\pi\_\{\\mu\}samples falling in blockμ\\muare uniformly distributed among itsCCtasks\. Therefore

𝔼⁡\[ϵμ∣πμ\]=0,\\mathbb\{E\}\[\\epsilon\_\{\\mu\}\\mid\\pi\_\{\\mu\}\]=0,\(45\)and

Cov⁡\(ϵμ∣πμ\)=πμB​C​\(IC−1C​JC\)\.\\operatorname\{Cov\}\\\!\\left\(\\epsilon\_\{\\mu\}\\mid\\pi\_\{\\mu\}\\right\)=\\frac\{\\pi\_\{\\mu\}\}\{BC\}\\left\(I\_\{C\}\-\\frac\{1\}\{C\}J\_\{C\}\\right\)\.\(46\)Define

zμ:=Xμ⊤​ϵμ\.z\_\{\\mu\}:=X\_\{\\mu\}^\{\\top\}\\epsilon\_\{\\mu\}\.SinceXμ⊤​JC=0X\_\{\\mu\}^\{\\top\}J\_\{C\}=0,

Cov⁡\(zμ∣πμ\)=πμB​C​IC−1,\\operatorname\{Cov\}\\\!\\left\(z\_\{\\mu\}\\mid\\pi\_\{\\mu\}\\right\)=\\frac\{\\pi\_\{\\mu\}\}\{BC\}I\_\{C\-1\},\(47\)and hence

𝔼⁡\[‖zμ‖22∣πμ\]=\(C−1\)​πμB​C\.\\mathbb\{E\}\\\!\\left\[\\\|z\_\{\\mu\}\\\|\_\{2\}^\{2\}\\mid\\pi\_\{\\mu\}\\right\]=\\frac\{\(C\-1\)\\pi\_\{\\mu\}\}\{BC\}\.\(48\)
Let

A:=I−S~,G~:=A​Q∥,E:=A​Q⟂\.A:=I\-\\widetilde\{S\},\\qquad\\widetilde\{G\}:=AQ^\{\\parallel\},\\qquad E:=AQ^\{\\perp\}\.For the fixed\-rank calculation below, work on the event

ℰB:=\{πμ\>0for everyμ\}\.\\mathcal\{E\}\_\{B\}:=\\\{\\pi\_\{\\mu\}\>0\\text\{ for every \}\\mu\\\}\.For fixedMM,

ℙ⁡\(ℰBc\)≤∑μ=1M\(1−p~μ\)B≤M​e−B​p~min,\\mathbb\{P\}\(\\mathcal\{E\}\_\{B\}^\{c\}\)\\leq\\sum\_\{\\mu=1\}^\{M\}\(1\-\\tilde\{p\}\_\{\\mu\}\)^\{B\}\\leq Me^\{\-B\\tilde\{p\}\_\{\\min\}\},soℙ⁡\(ℰB\)→1\\mathbb\{P\}\(\\mathcal\{E\}\_\{B\}\)\\to 1in the well\-sampled regime considered below\. OnℰB\\mathcal\{E\}\_\{B\},Q∥Q^\{\\parallel\}is invertible\. Moreover, for sufficiently small\|s\|\|s\|, the diagonal matrix

Q∥\+s​Q⟂Q^\{\\parallel\}\+sQ^\{\\perp\}remains invertible, so the path

G⁡\(s\):=A⁡\(Q∥\+s​Q⟂\)=G~\+s​EG\(s\):=A\\left\(Q^\{\\parallel\}\+sQ^\{\\perp\}\\right\)=\\widetilde\{G\}\+sEhas constant rankK−1K\-1in a neighborhood ofs=0s=0\.

Becausemsgn\\operatorname\{msgn\}is not an ambient\-smooth map at a rank\-deficient matrix, we define its derivative here along this constant\-rank path:

𝒟G~​\[E\]:=dd​s​msgn⁡\(G~\+s​E\)\|s=0\.\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]:=\\left\.\\frac\{d\}\{ds\}\\operatorname\{msgn\}\\\!\\left\(\\widetilde\{G\}\+sE\\right\)\\right\|\_\{s=0\}\.\(49\)
We will use the following standard fixed\-rank derivative formula\.

###### Lemma D\.1\(Derivative of the partial polar factor along a fixed\-rank path\)\.

LetG⁡\(s\)G\(s\)be a differentiable square\-matrix path of constant rankrrnears=0s=0, and write

G⁡\(0\)=U​\[Σ000\]​V⊤,Σ=diag⁡\(σ1,…,σr\),σi\>0\.G\(0\)=U\\begin\{bmatrix\}\\Sigma&0\\\\ 0&0\\end\{bmatrix\}V^\{\\top\},\\qquad\\Sigma=\\operatorname\{diag\}\(\\sigma\_\{1\},\\ldots,\\sigma\_\{r\}\),\\qquad\\sigma\_\{i\}\>0\.Let

E:=G′​\(0\),E^:=U⊤​E​V,E:=G^\{\\prime\}\(0\),\\qquad\\widehat\{E\}:=U^\{\\top\}EV,and let

P˙:=dd​s​msgn⁡\(G⁡\(s\)\)\|s=0\.\\dot\{P\}:=\\left\.\\frac\{d\}\{ds\}\\operatorname\{msgn\}\(G\(s\)\)\\right\|\_\{s=0\}\.PartitionU=\(U\+,U0\)U=\(U\_\{\+\},U\_\{0\}\)andV=\(V\+,V0\)V=\(V\_\{\+\},V\_\{0\}\)according to the positive and zero singular subspaces\. Then

\(U\+⊤​P˙​V\+\)i​j\\displaystyle\\left\(U\_\{\+\}^\{\\top\}\\dot\{P\}V\_\{\+\}\\right\)\_\{ij\}=\(E^\+⁣\+\)i​j−\(E^\+⁣\+\)j​iσi\+σj,\\displaystyle=\\frac\{\(\\widehat\{E\}\_\{\+\+\}\)\_\{ij\}\-\(\\widehat\{E\}\_\{\+\+\}\)\_\{ji\}\}\{\\sigma\_\{i\}\+\\sigma\_\{j\}\},\(50\)U\+⊤​P˙​V0\\displaystyle U\_\{\+\}^\{\\top\}\\dot\{P\}V\_\{0\}=Σ−1​E^\+0,\\displaystyle=\\Sigma^\{\-1\}\\widehat\{E\}\_\{\+0\},\(51\)U0⊤​P˙​V\+\\displaystyle U\_\{0\}^\{\\top\}\\dot\{P\}V\_\{\+\}=E^0\+​Σ−1,\\displaystyle=\\widehat\{E\}\_\{0\+\}\\Sigma^\{\-1\},\(52\)U0⊤​P˙​V0\\displaystyle U\_\{0\}^\{\\top\}\\dot\{P\}V\_\{0\}=0\.\\displaystyle=0\.\(53\)

###### Proof\.

Because the rank is constant, the column and row spaces vary smoothly locally\. Choose differentiable orthonormal framesU\+​\(s\)U\_\{\+\}\(s\)andV\+​\(s\)V\_\{\+\}\(s\)for these spaces and write

G⁡\(s\)=U\+​\(s\)​R​\(s\)​V\+​\(s\)⊤,G\(s\)=U\_\{\+\}\(s\)R\(s\)V\_\{\+\}\(s\)^\{\\top\},whereR⁡\(s\)∈ℝr×rR\(s\)\\in\\mathbb\{R\}^\{r\\times r\}is invertible\. We choose the frames so thatR⁡\(0\)=ΣR\(0\)=\\Sigma\. IfPR​\(s\)P\_\{R\}\(s\)denotes the polar factor ofR⁡\(s\)R\(s\), then

msgn⁡\(G⁡\(s\)\)=U\+​\(s\)​PR​\(s\)​V\+​\(s\)⊤,PR​\(0\)=Ir\.\\operatorname\{msgn\}\(G\(s\)\)=U\_\{\+\}\(s\)P\_\{R\}\(s\)V\_\{\+\}\(s\)^\{\\top\},\\qquad P\_\{R\}\(0\)=I\_\{r\}\.
Let

A:=U\+⊤​U˙\+,B:=V\+⊤​V˙\+,ΩR:=P˙R​\(0\)\.A:=U\_\{\+\}^\{\\top\}\\dot\{U\}\_\{\+\},\\qquad B:=V\_\{\+\}^\{\\top\}\\dot\{V\}\_\{\+\},\\qquad\\Omega\_\{R\}:=\\dot\{P\}\_\{R\}\(0\)\.BothAAandBBare skew\-symmetric\. Differentiating the polar decomposition of the invertible coreR⁡\(s\)R\(s\)ats=0s=0gives the Sylvester equation

Σ​ΩR\+ΩR​Σ=R˙−R˙⊤\.\\Sigma\\Omega\_\{R\}\+\\Omega\_\{R\}\\Sigma=\\dot\{R\}\-\\dot\{R\}^\{\\top\}\.On the positive singular subspaces,

E^\+⁣\+=A​Σ\+R˙−Σ​B,\\widehat\{E\}\_\{\+\+\}=A\\Sigma\+\\dot\{R\}\-\\Sigma B,while

U\+⊤​P˙​V\+=A\+ΩR−B\.U\_\{\+\}^\{\\top\}\\dot\{P\}V\_\{\+\}=A\+\\Omega\_\{R\}\-B\.Combining the last two displays yields

Σ⁡\(U\+⊤​P˙​V\+\)\+\(U\+⊤​P˙​V\+\)​Σ=E^\+⁣\+−E^\+⁣\+⊤,\\Sigma\\left\(U\_\{\+\}^\{\\top\}\\dot\{P\}V\_\{\+\}\\right\)\+\\left\(U\_\{\+\}^\{\\top\}\\dot\{P\}V\_\{\+\}\\right\)\\Sigma=\\widehat\{E\}\_\{\+\+\}\-\\widehat\{E\}\_\{\+\+\}^\{\\top\},which gives equation[50](https://arxiv.org/html/2610.31728#A4.E50)entrywise\.

For the cross blocks, differentiatingG⁡\(s\)​V0​\(s\)=0G\(s\)V\_\{0\}\(s\)=0andU0​\(s\)⊤​G​\(s\)=0U\_\{0\}\(s\)^\{\\top\}G\(s\)=0, or equivalently projecting the derivative ofG⁡\(s\)=U\+​\(s\)​R​\(s\)​V\+​\(s\)⊤G\(s\)=U\_\{\+\}\(s\)R\(s\)V\_\{\+\}\(s\)^\{\\top\}, gives

U\+⊤​P˙​V0=Σ−1​E^\+0,U0⊤​P˙​V\+=E^0\+​Σ−1\.U\_\{\+\}^\{\\top\}\\dot\{P\}V\_\{0\}=\\Sigma^\{\-1\}\\widehat\{E\}\_\{\+0\},\\qquad U\_\{0\}^\{\\top\}\\dot\{P\}V\_\{\+\}=\\widehat\{E\}\_\{0\+\}\\Sigma^\{\-1\}\.Finally, differentiatingmsgn⁡\(G⁡\(s\)\)=U\+​\(s\)​PR​\(s\)​V\+​\(s\)⊤\\operatorname\{msgn\}\(G\(s\)\)=U\_\{\+\}\(s\)P\_\{R\}\(s\)V\_\{\+\}\(s\)^\{\\top\}shows that its derivative has no null\-to\-null block, proving equation[53](https://arxiv.org/html/2610.31728#A4.E53)\. ∎

We next compute the perturbationE=A​Q⟂E=AQ^\{\\perp\}in the basis𝒰\\mathcal\{U\}\. By equation[39](https://arxiv.org/html/2610.31728#A4.E39),

Uμ⊤​A​Uμ=aμ​IC−1,Uμ⊤​A​Y=0,Y⊤​A​Uμ=0,Y⊤​A​Y=L\.U\_\{\\mu\}^\{\\top\}AU\_\{\\mu\}=a\_\{\\mu\}I\_\{C\-1\},\\qquad U\_\{\\mu\}^\{\\top\}AY=0,\\qquad Y^\{\\top\}AU\_\{\\mu\}=0,\\qquad Y^\{\\top\}AY=L\.Moreover,

Uμ⊤​Q⟂​Uμ\\displaystyle U\_\{\\mu\}^\{\\top\}Q^\{\\perp\}U\_\{\\mu\}=Dμ,Dμ:=Xμ⊤​diag⁡\(ϵμ\)​Xμ,\\displaystyle=D\_\{\\mu\},\\qquad D\_\{\\mu\}:=X\_\{\\mu\}^\{\\top\}\\operatorname\{diag\}\(\\epsilon\_\{\\mu\}\)X\_\{\\mu\},\(54\)Uμ⊤​Q⟂​Y\\displaystyle U\_\{\\mu\}^\{\\top\}Q^\{\\perp\}Y=1C​zμ​eμ⊤,\\displaystyle=\\frac\{1\}\{\\sqrt\{C\}\}z\_\{\\mu\}e\_\{\\mu\}^\{\\top\},\(55\)Y⊤​Q⟂​Uμ\\displaystyle Y^\{\\top\}Q^\{\\perp\}U\_\{\\mu\}=1C​eμ​zμ⊤,\\displaystyle=\\frac\{1\}\{\\sqrt\{C\}\}e\_\{\\mu\}z\_\{\\mu\}^\{\\top\},\(56\)Y⊤​Q⟂​Y\\displaystyle Y^\{\\top\}Q^\{\\perp\}Y=0\.\\displaystyle=0\.\(57\)Hereeμe\_\{\\mu\}is theμ\\mu\-th standard basis vector ofℝM\\mathbb\{R\}^\{M\}\. Hence

Uμ⊤​E​Uμ\\displaystyle U\_\{\\mu\}^\{\\top\}EU\_\{\\mu\}=aμ​Dμ,\\displaystyle=a\_\{\\mu\}D\_\{\\mu\},\(58\)Uμ⊤​E​Y\\displaystyle U\_\{\\mu\}^\{\\top\}EY=aμC​zμ​eμ⊤,\\displaystyle=\\frac\{a\_\{\\mu\}\}\{\\sqrt\{C\}\}z\_\{\\mu\}e\_\{\\mu\}^\{\\top\},\(59\)Y⊤​E​Uμ\\displaystyle Y^\{\\top\}EU\_\{\\mu\}=1C​L​eμ​zμ⊤,\\displaystyle=\\frac\{1\}\{\\sqrt\{C\}\}Le\_\{\\mu\}z\_\{\\mu\}^\{\\top\},\(60\)Y⊤​E​Y\\displaystyle Y^\{\\top\}EY=0\.\\displaystyle=0\.\(61\)Forμ≠ν\\mu\\neq\\nu,

Uμ⊤​E​Uν=0\.U\_\{\\mu\}^\{\\top\}EU\_\{\\nu\}=0\.

#### D\.5Well\-sampled regime: first\-order cancellation and residual coupling

We now apply Lemma[D\.1](https://arxiv.org/html/2610.31728#A4.Thmtheorem1)to the reference gradientG~\\widetilde\{G\}\.

##### Cancellation inside the contrast subspaces\.

Recall from equation[40](https://arxiv.org/html/2610.31728#A4.E40)that the singular value ofG~\\widetilde\{G\}onUμU\_\{\\mu\}is

σμ=aμ​rμ,rμ=πμC\.\\sigma\_\{\\mu\}=a\_\{\\mu\}r\_\{\\mu\},\\qquad r\_\{\\mu\}=\\frac\{\\pi\_\{\\mu\}\}\{C\}\.The matrixDμD\_\{\\mu\}in equation[54](https://arxiv.org/html/2610.31728#A4.E54)is symmetric\. Therefore

Uμ⊤​E​Uμ=\(Uμ⊤​E​Uμ\)⊤\.U\_\{\\mu\}^\{\\top\}EU\_\{\\mu\}=\\left\(U\_\{\\mu\}^\{\\top\}EU\_\{\\mu\}\\right\)^\{\\top\}\.By equation[50](https://arxiv.org/html/2610.31728#A4.E50),

Uμ⊤​𝒟G~​\[E\]​Uμ=0\.U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]U\_\{\\mu\}=0\.Forμ≠ν\\mu\\neq\\nu, both cross blocksUμ⊤​E​UνU\_\{\\mu\}^\{\\top\}EU\_\{\\nu\}andUν⊤​E​UμU\_\{\\nu\}^\{\\top\}EU\_\{\\mu\}vanish\. Hence

Uμ⊤​𝒟G~​\[E\]​Uν=0,∀μ,ν\.U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]U\_\{\\nu\}=0,\\qquad\\forall\\,\\mu,\\nu\.\(62\)Thus the direct first\-order effect of within\-block sampling fluctuations vanishes on every contrast\-to\-contrast block\.

##### Coupling to block\-average directions\.

Let

H=U^​diag⁡\(s1,…,sM\)​V^⊤H=\\widehat\{U\}\\operatorname\{diag\}\(s\_\{1\},\\ldots,s\_\{M\}\)\\widehat\{V\}^\{\\top\}be an SVD of the block\-average matrixH=L​RH=LR, with

s1,…,sM−1\>0,sM=0\.s\_\{1\},\\ldots,s\_\{M\-1\}\>0,\\qquad s\_\{M\}=0\.Writeu^k\\widehat\{u\}\_\{k\}andv^k\\widehat\{v\}\_\{k\}for the corresponding left and right singular vectors\.

Since

the singular\-vector identity

u^k⊤​H=sk​v^k⊤\\widehat\{u\}\_\{k\}^\{\\top\}H=s\_\{k\}\\widehat\{v\}\_\{k\}^\{\\top\}gives, for everykkandμ\\mu,

u^k⊤​L​eμ=skrμ​\(v^k\)μ\.\\widehat\{u\}\_\{k\}^\{\\top\}Le\_\{\\mu\}=\\frac\{s\_\{k\}\}\{r\_\{\\mu\}\}\(\\widehat\{v\}\_\{k\}\)\_\{\\mu\}\.\(63\)Forsk=0s\_\{k\}=0, this follows becauseRRis invertible andu^k⊤​L​R=0\\widehat\{u\}\_\{k\}^\{\\top\}LR=0\.

Using equations[59](https://arxiv.org/html/2610.31728#A4.E59)and[60](https://arxiv.org/html/2610.31728#A4.E60), and the derivative formulas in Lemma[D\.1](https://arxiv.org/html/2610.31728#A4.Thmtheorem1), we obtain

Uμ⊤𝒟G~\[E\]Yv^k=\(v^k\)μC​rμσμ−skσμ\+skzμ,k=1,…,M\.U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]Y\\widehat\{v\}\_\{k\}=\\frac\{\(\\widehat\{v\}\_\{k\}\)\_\{\\mu\}\}\{\\sqrt\{C\}\\,r\_\{\\mu\}\}\\frac\{\\sigma\_\{\\mu\}\-s\_\{k\}\}\{\\sigma\_\{\\mu\}\+s\_\{k\}\}z\_\{\\mu\},\\qquad k=1,\\ldots,M\.\(64\)Whensk=0s\_\{k\}=0, the fraction in equation[64](https://arxiv.org/html/2610.31728#A4.E64)equals one, and the formula agrees with the positive\-to\-null derivative in equation[51](https://arxiv.org/html/2610.31728#A4.E51)\.

Since

\|σμ−skσμ\+sk\|≤1\\left\|\\frac\{\\sigma\_\{\\mu\}\-s\_\{k\}\}\{\\sigma\_\{\\mu\}\+s\_\{k\}\}\\right\|\\leq 1andV^\\widehat\{V\}is orthogonal,

‖Uμ⊤​𝒟G~​\[E\]​Y‖F2\\displaystyle\\left\\\|U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]Y\\right\\\|\_\{F\}^\{2\}≤‖zμ‖22C​rμ2​∑k=1M\(v^k\)μ2\\displaystyle\\leq\\frac\{\\\|z\_\{\\mu\}\\\|\_\{2\}^\{2\}\}\{Cr\_\{\\mu\}^\{2\}\}\\sum\_\{k=1\}^\{M\}\(\\widehat\{v\}\_\{k\}\)\_\{\\mu\}^\{2\}=‖zμ‖22C​rμ2\.\\displaystyle=\\frac\{\\\|z\_\{\\mu\}\\\|\_\{2\}^\{2\}\}\{Cr\_\{\\mu\}^\{2\}\}\.\(65\)Conditioning onπμ\\pi\_\{\\mu\}and using equation[48](https://arxiv.org/html/2610.31728#A4.E48)together withrμ=πμ/Cr\_\{\\mu\}=\\pi\_\{\\mu\}/C,

𝔼⁡\[‖Uμ⊤​𝒟G~​\[E\]​Y‖F2\|πμ\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\left\\\|U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]Y\\right\\\|\_\{F\}^\{2\}\\biggm\|\\pi\_\{\\mu\}\\right\]≤1C​rμ2​\(C−1\)​πμB​C\\displaystyle\\leq\\frac\{1\}\{Cr\_\{\\mu\}^\{2\}\}\\frac\{\(C\-1\)\\pi\_\{\\mu\}\}\{BC\}=C−1B​πμ\.\\displaystyle=\\frac\{C\-1\}\{B\\pi\_\{\\mu\}\}\.\(66\)Therefore, conditionally onπμ\>0\\pi\_\{\\mu\}\>0,

‖Uμ⊤​𝒟G~​\[E\]​Y‖F=Oℙ​\(CB​πμ\)\.\\left\\\|U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]Y\\right\\\|\_\{F\}=O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{C\}\{B\\pi\_\{\\mu\}\}\}\\right\)\.\(67\)
The empirical block frequency satisfies

B​πμ∼Binom⁡\(B,p~μ\)\.B\\pi\_\{\\mu\}\\sim\\operatorname\{Binom\}\(B,\\tilde\{p\}\_\{\\mu\}\)\.Hence, whenever

B​p~μ→∞,B\\tilde\{p\}\_\{\\mu\}\\to\\infty,we have

πμp~μ→ℙ1\.\\frac\{\\pi\_\{\\mu\}\}\{\\tilde\{p\}\_\{\\mu\}\}\\xrightarrow\{\\mathbb\{P\}\}1\.It follows from equation[67](https://arxiv.org/html/2610.31728#A4.E67)that

‖Uμ⊤​𝒟G~​\[E\]​Y‖F=Oℙ​\(CB​p~μ\)\.\\left\\\|U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\widetilde\{G\}\}\[E\]Y\\right\\\|\_\{F\}=O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{C\}\{B\\tilde\{p\}\_\{\\mu\}\}\}\\right\)\.\(68\)Thus the first\-order contrast\-to\-average coupling vanishes when

B​p~μC→∞\.\\frac\{B\\tilde\{p\}\_\{\\mu\}\}\{C\}\\to\\infty\.\(69\)
For fixedMM, the sufficient uniform condition

B​p~minC→∞\\frac\{B\\tilde\{p\}\_\{\\min\}\}\{C\}\\to\\inftymakes the first\-order coupling small in every block simultaneously\. This condition should be interpreted as a large expected sample count per task\. It does not require every individual task to appear in every mini\-batch\.

#### D\.6Momentum: averaging across steps and tracking

We next analyze momentum as EMA averaging of the empirical frequency matrix\. This subsection also proves Proposition[5\.1](https://arxiv.org/html/2610.31728#S5.Thmtheorem1)\. The variance\-equivalent batch\-size notation is retained as a shorthand, but the underlying object is the exponentially weighted observation process derived explicitly below\. Let the matrix momentum obey

𝖬t=β​𝖬t−1\+\(1−β\)​Gt,Gt=\(I−St\)​Qt,\\mathsf\{M\}\_\{t\}=\\beta\\mathsf\{M\}\_\{t\-1\}\+\(1\-\\beta\)G\_\{t\},\\qquad G\_\{t\}=\(I\-S\_\{t\}\)Q\_\{t\},\(70\)with fresh independent mini\-batches and0≤β<10\\leq\\beta<1\. HereGtG\_\{t\}denotes the negative gradient in task coordinates, so the parameter update adds its normalized momentum\.

##### Taskwise stationary EMA and finite\-initialization correction\.

For a single taskjj, the empirical mini\-batch frequency satisfies

𝔼​qj,t=pj,Var⁡\(qj,t\)=pj​\(1−pj\)B,\\mathbb\{E\}q\_\{j,t\}=p\_\{j\},\\qquad\\operatorname\{Var\}\(q\_\{j,t\}\)=\\frac\{p\_\{j\}\(1\-p\_\{j\}\)\}\{B\},and the variables are independent across time\. For the stationary infinite\-past EMA

q¯j,t=\(1−β\)​∑r=0∞βr​qj,t−r,\\overline\{q\}\_\{j,t\}=\(1\-\\beta\)\\sum\_\{r=0\}^\{\\infty\}\\beta^\{r\}q\_\{j,t\-r\},independence gives

𝔼​q¯j,t\\displaystyle\\mathbb\{E\}\\overline\{q\}\_\{j,t\}=pj,\\displaystyle=p\_\{j\},Var⁡\(q¯j,t\)\\displaystyle\\operatorname\{Var\}\(\\overline\{q\}\_\{j,t\}\)=\(1−β\)2​∑r≥0β2​r​pj​\(1−pj\)B=1−β1\+β​pj​\(1−pj\)B\.\\displaystyle=\(1\-\\beta\)^\{2\}\\sum\_\{r\\geq 0\}\\beta^\{2r\}\\frac\{p\_\{j\}\(1\-p\_\{j\}\)\}\{B\}=\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{p\_\{j\}\(1\-p\_\{j\}\)\}\{B\}\.\(71\)Thus, with

Beff​\(β\)=B​1\+β1−β,B\_\{\\mathrm\{eff\}\}\(\\beta\)=B\\frac\{1\+\\beta\}\{1\-\\beta\},we have

Var⁡\(q¯j,t\)pj2=1−pjBeff​\(β\)​pj,\\frac\{\\operatorname\{Var\}\(\\overline\{q\}\_\{j,t\}\)\}\{p\_\{j\}^\{2\}\}=\\frac\{1\-p\_\{j\}\}\{B\_\{\\mathrm\{eff\}\}\(\\beta\)p\_\{j\}\},which proves the frozen\-state variance identity in Proposition[5\.1](https://arxiv.org/html/2610.31728#S5.Thmtheorem1)\.

For comparison with an implementation initialized at𝖬−1=0\\mathsf\{M\}\_\{\-1\}=0, define the bias\-corrected finite\-history frequency average

q^j,t:=1−β1−βt\+1​∑r=0tβr​qj,t−r\.\\widehat\{q\}\_\{j,t\}:=\\frac\{1\-\\beta\}\{1\-\\beta^\{t\+1\}\}\\sum\_\{r=0\}^\{t\}\\beta^\{r\}q\_\{j,t\-r\}\.Then𝔼​q^j,t=pj\\mathbb\{E\}\\widehat\{q\}\_\{j,t\}=p\_\{j\}and

Var⁡\(q^j,t\)pj2=1−pjB​pj​1−β1\+β​1\+βt\+11−βt\+1\.\\frac\{\\operatorname\{Var\}\(\\widehat\{q\}\_\{j,t\}\)\}\{p\_\{j\}^\{2\}\}=\\frac\{1\-p\_\{j\}\}\{Bp\_\{j\}\}\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{1\+\\beta^\{t\+1\}\}\{1\-\\beta^\{t\+1\}\}\.\(72\)The last factor tends to one ast→∞t\\to\\infty, recovering the stationaryBeffB\_\{\\mathrm\{eff\}\}\. Without bias correction, the zero\-initialized buffer is exactly the truncated stationary sum and has mean\(1−βt\+1\)​pj\(1\-\\beta^\{t\+1\}\)p\_\{j\}\.

For an evolving state, the stationary infinite\-past form of equation[70](https://arxiv.org/html/2610.31728#A4.E70)is

𝖬t=\(1−β\)​∑r≥0βr​\(I−St−r\)​Qt−r\.\\mathsf\{M\}\_\{t\}=\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}\(I\-S\_\{t\-r\}\)Q\_\{t\-r\}\.Adding and subtracting\(I−St\)​Qt−r\(I\-S\_\{t\}\)Q\_\{t\-r\}term by term gives the exact decomposition

𝖬t=\(I−St\)​Q¯t\+Rt,Rt=\(1−β\)​∑r≥0βr​\(St−St−r\)​Qt−r,\\mathsf\{M\}\_\{t\}=\(I\-S\_\{t\}\)\\overline\{Q\}\_\{t\}\+R\_\{t\},\\qquad R\_\{t\}=\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}\(S\_\{t\}\-S\_\{t\-r\}\)Q\_\{t\-r\},\(73\)whereQ¯t=\(1−β\)​∑r≥0βr​Qt−r\\overline\{Q\}\_\{t\}=\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}Q\_\{t\-r\}\. If‖St−St−r‖F≤LS​η​r\\\|S\_\{t\}\-S\_\{t\-r\}\\\|\_\{F\}\\leq L\_\{S\}\\eta rand‖Qt−r‖op≤1\\\|Q\_\{t\-r\}\\\|\_\{\\mathrm\{op\}\}\\leq 1, then

‖Rt‖F\\displaystyle\\\|R\_\{t\}\\\|\_\{F\}≤\(1−β\)​∑r≥0βr​‖St−St−r‖F​‖Qt−r‖op\\displaystyle\\leq\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}\\\|S\_\{t\}\-S\_\{t\-r\}\\\|\_\{F\}\\,\\\|Q\_\{t\-r\}\\\|\_\{\\mathrm\{op\}\}≤LS​η​\(1−β\)​∑r≥0r​βr=LS​η​β1−β,\\displaystyle\\leq L\_\{S\}\\eta\(1\-\\beta\)\\sum\_\{r\\geq 0\}r\\beta^\{r\}=L\_\{S\}\\eta\\frac\{\\beta\}\{1\-\\beta\},\(74\)proving the tracking part of Proposition[5\.1](https://arxiv.org/html/2610.31728#S5.Thmtheorem1)\. A zero\-initialized finite\-history buffer has the same decomposition with the sums truncated atr=tr=t; its bound is no larger, while bias correction multiplies both the frozen term and the remainder by\(1−βt\+1\)−1\(1\-\\beta^\{t\+1\}\)^\{\-1\}\.

##### Frozen\-state block representation\.

First freeze the state at a block\-symmetric matrixS~\\widetilde\{S\}, and let

A:=I−S~\.A:=I\-\\widetilde\{S\}\.After initialization transients,

𝖬t=\(1−β\)​∑k=0∞βk​A​Qt−k=A​Q¯t,\\mathsf\{M\}\_\{t\}=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}AQ\_\{t\-k\}=A\\overline\{Q\}\_\{t\},where

Q¯t:=\(1−β\)​∑k=0∞βk​Qt−k\.\\overline\{Q\}\_\{t\}:=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}Q\_\{t\-k\}\.\(75\)Thus, under a frozen state, momentum simply replaces the single\-batch frequency matrixQtQ\_\{t\}by its exponentially weighted average across steps\.

Decompose

Q¯t=Q¯t∥\+Q¯t⟂\\overline\{Q\}\_\{t\}=\\overline\{Q\}\_\{t\}^\{\\parallel\}\+\\overline\{Q\}\_\{t\}^\{\\perp\}exactly as before\. Define

π¯μ​\(t\):=\(1−β\)​∑k=0∞βk​πμ​\(t−k\),\\overline\{\\pi\}\_\{\\mu\}\(t\):=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}\\pi\_\{\\mu\}\(t\-k\),and letϵ¯μ​\(t\)\\overline\{\\epsilon\}\_\{\\mu\}\(t\)be the corresponding within\-block residual vector\.

The batch variables are independent across time\. Since

𝔼​\[ϵμ​\(t\)\]=0\\mathbb\{E\}\[\\epsilon\_\{\\mu\}\(t\)\]=0and, by averaging equation[46](https://arxiv.org/html/2610.31728#A4.E46)overπμ\\pi\_\{\\mu\},

Cov⁡\(ϵμ​\(t\)\)=p~μB​C​\(IC−1C​JC\),\\operatorname\{Cov\}\(\\epsilon\_\{\\mu\}\(t\)\)=\\frac\{\\tilde\{p\}\_\{\\mu\}\}\{BC\}\\left\(I\_\{C\}\-\\frac\{1\}\{C\}J\_\{C\}\\right\),we obtain

Cov⁡\(ϵ¯μ\)=1−β1\+β​p~μB​C​\(IC−1C​JC\)\.\\operatorname\{Cov\}\(\\overline\{\\epsilon\}\_\{\\mu\}\)=\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{\\tilde\{p\}\_\{\\mu\}\}\{BC\}\\left\(I\_\{C\}\-\\frac\{1\}\{C\}J\_\{C\}\\right\)\.\(76\)Similarly,

Var⁡\(π¯μ\)=1−β1\+β​p~μ​\(1−p~μ\)B\.\\operatorname\{Var\}\(\\overline\{\\pi\}\_\{\\mu\}\)=\\frac\{1\-\\beta\}\{1\+\\beta\}\\frac\{\\tilde\{p\}\_\{\\mu\}\(1\-\\tilde\{p\}\_\{\\mu\}\)\}\{B\}\.\(77\)
Define

Beff:=B​1\+β1−β\.B\_\{\\mathrm\{eff\}\}:=B\\frac\{1\+\\beta\}\{1\-\\beta\}\.\(78\)Then equations[76](https://arxiv.org/html/2610.31728#A4.E76)and[77](https://arxiv.org/html/2610.31728#A4.E77)are exactly the corresponding single\-batch*covariance*formulas withBBreplaced byBeffB\_\{\\mathrm\{eff\}\}\. This covariance identity is not a distributional identity:Q¯t\\overline\{Q\}\_\{t\}is a weighted average of multinomial batches rather than an empirical frequency vector fromBeffB\_\{\\mathrm\{eff\}\}independent samples\.

Let

z¯μ:=Xμ⊤​ϵ¯μ\.\\overline\{z\}\_\{\\mu\}:=X\_\{\\mu\}^\{\\top\}\\overline\{\\epsilon\}\_\{\\mu\}\.Thus equation[76](https://arxiv.org/html/2610.31728#A4.E76)gives

𝔼​‖z¯μ‖22=C−1C​p~μBeff\.\\mathbb\{E\}\\\|\\overline\{z\}\_\{\\mu\}\\\|\_\{2\}^\{2\}=\\frac\{C\-1\}\{C\}\\frac\{\\tilde\{p\}\_\{\\mu\}\}\{B\_\{\\mathrm\{eff\}\}\}\.\(79\)Moreover, if

Beff​p~μ→∞,B\_\{\\mathrm\{eff\}\}\\tilde\{p\}\_\{\\mu\}\\to\\infty,then

π¯μp~μ→ℙ1\\frac\{\\overline\{\\pi\}\_\{\\mu\}\}\{\\tilde\{p\}\_\{\\mu\}\}\\xrightarrow\{\\mathbb\{P\}\}1by equation[77](https://arxiv.org/html/2610.31728#A4.E77)\.

Combining the deterministic bound in equation[65](https://arxiv.org/html/2610.31728#A4.E65)with equation[79](https://arxiv.org/html/2610.31728#A4.E79)and the concentration ofπ¯μ\\overline\{\\pi\}\_\{\\mu\}therefore yields

‖Uμ⊤​𝒟G¯∥​\[E¯\]​Y‖F=Oℙ​\(CBeff​p~μ\),\\left\\\|U\_\{\\mu\}^\{\\top\}\\mathcal\{D\}\_\{\\overline\{G\}^\{\\parallel\}\}\[\\overline\{E\}\]Y\\right\\\|\_\{F\}=O\_\{\\mathbb\{P\}\}\\\!\\left\(\\sqrt\{\\frac\{C\}\{B\_\{\\mathrm\{eff\}\}\\tilde\{p\}\_\{\\mu\}\}\}\\right\),\(80\)where

G¯∥=A​Q¯∥,E¯=A​Q¯⟂\.\\overline\{G\}^\{\\parallel\}=A\\overline\{Q\}^\{\\parallel\},\\qquad\\overline\{E\}=A\\overline\{Q\}^\{\\perp\}\.
Hence, in probability at first order around the frozen symmetric state, momentum averaging suppresses the sampling distortion whenever

Beff​p~minC≫1\.\\frac\{B\_\{\\mathrm\{eff\}\}\\tilde\{p\}\_\{\\min\}\}\{C\}\\gg 1\.\(81\)Using equation[78](https://arxiv.org/html/2610.31728#A4.E78), this can also be written, up to the bounded factor1\+β1\+\\beta, as

B​p~min\(1−β\)​C≫1\.\\frac\{B\\tilde\{p\}\_\{\\min\}\}\{\(1\-\\beta\)C\}\\gg 1\.\(82\)

##### Tracking a slowly evolving state\.

We now return to the actual dynamics, whereStS\_\{t\}evolves\. The exact stationary\-form expansion of the momentum is

𝖬t=\(1−β\)​∑k=0∞βk​\(I−St−k\)​Qt−k\.\\mathsf\{M\}\_\{t\}=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}\(I\-S\_\{t\-k\}\)Q\_\{t\-k\}\.If the reference state is frozen at its current valueStS\_\{t\}, the corresponding approximation is

\(I−St\)​Q¯t\.\(I\-S\_\{t\}\)\\overline\{Q\}\_\{t\}\.Their difference is

𝖬t−\(I−St\)​Q¯t=\(1−β\)​∑k=0∞βk​\(St−St−k\)​Qt−k\.\\mathsf\{M\}\_\{t\}\-\(I\-S\_\{t\}\)\\overline\{Q\}\_\{t\}=\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}\\beta^\{k\}\(S\_\{t\}\-S\_\{t\-k\}\)Q\_\{t\-k\}\.\(83\)
In the late\-time regime, the block\-symmetric reference quantities evolve on the parameter timescale1/η1/\\eta\. Locally, if their relative change per iteration isO⁡\(η\)O\(\\eta\), then overkkiterations the mismatch isO⁡\(k​η\)O\(k\\eta\)\. Since

\(1−β\)​∑k=0∞k​βk=β1−β,\(1\-\\beta\)\\sum\_\{k=0\}^\{\\infty\}k\\beta^\{k\}=\\frac\{\\beta\}\{1\-\\beta\},equation[83](https://arxiv.org/html/2610.31728#A4.E83)gives the adiabatic tracking scale

O⁡\(η1−β\)\.O\\\!\\left\(\\frac\{\\eta\}\{1\-\\beta\}\\right\)\.\(84\)Under the stated localO⁡\(η\)O\(\\eta\)\-variation assumption, the frozen\-state mismatch is therefore small on the adiabatic scale

η≪1−β\.\\eta\\ll 1\-\\beta\.\(85\)This is a tracking estimate, not a global stability theorem for the stochastic trajectory\.

Thus the local momentum regime is described by two separate requirements:

B​p~min\(1−β\)​C≫1,\\frac\{B\\tilde\{p\}\_\{\\min\}\}\{\(1\-\\beta\)C\}\\gg 1,\(86\)for small relative sampling fluctuations under momentum, and

η1−β≪1,\\frac\{\\eta\}\{1\-\\beta\}\\ll 1,\(87\)for adiabatic tracking\. The first improves the frozen\-state sampling estimate relative to the no\-momentum conditionB​p~min/C≫1B\\tilde\{p\}\_\{\\min\}/C\\gg 1; the second prevents the averaging window from becoming so long that the underlying gradient changes substantially across it\.

#### D\.7Technical refinement: finite\-horizon well\-sampled continuum limit

The main text does not use convergence to the full\-batch continuum flow as the substantive higher\-rank claim; large expected sample counts making mini\-batch frequencies concentrate is expected\. We retain the following result because it formalizes how the local sampling analysis connects to the full\-batch trajectory on any fixed continuum\-time horizon\.

###### Proposition D\.2\(Finite\-horizon well\-sampled continuum limit\)\.

FixM,CM,Cand positive task frequencies\. LetW¯η,B​\(τ\)\\overline\{W\}^\{\\eta,B\}\(\\tau\)be the piecewise\-linear interpolation of the stochastic hierarchy on slow timeτ=η​t\\tau=\\eta t, and letW⋆W^\{\\star\}solve the full\-batch continuum flowW˙⋆=msgn⁡\(\(I−S⁡\(W⋆\)\)​P\)\\dot\{W\}^\{\\star\}=\\operatorname\{msgn\}\(\(I\-S\(W^\{\\star\}\)\)P\)fromW⋆​\(0\)=0W^\{\\star\}\(0\)=0\. For every finiteτ0\\tau\_\{0\}andε\>0\\varepsilon\>0,

limη↓0limB​pmin→∞ℙ⁡\(sup0≤τ≤τ0‖W¯η,B​\(τ\)−W⋆​\(τ\)‖F\>ε\)=0\.\\lim\_\{\\eta\\downarrow 0\}\\;\\lim\_\{Bp\_\{\\min\}\\to\\infty\}\\mathbb\{P\}\\\!\\left\(\\sup\_\{0\\leq\\tau\\leq\\tau\_\{0\}\}\\\|\\overline\{W\}^\{\\eta,B\}\(\\tau\)\-W^\{\\star\}\(\\tau\)\\\|\_\{F\}\>\\varepsilon\\right\)=0\.\(88\)The same holds along any joint sequenceηn↓0\\eta\_\{n\}\\downarrow 0,Bn​pmin→∞B\_\{n\}p\_\{\\min\}\\to\\inftysatisfyinglog⁡\(1/ηn\)=o⁡\(Bn​pmin\)\\log\(1/\\eta\_\{n\}\)=o\(B\_\{n\}p\_\{\\min\}\)\. Moreover, along the limiting flow,

Uμ⊤​W⋆​\(τ\)​Uν=δμ​ν​τ​IC−1,τ≥0\.U\_\{\\mu\}^\{\\top\}W^\{\\star\}\(\\tau\)U\_\{\\nu\}=\\delta\_\{\\mu\\nu\}\\,\\tau I\_\{C\-1\},\\qquad\\tau\\geq 0\.\(89\)

##### From large expected task count to the continuum trajectory\.

We now prove Proposition[D\.2](https://arxiv.org/html/2610.31728#A4.Thmtheorem2)\. Define

F⁡\(W\):=msgn⁡\(\(I−S⁡\(W\)\)​P\)\.F\(W\):=\\operatorname\{msgn\}\(\(I\-S\(W\)\)P\)\.For every finiteWW, the softmax matrixS⁡\(W\)S\(W\)has strictly positive entries and is column stochastic\. Perron–Frobenius therefore implies that its eigenvalue11is simple, soI−S⁡\(W\)I\-S\(W\), and hence\(I−S⁡\(W\)\)​P\(I\-S\(W\)\)P, has rankK−1K\-1\. The partial polar factor is smooth on the fixed\-rank manifold in a neighborhood of each such matrix\. Moreover

‖F⁡\(W\)‖F≤K−1,\\\|F\(W\)\\\|\_\{F\}\\leq\\sqrt\{K\-1\},so the ODE

W˙⋆​\(τ\)=F⁡\(W⋆​\(τ\)\),W⋆​\(0\)=0,\\dot\{W\}^\{\\star\}\(\\tau\)=F\(W^\{\\star\}\(\\tau\)\),\\qquad W^\{\\star\}\(0\)=0,has a unique global solution\. On every compact time interval\[0,τ0\]\[0,\\tau\_\{0\}\], the image ofW⋆W^\{\\star\}is compact and the smallest nonzero singular value of\(I−S⁡\(W⋆​\(τ\)\)\)​P\(I\-S\(W^\{\\star\}\(\\tau\)\)\)Phas a positive minimum\. ThusFFis uniformly Lipschitz on a sufficiently small tube around that trajectory\.

The ODE preserves permutations within every frequency block becausePPand the initialization share those symmetries and the polar map is orthogonally equivariant\. Therefore the block decomposition of Appendix[D\.3](https://arxiv.org/html/2610.31728#A4.SS3)applies for everyτ\\tau\. In particular,

Uμ⊤​W˙⋆​\(τ\)​Uν=δμ​ν​IC−1\.U\_\{\\mu\}^\{\\top\}\\dot\{W\}^\{\\star\}\(\\tau\)U\_\{\\nu\}=\\delta\_\{\\mu\\nu\}I\_\{C\-1\}\.\(90\)Integrating fromW⋆​\(0\)=0W^\{\\star\}\(0\)=0proves equation[89](https://arxiv.org/html/2610.31728#A4.E89)for everyτ≥0\\tau\\geq 0\.

We first prove the sequential limit in equation[88](https://arxiv.org/html/2610.31728#A4.E88)\. Fixη\>0\\eta\>0andτ0<∞\\tau\_\{0\}<\\infty\. Then onlyN=⌈τ0/η⌉N=\\lceil\\tau\_\{0\}/\\eta\\rceilmini\-batches are involved\. AsB​pmin→∞Bp\_\{\\min\}\\to\\infty, multinomial concentration and a union bound over these finitely many steps give

maxt<N⁡‖Qt​P−1−I‖op→ℙ0\.\\max\_\{t<N\}\\\|Q\_\{t\}P^\{\-1\}\-I\\\|\_\{\\mathrm\{op\}\}\\xrightarrow\{\\mathbb\{P\}\}0\.Uniform fixed\-rank smoothness of the polar map on a tube around the corresponding full\-batch Euler trajectory then implies that the stochastic recursion converges uniformly over theseNNsteps to

Vt\+1=Vt\+η​F​\(Vt\),V0=0\.V\_\{t\+1\}=V\_\{t\}\+\\eta F\(V\_\{t\}\),\\qquad V\_\{0\}=0\.Finally, sendingη↓0\\eta\\downarrow 0, the piecewise\-linear Euler interpolation converges uniformly on\[0,τ0\]\[0,\\tau\_\{0\}\]toW⋆W^\{\\star\}\. This proves the iterated well\-sampled/small\-step limit without any coupling betweenBBandη\\eta\.

For completeness, we now prove the convenient joint sufficient scaling stated in the proposition\. Let

Nn:=⌈τ0ηn⌉,κn=Bn​pmin\.N\_\{n\}:=\\left\\lceil\\frac\{\\tau\_\{0\}\}\{\\eta\_\{n\}\}\\right\\rceil,\\qquad\\kappa\_\{n\}=B\_\{n\}p\_\{\\min\}\.For a single mini\-batch, multiplicative Chernoff bounds give, for0<δ<10<\\delta<1,

ℙ⁡\(maxj⁡\|qj−pj\|pj\>δ\)≤2​K​exp⁡\(−c​δ2​κn\)\\mathbb\{P\}\\\!\\left\(\\max\_\{j\}\\frac\{\|q\_\{j\}\-p\_\{j\}\|\}\{p\_\{j\}\}\>\\delta\\right\)\\leq 2K\\exp\(\-c\\delta^\{2\}\\kappa\_\{n\}\)\(91\)for a universal constantc\>0c\>0\. Set

δn:=\(log⁡\(2​K​Nn\)κn\)1/4\.\\delta\_\{n\}:=\\left\(\\frac\{\\log\(2KN\_\{n\}\)\}\{\\kappa\_\{n\}\}\\right\)^\{1/4\}\.The assumptionlog⁡\(1/ηn\)=o⁡\(κn\)\\log\(1/\\eta\_\{n\}\)=o\(\\kappa\_\{n\}\)impliesδn→0\\delta\_\{n\}\\to 0and

Nn​2​K​exp⁡\(−c​δn2​κn\)→0\.N\_\{n\}\\,2K\\exp\(\-c\\delta\_\{n\}^\{2\}\\kappa\_\{n\}\)\\to 0\.Hence, with probability tending to one,

max0≤t<Nn⁡‖Qt​P−1−I‖op≤δn\.\\max\_\{0\\leq t<N\_\{n\}\}\\\|Q\_\{t\}P^\{\-1\}\-I\\\|\_\{\\mathrm\{op\}\}\\leq\\delta\_\{n\}\.\(92\)In particular all task counts are positive on this event for all sufficiently largenn, so every sampled gradient remains on the same rank\-\(K−1\)\(K\-1\)manifold as the full\-batch gradient\.

LetVt\+1=Vt\+ηn​F​\(Vt\)V\_\{t\+1\}=V\_\{t\}\+\\eta\_\{n\}F\(V\_\{t\}\),V0=0V\_\{0\}=0, denote the deterministic Euler discretization of the continuum flow\. Uniform fixed\-rank smoothness on the tube aroundW⋆​\(\[0,τ0\]\)W^\{\\star\}\(\[0,\\tau\_\{0\}\]\)gives constantsLdyn,asamp<∞L\_\{\\mathrm\{dyn\}\},a\_\{\\mathrm\{samp\}\}<\\inftysuch that, on the event in equation[92](https://arxiv.org/html/2610.31728#A4.E92),

‖msgn⁡\(\(I−S⁡\(W\)\)​Qt\)−F⁡\(W\)‖F≤asamp​δn,\\displaystyle\\\|\\operatorname\{msgn\}\(\(I\-S\(W\)\)Q\_\{t\}\)\-F\(W\)\\\|\_\{F\}\\leq a\_\{\\mathrm\{samp\}\}\\delta\_\{n\},\(93\)‖F⁡\(W\)−F⁡\(V\)‖F≤Ldyn​‖W−V‖F\.\\displaystyle\\\|F\(W\)\-F\(V\)\\\|\_\{F\}\\leq L\_\{\\mathrm\{dyn\}\}\\\|W\-V\\\|\_\{F\}\.A standard bootstrap then keeps both discrete trajectories inside the same tube and yields

maxt≤Nn⁡‖W¯t−Vt‖F≤asamp​τ0​eLdyn​τ0​δn=oℙ​\(1\)\.\\max\_\{t\\leq N\_\{n\}\}\\\|\\overline\{W\}\_\{t\}\-V\_\{t\}\\\|\_\{F\}\\leq a\_\{\\mathrm\{samp\}\}\\tau\_\{0\}e^\{L\_\{\\mathrm\{dyn\}\}\\tau\_\{0\}\}\\delta\_\{n\}=o\_\{\\mathbb\{P\}\}\(1\)\.BecauseFFis Lipschitz on the same compact tube, the usual Euler estimate gives

sup0≤τ≤τ0‖Vηn​\(τ\)−W⋆​\(τ\)‖F=O⁡\(ηn\),\\sup\_\{0\\leq\\tau\\leq\\tau\_\{0\}\}\\\|V^\{\\eta\_\{n\}\}\(\\tau\)\-W^\{\\star\}\(\\tau\)\\\|\_\{F\}=O\(\\eta\_\{n\}\),whereVηnV^\{\\eta\_\{n\}\}denotes the piecewise\-linear interpolation\. Combining the two estimates proves equation[88](https://arxiv.org/html/2610.31728#A4.E88)\.

Finally, equation[89](https://arxiv.org/html/2610.31728#A4.E89)implies for every fixedτ\>0\\tau\>0,

1τ​Uμ⊤​W¯ηn,Bn​\(τ\)​Uμ→ℙIC−1\.\\frac\{1\}\{\\tau\}U\_\{\\mu\}^\{\\top\}\\overline\{W\}^\{\\eta\_\{n\},B\_\{n\}\}\(\\tau\)U\_\{\\mu\}\\xrightarrow\{\\mathbb\{P\}\}I\_\{C\-1\}\.Hence also

limτ→∞limn→∞1τ​Uμ⊤​W¯ηn,Bn​\(τ\)​Uμ=IC−1in probability\.\\lim\_\{\\tau\\to\\infty\}\\;\\lim\_\{n\\to\\infty\}\\frac\{1\}\{\\tau\}U\_\{\\mu\}^\{\\top\}\\overline\{W\}^\{\\eta\_\{n\},B\_\{n\}\}\(\\tau\)U\_\{\\mu\}=I\_\{C\-1\}\\quad\\text\{in probability\}\.\(94\)This is an iterated long\-time statement rather than a claim of uniform stochastic approximation on the entire half\-line: the nonzero singular gap of the gradient may shrink as training proceeds\. The important distinction from the previous finite\-step bound is that the number of discrete optimizer steps already diverges as1/ηn1/\\eta\_\{n\}on every fixed continuum interval, while the limiting dynamics themselves are global inτ\\tauand have an exact asymptotic contrast rate\.

### Appendix EStochastic Labels: Exact Statistics, Projected Regimes, and Local Closure

We now derive the stochastic\-gradient structure under symmetric label noise, assuming0<α<10<\\alpha<1\.

##### Roadmap and status of the results\.

Appendices E\.1–E\.3 concern the complete categorical model and are exact: they give the gradient decomposition, covariance/reliability formulas, and the fixed\-dimensional local Gaussian limit\. Appendix E\.4 introduces the longitudinal target\-versus\-average\-nontarget projection; the counting formulas are exact*within that reduced model*and describe the occurrence\-limited and corruption\-sparse regimes\. Appendices E\.5–E\.6 add a moment\-matched isotropic covariance closure \(and a local frequency\-homogeneous sector\) to obtain the explicitp\\sqrt\{p\}coefficient in the corruption\-resolved local regime\. The exact categorical covariance is anisotropic, so the closure result should not be read as an exact formula for the full matrix process\.

Define

cα:=1−α\+αK,ρα:=1−cα=α​K−1K\.c\_\{\\alpha\}:=1\-\\alpha\+\\frac\{\\alpha\}\{K\},\\qquad\\rho\_\{\\alpha\}:=1\-c\_\{\\alpha\}=\\alpha\\frac\{K\-1\}\{K\}\.Thuscαc\_\{\\alpha\}is the probability that the observed label equals the target label andρα\\rho\_\{\\alpha\}is the probability that the observed label differs from the target label\. The transition matrix is

Dα=\(1−α\)​IK\+αK​𝟏𝟏⊤\.D\_\{\\alpha\}=\(1\-\\alpha\)I\_\{K\}\+\\frac\{\\alpha\}\{K\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\.

#### E\.1Complete categorical model: gradient dynamics

LetJJdenote the sampled task andYYthe observed label\. We have

ℙ⁡\(J=j\)=pj,ℙ⁡\(Y=i∣J=j\)=pi\|j,\\mathbb\{P\}\(J=j\)=p\_\{j\},\\qquad\\mathbb\{P\}\(Y=i\\mid J=j\)=p\_\{i\|j\},with

pi\|j=\{cα,i=j,α/K,i≠j\.p\_\{i\|j\}=\\begin\{cases\}c\_\{\\alpha\},&i=j,\\\\ \\alpha/K,&i\\neq j\.\\end\{cases\}The population loss under label noise is

ℒ\(W\)=−∑j=1Kpj∑i=1Kpi\|jlogSi​j\(W\)\.\\mathcal\{L\}\(W\)=\-\\sum\_\{j=1\}^\{K\}p\_\{j\}\\sum\_\{i=1\}^\{K\}p\_\{i\|j\}\\log S\_\{ij\}\(W\)\.In task coordinates, differentiation gives

∇ℒ​\(W\)=−E~​\(Dα−S\)​P​E⊤,P:=diag⁡\(p1,…,pK\)\.\\nabla\\mathcal\{L\}\(W\)=\-\\widetilde\{E\}\(D\_\{\\alpha\}\-S\)PE^\{\\top\},\\qquad P:=\\operatorname\{diag\}\(p\_\{1\},\\ldots,p\_\{K\}\)\.\(95\)
For a mini\-batch of sizeBB, letNi​jN\_\{ij\}be the number of samples with taskjjand observed labelii, and define the empirical joint frequency matrix

T^i​j:=Ni​jB\.\\widehat\{T\}\_\{ij\}:=\\frac\{N\_\{ij\}\}\{B\}\.The empirical task frequencies are

qj:=∑iT^i​j,Q:=diag⁡\(q1,…,qK\)\.q\_\{j\}:=\\sum\_\{i\}\\widehat\{T\}\_\{ij\},\\qquad Q:=\\operatorname\{diag\}\(q\_\{1\},\\ldots,q\_\{K\}\)\.The negative stochastic gradient in task coordinates is therefore

T^−S​Q\.\\widehat\{T\}\-SQ\.\(96\)Ifqj\>0q\_\{j\}\>0, one may equivalently writeT^⋅j=qj​R⋅j\\widehat\{T\}\_\{\\cdot j\}=q\_\{j\}R\_\{\\cdot j\}, whereR⋅jR\_\{\\cdot j\}is the empirical conditional label distribution\. Whenqj=0q\_\{j\}=0, the column ofRRis immaterial because onlyR​Q=T^RQ=\\widehat\{T\}enters the gradient\.

A useful decomposition is

T^−S​Q=\(Dα−S\)​P\+L\+T,\\widehat\{T\}\-SQ=\(D\_\{\\alpha\}\-S\)P\+L\+T,\(97\)where

L\\displaystyle L:=T^−Dα​Q,\\displaystyle:=\\widehat\{T\}\-D\_\{\\alpha\}Q,\(98\)T\\displaystyle T:=\(Dα−S\)​\(Q−P\)\.\\displaystyle:=\(D\_\{\\alpha\}\-S\)\(Q\-P\)\.\(99\)HereLLis the conditional label\-sampling fluctuation andTTis the task\-frequency fluctuation\. At the noisy population optimum

the task\-frequency termTTvanishes exactly, so the persistent local stochasticity comes fromLL\.

#### E\.2Complete categorical model: exact covariance and reliability

From this point through the local\-response analysis we work in task coordinates; the orthogonal embedding transforms preserve all covariance, norm, and polar\-equivariance statements used below\.

Let

dj:=Dα​ej,δj:=S​ej−dj,Γj:=diag⁡\(dj\)−dj​dj⊤\.d\_\{j\}:=D\_\{\\alpha\}e\_\{j\},\\qquad\\delta\_\{j\}:=Se\_\{j\}\-d\_\{j\},\\qquad\\Gamma\_\{j\}:=\\operatorname\{diag\}\(d\_\{j\}\)\-d\_\{j\}d\_\{j\}^\{\\top\}\.For one observation\(J,Y\)\(J,Y\), define the task\-jjpositive\-gradient contribution

Xj:=𝟏\{J=j\}\(Sej−eY\)\.X\_\{j\}:=\\mathbf\{1\}\\\{J=j\\\}\(Se\_\{j\}\-e\_\{Y\}\)\.Then

𝔼​Xj=pj​δj,𝔼⁡\[Xj​Xj⊤\]=pj​\(Γj\+δj​δj⊤\)\.\\mathbb\{E\}X\_\{j\}=p\_\{j\}\\delta\_\{j\},\\qquad\\mathbb\{E\}\[X\_\{j\}X\_\{j\}^\{\\top\}\]=p\_\{j\}\(\\Gamma\_\{j\}\+\\delta\_\{j\}\\delta\_\{j\}^\{\\top\}\)\.AveragingBBindependent observations gives

Cov⁡\(g^j\)=1B​\[pj​Γj\+pj​\(1−pj\)​δj​δj⊤\],\\operatorname\{Cov\}\(\\widehat\{g\}\_\{j\}\)=\\frac\{1\}\{B\}\\left\[p\_\{j\}\\Gamma\_\{j\}\+p\_\{j\}\(1\-p\_\{j\}\)\\delta\_\{j\}\\delta\_\{j\}^\{\\top\}\\right\],\(100\)and forj≠kj\\neq k,

Cov⁡\(g^j,g^k\)=−pj​pkB​δj​δk⊤\.\\operatorname\{Cov\}\(\\widehat\{g\}\_\{j\},\\widehat\{g\}\_\{k\}\)=\-\\frac\{p\_\{j\}p\_\{k\}\}\{B\}\\delta\_\{j\}\\delta\_\{k\}^\{\\top\}\.\(101\)Therefore

SNRj2=B​pj​‖δj‖22tr⁡Γj\+\(1−pj\)​‖δj‖22\.\\mathrm\{SNR\}\_\{j\}^\{2\}=\\frac\{Bp\_\{j\}\\\|\\delta\_\{j\}\\\|\_\{2\}^\{2\}\}\{\\operatorname\{tr\}\\Gamma\_\{j\}\+\(1\-p\_\{j\}\)\\\|\\delta\_\{j\}\\\|\_\{2\}^\{2\}\}\.\(102\)AtS∗=DαS^\{\*\}=D\_\{\\alpha\}, allδj=0\\delta\_\{j\}=0\. WritingG^:=S​Q−T^\\widehat\{G\}:=SQ\-\\widehat\{T\}for the positive stochastic gradient, the population Hessian is block diagonal across task columns with blockspj​Γjp\_\{j\}\\Gamma\_\{j\}, so

Cov⁡\(vec⁡G^​\(W∗\)\)=1B​H∗\\operatorname\{Cov\}\\\!\\left\(\\operatorname\{vec\}\\widehat\{G\}\(W^\{\*\}\)\\right\)=\\frac\{1\}\{B\}H^\{\*\}\(103\)on the columnwise zero\-sum subspace\. This likelihood identity motivates a local structured\-noise viewpoint, but it does not imply that applying a polar map implements\(H∗\)−1/2\(H^\{\*\}\)^\{\-1/2\}\.

We now isolate the conditional label\-sampling term\. Letℓj:=L​ej\\ell\_\{j\}:=Le\_\{j\}\. From equation[98](https://arxiv.org/html/2610.31728#A5.E98),

ℓj=1B∑b=1B𝟏\{Jb=j\}\(eYb−dj\)\.\\ell\_\{j\}=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\mathbf\{1\}\\\{J\_\{b\}=j\\\}\\bigl\(e\_\{Y\_\{b\}\}\-d\_\{j\}\\bigr\)\.Hence

𝔼⁡\[ℓj\]=0\.\\mathbb\{E\}\[\\ell\_\{j\}\]=0\.Different task columns are uncorrelated, and

Cov⁡\(ℓj\)=pjB​Γj\.\\operatorname\{Cov\}\(\\ell\_\{j\}\)=\\frac\{p\_\{j\}\}\{B\}\\Gamma\_\{j\}\.\(104\)In particular,

Var⁡\(Lj​j\)\\displaystyle\\operatorname\{Var\}\(L\_\{jj\}\)=pjB​cα​ρα,\\displaystyle=\\frac\{p\_\{j\}\}\{B\}c\_\{\\alpha\}\\rho\_\{\\alpha\},\(105\)Var⁡\(Li​j\)\\displaystyle\\operatorname\{Var\}\(L\_\{ij\}\)=pjB​αK​\(1−αK\),i≠j,\\displaystyle=\\frac\{p\_\{j\}\}\{B\}\\frac\{\\alpha\}\{K\}\\left\(1\-\\frac\{\\alpha\}\{K\}\\right\),\\qquad i\\neq j,\(106\)and, fori≠ki\\neq k,

Cov⁡\(Li​j,Lk​j\)=−pjB​pi\|j​pk\|j\.\\operatorname\{Cov\}\(L\_\{ij\},L\_\{kj\}\)=\-\\frac\{p\_\{j\}\}\{B\}p\_\{i\|j\}p\_\{k\|j\}\.
The geometry ofΓj\\Gamma\_\{j\}is especially simple on the zero\-sum subspace\. Let

P0:=IK−1K​𝟏𝟏⊤\.P\_\{0\}:=I\_\{K\}\-\\frac\{1\}\{K\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\.A direct calculation gives

Γj=αK​P0\+α⁡\(1−α\)​\(P0​ej\)​\(P0​ej\)⊤\.\\Gamma\_\{j\}=\\frac\{\\alpha\}\{K\}P\_\{0\}\+\\alpha\(1\-\\alpha\)\(P\_\{0\}e\_\{j\}\)\(P\_\{0\}e\_\{j\}\)^\{\\top\}\.\(107\)Define the correct\-versus\-average\-incorrect contrast

vj:=ej−1K−1​∑i≠jei\.v\_\{j\}:=e\_\{j\}\-\\frac\{1\}\{K\-1\}\\sum\_\{i\\neq j\}e\_\{i\}\.The covariance eigenvalue along the normalized version ofvjv\_\{j\}is

λ∥=α​cα,\\lambda\_\{\\parallel\}=\\alpha c\_\{\\alpha\},whereas the otherK−2K\-2zero\-sum directions have eigenvalue

λ⟂=αK\.\\lambda\_\{\\perp\}=\\frac\{\\alpha\}\{K\}\.Equivalently, ifζj\\zeta\_\{j\}is the coefficient ofℓj\\ell\_\{j\}along the unnormalized vectorvjv\_\{j\}, then

Var⁡\(ζj\)=pjB​cα​ρα\.\\operatorname\{Var\}\(\\zeta\_\{j\}\)=\\frac\{p\_\{j\}\}\{B\}c\_\{\\alpha\}\\rho\_\{\\alpha\}\.\(108\)
These formulas identify the two basic expected\-count scales\. A batch containsB​pjBp\_\{j\}task\-jjexamples on average and

B​pj​ρα=B​pj​α​K−1KBp\_\{j\}\\rho\_\{\\alpha\}=Bp\_\{j\}\\alpha\\frac\{K\-1\}\{K\}incorrectly labeled task\-jjexamples on average\. The latter scale resolves the aggregate correct\-versus\-incorrect fluctuation\. If one instead requires each individual incorrect class to be well sampled, the stronger scaleB​pj​α/KBp\_\{j\}\\alpha/Kis relevant\.

#### E\.3Complete categorical model: local Gaussian limit and structured response

The covariance calculation above determines the complete local sampling limit before any isotropic closure\. The complete categorical model admits a fixed\-dimensional limit that retains every zero\-sum fluctuation mode; this separates exact local sampling behavior from the auxiliary closures introduced later only to obtain explicit scalar rates\. Let

𝒮=\{A∈ℝK×K:𝟏⊤​A=0\}\\mathcal\{S\}=\\\{A\\in\\mathbb\{R\}^\{K\\times K\}:\\mathbf\{1\}^\{\\top\}A=0\\\}with the Frobenius inner product\. Choose the zero\-column\-mean representativeW∗W^\{\*\}of a population minimizer and let the Hessian operator satisfy

ℋ∗​\[A\]​ej=pj​Γj​A​ej\.\\mathcal\{H\}^\{\*\}\[A\]e\_\{j\}=p\_\{j\}\\Gamma\_\{j\}Ae\_\{j\}\.
###### Theorem E\.1\(Local sampling limit with complete categorical noise\)\.

FixKK,pj\>0p\_\{j\}\>0, and0<α<10<\\alpha<1\. For deterministicA∈𝒮A\\in\\mathcal\{S\}, letWB=W∗\+A/BW\_\{B\}=W^\{\*\}\+A/\\sqrt\{B\}\. Then the positive mini\-batch gradient obeys

B​G^B​\(WB\)→𝑑ℋ∗​\[A\]\+Z,\\sqrt\{B\}\\,\\widehat\{G\}\_\{B\}\(W\_\{B\}\)\\xrightarrow\{d\}\\mathcal\{H\}^\{\*\}\[A\]\+Z,\(109\)whereZ∈𝒮Z\\in\\mathcal\{S\}is centered Gaussian with vectorized covarianceℋ∗\\mathcal\{H\}^\{\*\}\. Moreover,

limB→∞𝔼​msgn⁡\(G^B​\(WB\)\)=𝔼​msgn⁡\(ℋ∗​\[A\]\+Z\)\.\\lim\_\{B\\to\\infty\}\\mathbb\{E\}\\,\\operatorname\{msgn\}\(\\widehat\{G\}\_\{B\}\(W\_\{B\}\)\)=\\mathbb\{E\}\\,\\operatorname\{msgn\}\(\\mathcal\{H\}^\{\*\}\[A\]\+Z\)\.\(110\)The limit keepsK,p,αK,p,\\alphafixed and is not uniform over vanishing task frequencies or vanishing label noise\.

###### Proof\.

Smoothness of the finite\-dimensional softmax loss givesB∇ℒ\(WB\)=ℋ∗\[A\]\+o\(1\)\\sqrt\{B\}\\,\\nabla\\mathcal\{L\}\(W\_\{B\}\)=\\mathcal\{H\}^\{\*\}\[A\]\+o\(1\)\. Per\-sample gradients are uniformly bounded nearW∗W^\{\*\}and their covariance converges toℋ∗\\mathcal\{H\}^\{\*\}, so the triangular\-array central limit theorem gives equation[109](https://arxiv.org/html/2610.31728#A5.E109)\. Choose an orthonormal basisX∈ℝK×\(K−1\)X\\in\\mathbb\{R\}^\{K\\times\(K\-1\)\}for the zero\-sum subspace\. In these coordinates the Gaussian covariance is positive definite, soX⊤​\(ℋ∗​\[A\]\+Z\)X^\{\\top\}\(\\mathcal\{H\}^\{\*\}\[A\]\+Z\)has full row rank almost surely; equivalently,ℋ∗​\[A\]\+Z\\mathcal\{H\}^\{\*\}\[A\]\+Zhas rankK−1K\-1almost surely\. Every stochastic gradient also lies in𝒮\\mathcal\{S\}\. Writing anyG∈𝒮G\\in\\mathcal\{S\}asG=X​BG=XBgivesmsgn⁡\(G\)=X​msgn⁡\(B\)\\operatorname\{msgn\}\(G\)=X\\operatorname\{msgn\}\(B\); the rectangular partial polar map is continuous on the open set of full\-row\-rank\(K−1\)×K\(K\-1\)\\times Kmatrices\. Hence the square partial polar map is continuous at every rank\-\(K−1\)\(K\-1\)point relative to𝒮\\mathcal\{S\}, and the continuous mapping theorem applies on this support\. Since every entry of a partial polar factor is bounded by one, convergence in distribution upgrades to convergence of expectations, and positive scale invariance removes the factorB\\sqrt\{B\}\. ∎

The theorem isolates the remaining question: how does the polar map respond to an anisotropic Gaussian law? In orthonormal coordinates on𝒮\\mathcal\{S\}, letZZbe centered nondegenerate Gaussian with covarianceΣ\\Sigmaand define

FΣ​\(A\)=𝔼​msgn⁡\(Z\+A\)\.F\_\{\\Sigma\}\(A\)=\\mathbb\{E\}\\,\\operatorname\{msgn\}\(Z\+A\)\.Gaussian density differentiation gives the exact linear\-response operator

D​FΣ​\(0\)​\[A\]=𝔼⁡\[msgn⁡\(Z\)​⟨Σ−1​vec𝒮​A,vec𝒮⁡Z⟩\]\.DF\_\{\\Sigma\}\(0\)\[A\]=\\mathbb\{E\}\\\!\\left\[\\operatorname\{msgn\}\(Z\)\\,\\bigl\\langle\\Sigma^\{\-1\}\\operatorname\{vec\}\_\{\\mathcal\{S\}\}A,\\operatorname\{vec\}\_\{\\mathcal\{S\}\}Z\\bigr\\rangle\\right\]\.\(111\)The inverse is taken only on the zero\-sum support, not on softmax gauge directions\. The same Gaussian\-smoothing argument as Appendix[A\.1](https://arxiv.org/html/2610.31728#A1.SS1)implies a cubic fixed\-dimensional remainder after this linear term\. Unlike the fully isotropic setting, however, no left–right orthogonal symmetry forces equation[111](https://arxiv.org/html/2610.31728#A5.E111)to be a scalar multiple ofAA\. Anisotropy is therefore necessary for nontrivial directional response in this local Gaussian picture, but is not by itself sufficient to guarantee improved conditioning\.

The exact Gaussian limit above retains the full anisotropic response but does not simplify to a scalar closed form\. The following subsections therefore study complementary reductions: first a longitudinal counting model that is exact within its projection, and then an isotropic covariance closure that preserves total label\-noise energy while sacrificing anisotropy\.

#### E\.4Projected counting model: task and corruption counts

We quantify the occurrence\-limited and corruption\-sparse regimes in a projected target\-versus\-average\-nontarget model\. The projection retains the longitudinal component alongvjv\_\{j\}and removes the transverse categorical fluctuations in equation[107](https://arxiv.org/html/2610.31728#A5.E107)\. Within this reduced model, the counting argument below is exact; the Gaussian approximation enters only later\.

Consider a symmetry\-equivalent task direction with frequencyppand marginΔ\\Delta\. Write

u⁡\(Δ\):=1−Sj​j=K−1eΔ\+K−1\.u\(\\Delta\):=1\-S\_\{jj\}=\\frac\{K\-1\}\{e^\{\\Delta\}\+K\-1\}\.LetNNbe the number of samples of this task in a batch and letMMbe the number of those samples whose observed label is incorrect\. For each of theBBdraws, the three possibilities have probabilities

p​cα,p​ρα,1−p,pc\_\{\\alpha\},\\qquad p\\rho\_\{\\alpha\},\\qquad 1\-p,corresponding to a correctly labeled occurrence of the task, an incorrectly labeled occurrence, and another task, respectively\. Hence

ℙ⁡\(N=n,M=m\)=B\!\(n−m\)\!​m\!​\(B−n\)\!​\(p​cα\)n−m​\(p​ρα\)m​\(1−p\)B−n,\\mathbb\{P\}\(N=n,M=m\)=\\frac\{B\!\}\{\(n\-m\)\!\\,m\!\\,\(B\-n\)\!\}\(pc\_\{\\alpha\}\)^\{n\-m\}\(p\\rho\_\{\\alpha\}\)^\{m\}\(1\-p\)^\{B\-n\},\(112\)for0≤m≤n≤B0\\leq m\\leq n\\leq B\.

The correct\-coordinate residual in the stochastic gradient column is

T^j​j−Sj​j​qj=N−M−N​Sj​jB\.\\widehat\{T\}\_\{jj\}\-S\_\{jj\}q\_\{j\}=\\frac\{N\-M\-NS\_\{jj\}\}\{B\}\.Since the residual column has zero sum, this is also its coefficient along the unnormalized contrastvjv\_\{j\}\. Therefore the longitudinal random coefficient is

A⁡\(Δ\):=N​u​\(Δ\)−MB\.A\(\\Delta\):=\\frac\{Nu\(\\Delta\)\-M\}\{B\}\.\(113\)The corresponding scalar sign response is

mB,p​\(Δ\):=𝔼⁡\[sgn⁡\(A⁡\(Δ\)\)\]=∑n=0B∑m=0nsgn⁡\(n​u​\(Δ\)−m\)​ℙ​\(N=n,M=m\),m\_\{B,p\}\(\\Delta\):=\\mathbb\{E\}\\\!\\left\[\\operatorname\{sgn\}\(A\(\\Delta\)\)\\right\]=\\sum\_\{n=0\}^\{B\}\\sum\_\{m=0\}^\{n\}\\operatorname\{sgn\}\\\!\\left\(nu\(\\Delta\)\-m\\right\)\\mathbb\{P\}\(N=n,M=m\),\(114\)with the conventionsgn⁡\(0\)=0\\operatorname\{sgn\}\(0\)=0\.

To connect this scalar response to the projected Muon dynamics, considerssexchangeable directions with coefficientsA1,…,AsA\_\{1\},\\ldots,A\_\{s\}, and let

X∈ℝs×\(s−1\)X\\in\\mathbb\{R\}^\{s\\times\(s\-1\)\}have orthonormal columns spanning the zero\-sum subspace\. Define

𝒜s​\(Δ\):=X⊤​diag⁡\(A1​\(Δ\),…,As​\(Δ\)\)​X\.\\mathcal\{A\}\_\{s\}\(\\Delta\):=X^\{\\top\}\\operatorname\{diag\}\(A\_\{1\}\(\\Delta\),\\ldots,A\_\{s\}\(\\Delta\)\)X\.Permutation symmetry implies

𝔼⁡\[msgn⁡\(𝒜s​\(Δ\)\)\]=rs​\(Δ\)​Is−1\\mathbb\{E\}\\\!\\left\[\\operatorname\{msgn\}\(\\mathcal\{A\}\_\{s\}\(\\Delta\)\)\\right\]=r\_\{s\}\(\\Delta\)I\_\{s\-1\}\(115\)for a scalarrs​\(Δ\)r\_\{s\}\(\\Delta\)\. By Cauchy interlacing between the eigenvalues of𝒜s\\mathcal\{A\}\_\{s\}and the diagonal entriesA1,…,AsA\_\{1\},\\ldots,A\_\{s\},

\|rs​\(Δ\)−mB,p​\(Δ\)\|≤2s−1\.\\left\|r\_\{s\}\(\\Delta\)\-m\_\{B,p\}\(\\Delta\)\\right\|\\leq\\frac\{2\}\{s\-1\}\.\(116\)ThusmB,pm\_\{B,p\}gives the large\-sector mean Muon response in the projected model, with an explicitO⁡\(s−1\)O\(s^\{\-1\}\)correction\.

##### Sparse\-task limit\.

SupposeB​p→0Bp\\to 0\. With probability1−O⁡\(\(B​p\)2\)1\-O\(\(Bp\)^\{2\}\), the task appears at most once\. Whenever it appears once, an observation matching the target label givesA⁡\(Δ\)\>0A\(\\Delta\)\>0, whereas a mismatched observation givesA⁡\(Δ\)<0A\(\\Delta\)<0, for every finiteΔ\\Delta\. Therefore

mB,p​\(Δ\)=B​p​\(cα−ρα\)\+O⁡\(\(B​p\)2\)=B​p​\(1−2​ρα\)\+O⁡\(\(B​p\)2\)\.m\_\{B,p\}\(\\Delta\)=Bp\(c\_\{\\alpha\}\-\\rho\_\{\\alpha\}\)\+O\\\!\\left\(\(Bp\)^\{2\}\\right\)=Bp\(1\-2\\rho\_\{\\alpha\}\)\+O\\\!\\left\(\(Bp\)^\{2\}\\right\)\.\(117\)This expansion is uniform forΔ\\Deltain compact sets\. Hence, forρα<1/2\\rho\_\{\\alpha\}<1/2, the scalar counting dynamics have positive mean margin drift proportional toB​pBp\. For the finite\-sector matrix responsersr\_\{s\}, the additive bound in equation[116](https://arxiv.org/html/2610.31728#A5.E116)alone does not establish the same leading rate asB​p→0Bp\\to 0; transferring this asymptotic requires additional control, such ass−1=o⁡\(B​p\)s^\{\-1\}=o\(Bp\)\.

##### Large\-margin response\.

We next letΔ→∞\\Delta\\to\\infty, so thatu⁡\(Δ\)↓0u\(\\Delta\)\\downarrow 0\. For each fixed batch realization,

sgn\(A\(Δ\)\)⟶𝟏\{N\>0,M=0\}−𝟏\{M\>0\}\.\\operatorname\{sgn\}\(A\(\\Delta\)\)\\longrightarrow\\mathbf\{1\}\\\{N\>0,M=0\\\}\-\\mathbf\{1\}\\\{M\>0\\\}\.Now

ℙ⁡\(M=0\)=\(1−p​ρα\)B,ℙ⁡\(N=0\)=\(1−p\)B\.\\mathbb\{P\}\(M=0\)=\(1\-p\\rho\_\{\\alpha\}\)^\{B\},\\qquad\\mathbb\{P\}\(N=0\)=\(1\-p\)^\{B\}\.Therefore dominated convergence gives

limΔ→∞mB,p​\(Δ\)=χB,p:=2​\(1−p​ρα\)B−\(1−p\)B−1\.\\lim\_\{\\Delta\\to\\infty\}m\_\{B,p\}\(\\Delta\)=\\chi\_\{B,p\}:=2\(1\-p\\rho\_\{\\alpha\}\)^\{B\}\-\(1\-p\)^\{B\}\-1\.\(118\)Sinceu⁡\(Δ\)u\(\\Delta\)decreases withΔ\\Delta, every summand in equation[114](https://arxiv.org/html/2610.31728#A5.E114)is nonincreasing inΔ\\Delta\. ThusmB,p​\(Δ\)m\_\{B,p\}\(\\Delta\)is nonincreasing andχB,p\\chi\_\{B,p\}is its minimum tail value\.

Consider the large\-sector projected mean\-field recursion

Δt\+1mf−Δtmf=η​mB,p​\(Δtmf\)\.\\Delta^\{\\mathrm\{mf\}\}\_\{t\+1\}\-\\Delta^\{\\mathrm\{mf\}\}\_\{t\}=\\eta m\_\{B,p\}\(\\Delta^\{\\mathrm\{mf\}\}\_\{t\}\)\.\(119\)IfχB,p\>0\\chi\_\{B,p\}\>0, thenmB,p​\(Δ\)≥χB,p\>0m\_\{B,p\}\(\\Delta\)\\geq\\chi\_\{B,p\}\>0for every finiteΔ\\Delta, so the limiting mean\-field recursion in equation[119](https://arxiv.org/html/2610.31728#A5.E119)satisfiesΔtmf→∞\\Delta\_\{t\}^\{\\mathrm\{mf\}\}\\to\\infty\. SincemB,p​\(Δ\)→χB,pm\_\{B,p\}\(\\Delta\)\\to\\chi\_\{B,p\}along that trajectory,

Δtmfη​t⟶χB,p\.\\frac\{\\Delta\_\{t\}^\{\\mathrm\{mf\}\}\}\{\\eta t\}\\longrightarrow\\chi\_\{B,p\}\.\(120\)For a finite projected sector, equation[116](https://arxiv.org/html/2610.31728#A5.E116)additionally guarantees positive mean drift whenever2/\(s−1\)<χB,p2/\(s\-1\)<\\chi\_\{B,p\}\. Consequently,

1−Sj​j​\(t\)≍\(K−1\)​e−η​χB,p​t1\-S\_\{jj\}\(t\)\\asymp\(K\-1\)e^\{\-\\eta\\chi\_\{B,p\}t\}along this target\-interpolating mean\-field trajectory\.

##### Corruption\-sparse window and corruption threshold\.

We use≪\\lland≫\\ggin this paragraph as scale\-separation notation\. For fixed nonzeroρα\\rho\_\{\\alpha\}, the corruption\-sparse case is the intermediate window

1≪B​p≪ρα−1\.1\\ll Bp\\ll\\rho\_\{\\alpha\}^\{\-1\}\.Within this window,\(1−p\)B≤e−B​p\(1\-p\)^\{B\}\\leq e^\{\-Bp\}and a binomial expansion gives

χB,p=1−2​B​p​ρα\+O⁡\(\(B​p​ρα\)2\+e−B​p\)\.\\chi\_\{B,p\}=1\-2Bp\\rho\_\{\\alpha\}\+O\\\!\\left\(\(Bp\\rho\_\{\\alpha\}\)^\{2\}\+e^\{\-Bp\}\\right\)\.\(121\)Thus the projected large\-margin growth rate is close to the unit deterministic\-label full\-batch rate and is no longer occurrence\-limited\.

A sharper crossover is obtained in the Poisson scaling

B​p→∞,p​ρα→0,B​p​ρα→λ\.Bp\\to\\infty,\\qquad p\\rho\_\{\\alpha\}\\to 0,\\qquad Bp\\rho\_\{\\alpha\}\\to\\lambda\.Then

χB,p⟶2​e−λ−1\.\\chi\_\{B,p\}\\longrightarrow 2e^\{\-\\lambda\}\-1\.\(122\)Hence the large\-margin regime has positive drift forλ<log⁡2\\lambda<\\log 2and negative drift forλ\>log⁡2\\lambda\>\\log 2\. Equivalently, once task absence is negligible, the crossover batch scale is

Bnoise≃log⁡2p​ρα\.B\_\{\\mathrm\{noise\}\}\\simeq\\frac\{\\log 2\}\{p\\rho\_\{\\alpha\}\}\.\(123\)The regimeB​p≫1Bp\\gg 1butB​p​ρα≪1Bp\\rho\_\{\\alpha\}\\ll 1therefore has a quantitatively stable large\-margin regime in the reduced model, while resolved corruption reverses the projected large\-margin drift\. The finite population optimum exists for every0<α<10<\\alpha<1, independently of batch size; the local response around it is analyzed separately below\.

#### E\.5Moment\-matched isotropic covariance closure

A Gaussian proxy with the exact column covariance ofLLcan be represented as

LG=1B​P0​\[αK​Z\+α⁡\(1−α\)​diag⁡\(g\)\]​P1/2,L\_\{\\mathrm\{G\}\}=\\frac\{1\}\{\\sqrt\{B\}\}P\_\{0\}\\left\[\\sqrt\{\\frac\{\\alpha\}\{K\}\}Z\+\\sqrt\{\\alpha\(1\-\\alpha\)\}\\operatorname\{diag\}\(g\)\\right\]P^\{1/2\},\(124\)whereZZhas i\.i\.d\. standard Gaussian entries andgghas i\.i\.d\. standard Gaussian coordinates, independently\. This proxy makes the anisotropic categorical covariance explicit: it contains both a projected dense component and a projected diagonal component\.

For a closed\-form polar response, we now replace this anisotropic covariance by a moment\-matched isotropic covariance on the zero\-sum subspace\. The exact covariance in equation[107](https://arxiv.org/html/2610.31728#A5.E107)has one longitudinal eigenvalueα​cα\\alpha c\_\{\\alpha\}andK−2K\-2transverse eigenvaluesα/K\\alpha/K; the closure preserves their total covariance energy while removing the directional anisotropy\. This is the additional approximation that allows us to reuse the Gaussian polar response from Section[3\.2](https://arxiv.org/html/2610.31728#S3.SS2)\.

From equation[107](https://arxiv.org/html/2610.31728#A5.E107),

tr⁡\(Γj\)=α⁡\(2−α\)​K−1K\.\\operatorname\{tr\}\(\\Gamma\_\{j\}\)=\\alpha\(2\-\\alpha\)\\frac\{K\-1\}\{K\}\.\(125\)The zero\-sum subspace has dimensionK−1K\-1\. We therefore define

τα2:=tr⁡\(Γj\)K−1=α⁡\(2−α\)K,\\tau\_\{\\alpha\}^\{2\}:=\\frac\{\\operatorname\{tr\}\(\\Gamma\_\{j\}\)\}\{K\-1\}=\\frac\{\\alpha\(2\-\\alpha\)\}\{K\},\(126\)and make the moment\-matched covariance approximation

Γj≈τα2​P0\.\\Gamma\_\{j\}\\approx\\tau\_\{\\alpha\}^\{2\}P\_\{0\}\.\(127\)This approximation preserves the total label\-noise variance on the zero\-sum subspace but removes the longitudinal anisotropy\.

Let

X∈ℝK×\(K−1\)X\\in\\mathbb\{R\}^\{K\\times\(K\-1\)\}have orthonormal columns spanning the zero\-sum subspace, so that

X⊤​X=IK−1,X​X⊤=P0\.X^\{\\top\}X=I\_\{K\-1\},\\qquad XX^\{\\top\}=P\_\{0\}\.Since every stochastic\-gradient column sums to zero, we may write

G=X​G^,G^:=X⊤​G∈ℝ\(K−1\)×K\.G=X\\widehat\{G\},\\qquad\\widehat\{G\}:=X^\{\\top\}G\\in\\mathbb\{R\}^\{\(K\-1\)\\times K\}\.The partial polar factors are related by

msgn⁡\(G\)=X​msgn⁡\(G^\)\.\\operatorname\{msgn\}\(G\)=X\\operatorname\{msgn\}\(\\widehat\{G\}\)\.\(128\)
For an explicit closed\-form response, we additionally use a local frequency\-homogeneous approximation: the symmetry\-equivalent columns under consideration are assigned a common frequency scalepp\. This is a local spectral proxy, not an exact reduction of a heterogeneous matrix: the polar factor couples columns, and their unequal noise scales generally break right\-orthogonal symmetry\. Varyingppcompares proxies at different local frequency scales; it does not vary the common probability of allKKtasks in one normalized uniform distribution\. Under equation[127](https://arxiv.org/html/2610.31728#A5.E127), the projected Gaussian label noise is then approximated by

L^≈σp,α​Ξ,σp,α2:=p​τα2B=p​α​\(2−α\)B​K,\\widehat\{L\}\\approx\\sigma\_\{p,\\alpha\}\\Xi,\\qquad\\sigma\_\{p,\\alpha\}^\{2\}:=\\frac\{p\\tau\_\{\\alpha\}^\{2\}\}\{B\}=\\frac\{p\\alpha\(2\-\\alpha\)\}\{BK\},\(129\)where

Ξ∈ℝ\(K−1\)×K\\Xi\\in\\mathbb\{R\}^\{\(K\-1\)\\times K\}has i\.i\.d\. standard Gaussian entries\.

We use the rectangular analogue of Proposition[3\.1](https://arxiv.org/html/2610.31728#S3.Thmtheorem1)\. For a rectangular matrixA=U​Σ​V⊤A=U\\Sigma V^\{\\top\}with singular valuesσi\\sigma\_\{i\}, we use the same partial\-polar definition

msgn\(A\):=Udiag\(𝟏\{σi\>0\}\)V⊤\.\\operatorname\{msgn\}\(A\):=U\\operatorname\{diag\}\(\\mathbf\{1\}\\\{\\sigma\_\{i\}\>0\\\}\)V^\{\\top\}\.Thus positive singular values are mapped to one and zero singular values remain zero\. IfΞ∈ℝm×n\\Xi\\in\\mathbb\{R\}^\{m\\times n\}has i\.i\.d\. standard Gaussian entries, define

cm,n:=1m​n​𝔼​‖Ξ‖∗\.c\_\{m,n\}:=\\frac\{1\}\{mn\}\\mathbb\{E\}\\\|\\Xi\\\|\_\{\*\}\.\(130\)For fixedm,nm,n, Gaussian convolution andO⁡\(m\)×O⁡\(n\)O\(m\)\\times O\(n\)equivariance give

𝔼⁡\[msgn⁡\(H\+σ​Ξ\)\]=cm,nσ​H\+Om,n​\(‖H‖F3σ3\)\\mathbb\{E\}\[\\operatorname\{msgn\}\(H\+\\sigma\\Xi\)\]=\\frac\{c\_\{m,n\}\}\{\\sigma\}H\+O\_\{m,n\}\\\!\\left\(\\frac\{\\\|H\\\|\_\{F\}^\{3\}\}\{\\sigma^\{3\}\}\\right\)\(131\)as‖H‖F/σ→0\\\|H\\\|\_\{F\}/\\sigma\\to 0\. The coefficient follows by taking the trace of the derivative:

m​n​cm,n=𝔼​⟨msgn⁡\(Ξ\),Ξ⟩F=𝔼​‖Ξ‖∗\.mn\\,c\_\{m,n\}=\\mathbb\{E\}\\langle\\operatorname\{msgn\}\(\\Xi\),\\Xi\\rangle\_\{F\}=\\mathbb\{E\}\\\|\\Xi\\\|\_\{\*\}\.Thus equation[131](https://arxiv.org/html/2610.31728#A5.E131)is the direct rectangular counterpart of the square Gaussian response derived in Appendix[A\.1](https://arxiv.org/html/2610.31728#A1.SS1)\.

#### E\.6Local Muon rate under the covariance closure

Under the equal\-incorrect\-logit reduction in a symmetry\-equivalent sector, define

Δj:=W¯j​j−1K−1​∑i≠jW¯i​j\.\\Delta\_\{j\}:=\\overline\{W\}\_\{jj\}\-\\frac\{1\}\{K\-1\}\\sum\_\{i\\neq j\}\\overline\{W\}\_\{ij\}\.Then

Sj​j=eΔjeΔj\+K−1\.S\_\{jj\}=\\frac\{e^\{\\Delta\_\{j\}\}\}\{e^\{\\Delta\_\{j\}\}\+K\-1\}\.The noisy population optimum satisfies

Sj​j∗=cα,S\_\{jj\}^\{\*\}=c\_\{\\alpha\},which gives

Δj∗=log⁡\(Kα−\(K−1\)\)\.\\Delta\_\{j\}^\{\*\}=\\log\\left\(\\frac\{K\}\{\\alpha\}\-\(K\-1\)\\right\)\.\(132\)
Define

rj:=cα−Sj​j\.r\_\{j\}:=c\_\{\\alpha\}\-S\_\{jj\}\.Under the equal\-incorrect\-logit reduction,

\(Dα−S\)​ej=rj​vj,vj=ej−1K−1​∑i≠jei\.\(D\_\{\\alpha\}\-S\)e\_\{j\}=r\_\{j\}v\_\{j\},\\qquad v\_\{j\}=e\_\{j\}\-\\frac\{1\}\{K\-1\}\\sum\_\{i\\neq j\}e\_\{i\}\.\(133\)For a task of local frequency scalepp, the projected deterministic signal is therefore

H^j=p​rj​xj​ej⊤,xj:=X⊤​vj\.\\widehat\{H\}\_\{j\}=pr\_\{j\}x\_\{j\}e\_\{j\}^\{\\top\},\\qquad x\_\{j\}:=X^\{\\top\}v\_\{j\}\.\(134\)Becausevjv\_\{j\}lies in the zero\-sum subspace,

‖xj‖22=‖vj‖22=KK−1\.\\\|x\_\{j\}\\\|\_\{2\}^\{2\}=\\\|v\_\{j\}\\\|\_\{2\}^\{2\}=\\frac\{K\}\{K\-1\}\.\(135\)
Combining equations[129](https://arxiv.org/html/2610.31728#A5.E129)and[131](https://arxiv.org/html/2610.31728#A5.E131)withm=K−1m=K\-1andn=Kn=Kgives

𝔼⁡\[msgn⁡\(H^j\+L^\)\]≈cK−1,Kσp,α​H^j\\mathbb\{E\}\[\\operatorname\{msgn\}\(\\widehat\{H\}\_\{j\}\+\\widehat\{L\}\)\]\\approx\\frac\{c\_\{K\-1,K\}\}\{\\sigma\_\{p,\\alpha\}\}\\widehat\{H\}\_\{j\}\(136\)in the small\-signal regime near the population optimum under label noise\.

The margin functional is

Δj=vj⊤​W¯​ej\.\\Delta\_\{j\}=v\_\{j\}^\{\\top\}\\overline\{W\}e\_\{j\}\.Using equations[128](https://arxiv.org/html/2610.31728#A5.E128),[134](https://arxiv.org/html/2610.31728#A5.E134), and[135](https://arxiv.org/html/2610.31728#A5.E135), the expected Muon margin increment becomes

𝔼⁡\[Δj​\(t\+1\)−Δj​\(t\)∣Δj​\(t\)\]\\displaystyle\\mathbb\{E\}\[\\Delta\_\{j\}\(t\+1\)\-\\Delta\_\{j\}\(t\)\\mid\\Delta\_\{j\}\(t\)\]≈η​cK−1,Kσp,α​p​rj​KK−1\.\\displaystyle\\approx\\eta\\frac\{c\_\{K\-1,K\}\}\{\\sigma\_\{p,\\alpha\}\}pr\_\{j\}\\frac\{K\}\{K\-1\}\.\(137\)
Finally,

∂Sj​j∂Δj=Sj​j​\(1−Sj​j\),\\frac\{\\partial S\_\{jj\}\}\{\\partial\\Delta\_\{j\}\}=S\_\{jj\}\(1\-S\_\{jj\}\),and at the optimum,

Sj​j∗​\(1−Sj​j∗\)=cα​ρα\.S\_\{jj\}^\{\*\}\(1\-S\_\{jj\}^\{\*\}\)=c\_\{\\alpha\}\\rho\_\{\\alpha\}\.Hence

rj=cα​ρα​\(Δj∗−Δj\)\+O⁡\(\(Δj−Δj∗\)2\)\.r\_\{j\}=c\_\{\\alpha\}\\rho\_\{\\alpha\}\\bigl\(\\Delta\_\{j\}^\{\*\}\-\\Delta\_\{j\}\\bigr\)\+O\\\!\\left\(\(\\Delta\_\{j\}\-\\Delta\_\{j\}^\{\*\}\)^\{2\}\\right\)\.\(138\)Since

ρα=α​K−1K,\\rho\_\{\\alpha\}=\\alpha\\frac\{K\-1\}\{K\},substituting equations[129](https://arxiv.org/html/2610.31728#A5.E129)and[138](https://arxiv.org/html/2610.31728#A5.E138)into equation[137](https://arxiv.org/html/2610.31728#A5.E137)yields

𝔼⁡\[Δj​\(t\+1\)−Δj​\(t\)∣Δj​\(t\)\]≈η​cK−1,K​B​K​p​cα​α2−α​\(Δj∗−Δj​\(t\)\)\.\\mathbb\{E\}\[\\Delta\_\{j\}\(t\+1\)\-\\Delta\_\{j\}\(t\)\\mid\\Delta\_\{j\}\(t\)\]\\approx\\eta\\,c\_\{K\-1,K\}\\sqrt\{BKp\}\\,\\frac\{c\_\{\\alpha\}\\sqrt\{\\alpha\}\}\{\\sqrt\{2\-\\alpha\}\}\\bigl\(\\Delta\_\{j\}^\{\*\}\-\\Delta\_\{j\}\(t\)\\bigr\)\.\(139\)Thus the covariance and locally homogeneous frequency approximations predict a local Muon conditional\-mean contraction coefficient proportional top\\sqrt\{p\}\.

For comparison, under the same symmetric\-logit reduction, the conditional mean SGD update obeys

𝔼⁡\[Δj​\(t\+1\)−Δj​\(t\)∣Wt\]=η​KK−1​pj​rj​\(t\),\\mathbb\{E\}\[\\Delta\_\{j\}\(t\+1\)\-\\Delta\_\{j\}\(t\)\\mid W\_\{t\}\]=\\eta\\frac\{K\}\{K\-1\}p\_\{j\}r\_\{j\}\(t\),so its exact local conditional\-mean contraction coefficient is

η​KK−1​pj​cα​ρα,\\eta\\frac\{K\}\{K\-1\}p\_\{j\}c\_\{\\alpha\}\\rho\_\{\\alpha\},\(140\)which scales linearly withpjp\_\{j\}\.

For largeKK, the quarter\-circle law gives

cK−1,K​K⟶83​π,c\_\{K\-1,K\}\\sqrt\{K\}\\longrightarrow\\frac\{8\}\{3\\pi\},\(141\)whilecα→1−αc\_\{\\alpha\}\\to 1\-\\alpha\. Therefore equation[139](https://arxiv.org/html/2610.31728#A5.E139)reduces to

𝔼⁡\[Δj​\(t\+1\)−Δj​\(t\)∣Δj​\(t\)\]≈η​83​π​\(1−α\)​α2−α​B​p​\(Δj∗−Δj​\(t\)\)\.\\mathbb\{E\}\[\\Delta\_\{j\}\(t\+1\)\-\\Delta\_\{j\}\(t\)\\mid\\Delta\_\{j\}\(t\)\]\\approx\\eta\\,\\frac\{8\}\{3\\pi\}\\frac\{\(1\-\\alpha\)\\sqrt\{\\alpha\}\}\{\\sqrt\{2\-\\alpha\}\}\\sqrt\{Bp\}\\,\\bigl\(\\Delta\_\{j\}^\{\*\}\-\\Delta\_\{j\}\(t\)\\bigr\)\.\(142\)
##### Scope of the approximation\.

The covariance formulas in equations[104](https://arxiv.org/html/2610.31728#A5.E104)–[107](https://arxiv.org/html/2610.31728#A5.E107)are exact\. The explicit rate equation[139](https://arxiv.org/html/2610.31728#A5.E139)uses two additional approximations: the anisotropic categorical covariance is replaced by the isotropic covariance equation[127](https://arxiv.org/html/2610.31728#A5.E127)with matching trace, and the columns in the local spectral sector are assigned a common frequency scalepp\. These approximations restore the left\-right isotropy needed for a closed\-form polar response while retaining the total label\-noise energy\. They are intended for the corruption\-resolved local regime; the occurrence\-limited and corruption\-sparse regimes are instead described by the counting analysis in Appendix[E\.4](https://arxiv.org/html/2610.31728#A5.SS4)\.

### Appendix FMuon–SignSGD Sparse\-Support Contrast: Complete Derivation

This appendix derives the two\-task SignSGD dynamics and makes explicit how the result depends on rotations of the embedding space\. We use the elementwise conventionsgn⁡\(0\)=0\\operatorname\{sgn\}\(0\)=0\.

#### F\.1Rank\-one reduction

In task coordinates, the two\-task negative stochastic gradient is

v​gt⊤,v=\[1−1\],gt=\[q1​\(t\)​u1​\(t\)−q2​\(t\)​u2​\(t\)\]\.vg\_\{t\}^\{\\top\},\\qquad v=\\begin\{bmatrix\}1\\\\ \-1\\end\{bmatrix\},\\qquad g\_\{t\}=\\begin\{bmatrix\}q\_\{1\}\(t\)u\_\{1\}\(t\)\\\\ \-q\_\{2\}\(t\)u\_\{2\}\(t\)\\end\{bmatrix\}\.Hence, in the original parameter coordinates, the negative gradient is

E~​v​gt⊤​E⊤=\(E~​v\)​\(E​gt\)⊤\.\\widetilde\{E\}vg\_\{t\}^\{\\top\}E^\{\\top\}=\(\\widetilde\{E\}v\)\(Eg\_\{t\}\)^\{\\top\}\.For arbitrary vectorsa,ba,b,

sgn⁡\(a​b⊤\)=sgn⁡\(a\)​sgn⁡\(b\)⊤,\\operatorname\{sgn\}\(ab^\{\\top\}\)=\\operatorname\{sgn\}\(a\)\\operatorname\{sgn\}\(b\)^\{\\top\},where the sign is applied elementwise\. Therefore the SignSGD update in task coordinates is

W¯t\+1−W¯t=η​E~⊤​sgn⁡\(E~​v\)​sgn​\(E​gt\)⊤​E\.\\overline\{W\}\_\{t\+1\}\-\\overline\{W\}\_\{t\}=\\eta\\widetilde\{E\}^\{\\top\}\\operatorname\{sgn\}\(\\widetilde\{E\}v\)\\operatorname\{sgn\}\(Eg\_\{t\}\)^\{\\top\}E\.
Define the margin vector

ht:=\[Δ1​\(t\)−Δ2​\(t\)\]=W¯t⊤​v\.h\_\{t\}:=\\begin\{bmatrix\}\\Delta\_\{1\}\(t\)\\\\ \-\\Delta\_\{2\}\(t\)\\end\{bmatrix\}=\\overline\{W\}\_\{t\}^\{\\top\}v\.Then

ht\+1−ht\\displaystyle h\_\{t\+1\}\-h\_\{t\}=η​E⊤​sgn⁡\(E​gt\)​sgn​\(E~​v\)⊤​E~​v\\displaystyle=\\eta E^\{\\top\}\\operatorname\{sgn\}\(Eg\_\{t\}\)\\operatorname\{sgn\}\(\\widetilde\{E\}v\)^\{\\top\}\\widetilde\{E\}v=η​c~​E⊤​sgn⁡\(E​gt\),\\displaystyle=\\eta\\widetilde\{c\}\\,E^\{\\top\}\\operatorname\{sgn\}\(Eg\_\{t\}\),where

c~:=sgn⁡\(E~​v\)⊤​E~​v=‖E~​v‖1\>0\.\\widetilde\{c\}:=\\operatorname\{sgn\}\(\\widetilde\{E\}v\)^\{\\top\}\\widetilde\{E\}v=\\\|\\widetilde\{E\}v\\\|\_\{1\}\>0\.\(143\)Thus the answer embedding contributes only the positive scalarc~\\widetilde\{c\}to the two margins, whereas the query embedding determines their coupling\.

#### F\.2Rotation\-dependent margin recursion

Consider the representative rotation

E=\[cs−sc\],c:=cos⁡θ,s:=sin⁡θ,0<θ<π2\.E=\\begin\{bmatrix\}c&s\\\\ \-s&c\\end\{bmatrix\},\\qquad c:=\\cos\\theta,\\qquad s:=\\sin\\theta,\\qquad 0<\\theta<\\frac\{\\pi\}\{2\}\.Then

E​gt=\[c​q1​u1−s​q2​u2−s​q1​u1−c​q2​u2\]\.Eg\_\{t\}=\\begin\{bmatrix\}cq\_\{1\}u\_\{1\}\-sq\_\{2\}u\_\{2\}\\\\ \-sq\_\{1\}u\_\{1\}\-cq\_\{2\}u\_\{2\}\\end\{bmatrix\}\.The second coordinate is strictly negative becauseq1\+q2=1q\_\{1\}\+q\_\{2\}=1andu1,u2\>0u\_\{1\},u\_\{2\}\>0\. Define

ξt:=sgn⁡\(c​q1​\(t\)​u1​\(t\)−s​q2​\(t\)​u2​\(t\)\)\.\\xi\_\{t\}:=\\operatorname\{sgn\}\\left\(cq\_\{1\}\(t\)u\_\{1\}\(t\)\-sq\_\{2\}\(t\)u\_\{2\}\(t\)\\right\)\.\(144\)Hence

sgn⁡\(E​gt\)=\[ξt−1\],\\operatorname\{sgn\}\(Eg\_\{t\}\)=\\begin\{bmatrix\}\\xi\_\{t\}\\\\ \-1\\end\{bmatrix\},and

E⊤​sgn⁡\(E​gt\)=\[c​ξt\+ss​ξt−c\]\.E^\{\\top\}\\operatorname\{sgn\}\(Eg\_\{t\}\)=\\begin\{bmatrix\}c\\xi\_\{t\}\+s\\\\ s\\xi\_\{t\}\-c\\end\{bmatrix\}\.Since the second entry ofhth\_\{t\}is−Δ2​\(t\)\-\\Delta\_\{2\}\(t\), the exact margin recursion is

Δ1​\(t\+1\)−Δ1​\(t\)\\displaystyle\\Delta\_\{1\}\(t\+1\)\-\\Delta\_\{1\}\(t\)=η​c~​\(c​ξt\+s\),\\displaystyle=\\eta\\widetilde\{c\}\(c\\xi\_\{t\}\+s\),\(145\)Δ2​\(t\+1\)−Δ2​\(t\)\\displaystyle\\Delta\_\{2\}\(t\+1\)\-\\Delta\_\{2\}\(t\)=η​c~​\(c−s​ξt\)\.\\displaystyle=\\eta\\widetilde\{c\}\(c\-s\\xi\_\{t\}\)\.\(146\)Equivalently, for

Dt:=Δ1​\(t\)−Δ2​\(t\),D\_\{t\}:=\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\),we have

Dt\+1−Dt=η​c~​\(\(c\+s\)​ξt\+s−c\)\.D\_\{t\+1\}\-D\_\{t\}=\\eta\\widetilde\{c\}\\left\(\(c\+s\)\\xi\_\{t\}\+s\-c\\right\)\.\(147\)In particular,

ξt=1⟹Dt\+1−Dt=2​η​c~​s,\\xi\_\{t\}=1\\quad\\Longrightarrow\\quad D\_\{t\+1\}\-D\_\{t\}=2\\eta\\widetilde\{c\}\\,s,and

ξt=−1⟹Dt\+1−Dt=−2​η​c~​c\.\\xi\_\{t\}=\-1\\quad\\Longrightarrow\\quad D\_\{t\+1\}\-D\_\{t\}=\-2\\eta\\widetilde\{c\}\\,c\.Ifξt=0\\xi\_\{t\}=0, then

Dt\+1−Dt=η​c~​\(s−c\)\.D\_\{t\+1\}\-D\_\{t\}=\\eta\\widetilde\{c\}\(s\-c\)\.

#### F\.3Full\-batch reference dynamics

Setqi​\(t\)=piq\_\{i\}\(t\)=p\_\{i\}\. Define the sum and gap

At:=Δ1​\(t\)\+Δ2​\(t\),Dt:=Δ1​\(t\)−Δ2​\(t\)\.A\_\{t\}:=\\Delta\_\{1\}\(t\)\+\\Delta\_\{2\}\(t\),\\qquad D\_\{t\}:=\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\)\.From equations[145](https://arxiv.org/html/2610.31728#A6.E145)–[146](https://arxiv.org/html/2610.31728#A6.E146),

At\+1−At=\{2​η​c~​c,ξt=1,η​c~​\(c\+s\),ξt=0,2​η​c~​s,ξt=−1\.A\_\{t\+1\}\-A\_\{t\}=\\begin\{cases\}2\\eta\\widetilde\{c\}\\,c,&\\xi\_\{t\}=1,\\\\ \\eta\\widetilde\{c\}\(c\+s\),&\\xi\_\{t\}=0,\\\\ 2\\eta\\widetilde\{c\}\\,s,&\\xi\_\{t\}=\-1\.\\end\{cases\}Therefore

At≥2​η​c~​min⁡\{c,s\}​t,A\_\{t\}\\geq 2\\eta\\widetilde\{c\}\\min\\\{c,s\\\}\\,t,\(148\)soAt→∞A\_\{t\}\\to\\infty\.

For fixedAA, write

Δ1=A\+D2,Δ2=A−D2\.\\Delta\_\{1\}=\\frac\{A\+D\}\{2\},\\qquad\\Delta\_\{2\}=\\frac\{A\-D\}\{2\}\.The switching variable in equation[144](https://arxiv.org/html/2610.31728#A6.E144)has the same sign as

Λ⁡\(A,D\):=log⁡p1​cp2​s\+log⁡\(1\+e\(A−D\)/2\)−log⁡\(1\+e\(A\+D\)/2\)\.\\Lambda\(A,D\):=\\log\\frac\{p\_\{1\}c\}\{p\_\{2\}s\}\+\\log\\left\(1\+e^\{\(A\-D\)/2\}\\right\)\-\\log\\left\(1\+e^\{\(A\+D\)/2\}\\right\)\.\(149\)For each fixedAA,Λ⁡\(A,D\)\\Lambda\(A,D\)is strictly decreasing inDD, with limits\+∞\+\\inftyand−∞\-\\inftyasD→−∞D\\to\-\\inftyandD→\+∞D\\to\+\\infty, respectively\. Hence there is a unique switching thresholdT⁡\(A\)T\(A\)satisfying

Λ⁡\(A,T⁡\(A\)\)=0\.\\Lambda\(A,T\(A\)\)=0\.Moreover,

T⁡\(A\)⟶Dθ:=log⁡p1p2\+log⁡cot⁡θas​A→∞\.T\(A\)\\longrightarrow D\_\{\\theta\}:=\\log\\frac\{p\_\{1\}\}\{p\_\{2\}\}\+\\log\\cot\\theta\\qquad\\text\{as \}A\\to\\infty\.\(150\)
On either strict side of the switching threshold, the recursion moves the gap toward that threshold:

Dt<T⁡\(At\)⟹ξt=1,D\_\{t\}<T\(A\_\{t\}\)\\quad\\Longrightarrow\\quad\\xi\_\{t\}=1,while

Dt\>T⁡\(At\)⟹ξt=−1\.D\_\{t\}\>T\(A\_\{t\}\)\\quad\\Longrightarrow\\quad\\xi\_\{t\}=\-1\.At equality,ξt=0\\xi\_\{t\}=0and

Dt\+1−Dt=η​c~​\(s−c\),D\_\{t\+1\}\-D\_\{t\}=\\eta\\widetilde\{c\}\(s\-c\),so the one\-step displacement is uniformly bounded\. SinceT⁡\(A\)T\(A\)is bounded forA≥0A\\geq 0, these one\-step bounds imply thatDtD\_\{t\}is bounded\. Combining this with equation[148](https://arxiv.org/html/2610.31728#A6.E148)gives

Δ1​\(t\)→∞,Δ2​\(t\)→∞\.\\Delta\_\{1\}\(t\)\\to\\infty,\\qquad\\Delta\_\{2\}\(t\)\\to\\infty\.Consequently, equation[150](https://arxiv.org/html/2610.31728#A6.E150)applies along the trajectory\.

More precisely, for everyε\>0\\varepsilon\>0, for all sufficiently largett,

Dθ−ε≤T⁡\(At\)≤Dθ\+ε\.D\_\{\\theta\}\-\\varepsilon\\leq T\(A\_\{t\}\)\\leq D\_\{\\theta\}\+\\varepsilon\.An upward crossing of the threshold changesDtD\_\{t\}by at most2​η​c~​s2\\eta\\widetilde\{c\}\\,s, while a downward crossing changes it by at most2​η​c~​c2\\eta\\widetilde\{c\}\\,c\. Therefore

Dθ−2​η​c~​c≤lim inft→∞Dt≤lim supt→∞Dt≤Dθ\+2​η​c~​s\.D\_\{\\theta\}\-2\\eta\\widetilde\{c\}\\,c\\leq\\liminf\_\{t\\to\\infty\}D\_\{t\}\\leq\\limsup\_\{t\\to\\infty\}D\_\{t\}\\leq D\_\{\\theta\}\+2\\eta\\widetilde\{c\}\\,s\.\(151\)Thus the discrete full\-batch gap need not converge to a single limit; rather, it remains in anO⁡\(η\)O\(\\eta\)switching band around the embedding\-dependent locationDθD\_\{\\theta\}\. In the small\-step limit this band collapses toDθD\_\{\\theta\}\.

We next compute the asymptotic growth rates\. Rearranging equation[147](https://arxiv.org/html/2610.31728#A6.E147)gives

1t​∑k=0t−1ξk=c−sc\+s\+Dt−D0η​c~​\(c\+s\)​t\.\\frac\{1\}\{t\}\\sum\_\{k=0\}^\{t\-1\}\\xi\_\{k\}=\\frac\{c\-s\}\{c\+s\}\+\\frac\{D\_\{t\}\-D\_\{0\}\}\{\\eta\\widetilde\{c\}\(c\+s\)t\}\.SinceDtD\_\{t\}is bounded,

1t​∑k=0t−1ξk⟶c−sc\+s\.\\frac\{1\}\{t\}\\sum\_\{k=0\}^\{t\-1\}\\xi\_\{k\}\\longrightarrow\\frac\{c\-s\}\{c\+s\}\.\(152\)Averaging equations[145](https://arxiv.org/html/2610.31728#A6.E145)and[146](https://arxiv.org/html/2610.31728#A6.E146)then yields

Δ1​\(t\)η​t⟶c~c\+s,Δ2​\(t\)η​t⟶c~c\+s\.\\frac\{\\Delta\_\{1\}\(t\)\}\{\\eta t\}\\longrightarrow\\frac\{\\widetilde\{c\}\}\{c\+s\},\\qquad\\frac\{\\Delta\_\{2\}\(t\)\}\{\\eta t\}\\longrightarrow\\frac\{\\widetilde\{c\}\}\{c\+s\}\.\(153\)Hence full\-batch SignSGD balances the two tasks at leading order, but both the switching location and the common growth rate depend on the embedding rotation\.

#### F\.4Sparse\-support negative transfer

Consider the event

𝒜t:=\{q2\(t\)=0\},\\mathcal\{A\}\_\{t\}:=\\\{q\_\{2\}\(t\)=0\\\},meaning that the batch contains no examples from task 2\. Since every sample is then task 1,

ℙ⁡\(𝒜t\)=p1B\.\\mathbb\{P\}\(\\mathcal\{A\}\_\{t\}\)=p\_\{1\}^\{B\}\.On𝒜t\\mathcal\{A\}\_\{t\},

and hence

Δ2​\(t\+1\)−Δ2​\(t\)=η​c~​\(c−s\)\.\\Delta\_\{2\}\(t\+1\)\-\\Delta\_\{2\}\(t\)=\\eta\\widetilde\{c\}\(c\-s\)\.\(154\)Therefore, whenever

θ\>π4,\\theta\>\\frac\{\\pi\}\{4\},a batch containing no task 2 examples decreases the task 2 margin\.

A stronger statement follows by bounding all remaining updates\. For every batch,

Δ2​\(t\+1\)−Δ2​\(t\)≤η​c~​\(c\+s\)\.\\Delta\_\{2\}\(t\+1\)\-\\Delta\_\{2\}\(t\)\\leq\\eta\\widetilde\{c\}\(c\+s\)\.Consequently,

Δ2​\(t\)≤Δ2​\(0\)\+η​c~​\[\(c−s\)​∑k<t𝟏𝒜k\+\(c\+s\)​∑k<t𝟏𝒜kc\]\.\\Delta\_\{2\}\(t\)\\leq\\Delta\_\{2\}\(0\)\+\\eta\\widetilde\{c\}\\left\[\(c\-s\)\\sum\_\{k<t\}\\mathbf\{1\}\_\{\\mathcal\{A\}\_\{k\}\}\+\(c\+s\)\\sum\_\{k<t\}\\mathbf\{1\}\_\{\\mathcal\{A\}\_\{k\}^\{c\}\}\\right\]\.By the strong law of large numbers,

1t​∑k<t𝟏𝒜k⟶p1Ba\.s\.\\frac\{1\}\{t\}\\sum\_\{k<t\}\\mathbf\{1\}\_\{\\mathcal\{A\}\_\{k\}\}\\longrightarrow p\_\{1\}^\{B\}\\qquad\\text\{a\.s\.\}Therefore

lim supt→∞Δ2​\(t\)η​c~​t≤c\+s⁡\(1−2​p1B\)a\.s\.\\limsup\_\{t\\to\\infty\}\\frac\{\\Delta\_\{2\}\(t\)\}\{\\eta\\widetilde\{c\}\\,t\}\\leq c\+s\\left\(1\-2p\_\{1\}^\{B\}\\right\)\\qquad\\text\{a\.s\.\}\(155\)If

p1B\>1\+cot⁡θ2,p\_\{1\}^\{B\}\>\\frac\{1\+\\cot\\theta\}\{2\},\(156\)then the right\-hand side of equation[155](https://arxiv.org/html/2610.31728#A6.E155)is strictly negative, and therefore

Δ2​\(t\)→−∞\\Delta\_\{2\}\(t\)\\to\-\\inftylinearly almost surely\. Condition \([156](https://arxiv.org/html/2610.31728#A6.E156)\) can only hold whenθ\>π/4\\theta\>\\pi/4\. In this regime, equation[145](https://arxiv.org/html/2610.31728#A6.E145)also gives

Δ1​\(t\+1\)−Δ1​\(t\)≥η​c~​\(s−c\)\>0\.\\Delta\_\{1\}\(t\+1\)\-\\Delta\_\{1\}\(t\)\\geq\\eta\\widetilde\{c\}\(s\-c\)\>0\.Thus insufficiently small relative task\-frequency fluctuations can be converted by the embedding rotation into persistent negative transfer from the frequent task to the rare task\.

For Muon, orthogonal equivariance allows the dynamics to be analyzed directly in task coordinates\. Whenq2​\(t\)=0q\_\{2\}\(t\)=0, the exact two\-task Muon recursion in equation[3](https://arxiv.org/html/2610.31728#A3.E3)gives

Δ2​\(t\+1\)−Δ2​\(t\)=0\.\\Delta\_\{2\}\(t\+1\)\-\\Delta\_\{2\}\(t\)=0\.The rotation\-induced negative\-transfer mechanism above is therefore specific to the coordinate\-sensitive elementwise sign operation\.

### Appendix GSupplementary EMA Bias–Variance Calculation

This optional scalar calculation makes the familiar EMA bias–variance trade\-off explicit for an exogenous affine drifting signal\. It is included only to interpret the tracking side of Proposition[5\.1](https://arxiv.org/html/2610.31728#S5.Thmtheorem1); it is not a separate Muon convergence result or a method\-level contribution\. For reference, write

R⁡\(β,v,b\):=v​1−β1\+β\+b2​β2\(1−β\)2,R\(\\beta;v,b\):=v\\frac\{1\-\\beta\}\{1\+\\beta\}\+b^\{2\}\\frac\{\\beta^\{2\}\}\{\(1\-\\beta\)^\{2\}\},\(157\)and denote the first\-order optimality condition by

b2​β​\(1\+β\)2=v​\(1−β\)3\.b^\{2\}\\beta\(1\+\\beta\)^\{2\}=v\(1\-\\beta\)^\{3\}\.\(158\)
#### G\.1Affine\-drift calculation

Letyt=at\+εty\_\{t\}=a\_\{t\}\+\\varepsilon\_\{t\}, whereat=a0\+b​ta\_\{t\}=a\_\{0\}\+btand the noise variables are independent with mean zero and variancev\>0v\>0\. Definemt=\(1−β\)​∑r≥0βr​yt−rm\_\{t\}=\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}y\_\{t\-r\}for0≤β<10\\leq\\beta<1\. The deterministic weighted sum converges absolutely, and the noise weights are square summable, so the infinite\-past EMA is well defined in mean square\. Its mean is

𝔼​mt=\(1−β\)​∑r≥0βr​\(at−b​r\)=at−β1−β​b,\\mathbb\{E\}m\_\{t\}=\(1\-\\beta\)\\sum\_\{r\\geq 0\}\\beta^\{r\}\(a\_\{t\}\-br\)=a\_\{t\}\-\\frac\{\\beta\}\{1\-\\beta\}b,while independence gives

Var⁡\(mt\)=v​\(1−β\)2​∑r≥0β2​r=v​1−β1\+β\.\\operatorname\{Var\}\(m\_\{t\}\)=v\(1\-\\beta\)^\{2\}\\sum\_\{r\\geq 0\}\\beta^\{2r\}=v\\frac\{1\-\\beta\}\{1\+\\beta\}\.Adding squared bias and variance proves equation[157](https://arxiv.org/html/2610.31728#A7.E157)\. For0<β<10<\\beta<1,

∂βR=−2​v\(1\+β\)2\+2​b2​β\(1−β\)3\.\\partial\_\{\\beta\}R=\-\\frac\{2v\}\{\(1\+\\beta\)^\{2\}\}\+\\frac\{2b^\{2\}\\beta\}\{\(1\-\\beta\)^\{3\}\}\.The functionβ​\(1\+β\)2/\(1−β\)3\\beta\(1\+\\beta\)^\{2\}/\(1\-\\beta\)^\{3\}increases strictly from zero to infinity, so forb≠0b\\neq 0the derivative has exactly one root and changes sign from negative to positive\. This gives equation[158](https://arxiv.org/html/2610.31728#A7.E158)\. Writingϵ=1−β∗\\epsilon=1\-\\beta^\{\*\}gives

ϵ3=\(b2/v\)​\(1−ϵ\)​\(2−ϵ\)2,\\epsilon^\{3\}=\(b^\{2\}/v\)\(1\-\\epsilon\)\(2\-\\epsilon\)^\{2\},which yields1−β∗∼\(4​b2/v\)1/31\-\\beta^\{\*\}\\sim\(4b^\{2\}/v\)^\{1/3\}asb2/v→0b^\{2\}/v\\to 0\.

###### Corollary G\.1\(Strict advantage over a shared constant memory\)\.

For finitely many directions, letvj\>0v\_\{j\}\>0,bj≠0b\_\{j\}\\neq 0, andwj\>0w\_\{j\}\>0, and letβj∗\\beta\_\{j\}^\{\*\}minimizeR⁡\(β,vj,bj\)R\(\\beta;v\_\{j\},b\_\{j\}\)\. If two ratiosbj2/vjb\_\{j\}^\{2\}/v\_\{j\}differ, then

∑jwj​R​\(βj∗,vj,bj\)<min⁡∑j0≤β<1⁡wj​R​\(β,vj,bj\)\.\\sum\_\{j\}w\_\{j\}R\(\\beta\_\{j\}^\{\*\};v\_\{j\},b\_\{j\}\)<\\min\_\{0\\leq\\beta<1\}\\sum\_\{j\}w\_\{j\}R\(\\beta;v\_\{j\},b\_\{j\}\)\.

###### Proof\.

The strict monotonicity in the optimality equation gives distinct unique minimizers whenever the ratiosbj2/vjb\_\{j\}^\{2\}/v\_\{j\}differ\. Every shared coefficient is therefore strictly suboptimal for at least one positive\-weight direction\. The shared objective is continuous and diverges asβ↑1\\beta\\uparrow 1, so it attains its minimum on a compact subset of\[0,1\)\[0,1\)\. Evaluating the strict pointwise inequality at that minimizer proves the claim\. ∎

The corollary compares constant coefficients in an exogenous affine\-signal estimation problem\. It is not a lower bound against every adaptive scalar\-momentum method and does not establish end\-to\-end acceleration for Muon\.

### Appendix HNumerical Experiments and Reproducibility Details

The synthetic experiments in this appendix diagnose the displayed theoretical mechanisms using the exact partial polar factor\. The preliminary language\-model pretraining comparison is documented separately in Appendix[H\.8](https://arxiv.org/html/2610.31728#A8.SS8)\.

#### H\.1Common protocol and map from main\-text figures

##### Where each main\-text figure is documented\.

##### Two replication notions\.

Unless specified otherwise, synthetic trajectory experiments useRtraj=64R\_\{\\rm traj\}=64independent mini\-batch streams per configuration, and frozen\-state checks useNMC=8192N\_\{\\rm MC\}=8192independent Monte Carlo replicates\. Experiment\-specific sample counts are given below, including larger samples for Gaussian coefficient estimates\. Trajectory runs measure variability of an entire optimization path, whereas frozen\-state replicates estimate a conditional expectation or coupling norm\. For antithetic estimators, uncertainty should be computed across independent pairs, because the two evaluations within each pair are dependent\. Unless stated otherwise, curves show means, trajectory ribbons show interquartile ranges, and confidence intervals are1\.961\.96standard errors over independent replicates\. Log–log exponents are ordinary least\-squares fits to the displayed grid points\. The pretraining comparison contains one run per method and has no repeated\-run uncertainty estimate\.

##### Exact Muon map and numerical invariances\.

Every synthetic Muon simulation uses the exact SVD partial polar factor rather than an approximate Newton–Schulz implementation\. IfA=U​diag⁡\(σ1,…\)​V⊤A=U\\operatorname\{diag\}\(\\sigma\_\{1\},\\ldots\)V^\{\\top\}andσmax\>0\\sigma\_\{\\max\}\>0, singular values are treated as numerical zeros only relative to the matrix scale, using a cutoff proportional toσmax\\sigma\_\{\\max\}; the zero matrix maps to zero\. This preserves positive\-scale invariance\. Deterministic regression tests verify positive\-scale invariance, left–right orthogonal equivariance, the rank\-one polar formula, and agreement between the full2×22\\times 2matrix update and the reduced two\-task recursion, all to errors below4×10−154\\times 10^\{\-15\}\.

##### Hierarchical trajectory experiment in Figure[2](https://arxiv.org/html/2610.31728#S4.F2)\.

We simulate the deterministic\-label associative\-memory recursion directly in task coordinates,

Wt\+1=Wt\+η​msgn⁡\(\(I−S⁡\(Wt\)\)​Qt\),B​qt∼Multinomial⁡\(B,p\),W0=0\.W\_\{t\+1\}=W\_\{t\}\+\\eta\\,\\operatorname\{msgn\}\\\!\\big\(\(I\-S\(W\_\{t\}\)\)Q\_\{t\}\\big\),\\qquad Bq\_\{t\}\\sim\\operatorname\{Multinomial\}\(B,p\),\\qquad W\_\{0\}=0\.There areM=4M=4frequency blocks of sizeC=6C=6\(K=24K=24tasks\) with block masses\(0\.55,0\.25,0\.13,0\.07\)\(0\.55,0\.25,0\.13,0\.07\), so each task in blockμ\\muhas probabilityp~μ/C\\widetilde\{p\}\_\{\\mu\}/C\. We useB∈\{8,64,256,1024\}B\\in\\\{8,64,256,1024\\\}, step sizeη=0\.02\\eta=0\.02, andT=800T=800updates\. For blockμ\\muwith index setℐμ\\mathcal\{I\}\_\{\\mu\}, the plotted margin is

mμ​\(t\)=1C​∑j∈ℐμ\(Wj​j​\(t\)−1C−1​∑i∈ℐμi≠jWi​j​\(t\)\)\.m\_\{\\mu\}\(t\)=\\frac\{1\}\{C\}\\sum\_\{j\\in\\mathcal\{I\}\_\{\\mu\}\}\\left\(W\_\{jj\}\(t\)\-\\frac\{1\}\{C\-1\}\\sum\_\{\\begin\{subarray\}\{c\}i\\in\\mathcal\{I\}\_\{\\mu\}\\\\ i\\neq j\\end\{subarray\}\}W\_\{ij\}\(t\)\\right\)\.The late growth rate is the OLS slope ofmμm\_\{\\mu\}against continuous timeτ=η​t\\tau=\\eta tover the last40%40\\%of the recorded trajectory\. The main text summarizes the resulting sampling collapse in panel \(b\) of Figure[2](https://arxiv.org/html/2610.31728#S4.F2); the raw small\- and large\-expected\-count trajectories are retained below for reference\.

Figure 7:Higher\-rank trajectory view underlying Figure[2](https://arxiv.org/html/2610.31728#S4.F2)\(b\)\.At small expected task count \(B=8B=8\), block margins grow at visibly different rates; at large expected task count \(B=1024B=1024\), all four blocks approach the unit full\-batch rate\. The main text uses the rate\-versus\-B​pjBp\_\{j\}collapse because it exposes the controlling expected\-count variable more directly\.Table 1:Quantitative summary of the controlled numerical checks\.Exact and reduced\-model validations are kept separate from transfer tests\. The isotropic and closure errors are finite\-signal/finite\-Monte\-Carlo quantities whose limiting prediction is zero; the full\-categorical label\-noise exponent is a stress test rather than a direct validation of the isotropic closure\.

#### H\.2Isotropic Gaussian mean response with an anisotropic signal

The first\-order theorem is directional, so we use a deliberately non\-isotropic signal rather than a multiple of the identity\. For eachd∈\{4,8\}d\\in\\\{4,8\\\}, define

Hd=diag⁡\(s1,…,sd\)‖diag⁡\(s1,…,sd\)‖F,\(s1,…,sd\)=geomspace⁡\(1,0\.08,d\)\.H\_\{d\}=\\frac\{\\operatorname\{diag\}\(s\_\{1\},\\ldots,s\_\{d\}\)\}\{\\\|\\operatorname\{diag\}\(s\_\{1\},\\ldots,s\_\{d\}\)\\\|\_\{F\}\},\\qquad\(s\_\{1\},\\ldots,s\_\{d\}\)=\\operatorname\{geomspace\}\(1,0\.08;d\)\.Forε∈\{0\.04,0\.07,0\.12,0\.20\}\\varepsilon\\in\\\{0\.04,0\.07,0\.12,0\.20\\\}we estimate

Fd​\(ε\)=𝔼​msgn⁡\(ε​Hd\+Ξ\)F\_\{d\}\(\\varepsilon\)=\\mathbb\{E\}\\,\\operatorname\{msgn\}\(\\varepsilon H\_\{d\}\+\\Xi\)using32,76832\{,\}768antithetic Gaussian pairs\. The coefficientcd=d−2​𝔼​‖Ξ‖∗c\_\{d\}=d^\{\-2\}\\mathbb\{E\}\\\|\\Xi\\\|\_\{\*\}is estimated independently from65,53665\{,\}536Gaussian matrices, so agreement does not arise from sharing Monte Carlo noise between response and theory\. We report both the scalar projection⟨F^d​\(ε\),Hd⟩/ε\\langle\\widehat\{F\}\_\{d\}\(\\varepsilon\),H\_\{d\}\\rangle/\\varepsilonand the full matrix residual

‖F^d​\(ε\)−cd​ε​Hd‖F‖cd​ε​Hd‖F\.\\frac\{\\\|\\widehat\{F\}\_\{d\}\(\\varepsilon\)\-c\_\{d\}\\varepsilon H\_\{d\}\\\|\_\{F\}\}\{\\\|c\_\{d\}\\varepsilon H\_\{d\}\\\|\_\{F\}\}\.Across the eight settings, the median full\-matrix relative error is3\.05%3\.05\\%\. For the directional\-gain panel in Figure[1](https://arxiv.org/html/2610.31728#S3.F1)\(a\), we additionally repeat thed=8d=8experiment with131,072131\{,\}072antithetic pairs and an independent131,072131\{,\}072\-sample estimate ofcdc\_\{d\}, then report the gain along each singular direction divided bycdc\_\{d\}\.

The corresponding response curves are shown in Figure[1](https://arxiv.org/html/2610.31728#S3.F1)in the main text\.

#### H\.3Two\-task finite\-batch transition, asymptotics, and momentum

##### Direct finite\-batch transition\.

To test Theorem[4\.1](https://arxiv.org/html/2610.31728#S4.Thmtheorem1)itself, we usepmin∈\{0\.02,0\.05,0\.10\}p\_\{\\min\}\\in\\\{0\.02,0\.05,0\.10\\\}and choose batch sizes spanning both sides ofBbalB\_\{\\rm bal\}\. For each configuration we estimate the late\-time growth rate of the gapD⁡\(t\)=Δ1​\(t\)−Δ2​\(t\)D\(t\)=\\Delta\_\{1\}\(t\)\-\\Delta\_\{2\}\(t\)across 64 trajectories and plot it againstB/BbalB/B\_\{\\rm bal\}\. In the small\-batch phase,

D⁡\(t\)η​t⟶2​\(2​p1B−1\),\\frac\{D\(t\)\}\{\\eta t\}\\longrightarrow\\sqrt\{2\}\\bigl\(2p\_\{1\}^\{B\}\-1\\bigr\),whereas the balanced phase has zero asymptotic gap slope\. The three frequency settings collapse aroundB/Bbal=1B/B\_\{\\rm bal\}=1and agree with the piecewise theoretical curve\.

The direct finite\-batch transition plot is panel \(a\) of Figure[2](https://arxiv.org/html/2610.31728#S4.F2)in the main text\.

##### Large\-batch asymptotics\.

We use the exact two\-task recursion of Appendix[C](https://arxiv.org/html/2610.31728#A3)withp2=0\.05p\_\{2\}=0\.05,p1=0\.95p\_\{1\}=0\.95,η=0\.003\\eta=0\.003,T=30,000T=30\{,\}000, and

B∈\{50,80,120,200,320,500,800\}\.B\\in\\\{50,80,120,200,320,500,800\\\}\.For each of the 64 trajectories we discard the first60%60\\%of updates\. Over the remaining segment we estimate the time\-averaged gapΔ1−Δ2\\Delta\_\{1\}\-\\Delta\_\{2\}and its stationary standard deviation \(sampling every five iterations\)\. The equilibrium target islog⁡\(p1/p2\)\\log\(p\_\{1\}/p\_\{2\}\)\. Regressing the mean equilibrium bias and the stationary standard deviation againstBBgives exponents−1\.011\-1\.011and−0\.531\-0\.531, respectively, close to the predicted−1\-1and−1/2\-1/2\.

Figure 8:Two\-task large\-batch asymptotics\.Both panels usep2=0\.05p\_\{2\}=0\.05,η=0\.003\\eta=0\.003, and 64 trajectories\. \(a\) Absolute late\-time bias ofΔ1−Δ2\\Delta\_\{1\}\-\\Delta\_\{2\}from the full\-batch targetlog⁡\(p1/p2\)\\log\(p\_\{1\}/p\_\{2\}\)versus batch sizeBBon log–log axes; the fitted exponent is−1\.011\-1\.011, compared with theB−1B^\{\-1\}prediction\. \(b\) Late\-time stationary standard deviation of the gap versusBB; the fitted exponent is−0\.531\-0\.531, compared with theB−1/2B^\{\-1/2\}diffusion prediction\.
##### Variance\-equivalent sampling under momentum\.

For the momentum\-collapse check we keepp2=0\.05p\_\{2\}=0\.05, useη=0\.002\\eta=0\.002andT=30,000T=30\{,\}000, and targetBeff∈\{40,80,160,320\}B\_\{\\rm eff\}\\in\\\{40,80,160,320\\\}\. For everyB∈\{2,4,8,16,32\}B\\in\\\{2,4,8,16,32\\\}withB≤BeffB\\leq B\_\{\\rm eff\}, we choose

β=Beff/B−1Beff/B\+1,\\beta=\\frac\{B\_\{\\rm eff\}/B\-1\}\{B\_\{\\rm eff\}/B\+1\},so thatB⁡\(1\+β\)/\(1−β\)=BeffB\(1\+\\beta\)/\(1\-\\beta\)=B\_\{\\rm eff\}\. The same 60% burn\-in convention is used\. For the tracking check we instead fixB=4B=4andβ=0\.95\\beta=0\.95, sweep

η∈\{×10−4,0\.002,0\.01,0\.03,0\.05,0\.10,0\.20\},\\eta\\in\\\{5\\\!\\times\\\!10^\{\-4\},0\.002,0\.01,0\.03,0\.05,0\.10,0\.20\\\},and choose

T=round⁡\(50/η\),T=\\operatorname\{round\}\(50/\\eta\),so that the continuous\-time horizon satisfiesτmax=η​T≈50\\tau\_\{\\max\}=\\eta T\\approx 50for every learning rate\. We again discard the first 60% of updates and compute the mean gap over the retained late\-time window; the plotted mean\-gap error is its absolute deviation from the full\-batch targetlog⁡\(p1/p2\)\\log\(p\_\{1\}/p\_\{2\}\)\. The measured error grows onceη/\(1−β\)\\eta/\(1\-\\beta\)is no longer small\.

The corresponding collapse/tracking summary is shown directly in Figure[5](https://arxiv.org/html/2610.31728#S5.F5)of the main text\.

#### H\.4Hierarchical well\-sampled perturbation scaling

These checks are deliberately*frozen\-state*Monte Carlo experiments rather than optimization trajectories\. We setM=3M=3,C=8C=8, block masses\(0\.5,0\.3,0\.2\)\(0\.5,0\.3,0\.2\), and test the third block\. To remove any unspecified initialization, the reference logit matrix is fixed deterministically as follows\. If columnjjbelongs to blockν\\nuand rowiito blockμ\\mu, then

Wi​j=\{2\+0\.1​ν,i=j,0\.3\+0\.05​ν,μ=ν,i≠j,−0\.2−0\.05​μ\+0\.03​ν,μ≠ν,W\_\{ij\}=\\begin\{cases\}2\+0\.1\\nu,&i=j,\\\\ 0\.3\+0\.05\\nu,&\\mu=\\nu,~i\\neq j,\\\\ \-0\.2\-0\.05\\mu\+0\.03\\nu,&\\mu\\neq\\nu,\\end\{cases\}with zero\-based block indicesμ,ν∈\{0,1,2\}\\mu,\\nu\\in\\\{0,1,2\\\}\. We setA=I−S⁡\(W\)A=I\-S\(W\), draw a multinomial frequency vectorqq, decompose it into its block\-average and within\-block partsq=q∥\+q⟂q=q^\{\\parallel\}\+q^\{\\perp\}, and estimate the derivative

msgn⁡\(A​diag⁡\(q∥\+h​q⟂\)\)−msgn⁡\(A​diag⁡\(q∥−h​q⟂\)\)2​h\.\\frac\{\\operatorname\{msgn\}\(A\\operatorname\{diag\}\(q^\{\\parallel\}\+hq^\{\\perp\}\)\)\-\\operatorname\{msgn\}\(A\\operatorname\{diag\}\(q^\{\\parallel\}\-hq^\{\\perp\}\)\)\}\{2h\}\.Each point below averagesNMCN\_\{\\rm MC\}independent draws ofqq\.

For cancellation we fixB=640B=640and useh∈\{10−2,×10−3,10−3,×10−4,10−4\}h\\in\\\{10^\{\-2\},3\\\!\\times\\\!10^\{\-3\},10^\{\-3\},3\\\!\\times\\\!10^\{\-4\},10^\{\-4\}\\\}\. For spatial scaling we fixh=10−4h=10^\{\-4\}and sweepB∈\{80,160,320,640,1280,2560\}B\\in\\\{80,160,320,640,1280,2560\\\}\. For momentum scaling we freeze the same state, setB=16B=16, and sweepβ∈\{0\.50,0\.75,0\.875,0\.94,0\.97\}\\beta\\in\\\{0\.50,0\.75,0\.875,0\.94,0\.97\\\}\. For eachβ\\beta, the EMA frequency vector is initialized atppand run for⌈12/\(1−β\)⌉\\lceil 12/\(1\-\\beta\)\\rceilmini\-batches before measuring the same finite\-difference derivative\. The resulting spatial and momentum exponents are−0\.512\-0\.512and−0\.509\-0\.509, respectively\.

Figure 9:Higher\-rank perturbation checks at a frozen block\-symmetric state\.\(a\) Mean finite\-difference derivative norm versus stephh: the contrast\-to\-contrast block decays to numerical zero while contrast\-to\-average coupling remainsO⁡\(1\)O\(1\), verifying first\-order cancellation\. \(b\) RMS contrast\-to\-average coupling versus expected task countB​p~μ/CB\\widetilde\{p\}\_\{\\mu\}/C; the fitted log–log slope is−0\.512\-0\.512, matching the predicted−1/2\-1/2\. \(c\) The same coupling versus variance\-equivalent task countBeff​p~μ/CB\_\{\\rm eff\}\\widetilde\{p\}\_\{\\mu\}/Cunder EMA momentum; the fitted slope is−0\.509\-0\.509\. Each point usesNMC=8192N\_\{\\rm MC\}=8192draws\.
#### H\.5Label\-noise large\-margin drift: projected validation and full\-matrix transfer

##### Projected large\-margin formula\.

The large\-margin driftχB,p\\chi\_\{B,p\}in equation[118](https://arxiv.org/html/2610.31728#A5.E118)is exact for the longitudinal projected counting model, not for the complete categorical matrix\. We validate that reduced model directly before testing transfer to full matrix Muon\. We useK=8K=8,pj=1/8p\_\{j\}=1/8, margin1212,α∈\{0\.05,0\.10,0\.20\}\\alpha\\in\\\{0\.05,0\.10,0\.20\\\}, and expected corrupted\-sample counts

λ=B​pj​ρα∈\{0\.20,0\.35,0\.50,log⁡2,1\.0,1\.4,2\.0\}\\lambda=Bp\_\{j\}\\rho\_\{\\alpha\}\\in\\\{0\.20,0\.35,0\.50,\\log 2,1\.0,1\.4,2\.0\\\}\(up to integer rounding ofBB\)\. Each point uses16,38416\{,\}384samples\. Direct projected Monte Carlo has RMSE0\.006180\.00618fromχB,p\\chi\_\{B,p\}\. The complete matrix update shows the same sign transition aroundλ≃log⁡2\\lambda\\simeq\\log 2but is not expected to equal the projected formula; its RMSE relative to the projected prediction is0\.02330\.0233\.

The full\-matrix transfer summary is shown in panel \(a\) of Figure[3](https://arxiv.org/html/2610.31728#S4.F3); the direct projected\-model validation is isolated below so that reduced\-model validation and full\-matrix transfer are not conflated\.

Figure 10:Direct validation of the projected large\-margin counting law\.Points are Monte Carlo estimates in the longitudinal projected model and dashed curves are the exact finite\-BBformula\. The full\-matrix transfer test is shown in Figure[3](https://arxiv.org/html/2610.31728#S4.F3)\(a\), where deviations from the projected formula are expected because transverse categorical modes are retained\.

#### H\.6Label\-noise local response: direct closure test and categorical stress test

##### Direct test of the stated isotropic covariance closure\.

We simulate exactly the Gaussian proxy used to derive equation[139](https://arxiv.org/html/2610.31728#A5.E139)\. We takeK=6K=6,α=0\.2\\alpha=0\.2,B=5000B=5000, local displacementh=0\.01h=0\.01, and frequency scales

p∈\{0\.02,0\.04,0\.08,0\.16,0\.32\}\.p\\in\\\{0\.02,0\.04,0\.08,0\.16,0\.32\\\}\.The projected signal and isotropic Gaussian covariance follow Appendix[E\.6](https://arxiv.org/html/2610.31728#A5.SS6), and each point uses32,76832\{,\}768antithetic samples\. Hereppis the frequency scale of the local proxy, not a common probability assigned to all six tasks in a normalized categorical distribution\. The fitted frequency exponent is0\.5020\.502and the mean relative error of the explicit finite\-KKcoefficient is0\.61%0\.61\\%\.

The direct Gaussian\-closure check is retained below; panel \(b\) of Figure[3](https://arxiv.org/html/2610.31728#S4.F3)instead shows the stronger full\-categorical frequency\-scaling stress test\.

Figure 11:Direct check of the moment\-matched isotropic covariance closure\.The Monte Carlo coefficient follows the predictedp1/2p^\{1/2\}scaling with fitted exponent0\.5020\.502\. This panel validates the stated Gaussian proxy itself; the main text uses the full\-categorical simulation as the more demanding stress test\.
##### Full\-categorical heterogeneous\-frequency stress test\.

We separately retain the stronger complete\-categorical simulation\. It usesK=6K=6,α=0\.2\\alpha=0\.2,B=5000B=5000, frequenciesp=\(0\.35,0\.25,0\.16,0\.11,0\.08,0\.05\)p=\(0\.35,0\.25,0\.16,0\.11,0\.08,0\.05\), and a local margin perturbationδ=0\.03\\delta=0\.03\. Each target task is evaluated withNMC=8192N\_\{\\rm MC\}=8192complete mini\-batches\. The empirical Muon frequency exponent is0\.493970\.49397, while the corresponding SGD exponent is11; the mean relative error against the auxiliary isotropic\-closure coefficient is3\.25%3\.25\\%\. Because the exact categorical covariance is anisotropic, this agreement is reported as a stress test beyond the closure assumptions, not as theorem\-level validation\.

The full\-categorical frequency\-scaling result is the quantity plotted in Figure[3](https://arxiv.org/html/2610.31728#S4.F3)\(b\)\.

#### H\.7Muon–SignSGD sparse\-support contrast

Figure[4](https://arxiv.org/html/2610.31728#S4.F4)uses the exact two\-task recursions from Appendices[C\.1](https://arxiv.org/html/2610.31728#A3.SS1)and[F](https://arxiv.org/html/2610.31728#A6)\. We setp1=0\.95p\_\{1\}=0\.95,p2=0\.05p\_\{2\}=0\.05,B=1B=1,η=0\.01\\eta=0\.01,T=3000T=3000, andθ=1\.15\\theta=1\.15radians, and run6464independent paired mini\-batch streams\. The same task draw is fed to Muon and SignSGD at every step\. For SignSGD, the positive answer\-embedding scalarc~\\widetilde\{c\}in equation[143](https://arxiv.org/html/2610.31728#A6.E143)is absorbed into the learning rate so that the experiment isolates the rotation\-dependent coupling\.

For Muon, aB=1B=1batch updates only the sampled task, giving asymptotic rates2​p1\\sqrt\{2\}p\_\{1\}and2​p2\\sqrt\{2\}p\_\{2\}per unit optimization time\. The rare\-task margin therefore remains nonnegative and grows on its occurrence clock\. For SignSGD,θ\>π/4\\theta\>\\pi/4makes every task\-1\-only step decrease the task\-2 margin, and the sufficient condition in Proposition[4\.5](https://arxiv.org/html/2610.31728#S4.Thmtheorem5)holds\. We record both margins every 20 steps and plot the median and interquartile range across the 64 paired runs\.

The final recorded medians atτ=η​T=30\\tau=\\eta T=30are approximately40\.2840\.28and2\.152\.15for the frequent and rare Muon margins, versus38\.4038\.40and−12\.35\-12\.35for SignSGD\. These trajectories are the two panels of Figure[4](https://arxiv.org/html/2610.31728#S4.F4)\.

#### H\.8Language\-model pretraining comparison

We compare standard Muon with the fast–slow construction of[Zhu et al\. \(2026\)](https://arxiv.org/html/2610.31728#bib.bib11)on a 0\.12B\-parameter dense language model trained for 30,000 steps\. The remaining training settings are matched\. The metric is validation loss, with one recorded trajectory per method\. The export contains 140 checkpoints; the first two have no matching Muon value, leaving 138 paired checkpoints at 200\-step intervals from step 2,600 to 30,000\. We use these paired observations without interpolation or smoothing\.

The fast–slow run has lower validation loss at every paired checkpoint\. Final losses are3\.0097363\.009736for Muon and2\.9888442\.988844for fast–slow momentum, a difference of0\.0208920\.020892\. The final\-ten\-checkpoint means are3\.0105683\.010568and2\.9906692\.990669, respectively\. Figure[6](https://arxiv.org/html/2610.31728#S6.F6)displays the 51 paired checkpoints from step 20,000 to 30,000\. Checkpoints within a trajectory are not independent training replications; these observations do not establish variability across seeds or isolate the contribution of each algorithmic component\.