Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

# Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
Source: [https://arxiv.org/html/2605.16341](https://arxiv.org/html/2605.16341)
Equal contribution:[tatsuhironm@keio\.jp,lpgomez@stanford\.edu](https://arxiv.org/html/2605.16341v1/[email protected],[email protected])Corresponding author:[hiroki11x@gmail\.com](https://arxiv.org/html/2605.16341v1/[email protected])

###### Abstract

Low\-rank gradient compression reduces communication in distributed training by representing updates with rank\-rrfactors\. Dion is a recent method that approximates Muon, a spectral optimizer that orthogonalizes momentum, using one step of power iteration followed by column normalization \(rescaling each column of the right factor to unit length\)\. This makes it compatible with fully sharded data parallel training, but it converges more slowly than full\-rank spectral methods\. We show that this gap is geometric: column normalization does not yield the rank\-rrpolar factor that Muon implicitly targets, so the resulting direction violates the dual\-norm constraint of the low\-rank spectral geometry, and the rate picks up an extra factor ofr\\sqrt\{r\}even though the low\-rank approximation of the gradient itself is accurate\. The same mismatch enters the smoothness term and the error\-feedback recursion in the analysis, which has a knock\-on effect on empirical performance\. We propose Orth\-Dion, which replaces column normalization with QR orthogonalization of the right factor\. Under non\-Euclidean smoothness, withLrL\_\{r\}the curvature constant along rank\-rrdirections, Orth\-Dion attains rateO​\(Lr/T\)O\(\\sqrt\{L\_\{r\}/T\}\), matching exact spectral methods at the same per\-step communication cost as Dion\. The proof removes the bounded\-drift assumption common in prior error\-feedback analyses via a self\-consistent fixed\-point argument, and uses a time\-averaged contraction that only requires the error sequence to contract on average rather than at every step\. Experiments on large\-scale language model pre\-training validate the predictedr\\sqrt\{r\}scaling and show that Orth\-Dion closes the convergence gap to Muon at Dion’s communication cost\.

## 1Introduction

Training large language models increasingly relies on distributed sharding schemes such as fully sharded data parallelism\(Zhaoet al\.,[2023](https://arxiv.org/html/2605.16341#bib.bib5)\), where optimizer design is constrained by both per\-step computation and communication\. Spectral optimizers such as Muon\(Jordanet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib8)\)are attractive in this setting because they exploit the matrix structure of neural network parameters\. Rather than applying an entrywise update, they orthogonalize momentum and take a step in a spectrally normalized direction\. This geometry has been empirically effective, but it is difficult to scale under sharding, since forming and orthogonalizing full momentum matrices requires additional collective communication\. Dion\(Ahnet al\.,[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\)addresses this bottleneck by replacing the full spectral update with a low\-rank factorized one\. Each matrix update is represented through rank\-rrfactors, which are computed by one step of power iteration and communicated at cost proportional to\(m\+n\)​r\(m\+n\)rrather thanm​nmn\.

At first glance, the remaining gap between Dion and full\-rank spectral methods appears to be the unavoidable price of using rank\-rrupdates\(Ahnet al\.,[2025b](https://arxiv.org/html/2605.16341#bib.bib3); Carlsonet al\.,[2015](https://arxiv.org/html/2605.16341#bib.bib2)\)\. Since Dion communicates only low\-rank factors, one might expect its slower convergence to come from missing gradient directions outside the selected subspace\. This interpretation is natural given the low\-rank, power\-iteration\-based compression lineage of PowerSGD and Dion\(Vogelset al\.,[2019](https://arxiv.org/html/2605.16341#bib.bib21); Ahnet al\.,[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\)\. Under this view, the problem is where the update points, and Dion loses because its rank\-rrsubspace is too small\. We show that this explanation is incomplete\. Dion’s power iteration can identify a useful low\-rank subspace, yet the subsequent update can still be shaped incorrectly inside that subspace\. The source is Dion’s final normalization step\. After power iteration, Dion normalizes the columns of the right factor independently\(Ahnet al\.,[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\)\. This preserves the span of the factor, but it does not produce the orthogonal right factor associated with a rank\-rrspectral update\(Carlsonet al\.,[2015](https://arxiv.org/html/2605.16341#bib.bib2); Bernstein and Newhouse,[2024b](https://arxiv.org/html/2605.16341#bib.bib1)\)\. As a result, Dion can move in essentially the same subspace as the desired low\-rank polar direction while using a direction that is incorrectly scaled for spectral descent\. We formalize this mismatch through a dual\-norm factorνt\\nu\_\{t\}, showing that column normalization can introduce a rank\-dependent penalty even when the low\-rank approximation itself is accurate\.

Orth\-Dion fixes this mismatch with the smallest possible change\. It keeps Dion’s power iteration, residual buffer, and low\-rank communication pattern unchanged, but replaces column normalization with QR orthogonalization of the right factor\. This makes the update geometrically aligned with the rank\-rrpolar direction targeted by spectral methods\. Under the same low\-rank communication cost as Dion, Orth\-Dion removes the rank\-dependent normalization penalty and recovers the leading convergence rate of exact rank\-rrspectral updates\. Because Dion\-family methods use error feedback, the proof must control the coupled evolution of the residual buffer and the tracked subspace\. We do this through a self\-consistent residual\-control argument and an amortized contraction condition, avoiding the bounded\-buffer\-drift assumption used in prior analyses\.

We measureνt\\nu\_\{t\}directly during Llama 3 320M pretraining\. Dion’sνt\\nu\_\{t\}rises with rank across every layer and logged step; Orth\-Dion’s holds atνt≈1\\nu\_\{t\}\\approx 1\. The mismatch is not a worst\-case anomaly; it is what Dion does in the regime it is meant for\. Holdingrrfixed and swapping column normalization for QR lowers validation loss at every rank we tested, which separates the QR fix from any rank\-capacity effect\. Because QR adds per\-step compute, we pair it with adaptive rank to obtain Ada\-Orth\-Dion: the working rank shrinks toward each layer’s intrinsic dimensionality, recovering Dion\-like step time on a 17\.1B model \(Figure[1](https://arxiv.org/html/2605.16341#S1.F1)\) while maintaining convergence gain\. The combined method gives a low\-rank distributed spectral optimizer with a better convergence–communication trade\-off than Dion\.

![Refer to caption](https://arxiv.org/html/2605.16341v1/x1.png)\(a\)Lower validation loss at matched rank\.LLaMA 320M, late\-stage convergence \(mean±\\pm2 std, full trajectory in Fig\.[3](https://arxiv.org/html/2605.16341#S8.F3)\)\.
![Refer to caption](https://arxiv.org/html/2605.16341v1/x2.png)\(b\)Faster wallclock at scale\.Per\-step time on a LLaMA 17\.1B 50\-step run \(mean±\\pm2 std\)\. Rank fractionrfr\_\{f\}is the spectral update’s rank divided by the parameter matrix’s full rank\.

Figure 1:Proposed methods improve both convergence and wallclock\.\(a\)At matched rank, Orth\-Dion and Ada\-Orth\-Dion reach Dion’s plateau earlier and keep going\.\(b\)Adaptive rank absorbs Orth\-Dion’s QR overhead and matches Dion’s per\-step time on the 17\.1B model\. \(Ada\-Orth\-Dion’s rank is pinned at0\.93​rf0\.93\\,r\_\{f\}, the steady\-state rank reached on the 320M model; see App\.[L\.2](https://arxiv.org/html/2605.16341#A12.SS2)\.\)#### Contributions\.

We \(i\) identify a geometric source of suboptimality in Dion: column normalization inflates the Ky Fan dual norm by up tor\\sqrt\{r\}, producing a rank\-dependent convergence penalty distinct from low\-rank approximation error; \(ii\) proposeOrth\-Dion, a one\-line ColNorm→\\toQR replacement that forcesνt=1\\nu\_\{t\}\{=\}1and recovers the exact low\-rank spectral rate at Dion’s communication cost; \(iii\) prove convergence under non\-Euclidean smoothness via a self\-consistent residual\-control argument that eliminates bounded\-buffer\-drift and allows amortized rather than per\-step contraction; and \(iv\) validate the mechanism and optimizer\-level impact through directνt\\nu\_\{t\}measurements, matched\-rank improvements on Llama 3 320M, and Ada\-Orth\-Dion matching Dion’s wall\-clock time at lower validation loss\.

## 2Background: Geometry of Rank\-Constrained Optimization

Steepest descent under norms\.Givenf:ℝm×n→ℝf:\\mathbb\{R\}^\{m\\times n\}\\to\\mathbb\{R\}, steepest descent under norm∥⋅∥\\\|\\cdot\\\|updatesXt\+1=Xt−η​Dt∗X\_\{t\+1\}=X\_\{t\}\-\\eta D\_\{t\}^\{\*\}withDt∗=arg​max‖D‖≤1⁡⟨∇f​\(Xt\),D⟩D\_\{t\}^\{\*\}=\\operatorname\*\{arg\\,max\}\_\{\\\|D\\\|\\leq 1\}\\langle\\nabla f\(X\_\{t\}\),D\\rangle\. SGD usesℓ2\\ell\_\{2\}; SignSGD\(Bernsteinet al\.,[2018](https://arxiv.org/html/2605.16341#bib.bib10)\)usesℓ∞\\ell\_\{\\infty\}; spectral methods use the operator norm\. For distributed training, we constrain updates to rank≤r\\leq r, yielding the*Ky Fanrr\-norm*geometry\. The Ky Fan r\-norm is the dual of the sum of the top\-r singular values, and can be written asDt∗=arg​max‖D‖\(r\)≤1⁡⟨Mt,D⟩D\_\{t\}^\{\*\}=\\operatorname\*\{arg\\,max\}\_\{\\\|D\\\|\_\{\(r\)\}\\leq 1\}\\langle M\_\{t\},D\\rangle, where‖D‖\(r\)=max⁡\{σ1​\(D\),‖D‖F/r\}\\\|D\\\|\_\{\(r\)\}=\\max\\\{\\sigma\_\{1\}\(D\),\\,\\\|D\\\|\_\{F\}/\\sqrt\{r\}\\\}\. The solution is the rank\-rrpolar factorPr​\(Mt\)=Ur​Vr⊤P\_\{r\}\(M\_\{t\}\)=U\_\{r\}V\_\{r\}^\{\\top\}from the truncated SVD, achieving⟨Mt,Dt∗⟩=‖Mt‖KF,r=∑i=1rσi​\(Mt\)\\langle M\_\{t\},D\_\{t\}^\{\*\}\\rangle=\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}=\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}\(M\_\{t\}\)\.

The two algorithms\.Both maintain bufferMt=Gt\+RtM\_\{t\}=G\_\{t\}\+R\_\{t\}and use one power iteration step\. The*sole difference*is the right\-factor normalization:

whereWt=Mt⊤​UtW\_\{t\}=M\_\{t\}^\{\\top\}U\_\{t\}andUt=orth⁡\(Mt​Vt−1\)U\_\{t\}=\\operatorname\{orth\}\(M\_\{t\}V\_\{t\-1\}\)\. The update isD^t=Ut​V¯t⊤\\hat\{D\}\_\{t\}=U\_\{t\}\\bar\{V\}\_\{t\}^\{\\top\}, and the error feedback isRt\+1=β​\(I−Ut​Ut⊤\)​MtR\_\{t\+1\}=\\beta\\,\(I\-U\_\{t\}U\_\{t\}^\{\\top\}\)\\,M\_\{t\}\. QR costsO​\(n​r2\)O\(nr^\{2\}\)which is negligible compared to theO​\(m​n​r\)O\(mnr\)power step sincer≪mr\\ll m\. The full procedure is given in Algorithm[1](https://arxiv.org/html/2605.16341#alg1)\.

Algorithm 1Orth\-Dion / Stripped Dion \(one step\)0:Gradient

Gt∈ℝm×nG\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}, residual

RtR\_\{t\}, right factor

Vt−1V\_\{t\-1\}, step size

η\\eta, EF coefficient

β\\beta
1:

Mt←Gt\+RtM\_\{t\}\\leftarrow G\_\{t\}\+R\_\{t\}⊳\\trianglerightBuffer = gradient \+ error feedback

2:

Ut←orth⁡\(Mt​Vt−1\)U\_\{t\}\\leftarrow\\operatorname\{orth\}\(M\_\{t\}V\_\{t\-1\}\)⊳\\trianglerightLeft factor via QR,O​\(m​r2\)O\(mr^\{2\}\)

3:

Wt←Mt⊤​UtW\_\{t\}\\leftarrow M\_\{t\}^\{\\top\}U\_\{t\}⊳\\trianglerightRight factor \(un\-normalized\),O​\(m​n​r\)O\(mnr\)

4:

V¯t←orth⁡\(Wt\)\\bar\{V\}\_\{t\}\\leftarrow\\operatorname\{orth\}\(W\_\{t\}\)⊳\\trianglerightOrth\-Dion: QR;Dion:ColNorm⁡\(Wt\)\\operatorname\{ColNorm\}\(W\_\{t\}\)

5:

D^t←Ut​V¯t⊤\\hat\{D\}\_\{t\}\\leftarrow U\_\{t\}\\bar\{V\}\_\{t\}^\{\\top\}⊳\\trianglerightLow\-rank update direction

6:

Xt\+1←Xt−η​D^tX\_\{t\+1\}\\leftarrow X\_\{t\}\-\\eta\\hat\{D\}\_\{t\}⊳\\trianglerightParameter update

7:

Rt\+1←β​\(Mt−Ut​\(Ut⊤​Mt\)\)R\_\{t\+1\}\\leftarrow\\beta\\,\(M\_\{t\}\-U\_\{t\}\(U\_\{t\}^\{\\top\}M\_\{t\}\)\)⊳\\trianglerightError feedback

8:

Vt←V¯tV\_\{t\}\\leftarrow\\bar\{V\}\_\{t\}⊳\\trianglerightWarm\-start for next step

FSDP communication\.Under FSDP, parameters are sharded across devices\. Each training step involves: \(1\) all\-gather to materialize full weights, \(2\) forward/backward pass, \(3\) reduce\-scatter to re\-shard gradients, \(4\) local optimizer step\. Muon requires an*additional*all\-gather/reduce\-scatter pair for the full momentum matrix during orthogonalization\. Dion and Orth\-Dion avoid this by communicating only the low\-rank factors \(O​\(\(m\+n\)​r\)O\(\(m\{\+\}n\)r\)vs\.O​\(m​n\)O\(mn\)\), making them compatible with FSDP sharding at rank\-proportional cost\.

Assumptions\.We use the following assumptions \(details in Appendix[A](https://arxiv.org/html/2605.16341#A1)\):\(A\)*Non\-Euclidean smoothness*:f​\(X\+Δ\)≤f​\(X\)\+⟨∇f​\(X\),Δ⟩\+Lr2​‖Δ‖\(r\)2f\(X\{\+\}\\Delta\)\\leq f\(X\)\+\\langle\\nabla f\(X\),\\Delta\\rangle\+\\frac\{L\_\{r\}\}\{2\}\\\|\\Delta\\\|\_\{\(r\)\}^\{2\}, whereLrL\_\{r\}captures curvature along low\-rank directions;\(A′\)*Frobenius gradient\-Lipschitz*:‖∇f​\(Y\)−∇f​\(X\)‖F≤LF​‖Y−X‖F\\\|\\nabla f\(Y\)\-\\nabla f\(X\)\\\|\_\{F\}\\leq L\_\{F\}\\\|Y\-X\\\|\_\{F\}\(independent of A in non\-Euclidean geometry\);\(B′\)*Gradient spectral gap*:σr​\(Gt\)−σr\+1​\(Gt\)≥Δgap\>0\\sigma\_\{r\}\(G\_\{t\}\)\-\\sigma\_\{r\+1\}\(G\_\{t\}\)\\geq\\Delta\_\{\\mathrm\{gap\}\}\>0;\(B′′\)*Small spectral tail*:σr\+1​\(Gt\)≤τ\\sigma\_\{r\+1\}\(G\_\{t\}\)\\leq\\tauwithτ=O​\(η\)\\tau=O\(\\eta\)on the relevant horizon;\(C′\)*Gradient bounds*:‖Gt‖F≤GF\\\|G\_\{t\}\\\|\_\{F\}\\leq G\_\{F\},κr​\(Gt\)≤κG\\kappa\_\{r\}\(G\_\{t\}\)\\leq\\kappa\_\{G\}\.

One\-step descent decomposition\.Under Assumption A, any updateD^t\\hat\{D\}\_\{t\}with‖D^t‖\(r\)=νt\\\|\\hat\{D\}\_\{t\}\\\|\_\{\(r\)\}=\\nu\_\{t\}yields \(proof in Appendix[D](https://arxiv.org/html/2605.16341#A4)\):

f​\(Xt\+1\)≤f​\(Xt\)−η​‖Gt‖KF,r\+η​\(δt\+\(1\+νt\)​‖Rt‖KF,r\)\+Lr​νt22​η2,f\(X\_\{t\+1\}\)\\leq f\(X\_\{t\}\)\-\\eta\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\+\\eta\\big\(\\delta\_\{t\}\+\(1\{\+\}\\nu\_\{t\}\)\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\big\)\+\\tfrac\{L\_\{r\}\\nu\_\{t\}^\{2\}\}\{2\}\\eta^\{2\},\(1\)whereδt=‖Mt‖KF,r−⟨Mt,D^t⟩\\delta\_\{t\}=\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\-\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangleis the oracle defect\. The dual norm factorνt\\nu\_\{t\}appears in two critical places: the smoothness penaltyLr​νt2​η2/2L\_\{r\}\\nu\_\{t\}^\{2\}\\eta^\{2\}/2and the residual coupling\(1\+νt\)​‖Rt‖KF,r\(1\{\+\}\\nu\_\{t\}\)\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\.

## 3Ther\\sqrt\{r\}Inefficiency

Column normalization producesV¯t\\bar\{V\}\_\{t\}with unit\-length columns, giving Gram matrixV¯t⊤​V¯t\\bar\{V\}\_\{t\}^\{\\top\}\\bar\{V\}\_\{t\}with diagonal entries 1 and potentially large off\-diagonal entries\. Since this is a correlation matrix:

###### Proposition 3\.1\(r\\sqrt\{r\}bound\)\.

For column\-normalizedV¯t\\bar\{V\}\_\{t\}:νt=‖D^t‖\(r\)=‖V¯t‖op∈\[1,r\]\\nu\_\{t\}=\\\|\\hat\{D\}\_\{t\}\\\|\_\{\(r\)\}=\\\|\\bar\{V\}\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\in\[1,\\sqrt\{r\}\]\. Equalityνt=1\\nu\_\{t\}\{=\}1holds only when columns ofWtW\_\{t\}are already orthogonal\.

Howνt\\nu\_\{t\}enters the rate\.Telescoping \([1](https://arxiv.org/html/2605.16341#S2.E1)\) and optimizingη\\eta:

mint⁡‖Gt‖KF,r≤2​\(f0−f∞\)​Lr​ν2T\+O​\(1/T\),ν=maxt⁡νt\.\\min\_\{t\}\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq\\sqrt\{\\tfrac\{2\(f\_\{0\}\{\-\}f\_\{\\infty\}\)L\_\{r\}\\nu^\{2\}\}\{T\}\}\+O\(1/T\),\\quad\\nu=\\max\_\{t\}\\nu\_\{t\}\.\(2\)Withν=r\\nu\{=\}\\sqrt\{r\}\(ColNorm\): rate=O​\(Lr​r/T\)=O\(\\sqrt\{L\_\{r\}r/T\}\)\. Withν=1\\nu\{=\}1\(Orth\-Dion\): rate=O​\(Lr/T\)=O\(\\sqrt\{L\_\{r\}/T\}\)\. Ther\\sqrt\{r\}gap is purely geometric\.

Geometric interpretation\.The Ky Fan dual ball\{D:‖D‖\(r\)≤1\}\\\{D:\\\|D\\\|\_\{\(r\)\}\{\\leq\}1\\\}constrains both‖D‖op≤1\\\|D\\\|\_\{\\mathrm\{op\}\}\{\\leq\}1and‖D‖F≤r\\\|D\\\|\_\{F\}\{\\leq\}\\sqrt\{r\}\. The polar factorPr​\(Mt\)P\_\{r\}\(M\_\{t\}\)lies on its boundary \(∥⋅∥op=1\\\|\\cdot\\\|\_\{\\mathrm\{op\}\}\{=\}1,∥⋅∥F=r\\\|\\cdot\\\|\_\{F\}\{=\}\\sqrt\{r\}\)\. ColNorm preserves∥⋅∥F=r\\\|\\cdot\\\|\_\{F\}\{=\}\\sqrt\{r\}but allows∥⋅∥op\\\|\\cdot\\\|\_\{\\mathrm\{op\}\}up tor\\sqrt\{r\}—pushing the update*outside*the dual ball and incurring a factor\-rrsmoothness penalty\. The operator norm constraint‖D^t‖op≤1\\\|\\hat\{D\}\_\{t\}\\\|\_\{\\mathrm\{op\}\}\{\\leq\}1is the binding constraint for any rank\-rrpartial isometry\. ColNorm violates it by concentrating singular values non\-uniformly \(σmax​\(D^t\)≫σmin​\(D^t\)\\sigma\_\{\\max\}\(\\hat\{D\}\_\{t\}\)\\gg\\sigma\_\{\\min\}\(\\hat\{D\}\_\{t\}\)\); QR orthogonalization forces all singular values to 1 by construction\. The key insight: the inefficiency is not in*which subspace*is selected \(Lemma[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3)shows both algorithms select the same subspace\) but in*how the update is scaled within that subspace*\.

## 4Orth\-Dion: Correcting the Geometry

###### Lemma 4\.1\(Partial isometry,νt=1\\nu\_\{t\}\{=\}1\)\.

IfU∈ℝm×rU\{\\in\}\\mathbb\{R\}^\{m\\times r\},V¯∈ℝn×r\\bar\{V\}\{\\in\}\\mathbb\{R\}^\{n\\times r\}both have orthonormal columns, thenD^=U​V¯⊤\\hat\{D\}\{=\}U\\bar\{V\}^\{\\top\}has‖D^‖\(r\)=1\\\|\\hat\{D\}\\\|\_\{\(r\)\}\{=\}1\.

###### Lemma 4\.2\(Non\-negative defect\)\.

For Orth\-Dion,δt≥0\\delta\_\{t\}\\geq 0for alltt\.*\(By Hölder:⟨Mt,D^t⟩≤‖Mt‖KF,r⋅1\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangle\\leq\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\cdot 1\.\)*

###### Lemma 4\.3\(Same tracking\)\.

The projector errorεt\\varepsilon\_\{t\}satisfies the same recursion for both algorithms, sincecol⁡\(orth⁡\(W\)\)=col⁡\(ColNorm⁡\(W\)\)\\operatorname\{col\}\(\\operatorname\{orth\}\(W\)\)=\\operatorname\{col\}\(\\operatorname\{ColNorm\}\(W\)\)and the error feedback depends only onUtU\_\{t\}\.

Consequence\.The improvement is*entirely*inνt\\nu\_\{t\}, not approximation quality\. Lemma[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3)ensures that both algorithms track the same gradient subspace, since the column spaces oforth⁡\(W\)\\operatorname\{orth\}\(W\)andColNorm⁡\(W\)\\operatorname\{ColNorm\}\(W\)are identical\. At typical ranks, the quantitative impact is substantial\. Forr=256r\{=\}256, the smoothness cost ratio isLrν2/Lr=r=256×L\_\{r\}\\nu^\{2\}/L\_\{r\}=r=256\\times, while the residual coupling ratio is\(1\+r\)/\(1\+1\)=\(1\+16\)/2=8\.5×\(1\{\+\}\\sqrt\{r\}\)/\(1\{\+\}1\)=\(1\{\+\}16\)/2=8\.5\\times\. These factors compound, inflating the leading convergence constant byr\\sqrt\{r\}through both the smoothness and coupling terms\.

![Refer to caption](https://arxiv.org/html/2605.16341v1/x3.png)\(a\)Layer\-meanν¯t\\bar\{\\nu\}\_\{t\}vs\. training step\.
![Refer to caption](https://arxiv.org/html/2605.16341v1/x4.png)\(b\)Post\-warmup pooledνt\\nu\_\{t\}distribution\.

Figure 2:The dual\-norm mismatch is real, rank\-ordered, and persistent\(Llama 3 320M,r∈\{96,192,384\}r\{\\in\}\\\{96,192,384\\\}, mean over33seeds; band±\\pmstd\)\.\(a\)Dion’sν¯t\\bar\{\\nu\}\_\{t\}separates cleanly by rank and stays well above11throughout training; the right axis showsν¯t2\\bar\{\\nu\}\_\{t\}^\{2\}, the factor through whichνt\\nu\_\{t\}enters the smoothness term in \([1](https://arxiv.org/html/2605.16341#S2.E1)\)\.\(b\)Per\-updateνt\\nu\_\{t\}pooled over layer, step, and seed: the inflation is dispersed, not transient\. Orth\-Dion collapses toνt≈1\\nu\_\{t\}\{\\approx\}1in both views \(max≈1\.003\\approx 1\.003\)\.The boundνt=r\\nu\_\{t\}=\\sqrt\{r\}is attained only when all columns ofWtW\_\{t\}are colinear\. This pathological case does not arise in practice\. But the inflation it predicts is real, rank\-ordered, and persistent throughout training\. Figure[2](https://arxiv.org/html/2605.16341#S4.F2)measuresνt\\nu\_\{t\}during Llama 3 320M pretraining atr∈\{96,192,384\}r\\in\\\{96,192,384\\\}\. Figure[2\(a\)](https://arxiv.org/html/2605.16341#S4.F2.sf1)shows the time\-course: Dion’s curves separate cleanly by rank \(peaks≈2\.1,2\.5,2\.9\\approx\\\!2\.1,2\.5,2\.9, i\.e\.ν¯t2≈4\.5−8\.6\\bar\{\\nu\}\_\{t\}^\{2\}\\approx 4\.5\{\-\}8\.6\) and stay strictly above11throughout\. Figure[2\(b\)](https://arxiv.org/html/2605.16341#S4.F2.sf2)shows the per\-update dispersion: atr=384r\{=\}384,∼33%\\sim\\\!33\\%of individual updates carryνt\>2\\nu\_\{t\}\>2and∼10%\\sim\\\!10\\%carryνt\>3\\nu\_\{t\}\>3\. In our runsνt\\nu\_\{t\}never approaches the worst\-caser\\sqrt\{r\}\. It does climb with rank, though, and stays elevated through training\. Orth\-Dion’sνt\\nu\_\{t\}holds at11everywhere we measured\. No hyperparameter setting fixes this; the problem is in the normalization step itself\. Proofs of Lemmas[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1)–[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3)are in Appendix[E](https://arxiv.org/html/2605.16341#A5)\.

## 5Convergence Theory

### 5\.1Self\-Consistent Drift Elimination

The original Dion analysis assumes bounded buffer drift \(‖Mt−Mt−1‖op≤Δdrift\\\|M\_\{t\}\{\-\}M\_\{t\-1\}\\\|\_\{\\mathrm\{op\}\}\\leq\\Delta\_\{\\mathrm\{drift\}\}, Assumption D\), but this is circular:Mt−Mt−1M\_\{t\}\{\-\}M\_\{t\-1\}contains the residual increment, which is itselfO​\(‖Gt‖\)O\(\\\|G\_\{t\}\\\|\)rather than small\. We eliminate this assumption by exploiting an asymmetry: under Assumption A′, the*gradient*subspace changes slowly \(sin⁡Θmax​\(PGt\+1,PGt\)≤LF​η​r/Δgap=O​\(η\)\\sin\\Theta\_\{\\max\}\(P\_\{G\}^\{t\+1\},P\_\{G\}^\{t\}\)\\leq L\_\{F\}\\,\\eta\\sqrt\{r\}/\\Delta\_\{\\mathrm\{gap\}\}=O\(\\eta\)\), while the buffer subspace may change abruptly\. We track the former and*derive*the latter via a self\-consistency loop on the coupled system\(εt,ℛt\)\(\\varepsilon\_\{t\},\\mathcal\{R\}\_\{t\}\), whereεt=sin⁡Θmax​\(PMt,PGt\)\\varepsilon\_\{t\}=\\sin\\Theta\_\{\\max\}\(P\_\{M\}^\{t\},P\_\{G\}^\{t\}\)is the tracking error andℛt=‖Rt‖F/‖Gt‖F\\mathcal\{R\}\_\{t\}=\\\|R\_\{t\}\\\|\_\{F\}/\\\|G\_\{t\}\\\|\_\{F\}the residual ratio\. The recursionsεt\+1≤γ~eff​\(1\+κG\)​εt\+O​\(η\)\\varepsilon\_\{t\+1\}\\leq\\tilde\{\\gamma\}\_\{\\mathrm\{eff\}\}\(1\{\+\}\\kappa\_\{G\}\)\\varepsilon\_\{t\}\+O\(\\eta\)andℛt\+1≤h​\(εt,ℛt\)\\mathcal\{R\}\_\{t\+1\}\\leq h\(\\varepsilon\_\{t\},\\mathcal\{R\}\_\{t\}\)form a contractive mapΦ\\Phion\[0,1\]2\[0,1\]^\{2\}; forη=O​\(Δgap/\(LF​r​κG\)\)\\eta=O\(\\Delta\_\{\\mathrm\{gap\}\}/\(L\_\{F\}\\sqrt\{r\}\\kappa\_\{G\}\)\), Banach fixed\-point yields a unique fixed point withℛ∞,ε∞=O​\(η\)\\mathcal\{R\}\_\{\\infty\},\\varepsilon\_\{\\infty\}=O\(\\eta\)*provided*the spectral tail is also small \(Assumption B′′\); without B′′the conclusion isℛ∞=O​\(τ\+η\)\\mathcal\{R\}\_\{\\infty\}=O\(\\tau\+\\eta\)\. The residual recursion is controlled by the spectral tailσr\+1​\(Mt\)\\sigma\_\{r\+1\}\(M\_\{t\}\)plus a tracking\-leakage termεt​σ1​\(Mt\)\\varepsilon\_\{t\}\\sigma\_\{1\}\(M\_\{t\}\)\(Appendix[F](https://arxiv.org/html/2605.16341#A6)\)\.

### 5\.2Amortized Contraction

The per\-step contractionρt=γ~t​\(1\+κG,t\)\\rho\_\{t\}=\\tilde\{\\gamma\}\_\{t\}\(1\{\+\}\\kappa\_\{G,t\}\)may exceed11during warmup or landscape transitions, so a uniform per\-step requirement is too strong\. We instead impose*uniform amortized contraction*: there existρ¯∈\(0,1\)\\bar\{\\rho\}\\in\(0,1\),Cρ≥1C\_\{\\rho\}\\geq 1such that

∏i=st−1ρi≤Cρ​ρ¯t−s,0≤s<t\.\\prod\_\{i=s\}^\{t\-1\}\\rho\_\{i\}\\;\\leq\\;C\_\{\\rho\}\\,\\bar\{\\rho\}^\{\\,t\-s\},\\qquad 0\\leq s<t\.Every suffix product decays geometrically up to a constant, allowing transientρt\>1\\rho\_\{t\}\>1while ruling out persistent growth\. Under this condition the tracking\-error recursionεt≤ρt−1​εt−1\+ct−1\\varepsilon\_\{t\}\\leq\\rho\_\{t\-1\}\\varepsilon\_\{t\-1\}\+c\_\{t\-1\}unrolls to geometric convergence up to a bounded bias term \(Theorem[G\.1](https://arxiv.org/html/2605.16341#A7.Thmtheorem1), Appendix[G](https://arxiv.org/html/2605.16341#A7)\)—the natural Lyapunov\-style stability condition for subspace tracking, where stability is governed by contraction over windows rather than at every step\.

### 5\.3Main Theorem

###### Theorem 5\.1\(Convergence of Orth\-Dion\)\.

Under Assumptions A, B’, C’ withβ=1\\beta\{=\}1,ρ′<1\\rho^\{\\prime\}\{<\}1, andη=c/T\\eta=c/\\sqrt\{T\}:

min0≤t<T⁡‖Gt‖KF,r≤2​\(f0−f∞\)​LrT\+O​\(1T\)\\min\_\{0\\leq t<T\}\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq\\sqrt\{\\frac\{2\(f\_\{0\}\-f\_\{\\infty\}\)L\_\{r\}\}\{T\}\}\+O\\\!\\left\(\\frac\{1\}\{T\}\\right\)\(3\)

The leading constant matches exact\-SVD Muon, whereν=1\\nu\{=\}1, and improves on Stripped Dion’s2​\(f0−f∞\)​Lr​r/T\\sqrt\{2\(f\_\{0\}\{\-\}f\_\{\\infty\}\)L\_\{r\}r/T\}, whereν=r\\nu\{=\}\\sqrt\{r\}, by a factor ofr\\sqrt\{r\}\. The full per\-step cost isO​\(m​n​r\)\+O​\(n​r2\)O\(mnr\)\+O\(nr^\{2\}\), matching Stripped Dion\. The improvement is geometric, since Orth\-Dion achievesν=1\\nu\{=\}1by construction and eliminates the rank\-dependent inflation in the smoothness term\. The proof appears in Appendix[H](https://arxiv.org/html/2605.16341#A8)\.

## 6Error Feedback and theβ<1\\beta<1Regimes

The one\-step descent bound \([1](https://arxiv.org/html/2605.16341#S2.E1)\) contains two rank\-dependent terms: the smoothness penaltyLr​νt2​η2/2L\_\{r\}\\nu\_\{t\}^\{2\}\\eta^\{2\}/2, addressed in Sections[3](https://arxiv.org/html/2605.16341#S3)–[4](https://arxiv.org/html/2605.16341#S4)viaνt=1\\nu\_\{t\}\{=\}1, and the residual coupling\(1\+νt\)​‖Rt‖KF,r\(1\{\+\}\\nu\_\{t\}\)\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}, whose dynamics depend onβ\\beta\. With fixedνt\\nu\_\{t\}, the residual recursionRt\+1=β​\(I−Ut​Ut⊤\)​\(Gt\+Rt\)R\_\{t\+1\}=\\beta\\,\(I\-U\_\{t\}U\_\{t\}^\{\\top\}\)\(G\_\{t\}\{\+\}R\_\{t\}\)is governed by the cross\-step correlation of the out\-of\-subspace gradient\. Characterizing this correlation yields a0\.3150\.315\-nat improvement on GPT\-2 45M at zero additional compute \(Table[2](https://arxiv.org/html/2605.16341#S8.T2)\) and shows that EF is mechanistically distinct from heavy\-ball momentum\. Define the*gradient subspace persistence*ϕt:=cos⁡∠​\(\(I−Pt\)​Gt,\(I−Pt−1\)​Gt−1\)\\phi\_\{t\}:=\\cos\\angle\\big\(\(I\{\-\}P\_\{t\}\)G\_\{t\},\\;\(I\{\-\}P\_\{t\-1\}\)G\_\{t\-1\}\\big\)and the*R\-norm ratio*ℛt:=‖Rt‖F/‖Gt‖F\\mathcal\{R\}\_\{t\}:=\\\|R\_\{t\}\\\|\_\{F\}/\\\|G\_\{t\}\\\|\_\{F\}\. Details in Appendix[K](https://arxiv.org/html/2605.16341#A11)\.

Regime 1 \(Coherent,ϕt\>0\\phi\_\{t\}\{\>\}0\):Out\-of\-subspace components persist across steps, soRT≈∑t\(I−Pt\)​GtR\_\{T\}\\approx\\sum\_\{t\}\(I\{\-\}P\_\{t\}\)G\_\{t\}accumulates coherently\. The subspace eventually rotates to capture this direction;β=1\\beta\{=\}1is optimal\.

Regime 2 \(Stochastic,ϕt≈0\\phi\_\{t\}\{\\approx\}0\):Typical of LLM training with stochastic mini\-batches\. The residual performs a random walk, acting as*implicit heavy\-ball momentum*with effective coefficientε^2\\hat\{\\varepsilon\}^\{2\}, whereε^t=‖\(I−Pt\)​Mt‖F/‖Mt‖F\\hat\{\\varepsilon\}\_\{t\}=\\\|\(I\{\-\}P\_\{t\}\)M\_\{t\}\\\|\_\{F\}/\\\|M\_\{t\}\\\|\_\{F\}is the tracking error\. Reducingβ\\betabelow 1 boosts this effect at zero extra compute\. The optimalβ\\betasatisfiesβ∗≈1−2​ε^\\beta^\{\*\}\\approx 1\{\-\}2\\hat\{\\varepsilon\}; for GPT\-2 with baseline tracking errorε^≈0\.35\\hat\{\\varepsilon\}\{\\approx\}0\.35measured atβ=1\\beta\{=\}1, this predictsβ∗≪1\\beta^\{\*\}\\\!\\ll\\\!1\. Empirically, val loss decreases monotonically fromβ=1\\beta\{=\}1down toβ≈0\.1\\beta\{\\approx\}0\.1\(Table[2](https://arxiv.org/html/2605.16341#S8.T2)\), andℛt∗\\mathcal\{R\}\_\{t\}^\{\*\}grows from 0\.3 to 3\.2 over the same range, directly tracking the implicit\-momentum accumulation the theory predicts\. Explicit Polyak momentum is not a substitute: PolyakDion \(μ=0\.95\\mu\{=\}0\.95,R≡0R\{\\equiv\}0\) reaches val\. loss3\.1623\.162vs\.3\.0563\.056for Dion atβ=0\.3\\beta\{=\}0\.3, since EF reshapes the buffer*before*compression while explicit momentum acts in the full parameter space\.

Regime 3 \(Anti\-correlated,ϕt<0\\phi\_\{t\}\{<\}0\):Residual components cancel across steps; the buffer diverges and EF is harmful\.β→0\\beta\{\\to\}0is necessary\.

This framework extends EF21\(Richtáriket al\.,[2021](https://arxiv.org/html/2605.16341#bib.bib20)\), which assumes independent lost signal \(Regime 2 only\) and misses the coherent and anti\-correlated cases\. Full derivations are in Appendix[K](https://arxiv.org/html/2605.16341#A11)\.

## 7Ada\-Orth\-Dion: Adaptive Rank as Intrinsic Dimensionality

Although Orth\-Dion achieves convergence improvement with a minimal fix, the extra QR orthogonalization adds compute overhead per step\. To mitigate this and match Dion’s wall\-clock speed, we couple Orth\-Dion with a per\-layer adaptive rank mechanism to reduce the rank and the overhead, yielding*Ada\-Orth\-Dion*\.

From contraction to intrinsic dimensionality\.The contraction conditionγ~i,r​\(1\+κG,i,r\)<1\\tilde\{\\gamma\}\_\{i,r\}\(1\{\+\}\\kappa\_\{G,i,r\}\)<1governing Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)decomposes into two competing terms per layerii:γ~i,r=σr\+1​\(Mi,t\)/σr​\(Mi,t\)\\tilde\{\\gamma\}\_\{i,r\}=\\sigma\_\{r\{\+\}1\}\(M\_\{i,t\}\)/\\sigma\_\{r\}\(M\_\{i,t\}\)*decreases*withrr\(more signal captured\), whileκG,i,r=σ1​\(Gi,t\)/σr​\(Gi,t\)\\kappa\_\{G,i,r\}=\\sigma\_\{1\}\(G\_\{i,t\}\)/\\sigma\_\{r\}\(G\_\{i,t\}\)*increases*withrr\(the tail is harder to track\)\. The critical rankri∗=max⁡\{r:γ~i,r​\(1\+κG,i,r\)<1\}r\_\{i\}^\{\*\}=\\max\\\{r:\\tilde\{\\gamma\}\_\{i,r\}\(1\{\+\}\\kappa\_\{G,i,r\}\)<1\\\}marks the signal\-to\-noise transition and is the layer’s*intrinsic dimensionality*: above it, power iteration diverges; below it, the spectral update is provably contracting\.

Lightweight estimator\.We approximateri∗r\_\{i\}^\{\*\}from Dion’s existing factors with no extra SVD:erank^​\(Mt\)=exp⁡\(−∑ip^i​log⁡p^i\)\\widehat\{\\operatorname\{erank\}\}\(M\_\{t\}\)=\\exp\(\-\\sum\_\{i\}\\hat\{p\}\_\{i\}\\log\\hat\{p\}\_\{i\}\)wherep^i=σ^i/∑jσ^j\\hat\{p\}\_\{i\}=\\hat\{\\sigma\}\_\{i\}/\\sum\_\{j\}\\hat\{\\sigma\}\_\{j\}andσ^i=‖\[Wt\]i‖2\\hat\{\\sigma\}\_\{i\}=\\\|\[W\_\{t\}\]\_\{i\}\\\|\_\{2\}\. The rank is updated via EMA smoothing \(α\\alpha\), a buffer multiplier \(γ=1\.1\\gamma\{=\}1\.1\), clipping, and rounding to multiples of88for GPU efficiency \(full policy in Algorithm[4](https://arxiv.org/html/2605.16341#alg4), Appendix[B\.2](https://arxiv.org/html/2605.16341#A2.SS2)\)\.

QR remains essential at adaptive rank\.Reducingrr*does not*substitute for fixingνt\\nu\_\{t\}\. Proposition[3\.1](https://arxiv.org/html/2605.16341#S3.Thmtheorem1)boundsνt\\nu\_\{t\}by the working rank, so a ColNorm\-based adaptive scheme inherits ther\\sqrt\{r\}penalty at whatever rank it selects\. Ada\-Orth\-Dion combines two complementary improvements, with QR removing the rank\-dependent geometric inflation, as shown in Lemma[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1), and adaptive rank trimming the working rank to its intrinsic dimensionality\. Empirically \(Table[1](https://arxiv.org/html/2605.16341#S8.T1)\), Ada\-Orth\-Dion settles at avg\. rank∼357\\sim\\\!357from ar=384r\{=\}384start, achieves the lowest validation loss of any method we evaluate that communicates strictly less than the parameter matrix \(3\.2823\.282\), and reaches Dion’s best loss18\.3%18\.3\\%faster than Dion at matched starting rank\.

## 8Experiments

Our experiments answer: \(1\) Does replacing ColNorm with QR improve convergence at matched rank, as predicted by Lemma 4\.1? \(2\) Does Ada\-Orth\-Dion recover Dion’s per\-step wall\-clock on a larger model? \(3\) How does the error\-feedback coefficientβ\\betainteract with the corrected geometry?

### 8\.1LLM Pre\-training: Llama 3 320M

Setup\.320M\-param Llama 3\(Grattafioriet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib13)\)on C4\(Raffelet al\.,[2020](https://arxiv.org/html/2605.16341#bib.bib14)\), 3\.2B tokens \(Chinchilla\-optimal\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.16341#bib.bib15)\)\), seq len 2048, 8×\\timesGH200 with FSDP\. All methods use cosine LR schedule with warmup,β=1\.0\\beta\{=\}1\.0\(unless stated\), and identical hyperparameters except the normalization\. Baselines: AdamW \(β1=0\.95\\beta\_\{1\}\{=\}0\.95,β2=0\.95\\beta\_\{2\}\{=\}0\.95\), Muon \(full\-rank with Newton–Schulz\), Dion \(r∈\{96,192,384\}r\{\\in\}\\\{96,192,384\\\}with ColNorm\), Orth\-Dion \(same ranks with QR\), and Ada\-Orth\-Dion \(adaptive rank, maximum rankrm​a​xr\_\{max\}at 384, rank initialized torm​a​xr\_\{max\}\)\.

Table 1:Llama 3 320M pre\-training on C4 \(3\.2B tokens, 8×\\timesGH200\)\. Mean±\\pm2 std over 3 seeds\. The steps reduction represents the percent reduction in steps to reach the Dion’s minimum validation loss at equivalent rank\.Findings\(Table[1](https://arxiv.org/html/2605.16341#S8.T1)\)\.*\(a\) Matched\-rank improvement\.*Orth\-Dion lowers val\. loss over Dion at every rank and reaches Dion’s best loss in1212–13%13\\%fewer steps, which equates to∼10%\\sim 10\\%reduction in wall\-clock time \(Figure[3\(a\)](https://arxiv.org/html/2605.16341#S8.F3.sf1)\)\. This confirms that replacing ColNorm with QR improves convergence\.*\(b\) Direct theory validation\.*Dion’s measuredν¯t\\bar\{\\nu\}\_\{t\}inflates monotonically withrr\(1\.20→1\.28→1\.341\.20\{\\to\}1\.28\{\\to\}1\.34\) while Orth\-Dion hasν¯t=1\.00\\bar\{\\nu\}\_\{t\}=1\.00across all ranks, confirming Proposition[3\.1](https://arxiv.org/html/2605.16341#S3.Thmtheorem1)and Lemma[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1)\.*\(c\) Adaptive rank\.*Ada\-Orth\-Dion beats all low\-rank baselines at an average rank of356\.9356\.9\(93% of Dion’sr=384r=384\)\. The rank reduction can lead to wall\-clock time improvement in large scale models, as discussed in Section[8\.3](https://arxiv.org/html/2605.16341#S8.SS3)\.

### 8\.2Measuringνt\\nu\_\{t\}at LLM Scale

The central theoretical prediction is that Dion’s convergence degrades asr\\sqrt\{r\}while Orth\-Dion is rank\-independent\. We validate this with two experiments\.

Mechanism check\.Figure[2](https://arxiv.org/html/2605.16341#S4.F2)\(§[3](https://arxiv.org/html/2605.16341#S3)\) measuresνt=‖V¯t‖op\\nu\_\{t\}=\\\|\\bar\{V\}\_\{t\}\\\|\_\{\\mathrm\{op\}\}for every matrix layer and logged step during Llama 3 320M pretraining atr∈\{96,192,384\}r\\in\\\{96,192,384\\\}\. Both the time\-course \(Figure[2\(a\)](https://arxiv.org/html/2605.16341#S4.F2.sf1)\) and per\-update distribution \(Figure[2\(b\)](https://arxiv.org/html/2605.16341#S4.F2.sf2)\) confirm Proposition[3\.1](https://arxiv.org/html/2605.16341#S3.Thmtheorem1): Dion’sνt\\nu\_\{t\}inflates monotonically withrrthroughout training, while Orth\-Dion saturates the partial\-isometry boundνt=1\\nu\_\{t\}=1across every \(rank, layer, step\) sample \(Lemma[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1)\)\.

Matched\-rank, hyperparameter\-controlled comparison\.The one\-step descent bound \([1](https://arxiv.org/html/2605.16341#S2.E1)\) couples three rank\-dependent terms: the captured\-gradient mass‖Gt‖KF,r\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}and the residual‖Rt‖KF,r\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}both improve withrr, while the smoothness penaltyLr​νt2​η2/2L\_\{r\}\\nu\_\{t\}^\{2\}\\eta^\{2\}/2inflates withνt2\\nu\_\{t\}^\{2\}\. Their relative weight is regime\-dependent, and the matched\-rrOrth\-Dion vs\. Dion comparison, where the only difference is ColNorm vs\. QR on Line 4 of Algorithm[1](https://arxiv.org/html/2605.16341#alg1), is the clean isolation of the geometric correction\.

On Llama 3 320M, the residual and descent gains from a largerrroutweigh the smoothness penalty, so Dion’s val\. loss falls asrrgrows \(Figure[3\(b\)](https://arxiv.org/html/2605.16341#S8.F3.sf2)\)\. Dion’sν¯t\\bar\{\\nu\}\_\{t\}does rise with rank, from1\.201\.20atr=96r\{=\}96to1\.341\.34atr=384r\{=\}384, but the residual benefit at higherrrwins\. Withrrfixed, the only difference between Dion and Orth\-Dion is ColNorm vs\. QR, which translates directly to the dual\-norm inflation\. Orth\-Dion beats Dion at every rank we tested \(Table[1](https://arxiv.org/html/2605.16341#S8.T1)\)\. Learning rates, momentum, and schedules are identical across runs\.

![Refer to caption](https://arxiv.org/html/2605.16341v1/x5.png)

![Refer to caption](https://arxiv.org/html/2605.16341v1/x6.png)\(a\)Wall\-clock time to match Dion, normalized to mean Dion loss=1\{=\}1\. Orth\-Dion: mean±\\pm2 std\.
![Refer to caption](https://arxiv.org/html/2605.16341v1/x7.png)\(b\)Validation loss on c4\_validation, logged every 50 steps\.
![Refer to caption](https://arxiv.org/html/2605.16341v1/x8.png)\(c\)Mean per\-block effective rank during training\.

Figure 3:Training dynamics for Llama 3 320M \(6,100 steps, 8×\\timesGH200 FSDP\)\.\(a\)Orth\-Dion reaches Dion’s best loss at matched rank in∼0\.89×\\sim 0\.89\\timeswall\-clock time acrossr∈\{96,192,384\}r\\in\\\{96,192,384\\\}\.\(b\)At every rank, Orth\-Dion lands at lower C4 validation loss than Dion, consistent with Lemma[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1)\.\(c\)Ada\-Orth\-Dion \(initr=384r\{=\}384\) hasr≈357r\\approx 357on average; the fixed\-rank curves are flat by construction\.
### 8\.3Wall\-Clock on Llama 3 17\.1B

Figure[1\(b\)](https://arxiv.org/html/2605.16341#S1.F1.sf2)measures per\-step time on Llama 3 with 17\.1B parameters on8×8\{\\times\}A40 with FSDP full\-shard\. The largest per\-block matrix is 16384×\\times43776, roughly 450×\\timesthe size of the largest 320M matrix \(768×\\times2048\)\. Combined with A40s’ lower per\-link bandwidth than NVLink\-class accelerators, this pushes NCCL collectives into the bandwidth\-bound regime, where communication volume dominates step time and the gap between low\-rank and full\-rank updates is most visible\. Two findings:*\(i\)*fixed\-rank Orth\-Dion is slightly slower than Dion at matchedrrdue to the QR step, and*\(ii\)*Ada\-Orth\-Dion closes that gap and matches Dion’s per\-step time\. Combined with the convergence advantage of Section[8\.1](https://arxiv.org/html/2605.16341#S8.SS1), Ada\-Orth\-Dion gives the same per\-step time as Dion at lower validation loss\. Full architecture, hardware, and rank\-policy details are in Appendix[L\.2](https://arxiv.org/html/2605.16341#A12.SS2)\.

### 8\.4Error Feedback:β\\beta\-Sweep

Table 2:β\\beta\-sweep on GPT\-2 45M/WikiText\-103\. Smallerβ\\betaimproves loss untilβ≈0\.05\\beta\{\\approx\}0\.05–0\.10\.1whileℛt∗\\mathcal\{R\}\_\{t\}^\{\*\}grows; disabling EF is worse\.Table[2](https://arxiv.org/html/2605.16341#S8.T2)validates theβ<1\\beta\{<\}1regime on GPT\-2 45M \(wikitext\-103,r=64r\{=\}64, 30k steps\), whereϕt≈0\\phi\_\{t\}\\approx 0\(Regime 2\)\. Full99\-point sweep in App\.[K](https://arxiv.org/html/2605.16341#A11)\. Val loss decreases monotonically asβ\\betais reduced from1\.01\.0down toβ≈0\.1\\beta\{\\approx\}0\.1, with the empirical optimum atβ≈0\.1\\beta\{\\approx\}0\.1\(val\. loss2\.9782\.978vs\.3\.2933\.293forβ=1\\beta\{=\}1\), a0\.3150\.315\-nat improvement obtained at zero additional compute, in line with the theoretical predictionβ∗≪1\\beta^\{\*\}\\\!\\ll\\\!1\. The R\-norm ratioℛt∗\\mathcal\{R\}\_\{t\}^\{\*\}grows from0\.30\.3atβ=1\\beta\{=\}1to3\.23\.2atβ=0\.1\\beta\{=\}0\.1, directly tracking the predicted implicit\-momentum accumulation\. Disabling EF entirely \(β=0\\beta\{=\}0\) hurts by0\.2610\.261nats relative toβ=1\\beta\{=\}1, confirming that EF’s contribution is essential even when heavily damped\.

## 9Analysis and Discussion

Ther\\sqrt\{r\}factor is rank\-dependent, and rank is the parameter practitioners scale up\. The realizedνt\\nu\_\{t\}in our Llama 3 runs sits well below the worst\-caser\\sqrt\{r\}envelope \(Fig\.[2](https://arxiv.org/html/2605.16341#S4.F2)\), so the absolute gap at any fixed rank is gentler than the worst case; nonetheless the inflation is monotone inrrand persists across layers and training steps\. This gap is geometric rather than statistical\. Lemma[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3)shows both algorithms track the same gradient subspace with identical error, so ColNorm does not lose information\. It produces a direction that overshoots the dual\-norm unit ball\. The QR fix removes the inflaall inflation from the norm geometry of the update, independent of how tight the worst\-case bound is\.

## 10Related Work

Spectral\-norm optimizers\.Shampoo\(Guptaet al\.,[2018](https://arxiv.org/html/2605.16341#bib.bib24)\)and SOAP\(Vyaset al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib25)\)precondition with Kronecker factors of second\-moment statistics\. Muon\(Jordanet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib8)\)instead takes the orthogonal polar factor of the gradient\. That is the closed\-form steepest\-descent direction under the spectral norm in the modular\-norm framework ofBernstein and Newhouse \([2024a](https://arxiv.org/html/2605.16341#bib.bib26)\)computed via Newton–Schulz iteration\. All three requireO​\(m​n\)O\(mn\)communication per matrix\.

Error feedback for biased compressors\.Error feedback originates with 1\-bit SGD\(Seideet al\.,[2014](https://arxiv.org/html/2605.16341#bib.bib18)\)and was developed through QSGD\(Alistarhet al\.,[2017](https://arxiv.org/html/2605.16341#bib.bib22)\), sparsified SGD\(Stichet al\.,[2018](https://arxiv.org/html/2605.16341#bib.bib19)\), and the EF21 framework\(Richtáriket al\.,[2021](https://arxiv.org/html/2605.16341#bib.bib20)\)\.Karimireddyet al\.\([2019](https://arxiv.org/html/2605.16341#bib.bib29)\)showed EF restores convergence for sign\-based methods\. All assume Euclidean smoothness, under which our setting’s bounds are loose bymin⁡\(m,n\)/r\\min\(m,n\)/r\. Ourϕt\\phi\_\{t\}\-framework adapts the EF analysis to operator\-norm geometry and resolves the cross\-step correlation between residual and signal that standard analyses implicitly absorb into a worst\-case constant \(Regimes 1 and 3\)\.

Low\-rank and adaptive distributed training\.PowerSGD\(Vogelset al\.,[2019](https://arxiv.org/html/2605.16341#bib.bib21)\)performs power iteration under Frobenius geometry, where column normalization is harmless\. The Ky Fan mismatch is specific to the operator\-norm setting\. Dion\(Ahnet al\.,[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\)inherits ColNorm from this lineage and \(as we show\) pays ar\\sqrt\{r\}penalty for it\. Dion2\(Ahnet al\.,[2025a](https://arxiv.org/html/2605.16341#bib.bib4)\)selects row/column subsets without touching the normalization step\. GaLore\(Zhaoet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib27)\)and ReLoRA\(Lialinet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib28)\)operate in Frobenius geometry and so neither encounter nor benefit from the QR fix\. MuonBP\(Khaledet al\.,[2025](https://arxiv.org/html/2605.16341#bib.bib6)\)keeps full\-rank updates but synchronizes only periodically, addressing communication along an orthogonal axis\. For adaptive rank, AdaLoRA\(Zhanget al\.,[2023](https://arxiv.org/html/2605.16341#bib.bib11)\)prunes singular values by importance during fine\-tuning, and adaptive GaLore variants set rank from gradient\-projection statistics\. Our contraction condition instead ties per\-layer rank to the gradient spectrum’s signal\-to\-noise transition in the from\-scratch optimization regime, where the relevant geometry is operator\-norm rather than Frobenius\.

## 11Conclusion

We identified ar\\sqrt\{r\}geometric inefficiency in low\-rank spectral optimization\. Column normalization inflates the dual normνt\\nu\_\{t\}by up tor\\sqrt\{r\}, producing a convergence penalty that grows with rank\. Replacing ColNorm with QR orthogonalization forcesνt=1\\nu\_\{t\}\{=\}1and recovers the exact polar rate targeted by Muon\. Directνt\\nu\_\{t\}measurement on Llama 3 320M confirms the predicted rank\-dependent inflation in Dion and that QR removes it identically\.

Ada\-Orth\-Dion couples the geometric fix with per\-layer adaptive rank that targets each layer’s intrinsic dimensionality\. The two corrections are complementary: QR removes the rank\-dependent inflation, and adaptive rank reduces the working rank to offset QR’s per\-step compute overhead\. It achieves the lowest validation loss \(3\.2823\.282\) of any low\-rank method evaluated, reaches within0\.0270\.027nats of full\-rank Muon \(3\.2553\.255\) at substantially lower per\-step communication, and attains Dion’s best loss in 18\.3% fewer steps\. On the 17\.1B model \(Figure[1\(b\)](https://arxiv.org/html/2605.16341#S1.F1.sf2), Appendix[L\.2](https://arxiv.org/html/2605.16341#A12.SS2)\), this convergence advantage holds at Dion\-comparable per\-step time\. Together, the fixed\-rank Orth\-Dion result establishes the geometric correction, and Ada\-Orth\-Dion turns it into a practical low\-rank spectral optimizer with a strictly better convergence–communication trade\-off than Dion\.

Limitations\.Our analysis assumes deterministic gradients; extending to the stochastic setting requires bounding the interaction between mini\-batch noise and subspace tracking\. The adaptive rank policy uses a heuristic estimator; tighter connections to the contraction condition would enable provable rank guarantees\.

## Acknowledgements

This research was conducted using the Supermicro ARS\-111GL\-DNHR\-LCC and FUJITSU Server PRIMERGY CX2550 M7 \(Miyabi\) at Joint Center for Advanced High Performance Computing \(JCAHPC\)\. The authors also acknowledge the support of Compute Canada and Mila computing clusters for experimental resources\. We are grateful to the Masason Foundation for supporting this work through computational resources and for fostering a collaborative research environment\.

## References

- Dion2: a simple method to shrink matrix in muon\.arXiv preprint arXiv:2512\.16928\.Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- K\. Ahn, B\. Xu, N\. Abreu, Y\. Fan, G\. Magakyan, P\. Sharma, Z\. Zhan, and J\. Langford \(2025b\)Dion: distributed orthonormalized updates\.arXiv preprint arXiv:2504\.05295\.Cited by:[§L\.1](https://arxiv.org/html/2605.16341#A12.SS1.p1.10),[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix15.p1.1),[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§1](https://arxiv.org/html/2605.16341#S1.p1.3),[§1](https://arxiv.org/html/2605.16341#S1.p2.4),[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- D\. Alistarh, D\. Grubic, J\. Li, R\. Tomioka, and M\. Vojnovic \(2017\)QSGD: communication\-efficient SGD via gradient quantization and encoding\.InAdvances in Neural Information Processing Systems,Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p2.2)\.
- J\. Bernstein and L\. Newhouse \(2024a\)Modular duality in deep learning\.arXiv preprint arXiv:2410\.21265\.Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p1.1)\.
- J\. Bernstein and L\. Newhouse \(2024b\)Old optimizer, new norm: an anthology\.InOPT 2024: Optimization for Machine Learning,Note:NeurIPS 2024 Workshop; also available as arXiv:2409\.20325External Links:[Document](https://dx.doi.org/10.48550/arXiv.2409.20325),[Link](https://openreview.net/forum?id=ux18f5nOpD)Cited by:[§1](https://arxiv.org/html/2605.16341#S1.p2.4)\.
- J\. Bernstein, Y\. Wang, K\. Azizzadenesheli, and A\. Anandkumar \(2018\)SignSGD: compressed optimisation for non\-convex problems\.InInternational Conference on Machine Learning,pp\. 560–569\.Cited by:[§2](https://arxiv.org/html/2605.16341#S2.p1.13)\.
- D\. E\. Carlson, E\. Collins, Y\. Hsieh, L\. Carin, and V\. Cevher \(2015\)Preconditioned spectral descent for deep learning\.InAdvances in Neural Information Processing Systems,Vol\.28,pp\. 2971–2979\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2015/hash/f50a6c02a3fc5a3a5d4d9391f05f3efc-Abstract.html)Cited by:[§1](https://arxiv.org/html/2605.16341#S1.p2.4)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§8\.1](https://arxiv.org/html/2605.16341#S8.SS1.p1.7)\.
- V\. Gupta, T\. Koren, and Y\. Singer \(2018\)Shampoo: preconditioned stochastic tensor optimization\.InInternational Conference on Machine Learning,pp\. 1842–1850\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§10](https://arxiv.org/html/2605.16341#S10.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§8\.1](https://arxiv.org/html/2605.16341#S8.SS1.p1.7)\.
- K\. Jordan, Y\. Jin, V\. Boza, Y\. Jiacheng, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024\)Muon: an optimizer for hidden layers in neural networks\.Note:[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)Accessed: 2025\-7\-3Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§1](https://arxiv.org/html/2605.16341#S1.p1.3),[§10](https://arxiv.org/html/2605.16341#S10.p1.1)\.
- S\. P\. Karimireddy, Q\. Rebjock, S\. U\. Stich, and M\. Jaggi \(2019\)Error feedback fixes SignSGD and other gradient compression schemes\.InInternational Conference on Machine Learning,pp\. 3252–3261\.Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p2.2)\.
- A\. Khaled, K\. Ozkara, T\. Yu, M\. Hong, and Y\. Park \(2025\)MuonBP: faster muon via block\-periodic orthogonalization\.arXiv preprint arXiv:2510\.16981\.Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- V\. Lialin, S\. Muckatira, N\. Shivagunde, and A\. Rumshisky \(2024\)ReLoRA: high\-rank training through low\-rank updates\.InInternational Conference on Learning Representations,Note:arXiv:2307\.05695Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§8\.1](https://arxiv.org/html/2605.16341#S8.SS1.p1.7)\.
- P\. Richtárik, I\. Sokolov, and I\. Fatkhullin \(2021\)EF21: a new, simpler, theoretically better, and practically faster error feedback\.Advances in Neural Information Processing Systems34\.Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p2.2),[§6](https://arxiv.org/html/2605.16341#S6.p5.1)\.
- N\. Sato, H\. Naganuma, and H\. Iiduka \(2025\)Convergence bound and critical batch size of muon optimizer\.arXiv preprint arXiv:2507\.01598\.Cited by:[Appendix I](https://arxiv.org/html/2605.16341#A9.p1.1)\.
- F\. Seide, H\. Fu, J\. Droppo, G\. Li, and D\. Yu \(2014\)1\-bit stochastic gradient descent and its application to data\-parallel distributed training of speech DNNs\.InFifteenth Annual Conference of the International Speech Communication Association,Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p2.2)\.
- S\. U\. Stich, J\. Cordonnier, and M\. Jaggi \(2018\)Sparsified SGD with memory\.InAdvances in Neural Information Processing Systems,Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p2.2)\.
- T\. Vogels, S\. P\. Karimireddy, and M\. Jaggi \(2019\)PowerSGD: practical low\-rank gradient compression for distributed optimization\.InAdvances in Neural Information Processing Systems,Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§1](https://arxiv.org/html/2605.16341#S1.p2.4),[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- N\. Vyas, D\. Morwani, R\. Zhao, M\. Kwun, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. Kakade \(2024\)SOAP: improving and stabilizing shampoo using Adam\.arXiv preprint arXiv:2409\.11321\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§10](https://arxiv.org/html/2605.16341#S10.p1.1)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)GaLore: memory\-efficient LLM training by gradient low\-rank projection\.InInternational Conference on Machine Learning,Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§10](https://arxiv.org/html/2605.16341#S10.p3.1)\.
- Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer,et al\.\(2023\)PyTorch fsdp: experiences on scaling fully sharded data parallel\.Proceedings of the VLDB Endowment16\(12\),pp\. 3848–3860\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2605.16341#Ax2.I1.ix36.p1.1),[§1](https://arxiv.org/html/2605.16341#S1.p1.3)\.

## Appendix

## Appendix AAssumptions \(Full Statements\)

###### Assumption A\.1\(Non\-Euclidean smoothness\)\.

There existsLr\>0L\_\{r\}\>0such that for allX,ΔX,\\Deltawithrank​\(Δ\)≤r\\mathrm\{rank\}\(\\Delta\)\\leq r:

f​\(X\+Δ\)≤f​\(X\)\+⟨∇f​\(X\),Δ⟩\+Lr2​‖Δ‖\(r\)2\.f\(X\+\\Delta\)\\leq f\(X\)\+\\langle\\nabla f\(X\),\\Delta\\rangle\+\\frac\{L\_\{r\}\}\{2\}\\\|\\Delta\\\|\_\{\(r\)\}^\{2\}\.This uses the Ky Fan dual norm∥⋅∥\(r\)\\\|\\cdot\\\|\_\{\(r\)\}rather than the Frobenius norm\. For rank\-rrupdatesΔ=U​V¯⊤\\Delta=U\\bar\{V\}^\{\\top\}with orthonormalUU,V¯\\bar\{V\}:‖Δ‖\(r\)=1\\\|\\Delta\\\|\_\{\(r\)\}=1, so the smoothness cost reduces toLr/2L\_\{r\}/2independent ofrr\.

Assumption A is a one\-sided function\-value inequality\. In Frobenius/Euclidean geometry, a matching descent lemma and a gradient\-Lipschitz bound are equivalent; this equivalence*fails*for the KF dual norm∥⋅∥\(r\)\\\|\\cdot\\\|\_\{\(r\)\}\. Assumption A therefore does not by itself imply any bound of the form‖∇f​\(Y\)−∇f​\(X\)‖∙≤Lr​‖Y−X‖∙\\\|\\nabla f\(Y\)\-\\nabla f\(X\)\\\|\_\{\\bullet\}\\leq L\_\{r\}\\,\\\|Y\-X\\\|\_\{\\bullet\}\. Steps that need such a bound \(the gradient\-subspace drift in Appendix[F](https://arxiv.org/html/2605.16341#A6)and the resulting fixed\-point argument\) require Assumption A′below\.

###### Assumption A\.2\(Gradient Lipschitz in Frobenius – A′\)\.

There existsLF\>0L\_\{F\}\>0such that for allX,Y∈ℝm×nX,Y\\in\\mathbb\{R\}^\{m\\times n\}:

‖∇f​\(Y\)−∇f​\(X\)‖F≤LF​‖Y−X‖F\.\\\|\\nabla f\(Y\)\-\\nabla f\(X\)\\\|\_\{F\}\\leq L\_\{F\}\\\|Y\-X\\\|\_\{F\}\.This is the standard Euclidean gradient\-Lipschitz condition\. It is independent of Assumption A in general, and is the constant that drives the Wedin\-type gradient\-subspace drift bound used in Appendix[F](https://arxiv.org/html/2605.16341#A6)\. \(Where the framework manuscript refers to a drift constant ofLr​rL\_\{r\}\\sqrt\{r\}, the rigorous constant isLF​rL\_\{F\}\\sqrt\{r\}\.\)

###### Assumption A\.3\(Gradient spectral gap – B′\)\.

There existΔgap\>0\\Delta\_\{\\mathrm\{gap\}\}\>0andσmin\>0\\sigma\_\{\\min\}\>0such that for allt≥0t\\geq 0:σr​\(Gt\)−σr\+1​\(Gt\)≥Δgap\\sigma\_\{r\}\(G\_\{t\}\)\-\\sigma\_\{r\+1\}\(G\_\{t\}\)\\geq\\Delta\_\{\\mathrm\{gap\}\}andσr​\(Gt\)≥σmin\\sigma\_\{r\}\(G\_\{t\}\)\\geq\\sigma\_\{\\min\}\. This is a property of the*gradient*\(the optimization landscape\), not the algorithm’s internal state\. It replaces the original Dion framework’s Assumption B, which imposed a gap on the*buffer*MtM\_\{t\}—a quantity that depends on the algorithm’s own error feedback and is therefore circular\.

*Local validity\.*Assumption B′is intended to hold over the regime in which the gradient still has its meaningful rank\-rrstructure \(early/mid training\)\. Near convergence, where‖Gt‖KF,r→0\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\to 0, the lower boundσr​\(Gt\)≥σmin\\sigma\_\{r\}\(G\_\{t\}\)\\geq\\sigma\_\{\\min\}may fail; the convergence results should be read as applying over any horizon on which B′is in force\.

###### Assumption A\.4\(Small spectral tail – B′′\)\.

There existsτ≥0\\tau\\geq 0such that for allt≥0t\\geq 0:σr\+1​\(Gt\)≤τ\\sigma\_\{r\+1\}\(G\_\{t\}\)\\leq\\tau, withτ=O​\(η\)\\tau=O\(\\eta\)in the regime where the convergence theorem is asserted \(cf\. Appendix[F](https://arxiv.org/html/2605.16341#A6)\)\.

*Why this is needed\.*The self\-consistent residual recursion of Appendix[F](https://arxiv.org/html/2605.16341#A6)yields the operator\-norm bound‖Rt\+1‖op≤σr\+1​\(Mt\)\+εt​σ1​\(Mt\)\\\|R\_\{t\+1\}\\\|\_\{\\mathrm\{op\}\}\\leq\\sigma\_\{r\+1\}\(M\_\{t\}\)\+\\varepsilon\_\{t\}\\sigma\_\{1\}\(M\_\{t\}\)\. Assumption B′controls the*gap*but not the*tail*; without B′′, the floorσr\+1​\(Mt\)\\sigma\_\{r\+1\}\(M\_\{t\}\)is notO​\(η\)O\(\\eta\), and the advertisedR∞=O​\(η\)R\_\{\\infty\}=O\(\\eta\)conclusion does not follow\. With B′′, the boxed rate of Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)acquires an additiveO​\(τ\)O\(\\tau\)term, which is absorbed whenτ=O​\(η\)\\tau=O\(\\eta\)\. When B′′fails \(large tail\), the rate becomes2​\(f0−f∞\)​Lr/T\+O​\(τ\)\+O​\(1/T\)\\sqrt\{2\(f\_\{0\}\-f\_\{\\infty\}\)L\_\{r\}/T\}\+O\(\\tau\)\+O\(1/T\), i\.e\. a floor of orderτ\\tau\.

###### Assumption A\.5\(Gradient bounds – C′\)\.

There existGF,GK,κG\>0G\_\{F\},G\_\{K\},\\kappa\_\{G\}\>0such that for allt≥0t\\geq 0:‖Gt‖F≤GF\\\|G\_\{t\}\\\|\_\{F\}\\leq G\_\{F\},‖Gt‖KF,r≤GK\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq G\_\{K\},κr​\(Gt\)≤κG\\kappa\_\{r\}\(G\_\{t\}\)\\leq\\kappa\_\{G\}\.

#### Notational map for stability constants\.

Three closely related contraction symbols appear across the paper\. To keep them straight:

- •ρt:=γ~t​\(1\+κG,t\)\\rho\_\{t\}:=\\tilde\{\\gamma\}\_\{t\}\(1\+\\kappa\_\{G,t\}\)is the*instantaneous*per\-step contraction\.
- •ρ′<1\\rho^\{\\prime\}<1denotes a*uniform per\-step*boundρt≤ρ′\\rho\_\{t\}\\leq\\rho^\{\\prime\}for alltt\(used in Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)for the cleanest statement\)\.
- •ρ¯∈\(0,1\)\\bar\{\\rho\}\\in\(0,1\)together withCρ≥1C\_\{\\rho\}\\geq 1are the*amortized / suffix\-product*constants used whenρt\>1\\rho\_\{t\}\>1is allowed transiently:∏i=st−1ρi≤Cρ​ρ¯t−s\\prod\_\{i=s\}^\{t\-1\}\\rho\_\{i\}\\leq C\_\{\\rho\}\\,\\bar\{\\rho\}^\{t\-s\}for alls<ts<t\(Section[5\.2](https://arxiv.org/html/2605.16341#S5.SS2)\)\.

The uniform condition is the special caseρ¯=ρ′\\bar\{\\rho\}=\\rho^\{\\prime\},Cρ=1C\_\{\\rho\}=1\. Theorem statements that say “ρ′<1\\rho^\{\\prime\}<1” are tight conditions; the amortized version \(Theorem[G\.1](https://arxiv.org/html/2605.16341#A7.Thmtheorem1)\) yields the same conclusion under the suffix\-product condition above\.

## Appendix BAlgorithm Details

### B\.1Full Algorithms

Algorithm 2Stripped Dion \(Baseline\)1:Input:learning rate

η\\eta, error\-feedback coefficient

β\\beta, rank

rr
2:for

t=0,1,…,T−1t=0,1,\\dots,T\-1do

3:

Mt←Gt\+RtM\_\{t\}\\leftarrow G\_\{t\}\+R\_\{t\}\(buffer = gradient \+ residual\)

4:

Ut←orth⁡\(Mt​Vt−1\)U\_\{t\}\\leftarrow\\operatorname\{orth\}\(M\_\{t\}V\_\{t\-1\}\)\(one step of subspace iteration\)

5:

Wt←Mt⊤​UtW\_\{t\}\\leftarrow M\_\{t\}^\{\\top\}U\_\{t\},

V¯t←ColNorm⁡\(Wt\)\\bar\{V\}\_\{t\}\\leftarrow\\operatorname\{ColNorm\}\(W\_\{t\}\)\(column normalization\)

6:

D^t←Ut​V¯t⊤\\hat\{D\}\_\{t\}\\leftarrow U\_\{t\}\\bar\{V\}\_\{t\}^\{\\top\}\(rank\-rrupdate direction\)

7:

Xt\+1←Xt−ηt​D^tX\_\{t\+1\}\\leftarrow X\_\{t\}\-\\eta\_\{t\}\\hat\{D\}\_\{t\}
8:

Bt←Ut​Ut⊤​MtB\_\{t\}\\leftarrow U\_\{t\}U\_\{t\}^\{\\top\}M\_\{t\},

Rt\+1←β​\(Mt−Bt\)R\_\{t\+1\}\\leftarrow\\beta\\,\(M\_\{t\}\-B\_\{t\}\)\(error feedback\)

9:endfor

Algorithm 3Orth\-Dion \(Proposed\)1:Input:learning rate

η\\eta, error\-feedback coefficient

β\\beta, rank

rr
2:for

t=0,1,…,T−1t=0,1,\\dots,T\-1do

3:

Mt←Gt\+RtM\_\{t\}\\leftarrow G\_\{t\}\+R\_\{t\}\(buffer = gradient \+ residual\)

4:

Ut←orth⁡\(Mt​Vt−1\)U\_\{t\}\\leftarrow\\operatorname\{orth\}\(M\_\{t\}V\_\{t\-1\}\)\(one step of subspace iteration\)

5:

Wt←Mt⊤​UtW\_\{t\}\\leftarrow M\_\{t\}^\{\\top\}U\_\{t\},

V¯t←orth⁡\(Wt\)\\bar\{V\}\_\{t\}\\leftarrow\\operatorname\{orth\}\(W\_\{t\}\)\(QR orthogonalization\)

6:

D^t←Ut​V¯t⊤\\hat\{D\}\_\{t\}\\leftarrow U\_\{t\}\\bar\{V\}\_\{t\}^\{\\top\}\(rank\-rrupdate direction\)

7:

Xt\+1←Xt−ηt​D^tX\_\{t\+1\}\\leftarrow X\_\{t\}\-\\eta\_\{t\}\\hat\{D\}\_\{t\}
8:

Bt←Ut​Ut⊤​MtB\_\{t\}\\leftarrow U\_\{t\}U\_\{t\}^\{\\top\}M\_\{t\},

Rt\+1←β​\(Mt−Bt\)R\_\{t\+1\}\\leftarrow\\beta\\,\(M\_\{t\}\-B\_\{t\}\)\(error feedback\)

9:endfor

### B\.2Ada\-Orth\-Dion Rank Selection

###### Definition B\.1\(Estimated effective rank\)\.

Given power\-iteration factorsUt∈ℝm×rU\_\{t\}\\in\\mathbb\{R\}^\{m\\times r\}andWt=Mt⊤​Ut∈ℝn×rW\_\{t\}=M\_\{t\}^\{\\top\}U\_\{t\}\\in\\mathbb\{R\}^\{n\\times r\}, defineσ^i=‖\[Wt\]i‖2\\hat\{\\sigma\}\_\{i\}=\\\|\[W\_\{t\}\]\_\{i\}\\\|\_\{2\}and:

erank^​\(Mt\)=exp⁡\(−∑i=1rp^i​log⁡p^i\),p^i=σ^i∑jσ^j\.\\widehat\{\\operatorname\{erank\}\}\(M\_\{t\}\)=\\exp\\\!\\Big\(\-\\sum\_\{i=1\}^\{r\}\\hat\{p\}\_\{i\}\\log\\hat\{p\}\_\{i\}\\Big\),\\qquad\\hat\{p\}\_\{i\}=\\frac\{\\hat\{\\sigma\}\_\{i\}\}\{\\sum\_\{j\}\\hat\{\\sigma\}\_\{j\}\}\.

Algorithm 4Ada\-Orth\-Dion Rank Selection0:estimate

ete\_\{t\}, current rank

rtr\_\{t\}, EMA

e¯t−1\\bar\{e\}\_\{t\-1\}, bounds

rmin,rmaxr\_\{\\min\},r\_\{\\max\}, smoothing

α\\alpha, buffer

γ\\gamma
1:

e¯t←α​et\+\(1−α\)​e¯t−1\\bar\{e\}\_\{t\}\\leftarrow\\alpha\\,e\_\{t\}\+\(1\-\\alpha\)\\,\\bar\{e\}\_\{t\-1\}
2:

rtarget←⌈γ​e¯t⌉r\_\{\\mathrm\{target\}\}\\leftarrow\\lceil\\gamma\\,\\bar\{e\}\_\{t\}\\rceil
3:

rt\+1←clip⁡\(rtarget,rmin,rmax\)r\_\{t\+1\}\\leftarrow\\operatorname\{clip\}\(r\_\{\\mathrm\{target\}\},\\;r\_\{\\min\},\\;r\_\{\\max\}\), rounded to multiple of 8

4:if

rt\+1≠rtr\_\{t\+1\}\\neq r\_\{t\}then

5:Resize

U,VU,Vto rank

rt\+1r\_\{t\+1\}\(truncate or pad\)

6:endif

Connection to contraction condition\.The estimated effective rank is a spectral entropy measure\. When the gradient has a sharp spectral gap at positionkk, the effective rank concentrates nearkkregardless of the nominalrr\. By trackingerank^\\widehat\{\\operatorname\{erank\}\}and adjustingrtr\_\{t\}, Ada\-Dion approximatesri∗=max⁡\{r:γ~i,r​\(1\+κG,i,r\)<1\}r\_\{i\}^\{\*\}=\\max\\\{r:\\tilde\{\\gamma\}\_\{i,r\}\(1\+\\kappa\_\{G,i,r\}\)<1\\\}without explicit SVD\.

## Appendix CNorms and Key Quantities

ForA∈ℝm×nA\\in\\mathbb\{R\}^\{m\\times n\}with singular valuesσ1​\(A\)≥σ2​\(A\)≥…\\sigma\_\{1\}\(A\)\\geq\\sigma\_\{2\}\(A\)\\geq\\ldots:

Key Hölder identity\.⟨A,B⟩≤‖A‖KF,r⋅‖B‖\(r\)\\langle A,B\\rangle\\leq\\\|A\\\|\_\{\\mathrm\{KF\},r\}\\cdot\\\|B\\\|\_\{\(r\)\}, with equality*attained*atB=Pr​\(A\)B=P\_\{r\}\(A\)\(the rank\-rrpolar factor\): then‖Pr​\(A\)‖\(r\)=1\\\|P\_\{r\}\(A\)\\\|\_\{\(r\)\}=1and⟨A,Pr​\(A\)⟩=‖A‖KF,r\\langle A,P\_\{r\}\(A\)\\rangle=\\\|A\\\|\_\{\\mathrm\{KF\},r\}\. Equality is not in general unique — e\.g\. forr=1r\{=\}1andA=diag⁡\(1,1\)A=\\operatorname\{diag\}\(1,1\), bothB1=diag⁡\(1,0\)B\_\{1\}=\\operatorname\{diag\}\(1,0\)andB2=diag⁡\(0,1\)B\_\{2\}=\\operatorname\{diag\}\(0,1\)saturate the bound\.

Key quantities per iteration\.εt=‖Ut​Ut⊤−Ur,t​Ur,t⊤‖op\\varepsilon\_\{t\}=\\\|U\_\{t\}U\_\{t\}^\{\\top\}\-U\_\{r,t\}U\_\{r,t\}^\{\\top\}\\\|\_\{\\mathrm\{op\}\}\(projector error\),δt=‖Mt‖KF,r−⟨Mt,D^t⟩\\delta\_\{t\}=\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\-\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangle\(oracle defect\),νt=‖D^t‖\(r\)\\nu\_\{t\}=\\\|\\hat\{D\}\_\{t\}\\\|\_\{\(r\)\}\(dual norm factor\),γ~t=σr\+1​\(Mt\)/σr​\(Mt\)\\tilde\{\\gamma\}\_\{t\}=\\sigma\_\{r\+1\}\(M\_\{t\}\)/\\sigma\_\{r\}\(M\_\{t\}\)\(buffer spectral ratio\)\.

## Appendix DProof of One\-Step Descent \(Equation[1](https://arxiv.org/html/2605.16341#S2.E1)\)

###### Proof\.

Step 1 \(Smoothness\):f​\(Xt\+1\)≤f​\(Xt\)\+⟨Gt,−η​D^t⟩\+Lr2​‖η​D^t‖\(r\)2=f​\(Xt\)−η​⟨Gt,D^t⟩\+Lr​νt22​η2f\(X\_\{t\+1\}\)\\leq f\(X\_\{t\}\)\+\\langle G\_\{t\},\-\\eta\\hat\{D\}\_\{t\}\\rangle\+\\frac\{L\_\{r\}\}\{2\}\\\|\\eta\\hat\{D\}\_\{t\}\\\|\_\{\(r\)\}^\{2\}=f\(X\_\{t\}\)\-\\eta\\langle G\_\{t\},\\hat\{D\}\_\{t\}\\rangle\+\\frac\{L\_\{r\}\\nu\_\{t\}^\{2\}\}\{2\}\\eta^\{2\}\.

Step 2 \(Gradient\-buffer decomposition\):⟨Gt,D^t⟩=⟨Mt,D^t⟩−⟨Rt,D^t⟩\\langle G\_\{t\},\\hat\{D\}\_\{t\}\\rangle=\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangle\-\\langle R\_\{t\},\\hat\{D\}\_\{t\}\\rangle\.

Step 3 \(LMO defect\):⟨Mt,D^t⟩=‖Mt‖KF,r−δt\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangle=\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\-\\delta\_\{t\}\.

Step 4 \(Triangle inequality\):‖Mt‖KF,r≥‖Gt‖KF,r−‖Rt‖KF,r\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\geq\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\-\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\.

Step 5 \(Hölder forRtR\_\{t\}term\):\|⟨Rt,D^t⟩\|≤‖Rt‖KF,r⋅νt\|\\langle R\_\{t\},\\hat\{D\}\_\{t\}\\rangle\|\\leq\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\cdot\\nu\_\{t\}\.

Assembly:−⟨Gt,D^t⟩≤−‖Gt‖KF,r\+δt\+\(1\+νt\)​‖Rt‖KF,r\-\\langle G\_\{t\},\\hat\{D\}\_\{t\}\\rangle\\leq\-\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\+\\delta\_\{t\}\+\(1\+\\nu\_\{t\}\)\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\. Substituting into Step 1 completes the proof\. ∎

## Appendix EOrth\-Dion Proofs

###### Proof of Lemma[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1)\.

D^⊤​D^=V¯​U⊤​U​V¯⊤=V¯​V¯⊤\\hat\{D\}^\{\\top\}\\hat\{D\}=\\bar\{V\}U^\{\\top\}U\\bar\{V\}^\{\\top\}=\\bar\{V\}\\bar\{V\}^\{\\top\}is an orthogonal projector with eigenvalues\{0,1\}\\\{0,1\\\}\. ThereforeD^\\hat\{D\}hasrrsingular values equal to11and the rest0\. Henceσ1​\(D^\)=1\\sigma\_\{1\}\(\\hat\{D\}\)=1and‖D^‖∗=r\\\|\\hat\{D\}\\\|\_\{\*\}=r, so‖D^‖\(r\)=max⁡\{1,‖D^‖∗/r\}=max⁡\{1,1\}=1\\\|\\hat\{D\}\\\|\_\{\(r\)\}=\\max\\\{1,\\;\\\|\\hat\{D\}\\\|\_\{\*\}/r\\\}=\\max\\\{1,1\\\}=1\. ∎

###### Proof of Lemma[4\.2](https://arxiv.org/html/2605.16341#S4.Thmtheorem2)\.

By Hölder:⟨Mt,D^t⟩≤‖Mt‖KF,r⋅‖D^t‖\(r\)=‖Mt‖KF,r⋅1\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangle\\leq\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\cdot\\\|\\hat\{D\}\_\{t\}\\\|\_\{\(r\)\}=\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\cdot 1\. Soδt=‖Mt‖KF,r−⟨Mt,D^t⟩≥0\\delta\_\{t\}=\\\|M\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\-\\langle M\_\{t\},\\hat\{D\}\_\{t\}\\rangle\\geq 0\.

Significance: For ColNorm,νt\>1\\nu\_\{t\}\>1meansD^t\\hat\{D\}\_\{t\}can violate the Hölder bound, makingδt\\delta\_\{t\}potentially negative and complicating the analysis\. ∎

###### Proof of Lemma[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3)\.

The tracking recursion depends on three quantities: \(1\) Column space ofVt−1V\_\{t\-1\}:col⁡\(orth⁡\(W\)\)=col⁡\(W\)=col⁡\(ColNorm⁡\(W\)\)\\operatorname\{col\}\(\\operatorname\{orth\}\(W\)\)=\\operatorname\{col\}\(W\)=\\operatorname\{col\}\(\\operatorname\{ColNorm\}\(W\)\)for full\-rankWW\. \(2\) Spectral gap ofMtM\_\{t\}: unchanged between algorithms\. \(3\) Buffer drift‖Mt−Mt−1‖op\\\|M\_\{t\}\-M\_\{t\-1\}\\\|\_\{\\mathrm\{op\}\}: depends onη,β\\eta,\\beta, unchanged\. Thereforeεt\\varepsilon\_\{t\}satisfies the same recursion and bounds\. ∎

Defect bound for Orth\-Dion\.Under the projector\-level bound with Lemma[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3):0≤δt≤Cδ′​‖Mt‖F⋅εt0\\leq\\delta\_\{t\}\\leq C^\{\\prime\}\_\{\\delta\}\\\|M\_\{t\}\\\|\_\{F\}\\cdot\\varepsilon\_\{t\}whereCδ′=\(1\+κG\)​2​rC^\{\\prime\}\_\{\\delta\}=\(1\+\\kappa\_\{G\}\)\\sqrt\{2r\}\. The constant is the same order as for Stripped Dion; the improvement comes entirely fromνt\\nu\_\{t\}\.

## Appendix FSelf\-Consistent Drift Elimination \(Full Details\)

### F\.1Why the Original Assumption D Was Circular

Assumption D required‖Mt−Mt−1‖op≤Δdrift\\\|M\_\{t\}\-M\_\{t\-1\}\\\|\_\{\\mathrm\{op\}\}\\leq\\Delta\_\{\\mathrm\{drift\}\}\. ButMt−Mt−1=\(Gt−Gt−1\)\+\(Rt−Rt−1\)M\_\{t\}\-M\_\{t\-1\}=\(G\_\{t\}\-G\_\{t\-1\}\)\+\(R\_\{t\}\-R\_\{t\-1\}\)\. Under Assumption A′\(Frobenius gradient\-Lipschitz, Appendix[A\.2](https://arxiv.org/html/2605.16341#A1.Thmtheorem2)\), the gradient difference is small:‖Gt\+1−Gt‖F≤LF​‖Xt\+1−Xt‖F=LF​η​‖D^t‖F≤LF​η​r\\\|G\_\{t\+1\}\-G\_\{t\}\\\|\_\{F\}\\leq L\_\{F\}\\\|X\_\{t\+1\}\-X\_\{t\}\\\|\_\{F\}=L\_\{F\}\\eta\\\|\\hat\{D\}\_\{t\}\\\|\_\{F\}\\leq L\_\{F\}\\eta\\sqrt\{r\}\. The residual difference is*not*small:Rt\+1−Rt=β​\(I−Ut​Ut⊤\)​Mt−RtR\_\{t\+1\}\-R\_\{t\}=\\beta\\,\(I\-U\_\{t\}U\_\{t\}^\{\\top\}\)M\_\{t\}\-R\_\{t\}, so‖Rt\+1−Rt‖op=O​\(‖Gt‖\)\\\|R\_\{t\+1\}\-R\_\{t\}\\\|\_\{\\mathrm\{op\}\}=O\(\\\|G\_\{t\}\\\|\)\. Assumption D is inconsistent with the algorithm’s actual dynamics\.

### F\.2Gradient Subspace Stability

###### Proposition F\.1\(Gradient subspace drift\)\.

Under Assumptions A′and B′:sin⁡Θmax​\(PGt\+1,PGt\)≤LF​η​r/Δgap\\sin\\Theta\_\{\\max\}\(P\_\{G\}^\{t\+1\},P\_\{G\}^\{t\}\)\\leq L\_\{F\}\\,\\eta\\,\\sqrt\{r\}/\\Delta\_\{\\mathrm\{gap\}\}\.

###### Proof\.

By A′:‖Gt\+1−Gt‖F≤LF​‖Xt\+1−Xt‖F=LF​η​‖D^t‖F≤LF​η​r\\\|G\_\{t\+1\}\-G\_\{t\}\\\|\_\{F\}\\leq L\_\{F\}\\,\\\|X\_\{t\+1\}\-X\_\{t\}\\\|\_\{F\}=L\_\{F\}\\eta\\\|\\hat\{D\}\_\{t\}\\\|\_\{F\}\\leq L\_\{F\}\\eta\\sqrt\{r\}, where the last step uses‖D^t‖F≤r\\\|\\hat\{D\}\_\{t\}\\\|\_\{F\}\\leq\\sqrt\{r\}\(everyD^t\\hat\{D\}\_\{t\}has at mostrrsingular values, each at most11\)\. Wedin’ssin⁡Θ\\sin\\Thetatheorem with eigengapΔgap\\Delta\_\{\\mathrm\{gap\}\}then yields the stated bound\.

*Remark on the constant\.*An earlier version of this proof attempted to derive a Frobenius gradient\-Lipschitz bound from Assumption A\. That step is not valid: A is a one\-sided function\-value smoothness inequality in the KF dual norm and does not imply a Frobenius gradient\-Lipschitz estimate\. We therefore introduce A′as a separate assumption and state the drift bound with constantLF​rL\_\{F\}\\sqrt\{r\}, notLr​rL\_\{r\}\\sqrt\{r\}\. This substitution propagates through all downstream uses of the drift bound \(in particular, the fixed\-point map below\)\. ∎

### F\.3Buffer Spectral Gap Transfer

###### Proposition F\.2\.

Under Assumptions B’, C’, if‖Rt‖op≤Δgap/2\\\|R\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\leq\\Delta\_\{\\mathrm\{gap\}\}/2: \(1\)σr​\(Mt\)≥σmin−Δgap/2\>0\\sigma\_\{r\}\(M\_\{t\}\)\\geq\\sigma\_\{\\min\}\-\\Delta\_\{\\mathrm\{gap\}\}/2\>0; \(2\)γ~eff≤\(σr\+1​\(Gt\)\+‖Rt‖op\)/\(σr​\(Gt\)−‖Rt‖op\)\\tilde\{\\gamma\}\_\{\\mathrm\{eff\}\}\\leq\(\\sigma\_\{r\+1\}\(G\_\{t\}\)\+\\\|R\_\{t\}\\\|\_\{\\mathrm\{op\}\}\)/\(\\sigma\_\{r\}\(G\_\{t\}\)\-\\\|R\_\{t\}\\\|\_\{\\mathrm\{op\}\}\); \(3\) Buffer gap:σr​\(Mt\)−σr\+1​\(Mt\)≥Δgap−2​‖Rt‖op\\sigma\_\{r\}\(M\_\{t\}\)\-\\sigma\_\{r\+1\}\(M\_\{t\}\)\\geq\\Delta\_\{\\mathrm\{gap\}\}\-2\\\|R\_\{t\}\\\|\_\{\\mathrm\{op\}\}\. All follow from Weyl’s inequality\.

### F\.4Self\-Consistent Residual Bound

Withβ=1\\beta=1and projector errorεt\\varepsilon\_\{t\}, the residual update yields the operator\-norm bound

‖Rt\+1‖op≤σr\+1​\(Mt\)\+εt⋅σ1​\(Mt\)\.\\\|R\_\{t\+1\}\\\|\_\{\\mathrm\{op\}\}\\;\\leq\\;\\sigma\_\{r\+1\}\(M\_\{t\}\)\+\\varepsilon\_\{t\}\\cdot\\sigma\_\{1\}\(M\_\{t\}\)\.This decomposition has two components: the spectral*tail*σr\+1​\(Mt\)\\sigma\_\{r\+1\}\(M\_\{t\}\)\(a property of the buffer’s low\-rank approximation error, controlled by Assumption B′′together with Weyl\-type buffer\-vs\-gradient comparison\), and the tracking\-leakage termεt​σ1​\(Mt\)\\varepsilon\_\{t\}\\sigma\_\{1\}\(M\_\{t\}\)\(controlled by the contractionρ′<1\\rho^\{\\prime\}<1\)\.

*Status \(added 2026\-04\-15\)\.*Assumption B′controls only the gradient gap, not the tail; even with perfect tracking \(εt=0\\varepsilon\_\{t\}=0\) the floorσr\+1​\(Mt\)\\sigma\_\{r\+1\}\(M\_\{t\}\)remains, and Weyl gives onlyσr\+1​\(Mt\)≤σr\+1​\(Gt\)\+‖Rt‖op\\sigma\_\{r\+1\}\(M\_\{t\}\)\\leq\\sigma\_\{r\+1\}\(G\_\{t\}\)\+\\\|R\_\{t\}\\\|\_\{\\mathrm\{op\}\}\. The advertised conclusionR∞=O​\(η\)R\_\{\\infty\}=O\(\\eta\)therefore requires the additional small\-tail Assumption B′′\(Appendix[A\.4](https://arxiv.org/html/2605.16341#A1.Thmtheorem4)\); withσr\+1​\(Gt\)≤τ\\sigma\_\{r\+1\}\(G\_\{t\}\)\\leq\\tau, the recursion has a fixed pointR∞=O​\(τ\+η\)R\_\{\\infty\}=O\(\\tau\+\\eta\), which collapses toO​\(η\)O\(\\eta\)whenτ=O​\(η\)\\tau=O\(\\eta\)\. When B′′fails \(e\.g\. on heavy\-tailed gradients\), the conclusion weakens to a tail floor of orderτ\\tauin Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)\.

### F\.5The Fixed Point

###### Proposition F\.3\(Self\-consistent fixed point\)\.

Assume A, A′, B′, B′′\(with tailτ\\tau\), C′, and letρ′=γ~eff​\(1\+κG\)<1\\rho^\{\\prime\}=\\tilde\{\\gamma\}\_\{\\mathrm\{eff\}\}\(1\+\\kappa\_\{G\}\)<1\. Forη=O​\(Δgap/\(LF​r​κG\)\)\\eta=O\(\\Delta\_\{\\mathrm\{gap\}\}/\(L\_\{F\}\\sqrt\{r\}\\kappa\_\{G\}\)\), the coupled system

ε∞′\\displaystyle\\varepsilon^\{\\prime\}\_\{\\infty\}=γ~eff⋅Deff1−ρ′,R∞≤σr\+1​\(M\)⏟≤τ\+‖R∞‖op​\(B′′\+Weyl\)\+ε∞′​σ1,\\displaystyle=\\frac\{\\tilde\{\\gamma\}\_\{\\mathrm\{eff\}\}\\cdot D\_\{\\mathrm\{eff\}\}\}\{1\-\\rho^\{\\prime\}\},\\quad R\_\{\\infty\}\\leq\\underbrace\{\\sigma\_\{r\+1\}\(M\)\}\_\{\\leq\\,\\tau\\,\+\\,\\\|R\_\{\\infty\}\\\|\_\{\\mathrm\{op\}\}\\,\(\\text\{B$\{\}^\{\\prime\\prime\}$\+Weyl\}\)\}\+\\varepsilon^\{\\prime\}\_\{\\infty\}\\sigma\_\{1\},Deff\\displaystyle D\_\{\\mathrm\{eff\}\}=LF​η​r\+R∞Δgap−2​R∞,γ~eff=σr\+1​\(G\)\+R∞σr​\(G\)−R∞\\displaystyle=\\frac\{L\_\{F\}\\eta\\sqrt\{r\}\+R\_\{\\infty\}\}\{\\Delta\_\{\\mathrm\{gap\}\}\-2R\_\{\\infty\}\},\\quad\\tilde\{\\gamma\}\_\{\\mathrm\{eff\}\}=\\frac\{\\sigma\_\{r\+1\}\(G\)\+R\_\{\\infty\}\}\{\\sigma\_\{r\}\(G\)\-R\_\{\\infty\}\}has a unique small fixed point withR∞=O​\(τ\+η\)R\_\{\\infty\}=O\(\\tau\+\\eta\)andε∞′=O​\(η\)\\varepsilon^\{\\prime\}\_\{\\infty\}=O\(\\eta\)\. In particular, whenτ=O​\(η\)\\tau=O\(\\eta\), both quantities areO​\(η\)O\(\\eta\)\.

###### Proof\.

Banach iteration from\(0,0\)\(0,0\)\. First iterate:Deff\(0\)=LF​η​r/Δgap=O​\(η\)D\_\{\\mathrm\{eff\}\}^\{\(0\)\}=L\_\{F\}\\eta\\sqrt\{r\}/\\Delta\_\{\\mathrm\{gap\}\}=O\(\\eta\)\(uses A′\),γ~eff\(0\)=γ~G\\tilde\{\\gamma\}\_\{\\mathrm\{eff\}\}^\{\(0\)\}=\\tilde\{\\gamma\}\_\{G\},ε′⁣\(0\)=O​\(η\)\\varepsilon^\{\\prime\(0\)\}=O\(\\eta\),R\(1\)≤τ\+O​\(η\)R^\{\(1\)\}\\leq\\tau\+O\(\\eta\)\(using B′′to bound the tail floor\)\. Each subsequent iterate perturbs byO​\(η2\)O\(\\eta^\{2\}\)\. The map is Lipschitz with constantO​\(η\)O\(\\eta\)near the fixed point, hence contractive\.

*What this proves precisely\.*Under A, A′, B′, B′′, C′, the coupled system has a fixed point withR∞,ε∞′R\_\{\\infty\},\\varepsilon^\{\\prime\}\_\{\\infty\}as stated, on any finite horizon over which the assumptions hold\. Assumption D is*derived*in this regime, not posited\. Outside this regime — in particular when B′′fails — the drift is not eliminated; it is re\-expressed in terms of the tailτ\\tau, which can be large\. ∎

## Appendix GAmortized Contraction \(Full Proof\)

###### Theorem G\.1\(Amortized contraction\)\.

Letρt=γ~t​\(1\+κG,t\)\\rho\_\{t\}=\\tilde\{\\gamma\}\_\{t\}\(1\+\\kappa\_\{G,t\}\)\. Suppose there exist constantsρ¯∈\(0,1\)\\bar\{\\rho\}\\in\(0,1\)andCρ≥1C\_\{\\rho\}\\geq 1such that the*suffix\-product condition*holds: for every0≤s<t0\\leq s<t,

∏i=st−1ρi≤Cρ​ρ¯t−s\.\\prod\_\{i=s\}^\{t\-1\}\\rho\_\{i\}\\;\\leq\\;C\_\{\\rho\}\\,\\bar\{\\rho\}^\{\\,t\-s\}\.Then the conclusion of Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)holds withρ′\\rho^\{\\prime\}replaced byρ¯\\bar\{\\rho\}and an additional multiplicative constantCρC\_\{\\rho\}in the tracking\-error and residual\-coupling contributions\.

###### Proof\.

The tracking recursion with variableρt\\rho\_\{t\}isεt≤ρt−1​εt−1\+ct−1\\varepsilon\_\{t\}\\leq\\rho\_\{t\-1\}\\varepsilon\_\{t\-1\}\+c\_\{t\-1\}\. Unrolling,

εT≤\(∏i=0T−1ρi\)​ε0\+∑s=0T−1cs​\(∏i=s\+1T−1ρi\)\.\\varepsilon\_\{T\}\\;\\leq\\;\\Big\(\\prod\_\{i=0\}^\{T\-1\}\\rho\_\{i\}\\Big\)\\varepsilon\_\{0\}\\;\+\\;\\sum\_\{s=0\}^\{T\-1\}c\_\{s\}\\,\\Big\(\\prod\_\{i=s\+1\}^\{T\-1\}\\rho\_\{i\}\\Big\)\.Apply the suffix\-product condition twice\.*Initial term:*∏i=0T−1ρi≤Cρ​ρ¯T→0\\prod\_\{i=0\}^\{T\-1\}\\rho\_\{i\}\\leq C\_\{\\rho\}\\,\\bar\{\\rho\}^\{T\}\\to 0\.*Sum term:*each partial product∏i=s\+1T−1ρi≤Cρ​ρ¯T−1−s\\prod\_\{i=s\+1\}^\{T\-1\}\\rho\_\{i\}\\leq C\_\{\\rho\}\\,\\bar\{\\rho\}^\{T\-1\-s\}, so∑scs​∏i=s\+1T−1ρi≤Cρ​maxs⁡cs1−ρ¯\\sum\_\{s\}c\_\{s\}\\prod\_\{i=s\+1\}^\{T\-1\}\\rho\_\{i\}\\leq C\_\{\\rho\}\\,\\frac\{\\max\_\{s\}c\_\{s\}\}\{1\-\\bar\{\\rho\}\}\. Both terms are uniformly controlled, and the rest of the proof of Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)\(Appendix[H](https://arxiv.org/html/2605.16341#A8)\) goes through with1/\(1−ρ′\)1/\(1\-\\rho^\{\\prime\}\)replaced byCρ/\(1−ρ¯\)C\_\{\\rho\}/\(1\-\\bar\{\\rho\}\)\.

*Why the suffix\-product condition \(not the time\-averagedρ¯=exp⁡\(\(1/T\)​∑tlog⁡ρt\)\\bar\{\\rho\}=\\exp\(\(1/T\)\\sum\_\{t\}\\log\\rho\_\{t\}\)\) is required\.*A condition on the*full\-horizon*geometric mean does not control*suffix*products: a sequence withρt<1\\rho\_\{t\}<1fort<T/2t<T/2andρt\>1\\rho\_\{t\}\>1fort≥T/2t\\geq T/2can have full\-horizon mean<1<1while every suffix product over\[T/2,T\)\[T/2,T\)grows\. The earlier statement “ρ¯=exp⁡\(\(1/T\)​∑tlog⁡ρt\)<1\\bar\{\\rho\}=\\exp\(\(1/T\)\\sum\_\{t\}\\log\\rho\_\{t\}\)<1implies each partial product is bounded by a power ofρ¯\\bar\{\\rho\}by AM\-GM” is therefore not valid as written\. The strong suffix\-product condition above is exactly the body’s Section[5\.2](https://arxiv.org/html/2605.16341#S5.SS2)statement and is the natural Lyapunov\-style stability condition for subspace tracking\. ∎

## Appendix HComplete Proof of Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)

###### Proof\.

Step 1 \(Telescoping\)\.Sum the Orth\-Dion descent bound overt=0,…,T−1t=0,\\ldots,T\{\-\}1:

∑tη​‖Gt‖KF,r≤\(f0−f∞\)\+∑tη​\(δt\+2​‖Rt‖KF,r\)\+Lr2​T​η2\.\\sum\_\{t\}\\eta\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq\(f\_\{0\}\-f\_\{\\infty\}\)\+\\sum\_\{t\}\\eta\(\\delta\_\{t\}\+2\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\)\+\\tfrac\{L\_\{r\}\}\{2\}T\\eta^\{2\}\.
Step 2 \(Error terms,β=1\\beta\{=\}1\)\.By the self\-consistent analysis:ε∞′=O​\(η\)\\varepsilon^\{\\prime\}\_\{\\infty\}=O\(\\eta\)\. Defect:δt≤Cδ′​GF​εt\\delta\_\{t\}\\leq C^\{\\prime\}\_\{\\delta\}G\_\{F\}\\varepsilon\_\{t\}\. Residual:‖Rt\+1‖KF,r≤CR​GF​εt\\\|R\_\{t\+1\}\\\|\_\{\\mathrm\{KF\},r\}\\leq C\_\{R\}G\_\{F\}\\varepsilon\_\{t\}\. Error sum splits:∑tεt≤ε0/\(1−ρ′\)\+T​ε∞′=O​\(1\)\+O​\(T​η\)\\sum\_\{t\}\\varepsilon\_\{t\}\\leq\\varepsilon\_\{0\}/\(1\{\-\}\\rho^\{\\prime\}\)\+T\\varepsilon^\{\\prime\}\_\{\\infty\}=O\(1\)\+O\(T\\eta\)\. Therefore∑tη​\(δt\+2​‖Rt‖KF,r\)=O​\(T​η2\)\\sum\_\{t\}\\eta\(\\delta\_\{t\}\+2\\\|R\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\)=O\(T\\eta^\{2\}\)\.

Step 3 \(Assembly\)\.η​T​mint⁡‖Gt‖KF,r≤\(f0−f∞\)\+\(Lr2\+C\)​T​η2\\eta T\\min\_\{t\}\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq\(f\_\{0\}\-f\_\{\\infty\}\)\+\(\\frac\{L\_\{r\}\}\{2\}\+C\)T\\eta^\{2\}\. Dividing:mint⁡‖Gt‖KF,r≤f0−f∞η​T\+Lr​η2\+O​\(1/T\)\\min\_\{t\}\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq\\frac\{f\_\{0\}\-f\_\{\\infty\}\}\{\\eta T\}\+\\frac\{L\_\{r\}\\eta\}\{2\}\+O\(1/T\)\.

Step 4 \(Optimizeη=c/T\\eta=c/\\sqrt\{T\}\)\.minc⁡\[f0−f∞c\+Lr​c2\]\\min\_\{c\}\[\\frac\{f\_\{0\}\-f\_\{\\infty\}\}\{c\}\+\\frac\{L\_\{r\}c\}\{2\}\]atc∗=2​\(f0−f∞\)/Lrc^\{\*\}=\\sqrt\{2\(f\_\{0\}\-f\_\{\\infty\}\)/L\_\{r\}\}\. Substituting:mint⁡‖Gt‖KF,r≤2​\(f0−f∞\)​Lr/T\+O​\(1/T\)\\min\_\{t\}\\\|G\_\{t\}\\\|\_\{\\mathrm\{KF\},r\}\\leq\\sqrt\{2\(f\_\{0\}\-f\_\{\\infty\}\)L\_\{r\}/T\}\+O\(1/T\)\. ∎

## Appendix IConvergence Rate Comparison

The exact\-Muon column in Table[3](https://arxiv.org/html/2605.16341#A9.T3)should be read as a geometry\-level comparison in the rank\-rrKy Fan setting\. It is complementary to recent full\-rank Muon analyses, which prove convergence for practical Muon variants with Nesterov momentum and weight decay and derive critical batch sizes\[Satoet al\.,[2025](https://arxiv.org/html/2605.16341#bib.bib30)\]\. Our focus is instead the low\-rank spectral update geometry and the rank\-dependent dual\-norm inflation caused by column normalization\.

Table 3:Complete rate comparison across algorithm variants\.
## Appendix JContraction Condition Comparison

The convergence proof requires the per\-step buffer\-tracking contraction rateγ~\\tilde\{\\gamma\}to lie strictly below an admissibility ceiling that depends on how tightly the residual amplification can be controlled\. The original Dion ceiling scales as1/\(1\+2​2​r​κ\)1/\(1\+2\\sqrt\{2r\}\\kappa\), so it shrinks as the rank grows\. Our analysis replaces this with the rank\-independent ceiling1/\(1\+κ\)1/\(1\+\\kappa\), which depends only on the conditioning of the gradient covariance\. Table[4](https://arxiv.org/html/2605.16341#A10.T4)reports both ceilings at the representative regimer=64r\{=\}64,κ=10\\kappa\{=\}10used throughout our LLM experiments: the original bound forcesγ~<0\.0044\\tilde\{\\gamma\}<0\.0044while ours permitsγ~<0\.091\\tilde\{\\gamma\}<0\.091, a22\.6×22\.6\\timesrelaxation\.

Table 4:Contraction condition atr=64r\{=\}64,κ=10\\kappa\{=\}10\. Our Ky Fan dual\-norm analysis \(Section[5](https://arxiv.org/html/2605.16341#S5)\) relaxes the original Dion bound by22\.6×22\.6\\times\.
## Appendix Kβ<1\\beta<1Regimes: Additional Details

This appendix supplements Section[6](https://arxiv.org/html/2605.16341#S6)with the full three\-regime taxonomy of error\-feedback dynamics, the completeβ\\beta\-sweep used to validate Regime 2 on GPT\-2 45M, and the implicit\-momentum derivation behind the predictionβ∗≈1−2​ε^\\beta^\{\*\}\\approx 1\-2\\hat\{\\varepsilon\}\. Withνt\\nu\_\{t\}fixed by Orth\-Dion, the residual recursionRt\+1=β​\(I−Ut​Ut⊤\)​\(Gt\+Rt\)R\_\{t\+1\}=\\beta\(I\-U\_\{t\}U\_\{t\}^\{\\top\}\)\(G\_\{t\}\+R\_\{t\}\)is driven entirely by the cross\-step correlation of the out\-of\-subspace gradient, summarised through the persistenceϕt:=cos⁡∠​\(\(I−Pt\)​Gt,\(I−Pt−1\)​Gt−1\)\\phi\_\{t\}:=\\cos\\angle\(\(I\{\-\}P\_\{t\}\)G\_\{t\},\(I\{\-\}P\_\{t\-1\}\)G\_\{t\-1\}\)\. The sign ofϕt\\phi\_\{t\}determines which of three qualitatively distinct dynamical regimes the optimizer is operating in, and therefore which value ofβ\\betais optimal\.

Table[5](https://arxiv.org/html/2605.16341#A11.T5)summarises the three regimes\. In Regime 1 \(ϕt\>0\\phi\_\{t\}\>0\), gradients are coherent across steps, residuals accumulate constructively, and the tracked subspace eventually rotates to absorb them; full retention \(β=1\\beta=1\) is optimal because it preserves the coherent signal\. In Regime 2 \(ϕt≈0\\phi\_\{t\}\\approx 0, the typical setting for stochastic LLM pretraining\), residual increments are nearly independent, soRTR\_\{T\}executes aT\\sqrt\{T\}random walk that, when projected back onto the tracked subspace, behaves like Polyak momentum with effective coefficientε^2\\hat\{\\varepsilon\}^\{2\}; reducingβ\\betaamplifies this implicit\-momentum signal at zero additional compute\. In Regime 3 \(ϕt<0\\phi\_\{t\}<0\), residuals anti\-correlate across steps and the EF accumulator chases a moving target, so vanishingβ\\betais required to suppress harmful interference\.

Table 5:Three regimes of error feedback dynamics, governed by the gradient persistenceϕt\\phi\_\{t\}\.Table[6](https://arxiv.org/html/2605.16341#A11.T6)reports the complete nine\-pointβ\\beta\-sweep on GPT\-2 45M \(WikiText\-103,r=64r\{=\}64, 30k steps\), of which the body Table[2](https://arxiv.org/html/2605.16341#S8.T2)is a six\-point subset\. All runs share identical optimizer hyperparameters apart fromβ\\beta, and the empirical persistenceϕt≈−0\.01\\phi\_\{t\}\\approx\-0\.01places the configuration squarely in Regime 2\. Validation loss decreases monotonically asβ\\betashrinks from1\.01\.0, with the empirical optimum atβ≈0\.1\\beta\{\\approx\}0\.1\(val\. loss2\.9782\.978vs\.3\.2933\.293atβ=1\\beta\{=\}1\); the R\-norm ratioℛt∗\\mathcal\{R\}\_\{t\}^\{\*\}grows monotonically from0\.30\.3atβ=1\\beta\{=\}1to4\.64\.6atβ=0\.05\\beta\{=\}0\.05, directly tracking the predicted implicit\-momentum accumulation\. Disabling EF entirely \(β=0\\beta\{=\}0\) is far worse than anyβ\>0\\beta\{\>\}0in the sweep, indicating that the gain fromβ<1\\beta\{<\}1comes from tuning the implicit\-momentum coefficient rather than from removing the residual altogether\.

Table 6:Fullβ\\beta\-sweep on GPT\-2 45M/WikiText\-103 \(r=64r\{=\}64, 30k steps; Regime 2,ϕt≈0\\phi\_\{t\}\{\\approx\}0\)\. Validation loss is monotonically non\-increasing inβ\\betafromβ=1\\beta\{=\}1down toβ≈0\.05\\beta\{\\approx\}0\.05–0\.10\.1; below this the implicit\-momentum mechanism saturates \(Section[6](https://arxiv.org/html/2605.16341#S6)\)\. Disabling EF entirely \(β=0\\beta\{=\}0\) is substantially worse than anyβ\>0\\beta\{\>\}0, confirming EF is essential even when heavily damped\. Body Table[2](https://arxiv.org/html/2605.16341#S8.T2)reports the subset \{0, 0\.05, 0\.1, 0\.3, 0\.5, 1\.0\}\.### K\.1Key Quantities

Tracking error:ε^t:=‖\(I−Ut​Ut⊤\)​Mt‖F/‖Mt‖F\\hat\{\\varepsilon\}\_\{t\}:=\\\|\(I\-U\_\{t\}U\_\{t\}^\{\\top\}\)M\_\{t\}\\\|\_\{F\}/\\\|M\_\{t\}\\\|\_\{F\}\. For ColNorm:ε^t≈0\.35\\hat\{\\varepsilon\}\_\{t\}\\approx 0\.35in LLM training\. For Orth\-Dion:ε^t≈0\.007\\hat\{\\varepsilon\}\_\{t\}\\approx 0\.007\.

Gradient subspace persistence:ϕt:=cos⁡∠​\(\(I−Pt\)​Gt,\(I−Pt−1\)​Gt−1\)\\phi\_\{t\}:=\\cos\\angle\(\(I\{\-\}P\_\{t\}\)G\_\{t\},\\;\(I\{\-\}P\_\{t\-1\}\)G\_\{t\-1\}\)\(distinct from the per\-step contractionρt\\rho\_\{t\}of Section[5\.2](https://arxiv.org/html/2605.16341#S5.SS2)\)\.ϕt\>0\\phi\_\{t\}\>0: directions coherent\.ϕt≈0\\phi\_\{t\}\\approx 0: approximately independent \(LLM typical:ϕt≈−0\.01\\phi\_\{t\}\\approx\-0\.01\)\.ϕt<0\\phi\_\{t\}<0: anti\-aligned\.

R\-norm ratio:ℛt:=‖Rt‖F/‖Gt‖F\\mathcal\{R\}\_\{t\}:=\\\|R\_\{t\}\\\|\_\{F\}/\\\|G\_\{t\}\\\|\_\{F\}\. Empiricallyℛt≈0\\mathcal\{R\}\_\{t\}\\approx 0forβ=1\\beta\{=\}1; grows monotonically asβ\\betadecreases\. Optimal sweet spot:ℛt∗∈\[1\.5,2\.0\]\\mathcal\{R\}\_\{t\}^\{\*\}\\in\[1\.5,2\.0\]\.

### K\.2Implicit Momentum Mechanism \(Regime 2\)

Whenϕt≈0\\phi\_\{t\}\\approx 0, the residualRTR\_\{T\}is a sum of approximately independent terms:RT≈∑t=0T−1\(I−Pt\)​GtR\_\{T\}\\approx\\sum\_\{t=0\}^\{T\-1\}\(I\-P\_\{t\}\)G\_\{t\}\. By random\-walk scaling:‖RT‖F∼T⋅ε^⋅‖G‖F\\\|R\_\{T\}\\\|\_\{F\}\\sim\\sqrt\{T\}\\cdot\\hat\{\\varepsilon\}\\cdot\\\|G\\\|\_\{F\}\.

The projected residualPT​RTP\_\{T\}R\_\{T\}acts as a momentum term:𝔼​\[PT​RT\]≈ε^2⋅∑t<TγT−t​∇f​\(wt\)\\mathbb\{E\}\[P\_\{T\}R\_\{T\}\]\\approx\\hat\{\\varepsilon\}^\{2\}\\cdot\\sum\_\{t<T\}\\gamma^\{T\-t\}\\nabla f\(w\_\{t\}\)for effective decayγ∈\(0,1\)\\gamma\\in\(0,1\)depending on subspace rotation rate\. This is equivalent to heavy\-ball momentum with coefficientε^2\\hat\{\\varepsilon\}^\{2\}\.

For ColNorm \(ε^≈0\.35\\hat\{\\varepsilon\}\\approx 0\.35\): implicit momentum≈0\.12\\approx 0\.12—substantial\. For Orth\-Dion \(ε^≈0\.007\\hat\{\\varepsilon\}\\approx 0\.007\): implicit momentum≈5×10−5\\approx 5\\times 10^\{\-5\}—negligible\.*This is why ColNorm\+EF empirically outperforms Orth\-Dion\+EF even whenϕt≈0\\phi\_\{t\}\\approx 0\.*

### K\.3Experimental Evidence: EF≠\\neqExplicit Momentum

PolyakDion \(explicit Polyak momentumμ=0\.95\\mu\{=\}0\.95,R≡0R\{\\equiv\}0by design\) achieves val loss 3\.162\. Dion withβ=0\.3\\beta\{=\}0\.3achieves 3\.056—a gap of 0\.106 nats\. The mechanism: EF accumulation reshapesMt=Gt\+RtM\_\{t\}=G\_\{t\}\+R\_\{t\}*before*the rank\-rrcompression step, directly influencing which subspace is tracked\. Explicit momentum is applied before compression and does not interact with subspace selection in the same way\.

## Appendix LLLM Pre\-training Details

### L\.1320M Pre\-training

Table 7:Llama 3 320M architecture\.Table 8:Training hyperparameters \(320M\)\.Table 9:Optimizer hyperparameters \(320M\)\. Values marked with†were tuned via sweep\. For Muon, Dion, and Ada\-Orth\-Dion, the scalar\-parameter group \(e\.g\., embeddings, biases, layer norms\) is optimized with AdamW\-style updates whose hyperparameters are reused from the best AdamW configuration\.In this paper, we used torchtitan stack to train a Llama 3 architecture model, using C4 data, on the Miyabi\-G supercomputer at JCAHPC, with each run using 8 GH200 \(120GB\) GPUs distributed across 8 nodes interconnected via InfiniBand NDR200\. The number of tokens \(3\.2B\) for the training was calculated based on the Chinchilla optimal heuristic\. For Dion, we used the microsoft implementationAhnet al\.\[[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\], and our proposed Dion variants extended that implementation\. For AdamW, we swept the joint grid of learning rate\{0\.004,0\.008,0\.012,0\.016\}\\\{0\.004,0\.008,0\.012,0\.016\\\}andβ1\\beta\_\{1\}\{0\.9,0\.925,0\.95,0\.975\}\\\{0\.9,0\.925,0\.95,0\.975\\\}\. For Muon and Dion, we swept the joint grid of learning rate\{0\.004,0\.008,0\.012,0\.016\}\\\{0\.004,0\.008,0\.012,0\.016\\\}and output head LR scaling\{True,False\}\\\{\\text\{True\},\\text\{False\}\\\}, where the latter controls whether a shape\-dependent rescaling is applied to the learning rate of the final output projection \(whose fan\-out, the vocabulary size, is much larger than that of the hidden matrices\)\. We reused Dion’s best\-performing configuration for our proposed methods without further tuning\. For Ada\-Orth\-Dion, we set the maximum rank atrm​a​x=384r\_\{max\}=384and initialized the rank atrm​a​xr\_\{max\}to conduct comparison with Dion atr=384r=384\. Table[7](https://arxiv.org/html/2605.16341#A12.T7), Table[8](https://arxiv.org/html/2605.16341#A12.T8), and Table[9](https://arxiv.org/html/2605.16341#A12.T9)list the model architecture, training hyperparameters, and optimizer hyperparameters used for the experiments, respectively\. A single 320M run \(6,100 steps, 3\.2B tokens\) takes 58–76 minutes on 8×\\timesGH200 depending on the optimizer, with peak VRAM of∼40\{\\sim\}40GB/GPU under FSDP full\-shard\.

### L\.2Wide\-and\-Shallow Large\-Scale Wall\-Clock Study

Table 10:w16k\_l4architecture\.HyperparameterValueHidden dimension16,384Intermediate dimension \(FFN\)43,776Number of layers4Number of attention heads64Number of KV heads64Head dimension256Sequence length2048Vocabulary size128,256Total parameters∼\\sim17\.1BTable 11:Training hyperparameters for thew16k\_l4wall\-clock study\.To examine wall\-clock behavior in the bandwidth\-bound regime, we additionally measured per\-step time on a wide\-and\-shallow Llama 3 architecture \(referred to asw16k\_l4\) with approximately 17\.1B parameters \(Figure[1\(b\)](https://arxiv.org/html/2605.16341#S1.F1.sf2)\)\. Table[10](https://arxiv.org/html/2605.16341#A12.T10)and Table[11](https://arxiv.org/html/2605.16341#A12.T11)list the model architecture and training used for the wall\-clock study\. Each 50\-step measurement run takes approximately 40 minutes on 8×\\timesA40 \(48 GB GDDR6 per GPU,∼44\.5\{\\sim\}44\.5GB usable after framework overhead\), with peak VRAM of∼43\{\\sim\}43GB/GPU under FSDP full\-shard\. Aside from the rank fractions set to different values, we reused the best\-performing configurations from the 320M sweep for optimizer hyperparameters\. The model architecture is intentionally wide and shallow so that the per\-block matrices \(e\.g\., FFN matrices of shape43,776×16,38443\{,\}776\\times 16\{,\}384\) are large enough for NCCL collectives to transition from the latency\-bound regime to the bandwidth\-bound regime\. The 8×\\timesA40 hardware was chosen for the same reason: A40s have substantially lower inter\-GPU bandwidth than current\-generation training accelerators \(e\.g\., H100s with NVLink\), so the bandwidth\-bound regime dominates at smaller collective payload sizes than it would on a higher\-end system, making the wall\-clock differences between communication\-light and communication\-heavy optimizers more visible\. In this regime, the reduced communication volume of low\-rank Dion\-family updates relative to Muon’s full\-rank Newton–Schulz updates translates directly into wall\-clock savings\.

For Ada\-Orth\-Dion, we executed the full adaptive rank algorithm, including the per\-block effective rank computation and bookkeeping, but overrode the resulting rank assignment so that the actual working rank was always held fixed atρ=0\.93×0\.25=0\.2325\\rho=0\.93\\times 0\.25=0\.2325, i\.e\., 93% of Dion’s fixed rank fraction, regardless of the computed effective rank or the schedule’s shrink/grow decision\. The 93% factor was chosen to match the average effective rank that Ada\-Orth\-Dion’s adaptive schedule settled at in the 320M experiment, so the wide\-model setting reflects its empirically observed steady\-state\. Since the run still goes through the full Ada\-Orth\-Dion code path \(sort\-and\-pack batching, per\-block adaptive bookkeeping, Cholesky\-QR\), any algorithmic and implementation overhead inherent to Ada\-Orth\-Dion is reflected in the measurement, while holding the rank constant isolates whether the rank reduction enabled by the adaptive mechanism is sufficient to recover this overhead and yield a net wall\-clock improvement over fixed\-rank Orth\-Dion or Dion\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The introduction \(Section[1](https://arxiv.org/html/2605.16341#S1)\) explicitly enumerates the four contributions \(geometric diagnosis of ther\\sqrt\{r\}Ky Fan dual\-norm inflation in Dion; Orth\-Dion as a one\-line ColNorm→\\toQR fix; convergence theory under non\-Euclidean smoothness with self\-consistent residual control and amortized contraction; and Ada\-Orth\-Dion at LLM scale\) and the scope is constrained to the empirical regimes actually studied \(Llama 3 320M pretraining on C4, GPT\-2 45M on WikiText\-103 for theβ\\beta\-sweep, and a Llama 3 17\.1B wide\-and\-shallow wall\-clock measurement\)\.
5. 2\.Limitations
6. Question: Does the paper discuss the limitations of the work performed by the authors?
7. Answer:\[Yes\]
8. Justification: Section[11](https://arxiv.org/html/2605.16341#S11)\(“Limitations” paragraph\) states that the convergence analysis assumes deterministic gradients \(the stochastic extension requires bounding mini\-batch noise/subspace\-tracking interaction\) and that the adaptive\-rank policy currently uses a heuristic effective\-rank estimator rather than a provably\-tight estimate of the contraction\-condition critical rank\. Section[8\.4](https://arxiv.org/html/2605.16341#S8.SS4)additionally notes that theβ\\beta\-sweep is a single\-seed study, and Section[8\.3](https://arxiv.org/html/2605.16341#S8.SS3)notes that the 17\.1B wall\-clock study fixes Ada\-Orth\-Dion’s working rank to its 320M steady\-state value rather than running the adaptive schedule\.
9. 3\.Theory assumptions and proofs
10. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
11. Answer:\[Yes\]
12. Justification: Assumptions A, B′, C′are stated in Section[2](https://arxiv.org/html/2605.16341#S2)and elaborated in Appendix[A](https://arxiv.org/html/2605.16341#A1)\. The one\-step descent inequality is proved in Appendix[D](https://arxiv.org/html/2605.16341#A4); the Orth\-Dion structural lemmas \(Lemmas[4\.1](https://arxiv.org/html/2605.16341#S4.Thmtheorem1)–[4\.3](https://arxiv.org/html/2605.16341#S4.Thmtheorem3), Proposition[3\.1](https://arxiv.org/html/2605.16341#S3.Thmtheorem1)\) in Appendix[E](https://arxiv.org/html/2605.16341#A5); the self\-consistent residual fixed point in Appendix[F](https://arxiv.org/html/2605.16341#A6); the amortized\-contraction tracking\-error bound in Appendix[G](https://arxiv.org/html/2605.16341#A7); the main convergence theorem \(Theorem[5\.1](https://arxiv.org/html/2605.16341#S5.Thmtheorem1)\) in Appendix[H](https://arxiv.org/html/2605.16341#A8); and theβ\\beta\-regime derivations in Appendix[K](https://arxiv.org/html/2605.16341#A11)\.
13. 4\.Experimental result reproducibility
14. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
15. Answer:\[Yes\]
16. Justification: Section[8\.1](https://arxiv.org/html/2605.16341#S8.SS1)specifies the 320M experimental setup \(model, dataset, token budget, hardware, FSDP configuration, baselines, schedule,β\\beta\)\. Appendix[L](https://arxiv.org/html/2605.16341#A12)lists the full Llama 3 320M architecture \(Table[7](https://arxiv.org/html/2605.16341#A12.T7)\), training hyperparameters \(Table[8](https://arxiv.org/html/2605.16341#A12.T8)\), per\-optimizer hyperparameters and the LR / output\-head\-scaling sweep grid \(Table[9](https://arxiv.org/html/2605.16341#A12.T9)\), the Dion implementation reused, and the upstream Microsoft Dion implementation our variants extend\. Appendix[L\.2](https://arxiv.org/html/2605.16341#A12.SS2)mirrors this for the 17\.1B wall\-clock study \(architecture, sequence length, batch size, FSDP/precision, hardware rationale, and the rank\-pinning protocol used for Ada\-Orth\-Dion\)\. Theβ\\beta\-sweep configuration \(model, dataset,rr, step count, schedule\) is given in Section[8\.4](https://arxiv.org/html/2605.16341#S8.SS4)and Table[2](https://arxiv.org/html/2605.16341#S8.T2)\. Algorithms[1](https://arxiv.org/html/2605.16341#alg1)\(Orth\-Dion\) and[4](https://arxiv.org/html/2605.16341#alg4)\(adaptive\-rank policy, Appendix[B\.2](https://arxiv.org/html/2605.16341#A2.SS2)\) specify the methods at the line\-of\-pseudocode level\.
17. 5\.Open access to data and code
18. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
19. Answer:\[Yes\]
20. Justification: An anonymized code release accompanying the submission contains the Orth\-Dion / Ada\-Orth\-Dion implementation \(built on top of the public Microsoft Dion implementationAhnet al\.\[[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\]and the torchtitan training stack\), launch scripts, configuration files for the 320M, 17\.1B, and GPT\-2 45M experiments, and the analysis scripts that produce theνt\\nu\_\{t\}measurements and figures\. The dataset \(C4\) and tokenizer are public; the training corpus is loaded directly from the standard release\. We do not redistribute pretrained weights\.
21. 6\.Experimental setting/details
22. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer, etc\.\) necessary to understand the results?
23. Answer:\[Yes\]
24. Justification: Section[8](https://arxiv.org/html/2605.16341#S8)\(subsections[8\.1](https://arxiv.org/html/2605.16341#S8.SS1),[8\.2](https://arxiv.org/html/2605.16341#S8.SS2),[8\.3](https://arxiv.org/html/2605.16341#S8.SS3),[8\.4](https://arxiv.org/html/2605.16341#S8.SS4)\) reports the per\-experiment training/eval configuration\. Appendix[L](https://arxiv.org/html/2605.16341#A12)provides the full set of hyperparameters and the sweep grids used for each optimizer \(LR,β1\\beta\_\{1\}, output\-head LR scaling\), and explicitly states that the proposed methods reuse Dion’s best\-tuned configuration without further tuning so that Dion vs\. Orth\-Dion comparisons are hyperparameter\-controlled\. Validation is on the C4 validation split \(320M\) and the standard WikiText\-103 split \(GPT\-2 45M\)\.
25. 7\.Experiment statistical significance
26. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27. Answer:\[Yes\]
28. Justification: Table[1](https://arxiv.org/html/2605.16341#S8.T1)and Figure[1\(a\)](https://arxiv.org/html/2605.16341#S1.F1.sf1)report mean±\\pm2 std over33seeds for the 320M Llama 3 results\. Figure[2\(a\)](https://arxiv.org/html/2605.16341#S4.F2.sf1)reports mean over33Dion seeds with a±\\pm1 std band, and Figure[2\(b\)](https://arxiv.org/html/2605.16341#S4.F2.sf2)pools over \(layer, step, seed\)\. Figure[1\(b\)](https://arxiv.org/html/2605.16341#S1.F1.sf2)\(and the 17\.1B wall\-clock plots in Appendix[L\.2](https://arxiv.org/html/2605.16341#A12.SS2)\) report mean±\\pm2 std across the measured per\-step times within each run\. The GPT\-2 45Mβ\\beta\-sweep \(Table[2](https://arxiv.org/html/2605.16341#S8.T2)\) is single\-seed due to compute constraints; this is acknowledged as a limitation, and theβ\\beta\-vs\.\-ℛt∗\\mathcal\{R\}\_\{t\}^\{\*\}trend is monotone over a wide range ofβ\\beta, which we report as evidence of a genuine effect rather than seed noise\.
29. 8\.Experiments compute resources
30. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
31. Answer:\[Yes\]
32. Justification: Appendix[L\.1](https://arxiv.org/html/2605.16341#A12.SS1)states that each 320M run \(6,100 steps, 3\.2B tokens\) takes 58–76 minutes on8×8\\timesGH200 \(120GB\) interconnected via InfiniBand NDR200, with peak∼40\{\\sim\}40GB/GPU under FSDP full\-shard\. Appendix[L\.2](https://arxiv.org/html/2605.16341#A12.SS2)states that each 50\-step 17\.1B measurement run takes∼40\{\\sim\}40minutes on8×8\\timesA40 \(44\.45 GB usable\) under FSDP full\-shard, with peak∼43\{\\sim\}43GB/GPU, and explains the choice of A40 hardware \(lower inter\-GPU bandwidth\) for the bandwidth\-bound regime study\. The GPT\-2 45Mβ\\beta\-sweep configuration \(r=64r\{=\}64, 30k steps\) is given in Section[8\.4](https://arxiv.org/html/2605.16341#S8.SS4)\.
33. 9\.Code of ethics
34. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics?
35. Answer:\[Yes\]
36. Justification: This work studies optimizer geometry for distributed pretraining and does not involve human subjects, sensitive data, deception, dual\-use models, or deployment risks beyond those already present in standard language\-model pretraining\.
37. 10\.Broader impacts
38. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
39. Answer:\[N/A\]
40. Justification: The contribution is a low\-rank distributed optimizer for matrix\-shaped parameters\. Like other algorithmic improvements to pretraining throughput, any societal impact is mediated entirely by the downstream models that practitioners may train more efficiently as a result, and is not specific to this method\.
41. 11\.Safeguards
42. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse?
43. Answer:\[N/A\]
44. Justification: The paper does not release pretrained weights or new datasets\. The released code is an optimizer implementation and supporting analysis scripts\.
45. 12\.Licenses for existing assets
46. Question: Are the creators or original owners of assets used in the paper properly credited and are the license and terms of use explicitly mentioned and properly respected?
47. Answer:\[Yes\]
48. Justification: All upstream assets are cited at point of use: the C4 dataset\[Raffelet al\.,[2020](https://arxiv.org/html/2605.16341#bib.bib14)\], the Llama 3 architecture\[Grattafioriet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib13)\], the Chinchilla token\-budget rule\[Hoffmannet al\.,[2022](https://arxiv.org/html/2605.16341#bib.bib15)\], FSDP\[Zhaoet al\.,[2023](https://arxiv.org/html/2605.16341#bib.bib5)\], Muon\[Jordanet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib8)\], and the Dion algorithm and Microsoft implementation we extend\[Ahnet al\.,[2025b](https://arxiv.org/html/2605.16341#bib.bib3)\], along with related baselines \(PowerSGD\[Vogelset al\.,[2019](https://arxiv.org/html/2605.16341#bib.bib21)\], GaLore\[Zhaoet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib27)\], ReLoRA\[Lialinet al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib28)\], Shampoo\[Guptaet al\.,[2018](https://arxiv.org/html/2605.16341#bib.bib24)\], SOAP\[Vyaset al\.,[2024](https://arxiv.org/html/2605.16341#bib.bib25)\]\)\. The torchtitan training stack used for the 320M and 17\.1B runs is acknowledged in Appendix[L\.1](https://arxiv.org/html/2605.16341#A12.SS1)\. We use these assets within their stated open\-source licenses\.
49. 13\.New assets
50. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
51. Answer:\[Yes\]
52. Justification: The accompanying anonymized code release contains the Orth\-Dion and Ada\-Orth\-Dion optimizer implementations together with launch scripts, configuration files, and the analysis pipeline that generates the reported figures and tables, with a README documenting environment requirements and entry points\.
53. 14\.Crowdsourcing and research with human subjects
54. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants?
55. Answer:\[N/A\]
56. Justification: The paper does not involve crowdsourcing or human subjects\.
57. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
58. Question: Does the paper describe potential risks incurred by study participants?
59. Answer:\[N/A\]
60. Justification: The paper does not involve human subjects\.
61. 16\.Declaration of LLM usage
62. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research?
63. Answer:\[N/A\]
64. Justification: LLMs \(Llama 3, GPT\-2\) appear only as the training targets used to evaluate the proposed optimizer; they are not a component of the method, and no LLM is part of the algorithmic pipeline being analyzed\.