RODE: A Radial-Orthogonal Decoupled Engine for Optimization

# RODE: A Radial-Orthogonal Decoupled Engine for Optimization
Source: [https://arxiv.org/html/2608.21024](https://arxiv.org/html/2608.21024)
Cheng ZhuoZhejiang University, Hangzhou, China\{12647046,12621175\}@zju\.edu\.cnAffiliation:Qi SunAffiliation:Correspondence:\{qisunchn,czhuo\}@zju\.edu\.cn

###### Abstract

Modern neural network training increasingly uses matrix\-aware optimizers, yet their conditioned matrix step is typically added directly to the weight, jointly changing its norm and direction\. This interaction matters because the current norm determines angular motion, while directional learning can drive norm growth and thereby alter later steps\. We introduce RODE, which gives the radial and directional components separate update rules and step sizes\. RODE explicitly updates the matrix Frobenius norm through a scalar radial rule, while its directional channel performs Newton–Schulz\-conditioned updates in the tangent space\. Controlled GPT\-2 interventions show gains from both direct norm control and RODE’s directional update\. Across two language\-modeling and two image\-classification tasks, RODE outperforms both Muon variants in every direct comparison and ends with lower full\-model norms\. At 1\.5B scale, using the learning rate transferred directly from the Qwen2\-style LM sweep, RODE lowers loss from4\.1454\.145to3\.3463\.346and final global norm from1196411964to21832183relative to Muon RMS, with fixed\-radius RODE improving further\. For Qwen3\.5\-9B full\-parameter fine\-tuning, all six optimizers use the same tuning budget and the same formal\-training and evaluation settings; RODE outperforms both Muon variants on all four evaluation tasks and attains the highest mean on GSM8K and MATH\-500\. Thus, decoupling radial and directional dynamics offers a more effective and controllable approach to matrix optimization\.

*Keywords*Optimization⋅\\cdotMatrix parameters⋅\\cdotNewton\-Schulz iteration⋅\\cdotRadial\-directional optimization⋅\\cdotDeep learning

## 1Introduction

Modern neural networks are trained largely by updating weight matrices\. Adam and AdamW adapt individual coordinates\([18](https://arxiv.org/html/2608.21024#bib.bib1);[24](https://arxiv.org/html/2608.21024#bib.bib2)\), whereas matrix\-aware optimizers exploit two\-dimensional structure; Muon, for example, applies Newton–Schulz iterations to momentum\([16](https://arxiv.org/html/2608.21024#bib.bib25);[13](https://arxiv.org/html/2608.21024#bib.bib12)\)\. Despite different constructions, most ultimately add a matrix\-valued step to the current weight,

Wt\+1=Wt\+Δ​Wt\.W\_\{t\+1\}=W\_\{t\}\+\\Delta W\_\{t\}\.\(1\)We refer to this operation as an*additive matrix update*\.

An additive matrix update has two geometrically distinct effects\. Write

Wt=ρt​Ut,ρt=‖Wt‖F,Ut=Wtρt,‖Ut‖F=1,W\_\{t\}=\\rho\_\{t\}U\_\{t\},\\qquad\\rho\_\{t\}=\\\|W\_\{t\}\\\|\_\{\\mathrm\{F\}\},\\qquad U\_\{t\}=\\frac\{W\_\{t\}\}\{\\rho\_\{t\}\},\\qquad\\\|U\_\{t\}\\\|\_\{\\mathrm\{F\}\}=1,\(2\)whereUtU\_\{t\}is direction andρt\\rho\_\{t\}is scale\. In normalized or scale\-invariant blocks, rescaling may preserve the represented function but still change the angular motion of the next update\([33](https://arxiv.org/html/2608.21024#bib.bib15);[14](https://arxiv.org/html/2608.21024#bib.bib14);[12](https://arxiv.org/html/2608.21024#bib.bib33);[19](https://arxiv.org/html/2608.21024#bib.bib16)\); elsewhere, scale can also affect the function directly\. Thus norm and direction play different roles\.

LetQtQ\_\{t\}denote a matrix update produced by a conditioner such as Muon, so thatWt\+1=Wt−η​QtW\_\{t\+1\}=W\_\{t\}\-\\eta Q\_\{t\}\. Then

‖Wt−η​Qt‖F2=ρt2−2​η​ρt​⟨Ut,Qt⟩F\+η2​‖Qt‖F2,\\\|W\_\{t\}\-\\eta Q\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\rho\_\{t\}^\{2\}\-2\\eta\\rho\_\{t\}\\langle U\_\{t\},Q\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\+\\eta^\{2\}\\\|Q\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\},The component parallel toUtU\_\{t\}changes Frobenius norm to first order, whileQt⟂Q\_\{t\}^\{\\perp\}changes direction byη​‖Qt⟂‖F/ρt\+𝒪⁡\(η2\)\\eta\\\|Q\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}/\\rho\_\{t\}\+\\mathcal\{O\}\(\\eta^\{2\}\)\. Even ifQtQ\_\{t\}is tangent, the quadratic term grows the squared norm\. Directional learning can therefore increase norm and reduce the angle realized by later absolute steps\. Muon’s update is generally not tangent toUtU\_\{t\}, so one conditioned step mixes both effects\. Weight decay can oppose growth, but only through an optimizer\- and schedule\-dependent balance\([19](https://arxiv.org/html/2608.21024#bib.bib16);[35](https://arxiv.org/html/2608.21024#bib.bib17)\)\. Explicit radial control instead makes this choice visible and tunable\.

We introduceRODE, aRadial\-OrthogonalDecoupledEngine for matrix optimization\. RODE extracts radial and directional signals, gives them different update rules and learning rates, and retains Newton–Schulz conditioning for direction\. It uses tangent projections and a spherical directional realization, while a scalar rule updates radius\. Radius evolution and directional learning thus become separately adjustable, while momentum across changing tangent spaces introduces a residual interaction that is accounted for in our analysis \(Appendix[D](https://arxiv.org/html/2608.21024#A4)\)\.

RODE retains matrix\-aware conditioning while making norm and directional dynamics explicitly controllable\. We evaluate its components through controlled interventions, its overall optimization performance across four language and vision tasks, hyperparameter transfer at 1\.5B scale, and full\-parameter fine\-tuning at 9B scale\.

Our contributions are:

- •We introduce RODE, a matrix optimizer that reformulates additive matrix optimization in radial–directional coordinates, turning weight scale and direction from implicitly coupled consequences of an update into separately controllable optimization variables\.
- •First, RODE introduces an explicit radial optimization channel\. It extracts the radial gradient of each managed matrix and updates its Frobenius norm with a dedicated scalar rule and learning rate, preventing norm evolution from being an uncontrolled by\-product of directional learning\.
- •Second, RODE develops an orthogonal directional engine for matrix\-aware optimization\. It reprojects momentum onto the current tangent space, applies projected Newton–Schulz conditioning, and realizes the resulting update as an on\-sphere directional step with an independent learning rate\. This construction preserves matrix\-aware conditioning while decoupling directional progress from the current parameter norm\. Our analysis further accounts for the residual interaction induced by momentum across changing tangent spaces and provides convergence guarantees under explicit assumptions \(Appendices[D](https://arxiv.org/html/2608.21024#A4)and[E](https://arxiv.org/html/2608.21024#A5)\)\.
- •We validate both the mechanism and the resulting optimizer empirically\. Controlled interventions separately identify the gains from explicit norm control and the directional engine, while experiments across language modeling and image classification, together with the 1\.5B hyperparameter\-transfer study, demonstrate consistent improvements over Muon variants with substantially more controlled norm growth\. In Qwen3\.5\-9B full\-parameter fine\-tuning, all six optimizers use the same tuning budget and formal\-training and evaluation settings; RODE outperforms both Muon variants on all four evaluation tasks and attains the highest mean on GSM8K and MATH\-500\.

## 2Related Work

RODE connects two often separate lines of work: matrix\-valued update directions and parameter\-scale control\.

#### Adaptive and matrix\-aware optimizers\.

Adam\(W\) combines momentum with coordinate\-wise second moments\([18](https://arxiv.org/html/2608.21024#bib.bib1);[24](https://arxiv.org/html/2608.21024#bib.bib2)\); Adafactor factorizes that state\([32](https://arxiv.org/html/2608.21024#bib.bib7)\), and AdEMAMix mixes fast and slow momenta\([26](https://arxiv.org/html/2608.21024#bib.bib8)\)\. Shampoo uses Kronecker\-factored statistics\([7](https://arxiv.org/html/2608.21024#bib.bib3)\); SOAP updates Adam\-like statistics in the preconditioner’s eigenbasis\([34](https://arxiv.org/html/2608.21024#bib.bib4)\); and MARS\-M adapts variance reduction to matrix updates\([41](https://arxiv.org/html/2608.21024#bib.bib5);[23](https://arxiv.org/html/2608.21024#bib.bib6)\)\. Muon applies short Newton–Schulz iterations to momentum\([16](https://arxiv.org/html/2608.21024#bib.bib25)\), with later work studying its large\-scale recipe\([21](https://arxiv.org/html/2608.21024#bib.bib30)\)\. Muon adds its conditioned step directly; RODE gives radial and matrix\-conditioned directional signals different rules\.

#### Scale, direction, and manifold methods\.

Weight normalization exposes magnitude and direction in the model parameterization\([31](https://arxiv.org/html/2608.21024#bib.bib13)\)\. For scale\-invariant normalized weights, the norm controls the effective learning rate: update\-induced norm growth slows subsequent directional change, and weight decay can counteract this effect\([33](https://arxiv.org/html/2608.21024#bib.bib15);[14](https://arxiv.org/html/2608.21024#bib.bib14);[30](https://arxiv.org/html/2608.21024#bib.bib34)\)\. AdamP removes norm\-increasing radial momentum components to prevent premature effective\-step decay\([12](https://arxiv.org/html/2608.21024#bib.bib33)\)\. Rotational Equilibrium shows that update\-induced growth and weight decay can settle into an equilibrium norm and angular update, and that controlling rotation explicitly recovers important benefits of that balance\([19](https://arxiv.org/html/2608.21024#bib.bib16)\)\. The appropriate AdamW decay also changes with model and dataset scale and can determine whether learning\-rate transfer remains valid\([35](https://arxiv.org/html/2608.21024#bib.bib17)\)\. Together, these results establish weight norm as a controller of optimization dynamics, not merely a regularization statistic\.

LARS/LAMB scale additive steps by parameter norms\([39](https://arxiv.org/html/2608.21024#bib.bib18);[40](https://arxiv.org/html/2608.21024#bib.bib19)\); Fromage uses layer\-relative updates\([3](https://arxiv.org/html/2608.21024#bib.bib31)\); and Nero performs angle\-controlled projected updates per neuron\([22](https://arxiv.org/html/2608.21024#bib.bib32)\)\. RODE instead exposes one matrix\-wide radius as an optimizer coordinate and applies its orthogonalizing kernel to tangent\-projected momentum\. Riemannian optimization methods based on tangent\-space updates and manifold\-preserving maps motivate RODE’s constrained directional update\([1](https://arxiv.org/html/2608.21024#bib.bib9);[4](https://arxiv.org/html/2608.21024#bib.bib10);[2](https://arxiv.org/html/2608.21024#bib.bib11)\)\.

#### Recent radius–direction methods\.

Hyperball wraps a base optimizer by constraining each managed matrix to the sphere defined by its initial Frobenius norm and normalizing the base\-optimizer update, so the learning rate directly sets the relative update length\([37](https://arxiv.org/html/2608.21024#bib.bib36)\)\. The radius remains fixed throughout training\. Magnitude–Direction Decoupling also maintains a fixed\-norm directional factor, while scale is represented by separately optimized row and column gains; the directional factor can be updated by Adam, Muon, or another base optimizer\([8](https://arxiv.org/html/2608.21024#bib.bib35)\)\. In its default form, RODE learns one matrix\-wide scalar radius directly from the radial gradient\. Its directional channel applies projected Newton–Schulz conditioning to tangent\-projected momentum\. The radial and directional channels use independent learning rates\.

## 3Method

RODE decomposes the current signal into radial and directional components, gives them different rules and step sizes, and retains matrix conditioning for direction\. Figure[1](https://arxiv.org/html/2608.21024#S3.F1)contrasts this interface with a direct additive update\. We describe one matrix; other parameters use AdamW\.

Figure 1:Muon conditions momentum and adds the resulting matrix step directly toWtW\_\{t\}, so its effects on norm and direction are jointly determined\. RODE exposes a scalar radial signal and a tangent\-projected directional signal, assigns them separate step sizes, and reconstructs the managed matrix from the resultingρt\+1\\rho\_\{t\+1\}andUt\+1U\_\{t\+1\}\.### 3\.1Radial–tangent gradient decomposition

Letgtg\_\{t\}be a stochastic estimator ofGt=∇f​\(Wt\)G\_\{t\}=\\nabla f\(W\_\{t\}\), and let⟨A,B⟩F=tr⁡\(A⊤​B\)\\langle A,B\\rangle\_\{\\mathrm\{F\}\}=\\mathrm\{tr\}\(A^\{\\top\}B\)\. We use𝒫t\\mathcal\{P\}\_\{t\}to denote the orthogonal projection onto the tangent space at the current directionUtU\_\{t\}\. Specifically,

𝒫t​\(X\)\\displaystyle\\mathcal\{P\}\_\{t\}\(X\)=X−⟨X,Ut⟩F​Ut,\\displaystyle=X\-\\langle X,U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}U\_\{t\},gtρ\\displaystyle g\_\{t\}^\{\\rho\}=⟨gt,Ut⟩F,\\displaystyle=\\langle g\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\},gt⟂\\displaystyle g\_\{t\}^\{\\perp\}=𝒫t​\(gt\)\.\\displaystyle=\\mathcal\{P\}\_\{t\}\(g\_\{t\}\)\.\(3\)Then⟨gt⟂,Ut⟩F=0\\langle g\_\{t\}^\{\\perp\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}=0, sogtρg\_\{t\}^\{\\rho\}andgt⟂g\_\{t\}^\{\\perp\}are the instantaneous radial and tangent components of the current gradient\. Under the parameterizationWt=ρt​UtW\_\{t\}=\\rho\_\{t\}U\_\{t\}, the gradient with respect to the directional coordinateUUisρt​gt⟂\\rho\_\{t\}g\_\{t\}^\{\\perp\}\. RODE omits this scale factor when forming the directional signal, so that the directional update is not directly scaled by the current matrix norm\. Since a positive scalar does not change the normalized polar direction used by the Newton–Schulz conditioner,ηdir\\eta\_\{\\mathrm\{dir\}\}can directly control the directional step\.

### 3\.2Radial update

LetBt−1B\_\{t\-1\}denote the raw momentum buffer stored from the previous step, and recall that𝒫t\\mathcal\{P\}\_\{t\}projects a matrix onto the tangent space at the current directionUtU\_\{t\}\. The radial channel updates only the matrix Frobenius norm, using the current gradient component parallel toUtU\_\{t\}\. RODE first reprojects the historical momentum onto the current tangent space and then adds the current gradient:

B~t\\displaystyle\\widetilde\{B\}\_\{t\}=β​𝒫t​\(Bt−1\)\+gt,\\displaystyle=\\beta\\mathcal\{P\}\_\{t\}\(B\_\{t\-1\}\)\+g\_\{t\},st\\displaystyle s\_\{t\}=⟨B~t,Ut⟩F=gtρ,\\displaystyle=\\langle\\widetilde\{B\}\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}=g\_\{t\}^\{\\rho\},ρt\+1\\displaystyle\\rho\_\{t\+1\}=max⁡\{ρt−ηρ​st,κfloor​ρt\}\.\\displaystyle=\\max\\\{\\rho\_\{t\}\-\\eta\_\{\\rho\}s\_\{t\},\\kappa\_\{\\mathrm\{floor\}\}\\rho\_\{t\}\\\}\.\(4\)Because𝒫t​\(Bt−1\)\\mathcal\{P\}\_\{t\}\(B\_\{t\-1\}\)lies in the tangent space atUtU\_\{t\}, it is orthogonal toUtU\_\{t\}and therefore contributes no radial component\. Hence the radial signal satisfiesst=gtρs\_\{t\}=g\_\{t\}^\{\\rho\}and is determined only by the current gradient\. The scalar learning rateηρ\\eta\_\{\\rho\}directly controls the change in the matrix Frobenius norm\. The multiplicative floor prevents a single update from collapsing the radius or crossing zero; our analysis considers the interior regime in which this safeguard is inactive\.

### 3\.3Tangent momentum and conditioning

RODE forms the tangent momentum by projecting the updated raw buffer,

Mt⟂=𝒫t​\(B~t\),M\_\{t\}^\{\\perp\}=\\mathcal\{P\}\_\{t\}\(\\widetilde\{B\}\_\{t\}\),\(5\)while storingBt=B~tB\_\{t\}=\\widetilde\{B\}\_\{t\}for the next step\. Because the directionUtU\_\{t\}changes during training, radial content retained in the raw buffer can enter a future tangent space after reprojection\. Appendix[D](https://arxiv.org/html/2608.21024#A4)bounds this cross\-step residual\.

For wide matrices, RODE first transposes the matrix and initializes

X0=Mt⟂‖Mt⟂‖F\.X\_\{0\}=\\frac\{M\_\{t\}^\{\\perp\}\}\{\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\}\.\(6\)It then applies the projected Newton–Schulz iterations

Ak\\displaystyle A\_\{k\}=Xk⊤​Xk,\\displaystyle=X\_\{k\}^\{\\top\}X\_\{k\},Xk\+1\\displaystyle X\_\{k\+1\}=𝒫t​\[3\.4445​Xk\+Xk​\{−4\.7750​Ak\+2\.0315​Ak2\}\]\.\\displaystyle=\\mathcal\{P\}\_\{t\}\\\!\\left\[3\.4445X\_\{k\}\+X\_\{k\}\\\{\-4\.7750A\_\{k\}\+2\.0315A\_\{k\}^\{2\}\\\}\\right\]\.\(7\)Newton–Schulz iterations approximate matrix sign and polar factors\([13](https://arxiv.org/html/2608.21024#bib.bib12)\)\. Each polynomial step can introduce a component parallel toUtU\_\{t\}, so RODE projects the result back onto the current tangent space after every iteration\. AfterKKiterations, we denote the resulting conditioned tangent kernel by

Q^t=XK\.\\widehat\{Q\}\_\{t\}=X\_\{K\}\.\(8\)Its Frobenius norm is used in the directional update below\.

### 3\.4Spherical directional realization

Let

qt=‖Q^t‖F\.q\_\{t\}=\\\|\\widehat\{Q\}\_\{t\}\\\|\_\{\\mathrm\{F\}\}\.\(9\)Forqt\>0q\_\{t\}\>0, define the normalized tangent directionDt⟂D\_\{t\}^\{\\perp\}and rotation angleθt\\theta\_\{t\}as

Dt⟂\\displaystyle D\_\{t\}^\{\\perp\}=Q^t/qt,\\displaystyle=\\widehat\{Q\}\_\{t\}/q\_\{t\},θt\\displaystyle\\theta\_\{t\}=ηdir​qt,\\displaystyle=\\eta\_\{\\mathrm\{dir\}\}q\_\{t\},Ut\+1\\displaystyle U\_\{t\+1\}=Utcosθt−Dt⟂sinθt,\\displaystyle=U\_\{t\}\\cos\\theta\_\{t\}\-D\_\{t\}^\{\\perp\}\\sin\\theta\_\{t\},Wt\+1\\displaystyle W\_\{t\+1\}=ρt\+1​Ut\+1\.\\displaystyle=\\rho\_\{t\+1\}U\_\{t\+1\}\.\(10\)BecauseDt⟂D\_\{t\}^\{\\perp\}lies in the tangent space atUtU\_\{t\}, we have⟨Dt⟂,Ut⟩F=0\\langle D\_\{t\}^\{\\perp\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}=0, which gives‖Ut\+1‖F=1\\\|U\_\{t\+1\}\\\|\_\{\\mathrm\{F\}\}=1in exact arithmetic\. The implementation renormalizesUt\+1U\_\{t\+1\}for numerical stability and handles theqt=0q\_\{t\}=0case separately; Appendix[A](https://arxiv.org/html/2608.21024#A1)gives pseudocode\.

### 3\.5Optional scalar damping

RODE\-SNR optionally damps the directional step using a running estimate of the tangent\-gradient scale\. We maintain a scalar statevtv\_\{t\}that accumulates the recent magnitude ofgt⟂g\_\{t\}^\{\\perp\}:

vt=β​vt−1\+‖gt⟂‖F\+ϵ\.v\_\{t\}=\\beta v\_\{t\-1\}\+\\\|g\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\+\\epsilon\.\(11\)The damping factor compares the current tangent\-momentum norm with this reference scale:

γt=clip⁡\(‖Mt⟂‖Fvt,γmin,1\)\.\\gamma\_\{t\}=\\mathrm\{clip\}\\\!\\left\(\\frac\{\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\}\{v\_\{t\}\},\\gamma\_\{\\min\},1\\right\)\.\(12\)RODE\-SNR then uses

θt=ηdir​γt​qt,\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\},\(13\)soγt\\gamma\_\{t\}changes only the directional step magnitude and leavesDt⟂D\_\{t\}^\{\\perp\}unchanged\. Default RODE setsγt=1\\gamma\_\{t\}=1; we report RODE\-SNR as an optional, separately tuned variant\.

## 4Theory

Appendix[D](https://arxiv.org/html/2608.21024#A4)analyzes the implemented raw\-buffer recurrence in the nondegenerate interior regime, where the radial floor is inactive and the projected tangent kernel remains nonzero\. The analysis assumes smoothness, bounded stochastic noise and iterates, local tangent\-gradient regularity, bounded rotation angles, and positive alignment between the projected Newton–Schulz kernel and the tangent momentum\.

For the full gradientGt=∇f​\(Wt\)G\_\{t\}=\\nabla f\(W\_\{t\}\), letGtρ=⟨Gt,Ut⟩FG\_\{t\}^\{\\rho\}=\\langle G\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}andGt⟂=𝒫t​\(Gt\)G\_\{t\}^\{\\perp\}=\\mathcal\{P\}\_\{t\}\(G\_\{t\}\)denote its radial and tangent components\. Under the stated assumptions, an exact expansion of the displacementWt\+1−WtW\_\{t\+1\}\-W\_\{t\}separates radial descent, tangent descent, geodesic curvature, and cross\-step leakage from the raw momentum buffer\. A Lyapunov function combiningf⁡\(Wt\)f\(W\_\{t\}\)with the tangent\-momentum tracking error yields

lim supT→∞1T​∑t=1T𝔼⁡\[\(Gtρ\)2\]\\displaystyle\\limsup\_\{T\\to\\infty\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\\\!\\left\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\\right\]≤ϵρ,\\displaystyle\\leq\\epsilon\_\{\\rho\},lim supT→∞1T​∑t=1T𝔼⁡\[‖Gt⟂‖F2\]\\displaystyle\\limsup\_\{T\\to\\infty\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\\\!\\left\[\\\|G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\]≤ϵ⟂\.\\displaystyle\\leq\\epsilon\_\{\\perp\}\.\(14\)The explicit expressions forϵρ\\epsilon\_\{\\rho\}andϵ⟂\\epsilon\_\{\\perp\}are given in Theorem[1](https://arxiv.org/html/2608.21024#Thmtheorem1); they depend on the radial and directional step sizes, the stochastic\-noise levels, and the constants in the stated assumptions\. These bounds cover both gradient components even though the stored raw momentum need not remain perfectly tangent as the directionUtU\_\{t\}changes\.

Appendix[E](https://arxiv.org/html/2608.21024#A5)separately studies the first\-order approximation

Wt\+1=Wt−η​Ht,W\_\{t\+1\}=W\_\{t\}\-\\eta H\_\{t\},\(15\)whereHtH\_\{t\}is the effective update direction andη\>0\\eta\>0is its step size\. The approximation assumes thatHtH\_\{t\}has uniformly bounded Frobenius norm and remains positively aligned with the full gradient:

⟨Gt,Ht⟩F≥α​‖Gt‖F2,α\>0,\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\geq\\alpha\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\\qquad\\alpha\>0,\(16\)whereα\\alphaquantifies the minimum degree of alignment\. The objective is also assumed to satisfy the Polyak–Lojasiewicz condition\([17](https://arxiv.org/html/2608.21024#bib.bib26)\),

12​‖Gt‖F2≥μ⁡\(f⁡\(Wt\)−f∗\),μ\>0,\\frac\{1\}\{2\}\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\geq\\mu\\bigl\(f\(W\_\{t\}\)\-f^\{\*\}\\bigr\),\\qquad\\mu\>0,\(17\)whereμ\\muis the PL constant andf∗=infWf⁡\(W\)f^\{\*\}=\\inf\_\{W\}f\(W\)is the optimal objective value\. If0<2​η​α​μ<10<2\\eta\\alpha\\mu<1, smoothness gives

f⁡\(Wt\)−f∗≤\(1−2​η​α​μ\)t​\(f⁡\(W0\)−f∗\)\+𝒪⁡\(η\)\.f\(W\_\{t\}\)\-f^\{\*\}\\leq\(1\-2\\eta\\alpha\\mu\)^\{t\}\\bigl\(f\(W\_\{0\}\)\-f^\{\*\}\\bigr\)\+\\mathcal\{O\}\(\\eta\)\.\(18\)Thus the aligned first\-order model converges linearly to a neighborhood whose size is controlled byη\\eta\. Its contraction factor1−2​η​α​μ1\-2\\eta\\alpha\\mudepends jointly on the alignment strengthα\\alphaand the PL constantμ\\mu\.

## 5Experiments

We test optimization quality and norm behavior, separate radial and directional contributions, then examine scale, modality, components, and systems cost\.

### 5\.1Experimental questions and setup

We study two WikiText\-103\([25](https://arxiv.org/html/2608.21024#bib.bib21)\)decoders: standard GPT\-2\([29](https://arxiv.org/html/2608.21024#bib.bib20)\)for 8000 steps and a randomly initialized Qwen2\-style language model for 4000 steps\. The latter has 114M parameters and uses RoPE, RMSNorm, grouped\-query attention, and SwiGLU\([38](https://arxiv.org/html/2608.21024#bib.bib27)\); it is a locally instantiated architecture rather than an official Qwen checkpoint\. Vision experiments train ResNet\-50\([9](https://arxiv.org/html/2608.21024#bib.bib22)\)on CIFAR\-100\([20](https://arxiv.org/html/2608.21024#bib.bib23)\)and ImageNet\-1K\([6](https://arxiv.org/html/2608.21024#bib.bib24)\)\. For a separate scaling study, we scale the Qwen2\-style architecture to a 1\.5B\-parameter decoder and train it for 20000 steps on FineWeb\([27](https://arxiv.org/html/2608.21024#bib.bib28)\)\. We additionally conduct a full\-parameter fine\-tuning benchmark on Qwen3\.5\-9B\([28](https://arxiv.org/html/2608.21024#bib.bib37)\); all six optimizers use the same tuning budget and formal\-training and evaluation settings\.

All reported optimization\-quality comparisons use three seeds\. In language pretraining, RODE manages two\-dimensional non\-embedding, non\-head matrices; in vision it manages flattened multi\-dimensional weights\. Remaining parameters use AdamW\. A within\-task formal comparison shares the model, data, horizon, seed set, and checkpoint rule; optimizer\-specific rates are selected by Optuna or transferred as marked\. The original vision suite reuses standard\-GPT\-2 settings, added baselines reuse Qwen2\-style\-LM settings, and the 1\.5B test reuses Qwen2\-style\-LM settings without another sweep\. Appendix[B](https://arxiv.org/html/2608.21024#A2)specifies routing, search spaces, schedules, and selected values\. Evaluation metrics and all tabulated values use raw logs\.

Table 1:Cross\-task optimization performance on language modeling and image classification benchmarks\. Results are reported as mean±\\pmsample standard deviation over three seeds\. Vision reports best accuracy on the official CIFAR\-100 test set and the ImageNet\-1K validation set\.†\\daggerRates transferred from the Qwen2\-style LM;‡\\ddaggerrates transferred from GPT\-2\. Best available mean per column is bold\.
### 5\.2Optimization quality and norm control

RODE shows consistent gains across all four main benchmarks \(Table[1](https://arxiv.org/html/2608.21024#S5.T1)\)\. Compared with the stronger Muon variant on each task, it reduces loss by0\.04910\.0491on GPT\-2 and0\.04540\.0454on the Qwen2\-style LM, while improving accuracy by2\.362\.36points on CIFAR\-100 and4\.484\.48points on ImageNet\-1K\. Its performance also remains competitive against the broader optimizer set: SOAP gives the best GPT\-2 result, RODE\-SNR leads on the Qwen2\-style LM and ImageNet\-1K, and RODE leads on CIFAR\-100\. All comparisons are made within task, with tuning and transfer sources indicated in Table[1](https://arxiv.org/html/2608.21024#S5.T1)\.

The norm trajectories reveal a complementary aspect of RODE’s behavior\. Among non\-SGD methods, a RODE variant attains the lowest observed final full\-model norm on every task; on GPT\-2, CIFAR\-100, and ImageNet\-1K, the two lowest values are both obtained by RODE variants \(Table[2](https://arxiv.org/html/2608.21024#S5.T2)\)\. Muon RMS finishes at2\.52\.5–5\.7×5\.7\\timesthe norm of RODE\. This pattern aligns with the motivation for explicit norm control: for scale\-invariant weights, parameter norm governs effective directional progress, and sustained update\-induced norm growth can progressively reduce subsequent angular steps\([33](https://arxiv.org/html/2608.21024#bib.bib15);[12](https://arxiv.org/html/2608.21024#bib.bib33);[19](https://arxiv.org/html/2608.21024#bib.bib16)\)\. Across these experiments, improved task performance under RODE is accompanied by substantially more controlled norm evolution\. Full\-model norm includes all trainable parameters; Appendix[C\.2](https://arxiv.org/html/2608.21024#A3.SS2)gives the exact definition\. Figure[2](https://arxiv.org/html/2608.21024#S5.F2)shows the GPT\-2 trajectories, with the Qwen2\-style LM results reported in Appendix Figure[5](https://arxiv.org/html/2608.21024#A3.F5)\.

Table 2:Final observed full\-model weight norm at the end of training \(three\-seed mean±\\pmsample SD\)\. Values include every trainable parameter, including tensors delegated to AdamW, and are comparable only within a task\. A dash indicates that the task–optimizer combination was not run; values are descriptive rather than ranked\.
Figure 2:Standard GPT\-2/WikiText\-103 over 8000 steps \(three\-seed mean±\\pmsample SD\): validation loss \(left\) and observed full\-model weight norm \(right\)\. The norm aggregates all trainable parameters, including those delegated to AdamW; both RODE variants show substantially less growth than the Muon runs\.
### 5\.3Separating norm control from directional conditioning

Lower norms alone do not establish an optimization benefit, so Table[3](https://arxiv.org/html/2608.21024#S5.T3)introduces the radial and directional interventions sequentially\. Fixing the radii of Muon\-managed matrices lowers validation loss from3\.0273\.027to3\.0083\.008\. Replacing Muon’s additive directional update with RODE’s tangent directional update further lowers the loss to2\.9772\.977, while full RODE with learned radii reaches the same level\. These interventions show a measurable benefit from explicit norm control and a further gain from RODE’s directional update\. Muon RMS and full RODE use learning rates selected from their respective GPT\-2 sweeps; Muon \+ NormProj inherits the Muon RMS settings, and fixed\-radius RODE inherits the full RODE settings\.

Table 3:Sequential GPT\-2 norm–direction intervention\. Three\-seed values are mean±\\pmsample SD;Δ\\Deltais relative to Muon RMS\.
### 5\.4Hyperparameter transfer at 1\.5B scale

We train a Qwen2\-style 1\.5B decoder for 20000 FineWeb steps, transferring the smaller model’s rates without a new sweep \(Table[4](https://arxiv.org/html/2608.21024#S5.T4)\)\. With the transferred rates, RODE achieves lower loss and a substantially lower global norm than Muon RMS\. Fixed\-radius RODE improves both measures further\. Figure[3](https://arxiv.org/html/2608.21024#S5.F3)shows the trajectories; Appendices[C\.3](https://arxiv.org/html/2608.21024#A3.SS3)and[B](https://arxiv.org/html/2608.21024#A2)give the cache construction and complete protocol\.

Table 4:1\.5B scale transfer with rates selected on the smaller Qwen2\-style LM \(three seeds, mean±\\pmsample SD\)\.
Figure 3:Qwen2\-style 1\.5B/FineWeb scale transfer over 20000 steps \(three\-seed mean±\\pmsample SD\): unsmoothed fixed\-cache loss and observed global weight norm\. Fixed\-radius RODE freezes managed matrix radii; parameters delegated to AdamW can still change the global norm\.
### 5\.5Full\-parameter fine\-tuning at 9B scale

We full\-parameter fine\-tune Qwen3\.5\-9B on GSM8K\([5](https://arxiv.org/html/2608.21024#bib.bib38)\)\. Each of the six optimizers receives the same tuning budget of 20 trials of 800 steps, followed by three 3000\-step runs with distinct seeds; the minimum\-validation\-loss checkpoint from each run is evaluated on GSM8K, MATH\-500\([11](https://arxiv.org/html/2608.21024#bib.bib39)\), MMLU\-STEM\([10](https://arxiv.org/html/2608.21024#bib.bib40)\), and MMLU\-Pro Math\([36](https://arxiv.org/html/2608.21024#bib.bib41)\), as reported in Table[5](https://arxiv.org/html/2608.21024#S5.T5)\. Appendix[B](https://arxiv.org/html/2608.21024#A2)gives the complete training protocol, search spaces, selected learning rates, parameter routing, and task definitions\.

Table 5:Qwen3\.5\-9B full\-parameter fine\-tuning and task evaluation \(three\-seed mean±\\pmsample SD\)\. All task columns report accuracy in percent\. Each optimizer is independently tuned, and downstream evaluation uses the minimum\-validation\-loss checkpoint from each run\. Best mean per column is bold\.RODE outperforms both Muon variants on all four evaluation tasks\. Across all six optimizers, it achieves the highest mean on GSM8K and MATH\-500, while Adafactor leads on MMLU\-STEM and MMLU\-Pro Math\. Appendix Figure[7](https://arxiv.org/html/2608.21024#A3.F7)provides the full tuning and validation\-loss trajectories\.

### 5\.6Transfer to computer vision

With transferred hyperparameters, RODE reaches73\.75±0\.2973\.75\\pm 0\.29best accuracy on CIFAR\-100 and69\.74±0\.4769\.74\\pm 0\.47on ImageNet\-1K, exceeding both Muon variants; RODE\-SNR has the highest ImageNet\-1K mean\. RODE’s final norms are348\.6±0\.3348\.6\\pm 0\.3and801\.8±27\.6801\.8\\pm 27\.6, versus1142\.0±1\.21142\.0\\pm 1\.2and4547\.4±6\.94547\.4\\pm 6\.9for Muon RMS\. Thus the joint accuracy and norm behavior transfers beyond language models \(Figure[4](https://arxiv.org/html/2608.21024#S5.F4)\); Appendix[C](https://arxiv.org/html/2608.21024#A3)gives norm trajectories\.

Figure 4:ResNet\-50 test accuracy on CIFAR\-100 \(left\) and validation accuracy on ImageNet\-1K \(right\), each over three seeds\.
### 5\.7Component necessity

As shown in Table[6](https://arxiv.org/html/2608.21024#S5.T6), all six Qwen2\-style\-LM variants follow the same 4000\-step protocol\. Structural variants inherit RODE’s learning rates, while RODE\-SNR is independently tuned\. Removing projected Newton–Schulz conditioning or directional learning substantially degrades performance, confirming that both are important components of RODE\.

Table 6:Three\-seed 114M component study \(mean±\\pmsample SD\)\. Structural ablations inherit RODE rates and are not retuned; RODE\-SNR is an independently tuned diagnostic\.Δ\\Deltais relative to full RODE\.
### 5\.8Systems cost

The radial coordinate adds computation but no matrix\-sized state\. On the Qwen2\-style\-LM benchmark, RODE stores611611MB, equal to Muon and33%33\\%below AdamW; shape\-batched fusion raises throughput from0\.59×0\.59\\timesto0\.72×0\.72\\timesAdamW, versus0\.77×0\.77\\timesfor Muon RMS\. On ResNet\-50, fused RODE stores102\.4102\.4MB and reaches0\.64×0\.64\\times, versus0\.70×0\.70\\timesfor Muon RMS\. Table[7](https://arxiv.org/html/2608.21024#S5.T7)gives controlled measurements; Appendix[A\.2](https://arxiv.org/html/2608.21024#A1.SS2)details the implementation\.

Table 7:Compute and memory benchmark\. Measurements use a single RTX 5090 D v2 withbf16autocast: the Qwen2\-style LM uses micro\-batch 16 and sequence length 512, while ResNet\-50 uses batch 64 and224×224224\\times 224inputs\. Each of three runs has 8 warm\-up and 25 timed steps; Step and Opt\. are medians of the three per\-run medians\. Peak is PyTorch peak allocated memory, State is exact optimizer\-state tensor storage, Opt\. includesstep\(\)and gradient clearing, and throughput is relative to AdamW\.

## 6Discussion

#### What RODE contributes\.

RODE combines a learned matrix\-radius policy, a separately tuned directional policy, and matrix conditioning retained for direction\. It turns two effects jointly determined by direct matrix addition into visible optimizer choices\.

#### Why radius control matters\.

An additive step makes norm both an outcome of directional learning and a determinant of later angular progress\. Growth can therefore suppress directional motion, while weight decay regulates the interaction only indirectly\([33](https://arxiv.org/html/2608.21024#bib.bib15);[14](https://arxiv.org/html/2608.21024#bib.bib14);[19](https://arxiv.org/html/2608.21024#bib.bib16)\)\. RODE gives the radial signal its own rule and removes the explicit1/ρt1/\\rho\_\{t\}factor from directional step size\. Its goal is not to minimize norm, but to keep unintended norm growth from silently governing directional optimization\.

#### What the evidence supports\.

RODE reduces observed norm growth while improving over both Muon variants in every main language\-modeling and vision comparison\. The GPT\-2 intervention attributes gains to radial control and then the directional engine; in the 1\.5B hyperparameter\-transfer study, learned\-radius RODE improves both loss and global norm, and fixed radius improves further\. At 9B scale, each of the six optimizers is tuned with the same budget, then trained and evaluated under the same settings\. RODE outperforms both Muon variants on GSM8K, MATH\-500, MMLU\-STEM, and MMLU\-Pro Math; among all six optimizers, it attains the highest mean on GSM8K and MATH\-500\. Together, these results support explicit norm control as an effective mechanism in matrix optimization\.

#### Limitations\.

RODE currently applies its radial–directional update only to matrix\-valued parameters, while the remaining parameters are delegated to AdamW\. The additional tangent projections and directional updates introduce extra computational overhead\. We evaluate full\-parameter fine\-tuning of Qwen3\.5\-9B and training of a randomly initialized Qwen2\-style 1\.5B model, but the current study covers a limited set of language and vision architectures and training budgets; broader scales and model families remain to be explored\. The theoretical convergence guarantees and rates are established under idealized regularity and alignment assumptions\.

## 7Conclusion

Direct additive matrix updates couple weight\-norm evolution with directional progress, allowing norm growth to alter subsequent optimization dynamics\. RODE separates these effects by treating the matrix Frobenius norm and the matrix\-conditioned direction as distinct optimization coordinates with independent update rules and step sizes\. Controlled interventions isolate gains from both radial control and directional learning, while experiments across language modeling and image classification show consistent improvements over Muon variants together with substantially more controlled norm growth\. The 1\.5B hyperparameter\-transfer study further shows that these gains persist when learning rates are transferred from a smaller model, with fixed\-radius RODE improving further\. In Qwen3\.5\-9B full\-parameter fine\-tuning, RODE outperforms both Muon variants on all four evaluation tasks and leads the six\-optimizer comparison on GSM8K and MATH\-500\. These results support radial–directional decoupling as an effective and practical design strategy for matrix\-aware optimization\.

## References

- Absilet al\.\(2008\)P\.\-A\. Absil, R\. Mahony, and R\. SepulchreOptimization algorithms on matrix manifolds\.Princeton University Press\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Bécigneul and Ganea \(2019\)G\. Bécigneul and O\. GaneaRiemannian adaptive optimization methods\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Bernsteinet al\.\(2020\)J\. Bernstein, A\. Vahdat, Y\. Yue, and M\. LiuOn the distance between two neural networks and the stability of learning\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Bonnabel \(2013\)S\. BonnabelStochastic gradient descent on Riemannian manifolds\.IEEE Transactions on Automatic Control58\(9\),pp\. 2217–2229\.External Links:[Document](https://dx.doi.org/10.1109/TAC.2013.2254619)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§5\.5](https://arxiv.org/html/2608.21024#S5.SS5.p1.1)\.
- Denget al\.\(2009\)J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-FeiImageNet: a large\-scale hierarchical image database\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 248–255\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2009.5206848)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Guptaet al\.\(2018\)V\. Gupta, T\. Koren, and Y\. SingerShampoo: preconditioned stochastic tensor optimization\.InInternational Conference on Machine Learning,Vol\.80,pp\. 1842–1850\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Hägeleet al\.\(2026\)A\. Hägele, A\. Hernández\-Cano, A\. Kosson, and M\. JaggiImproving neural network training by decoupling the magnitude and direction of weight vectors\.arXiv preprint arXiv:2606\.25971\.External Links:[Link](https://arxiv.org/abs/2606.25971)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px3.p1.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.90)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§5\.5](https://arxiv.org/html/2608.21024#S5.SS5.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Vol\.1\.External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html)Cited by:[§5\.5](https://arxiv.org/html/2608.21024#S5.SS5.p1.1)\.
- Heoet al\.\(2021\)B\. Heo, S\. Chun, S\. J\. Oh, D\. Han, S\. Yun, G\. Kim, Y\. Uh, and J\. HaAdamP: slowing down the slowdown for momentum optimizers on scale\-invariant weights\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p2.2),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2608.21024#S5.SS2.p2.1)\.
- Higham \(2008\)N\. J\. HighamFunctions of matrices: theory and computation\.Society for Industrial and Applied Mathematics\.External Links:[Document](https://dx.doi.org/10.1137/1.9780898717778)Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p1.1),[§3\.3](https://arxiv.org/html/2608.21024#S3.SS3.p2.3)\.
- Hofferet al\.\(2018\)E\. Hoffer, R\. Banner, I\. Golan, and D\. SoudryNorm matters: efficient and accurate normalization schemes in deep networks\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p2.2),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2608.21024#S6.SS0.SSS0.Px2.p1.1)\.
- HuggingFaceFW \(2025\)HuggingFaceFWFineWeb dataset card\.Note:Version 1\.2\.0; accessed July 31, 2026External Links:[Document](https://dx.doi.org/10.57967/hf/2493),[Link](https://huggingface.co/datasets/HuggingFaceFW/fineweb)Cited by:[§C\.3](https://arxiv.org/html/2608.21024#A3.SS3.p1.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.Note:[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)Accessed August 17, 2026Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p1.1),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Karimiet al\.\(2016\)H\. Karimi, J\. Nutini, and M\. SchmidtLinear convergence of gradient and proximal\-gradient methods under the polyak–Łojasiewicz condition\.InEuropean Conference on Machine Learning and Knowledge Discovery in Databases,pp\. 795–811\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-46128-1%5F50)Cited by:[§4](https://arxiv.org/html/2608.21024#S4.p3.3)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p1.1),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Kossonet al\.\(2024\)A\. Kosson, B\. Messmer, and M\. JaggiRotational equilibrium: how weight decay balances learning across neural networks\.InInternational Conference on Machine Learning,Vol\.235,pp\. 25333–25369\.Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p2.2),[§1](https://arxiv.org/html/2608.21024#S1.p3.2),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2608.21024#S5.SS2.p2.1),[§6](https://arxiv.org/html/2608.21024#S6.SS0.SSS0.Px2.p1.1)\.
- Krizhevsky \(2009\)A\. KrizhevskyLearning multiple layers of features from tiny images\.Technical Report\.External Links:[Link](https://www.cs.toronto.edu/%CB%9Ckriz/learning-features-2009-TR.pdf)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Liuet al\.\(2025a\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.Muon is scalable for LLM training\.arXiv preprint arXiv:2502\.16982\.External Links:[Link](https://arxiv.org/abs/2502.16982)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2021\)Y\. Liu, J\. Bernstein, M\. Meister, and Y\. YueLearning by turning: neural architecture aware optimisation\.InInternational Conference on Machine Learning,Vol\.139,pp\. 6748–6758\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Liuet al\.\(2025b\)Y\. Liu, A\. Yuan, and Q\. GuMARS\-M: when variance reduction meets matrices\.External Links:2510\.21800,[Link](https://arxiv.org/abs/2510.21800)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p1.1),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.InInternational Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Pagliardiniet al\.\(2025\)M\. Pagliardini, P\. Ablin, and D\. GrangierThe AdEMAMix optimizer: better, faster, older\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfThe FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-0970)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.Note:Accessed August 21, 2026External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.OpenAI Technical Report\.External Links:[Link](https://cdn.openai.com/better-language-models/language-models.pdf)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Roburinet al\.\(2022\)S\. Roburin, Y\. de Mont\-Marin, A\. Bursuc, R\. Marlet, P\. Pérez, and M\. AubrySpherical perspective on learning with normalization layers\.Neurocomputing487,pp\. 66–74\.External Links:[Document](https://dx.doi.org/10.1016/j.neucom.2022.02.021)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1)\.
- Salimans and Kingma \(2016\)T\. Salimans and D\. P\. KingmaWeight normalization: a simple reparameterization to accelerate training of deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1)\.
- Shazeer and Stern \(2018\)N\. Shazeer and M\. SternAdafactor: adaptive learning rates with sublinear memory cost\.InInternational Conference on Machine Learning,Vol\.80,pp\. 4596–4604\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- van Laarhoven \(2017\)T\. van LaarhovenL2 regularization versus batch and weight normalization\.arXiv preprint arXiv:1706\.05350\.External Links:[Link](https://arxiv.org/abs/1706.05350)Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p2.2),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2608.21024#S5.SS2.p2.1),[§6](https://arxiv.org/html/2608.21024#S6.SS0.SSS0.Px2.p1.1)\.
- Vyaset al\.\(2025\)N\. Vyas, D\. Morwani, R\. Zhao, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. M\. KakadeSOAP: improving and stabilizing Shampoo using Adam for language modeling\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.
- Wang and Aitchison \(2025\)X\. Wang and L\. AitchisonHow to set AdamW’s weight decay as you scale model and dataset size\.InInternational Conference on Machine Learning,Vol\.267,pp\. 62222–62250\.Cited by:[§1](https://arxiv.org/html/2608.21024#S1.p3.2),[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-3018),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§5\.5](https://arxiv.org/html/2608.21024#S5.SS5.p1.1)\.
- Wenet al\.\(2026\)K\. Wen, X\. Dang, K\. Lyu, T\. Ma, and P\. LiangFantastic pretraining optimizers and where to find them II: hyperball optimization\.arXiv preprint arXiv:2606\.16899\.External Links:[Link](https://arxiv.org/abs/2606.16899)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)A\. Yanget al\.Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.External Links:[Link](https://arxiv.org/abs/2407.10671)Cited by:[§5\.1](https://arxiv.org/html/2608.21024#S5.SS1.p1.1)\.
- Youet al\.\(2017\)Y\. You, I\. Gitman, and B\. GinsburgLarge batch training of convolutional networks\.External Links:1708\.03888,[Link](https://arxiv.org/abs/1708.03888)Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Youet al\.\(2020\)Y\. You, J\. Li, S\. Reddi, J\. Hseu, S\. Kumar, S\. Bhojanapalli, X\. Song, J\. Demmel, K\. Keutzer, and C\. HsiehLarge batch optimization for deep learning: training BERT in 76 minutes\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px2.p2.1)\.
- Yuanet al\.\(2025\)H\. Yuan, Y\. Liu, S\. Wu, X\. Zhou, and Q\. GuMARS: unleashing the power of variance reduction for training large models\.InInternational Conference on Machine Learning,Vol\.267,pp\. 73553–73587\.Cited by:[§2](https://arxiv.org/html/2608.21024#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AComplete update and fused implementation

### A\.1Per\-matrix update

Algorithm 1:Default RODE update for one matrix parameterInput:parameterWtW\_\{t\}, gradientgtg\_\{t\}, raw bufferBt−1B\_\{t\-1\}, learning ratesηρ,ηdir\\eta\_\{\\rho\},\\eta\_\{\\mathrm\{dir\}\}, momentumβ\\beta, floorκfloor\\kappa\_\{\\mathrm\{floor\}\}, NS stepsKK, and safeguardsϵ,ϵNS\\epsilon,\\epsilon\_\{\\mathrm\{NS\}\}\.1\.If‖Wt‖F\>ϵ\\\|W\_\{t\}\\\|\_\{\\mathrm\{F\}\}\>\\epsilon, formρt=‖Wt‖F\\rho\_\{t\}=\\\|W\_\{t\}\\\|\_\{\\mathrm\{F\}\},Ut=Wt/ρtU\_\{t\}=W\_\{t\}/\\rho\_\{t\}, and𝒫t​\(X\)=X−⟨X,Ut⟩F​Ut\\mathcal\{P\}\_\{t\}\(X\)=X\-\\langle X,U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}U\_\{t\}\.2\.SetB~t=β​𝒫t​\(Bt−1\)\+gt\\widetilde\{B\}\_\{t\}=\\beta\\mathcal\{P\}\_\{t\}\(B\_\{t\-1\}\)\+g\_\{t\}\.3\.Setst=⟨B~t,Ut⟩Fs\_\{t\}=\\langle\\widetilde\{B\}\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}andρt\+1=max⁡\{ρt−ηρ​st,κfloor​ρt\}\\rho\_\{t\+1\}=\\max\\\{\\rho\_\{t\}\-\\eta\_\{\\rho\}s\_\{t\},\\kappa\_\{\\mathrm\{floor\}\}\\rho\_\{t\}\\\}\.4\.SetMt⟂=𝒫t​\(B~t\)=B~t−st​UtM\_\{t\}^\{\\perp\}=\\mathcal\{P\}\_\{t\}\(\\widetilde\{B\}\_\{t\}\)=\\widetilde\{B\}\_\{t\}\-s\_\{t\}U\_\{t\}\.5\.TransposeMt⟂M\_\{t\}^\{\\perp\}andUtU\_\{t\}if needed so rows≥\\geqcolumns; initializeX0=Mt⟂/‖Mt⟂‖FX\_\{0\}=M\_\{t\}^\{\\perp\}/\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}in the nonzero branch\.6\.Fork=0,…,K−1k=0,\\ldots,K\-1, setXk\+1=𝒫t​\[3\.4445​Xk\+Xk​\{−4\.7750​Xk⊤​Xk\+2\.0315​\(Xk⊤​Xk\)2\}\]\.X\_\{k\+1\}=\\mathcal\{P\}\_\{t\}\\\!\\left\[3\.4445X\_\{k\}\+X\_\{k\}\\\{\-4\.7750X\_\{k\}^\{\\top\}X\_\{k\}\+2\.0315\(X\_\{k\}^\{\\top\}X\_\{k\}\)^\{2\}\\\}\\right\]\.Undo any transpose and setQ^t=XK\\widehat\{Q\}\_\{t\}=X\_\{K\}\.7\.Forqt=‖Q^t‖F\>0q\_\{t\}=\\\|\\widehat\{Q\}\_\{t\}\\\|\_\{\\mathrm\{F\}\}\>0, setDt⟂=Q^t/qtD\_\{t\}^\{\\perp\}=\\widehat\{Q\}\_\{t\}/q\_\{t\},θt=ηdir​qt\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}q\_\{t\}, andUt\+1=Utcosθt−Dt⟂sinθtU\_\{t\+1\}=U\_\{t\}\\cos\\theta\_\{t\}\-D\_\{t\}^\{\\perp\}\\sin\\theta\_\{t\}\. RenormalizeUt\+1U\_\{t\+1\}\.8\.WriteWt\+1=ρt\+1​Ut\+1W\_\{t\+1\}=\\rho\_\{t\+1\}U\_\{t\+1\}and storeBt=B~tB\_\{t\}=\\widetilde\{B\}\_\{t\}\.Zero parameter norms and zero projected kernels take safeguarded no\-direction branches\. RODE\-SNR changesθt\\theta\_\{t\}toηdir​γt​qt\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\}\.

### A\.2Shape\-batched implementation

The per\-parameter form of RODE iterates over matrices in Python, issuing many small kernels for the split, projection, Newton–Schulz conditioning, and geodesic step\. Table[7](https://arxiv.org/html/2608.21024#S5.T7)shows0\.59×0\.59\\timesAdamW throughput on the Qwen2\-style\-LM configuration\.

The fused implementation removes the Python loop without changing the per\-matrix computation\. Within a parameter group, parameters are bucketed by shape; theBBmatrices of a common shapem×nm\\times nare stacked into a single tensor𝒲∈ℝB×m×n\\mathcal\{W\}\\in\\mathbb\{R\}^\{B\\times m\\times n\}, and likewise their gradients and momentum buffers\. Each step of the engine then becomes a batched operation that broadcasts over the leading dimension:

- •Polar split\.ρ=‖𝒲‖F\\rho=\\\|\\mathcal\{W\}\\\|\_\{\\mathrm\{F\}\}and𝒰=𝒲/ρ\\mathcal\{U\}=\\mathcal\{W\}/\\rhouse a Frobenius norm reduced over the last two dimensions, givingρ∈ℝB×1×1\\rho\\in\\mathbb\{R\}^\{B\\times 1\\times 1\}\.
- •Tangent momentum\.The projection𝒫𝒰​\(𝒳\)=𝒳−⟨𝒳,𝒰⟩​𝒰\\mathcal\{P\}\_\{\\mathcal\{U\}\}\(\\mathcal\{X\}\)=\\mathcal\{X\}\-\\langle\\mathcal\{X\},\\mathcal\{U\}\\rangle\\,\\mathcal\{U\}uses a batched inner product \(an elementwise product summed over the matrix dimensions, kept asB×1×1B\\times 1\\times 1\)\. The fused code first forms the raw bufferℳ~←β​𝒫𝒰​\(ℳold\)\+𝒢\\widetilde\{\\mathcal\{M\}\}\\leftarrow\\beta\\mathcal\{P\}\_\{\\mathcal\{U\}\}\(\\mathcal\{M\}\_\{\\mathrm\{old\}\}\)\+\\mathcal\{G\}, then removes the radial component for the directional engine:ℳ⟂←𝒫𝒰​\(ℳ~\)=β​𝒫𝒰​\(ℳold\)\+𝒫𝒰​\(𝒢\)\\mathcal\{M\}^\{\\perp\}\\leftarrow\\mathcal\{P\}\_\{\\mathcal\{U\}\}\(\\widetilde\{\\mathcal\{M\}\}\)=\\beta\\mathcal\{P\}\_\{\\mathcal\{U\}\}\(\\mathcal\{M\}\_\{\\mathrm\{old\}\}\)\+\\mathcal\{P\}\_\{\\mathcal\{U\}\}\(\\mathcal\{G\}\)\. The raw bufferℳ~\\widetilde\{\\mathcal\{M\}\}is what is written back; the next step begins by reprojecting it to the then\-current tangent space\.
- •Newton–Schulz\.The quintic iteration is batched matrix multiplication:𝒜=𝒳⊤​𝒳\\mathcal\{A\}=\\mathcal\{X\}^\{\\top\}\\mathcal\{X\}and𝒳←3\.4445​𝒳\+𝒳⁡\(−4\.7750​𝒜\+2\.0315​𝒜2\)\\mathcal\{X\}\\\!\\leftarrow\\\!3\.4445\\,\\mathcal\{X\}\+\\mathcal\{X\}\\,\(\-4\.7750\\,\\mathcal\{A\}\+2\.0315\\,\\mathcal\{A\}^\{2\}\)are all batchedbmms, with the tangent reprojection applied batched after each iteration\. Transposition \(them<nm<ncase\) is uniform within a bucket\.
- •Geodesic step\.cos\\cosandsin\\sinof the per\-matrix angleθ∈ℝB×1×1\\theta\\in\\mathbb\{R\}^\{B\\times 1\\times 1\}are applied elementwise, and the updated matrices are written back in bulk withtorch\.\_foreach\_copy\_\.

Matrices of different shapes form separate buckets\. Loop and batched implementations agree tobf16numerical tolerance \(9×10−49\\times 10^\{\-4\}in a five\-step check\)\. Batching raises Qwen2\-style\-LM throughput from0\.59×0\.59\\timesto0\.72×0\.72\\timesAdamW \(Table[7](https://arxiv.org/html/2608.21024#S5.T7)\), compared with0\.77×0\.77\\timesfor Muon RMS at equal state memory\. RODE performs additional projection and geodesic operations in this comparison\.

## Appendix BOptimizer Definitions and Hyperparameters

This appendix gives the exact optimizer routing, sweep budgets, and selected hyperparameters used in the reported experiments\. We writeη2​d/dir\\eta\_\{\\mathrm\{2d/dir\}\}for the matrix\-parameter learning rate; for RODE\-family optimizers this is the angular learning rateηdir\\eta\_\{\\mathrm\{dir\}\}, while for non\-RODE matrix optimizers it is the ordinary matrix\-parameter learning rate\. We writeηρ\\eta\_\{\\rho\}for RODE’s radial learning rate andη1​d\\eta\_\{\\mathrm\{1d\}\}for the non\-matrix learning rate\. A dash denotes an unused parameter\. Unless stated otherwise, language runs use two\-dimensional non\-embedding, non\-head parameters as the matrix set, with all remaining parameters routed to the listed non\-matrix optimizer\. Vision runs route all parameters withndim \>= 2to the matrix optimizer; for Muon baselines, convolutional tensors are reshaped to matrices before the Muon step\.

#### Hyperparameter selection protocol\.

For each sweep group in Table[9](https://arxiv.org/html/2608.21024#A2.T9), all optimizers in that group use the listed number of Optuna trials, training horizon, data source, model configuration, evaluation protocol, and log\-uniform search spaces\. The Qwen2\-style sweeps evaluate once at the end of the sweep horizon and select the trial with the lowest validation loss\. The standard GPT\-2 sweep selects the trial with the lowest mean training loss over the final 30 steps of its 300\-step protocol\. After selection, the chosen values are fixed for the formal multi\-seed runs\. When a row is marked as transferred in Tables[10](https://arxiv.org/html/2608.21024#A2.T10)–[15](https://arxiv.org/html/2608.21024#A2.T15), the target task is not re\-swept: the hyperparameters are copied from the stated source sweep before running the target\-task experiment\. CIFAR\-100, ImageNet\-1K, and the Qwen2\-style 1\.5B FineWeb experiment are transfer evaluations, not task\-specific tuning studies\. The separate Qwen3\.5\-9B sweep and training protocol is specified below and in Table[13](https://arxiv.org/html/2608.21024#A2.T13)\.

Table 8:Operational baseline definitions used in the code\. Matrix weight decay is the task\-level\-\-wd; non\-matrix AdamW uses weight decay9×10−49\\times 10^\{\-4\}unless the row states otherwise\.Table 9:Learning\-rate sweep budgets and search spaces\. All sweep ranges are log\-uniform\. The selected values from these sweeps are reported in the task\-specific hyperparameter tables below\. The added baselines are swept on the Qwen2\-style LM/WikiText\-103 benchmark and then transferred to standard GPT\-2 and CIFAR\-100\.Table 10:Standard GPT\-2/WikiText\-103 hyperparameters for the 8000\-step main benchmark\. Runs use sequence length 512, effective batch size 128, micro\-batch size 16,bf16autocast, evaluation every 500 steps, 100 warmup steps, and cosine decay\.Table 11:Qwen2\-style LM/WikiText\-103 hyperparameters for the aligned 4000\-step main benchmark and component ablation\. The model has 114M parameters\. All runs use sequence length 512, effective batch size 128, micro\-batch size 16,bf16autocast, the official validation split, evaluation every 200 steps, 200 warmup steps, and cosine decay\. This locally instantiated architecture uses a separate protocol from Table[10](https://arxiv.org/html/2608.21024#A2.T10)\.Table 12:Qwen2\-style 1\.5B/FineWeb transfer hyperparameters\. This three\-seed experiment reuses learning rates selected on the smaller Qwen2\-style LM without a new sweep\. Runs use streaming FineWebsample\-100BT, sequence length 1024, effective batch size 128, micro\-batch size 2 per GPU,bf16parameters/autocast, 20000 optimizer steps, evaluation every 500 steps, checkpointing every 100 steps, 200 warmup steps, cosine decay, andtorch\.compile\(default\)\. The fixed evaluation cache is identical across optimizers\.
#### Qwen3\.5\-9B full\-parameter fine\-tuning\.

All six optimizers start from the same non\-Base Qwen3\.5\-9B checkpoint and use the same materialized GSM8K split: 7099 training examples and 374 validation examples\. Each optimizer receives 20 Optuna trials of 800 steps\. The Adafactor, Muon Original, and SOAP studies enqueue three, four, and four fixed candidate settings, respectively, before continuing with log\-uniform sampling; these candidates count toward the same 20\-trial budget\. Validation is run every 100 steps on all 374 examples, and the trial with the lowest observed validation loss supplies the learning rates for the formal runs\. Formal training uses three distinct seeds for 3000 steps, sequence length 1024, micro\-batch size 1, gradient accumulation 8, BF16 full\-parameter training, gradient checkpointing, gradient clipping at norm 1\.0, and zero weight decay\. The schedule applies 3% linear warmup followed by cosine decay\. Evaluation is run every 100 steps, and the checkpoint with the minimum validation loss is saved for downstream evaluation\.

The parameter routing in this experiment differs from the language\-model default above: every trainable two\-dimensional tensor, including any embedding and output\-head matrices, belongs to the matrix set\. Muon Original, Muon RMS, SOAP, and RODE update this set with their matrix rule and route all remaining tensors to AdamW with betas\(0\.9,0\.95\)\(0\.9,0\.95\)\. AdamW and Adafactor each update all trainable parameters with one optimizer\. Here Adafactor uses an explicit learning rate with relative\-step scaling, parameter scaling, and warmup initialization disabled\. SOAP uses betas\(0\.9,0\.95\)\(0\.9,0\.95\), precondition frequency 100, maximum precondition dimension 1024, and disables dimension merging, one\-dimensional preconditioning, and gradient normalization\.

For downstream evaluation, every run loads its minimum\-validation\-loss checkpoint\. GSM8K \(1319 examples\) and MATH\-500 \(500 examples\) use greedy generation with at most 512 new tokens and normalized exact\-answer matching\. MMLU\-STEM \(3153 examples\) and MMLU\-Pro Math \(1351 examples\) use a fixed zero\-shot multiple\-choice prompt: each option letter is scored by its summed sequence log\-likelihood, and the highest\-scoring option is selected\. Table[5](https://arxiv.org/html/2608.21024#S5.T5)reports accuracy over each complete materialized evaluation set\.

Table 13:Qwen3\.5\-9B learning\-rate searches and selected values\. Every optimizer receives 20 trials of 800 steps; all searched ranges are log\-uniform\. “Same” denotes a single learning rate applied to all trainable parameters\.Table 14:ResNet\-50/CIFAR\-100 hyperparameters\. Runs use 100 epochs, image size 224, effective batch size 128, three distinct seeds, and epoch\-wise cosine annealing\. The implemented scheduler isCosineAnnealingLR\(T\_max=epochs\)with no warmup stage\. We report best test accuracy\. No CIFAR\-specific sweep is used: the original suite uses standard\-GPT\-2\-selected settings, while the added baselines use settings selected on the Qwen2\-style LM\.Table 15:ResNet\-50/ImageNet\-1K hyperparameters\. Runs use 100 epochs, image size 224, effective batch size 256, three distinct seeds, and epoch\-wise cosine annealing without an implemented warmup stage\. Table[1](https://arxiv.org/html/2608.21024#S5.T1)reports, for each seed, the best validation accuracy observed over 100 epochs\. The added SOAP/MARS\-M/AdEMAMix baselines are not run on ImageNet\-1K\.Table 16:Ablation protocols\. Each experiment uses three distinct seeds\.

## Appendix CAdditional Experimental Diagnostics and Curves

This appendix collects supporting diagnostics and full trajectories that complement the main results\. The same plotting code and color mapping are used as in the main figures\.

### C\.1Qwen2\-style LM validation trajectories

Figure 5:Qwen2\-style LM/WikiText\-103 validation trajectories over 4000 steps \(three\-seed mean±\\pmsample SD\): full trajectory \(left\) and late\-stage detail \(right\)\.
### C\.2Metric definitions

Diagnostics include every trainable parameter, including parameters delegated to AdamW\. For the diagnostic parameter set𝒫diag\\mathcal\{P\}\_\{\\mathrm\{diag\}\},

‖Wt‖global\\displaystyle\\\|W\_\{t\}\\\|\_\{\\mathrm\{global\}\}=\(∑p∈𝒫diag‖pt‖F2\)1/2,\\displaystyle=\\left\(\\sum\_\{p\\in\\mathcal\{P\}\_\{\\mathrm\{diag\}\}\}\\\|p\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\)^\{1/2\},\(19\)‖Δ​Wt‖global\\displaystyle\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{global\}\}=\(∑p∈𝒫diag‖pt\+1−pt‖F2\)1/2\.\\displaystyle=\\left\(\\sum\_\{p\\in\\mathcal\{P\}\_\{\\mathrm\{diag\}\}\}\\\|p\_\{t\+1\}\-p\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\)^\{1/2\}\.\(20\)The update ratio divides the latter by‖Wt‖global\+10−12\\\|W\_\{t\}\\\|\_\{\\mathrm\{global\}\}\+10^\{\-12\}\. Gradient norm is the analogous root\-sum\-square immediately before the optimizer step\. Only high\-frequency training, gradient, and update diagnostics are EMA\-smoothed for visualization; validation, accuracy, weight norm, schedules, and tabulated values use raw logs\.

### C\.31\.5B scale\-transfer protocol

The 1\.5B runs use sequence length 1024, effective batch size 128, 200 warmup steps, cosine decay, and learning rates selected on the smaller Qwen2\-style LM\. Fixed\-radius RODE uses the same directional and delegated\-parameter rates as full RODE, setsηρ=0\\eta\_\{\\rho\}=0, and holds each managed matrix’s radius at initialization\. All three methods use the same three distinct seeds for 20000 steps\. The fixed evaluation cache contains 1024 length\-1024 sequences built from FineWebsample\-10BT; document\-level exclusion from thesample\-100BTtraining stream is not enforced\[[15](https://arxiv.org/html/2608.21024#bib.bib29)\]\. Every optimizer is evaluated on the identical cache\. Figure[3](https://arxiv.org/html/2608.21024#S5.F3)reports cache loss and global norm; Figure[6](https://arxiv.org/html/2608.21024#A3.F6)reports dense training diagnostics\. Raw per\-seed scalars and provenance are archived with the paper\.

Figure 6:Qwen2\-style 1\.5B/FineWeb training diagnostics over 20000 steps \(three seeds per method\): EMA\-smoothed training loss \(left\) and gradient norm \(right\)\. Gradient norm uses a logarithmic axis\.
### C\.4Qwen3\.5\-9B tuning and training diagnostics

Figure 7:Qwen3\.5\-9B full\-parameter fine\-tuning diagnostics for the six optimizers in Table[5](https://arxiv.org/html/2608.21024#S5.T5)\. Left: validation loss over the 3000\-step formal runs \(three\-seed mean±\\pmsample SD\)\. Right: cumulative best objective over the 20 800\-step Optuna trials for each optimizer; each trial’s objective is its minimum validation loss\. Both panels use raw, unsmoothed values\.#### GPT\-2/WikiText\-103 dynamics\.

Figure[8](https://arxiv.org/html/2608.21024#A3.F8)shows the training loss, update step size, and gradient norm for the GPT\-2 runs; the GPT\-2 weight norm is in the main text \(Figure[2](https://arxiv.org/html/2608.21024#S5.F2)\)\.

Figure 8:GPT\-2/WikiText\-103 training dynamics \(three seeds\): training loss, update step size, and gradient norm\.
#### ResNet\-50 evaluation loss and weight norm\.

Figures[9](https://arxiv.org/html/2608.21024#A3.F9)and[10](https://arxiv.org/html/2608.21024#A3.F10)show test loss for CIFAR\-100, validation loss for ImageNet\-1K, and observed full\-model weight norm for both datasets\.

Figure 9:ResNet\-50 test loss on CIFAR\-100 \(left\) and validation loss on ImageNet\-1K \(right\), shown as three\-seed mean±\\pmsample SD\.
Figure 10:Observed full\-model weight norms on CIFAR\-100 \(left\) and ImageNet\-1K \(right\), shown as three\-seed mean±\\pmsample SD\. RODE directly updates managed\-matrix radii; these curves additionally include parameters delegated to AdamW\.
Figure 11:Additional ResNet\-50/CIFAR\-100 curves: training accuracy, training loss, gradient norm, and update\-to\-weight ratio\.
Figure 12:Additional ResNet\-50/ImageNet\-1K curves from the exported per\-step diagnostics: three\-seed means with sample\-standard\-deviation bands for training accuracy, training loss, gradient norm, and update\-to\-weight ratio\. Main ImageNet validation results are reported in Table[1](https://arxiv.org/html/2608.21024#S5.T1)and Figure[4](https://arxiv.org/html/2608.21024#S5.F4)\.

## Appendix DDetailed Global Convergence Proof

This appendix expands the convergence\-neighborhood statement in Section[4](https://arxiv.org/html/2608.21024#S4)\. We derive the geodesic displacement, split radial and tangent descent, bound the tangent tracking error, and combine the potential and kinetic inequalities with a Lyapunov function\.

### D\.1Notation and algorithmic equations

Letf⁡\(W\):ℝm×n→ℝf\(W\):\\mathbb\{R\}^\{m\\times n\}\\rightarrow\\mathbb\{R\}be the objective for a matrix parameter\. For clarity assumem≥nm\\geq n; ifm<nm<n, the implementation transposes the matrix before applying Newton–Schulz and transposes the result back\. The Frobenius inner product is

⟨A,B⟩F=tr⁡\(A⊤​B\),‖A‖F=⟨A,A⟩F\.\\langle A,B\\rangle\_\{\\mathrm\{F\}\}=\\mathrm\{tr\}\(A^\{\\top\}B\),\\qquad\\\|A\\\|\_\{\\mathrm\{F\}\}=\\sqrt\{\\langle A,A\\rangle\_\{\\mathrm\{F\}\}\}\.\(21\)At iterationtt, RODE writes

Wt=ρt​Ut,ρt=‖Wt‖F,Ut=Wtρt,‖Ut‖F=1\.W\_\{t\}=\\rho\_\{t\}U\_\{t\},\\qquad\\rho\_\{t\}=\\\|W\_\{t\}\\\|\_\{\\mathrm\{F\}\},\\qquad U\_\{t\}=\\frac\{W\_\{t\}\}\{\\rho\_\{t\}\},\\qquad\\\|U\_\{t\}\\\|\_\{\\mathrm\{F\}\}=1\.\(22\)Letℱt\\mathcal\{F\}\_\{t\}denote the history immediately before sampling the stochastic gradient at steptt, and write𝔼t\[⋅\]=𝔼\[⋅∣ℱt\]\\mathbb\{E\}\_\{t\}\[\\cdot\]=\\mathbb\{E\}\[\\cdot\\mid\\mathcal\{F\}\_\{t\}\]\. LetGt=∇f​\(Wt\)G\_\{t\}=\\nabla f\(W\_\{t\}\)denote the full gradient and letgtg\_\{t\}denote a stochastic gradient satisfying𝔼t​\[gt\]=Gt\\mathbb\{E\}\_\{t\}\[g\_\{t\}\]=G\_\{t\}\. We decompose both into radial and tangent components:

Gtρ\\displaystyle G\_\{t\}^\{\\rho\}=⟨Gt,Ut⟩F,\\displaystyle=\\langle G\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\},Gt⟂\\displaystyle G\_\{t\}^\{\\perp\}=Gt−Gtρ​Ut,\\displaystyle=G\_\{t\}\-G\_\{t\}^\{\\rho\}U\_\{t\},\(23\)gtρ\\displaystyle g\_\{t\}^\{\\rho\}=⟨gt,Ut⟩F,\\displaystyle=\\langle g\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\},gt⟂\\displaystyle g\_\{t\}^\{\\perp\}=gt−gtρ​Ut\.\\displaystyle=g\_\{t\}\-g\_\{t\}^\{\\rho\}U\_\{t\}\.\(24\)ThusWtW\_\{t\},UtU\_\{t\},ρt\\rho\_\{t\}, andGtG\_\{t\}are fixed under𝔼t\\mathbb\{E\}\_\{t\}\. Unsubscripted𝔼\\mathbb\{E\}denotes total expectation; below we pass from conditional one\-step inequalities to total expectations using the tower property\. The radial update is

ρt\+1=ρt−ηρ​gtρ\.\\rho\_\{t\+1\}=\\rho\_\{t\}\-\\eta\_\{\\rho\}g\_\{t\}^\{\\rho\}\.\(25\)The implementation uses the guarded scalar update

ρt\+1=max⁡\{ρt−ηρ​gtρ,κfloor​ρt\}\.\\rho\_\{t\+1\}=\\max\\\{\\rho\_\{t\}\-\\eta\_\{\\rho\}g\_\{t\}^\{\\rho\},\\kappa\_\{\\mathrm\{floor\}\}\\rho\_\{t\}\\\}\.\(26\)The proof below analyzes the interior regime in which this floor is inactive\. If the floor activates, the radial channel becomes a projected one\-dimensional step and the radial descent inequality must be replaced by the corresponding projected\-gradient inequality; we do not claim that branch in this appendix\. The implemented optimizer stores a raw buffer rather than only a tangent buffer:

Bt=β​𝒫t​\(Bt−1\)\+gt,Mraw,t⟂=𝒫t​\(Bt\)\.B\_\{t\}=\\beta\\mathcal\{P\}\_\{t\}\(B\_\{t\-1\}\)\+g\_\{t\},\\qquad M\_\{\\mathrm\{raw\},t\}^\{\\perp\}=\\mathcal\{P\}\_\{t\}\(B\_\{t\}\)\.\(27\)At the current step this gives the same tangent direction as projecting the newly formed buffer\. Across steps, however, the radial part retained inBtB\_\{t\}can re\-enter the next tangent space through𝒫t\+1​\(Bt\)\\mathcal\{P\}\_\{t\+1\}\(B\_\{t\}\)\. Write

Bt=Mraw,t⟂\+at​Ut,at=⟨Bt,Ut⟩F\.B\_\{t\}=M\_\{\\mathrm\{raw\},t\}^\{\\perp\}\+a\_\{t\}U\_\{t\},\\qquad a\_\{t\}=\\langle B\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\.\(28\)Then the next projected raw tangent buffer obeys

Mraw,t\+1⟂\\displaystyle M\_\{\\mathrm\{raw\},t\+1\}^\{\\perp\}=𝒫t\+1​\(Bt\+1\)\\displaystyle=\\mathcal\{P\}\_\{t\+1\}\(B\_\{t\+1\}\)=β​𝒫t\+1​\(Mraw,t⟂\)\+gt\+1⟂\+rt\+1,\\displaystyle=\\beta\\mathcal\{P\}\_\{t\+1\}\(M\_\{\\mathrm\{raw\},t\}^\{\\perp\}\)\+g\_\{t\+1\}^\{\\perp\}\+r\_\{t\+1\},\(29\)with the cross\-step radial leakage residual

rt\+1=β​at​𝒫t\+1​\(Ut\)\.r\_\{t\+1\}=\\beta a\_\{t\}\\mathcal\{P\}\_\{t\+1\}\(U\_\{t\}\)\.\(30\)The implementation stores the unnormalized momentum above\. For the tracking argument we analyze its normalized counterpart

Mt⟂=\(1−β\)​Mraw,t⟂,r¯t\+1=\(1−β\)​rt\+1,M\_\{t\}^\{\\perp\}=\(1\-\\beta\)M\_\{\\mathrm\{raw\},t\}^\{\\perp\},\\qquad\\bar\{r\}\_\{t\+1\}=\(1\-\\beta\)r\_\{t\+1\},\(31\)so that

Mt\+1⟂=β​𝒫t\+1​\(Mt⟂\)\+\(1−β\)​gt\+1⟂\+r¯t\+1\.M\_\{t\+1\}^\{\\perp\}=\\beta\\mathcal\{P\}\_\{t\+1\}\(M\_\{t\}^\{\\perp\}\)\+\(1\-\\beta\)g\_\{t\+1\}^\{\\perp\}\+\\bar\{r\}\_\{t\+1\}\.\(32\)This rescaling does not change the Newton–Schulz direction because the engine starts fromM/‖M‖FM/\\\|M\\\|\_\{\\mathrm\{F\}\}\. The implemented raw buffer therefore induces an additional cross\-step leakage residual; below we bound its squared contribution and absorb it into the curvature\-order residual\. From here on,Mt⟂M\_\{t\}^\{\\perp\}denotes the normalized analysis momentum, and the appendix analyzes the nondegenerate, inactive\-floor raw\-buffer branch rather than the zero\-norm safeguard or active\-floor branches\. The projected Newton–Schulz engine starts from

X0=Mt⟂‖Mt⟂‖FX\_\{0\}=\\frac\{M\_\{t\}^\{\\perp\}\}\{\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\}\(33\)in the nondegenerate branch and, matching the default implementation, applies tangent reprojection after each quintic polynomial step:

Yk\+1\\displaystyle Y\_\{k\+1\}=3\.4445​Xk\+Xk​\(−4\.7750​Xk⊤​Xk\+2\.0315​\(Xk⊤​Xk\)2\),\\displaystyle=3\.4445X\_\{k\}\+X\_\{k\}\\left\(\-4\.7750X\_\{k\}^\{\\top\}X\_\{k\}\+2\.0315\(X\_\{k\}^\{\\top\}X\_\{k\}\)^\{2\}\\right\),\(34\)Xk\+1\\displaystyle X\_\{k\+1\}=Yk\+1−⟨Yk\+1,Ut⟩FUt,k=0,…,K−1\.\\displaystyle=Y\_\{k\+1\}\-\\langle Y\_\{k\+1\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}U\_\{t\},\\qquad k=0,\\ldots,K\-1\.\(35\)AfterKKprojected steps, define

Q^t=XK\.\\widehat\{Q\}\_\{t\}=X\_\{K\}\.\(36\)Define its actual post\-projection norm

qt=‖Q^t‖F\.q\_\{t\}=\\\|\\widehat\{Q\}\_\{t\}\\\|\_\{\\mathrm\{F\}\}\.\(37\)Projection removes the component parallel toUtU\_\{t\}, soqtq\_\{t\}is generally different fromn\\sqrt\{n\}\. The proof assumes thatQ^t\\widehat\{Q\}\_\{t\}retains a constant fraction of the polar\-target alignment withMt⟂M\_\{t\}^\{\\perp\}and has nondegenerate projected norm, so that for somecNS∈\(0,1\]c\_\{\\mathrm\{NS\}\}\\in\(0,1\],

⟨Mt⟂,Q^t⟩F≥cNS​‖Mt⟂‖∗,0<qmin≤qt≤qmax<∞\.\\langle M\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\geq c\_\{\\mathrm\{NS\}\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\*\},\\qquad 0<q\_\{\\min\}\\leq q\_\{t\}\\leq q\_\{\\max\}<\\infty\.\(38\)Define the unit tangent direction

Dt⟂=Q^tqt,⟨Dt⟂,Ut⟩F=0\.D\_\{t\}^\{\\perp\}=\\frac\{\\widehat\{Q\}\_\{t\}\}\{q\_\{t\}\},\\qquad\\langle D\_\{t\}^\{\\perp\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}=0\.\(39\)Default RODE uses no scalar damping, i\.e\.γt=1\\gamma\_\{t\}=1andγmin=1\\gamma\_\{\\min\}=1\. The proof also covers any optional scalar damping satisfying0<γmin≤γt≤10<\\gamma\_\{\\min\}\\leq\\gamma\_\{t\}\\leq 1, including the RODE\-SNR ablation\. The geodesic angle is

θt=ηdir​γt​qt\.\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\}\.\(40\)Finally,

Ut\+1\\displaystyle U\_\{t\+1\}=Utcosθt−Dt⟂sinθt,\\displaystyle=U\_\{t\}\\cos\\theta\_\{t\}\-D\_\{t\}^\{\\perp\}\\sin\\theta\_\{t\},\(41\)Wt\+1\\displaystyle W\_\{t\+1\}=ρt\+1​Ut\+1\.\\displaystyle=\\rho\_\{t\+1\}U\_\{t\+1\}\.\(42\)

### D\.2Assumptions

###### Assumption 1\(Smoothness\)\.

For all matricesX,YX,Y,

f⁡\(Y\)≤f⁡\(X\)\+⟨∇f​\(X\),Y−X⟩F\+L2​‖Y−X‖F2\.f\(Y\)\\leq f\(X\)\+\\langle\\nabla f\(X\),Y\-X\\rangle\_\{\\mathrm\{F\}\}\+\\frac\{L\}\{2\}\\\|Y\-X\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(43\)

###### Assumption 2\(Unbiased noise with bounded variance\)\.

The stochastic gradient is unbiased, and the radial and tangent noises are bounded:

𝔼t​\[gtρ−Gtρ\]\\displaystyle\\mathbb\{E\}\_\{t\}\[g\_\{t\}^\{\\rho\}\-G\_\{t\}^\{\\rho\}\]=0,\\displaystyle=0,𝔼t​\[\(gtρ−Gtρ\)2\]\\displaystyle\\mathbb\{E\}\_\{t\}\[\(g\_\{t\}^\{\\rho\}\-G\_\{t\}^\{\\rho\}\)^\{2\}\]≤σρ2,\\displaystyle\\leq\\sigma\_\{\\rho\}^\{2\},\(44\)𝔼t​\[gt⟂−Gt⟂\]\\displaystyle\\mathbb\{E\}\_\{t\}\[g\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\]=0,\\displaystyle=0,𝔼t​\[‖gt⟂−Gt⟂‖F2\]\\displaystyle\\mathbb\{E\}\_\{t\}\[\\\|g\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]≤σ⟂2\.\\displaystyle\\leq\\sigma\_\{\\perp\}^\{2\}\.\(45\)

###### Assumption 3\(Bounded physical states\)\.

There exist constants such that

0<ρmin≤ρt≤ρmax,\|Gtρ\|≤Gmax,∥Mt⟂∥F≤Mmax,\|at\|≤Amaxfor someMmax,Amax\>0\.0<\\rho\_\{\\min\}\\leq\\rho\_\{t\}\\leq\\rho\_\{\\max\},\\quad\|G\_\{t\}^\{\\rho\}\|\\leq G\_\{\\max\},\\quad\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\\leq M\_\{\\max\},\\quad\|a\_\{t\}\|\\leq A\_\{\\max\}\\quad\\text\{for some \}M\_\{\\max\},A\_\{\\max\}\>0\.\(46\)Hereat=⟨Bt,Ut⟩Fa\_\{t\}=\\langle B\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}is the radial component retained in the implemented raw buffer\.

###### Assumption 4\(Local tangent\-gradient regularity\)\.

Let𝒫t\+1​\(X\)=X−⟨X,Ut\+1⟩F​Ut\+1\\mathcal\{P\}\_\{t\+1\}\(X\)=X\-\\langle X,U\_\{t\+1\}\\rangle\_\{\\mathrm\{F\}\}U\_\{t\+1\}\. For the small geodesic moves considered here,

𝔼⁡\[‖𝒫t\+1​\(Gt⟂\)−Gt\+1⟂‖F2\]≤L2​𝔼​\[‖Δ​Wt‖F2\],\\mathbb\{E\}\\left\[\\\|\\mathcal\{P\}\_\{t\+1\}\(G\_\{t\}^\{\\perp\}\)\-G\_\{t\+1\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\right\]\\leq L^\{2\}\\mathbb\{E\}\[\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\],\(47\)whereΔ​Wt=Wt\+1−Wt\\Delta W\_\{t\}=W\_\{t\+1\}\-W\_\{t\}\.

###### Assumption 5\(Aligned projected Newton–Schulz kernel\)\.

After the finite projected Newton–Schulz iterations, the constant\-step engine returns a tangent kernel satisfying, for somecNS∈\(0,1\]c\_\{\\mathrm\{NS\}\}\\in\(0,1\],

⟨Mt⟂,Q^t⟩F≥cNS​‖Mt⟂‖∗,0<qmin≤‖Q^t‖F≤qmax<∞\.\\langle M\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\geq c\_\{\\mathrm\{NS\}\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\*\},\\qquad 0<q\_\{\\min\}\\leq\\\|\\widehat\{Q\}\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq q\_\{\\max\}<\\infty\.\(48\)This assumption isolates the optimizer dynamics from finite\-iteration polynomial approximation errors, projection\-induced alignment loss, and reduced\-precision effects, and excludes degenerate steps where tangent projection nearly annihilates the conditioned kernel\. The exact polar identity is recovered in the ideal casecNS=1c\_\{\\mathrm\{NS\}\}=1\.

###### Assumption 6\(Bounded geodesic angle\)\.

There existsθmax∈\(0,π/4\]\\theta\_\{\\max\}\\in\(0,\\pi/4\]such that

ηdir≤θmaxqmax\.\\eta\_\{\\mathrm\{dir\}\}\\leq\\frac\{\\theta\_\{\\max\}\}\{q\_\{\\max\}\}\.\(49\)Sinceγt≤1\\gamma\_\{t\}\\leq 1, this givesθt≤θmax\\theta\_\{t\}\\leq\\theta\_\{\\max\}and ensures the global lower bound

sin⁡\(θt\)≥k​θt,k=sin⁡\(θmax\)θmax≥2​2π\.\\sin\(\\theta\_\{t\}\)\\geq k\\theta\_\{t\},\\qquad k=\\frac\{\\sin\(\\theta\_\{\\max\}\)\}\{\\theta\_\{\\max\}\}\\geq\\frac\{2\\sqrt\{2\}\}\{\\pi\}\.\(50\)

### D\.3Raw\-buffer leakage bound

The geodesic update gives

Ut\+1=Utcosθt−Dt⟂sinθt,⟨Ut,Dt⟂⟩F=0\.U\_\{t\+1\}=U\_\{t\}\\cos\\theta\_\{t\}\-D\_\{t\}^\{\\perp\}\\sin\\theta\_\{t\},\\qquad\\langle U\_\{t\},D\_\{t\}^\{\\perp\}\\rangle\_\{\\mathrm\{F\}\}=0\.\(51\)Hence⟨Ut,Ut\+1⟩F=cos⁡θt\\langle U\_\{t\},U\_\{t\+1\}\\rangle\_\{\\mathrm\{F\}\}=\\cos\\theta\_\{t\}and

‖𝒫t\+1​\(Ut\)‖F2=1−⟨Ut,Ut\+1⟩F2=sin2⁡θt\.\\\|\\mathcal\{P\}\_\{t\+1\}\(U\_\{t\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}=1\-\\langle U\_\{t\},U\_\{t\+1\}\\rangle\_\{\\mathrm\{F\}\}^\{2\}=\\sin^\{2\}\\theta\_\{t\}\.\(52\)Usingθt=ηdir​γt​qt\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\},γt≤1\\gamma\_\{t\}\\leq 1, andqt≤qmaxq\_\{t\}\\leq q\_\{\\max\},

‖𝒫t\+1​\(Ut\)‖F≤θt≤ηdir​qmax\.\\\|\\mathcal\{P\}\_\{t\+1\}\(U\_\{t\}\)\\\|\_\{\\mathrm\{F\}\}\\leq\\theta\_\{t\}\\leq\\eta\_\{\\mathrm\{dir\}\}q\_\{\\max\}\.\(53\)Therefore the normalized raw\-buffer leakage residual satisfies

‖r¯t\+1‖F2≤\(1−β\)2​β2​Amax2​qmax2​ηdir2=:Kleak​ηdir2\.\\\|\\bar\{r\}\_\{t\+1\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\leq\(1\-\\beta\)^\{2\}\\beta^\{2\}A\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}=:K\_\{\\mathrm\{leak\}\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(54\)

### D\.4Exact displacement expansion and second\-order bound

The physical displacement is

ΔWt=Wt\+1−Wt=ρt\+1\(Utcosθt−Dt⟂sinθt\)−ρtUt\.\\Delta W\_\{t\}=W\_\{t\+1\}\-W\_\{t\}=\\rho\_\{t\+1\}\(U\_\{t\}\\cos\\theta\_\{t\}\-D\_\{t\}^\{\\perp\}\\sin\\theta\_\{t\}\)\-\\rho\_\{t\}U\_\{t\}\.\(55\)Collecting the orthogonal basis termsUtU\_\{t\}andDt⟂D\_\{t\}^\{\\perp\}gives

ΔWt=\(ρt\+1cosθt−ρt\)Ut−ρt\+1sinθtDt⟂\.\\Delta W\_\{t\}=\(\\rho\_\{t\+1\}\\cos\\theta\_\{t\}\-\\rho\_\{t\}\)U\_\{t\}\-\\rho\_\{t\+1\}\\sin\\theta\_\{t\}D\_\{t\}^\{\\perp\}\.\(56\)Because⟨Ut,Dt⟂⟩F=0\\langle U\_\{t\},D\_\{t\}^\{\\perp\}\\rangle\_\{\\mathrm\{F\}\}=0and both directions have unit Frobenius norm,

‖Δ​Wt‖F2\\displaystyle\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\(ρt\+1cosθt−ρt\)2\+ρt\+12sin2θt\\displaystyle=\(\\rho\_\{t\+1\}\\cos\\theta\_\{t\}\-\\rho\_\{t\}\)^\{2\}\+\\rho\_\{t\+1\}^\{2\}\\sin^\{2\}\\theta\_\{t\}\(57\)=ρt\+12cos2θt−2ρt\+1ρtcosθt\+ρt2\+ρt\+12sin2θt\\displaystyle=\\rho\_\{t\+1\}^\{2\}\\cos^\{2\}\\theta\_\{t\}\-2\\rho\_\{t\+1\}\\rho\_\{t\}\\cos\\theta\_\{t\}\+\\rho\_\{t\}^\{2\}\+\\rho\_\{t\+1\}^\{2\}\\sin^\{2\}\\theta\_\{t\}\(58\)=ρt\+12\+ρt2−2ρt\+1ρtcosθt\\displaystyle=\\rho\_\{t\+1\}^\{2\}\+\\rho\_\{t\}^\{2\}\-2\\rho\_\{t\+1\}\\rho\_\{t\}\\cos\\theta\_\{t\}\(59\)=\(ρt\+1−ρt\)2\+2​ρt\+1​ρt​\(1−cos⁡θt\)\.\\displaystyle=\(\\rho\_\{t\+1\}\-\\rho\_\{t\}\)^\{2\}\+2\\rho\_\{t\+1\}\\rho\_\{t\}\(1\-\\cos\\theta\_\{t\}\)\.\(60\)Using1−cos⁡θ≤θ2/21\-\\cos\\theta\\leq\\theta^\{2\}/2,

‖Δ​Wt‖F2≤\(ρt\+1−ρt\)2\+ρt\+1​ρt​θt2\.\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\leq\(\\rho\_\{t\+1\}\-\\rho\_\{t\}\)^\{2\}\+\\rho\_\{t\+1\}\\rho\_\{t\}\\theta\_\{t\}^\{2\}\.\(61\)Substitutingρt\+1−ρt=−ηρ​gtρ\\rho\_\{t\+1\}\-\\rho\_\{t\}=\-\\eta\_\{\\rho\}g\_\{t\}^\{\\rho\}andθt=ηdir​γt​qt\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\},

‖Δ​Wt‖F2≤ηρ2​\(gtρ\)2\+ρt\+1​ρt​ηdir2​γt2​qt2\.\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\leq\\eta\_\{\\rho\}^\{2\}\(g\_\{t\}^\{\\rho\}\)^\{2\}\+\\rho\_\{t\+1\}\\rho\_\{t\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\\gamma\_\{t\}^\{2\}q\_\{t\}^\{2\}\.\(62\)By Assumptions[3](https://arxiv.org/html/2608.21024#Thmassumption3)–[5](https://arxiv.org/html/2608.21024#Thmassumption5),γt≤1\\gamma\_\{t\}\\leq 1, andqt≤qmaxq\_\{t\}\\leq q\_\{\\max\},

‖Δ​Wt‖F2≤ηρ2​\(gtρ\)2\+ρmax2​qmax2​ηdir2\.\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\leq\\eta\_\{\\rho\}^\{2\}\(g\_\{t\}^\{\\rho\}\)^\{2\}\+\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(63\)Taking conditional expectation and using

𝔼t​\[\(gtρ\)2\]=\(𝔼t​\[gtρ\]\)2\+Vart⁡\(gtρ\)≤\(Gtρ\)2\+σρ2,\\mathbb\{E\}\_\{t\}\[\(g\_\{t\}^\{\\rho\}\)^\{2\}\]=\(\\mathbb\{E\}\_\{t\}\[g\_\{t\}^\{\\rho\}\]\)^\{2\}\+\\operatorname\{Var\}\_\{t\}\(g\_\{t\}^\{\\rho\}\)\\leq\(G\_\{t\}^\{\\rho\}\)^\{2\}\+\\sigma\_\{\\rho\}^\{2\},\(64\)we obtain the second\-order smoothness contribution

L2​𝔼t​\[‖Δ​Wt‖F2\]≤L2​ηρ2​\(Gtρ\)2\+L2​ηρ2​σρ2\+L​ρmax2​qmax22​ηdir2\.\\frac\{L\}\{2\}\\mathbb\{E\}\_\{t\}\[\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\\leq\\frac\{L\}\{2\}\\eta\_\{\\rho\}^\{2\}\(G\_\{t\}^\{\\rho\}\)^\{2\}\+\\frac\{L\}\{2\}\\eta\_\{\\rho\}^\{2\}\\sigma\_\{\\rho\}^\{2\}\+\\frac\{L\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\}\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(65\)

### D\.5First\-order descent: radial part

The first\-order term in smoothness is

⟨Gt,Δ​Wt⟩F\.\\langle G\_\{t\},\\Delta W\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\.\(66\)UsingGt=Gtρ​Ut\+Gt⟂G\_\{t\}=G\_\{t\}^\{\\rho\}U\_\{t\}\+G\_\{t\}^\{\\perp\}and the displacement expansion,

⟨Gt,ΔWt⟩F=Gtρ\(ρt\+1cosθt−ρt\)−ρt\+1sinθt⟨Gt⟂,Dt⟂⟩F\.\\langle G\_\{t\},\\Delta W\_\{t\}\\rangle\_\{\\mathrm\{F\}\}=G\_\{t\}^\{\\rho\}\(\\rho\_\{t\+1\}\\cos\\theta\_\{t\}\-\\rho\_\{t\}\)\-\\rho\_\{t\+1\}\\sin\\theta\_\{t\}\\langle G\_\{t\}^\{\\perp\},D\_\{t\}^\{\\perp\}\\rangle\_\{\\mathrm\{F\}\}\.\(67\)For the radial component,

Iρ\\displaystyle I\_\{\\rho\}=Gtρ\(ρt\+1cosθt−ρt\)\\displaystyle=G\_\{t\}^\{\\rho\}\(\\rho\_\{t\+1\}\\cos\\theta\_\{t\}\-\\rho\_\{t\}\)\(68\)=Gtρ​\(ρt\+1−ρt\)\+Gtρ​ρt\+1​\(cos⁡θt−1\)\\displaystyle=G\_\{t\}^\{\\rho\}\(\\rho\_\{t\+1\}\-\\rho\_\{t\}\)\+G\_\{t\}^\{\\rho\}\\rho\_\{t\+1\}\(\\cos\\theta\_\{t\}\-1\)\(69\)=−ηρ​Gtρ​gtρ−ρt\+1​Gtρ​\(1−cos⁡θt\)\.\\displaystyle=\-\\eta\_\{\\rho\}G\_\{t\}^\{\\rho\}g\_\{t\}^\{\\rho\}\-\\rho\_\{t\+1\}G\_\{t\}^\{\\rho\}\(1\-\\cos\\theta\_\{t\}\)\.\(70\)Taking conditional expectation, the first term becomes

𝔼t​\[−ηρ​Gtρ​gtρ\]=−ηρ​Gtρ​𝔼t​\[gtρ\]=−ηρ​\(Gtρ\)2\.\\mathbb\{E\}\_\{t\}\[\-\\eta\_\{\\rho\}G\_\{t\}^\{\\rho\}g\_\{t\}^\{\\rho\}\]=\-\\eta\_\{\\rho\}G\_\{t\}^\{\\rho\}\\mathbb\{E\}\_\{t\}\[g\_\{t\}^\{\\rho\}\]=\-\\eta\_\{\\rho\}\(G\_\{t\}^\{\\rho\}\)^\{2\}\.\(71\)For the curvature contamination term,

−ρt\+1​Gtρ​\(1−cos⁡θt\)\\displaystyle\-\\rho\_\{t\+1\}G\_\{t\}^\{\\rho\}\(1\-\\cos\\theta\_\{t\}\)≤\|ρt\+1​Gtρ\|\(1−cos⁡θt\)\\displaystyle\\leq\|\\rho\_\{t\+1\}G\_\{t\}^\{\\rho\}\|\(1\-\\cos\\theta\_\{t\}\)\(72\)≤ρmax​Gmax​θt22\\displaystyle\\leq\\rho\_\{\\max\}G\_\{\\max\}\\frac\{\\theta\_\{t\}^\{2\}\}\{2\}\(73\)≤Gmax​ρmax​qmax22​ηdir2\.\\displaystyle\\leq\\frac\{G\_\{\\max\}\\rho\_\{\\max\}q\_\{\\max\}^\{2\}\}\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(74\)The last line usesθt=ηdir​γt​qt\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\},γt≤1\\gamma\_\{t\}\\leq 1, and the projected\-kernel boundqt≤qmaxq\_\{t\}\\leq q\_\{\\max\}\. Therefore

𝔼t​\[Iρ\]≤−ηρ​\(Gtρ\)2\+Gmax​ρmax​qmax22​ηdir2\.\\mathbb\{E\}\_\{t\}\[I\_\{\\rho\}\]\\leq\-\\eta\_\{\\rho\}\(G\_\{t\}^\{\\rho\}\)^\{2\}\+\\frac\{G\_\{\\max\}\\rho\_\{\\max\}q\_\{\\max\}^\{2\}\}\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(75\)

### D\.6First\-order descent: tangent part

The tangent component in \([67](https://arxiv.org/html/2608.21024#A4.E67)\) is

I⟂=−ρt\+1sinθt⟨Gt⟂,Dt⟂⟩F\.I\_\{\\perp\}=\-\\rho\_\{t\+1\}\\sin\\theta\_\{t\}\\langle G\_\{t\}^\{\\perp\},D\_\{t\}^\{\\perp\}\\rangle\_\{\\mathrm\{F\}\}\.\(76\)Insert the identityGt⟂=Mt⟂−\(Mt⟂−Gt⟂\)G\_\{t\}^\{\\perp\}=M\_\{t\}^\{\\perp\}\-\(M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\)and useDt⟂=Q^t/qtD\_\{t\}^\{\\perp\}=\\widehat\{Q\}\_\{t\}/q\_\{t\}:

I⟂\\displaystyle I\_\{\\perp\}=−ρt\+1​sin⁡θtqt​⟨Mt⟂,Q^t⟩F\+ρt\+1​sin⁡θtqt​⟨Mt⟂−Gt⟂,Q^t⟩F\\displaystyle=\-\\rho\_\{t\+1\}\\frac\{\\sin\\theta\_\{t\}\}\{q\_\{t\}\}\\langle M\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\+\\rho\_\{t\+1\}\\frac\{\\sin\\theta\_\{t\}\}\{q\_\{t\}\}\\langle M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\(77\)≤−ρt\+1​sin⁡θtqt​cNS​‖Mt⟂‖∗\+ρt\+1​sin⁡θtqt​⟨Mt⟂−Gt⟂,Q^t⟩F,\\displaystyle\\leq\-\\rho\_\{t\+1\}\\frac\{\\sin\\theta\_\{t\}\}\{q\_\{t\}\}c\_\{\\mathrm\{NS\}\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\*\}\+\\rho\_\{t\+1\}\\frac\{\\sin\\theta\_\{t\}\}\{q\_\{t\}\}\\langle M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\},\(78\)where the second line uses Assumption[5](https://arxiv.org/html/2608.21024#Thmassumption5); the negative coefficient turns the alignment lower bound into an upper bound onI⟂I\_\{\\perp\}\. By Assumption[6](https://arxiv.org/html/2608.21024#Thmassumption6),sin⁡θt≥k​θt\\sin\\theta\_\{t\}\\geq k\\theta\_\{t\}for the negative term\. The remaining inner product can have either sign, so we first upper\-bound it by its absolute value and then usesin⁡θt≤θt\\sin\\theta\_\{t\}\\leq\\theta\_\{t\}\. Sinceθt=ηdir​γt​qt\\theta\_\{t\}=\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}q\_\{t\}, the actual post\-projection norm cancels:

I⟂≤−k​ρt\+1​ηdir​cNS​γt​‖Mt⟂‖∗\+ρt\+1​ηdir​γt​\|⟨Mt⟂−Gt⟂,Q^t⟩F\|\.I\_\{\\perp\}\\leq\-k\\rho\_\{t\+1\}\\eta\_\{\\mathrm\{dir\}\}c\_\{\\mathrm\{NS\}\}\\gamma\_\{t\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\*\}\+\\rho\_\{t\+1\}\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}\\left\|\\langle M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\right\|\.\(79\)We next lower bound the possibly damped and finite\-NS\-aligned nuclear norm\. Sinceγt≥γmin\\gamma\_\{t\}\\geq\\gamma\_\{\\min\},‖M‖∗≥‖M‖F\\\|M\\\|\_\{\*\}\\geq\\\|M\\\|\_\{\\mathrm\{F\}\}, and‖Mt⟂‖F≤Mmax\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\\leq M\_\{\\max\},

cNS​γt​‖Mt⟂‖∗≥cNS​γmin​‖Mt⟂‖F≥cNS​γminMmax​‖Mt⟂‖F2\.c\_\{\\mathrm\{NS\}\}\\gamma\_\{t\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\*\}\\geq c\_\{\\mathrm\{NS\}\}\\gamma\_\{\\min\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\\geq\\frac\{c\_\{\\mathrm\{NS\}\}\\gamma\_\{\\min\}\}\{M\_\{\\max\}\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(80\)Usingρt\+1≥ρmin\\rho\_\{t\+1\}\\geq\\rho\_\{\\min\}and defining

αdir=k​cNS​ρmin​γminMmax,\\alpha\_\{\\mathrm\{dir\}\}=\\frac\{kc\_\{\\mathrm\{NS\}\}\\rho\_\{\\min\}\\gamma\_\{\\min\}\}\{M\_\{\\max\}\},\(81\)the main tangent term is bounded by

−k​ρt\+1​ηdir​cNS​γt​‖Mt⟂‖∗≤−αdir​ηdir​‖Mt⟂‖F2\.\-k\\rho\_\{t\+1\}\\eta\_\{\\mathrm\{dir\}\}c\_\{\\mathrm\{NS\}\}\\gamma\_\{t\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\*\}\\leq\-\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(82\)For the tracking\-error term, Cauchy\-Schwarz gives

ρt\+1​ηdir​γt​\|⟨Mt⟂−Gt⟂,Q^t⟩F\|\\displaystyle\\rho\_\{t\+1\}\\eta\_\{\\mathrm\{dir\}\}\\gamma\_\{t\}\\left\|\\langle M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\},\\widehat\{Q\}\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\right\|≤ρmax​ηdir​‖Mt⟂−Gt⟂‖F​‖Q^t‖F\\displaystyle\\leq\\rho\_\{\\max\}\\eta\_\{\\mathrm\{dir\}\}\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\\\|\\widehat\{Q\}\_\{t\}\\\|\_\{\\mathrm\{F\}\}\(83\)≤ρmax​qmax​ηdir​‖Mt⟂−Gt⟂‖F\.\\displaystyle\\leq\\rho\_\{\\max\}q\_\{\\max\}\\eta\_\{\\mathrm\{dir\}\}\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}\.\(84\)Applying Young’s inequalitya​b≤a2/2\+b2/2ab\\leq a^\{2\}/2\+b^\{2\}/2witha=ρmax​qmax​ηdira=\\rho\_\{\\max\}q\_\{\\max\}\\eta\_\{\\mathrm\{dir\}\}andb=‖Mt⟂−Gt⟂‖Fb=\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\},

≤12​ρmax2​qmax2​ηdir2\+12​‖Mt⟂−Gt⟂‖F2\.\\leq\\frac\{1\}\{2\}\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+\\frac\{1\}\{2\}\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(85\)Taking conditional expectation,

𝔼t​\[I⟂\]≤−αdir​ηdir​𝔼t​\[‖Mt⟂‖F2\]\+12​𝔼t​\[‖Mt⟂−Gt⟂‖F2\]\+12​ρmax2​qmax2​ηdir2\.\\mathbb\{E\}\_\{t\}\[I\_\{\\perp\}\]\\leq\-\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}\\mathbb\{E\}\_\{t\}\[\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+\\frac\{1\}\{2\}\\mathbb\{E\}\_\{t\}\[\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+\\frac\{1\}\{2\}\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(86\)

### D\.7Potential\-energy inequality

By smoothness,

𝔼t​\[f⁡\(Wt\+1\)\]−f⁡\(Wt\)≤𝔼t​\[⟨Gt,Δ​Wt⟩F\]\+L2​𝔼t​\[‖Δ​Wt‖F2\]\.\\mathbb\{E\}\_\{t\}\[f\(W\_\{t\+1\}\)\]\-f\(W\_\{t\}\)\\leq\\mathbb\{E\}\_\{t\}\[\\langle G\_\{t\},\\Delta W\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\]\+\\frac\{L\}\{2\}\\mathbb\{E\}\_\{t\}\[\\\|\\Delta W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\.\(87\)Substituting \([65](https://arxiv.org/html/2608.21024#A4.E65)\), \([75](https://arxiv.org/html/2608.21024#A4.E75)\), and \([86](https://arxiv.org/html/2608.21024#A4.E86)\), then taking total expectation and using the tower property, we obtain

𝔼⁡\[f⁡\(Wt\+1\)\]−𝔼⁡\[f⁡\(Wt\)\]≤\\displaystyle\\mathbb\{E\}\[f\(W\_\{t\+1\}\)\]\-\\mathbb\{E\}\[f\(W\_\{t\}\)\]\\leq−\(ηρ−L2​ηρ2\)​𝔼​\[\(Gtρ\)2\]−αdir​ηdir​𝔼​\[‖Mt⟂‖F2\]\\displaystyle\-\\left\(\\eta\_\{\\rho\}\-\\frac\{L\}\{2\}\\eta\_\{\\rho\}^\{2\}\\right\)\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]\-\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+12​𝔼​\[‖Mt⟂−Gt⟂‖F2\]\+C1​ηdir2\+L2​σρ2​ηρ2,\\displaystyle\+\\frac\{1\}\{2\}\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+C\_\{1\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+\\frac\{L\}\{2\}\\sigma\_\{\\rho\}^\{2\}\\eta\_\{\\rho\}^\{2\},\(88\)where

C1=12​ρmax2​qmax2\+L2​ρmax2​qmax2\+12​Gmax​ρmax​qmax2\.C\_\{1\}=\\frac\{1\}\{2\}\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\+\\frac\{L\}\{2\}\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\+\\frac\{1\}\{2\}G\_\{\\max\}\\rho\_\{\\max\}q\_\{\\max\}^\{2\}\.\(89\)Ifηρ<2/L\\eta\_\{\\rho\}<2/L, thenηρ−L2​ηρ2\>0\\eta\_\{\\rho\}\-\\frac\{L\}\{2\}\\eta\_\{\\rho\}^\{2\}\>0\.

### D\.8Tangent tracking\-error dynamics

Define the tangent tracking error

Et⟂=𝔼⁡\[‖Mt⟂−Gt⟂‖F2\]\.E\_\{t\}^\{\\perp\}=\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\.\(90\)HereMt⟂M\_\{t\}^\{\\perp\}is the normalized analysis momentum defined in Appendix[D](https://arxiv.org/html/2608.21024#A4); the implemented projected raw tangent bufferMraw,t⟂M\_\{\\mathrm\{raw\},t\}^\{\\perp\}differs from it only by the positive scalar\(1−β\)−1\(1\-\\beta\)^\{\-1\}before Newton–Schulz normalization\. For the actual raw\-buffer recurrence, the normalized analysis momentum obeys

Mt\+1⟂=β​𝒫t\+1​\(Mt⟂\)\+\(1−β\)​gt\+1⟂\+r¯t\+1\.M\_\{t\+1\}^\{\\perp\}=\\beta\\mathcal\{P\}\_\{t\+1\}\(M\_\{t\}^\{\\perp\}\)\+\(1\-\\beta\)g\_\{t\+1\}^\{\\perp\}\+\\bar\{r\}\_\{t\+1\}\.\(91\)SubtractGt\+1⟂G\_\{t\+1\}^\{\\perp\}and add/subtract𝒫t\+1​\(Gt⟂\)\\mathcal\{P\}\_\{t\+1\}\(G\_\{t\}^\{\\perp\}\):

Mt\+1⟂−Gt\+1⟂\\displaystyle M\_\{t\+1\}^\{\\perp\}\-G\_\{t\+1\}^\{\\perp\}=β​𝒫t\+1​\(Mt⟂−Gt⟂\)\+β⁡\(𝒫t\+1​\(Gt⟂\)−Gt\+1⟂\)\\displaystyle=\\beta\\mathcal\{P\}\_\{t\+1\}\(M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\)\+\\beta\(\\mathcal\{P\}\_\{t\+1\}\(G\_\{t\}^\{\\perp\}\)\-G\_\{t\+1\}^\{\\perp\}\)\+\(1−β\)​\(gt\+1⟂−Gt\+1⟂\)\+r¯t\+1\.\\displaystyle\\quad\+\(1\-\\beta\)\(g\_\{t\+1\}^\{\\perp\}\-G\_\{t\+1\}^\{\\perp\}\)\+\\bar\{r\}\_\{t\+1\}\.\(92\)Let

xt\\displaystyle x\_\{t\}=β​𝒫t\+1​\(Mt⟂−Gt⟂\),\\displaystyle=\\beta\\mathcal\{P\}\_\{t\+1\}\(M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\),yt\\displaystyle y\_\{t\}=β⁡\(𝒫t\+1​\(Gt⟂\)−Gt\+1⟂\),\\displaystyle=\\beta\(\\mathcal\{P\}\_\{t\+1\}\(G\_\{t\}^\{\\perp\}\)\-G\_\{t\+1\}^\{\\perp\}\),\(93\)zt\\displaystyle z\_\{t\}=\(1−β\)​\(gt\+1⟂−Gt\+1⟂\),\\displaystyle=\(1\-\\beta\)\(g\_\{t\+1\}^\{\\perp\}\-G\_\{t\+1\}^\{\\perp\}\),wt\\displaystyle w\_\{t\}=r¯t\+1\.\\displaystyle=\\bar\{r\}\_\{t\+1\}\.\(94\)Using

‖x\+y\+z\+w‖2≤\(1\+δ\)​‖x‖2\+3​\(1\+1δ\)​\(‖y‖2\+‖z‖2\+‖w‖2\),\\\|x\+y\+z\+w\\\|^\{2\}\\leq\(1\+\\delta\)\\\|x\\\|^\{2\}\+3\\left\(1\+\\frac\{1\}\{\\delta\}\\right\)\(\\\|y\\\|^\{2\}\+\\\|z\\\|^\{2\}\+\\\|w\\\|^\{2\}\),\(95\)we obtain

Et\+1⟂\\displaystyle E\_\{t\+1\}^\{\\perp\}≤β2​\(1\+δ\)​𝔼​\[‖𝒫t\+1​\(Mt⟂−Gt⟂\)‖F2\]\\displaystyle\\leq\\beta^\{2\}\(1\+\\delta\)\\mathbb\{E\}\[\\\|\\mathcal\{P\}\_\{t\+1\}\(M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+3​β2​\(1\+1δ\)​𝔼​\[‖𝒫t\+1​\(Gt⟂\)−Gt\+1⟂‖F2\]\\displaystyle\\quad\+3\\beta^\{2\}\\left\(1\+\\frac\{1\}\{\\delta\}\\right\)\\mathbb\{E\}\[\\\|\\mathcal\{P\}\_\{t\+1\}\(G\_\{t\}^\{\\perp\}\)\-G\_\{t\+1\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+3​\(1−β\)2​\(1\+1δ\)​σ⟂2\\displaystyle\\quad\+3\(1\-\\beta\)^\{2\}\\left\(1\+\\frac\{1\}\{\\delta\}\\right\)\\sigma\_\{\\perp\}^\{2\}\+3​\(1\+1δ\)​𝔼​\[‖r¯t\+1‖F2\]\.\\displaystyle\\quad\+3\\left\(1\+\\frac\{1\}\{\\delta\}\\right\)\\mathbb\{E\}\[\\\|\\bar\{r\}\_\{t\+1\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\.\(96\)Orthogonal projection is non\-expansive:

‖𝒫t\+1​\(Mt⟂−Gt⟂\)‖F2≤‖Mt⟂−Gt⟂‖F2\.\\\|\\mathcal\{P\}\_\{t\+1\}\(M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}\\leq\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(97\)Chooseδ=\(1−β\)/\(2​β\)\\delta=\(1\-\\beta\)/\(2\\beta\)\. Then

β2​\(1\+δ\)=β2​\(1\+1−β2​β\)=β​1\+β2<β,\\beta^\{2\}\(1\+\\delta\)=\\beta^\{2\}\\left\(1\+\\frac\{1\-\\beta\}\{2\\beta\}\\right\)=\\beta\\frac\{1\+\\beta\}\{2\}<\\beta,\(98\)and

3​\(1\+1δ\)=3​\(1\+2​β1−β\)=3​1\+β1−β≤61−β\.3\\left\(1\+\\frac\{1\}\{\\delta\}\\right\)=3\\left\(1\+\\frac\{2\\beta\}\{1\-\\beta\}\\right\)=3\\frac\{1\+\\beta\}\{1\-\\beta\}\\leq\\frac\{6\}\{1\-\\beta\}\.\(99\)By Assumption[4](https://arxiv.org/html/2608.21024#Thmassumption4), the displacement bound, and \([54](https://arxiv.org/html/2608.21024#A4.E54)\),

Et\+1⟂≤β​Et⟂\+6​β2​L21−β​𝔼​\[ηρ2​\(gtρ\)2\+ρmax2​qmax2​ηdir2\]\+6​\(1−β\)​σ⟂2\+6​Kleak1−β​ηdir2\.\\displaystyle E\_\{t\+1\}^\{\\perp\}\\leq\\beta E\_\{t\}^\{\\perp\}\+\\frac\{6\\beta^\{2\}L^\{2\}\}\{1\-\\beta\}\\mathbb\{E\}\[\\eta\_\{\\rho\}^\{2\}\(g\_\{t\}^\{\\rho\}\)^\{2\}\+\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\]\+6\(1\-\\beta\)\\sigma\_\{\\perp\}^\{2\}\+\\frac\{6K\_\{\\mathrm\{leak\}\}\}\{1\-\\beta\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\.\(100\)Using𝔼⁡\[\(gtρ\)2\]≤𝔼⁡\[\(Gtρ\)2\]\+σρ2\\mathbb\{E\}\[\(g\_\{t\}^\{\\rho\}\)^\{2\}\]\\leq\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]\+\\sigma\_\{\\rho\}^\{2\},

Et\+1⟂−Et⟂≤\\displaystyle E\_\{t\+1\}^\{\\perp\}\-E\_\{t\}^\{\\perp\}\\leq−\(1−β\)​Et⟂\+6​β2​L21−β​ηρ2​𝔼​\[\(Gtρ\)2\]\+C2​ηdir2\\displaystyle\-\(1\-\\beta\)E\_\{t\}^\{\\perp\}\+\\frac\{6\\beta^\{2\}L^\{2\}\}\{1\-\\beta\}\\eta\_\{\\rho\}^\{2\}\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]\+C\_\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+6​\(1−β\)​σ⟂2\+6​β2​L21−β​σρ2​ηρ2,\\displaystyle\+6\(1\-\\beta\)\\sigma\_\{\\perp\}^\{2\}\+\\frac\{6\\beta^\{2\}L^\{2\}\}\{1\-\\beta\}\\sigma\_\{\\rho\}^\{2\}\\eta\_\{\\rho\}^\{2\},\(101\)where

C2=6​β2​L2​ρmax2​qmax21−β\+6​Kleak1−β\.C\_\{2\}=\\frac\{6\\beta^\{2\}L^\{2\}\\rho\_\{\\max\}^\{2\}q\_\{\\max\}^\{2\}\}\{1\-\\beta\}\+\\frac\{6K\_\{\\mathrm\{leak\}\}\}\{1\-\\beta\}\.\(102\)

### D\.9Lyapunov synthesis and convergence neighborhood

Define the Lyapunov function

Φt=𝔼⁡\[f⁡\(Wt\)\]\+c⟂​Et⟂\.\\Phi\_\{t\}=\\mathbb\{E\}\[f\(W\_\{t\}\)\]\+c\_\{\\perp\}E\_\{t\}^\{\\perp\}\.\(103\)Choose

c⟂=11−β,c\_\{\\perp\}=\\frac\{1\}\{1\-\\beta\},\(104\)so that theEt⟂E\_\{t\}^\{\\perp\}coefficient after combining \([88](https://arxiv.org/html/2608.21024#A4.E88)\) andc⟂c\_\{\\perp\}times \([101](https://arxiv.org/html/2608.21024#A4.E101)\) becomes

12−c⟂​\(1−β\)=−12\.\\frac\{1\}\{2\}\-c\_\{\\perp\}\(1\-\\beta\)=\-\\frac\{1\}\{2\}\.\(105\)The radial coefficient becomes

−\(ηρ−L2​ηρ2\)\+c⟂​6​β2​L21−β​ηρ2\\displaystyle\-\\left\(\\eta\_\{\\rho\}\-\\frac\{L\}\{2\}\\eta\_\{\\rho\}^\{2\}\\right\)\+c\_\{\\perp\}\\frac\{6\\beta^\{2\}L^\{2\}\}\{1\-\\beta\}\\eta\_\{\\rho\}^\{2\}=−ηρ\+L2​ηρ2\+6​β2​L2\(1−β\)2​ηρ2\\displaystyle=\-\\eta\_\{\\rho\}\+\\frac\{L\}\{2\}\\eta\_\{\\rho\}^\{2\}\+\\frac\{6\\beta^\{2\}L^\{2\}\}\{\(1\-\\beta\)^\{2\}\}\\eta\_\{\\rho\}^\{2\}\(106\)=−ηρ​\(1−L2​ηρ−6​β2​L2\(1−β\)2​ηρ\)\.\\displaystyle=\-\\eta\_\{\\rho\}\\left\(1\-\\frac\{L\}\{2\}\\eta\_\{\\rho\}\-\\frac\{6\\beta^\{2\}L^\{2\}\}\{\(1\-\\beta\)^\{2\}\}\\eta\_\{\\rho\}\\right\)\.\(107\)For sufficiently smallηρ\\eta\_\{\\rho\}, the parenthesized term is positive\. Denote the resulting positive constant byα^ρ\>0\\widehat\{\\alpha\}\_\{\\rho\}\>0\. Combining all constant terms, including the raw\-buffer leakage contributionc⟂​C2​ηdir2c\_\{\\perp\}C\_\{2\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}, into𝒦curv\\mathcal\{K\}\_\{\\mathrm\{curv\}\},𝒦n⟂\\mathcal\{K\}\_\{n\}^\{\\perp\}, and𝒦nρ\\mathcal\{K\}\_\{n\}^\{\\rho\}gives

Φt\+1−Φt≤\\displaystyle\\Phi\_\{t\+1\}\-\\Phi\_\{t\}\\leq−α^ρ​ηρ​𝔼​\[\(Gtρ\)2\]−αdir​ηdir​𝔼​\[‖Mt⟂‖F2\]−12​Et⟂\\displaystyle\-\\widehat\{\\alpha\}\_\{\\rho\}\\eta\_\{\\rho\}\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]\-\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\-\\frac\{1\}\{2\}E\_\{t\}^\{\\perp\}\+𝒦curv​ηdir2\+𝒦n⟂​σ⟂2\+𝒦nρ​σρ2​ηρ2\.\\displaystyle\+\\mathcal\{K\}\_\{\\mathrm\{curv\}\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+\\mathcal\{K\}\_\{n\}^\{\\perp\}\\sigma\_\{\\perp\}^\{2\}\+\\mathcal\{K\}\_\{n\}^\{\\rho\}\\sigma\_\{\\rho\}^\{2\}\\eta\_\{\\rho\}^\{2\}\.\(108\)Summing fromt=1t=1toTT,

ΦT\+1−Φ1≤∑t=1T\[−α^ρ​ηρ​𝔼​\[\(Gtρ\)2\]−αdir​ηdir​𝔼​\[‖Mt⟂‖F2\]−12​Et⟂\+𝒦curv​ηdir2\+𝒦n⟂​σ⟂2\+𝒦nρ​σρ2​ηρ2\]\.\\displaystyle\\Phi\_\{T\+1\}\-\\Phi\_\{1\}\\leq\\sum\_\{t=1\}^\{T\}\\left\[\-\\widehat\{\\alpha\}\_\{\\rho\}\\eta\_\{\\rho\}\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]\-\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\-\\frac\{1\}\{2\}E\_\{t\}^\{\\perp\}\+\\mathcal\{K\}\_\{\\mathrm\{curv\}\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+\\mathcal\{K\}\_\{n\}^\{\\perp\}\\sigma\_\{\\perp\}^\{2\}\+\\mathcal\{K\}\_\{n\}^\{\\rho\}\\sigma\_\{\\rho\}^\{2\}\\eta\_\{\\rho\}^\{2\}\\right\]\.\(109\)Sinceffis lower bounded andEt⟂≥0E\_\{t\}^\{\\perp\}\\geq 0, there is a finiteΦinf\\Phi\_\{\\inf\}withΦt≥Φinf\\Phi\_\{t\}\\geq\\Phi\_\{\\inf\}\. Moving the nonnegative descent terms to the left and usingΦT\+1≥Φinf\\Phi\_\{T\+1\}\\geq\\Phi\_\{\\inf\}gives the finite\-time bound

1T​∑t=1T\[α^ρ​ηρ​𝔼​\[\(Gtρ\)2\]\+αdir​ηdir​𝔼​\[‖Mt⟂‖F2\]\+12​Et⟂\]≤𝒦curv​ηdir2\+𝒦n⟂​σ⟂2\+𝒦nρ​σρ2​ηρ2\+Φ1−ΦinfT\.\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\left\[\\widehat\{\\alpha\}\_\{\\rho\}\\eta\_\{\\rho\}\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]\+\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\+\\frac\{1\}\{2\}E\_\{t\}^\{\\perp\}\\right\]\\leq\\mathcal\{K\}\_\{\\mathrm\{curv\}\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+\\mathcal\{K\}\_\{n\}^\{\\perp\}\\sigma\_\{\\perp\}^\{2\}\+\\mathcal\{K\}\_\{n\}^\{\\rho\}\\sigma\_\{\\rho\}^\{2\}\\eta\_\{\\rho\}^\{2\}\+\\frac\{\\Phi\_\{1\}\-\\Phi\_\{\\inf\}\}\{T\}\.\(110\)Takinglim supT→∞\\limsup\_\{T\\to\\infty\}removes the last term\.

###### Theorem 1\(Interior\-regime decoupled convergence neighborhood\)\.

Under Assumptions[1](https://arxiv.org/html/2608.21024#Thmassumption1)–[6](https://arxiv.org/html/2608.21024#Thmassumption6), sufficiently smallηρ\\eta\_\{\\rho\}, inactive radial floors, and the nondegenerate raw\-buffer recurrence above, the time\-averaged radial gradient, normalized tangent momentum, and tangent tracking error admit the following asymptotic upper bounds:

lim supT→∞1T​∑t=1T𝔼⁡\[\(Gtρ\)2\]\\displaystyle\\limsup\_\{T\\rightarrow\\infty\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\[\(G\_\{t\}^\{\\rho\}\)^\{2\}\]≤𝒦curvα^ρ​ηdir2ηρ\+𝒦n⟂α^ρ​σ⟂2ηρ\+𝒦nρα^ρ​ηρ​σρ2=:ϵρ,\\displaystyle\\leq\\frac\{\\mathcal\{K\}\_\{\\mathrm\{curv\}\}\}\{\\widehat\{\\alpha\}\_\{\\rho\}\}\\frac\{\\eta\_\{\\mathrm\{dir\}\}^\{2\}\}\{\\eta\_\{\\rho\}\}\+\\frac\{\\mathcal\{K\}\_\{n\}^\{\\perp\}\}\{\\widehat\{\\alpha\}\_\{\\rho\}\}\\frac\{\\sigma\_\{\\perp\}^\{2\}\}\{\\eta\_\{\\rho\}\}\+\\frac\{\\mathcal\{K\}\_\{n\}^\{\\rho\}\}\{\\widehat\{\\alpha\}\_\{\\rho\}\}\\eta\_\{\\rho\}\\sigma\_\{\\rho\}^\{2\}=:\\epsilon\_\{\\rho\},\(111\)lim supT→∞1T​∑t=1T𝔼⁡\[‖Mt⟂‖F2\]\\displaystyle\\limsup\_\{T\\rightarrow\\infty\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\[\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]≤𝒦curvαdir​ηdir\+𝒦n⟂αdir​σ⟂2ηdir\+𝒦nραdir​ηρ2ηdir​σρ2,\\displaystyle\\leq\\frac\{\\mathcal\{K\}\_\{\\mathrm\{curv\}\}\}\{\\alpha\_\{\\mathrm\{dir\}\}\}\\eta\_\{\\mathrm\{dir\}\}\+\\frac\{\\mathcal\{K\}\_\{n\}^\{\\perp\}\}\{\\alpha\_\{\\mathrm\{dir\}\}\}\\frac\{\\sigma\_\{\\perp\}^\{2\}\}\{\\eta\_\{\\mathrm\{dir\}\}\}\+\\frac\{\\mathcal\{K\}\_\{n\}^\{\\rho\}\}\{\\alpha\_\{\\mathrm\{dir\}\}\}\\frac\{\\eta\_\{\\rho\}^\{2\}\}\{\\eta\_\{\\mathrm\{dir\}\}\}\\sigma\_\{\\rho\}^\{2\},\(112\)lim supT→∞1T​∑t=1TEt⟂\\displaystyle\\limsup\_\{T\\rightarrow\\infty\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}E\_\{t\}^\{\\perp\}≤2​𝒦curv​ηdir2\+2​𝒦n⟂​σ⟂2\+2​𝒦nρ​σρ2​ηρ2\.\\displaystyle\\leq 2\\mathcal\{K\}\_\{\\mathrm\{curv\}\}\\eta\_\{\\mathrm\{dir\}\}^\{2\}\+2\\mathcal\{K\}\_\{n\}^\{\\perp\}\\sigma\_\{\\perp\}^\{2\}\+2\\mathcal\{K\}\_\{n\}^\{\\rho\}\\sigma\_\{\\rho\}^\{2\}\\eta\_\{\\rho\}^\{2\}\.\(113\)Consequently,

lim supT→∞1T​∑t=1T𝔼⁡\[‖Gt⟂‖F2\]≤𝒪⁡\(ηdir\)\+𝒪⁡\(σ⟂2ηdir\)\+𝒪⁡\(ηρ2ηdir​σρ2\)=:ϵ⟂\.\\limsup\_\{T\\rightarrow\\infty\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\[\\\|G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\]\\leq\\mathcal\{O\}\(\\eta\_\{\\mathrm\{dir\}\}\)\+\\mathcal\{O\}\\left\(\\frac\{\\sigma\_\{\\perp\}^\{2\}\}\{\\eta\_\{\\mathrm\{dir\}\}\}\\right\)\+\\mathcal\{O\}\\left\(\\frac\{\\eta\_\{\\rho\}^\{2\}\}\{\\eta\_\{\\mathrm\{dir\}\}\}\\sigma\_\{\\rho\}^\{2\}\\right\)=:\\epsilon\_\{\\perp\}\.\(114\)

###### Proof\.

The first three bounds are obtained by dividing \([110](https://arxiv.org/html/2608.21024#A4.E110)\) byα^ρ​ηρ\\widehat\{\\alpha\}\_\{\\rho\}\\eta\_\{\\rho\},αdir​ηdir\\alpha\_\{\\mathrm\{dir\}\}\\eta\_\{\\mathrm\{dir\}\}, and1/21/2, respectively\. For the final true tangent\-gradient bound, use

‖Gt⟂‖F2=‖Mt⟂−\(Mt⟂−Gt⟂\)‖F2≤2​‖Mt⟂‖F2\+2​‖Mt⟂−Gt⟂‖F2,\\\|G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\\|M\_\{t\}^\{\\perp\}\-\(M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}\\leq 2\\\|M\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+2\\\|M\_\{t\}^\{\\perp\}\-G\_\{t\}^\{\\perp\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\(115\)then take expectation and substitute the tangent\-momentum and tracking\-error bounds above\. ∎

## Appendix EPL Recursion for an Aligned First\-Order Model

This appendix analyzes the first\-order modelWt\+1=Wt−η​HtW\_\{t\+1\}=W\_\{t\}\-\\eta H\_\{t\}for RODE’s effective displacement\. It is separate from the exact raw\-buffer recurrence proved in Appendix[D](https://arxiv.org/html/2608.21024#A4): finite geodesic curvature, unequal coordinate stepsizes, the radial floor, and finite projected Newton–Schulz error must be absorbed intoHtH\_\{t\}or controlled as residual terms before this model applies\. Within this scope, boundedness, effective alignment, smoothness, and the PL condition yield linear convergence to a stepsize\-controlled neighborhood\.

### E\.1Additional assumptions for PL analysis

###### Assumption 7\(PL smooth setting\)\.

The objective isLL\-smooth and satisfies the Polyak\-Lojasiewicz condition

12​‖∇f​\(Wt\)‖F2≥μ⁡\(f⁡\(Wt\)−f∗\)\.\\frac\{1\}\{2\}\\\|\\nabla f\(W\_\{t\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}\\geq\\mu\(f\(W\_\{t\}\)\-f^\{\*\}\)\.\(116\)

###### Assumption 8\(Bounded effective direction\)\.

The full effective update directionHtH\_\{t\}satisfies‖Ht‖F≤G\\\|H\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq G\.

###### Assumption 9\(Local reprojected tangent\-gradient regularity\)\.

For the momentum history used in the PL argument, letGitanG\_\{i\}^\{\\mathrm\{tan\}\}denote the tangent gradient from stepiireprojected into the current tangent space atUtU\_\{t\}, and letGtan=Gt⟂G\_\{\\mathrm\{tan\}\}=G\_\{t\}^\{\\perp\}\. There is a constantLtanL\_\{\\mathrm\{tan\}\}such that

‖Gitan−Gtan‖F≤Ltan​‖Wi−Wt‖F\.\\\|G\_\{i\}^\{\\mathrm\{tan\}\}\-G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\\leq L\_\{\\mathrm\{tan\}\}\\\|W\_\{i\}\-W\_\{t\}\\\|\_\{\\mathrm\{F\}\}\.\(117\)This local condition accounts for both ordinary gradient variation and projection drift from moving tangent spaces; it is not implied by EuclideanLL\-smoothness alone\.

###### Assumption 10\(Bounded tangent gradient and local polar stability\)\.

Along the trajectory considered in the PL argument,‖Gtan‖F≤Gmax\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\\leq G\_\{\\max\}\. Moreover, the projected polar map is locally Lipschitz around the tangent gradients: for the momentum tracking errors considered below,

‖𝒫t​\(Polar⁡\(Gtan\+E\)\)−𝒫t​\(Polar⁡\(Gtan\)\)‖F≤Cpol​‖E‖F\.\\\|\\mathcal\{P\}\_\{t\}\(\\mathrm\{Polar\}\(G\_\{\\mathrm\{tan\}\}\+E\)\)\-\\mathcal\{P\}\_\{t\}\(\\mathrm\{Polar\}\(G\_\{\\mathrm\{tan\}\}\)\)\\\|\_\{\\mathrm\{F\}\}\\leq C\_\{\\mathrm\{pol\}\}\\\|E\\\|\_\{\\mathrm\{F\}\}\.\(118\)This assumption excludes degenerate points where the polar factor is unstable\. Since𝒫t\\mathcal\{P\}\_\{t\}is non\-expansive, the displayed condition follows from the corresponding local Lipschitz bound for the unprojected polar map; the global\-neighborhood proof above instead isolates polar accuracy through Assumption[5](https://arxiv.org/html/2608.21024#Thmassumption5)\.

###### Assumption 11\(Strong directional alignment\)\.

There existsα\>0\\alpha\>0such that

⟨Gt,Ht⟩F≥α​‖Gt‖F2\.\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\geq\\alpha\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(119\)

The next two lemmas give sufficient alignment conditions for a projected\-polar first\-order model\. They do not establish the condition for every finite\-step implementation branch; Theorem[2](https://arxiv.org/html/2608.21024#Thmtheorem2)assumes effective alignment directly\.

### E\.2Radial alignment

###### Lemma 1\(Radial alignment\)\.

The radial update direction has positive alignment with the radial gradient\.

###### Proof\.

WriteWt=ρt​UtW\_\{t\}=\\rho\_\{t\}U\_\{t\}\. The radial gradient vector is

Grad=⟨Gt,Ut⟩F​Ut\.G\_\{\\mathrm\{rad\}\}=\\langle G\_\{t\},U\_\{t\}\\rangle\_\{\\mathrm\{F\}\}U\_\{t\}\.\(120\)The ideal radial descent direction is this gradient component; the outer PL recursion applies the scalar stepsize separately:

Hrad=Grad\.H\_\{\\mathrm\{rad\}\}=G\_\{\\mathrm\{rad\}\}\.\(121\)Therefore

⟨Grad,Hrad⟩F\\displaystyle\\langle G\_\{\\mathrm\{rad\}\},H\_\{\\mathrm\{rad\}\}\\rangle\_\{\\mathrm\{F\}\}=‖Grad‖F2\.\\displaystyle=\\\|G\_\{\\mathrm\{rad\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(122\)LetKrad=1K\_\{\\mathrm\{rad\}\}=1\. Then

⟨Grad,Hrad⟩F≥Krad​‖Grad‖F2\.\\langle G\_\{\\mathrm\{rad\}\},H\_\{\\mathrm\{rad\}\}\\rangle\_\{\\mathrm\{F\}\}\\geq K\_\{\\mathrm\{rad\}\}\\\|G\_\{\\mathrm\{rad\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(123\)∎

### E\.3Tangent alignment

###### Lemma 2\(Tangent alignment in the projected\-polar model\)\.

Under Assumptions[8](https://arxiv.org/html/2608.21024#Thmassumption8)–[10](https://arxiv.org/html/2608.21024#Thmassumption10)and sufficiently small learning rate, the projected\-polar tangent step in the first\-order model remains positively aligned with the true tangent gradient\. The same result holds for scalar damping bounded below by a positive constant\.

###### Proof\.

LetGtanG\_\{\\mathrm\{tan\}\}denote the tangent component ofGtG\_\{t\}at the current directionUtU\_\{t\}\. The tangent update is based on momentum\. To show that momentum preserves alignment, use the projection\-based momentum form of the EMA:GitanG\_\{i\}^\{\\mathrm\{tan\}\}denotes a past tangent gradient reprojected into the current tangent space, and

Mt=∑i=0twi​Gitan,M\_\{t\}=\\sum\_\{i=0\}^\{t\}w\_\{i\}G\_\{i\}^\{\\mathrm\{tan\}\},\(124\)where, with initialization at step00,

wi=\{\(1−β\)​βt−i,i≥1,βt,i=0\.w\_\{i\}=\\begin\{cases\}\(1\-\\beta\)\\beta^\{t\-i\},&i\\geq 1,\\\\ \\beta^\{t\},&i=0\.\\end\{cases\}\(125\)The weights sum to one:

∑i=0twi\\displaystyle\\sum\_\{i=0\}^\{t\}w\_\{i\}=\(∑i=1t\(1−β\)​βt−i\)\+βt\\displaystyle=\\left\(\\sum\_\{i=1\}^\{t\}\(1\-\\beta\)\\beta^\{t\-i\}\\right\)\+\\beta^\{t\}\(126\)=\(1−βt\)\+βt=1\.\\displaystyle=\(1\-\\beta^\{t\}\)\+\\beta^\{t\}=1\.\(127\)Define the momentum tracking errorEt=Mt−GtanE\_\{t\}=M\_\{t\}\-G\_\{\\mathrm\{tan\}\}\. Because∑iwi=1\\sum\_\{i\}w\_\{i\}=1,

Et\\displaystyle E\_\{t\}=∑i=0twi​Gitan−∑i=0twi​Gtan\\displaystyle=\\sum\_\{i=0\}^\{t\}w\_\{i\}G\_\{i\}^\{\\mathrm\{tan\}\}\-\\sum\_\{i=0\}^\{t\}w\_\{i\}G\_\{\\mathrm\{tan\}\}\(128\)=∑i=0twi​\(Gitan−Gtan\)\.\\displaystyle=\\sum\_\{i=0\}^\{t\}w\_\{i\}\(G\_\{i\}^\{\\mathrm\{tan\}\}\-G\_\{\\mathrm\{tan\}\}\)\.\(129\)By the triangle inequality,

‖Et‖F≤∑i=0twi​‖Gitan−Gtan‖F\.\\\|E\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq\\sum\_\{i=0\}^\{t\}w\_\{i\}\\\|G\_\{i\}^\{\\mathrm\{tan\}\}\-G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\.\(130\)Using Assumption[9](https://arxiv.org/html/2608.21024#Thmassumption9),

‖Gitan−Gtan‖F≤Ltan​‖Wi−Wt‖F\.\\\|G\_\{i\}^\{\\mathrm\{tan\}\}\-G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\\leq L\_\{\\mathrm\{tan\}\}\\\|W\_\{i\}\-W\_\{t\}\\\|\_\{\\mathrm\{F\}\}\.\(131\)If each update has norm at mostGGand learning rateη\\eta, then

‖Wi−Wt‖F\\displaystyle\\\|W\_\{i\}\-W\_\{t\}\\\|\_\{\\mathrm\{F\}\}=‖∑k=i\+1tη​Hk‖F\\displaystyle=\\left\\\|\\sum\_\{k=i\+1\}^\{t\}\\eta H\_\{k\}\\right\\\|\_\{\\mathrm\{F\}\}\(132\)≤∑k=i\+1tη​‖Hk‖F\\displaystyle\\leq\\sum\_\{k=i\+1\}^\{t\}\\eta\\\|H\_\{k\}\\\|\_\{\\mathrm\{F\}\}\(133\)≤\(t−i\)​η​G\.\\displaystyle\\leq\(t\-i\)\\eta G\.\(134\)Thus

‖Et‖F≤Ltan​G​η​∑i=0twi​\(t−i\)\.\\\|E\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq L\_\{\\mathrm\{tan\}\}G\\eta\\sum\_\{i=0\}^\{t\}w\_\{i\}\(t\-i\)\.\(135\)Let

S=∑i=0twi​\(t−i\)=t​βt\+\(1−β\)​∑k=1t−1k​βk\.S=\\sum\_\{i=0\}^\{t\}w\_\{i\}\(t\-i\)=t\\beta^\{t\}\+\(1\-\\beta\)\\sum\_\{k=1\}^\{t\-1\}k\\beta^\{k\}\.\(136\)ForA=∑k=1t−1k​βkA=\\sum\_\{k=1\}^\{t\-1\}k\\beta^\{k\},

A=β\+2​β2\+⋯\+\(t−1\)​βt−1,A=\\beta\+2\\beta^\{2\}\+\\cdots\+\(t\-1\)\\beta^\{t\-1\},\(137\)and

β​A=β2\+2​β3\+⋯\+\(t−2\)​βt−1\+\(t−1\)​βt\.\\beta A=\\beta^\{2\}\+2\\beta^\{3\}\+\\cdots\+\(t\-2\)\\beta^\{t\-1\}\+\(t\-1\)\\beta^\{t\}\.\(138\)Subtracting,

\(1−β\)​A\\displaystyle\(1\-\\beta\)A=β\+β2\+⋯\+βt−1−\(t−1\)​βt\\displaystyle=\\beta\+\\beta^\{2\}\+\\cdots\+\\beta^\{t\-1\}\-\(t\-1\)\\beta^\{t\}\(139\)=β⁡\(1−βt−1\)1−β−\(t−1\)​βt\.\\displaystyle=\\frac\{\\beta\(1\-\\beta^\{t\-1\}\)\}\{1\-\\beta\}\-\(t\-1\)\\beta^\{t\}\.\(140\)Substituting intoSS,

S\\displaystyle S=t​βt\+β−βt1−β−t​βt\+βt\\displaystyle=t\\beta^\{t\}\+\\frac\{\\beta\-\\beta^\{t\}\}\{1\-\\beta\}\-t\\beta^\{t\}\+\\beta^\{t\}\(141\)=β−βt\+11−β<β1−β\.\\displaystyle=\\frac\{\\beta\-\\beta^\{t\+1\}\}\{1\-\\beta\}<\\frac\{\\beta\}\{1\-\\beta\}\.\(142\)Therefore

‖Et‖F≤Ltan​G​η​β1−β=C1​η\.\\\|E\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq L\_\{\\mathrm\{tan\}\}G\\eta\\frac\{\\beta\}\{1\-\\beta\}=C\_\{1\}\\eta\.\(143\)Now consider the projected polar factor\. Let

Ot=𝒫t​\(Polar⁡\(Mt\)\)=𝒫t​\(Polar⁡\(Gtan\+Et\)\),Og=𝒫t​\(Polar⁡\(Gtan\)\)\.O\_\{t\}=\\mathcal\{P\}\_\{t\}\(\\mathrm\{Polar\}\(M\_\{t\}\)\)=\\mathcal\{P\}\_\{t\}\(\\mathrm\{Polar\}\(G\_\{\\mathrm\{tan\}\}\+E\_\{t\}\)\),\\qquad O\_\{g\}=\\mathcal\{P\}\_\{t\}\(\\mathrm\{Polar\}\(G\_\{\\mathrm\{tan\}\}\)\)\.\(144\)By Assumption[10](https://arxiv.org/html/2608.21024#Thmassumption10), the polar factor is locally stable, so

Ot=Og\+Δo,‖Δo‖F≤Cpol​‖Et‖F≤C1​Cpol​η\.O\_\{t\}=O\_\{g\}\+\\Delta\_\{o\},\\qquad\\\|\\Delta\_\{o\}\\\|\_\{\\mathrm\{F\}\}\\leq C\_\{\\mathrm\{pol\}\}\\\|E\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq C\_\{1\}C\_\{\\mathrm\{pol\}\}\\eta\.\(145\)The first\-order tangent contribution to the effective direction isHtan=γt​OtH\_\{\\mathrm\{tan\}\}=\\gamma\_\{t\}O\_\{t\}, which is tangent by construction because of the projection𝒫t\\mathcal\{P\}\_\{t\}\. For default RODE,γt=1\\gamma\_\{t\}=1; for an optional damped variant, assume

0<γmin≤γt≤1\.0<\\gamma\_\{\\min\}\\leq\\gamma\_\{t\}\\leq 1\.\(146\)Then

⟨Gtan,Htan⟩F\\displaystyle\\langle G\_\{\\mathrm\{tan\}\},H\_\{\\mathrm\{tan\}\}\\rangle\_\{\\mathrm\{F\}\}=γt​\(⟨Gtan,Og⟩F\+⟨Gtan,Δo⟩F\)\.\\displaystyle=\\gamma\_\{t\}\\left\(\\langle G\_\{\\mathrm\{tan\}\},O\_\{g\}\\rangle\_\{\\mathrm\{F\}\}\+\\langle G\_\{\\mathrm\{tan\}\},\\Delta\_\{o\}\\rangle\_\{\\mathrm\{F\}\}\\right\)\.\(147\)For the projected polar target, sinceGtanG\_\{\\mathrm\{tan\}\}lies in the current tangent space,

⟨Gtan,Og⟩F=⟨Gtan,Polar⁡\(Gtan\)⟩F=‖Gtan‖∗≥‖Gtan‖F\.\\langle G\_\{\\mathrm\{tan\}\},O\_\{g\}\\rangle\_\{\\mathrm\{F\}\}=\\langle G\_\{\\mathrm\{tan\}\},\\mathrm\{Polar\}\(G\_\{\\mathrm\{tan\}\}\)\\rangle\_\{\\mathrm\{F\}\}=\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\*\}\\geq\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\.\(148\)For the perturbation term,

\|⟨Gtan,Δo⟩F\|≤‖Gtan‖F​‖Δo‖F≤C1​Cpol​η​‖Gtan‖F\.\|\\langle G\_\{\\mathrm\{tan\}\},\\Delta\_\{o\}\\rangle\_\{\\mathrm\{F\}\}\|\\leq\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\\\|\\Delta\_\{o\}\\\|\_\{\\mathrm\{F\}\}\\leq C\_\{1\}C\_\{\\mathrm\{pol\}\}\\eta\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\.\(149\)Hence

⟨Gtan,Htan⟩F\\displaystyle\\langle G\_\{\\mathrm\{tan\}\},H\_\{\\mathrm\{tan\}\}\\rangle\_\{\\mathrm\{F\}\}≥γt​\(1−C1​Cpol​η\)​‖Gtan‖F\\displaystyle\\geq\\gamma\_\{t\}\(1\-C\_\{1\}C\_\{\\mathrm\{pol\}\}\\eta\)\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\(150\)≥γmin​\(1−C1​Cpol​η\)​‖Gtan‖F\.\\displaystyle\\geq\\gamma\_\{\\min\}\(1\-C\_\{1\}C\_\{\\mathrm\{pol\}\}\\eta\)\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\.\(151\)If‖Gtan‖F≤Gmax\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\\leq G\_\{\\max\}, then

‖Gtan‖F≥‖Gtan‖F2Gmax\.\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}\\geq\\frac\{\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{G\_\{\\max\}\}\.\(152\)Therefore

⟨Gtan,Htan⟩F≥γmin​\(1−C1​Cpol​η\)Gmax​‖Gtan‖F2\.\\langle G\_\{\\mathrm\{tan\}\},H\_\{\\mathrm\{tan\}\}\\rangle\_\{\\mathrm\{F\}\}\\geq\\frac\{\\gamma\_\{\\min\}\(1\-C\_\{1\}C\_\{\\mathrm\{pol\}\}\\eta\)\}\{G\_\{\\max\}\}\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(153\)Forη<1/\(C1​Cpol\)\\eta<1/\(C\_\{1\}C\_\{\\mathrm\{pol\}\}\), define

Ktan=γmin​\(1−C1​Cpol​η\)Gmax\>0\.K\_\{\\mathrm\{tan\}\}=\\frac\{\\gamma\_\{\\min\}\(1\-C\_\{1\}C\_\{\\mathrm\{pol\}\}\\eta\)\}\{G\_\{\\max\}\}\>0\.\(154\)This proves tangent alignment\. ∎

### E\.4Orthogonal synthesis

###### Lemma 3\(Alignment synthesis in the first\-order model\)\.

The radial and tangent alignment bounds imply Assumption[11](https://arxiv.org/html/2608.21024#Thmassumption11)\.

###### Proof\.

Decompose

Gt=Grad\+Gtan,Ht=Hrad\+Htan\.G\_\{t\}=G\_\{\\mathrm\{rad\}\}\+G\_\{\\mathrm\{tan\}\},\\qquad H\_\{t\}=H\_\{\\mathrm\{rad\}\}\+H\_\{\\mathrm\{tan\}\}\.\(155\)The radial and tangent subspaces are orthogonal, so

⟨Grad,Htan⟩F=0,⟨Gtan,Hrad⟩F=0,\\langle G\_\{\\mathrm\{rad\}\},H\_\{\\mathrm\{tan\}\}\\rangle\_\{\\mathrm\{F\}\}=0,\\qquad\\langle G\_\{\\mathrm\{tan\}\},H\_\{\\mathrm\{rad\}\}\\rangle\_\{\\mathrm\{F\}\}=0,\(156\)and

‖Gt‖F2=‖Grad‖F2\+‖Gtan‖F2\.\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\\|G\_\{\\mathrm\{rad\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(157\)Thus

⟨Gt,Ht⟩F\\displaystyle\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}=⟨Grad,Hrad⟩F\+⟨Gtan,Htan⟩F\\displaystyle=\\langle G\_\{\\mathrm\{rad\}\},H\_\{\\mathrm\{rad\}\}\\rangle\_\{\\mathrm\{F\}\}\+\\langle G\_\{\\mathrm\{tan\}\},H\_\{\\mathrm\{tan\}\}\\rangle\_\{\\mathrm\{F\}\}\(158\)≥Krad​‖Grad‖F2\+Ktan​‖Gtan‖F2\.\\displaystyle\\geq K\_\{\\mathrm\{rad\}\}\\\|G\_\{\\mathrm\{rad\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+K\_\{\\mathrm\{tan\}\}\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(159\)LetK=min⁡\{Krad,Ktan\}\>0K=\\min\\\{K\_\{\\mathrm\{rad\}\},K\_\{\\mathrm\{tan\}\}\\\}\>0\. Then

⟨Gt,Ht⟩F≥K⁡\(‖Grad‖F2\+‖Gtan‖F2\)=K​‖Gt‖F2\.\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\geq K\(\\\|G\_\{\\mathrm\{rad\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\\|G\_\{\\mathrm\{tan\}\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\)=K\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(160\)Takingα=K\\alpha=Kproves the claim\. ∎

### E\.5PL linear convergence

###### Theorem 2\(PL recursion for an aligned first\-order model\)\.

Assume smoothness, the PL condition, bounded updates, directional alignment for the first\-order updateWt\+1=Wt−η​HtW\_\{t\+1\}=W\_\{t\}\-\\eta H\_\{t\}, and0<2​η​α​μ<10<2\\eta\\alpha\\mu<1\. Then this aligned effective update satisfies

f⁡\(Wt\)−f∗≤\(1−2​η​α​μ\)t​\(f⁡\(W0\)−f∗\)\+Kerr​η2​α​μ,f\(W\_\{t\}\)\-f^\{\*\}\\leq\(1\-2\\eta\\alpha\\mu\)^\{t\}\(f\(W\_\{0\}\)\-f^\{\*\}\)\+\\frac\{K\_\{\\mathrm\{err\}\}\\eta\}\{2\\alpha\\mu\},\(161\)whereKerr=L2​G2K\_\{\\mathrm\{err\}\}=\\frac\{L\}\{2\}G^\{2\}\.

###### Proof\.

Consider the first\-order effective update

Wt\+1=Wt−η​Ht\.W\_\{t\+1\}=W\_\{t\}\-\\eta H\_\{t\}\.\(162\)ByLL\-smoothness,

f⁡\(Wt\+1\)\\displaystyle f\(W\_\{t\+1\}\)≤f⁡\(Wt\)\+⟨∇f​\(Wt\),Wt\+1−Wt⟩F\+L2​‖Wt\+1−Wt‖F2\\displaystyle\\leq f\(W\_\{t\}\)\+\\langle\\nabla f\(W\_\{t\}\),W\_\{t\+1\}\-W\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\+\\frac\{L\}\{2\}\\\|W\_\{t\+1\}\-W\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\(163\)=f⁡\(Wt\)−η​⟨Gt,Ht⟩F\+L2​η2​‖Ht‖F2\.\\displaystyle=f\(W\_\{t\}\)\-\\eta\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\+\\frac\{L\}\{2\}\\eta^\{2\}\\\|H\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(164\)Using‖Ht‖F≤G\\\|H\_\{t\}\\\|\_\{\\mathrm\{F\}\}\\leq G,

f⁡\(Wt\+1\)≤f⁡\(Wt\)−η​⟨Gt,Ht⟩F\+Kerr​η2,Kerr=L2​G2\.f\(W\_\{t\+1\}\)\\leq f\(W\_\{t\}\)\-\\eta\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\},\\qquad K\_\{\\mathrm\{err\}\}=\\frac\{L\}\{2\}G^\{2\}\.\(165\)Directional alignment gives

−η​⟨Gt,Ht⟩F≤−η​α​‖Gt‖F2\.\-\\eta\\langle G\_\{t\},H\_\{t\}\\rangle\_\{\\mathrm\{F\}\}\\leq\-\\eta\\alpha\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(166\)Therefore

f⁡\(Wt\+1\)≤f⁡\(Wt\)−η​α​‖Gt‖F2\+Kerr​η2\.f\(W\_\{t\+1\}\)\\leq f\(W\_\{t\}\)\-\\eta\\alpha\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\.\(167\)Subtractf∗f^\{\*\}from both sides:

f⁡\(Wt\+1\)−f∗≤f⁡\(Wt\)−f∗−η​α​‖Gt‖F2\+Kerr​η2\.f\(W\_\{t\+1\}\)\-f^\{\*\}\\leq f\(W\_\{t\}\)\-f^\{\*\}\-\\eta\\alpha\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\.\(168\)The PL condition implies

‖Gt‖F2≥2​μ​\(f⁡\(Wt\)−f∗\)\.\\\|G\_\{t\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\geq 2\\mu\(f\(W\_\{t\}\)\-f^\{\*\}\)\.\(169\)Thus

f⁡\(Wt\+1\)−f∗\\displaystyle f\(W\_\{t\+1\}\)\-f^\{\*\}≤f⁡\(Wt\)−f∗−2​η​α​μ​\(f⁡\(Wt\)−f∗\)\+Kerr​η2\\displaystyle\\leq f\(W\_\{t\}\)\-f^\{\*\}\-2\\eta\\alpha\\mu\(f\(W\_\{t\}\)\-f^\{\*\}\)\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\(170\)=\(1−2​η​α​μ\)​\(f⁡\(Wt\)−f∗\)\+Kerr​η2\.\\displaystyle=\(1\-2\\eta\\alpha\\mu\)\(f\(W\_\{t\}\)\-f^\{\*\}\)\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\.\(171\)LetΔt=f⁡\(Wt\)−f∗\\Delta\_\{t\}=f\(W\_\{t\}\)\-f^\{\*\}andρ=2​η​α​μ\\rho=2\\eta\\alpha\\mu\. The recursion is

Δt\+1≤\(1−ρ\)​Δt\+Kerr​η2\.\\Delta\_\{t\+1\}\\leq\(1\-\\rho\)\\Delta\_\{t\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\.\(172\)Expanding the first few steps makes the geometric structure explicit:

Δ1\\displaystyle\\Delta\_\{1\}≤\(1−ρ\)​Δ0\+Kerr​η2,\\displaystyle\\leq\(1\-\\rho\)\\Delta\_\{0\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\},\(173\)Δ2\\displaystyle\\Delta\_\{2\}≤\(1−ρ\)2​Δ0\+\(1−ρ\)​Kerr​η2\+Kerr​η2,\\displaystyle\\leq\(1\-\\rho\)^\{2\}\\Delta\_\{0\}\+\(1\-\\rho\)K\_\{\\mathrm\{err\}\}\\eta^\{2\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\},\(174\)Δ3\\displaystyle\\Delta\_\{3\}≤\(1−ρ\)3​Δ0\+\(1−ρ\)2​Kerr​η2\+\(1−ρ\)​Kerr​η2\+Kerr​η2\.\\displaystyle\\leq\(1\-\\rho\)^\{3\}\\Delta\_\{0\}\+\(1\-\\rho\)^\{2\}K\_\{\\mathrm\{err\}\}\\eta^\{2\}\+\(1\-\\rho\)K\_\{\\mathrm\{err\}\}\\eta^\{2\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\.\(175\)By induction,

Δt≤\(1−ρ\)t​Δ0\+Kerr​η2​∑i=0t−1\(1−ρ\)i\.\\Delta\_\{t\}\\leq\(1\-\\rho\)^\{t\}\\Delta\_\{0\}\+K\_\{\\mathrm\{err\}\}\\eta^\{2\}\\sum\_\{i=0\}^\{t\-1\}\(1\-\\rho\)^\{i\}\.\(176\)Since

∑i=0t−1\(1−ρ\)i=1−\(1−ρ\)tρ<1ρ,\\sum\_\{i=0\}^\{t\-1\}\(1\-\\rho\)^\{i\}=\\frac\{1\-\(1\-\\rho\)^\{t\}\}\{\\rho\}<\\frac\{1\}\{\\rho\},\(177\)we obtain

Δt\\displaystyle\\Delta\_\{t\}≤\(1−ρ\)t​Δ0\+Kerr​η2ρ\\displaystyle\\leq\(1\-\\rho\)^\{t\}\\Delta\_\{0\}\+\\frac\{K\_\{\\mathrm\{err\}\}\\eta^\{2\}\}\{\\rho\}\(178\)=\(1−2​η​α​μ\)t​\(f⁡\(W0\)−f∗\)\+Kerr​η2​α​μ\.\\displaystyle=\(1\-2\\eta\\alpha\\mu\)^\{t\}\(f\(W\_\{0\}\)\-f^\{\*\}\)\+\\frac\{K\_\{\\mathrm\{err\}\}\\eta\}\{2\\alpha\\mu\}\.\(179\)This proves linear convergence to an𝒪⁡\(η\)\\mathcal\{O\}\(\\eta\)neighborhood for the aligned first\-order model\. It does not extend the PL rate automatically to every branch of the finite implementation; Appendix[D](https://arxiv.org/html/2608.21024#A4)provides the separate convergence\-neighborhood guarantee for the exact raw\-buffer update under its stated assumptions\. ∎