Stiefel-AdamW：一种面向线性分解模块的几何感知优化器

# Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
Source: [https://arxiv.org/html/2609.21039](https://arxiv.org/html/2609.21039)
EMANUELE ZANGRANDO††thanks:Gran Sasso Science Institute, L’Aquila, Italy \([emanuele\.zangrando@gssi\.it](mailto:[email protected])\)\.[https://orcid.org/0000-0002-8410-1372](https://orcid.org/0000-0002-8410-1372)MARCO SUTTI††thanks:Gran Sasso Science Institute, L’Aquila, Italy \([marco\.sutti@gssi\.it](mailto:[email protected])\)\. M\. Sutti is a member of INdAM GNCS\.[https://orcid.org/](https://orcid.org/)

###### Abstract

A pervasive structural pattern in modern deep learning is the*linear factorization block*: a submodule of the formW=B​AW=BAin which two parameter matrices are multiplied directly, with no intervening nonlinearity\. Such blocks appear in LoRA adapters, low\-rank compressed layers, query\-key products of self\-attention, and share a common pathology: the factorization is non\-unique, which can destabilize training and limit usable learning rates\. Despite this, factorization blocks are typically optimized with standard Euclidean methods that ignore the underlying geometry\. We introduce Stiefel\-AdamW, a near drop\-in replacement for AdamW for use wherever such blocks appear\. By constraining one factor on the Stiefel manifold while leaving the other Euclidean, Stiefel\-AdamW relaxes the fullGL⁡\(ℝr\)\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\)gauge symmetry to a compact orthogonal symmetry, ruling out factor blow\-up while retaining the coordinate\-wise diagonal preconditioning that gives AdamW its practical strength\. Moment estimation is performed in the ambient Euclidean space, with geometry entering only through a tangent\-space projection and a manifold retraction\. The implementation overhead over AdamW is minimal, and we show that the resulting optimizer inherits both the stability benefits of Riemannian methods and standard convergence guarantees\. We validate Stiefel\-AdamW on LoRA\-style fine\-tuning of GPT2, ViT, and Mistral 7B and on full pretraining of GPT2 on OpenWebText, showing consistent improvements over strong baselines at essentially no additional cost over AdamW\.

## 1\. Introduction

A pervasive structural pattern in modern deep\-learning architectures is the presence of*linear factorization blocks*: trainable submodules of the formW=B​AW=BA, in which two parameter matrices are multiplied directly, with no intervening nonlinearity\. Such blocks arise in many guises across the modern model zoo\. In LoRA\-style parameter\-efficient fine\-tuning[Hu et al\. \(2022\)](https://arxiv.org/html/2609.21039#bib.bib21);[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.21039#bib.bib60);[Hayou et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib17);[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61);[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib62);[Lialin et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib30);[Schotthöfer et al\. \(2025b\)](https://arxiv.org/html/2609.21039#bib.bib51), weight correctors are parametrized asΔ​W=B​A\\varDelta W=BA, withB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}andA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}\. In low\-rank pretraining[Wang et al\. \(2021\)](https://arxiv.org/html/2609.21039#bib.bib34);[Khodak et al\. \(2021\)](https://arxiv.org/html/2609.21039#bib.bib24);[Schotthöfer et al\. \(2025a\)](https://arxiv.org/html/2609.21039#bib.bib50)and network compression[Vogels et al\. \(2019\)](https://arxiv.org/html/2609.21039#bib.bib53);[Saha et al\. \(2023\)](https://arxiv.org/html/2609.21039#bib.bib42);[Mo et al\. \(2025\)](https://arxiv.org/html/2609.21039#bib.bib35);[Schotthöfer et al\. \(2022\)](https://arxiv.org/html/2609.21039#bib.bib49), weight matrices are similarly factorized into two trainable factors\. The same structure also appears*inside*standard architectures: in self\-attention[Vaswani et al\. \(2017\)](https://arxiv.org/html/2609.21039#bib.bib37), the score matrixWQ​WK⊤W\_\{Q\}W\_\{K\}^\{\\top\}factorizes through the rank\-rrkey/query head dimension, and analogous matrix\-matrix factorizations appear in the recurrence kernels of several structured state\-space models \(SSMs\)\. In all these cases, two parameter blocks combine multiplicatively, without an intervening nonlinearity, to produce a single effective linear operator\.

These blocks share a common geometric feature: the product mapΦ⁡\(A,B\)=B​A\\Phi\(A,B\)=BAis highly non\-unique\. For anyM∈GL⁡\(ℝr\)M\\in\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\),Φ⁡\(A,B\)=Φ⁡\(M−1​A,B​M\)\\Phi\(A,B\)=\\Phi\(M^\{\-1\}A,\\,BM\), so each effective weightWWcorresponds to a continuous family of parameter pairs\. This gauge symmetry has direct consequences for optimization\. If the lossℒ\{\\mathcal\{L\}\}, viewed as a function of the effective matrixWW, has a stationary point, thenℒ∘Φ\{\\mathcal\{L\}\}\\circ\\Phihas an entire orbit of equivalent stationary points parametrized byGL⁡\(ℝr\)\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\)\. More worryingly, training along these blocks can be numerically unstable even without spurious minimizers: a sequence\(Mn−1​An,Bn​Mn\)\(M\_\{n\}^\{\-1\}A\_\{n\},\\,B\_\{n\}M\_\{n\}\)withMn→0M\_\{n\}\\to 0produces a bounded productWnW\_\{n\}even though one factor diverges\. These issues are well documented in practice[Mishra et al\. \(2014\)](https://arxiv.org/html/2609.21039#bib.bib33);[Schotthöfer et al\. \(2022\)](https://arxiv.org/html/2609.21039#bib.bib49);[Zangrando et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib59)and become especially pronounced under unbalanced initializations, such as those typical of LoRA\.

Despite the prevalence of this structure, standard practice for training linear factorization blocks is to apply Euclidean optimizers, most commonly AdamW[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2609.21039#bib.bib31), directly to the unconstrained pair\(A,B\)\(A,B\), ignoring the underlying geometry entirely\.

The natural geometric remedy is to optimize on the fixed\-rank matrix manifold

ℳr=ℝ∗r×n×ℝ∗m×r/GL⁡\(ℝr\),\{\\mathcal\{M\}\}\_\{r\}=\\mathbb\{R\}^\{r\\times n\}\_\{\*\}\\times\\mathbb\{R\}^\{m\\times r\}\_\{\*\}/\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\),viewed as a quotient[Mishra et al\. \(2014\)](https://arxiv.org/html/2609.21039#bib.bib33)\. Here,ℝ∗m×r\\mathbb\{R\}^\{m\\times r\}\_\{\*\}denotes the set of full\-rank matrices of sizem×rm\\times r\. While geometrically clean, this approach is at odds with adaptive optimizers such as Adam, since the fullGL⁡\(ℝr\)\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\)invariance forces nonlinear adaptive updates to satisfy strong equivariance constraints that, in general, cannot be reconciled with the coordinate\-wise diagonal preconditioning that gives AdamW much of its practical strength\. As a result, prior Riemannian adaptive methods either give up coordinate\-wise adaptivity in favor of scalar preconditioners, or modify the gradient structure rather than the preconditioner itself[Bécigneul and Ganea \(2019\)](https://arxiv.org/html/2609.21039#bib.bib5);[Sakai and Iiduka \(2025\)](https://arxiv.org/html/2609.21039#bib.bib43);[Schotthöfer et al\. \(2025a\)](https://arxiv.org/html/2609.21039#bib.bib50);[Bian et al\. \(2025\)](https://arxiv.org/html/2609.21039#bib.bib6)\.

In this work, we adopt a different trade\-off\. Rather than enforcing full quotient invariance, we relax it to an orthogonal symmetry by restricting to the product manifoldSt⁡\(n,r\)×ℝ∗m×r\\mathrm\{St\}\(n,r\)\\times\\mathbb\{R\}^\{m\\times r\}\_\{\*\}, on whichΦ\\Phiis invariant only underO⁡\(r\)\\mathrm\{O\}\(r\):

Φ⁡\(A,B\)=Φ⁡\(Q⊤​A,B​Q\),∀Q∈O⁡\(r\)\.\\Phi\(A,B\)=\\Phi\(Q^\{\\top\}\\\!A,\\,BQ\),\\quad\\forall Q\\in\\mathrm\{O\}\(r\)\.This relaxation accomplishes two things at once\.*Geometrically*, it makes the fibers ofΦ\\Phicompact, which rules out the kind of factor blow\-up illustrated above and stabilizes training under aggressive learning rates\.*Algorithmically*, because the constrained factor lives on an embedded submanifold and the other factor in a flat Euclidean space, we can perform all moment accumulation in the ambient space and only project onto the tangent space of the Stiefel manifold immediately before the update\. This preserves AdamW’s coordinate\-wise diagonal preconditioning essentially unchanged\.

The resulting algorithm,Stiefel\-AdamW, is best understood as a small, structural modification of AdamW\. The Euclidean factor is updated by the standard AdamW step; the Stiefel factor is updated by computing the first and second moments in the ambient space, projecting the preconditioned direction onto the tangent space, and retracting back to the manifold\. The retraction step is treated as a modular component: any efficient retraction on the Stiefel manifold can be used, including the Cayley transform, QR\-based retraction, polar decomposition, or Newton–Schulz iteration[Kovarik \(1970\)](https://arxiv.org/html/2609.21039#bib.bib27);[Björck and Bowie \(1971\)](https://arxiv.org/html/2609.21039#bib.bib7);[Higham \(2008\)](https://arxiv.org/html/2609.21039#bib.bib18)\. In our experiments, the choice of retraction has only a marginal effect on final performance, with the best option mildly model\- and problem\-dependent; see[SectionB\.1](https://arxiv.org/html/2609.21039#A2.SS1)and[Table3](https://arxiv.org/html/2609.21039#S5.T3)for a comparison\. The overall implementation overhead over standard AdamW is minimal, yet the resulting optimizer inherits the principal benefits of geometric optimization: numerical stability, parameter invariance to orthogonal reparametrization, low memory footprint, and provable convergence under standard assumptions\.

We emphasize that Stiefel\-AdamW is not a method specific to LoRA or to any particular architectural family\. It is a simple, near drop\-in replacement for AdamW that can \(and should\) be used on*any*factorization block of the formW=B​AW=BAin which the two factors are not separated by a nonlinearity, regardless of where such a block appears in the model\. This includes LoRA adapters, low\-rank compressed layers, and query/key projections in attention blocks\. In all these cases, applying Stiefel\-AdamW rather than vanilla AdamW replaces a Euclidean parametrization with hidden gauge symmetry by one in which \(most of\) the symmetry has been quotiented out, and does so essentially for free\.

Contributions\.Our contributions are as follows\.

1. 1\.We identify factorization blocksW=B​AW=BAubiquitous in modern deep\-learning architectures \(LoRA, low\-rank compressed layers, attention, SSMs\) as a unifying setting in which geometric optimization can be applied transparently, and we propose Stiefel\-AdamW, a near drop\-in replacement for AdamW that should be used wherever such blocks appear\.
2. 2\.We propose a structured relaxation of the fullGL⁡\(ℝr\)\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\)quotient invariance to orthogonal invariance, working on the product manifoldSt⁡\(n,r\)×ℝ∗m×r\\mathrm\{St\}\(n,r\)\\times\\mathbb\{R\}^\{m\\times r\}\_\{\*\}\. This is precisely the relaxation that allows full coordinate\-wise Adam\-style adaptive preconditioning to coexist with geometry\-aware updates, in contrast to prior Riemannian Adam\-type methods that rely on simple scalar preconditioners\.
3. 3\.We design the algorithm so that all adaptive moment estimation is performed in ambient Euclidean space, with geometric corrections \(tangent projection and retraction\) applied only at the update step\. As a consequence, the per\-step cost and memory footprint are essentially those of AdamW\.
4. 4\.We treat the Stiefel retraction as a modular, plug\-and\-play component: any efficient retraction \(Cayley transform, QR, polar decomposition, Newton–Schulz\) can be used, with only marginal differences in practice\. Approximate retractions, when second\-order accurate, remain compatible with the standard convergence theory\.
5. 5\.We establish theoretical guarantees of boundedness of gradients and convergence of the regret function under standard assumptions\.
6. 6\.We validate Stiefel\-AdamW empirically across a range of representative tasks, from LoRA\-style fine\-tuning \(GPT2, ViT, Mistral 7B\) to full LLM pretraining \(GPT2 on OpenWebText\), demonstrating that the additional cost over AdamW is minimal while consistently improving stability and final performance\.

## 2\. Related Work

Riemannian Optimization\.Riemannian gradient methods on embedded and quotient manifolds, including the Stiefel and Grassmann manifolds, have a long history in numerical optimization[Luenberger \(1972\)](https://arxiv.org/html/2609.21039#bib.bib32);[Gabay and Luenberger \(1976\)](https://arxiv.org/html/2609.21039#bib.bib15);[Edelman et al\. \(1998\)](https://arxiv.org/html/2609.21039#bib.bib14);[Yang \(2007\)](https://arxiv.org/html/2609.21039#bib.bib57), and have since been extended to trust\-region[Absil et al\. \(2007\)](https://arxiv.org/html/2609.21039#bib.bib1), quasi\-Newton[Ring and Wirth \(2012\)](https://arxiv.org/html/2609.21039#bib.bib41), and conjugate\-gradient[Sato and Iwai \(2015\)](https://arxiv.org/html/2609.21039#bib.bib48);[Sato \(2016\)](https://arxiv.org/html/2609.21039#bib.bib45);[Sato \(2022\)](https://arxiv.org/html/2609.21039#bib.bib47)methods\. Key tools including retraction mappings[Absil and Malick \(2012\)](https://arxiv.org/html/2609.21039#bib.bib2);[Absil and Oseledets \(2015\)](https://arxiv.org/html/2609.21039#bib.bib3), efficient preconditioners[Vandereycken and Vandewalle \(2010\)](https://arxiv.org/html/2609.21039#bib.bib52);[Boumal and Absil \(2015\)](https://arxiv.org/html/2609.21039#bib.bib10), and quotient geometries[Mishra et al\. \(2014\)](https://arxiv.org/html/2609.21039#bib.bib33)have been thoroughly developed; comprehensive treatments can be found in[Absil et al\. \(2008\)](https://arxiv.org/html/2609.21039#bib.bib4);[Sato \(2021\)](https://arxiv.org/html/2609.21039#bib.bib46);[Boumal \(2023\)](https://arxiv.org/html/2609.21039#bib.bib9)\.

Adaptive Methods and their Riemannian Extensions\.In deep learning, adaptive optimizers such as AdaGrad[Duchi et al\. \(2011\)](https://arxiv.org/html/2609.21039#bib.bib13), RMSProp[Hinton et al\. \(2012\)](https://arxiv.org/html/2609.21039#bib.bib19), Adam[Kingma and Ba \(2015\)](https://arxiv.org/html/2609.21039#bib.bib25), AMSGrad[Reddi et al\. \(2018\)](https://arxiv.org/html/2609.21039#bib.bib11), and AdamW[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2609.21039#bib.bib31), are the de facto standard, combining fast convergence, robustness, and low memory overhead\. Extending them to manifold\-constrained settings is nontrivial, because adaptive moments and preconditioning must respect the underlying geometry\. Stochastic Riemannian optimization began with Riemannian SGD[Bonnabel \(2013\)](https://arxiv.org/html/2609.21039#bib.bib8)and variance\-reduced variants[Zhang et al\. \(2016\)](https://arxiv.org/html/2609.21039#bib.bib36);[Kasai et al\. \(2018\)](https://arxiv.org/html/2609.21039#bib.bib39)\. More recent work has proposed Riemannian AdaGrad and AMSGrad[Bécigneul and Ganea \(2019\)](https://arxiv.org/html/2609.21039#bib.bib5), modified AMSGrad schemes[Sakai and Iiduka \(2022\)](https://arxiv.org/html/2609.21039#bib.bib44), RASA[Kasai et al\. \(2019\)](https://arxiv.org/html/2609.21039#bib.bib23), and Riemannian adaptive gradient methods with theoretical guarantees[Bian et al\. \(2025\)](https://arxiv.org/html/2609.21039#bib.bib6);[Sakai and Iiduka \(2025\)](https://arxiv.org/html/2609.21039#bib.bib43)\. A common limitation of these approaches is that they rely on scalar or geometry\-compatible preconditioners; the fully coordinate\-wise diagonal preconditioning central to AdamW is generally incompatible with strict manifold invariance, which is the gap Stiefel\-AdamW is designed to close\.

Linear Factorization Blocks in Deep Learning\.Matrix factorization blocks of the formW=B​AW=BA, in which two parameter matrices multiply directly without an intervening nonlinearity, appear throughout modern architectures\. They arise explicitly in LoRA\-style parameter\-efficient fine\-tuning[Hu et al\. \(2022\)](https://arxiv.org/html/2609.21039#bib.bib21)and its variants[Hayou et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib17);[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61);[Zhu et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib63);[Wang et al\. \(2025\)](https://arxiv.org/html/2609.21039#bib.bib55);[Schotthöfer et al\. \(2025b\)](https://arxiv.org/html/2609.21039#bib.bib51), in network compression[Vogels et al\. \(2019\)](https://arxiv.org/html/2609.21039#bib.bib53);[Saha et al\. \(2023\)](https://arxiv.org/html/2609.21039#bib.bib42);[Schotthöfer et al\. \(2022\)](https://arxiv.org/html/2609.21039#bib.bib49), and implicitly inside standard architectures such as the query\-key product in self\-attention[Vaswani et al\. \(2017\)](https://arxiv.org/html/2609.21039#bib.bib37)\. The non\-uniqueness of such factorizations and its consequences for training stability have been studied through dynamical low\-rank approximation[Koch and Lubich \(2007\)](https://arxiv.org/html/2609.21039#bib.bib26);[Hnatiuk et al\. \(2026\)](https://arxiv.org/html/2609.21039#bib.bib20);[Zangrando et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib59)and quotient\-manifold optimization[Mishra et al\. \(2014\)](https://arxiv.org/html/2609.21039#bib.bib33)\. Methods such as GeoLoRA[Schotthöfer et al\. \(2025b\)](https://arxiv.org/html/2609.21039#bib.bib51)have explicitly exploited Grassmannian geometry in low\-rank adaptation, and recent work has combined momentum and adaptivity with Riemannian updates[Schotthöfer et al\. \(2025a\)](https://arxiv.org/html/2609.21039#bib.bib50);[Sakai and Iiduka \(2025\)](https://arxiv.org/html/2609.21039#bib.bib43)\.

Relation to Prior Work\.The closest methods to Stiefel\-AdamW are GeoLoRA[Schotthöfer et al\. \(2025b\)](https://arxiv.org/html/2609.21039#bib.bib51)and RAdam[Bécigneul and Ganea \(2019\)](https://arxiv.org/html/2609.21039#bib.bib5)or its Stiefel\-specific version, Cayley Adam[Li et al\. \(2020\)](https://arxiv.org/html/2609.21039#bib.bib29)\. GeoLoRA enforces geometry on the full factorization via a quotient\-manifold formulation, which precludes coordinate\-wise adaptive preconditioning\. RAdam defines a Riemannian Adam on the Stiefel manifold but uses a scalar preconditioner, departing from AdamW’s diagonal adaptivity\. In contrast, Stiefel\-AdamW constrains only one factor to the Stiefel manifold while leaving the other in Euclidean space\. This product\-manifold structure is precisely what makes full coordinate\-wise adaptive preconditioning tractable: moment accumulation is performed in the ambient space for both factors, and geometry enters only through one factor via a tangent\-space projection followed by a retraction on the Stiefel manifold\. The result is an optimizer that inherits the stability benefits of Riemannian methods and the practical performance of AdamW, at minimal additional cost\.

## 3\. The Proposed Method: Stiefel\-AdamW

### 3\.1\. Problem Setup

We consider a trainable linear factorization block of the formW=B​AW=BA, whereB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}andA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}\. In order to avoid potential instabilities due to non\-uniqueness of this representation, we impose a row\-orthonormality constraint onAA, namelyA​A⊤=IrAA^\{\\top\}=I\_\{r\}, i\.e\., we requireA⊤A^\{\\top\}to lie on the Stiefel manifoldSt⁡\(n,r\)=\{X∈ℝn×r:X⊤​X=Ir\},\\mathrm\{St\}\(n,r\)=\\\{X\\in\\mathbb\{R\}^\{n\\times r\}\\colon X^\{\\top\}\\\!X=I\_\{r\}\\\},while leavingBBunconstrained\. This restriction reduces the invariance group fromGL⁡\(ℝr\)\\mathrm\{GL\}\(\\mathbb\{R\}^\{r\}\)to the compact groupO⁡\(r\)\\mathrm\{O\}\(r\), making the fibers ofΦ\\Phicompact and ruling out the factor blow\-up illustrated in[Section1](https://arxiv.org/html/2609.21039#S1); see also[Section4\.2](https://arxiv.org/html/2609.21039#S4.SS2)for further details\. The resulting product\-manifold structureSt⁡\(n,r\)×ℝm×r\\mathrm\{St\}\(n,r\)\\times\\mathbb\{R\}^\{m\\times r\}enables an efficient fully coordinate\-wise adaptive update: moment accumulation is performed in the ambient Euclidean space for both factors, with geometric corrections entering only through a tangent\-space projection and a retraction onto the Stiefel manifold for theAAfactor\.

### 3\.2\. Description of the Algorithm

In this section, we describe one iteration of Stiefel\-AdamW; the pseudocode is given in[Algorithm1](https://arxiv.org/html/2609.21039#alg1)\. For simplicity of exposition, we omit bias correction and explicit weight decay; both can be incorporated straightforwardly as in AdamW\.

Algorithm 1Single iteration of Stiefel\-AdamW\.1:

AtA\_\{t\}with

At​At⊤=IrA\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\},

BtB\_\{t\},

Mt−1AM^\{A\}\_\{t\-1\},

Mt−1BM^\{B\}\_\{t\-1\},

Vt−1AV^\{A\}\_\{t\-1\},

Vt−1BV^\{B\}\_\{t\-1\},

ηt\\eta\_\{t\},

β1\\beta\_\{1\},

β2\\beta\_\{2\},

ε\\varepsilon
2:

3:

GtB←∇Bℒ​\(Bt​At\)​At⊤G\_\{t\}^\{B\}\\leftarrow\\nabla\_\{B\}\{\\mathcal\{L\}\}\(B\_\{t\}A\_\{t\}\)\\,A\_\{t\}^\{\\top\}⊳\\trianglerightEuclidean gradient w\.r\.t\.BB

4:

GtA←Bt⊤​∇Aℒ​\(Bt​At\)G\_\{t\}^\{A\}\\leftarrow B\_\{t\}^\{\\top\}\\nabla\_\{A\}\{\\mathcal\{L\}\}\(B\_\{t\}A\_\{t\}\)⊳\\trianglerightEuclidean gradient w\.r\.t\.AA

5:

6:

MtB←β1​Mt−1B\+\(1−β1\)​GtBM^\{B\}\_\{t\}\\leftarrow\\beta\_\{1\}M^\{B\}\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)\\,G\_\{t\}^\{B\}⊳\\trianglerightFirst moment,BB

7:

MtA←β1​Mt−1A\+\(1−β1\)​GtAM^\{A\}\_\{t\}\\leftarrow\\beta\_\{1\}M^\{A\}\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)\\,G\_\{t\}^\{A\}⊳\\trianglerightFirst moment,AA

8:

VtB←β2​Vt−1B\+\(1−β2\)​\(GtB\)∘2V^\{B\}\_\{t\}\\leftarrow\\beta\_\{2\}V^\{B\}\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\,\(G\_\{t\}^\{B\}\)^\{\\circ 2\}⊳\\trianglerightSecond moment,BB

9:

VtA←β2​Vt−1A\+\(1−β2\)​\(GtA\)∘2V^\{A\}\_\{t\}\\leftarrow\\beta\_\{2\}V^\{A\}\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\,\(G\_\{t\}^\{A\}\)^\{\\circ 2\}⊳\\trianglerightSecond moment,AA

10:

Bt\+1←Bt−ηt​\(MtB/\(VtB\+ε\)\+λ​Bt\)B\_\{t\+1\}\\leftarrow B\_\{t\}\-\\eta\_\{t\}\\Bigl\(M^\{B\}\_\{t\}/\\\!\\left\(\\sqrt\{V^\{B\}\_\{t\}\+\\varepsilon\}\\right\)\+\\lambda B\_\{t\}\\Bigr\)⊳\\trianglerightStandard AdamW step onBB

11:

Xt←At⊤X\_\{t\}\\leftarrow A\_\{t\}^\{\\top\}⊳\\trianglerightColumn convention,Xt∈St⁡\(n,r\)X\_\{t\}\\in\\mathrm\{St\}\(n,r\)

12:

Dt←MtA/\(VtA\+ε\)D\_\{t\}\\leftarrow M^\{A\}\_\{t\}/\(\\sqrt\{V^\{A\}\_\{t\}\+\\varepsilon\}\)⊳\\trianglerightPreconditioned direction

13:

ξt←−ηt​PXt⁡\(Dt\)\\xi\_\{t\}\\leftarrow\{\-\\eta\_\{t\}\}\\,\\Proj\_\{X\_\{t\}\}\(D\_\{t\}\)⊳\\trianglerightProject ontoTXt​St​\(n,r\)\\mathrm\{T\}\_\{X\_\{t\}\}\\mathrm\{St\}\(n,r\),PX⁡\(Z\)=X​skew​\(X⊤​Z\)\+\(I−X​X⊤\)​Z\\Proj\_\{X\}\(Z\)=X\\,\\mathrm\{skew\}\(X^\{\\top\}Z\)\+\(I\-XX^\{\\top\}\)Z

14:

Xt\+1←RetrXt⁡\(ξt\),At\+1←Xt\+1⊤X\_\{t\+1\}\\leftarrow\\Retraction\_\{X\_\{t\}\}\(\\xi\_\{t\}\),\\,A\_\{t\+1\}\\leftarrow X\_\{t\+1\}^\{\\top\}⊳\\trianglerightRetract toSt⁡\(n,r\)\\mathrm\{St\}\(n,r\); see[Section3\.3](https://arxiv.org/html/2609.21039#S3.SS3)

LetWt∈ℝm×nW\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}be a weight matrix at iterationtt, represented by the factorizationWt=Bt​AtW\_\{t\}=B\_\{t\}A\_\{t\}, whereAt∈ℝr×nA\_\{t\}\\in\\mathbb\{R\}^\{r\\times n\}andBt∈ℝm×rB\_\{t\}\\in\\mathbb\{R\}^\{m\\times r\}\. The objective function is evaluated asℒ⁡\(Bt​At\)\{\\mathcal\{L\}\}\(B\_\{t\}A\_\{t\}\)\. Next, the algorithm computes the Euclidean gradientsGtB=∇Bℒ​\(Bt​At\)G\_\{t\}^\{B\}=\\nabla\_\{B\}\{\\mathcal\{L\}\}\(B\_\{t\}A\_\{t\}\)andGtA=∇Aℒ​\(Bt​At\)G\_\{t\}^\{A\}=\\nabla\_\{A\}\{\\mathcal\{L\}\}\(B\_\{t\}A\_\{t\}\)\. As in Adam and AdamW, Stiefel\-AdamW then computes the first and second moments for both factors, i\.e\.,

\{MtB=β1​Mt−1B\+\(1−β1\)​GtB,MtA=β1​Mt−1A\+\(1−β1\)​GtA,\{VtB=β2​Vt−1B\+\(1−β2\)​\(GtB\)∘2,VtA=β2​Vt−1A\+\(1−β2\)​\(GtA\)∘2,\\begin\{cases\}M^\{B\}\_\{t\}=\\beta\_\{1\}M^\{B\}\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)\\,G\_\{t\}^\{B\},\\\\ M^\{A\}\_\{t\}=\\beta\_\{1\}M^\{A\}\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)\\,G\_\{t\}^\{A\},\\end\{cases\}\\qquad\\begin\{cases\}V^\{B\}\_\{t\}=\\beta\_\{2\}V^\{B\}\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\,\(G\_\{t\}^\{B\}\)^\{\\circ 2\},\\\\ V^\{A\}\_\{t\}=\\beta\_\{2\}V^\{A\}\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\,\(G\_\{t\}^\{A\}\)^\{\\circ 2\},\\end\{cases\}where∘2denotes elementwise squaring\.

Up to this step, both factors are treated in the same way; however, the subsequent update steps do differ\. Indeed, sinceBtB\_\{t\}is unconstrained, the algorithm performs the usual AdamW update, i\.e\.,Bt\+1=Bt−ηt​\(MtB/\(VtB\+ε\)\+λ​Bt\)B\_\{t\+1\}=B\_\{t\}\-\\eta\_\{t\}\\Bigl\(M^\{B\}\_\{t\}/\(\\sqrt\{V^\{B\}\_\{t\}\+\\varepsilon\}\)\+\\lambda B\_\{t\}\\Bigr\), whereηt\\eta\_\{t\}is the learning rate,//indicates elementwise division, the square root is also meant to be performed componentwise, and theε\>0\\varepsilon\>0is a small constant to avoid blowup of the metric\. This factor requires no Riemannian machinery, which keeps the method simple and efficient\.

For the orthonormal factor, we employ a retraction\-based Riemannian update\. For convenience, we switch to a column\-orthonormal representation by definingXt≔At⊤∈ℝn×r,X\_\{t\}\\coloneqq A\_\{t\}^\{\\top\}\\in\\mathbb\{R\}^\{n\\times r\},so thatXt∈St⁡\(n,r\)X\_\{t\}\\in\\mathrm\{St\}\(n,r\), i\.e\.,Xt⊤​Xt=IrX\_\{t\}^\{\\top\}\\\!X\_\{t\}=I\_\{r\}\. A key feature of Stiefel\-AdamW is that it performs adaptive moment estimation in the ambient Euclidean space before projecting onto the tangent space\. This allows us to use coordinate\-wise preconditioning, as in AdamW\. In contrast, many existing Riemannian adaptive methods restrict the preconditioner to be scalar or geometry\-compatible to preserve invariance, thereby limiting their practical effectiveness\. More precisely, the Euclidean adaptive direction is first formed asDt=MtA/\(VtA\+ε\)D\_\{t\}=M^\{A\}\_\{t\}/\(\\sqrt\{V^\{A\}\_\{t\}\+\\varepsilon\}\), and then projected onto the tangent spaceTXt​St​\(n,r\)\\mathrm\{T\}\_\{X\_\{t\}\}\\mathrm\{St\}\(n,r\)to obtain the directionξt=−ηt​PXt⁡\(Dt\)\\xi\_\{t\}=\-\\eta\_\{t\}\\Proj\_\{X\_\{t\}\}\(D\_\{t\}\), wherePXt:ℝn×r→TXt​St​\(n,r\)\\Proj\_\{X\_\{t\}\}\\colon\\mathbb\{R\}^\{n\\times r\}\\to\\mathrm\{T\}\_\{X\_\{t\}\}\\mathrm\{St\}\(n,r\)is the orthogonal projection onto the tangent space toSt⁡\(n,r\)\\mathrm\{St\}\(n,r\)atXtX\_\{t\},PX⁡\(Z\)=X​skew​\(X⊤​Z\)\+\(I−X​X⊤\)​Z\\Proj\_\{X\}\(Z\)=X\\mathrm\{skew\}\(X^\{\\top\}\\\!Z\)\+\(I\-XX^\{\\top\}\)\\,Z, withskew⁡\(M\)=\(M−M⊤\)/2\\mathrm\{skew\}\(M\)=\(M\-M^\{\\top\}\)/2\. See[SectionA\.1](https://arxiv.org/html/2609.21039#A1.SS1)for more details on the geometry of the Stiefel manifold\.

### 3\.3\. Choice of Retraction

To map a tangent vectorξt\\xi\_\{t\}back onto the manifold, we need to apply a retraction mapping,Xt\+1=RetrXt⁡\(ξt\)X\_\{t\+1\}=\\Retraction\_\{X\_\{t\}\}\(\\xi\_\{t\}\)\. While the exponential map provides the most geometrically accurate geodesic path, it is often computationally prohibitive for large\-scale problems because it requires full eigenvalue decompositions or matrix exponentials\([Absil et al\., 2008](https://arxiv.org/html/2609.21039#bib.bib4)\)\. In practice, a retraction is any mapping that agrees to first order with the exponential map \(i\.e\., is centered at the point and has the differential at the origin equal to the identity map\)\.

For the Stiefel manifoldSt⁡\(n,r\)\\mathrm\{St\}\(n,r\), several efficient retractions exist with a computational complexity ofO⁡\(n​r2\+r3\)O\(nr^\{2\}\+r^\{3\}\), which is ideal for settings wherer≪nr\\ll n:

- •QR Decomposition:A standard choice that performs aQ​RQRfactorization ofXt\+DtX\_\{t\}\+D\_\{t\}and extracts the orthogonal factorQQ\([Absil et al\., 2008](https://arxiv.org/html/2609.21039#bib.bib4), \(4\.8\)\)\.
- •Polar Decomposition:Maps the tangent vector to the manifold by finding the closest orthogonal matrix in the Frobenius norm, typically implemented via iterative Newton–Schulz methods\([Zhu and Sato, 2020](https://arxiv.org/html/2609.21039#bib.bib64)\)\.
- •Cayley Transform:An algebraic alternative using a skew\-symmetric mapping\. When implemented with the Sherman–Morrison–Woodbury \(SMW\) identity, it avoids large matrix inversions, reducing the cost to a2​r×2​r2r\\times 2rsystem\([Wen and Yin, 2013](https://arxiv.org/html/2609.21039#bib.bib56), §2\.2\)\. It can also be computed implicitly via the fixed\-point iterationYk\+1=X\+α2​Ω​\(X\+Yk\)Y\_\{k\+1\}=X\+\\frac\{\\alpha\}\{2\}\\varOmega\\bigl\(X\+Y\_\{k\}\\bigr\), which converges quadratically aso⁡\(α2\+k\)o\(\\alpha^\{2\+k\}\)\.

In[Section3\.3](https://arxiv.org/html/2609.21039#S3.SS3)and in the right part of[Table3](https://arxiv.org/html/2609.21039#S5.T3), we present numerical results comparing different kinds of retractions\. In the remaining numerical experiments, we use the Cayley retraction as the standard choice, approximated via fixed\-point iteration, because of its simplicity of implementation and good performance in the comparison tests\. Moreover, our analysis shows that the framework is robust toapproximateretractions: as established in[Theorem4\.2](https://arxiv.org/html/2609.21039#S4.Thmtheorem2), the introduction of a maximal Frobenius errorδ\\deltain the retraction mapping merely adds a manageable linear term to the regret bound\. This theoretical guarantee justifies using truncated or iterative retraction methods that can run for only a few iterations without reaching machine precision, while offering significant speedups\.

## 4\. Theoretical Guarantees

### 4\.1\. Regret Analysis

In this section, we present a convex regret analysis for[Algorithm1](https://arxiv.org/html/2609.21039#alg1)\. Regret analysis is a standard tool in convex optimization that quantifies how much an optimization algorithm, when running dynamically on a family of convex objective functionsℒt\{\\mathcal\{L\}\}\_\{t\}, is suboptimal with respect to the optimal objective ahead of time\. In particular, given a sequence of iterates\{Wt\}t=1,…,T\\\{W\_\{t\}\\\}\_\{t=1,\\dots,T\}, we define the regret function as

R⁡\(T\)≔∑t=1Tℒt​\(Wt\)−min⁡∑t=1TW⁡ℒt​\(W\)\.R\(T\)\\coloneqq\\sum\_\{t=1\}^\{T\}\{\\mathcal\{L\}\}\_\{t\}\(W\_\{t\}\)\-\\min\_\{W\}\\sum\_\{t=1\}^\{T\}\{\\mathcal\{L\}\}\_\{t\}\(W\)\.We recall that an algorithm is said to be zero regret ifR⁡\(T\)/T→0R\(T\)/T\\to 0asT→\+∞T\\to\+\\infty\. Our aim is to show that[Algorithm1](https://arxiv.org/html/2609.21039#alg1)indeed produces arbitrarily small regret for a small enough learning rate and retraction error\. To prove this result, we will make the following assumptions:

###### Assumptions 4\.1\(Setting of regret analysis\)\.

1. \(H1\)The familyℒt:ℝm×r×St⁡\(n,r\)→ℝ\{\\mathcal\{L\}\}\_\{t\}\\colon\\mathbb\{R\}^\{m\\times r\}\\times\\mathrm\{St\}\(n,r\)\\to\\mathbb\{R\}is the restriction of a family of Euclidean strictly convex functions defined onℝr×m×ℝr×n\\mathbb\{R\}^\{r\\times m\}\\times\\mathbb\{R\}^\{r\\times n\}\. By a small abuse of notation, we will also denote the extension family withℒt\{\\mathcal\{L\}\}\_\{t\}, and we will denote the minimizer with\(B∗,A∗\)\(B^\{\*\},A^\{\*\}\)\.
2. \(H2\)The first momentum coefficientsβ1,t=β1​bt\\beta\_\{1,t\}=\\beta\_\{1\}b^\{t\}decrease geometrically in time for a constant0<b<10<b<1, and withβ1<β2\\beta\_\{1\}<\\sqrt\{\\beta\_\{2\}\}\.
3. \(H3\)The iteratesBtB\_\{t\}stay bounded, i\.e\.,supt‖Bt‖max≤D∞\\sup\_\{t\}\\\|B\_\{t\}\\\|\_\{\\max\}\\leq D\_\{\\infty\}\.
4. \(H4\)The Euclidean gradient∇ℒt​\(Bt,At\)\\nabla\{\\mathcal\{L\}\}\_\{t\}\(B\_\{t\},A\_\{t\}\)stays bounded, i\.e\.,supt‖∇ℒt​\(Bt,At\)‖max≤G∞\\sup\_\{t\}\\\|\\nabla\{\\mathcal\{L\}\}\_\{t\}\(B\_\{t\},A\_\{t\}\)\\\|\_\{\\max\}\\leq G\_\{\\infty\}\.
5. \(H5\)The second momentum update in[Algorithm1](https://arxiv.org/html/2609.21039#alg1)is followed by an entrywise maximum, i\.e\.,Vt\+1=max⁡\(β2​Vt−1\+\(1−β2\)​Gt2,Vt−1\),V\_\{t\+1\}=\\max\(\\beta\_\{2\}V\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\,G\_\{t\}^\{2\},\\ V\_\{t\-1\}\),as in AMSGrad\([Reddi et al\., 2018](https://arxiv.org/html/2609.21039#bib.bib11), Algorithm 2\)\.
6. \(H6\)The projected direction is aligned with the globally correct direction,⟨ξt,A∗−At⟩≥0\\langle\\xi\_\{t\},A^\{\*\}\-A\_\{t\}\\rangle\\geq 0\.

We emphasize that[Assumptions4\.1](https://arxiv.org/html/2609.21039#S4.Thmtheorem1)are fairly standard assumptions used to study convergence of Adam\-like algorithms, and they were already employed in, e\.g\.,[Reddi et al\. \(2018\)](https://arxiv.org/html/2609.21039#bib.bib11)\. Assumption \(H6\) is a hypothesis often used in Euclidean optimizers to ensure that the current local descent direction is aligned with the global direction to the minimizer\.

###### Theorem 4\.2\.

\(Regret bound\) Under[Assumptions4\.1](https://arxiv.org/html/2609.21039#S4.Thmtheorem1), consider the sequence of iterates produced by[Algorithm1](https://arxiv.org/html/2609.21039#alg1)with decreasing learning ratesηt=η/t\\eta\_\{t\}=\\eta/\\sqrt\{t\}, and no weight decay\. Then,

R⁡\(T\)\\displaystyle R\(T\)≤C1\+C2T\+C31\+log⁡T\+C4logT\+C5T−1/2,\\displaystyle\\leq C\_\{1\}\+C\_\{2\}\\sqrt\{T\}\+C\_\{3\}\\sqrt\{1\+\\log T\}\+C\_\{4\}\\log T\+C\_\{5\}T^\{\-1/2\},\(4\.1\)whereC1,C2,C3,C4,C5C\_\{1\},C\_\{2\},C\_\{3\},C\_\{4\},C\_\{5\}are constants independent ofTT\. In particular,limT→\+∞R⁡\(T\)/T=0\\lim\_\{T\\to\+\\infty\}R\(T\)/T=0\.

We note that, although the theoretical result requires an AMSGrad\-like assumption \(H5\), in practice the algorithm can be used without themax\\maxupdate with no loss in performance\. The proof of[Theorem4\.2](https://arxiv.org/html/2609.21039#S4.Thmtheorem2)can be found in[AppendixB](https://arxiv.org/html/2609.21039#A2)\.

In most practical implementations, the retraction is computed only approximately via a numerical algorithm, such as the fixed\-point method used in most of our experiments; see also[SectionA\.2\.2](https://arxiv.org/html/2609.21039#A1.SS2.SSS2)\. The proof of[Theorem4\.2](https://arxiv.org/html/2609.21039#S4.Thmtheorem2)above extends straightforwardly to that case: assuming that the computed retraction has an error ofδ\>0\\delta\>0, then a termC6​δC\_\{6\}\\deltahas to be added to \([4\.1](https://arxiv.org/html/2609.21039#S4.E1)\), without significantly affecting the main result of the theorem\.

### 4\.2\. Gradient Boundedness and Stability to Large Learning Rates

Working with an orthonormal factor yields a method with bounded gradients, potentially improving stability at large learning rates\. Here, we make this point more concrete with an example\. Let us consider the rank\-rrrecovery problem inℝn×n\\mathbb\{R\}^\{n\\times n\},

ℒ⁡\(A,B\)=12​‖B​A−α​In‖F2,B∈ℝn×r,A∈ℝr×n,\{\\mathcal\{L\}\}\(A,B\)=\\tfrac\{1\}\{2\}\\\|BA\-\\alpha I\_\{n\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\\qquad B\\in\\mathbb\{R\}^\{n\\times r\},\\ A\\in\\mathbb\{R\}^\{r\\times n\},trained by plain gradient descent from the canonical LoRA initializationB0=0B\_\{0\}=0,A0A\_\{0\}arbitrary\. The first GD step givesB1=η​α​A0⊤B\_\{1\}=\\eta\\alpha A\_\{0\}^\{\\top\}andA1=A0A\_\{1\}=A\_\{0\}, so for the scalingα=1/η\\alpha=1/\\etathe two factors align already after one step\. Specifically, we haveB1=A1⊤B\_\{1\}=A\_\{1\}^\{\\top\}and a direct induction argument shows that the iterates preserveBk=Ak⊤B\_\{k\}=A\_\{k\}^\{\\top\}thereafter\. SettingXk≔Bk=Ak⊤X\_\{k\}\\coloneqq B\_\{k\}=A\_\{k\}^\{\\top\}, the dynamics take the formXk\+1=Xk−η⁡\(Xk​Xk⊤−η−1​I\)​Xk=\(2​I−η​Xk​Xk⊤\)​XkX\_\{k\+1\}=X\_\{k\}\-\\eta\(X\_\{k\}X\_\{k\}^\{\\top\}\-\\eta^\{\-1\}I\)X\_\{k\}=\(2I\-\\eta X\_\{k\}X\_\{k\}^\{\\top\}\)X\_\{k\}, thus, using the SVDXk=Uk​Σk​Vk⊤X\_\{k\}=U\_\{k\}\\varSigma\_\{k\}V\_\{k\}^\{\\top\}, the dynamics decouple across singular values into the scalar recursionσi,k\+1=σi,k​\(2−η​σi,k2\)\\sigma\_\{i,k\+1\}=\\sigma\_\{i,k\}\\bigl\(2\-\\eta\\,\\sigma\_\{i,k\}^\{2\}\\bigr\), which diverges as soon\|2−η​σi,k2\|\>1\|2\-\\eta\\,\\sigma\_\{i,k\}^\{2\}\|\>1, i\.e\., when someσi,k\>3/η\\sigma\_\{i,k\}\>\\sqrt\{3/\\eta\}\. The largest stable learning rate is therefore dictated by the largest singular value of the iterate, a property of the*parameterization*, not of the underlying optimization landscape\.

If instead the constraintAk​Ak⊤=IrA\_\{k\}A\_\{k\}^\{\\top\}=I\_\{r\}is enforced, the gradient descent update onBBsimplifies to the affine recursionBk\+1=\(1−η\)​Bk\+η​α​Ak⊤B\_\{k\+1\}\\;=\\;\(1\-\\eta\)\\,B\_\{k\}\+\\eta\\alpha\\,A\_\{k\}^\{\\top\}, which is bounded for every0<η<20<\\eta<2as‖Ak‖=1\\\|A\_\{k\}\\\|=1, regardless of singular\-value scale, removing the dependence of the stable learning rate on the iterate\. This phenomenon is well documented in the Riemannian optimization literature[Schotthöfer et al\. \(2025a\)](https://arxiv.org/html/2609.21039#bib.bib50), and we provide a more precise result in the next[Proposition4\.3](https://arxiv.org/html/2609.21039#S4.Thmtheorem3), whose proof can be found in[AppendixC](https://arxiv.org/html/2609.21039#A3)\.

###### Proposition 4\.3\(Gradient boundedness on fibers\)\.

Consider the mapsΦ:ℝm×r×ℝr×n→ℝm×n\\Phi\\colon\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}\\to\\mathbb\{R\}^\{m\\times n\}given byΦ⁡\(B,A\)=B​A\\Phi\(B,A\)=BAand the mapΦ~=Φ\|St⁡\(n,r\)×ℝm×r\\widetilde\{\\Phi\}=\\Phi\|\_\{\\mathrm\{St\}\(n,r\)\\times\\mathbb\{R\}^\{m\\times r\}\}\. LetW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}be a fixed matrix withrank⁡\(W\)≤r\\rank\(W\)\\leq rsuch that∇ℒ​\(W\)≠0\\nabla\{\\mathcal\{L\}\}\(W\)\\neq 0\. Letℱ≔Φ−1​\(W\)\\mathcal\{F\}\\coloneqq\\Phi^\{\-1\}\(W\)andℱ~≔Φ~−1​\(W\)\\widetilde\{\\mathcal\{F\}\}\\coloneqq\\widetilde\{\\Phi\}^\{\-1\}\(W\)\. Then,

‖∇\(ℒ∘Φ\)‖L∞​\(ℱ\)=\+∞,‖∇\(ℒ∘Φ~\)‖L∞​\(ℱ~\)<\+∞\.\\\|\\nabla\(\{\\mathcal\{L\}\}\\circ\\Phi\)\\\|\_\{L^\{\\infty\}\(\\mathcal\{F\}\)\}=\+\\infty,\\quad\\\|\\nabla\(\{\\mathcal\{L\}\}\\circ\\widetilde\{\\Phi\}\)\\\|\_\{L^\{\\infty\}\(\\widetilde\{\\mathcal\{F\}\}\)\}<\+\\infty\.

In particular, it is known that the boundedness of∇\(ℒ∘Φ\)\\nabla\(\{\\mathcal\{L\}\}\\circ\\Phi\)is closely related to the range of stable learning rates\. This suggests that optimization algorithms on the parameterizationΦ~\\widetilde\{\\Phi\}are more stable than the ones on the more redundant representationΦ\\Phi\.

## 5\. Numerical Experiments

In this section, to show the effectiveness and scalability of Stiefel\-AdamW, we present several numerical experiments for both fine\-tuning pretrained models with LoRA adapters\([Hu et al\., 2022](https://arxiv.org/html/2609.21039#bib.bib21)\)and LLM pretraining\. We compare against six baselines: standardAdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.21039#bib.bib31)\);Scaled AdamW\([Zhang and Pilanci, 2024](https://arxiv.org/html/2609.21039#bib.bib61)\), which introduces a coupled preconditioner accounting for the product structure;GeoLoRA\([Schotthöfer et al\., 2025b](https://arxiv.org/html/2609.21039#bib.bib51)\), which uses two Stiefel representations;LoRA\-RITE\([Yen et al\., 2025](https://arxiv.org/html/2609.21039#bib.bib58)\)andLoRA\-Pro\([Wang et al\., 2025](https://arxiv.org/html/2609.21039#bib.bib55)\), which modify the gradient structure to reduce sensitivity to the non\-uniqueness of the factorization; andCayley Adam\([Li et al\., 2020](https://arxiv.org/html/2609.21039#bib.bib29)\), which uses the Riemannian Adam variant proposed in\([Bécigneul and Ganea, 2019](https://arxiv.org/html/2609.21039#bib.bib5)\)on the Stiefel manifold\. We emphasize that the latter method uses a scalar preconditioner, whereas Stiefel\-AdamW uses a diagonal preconditioner\.

### 5\.1\. LoRA Fine\-Tuning

##### GPT2

In this experiment, we tested Stiefel\-AdamW for fine\-tuning GPT2 on the E2E Natural Language Generation challenge\([Novikova et al\., 2017](https://arxiv.org/html/2609.21039#bib.bib38)\)with LoRA of rank 4\. We report the results in[Table1](https://arxiv.org/html/2609.21039#S5.T1)\. In all experiments, we trained the models for55epochs with a batch size of88\. For details on the hyperparameter settings, see[Table5](https://arxiv.org/html/2609.21039#A4.T5)\. As shown in[Table3](https://arxiv.org/html/2609.21039#S5.T3), Stiefel\-AdamW outperforms all baselines across all tasks except ROUGE\-L\. Interestingly, we observe that, consistently with the findings of[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61)for Scaled AdamW, Stiefel\-AdamW achieves better performance when using more aggressive moving average parametersβ1\\beta\_\{1\},β2\\beta\_\{2\}\. This suggests that part of AdamW’s update may be spent along invariant directions, while Riemannian\-informed approaches avoid this and allow greater emphasis to be placed on the current gradient direction\. All numerical experiments were performed on a single NVIDIA A100 80GB, except for GPT2 pretraining, which was performed on two NVIDIA H100 80GB via Modal\.

Table 1:Fine\-tuning performance with low\-rank adapters\. Best results highlighted in bold\.Left:GPT2 on the E2E Natural Language Generation challenge, with rank = 4\. AdamW and Scaled AdamW are reported from[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61)\.Right:ViT\-Base on CIFAR\-10, for three different choices of adapter rank\. We report with±σ\\pm\\sigmathe standard deviation over55random initializations\.
##### Vision Transformers

In this experiment, we fine\-tuned the base Vision Transformer from[Dosovitskiy et al\. \(2021\)](https://arxiv.org/html/2609.21039#bib.bib12)on CIFAR\-10[Krizhevsky and Hinton \(2009\)](https://arxiv.org/html/2609.21039#bib.bib28), with results shown in the right part of[Table1](https://arxiv.org/html/2609.21039#S5.T1)\. In[Figure1](https://arxiv.org/html/2609.21039#S5.F1), we compare loss descent and time per iteration against the best loss achieved\. All models have been trained for5050epochs with a batch size of6464, LoRA alpha3232, learning rate10−310^\{\-3\}, and no scheduler\. For all optimizers, we used weight decay of10−510^\{\-5\}on all adapters, applied to the key\-query attention matrices, attention projection, and the last two fully connected layers\. We did not optimize biases and left them as in the pretrained model\. As we can observe from the results in the right panel of[Table3](https://arxiv.org/html/2609.21039#S5.T3)and[Figure1](https://arxiv.org/html/2609.21039#S5.F1), Stiefel\-AdamW is able to outperform all baselines in terms of performance, with convergence speed comparable to that of Scaled AdamW\([Zhang and Pilanci, 2024](https://arxiv.org/html/2609.21039#bib.bib61)\)and AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.21039#bib.bib31)\)\.

Figure 1:Left and center:Loss function descent for ViT Base on CIFAR\-10 LoRA fine\-tuning \(LoRA rank6464and learning rate10−310^\{\-3\}\)\.Right:Learning rate versus best training loss for different optimizers when fine\-tuning ViT Base on CIFAR\-10\. In this experiment, we fixed the LoRA rank at3232and trained all models for5050epochs\.
##### Mistral 7B

In this experiment, we tested the effectiveness of Stiefel\-AdamW for fine\-tuning Mistral 7B\([Jiang et al\., 2023](https://arxiv.org/html/2609.21039#bib.bib22)\)on the GLUE benchmark\([Wang et al\., 2019](https://arxiv.org/html/2609.21039#bib.bib54)\)for natural language understanding, following the implementation in\([Zhang and Pilanci, 2024](https://arxiv.org/html/2609.21039#bib.bib61)\)\. LoRA adapters of rank1616have been applied to all query, key, value projection, and gate matrices of multihead attention\. We did not train biases, and for all optimizers, we set the LoRA alpha learning\-rate scaling parameter to1616\. We used mixed precision for all optimizers: the base model was loaded in its44\-bit quantized version, optimizer states were in float3232, and operations were performed in mixed\-precision bfloat16\. We trained all models using the codebase of[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61), with a dropout of0\.10\.1and a batch size of88across all models\. For AdamW, Scaled AdamW, and GeoLoRA, we used the optimal hyperparameters \(learning rate andβ1,β2\\beta\_\{1\},\\beta\_\{2\}\) from[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61); for Stiefel\-AdamW, we usedβ1=β2=0\.95\\beta\_\{1\}=\\beta\_\{2\}=0\.95\. For each GLUE task, we report the value of the standard test metric \(either accuracy or correlation\) and the percentage deviation from the best performer among all optimizers\. As shown in[Table2](https://arxiv.org/html/2609.21039#S5.T2), Stiefel\-AdamW outperforms all baselines in terms of average score\. On single tasks, Stiefel\-AdamW outperforms all baselines on MNLI, MRPC, STS\-B, and WNLI, while maintaining a competitive performance on all other tasks\.

Table 2:Scores for rank1616LoRA fine\-tuning of the 4\-bit quantized Mistral 7B model on the GLUE benchmark for Natural Language Understanding \(NLU\) challenges with different optimizers \(best results highlighted in bold\)\. In parentheses, we report, for each task, the percentage deviation from the best performer\. SGD, Scaled GD, and Scaled AdamW results reported from\([Zhang and Pilanci, 2024](https://arxiv.org/html/2609.21039#bib.bib61), Table 2\)\.

### 5\.2\. GPT2 Pretraining

As the proposed method works for all problems in which the manifold ofrank\\rank\-rrmatrices appears, it also applies directly to pretraining transformer\-based architectures\. In particular, self\-attention naturally respects this structure, as the image of the map\(WQ,WK\)∈ℝn×r×ℝn×r↦WQ​WK⊤\(W\_\{Q\},W\_\{K\}\)\\in\\mathbb\{R\}^\{n\\times r\}\\times\\mathbb\{R\}^\{n\\times r\}\\mapsto W\_\{Q\}W\_\{K\}^\{\\top\}is exactly the setℳr\{\\mathcal\{M\}\}\_\{r\}\. Despite this, the parametrization map is highly non\-injective, and therefore the problem could be restated equivalently by minimizing onℳr≅St⁡\(n,r\)×ℝ∗n×r/O⁡\(r\)\{\\mathcal\{M\}\}\_\{r\}\\cong\\mathrm\{St\}\(n,r\)\\times\\mathbb\{R\}^\{n\\times r\}\_\{\*\}/\\mathrm\{O\}\(r\)instead ofℝn×r×ℝn×r\\mathbb\{R\}^\{n\\times r\}\\times\\mathbb\{R\}^\{n\\times r\}\. In contrast, we optimize all other parameters, such as biases or non\-structured matrices, with the standard AdamW step\. In[Table3](https://arxiv.org/html/2609.21039#S5.T3)we present the results for pretraining GPT2\([Radford et al\., 2019](https://arxiv.org/html/2609.21039#bib.bib40)\)on OpenWebText\([Gokaslan and Cohen, 2019](https://arxiv.org/html/2609.21039#bib.bib16)\)using Karpathy’s reproduction111https://github\.com/karpathy/nanoGPT\. We reproduced the pretraining for Stiefel\-AdamW using exactly the same AdamW hyperparameters from the repository\. We trained both AdamW and Stiefel\-AdamW for70007000iterations\. As shown, Stiefel\-AdamW produces results comparable to standard AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.21039#bib.bib31)\)in both performance and peak GPU memory usage\. We performed no hyperparameter tuning, and Stiefel\-AdamW uses the same hyperparameters as AdamW, as shown in the reproduced repository\. This experiment highlights that Stiefel\-AdamW is not limited to fine\-tuning scenarios such as LoRA, but can be applied directly in pretraining settings where low\-rank structure arises naturally\.

Table 3:Left: Full GPT2 pretraining on OpenWeb Text for 6000 iterations\.Right:Ablation over different retractions on ViT Base\.
### 5\.3\. Qwen2 Pretraining

To showcase compatibility of our proposed approach with the presence of positional embeddings such as RoPE[Su et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib67), in[Table4](https://arxiv.org/html/2609.21039#S5.T4)we also present numerical results for Qwen2[Yang et al\. \(2024\)](https://arxiv.org/html/2609.21039#bib.bib66)LoRA pretraining on WikiText\-103[Merity et al\. \(2016\)](https://arxiv.org/html/2609.21039#bib.bib65)\. The model with the adapters has 100M parameters; it was trained for 20K steps with a batch size of1616and two steps of gradient accumulation\. As shown in[Table4](https://arxiv.org/html/2609.21039#S5.T4), Stiefel\-AdamW consistently outperforms the Euclidean version of AdamW in both mean and variance\.

Table 4:Qwen2 pretraining on Wikitext\-103, standard deviation reported over55random seeds\.
### 5\.4\. Stepsize Stability

Motivated by[Proposition4\.3](https://arxiv.org/html/2609.21039#S4.Thmtheorem3), we show the stability of Stiefel\-AdamW with respect to the learning rate, numerically demonstrating that methods with compact fibers are more stable with respect to learning\-rate size\. We fine\-tuned multiple vision transformers \(ViT Base\) on CIFAR\-10 for different learning rates, keeping all other hyperparameters fixed as in[Section5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px2)\. In the right panel of[Figure1](https://arxiv.org/html/2609.21039#S5.F1), we plot the learning rate against the best loss obtained during training\. As expected, the results in[Figure1](https://arxiv.org/html/2609.21039#S5.F1)show that “pure” Riemannian methods such as GeoLoRA[Schotthöfer et al\. \(2025b\)](https://arxiv.org/html/2609.21039#bib.bib51)and the proposed Stiefel\-AdamW are more stable with respect to learning\-rate magnitude\. In particular, Riemannian methods in which the fiber ofΦ\\Phiis not compact \(such as AdamW and Scaled AdamW, which are defined inℝr×n×ℝm×r\\mathbb\{R\}^\{r\\times n\}\\times\\mathbb\{R\}^\{m\\times r\}\), appear to be less stable with respect to larger learning rates, despite the preconditioning \(i\.e\., different metric\) employed in Scaled AdamW[Zhang and Pilanci \(2024\)](https://arxiv.org/html/2609.21039#bib.bib61)\.

## 6\. Conclusions, Limitations, and Future Work

In this work, we presented Stiefel\-AdamW, a stochastic Riemannian variant of AdamW that naturally provides convergence guarantees\. The compactness of the space in which one factor lives helps avoid potential numerical instabilities that can arise from unbalanced initializations\. One key advantage of the proposed method is its simplicity of implementation, which requires only minimal machinery from Riemannian optimization theory while maintaining guarantees\. We demonstrated the method’s effectiveness and scalability across a range of problems, from fine\-tuning to pretraining LLMs\. One limitation of the current method is that it still retains a set of orthogonal invariances, which stems from the entrywise nature of the Adam algorithm\. While this can be solved by imposing a gauge condition and working on horizontal spaces on the quotient spaceℳr\{\\mathcal\{M\}\}\_\{r\}, it requires additional computational effort, as the gradient needs to be computed using joint information on the pair\(B,A\)\(B,A\),\(∇Bℒ,∇Aℒ\)\(\\nabla\_\{B\}\{\\mathcal\{L\}\},\\nabla\_\{A\}\{\\mathcal\{L\}\}\), and therefore does not allow for full parallelization\. Future research could extend the current method to different preconditioners and propose a similarly simple version that is fully invariant on the manifold of fixed\-rank matrices and can partially mitigate this extra computational effort\.

## References

- \[1\]P\.\-A\. Absil, C\. G\. Baker, and K\. A\. Gallivan\(2007\)Trust\-region methods on Riemannian manifolds\.Found\. of Comput\. Math\.7,pp\. 303–330\.External Links:[Document](https://dx.doi.org/10.1007/s10208-005-0179-9),ISSN 1572\-9036Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[2\]P\.\-A\. Absil, R\. Mahony, and R\. Sepulchre\(2008\)Optimization Algorithms on Matrix Manifolds\.Princeton University Press,Princeton, NJ\.External Links:ISBN 978\-0\-691\-13298\-3Cited by:[§A\.1\.1](https://arxiv.org/html/2609.21039#A1.SS1.SSS1.p2.2),[§A\.2\.1](https://arxiv.org/html/2609.21039#A1.SS2.SSS1.p2.3),[§A\.2](https://arxiv.org/html/2609.21039#A1.SS2.p2.3),[§2](https://arxiv.org/html/2609.21039#S2.p1.1),[1st item](https://arxiv.org/html/2609.21039#S3.I1.i1.p1.1),[§3\.3](https://arxiv.org/html/2609.21039#S3.SS3.p1.1)\.
- \[3\]P\.\-A\. Absil and J\. Malick\(2012\)Projection\-like Retractions on Matrix Manifolds\.SIAM J\. Optim\.22\(1\),pp\. 135–158\.External Links:[Document](https://dx.doi.org/10.1137/100802529)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[4\]P\.\-A\. Absil and I\. V\. Oseledets\(2015\)Low\-rank retractions: a survey and new results\.Computational Optimization and Applications62\(1\),pp\. 5–29\.External Links:[Document](https://dx.doi.org/10.1007/s10589-014-9714-4),ISSN 1573\-2894,[Link](https://doi.org/10.1007/s10589-014-9714-4)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[5\]G\. Bécigneul and O\. Ganea\(2019\)Riemannian adaptive optimization methods\.InInternational Conference on Learning Representations \(ICLR 2019\),Vol\.9,pp\. 6384–6399\.Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p4.2),[§2](https://arxiv.org/html/2609.21039#S2.p2.1),[§2](https://arxiv.org/html/2609.21039#S2.p4.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[6\]F\. Bian, J\. Zheng, Z\. Liu, J\. Luo, and J\. Cai\(2025\)Finding Low\-Rank Matrix Weights in DNNs via Riemannian Optimization: RAdaGrad and RAdamW\.InThe 39th Annual Conference on Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://openreview.net/forum?id=tiGFiCrmKm)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p4.2),[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[7\]Å\. Björck and C\. Bowie\(1971\)An Iterative Algorithm for Computing the Best Estimate of an Orthogonal Matrix\.SIAM Journal on Numerical Analysis8\(2\),pp\. 358–364\.External Links:ISSN 00361429,[Link](http://www.jstor.org/stable/2949484)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p6.1)\.
- \[8\]S\. Bonnabel\(2013\)Stochastic Gradient Descent on Riemannian Manifolds\.IEEE Transactions on Automatic Control58\(9\),pp\. 2217–2229\.External Links:[Document](https://dx.doi.org/10.1109/TAC.2013.2254619)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[9\]N\. Boumal and P\.\-A\. Absil\(2015\)Low\-rank matrix completion via preconditioned optimization on the Grassmann manifold\.Linear Algebra and its Applications475,pp\. 200–239\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.laa.2015.02.027),ISSN 0024\-3795,[Link](https://www.sciencedirect.com/science/article/pii/S0024379515001342)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[10\]N\. Boumal\(2023\)An Introduction to Optimization on Smooth Manifolds\.Cambridge University Press\.External Links:[Document](https://dx.doi.org/10.1017/9781009166164)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[11\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YicbFdNTTy)Cited by:[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px2.p1.1)\.
- \[12\]J\. Duchi, E\. Hazan, and Y\. Singer\(2011\)Adaptive Subgradient Methods for Online Learning and Stochastic Optimization\.Journal of Machine Learning Research12\(61\),pp\. 2121–2159\.External Links:[Link](http://jmlr.org/papers/v12/duchi11a.html)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[13\]A\. Edelman, T\. A\. Arias, and S\. T\. Smith\(1998\)The geometry of algorithms with orthogonality constraints\.SIAM J\. Matrix Anal\. Appl\.20\(2\),pp\. 303–353\.External Links:[Document](https://dx.doi.org/10.1137/S0895479895290954),[Link](https://doi.org/10.1137/S0895479895290954),https://doi\.org/10\.1137/S0895479895290954Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[14\]D\. Gabay and D\. G\. Luenberger\(1976\)Efficiently Converging Minimization Methods Based on the Reduced Gradient\.SIAM Journal on Control and Optimization14\(1\),pp\. 42–61\.External Links:[Document](https://dx.doi.org/10.1137/0314004),[Link](https://doi.org/10.1137/0314004),https://doi\.org/10\.1137/0314004Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[15\]A\. Gokaslan and V\. Cohen\(2019\)OpenWebText Corpus\.Note:[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§5\.2](https://arxiv.org/html/2609.21039#S5.SS2.p1.1)\.
- \[16\]S\. Hayou, N\. Ghosh, and B\. Yu\(2024\)LoRA\+: Efficient low rank adaptation of large models\.InProceedings of the 41st International Conference on Machine Learning,pp\. 17783–17806\.Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[17\]N\. J\. Higham\(2008\)Functions of matrices\.Society for Industrial and Applied Mathematics\.External Links:[Document](https://dx.doi.org/10.1137/1.9780898717778),[Link](https://epubs.siam.org/doi/abs/10.1137/1.9780898717778),https://epubs\.siam\.org/doi/pdf/10\.1137/1\.9780898717778Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p6.1)\.
- \[18\]G\. Hinton, N\. Srivastava, and K\. Swersky\(2012\)Neural Networks for Machine Learning Lecture 6a Overview of mini\-batch gradient descent\.Note:[https://www\.cs\.toronto\.edu/~tijmen/csc321/slides/lecture\_slides\_lec6\.pdf](https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[19\]A\. Hnatiuk, J\. Kusch, L\. Kusch, N\. R\. Gauger, and A\. Walther\(2026\)Stochastic Dynamical Low\-Rank Approximation in the Context of Machine Learning\.Journal of Optimization Theory and Applications208\(1\),pp\. 1–33\.External Links:[Document](https://dx.doi.org/10.1007/s10957-025-02892-1),[Link](https://ideas.repec.org/a/spr/joptap/v208y2026i1d10.1007_s10957-025-02892-1.html)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[20\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: Low\-Rank Adaptation of Large Language Models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[21\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7B\.External Links:[Link](https://arxiv.org/abs/2310.06825),2310\.06825Cited by:[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px3.p1.1)\.
- \[22\]H\. Kasai, P\. Jawanpuria, and B\. Mishra\(2019\)Riemannian adaptive stochastic gradient algorithms on matrix manifolds\.InInternational conference on machine learning,pp\. 3262–3271\.Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[23\]H\. Kasai, H\. Sato, and B\. Mishra\(2018\)Riemannian Stochastic Recursive Gradient Algorithm\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 2516–2524\.External Links:[Link](https://proceedings.mlr.press/v80/kasai18a.html)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[24\]M\. Khodak, N\. A\. Tenenholtz, L\. Mackey, and N\. Fusi\(2021\)Initialization and Regularization of Factorized Neural Layers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=KTlJT1nof6d)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1)\.
- \[25\]D\. P\. Kingma and J\. Ba\(2015\)Adam: A Method for Stochastic Optimization\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1412.6980)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[26\]O\. Koch and C\. Lubich\(2007\)Dynamical Low‐Rank Approximation\.SIAM Journal on Matrix Analysis and Applications29\(2\),pp\. 434–454\.External Links:[Document](https://dx.doi.org/10.1137/050639703),[Link](https://doi.org/10.1137/050639703),https://doi\.org/10\.1137/050639703Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[27\]Z\. V\. Kovarik\(1970\)Some Iterative Methods for Improving Orthonormality\.SIAM Journal on Numerical Analysis7,pp\. 386–389\.External Links:[Link](https://api.semanticscholar.org/CorpusID:121623483)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p6.1)\.
- \[28\]A\. Krizhevsky and G\. Hinton\(2009\)Learning multiple layers of features from tiny images\.Technical reportTechnical Report0,Technical report, University of Toronto,University of Toronto,Toronto, Ontario\.External Links:[Link](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf)Cited by:[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px2.p1.1)\.
- \[29\]J\. Li, F\. Li, and S\. Todorovic\(2020\)Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2002.01113),2002\.01113Cited by:[§A\.2\.1](https://arxiv.org/html/2609.21039#A1.SS2.SSS1.p2.1),[§2](https://arxiv.org/html/2609.21039#S2.p4.1),[Table 1](https://arxiv.org/html/2609.21039#S5.T1.6.1.9.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[30\]V\. Lialin, S\. Muckatira, N\. Shivagunde, and A\. Rumshisky\(2024\)ReLoRA: High\-Rank Training Through Low\-Rank Updates\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=DLJznSp6X3)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1)\.
- \[31\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled Weight Decay Regularization\.External Links:[Link](https://arxiv.org/abs/1711.05101),1711\.05101Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p3.1),[§2](https://arxiv.org/html/2609.21039#S2.p2.1),[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2609.21039#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.21039#S5.T1.6.1.3.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[32\]D\. G\. Luenberger\(1972\)The Gradient Projection Method along Geodesics\.Manage\. Sci\.18\(11\),pp\. 620–631\.External Links:ISSN 00251909, 15265501,[Link](http://www.jstor.org/stable/2629156)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[33\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2016\)Pointer sentinel mixture models\.External Links:1609\.07843Cited by:[§5\.3](https://arxiv.org/html/2609.21039#S5.SS3.p1.1)\.
- \[34\]B\. Mishra, G\. Meyer, S\. Bonnabel, and R\. Sepulchre\(2014\)Fixed\-rank matrix factorizations and Riemannian low\-rank optimization\.Computational Statistics29\(3\),pp\. 591–621\.External Links:[Document](https://dx.doi.org/10.1007/s00180-013-0464-z),ISSN 1613\-9658,[Link](https://doi.org/10.1007/s00180-013-0464-z)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p2.1),[§1](https://arxiv.org/html/2609.21039#S1.p4.2),[§2](https://arxiv.org/html/2609.21039#S2.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[35\]Z\. Mo, L\. Huang, and S\. J\. Pan\(2025\)Parameter and Memory Efficient Pretraining via Low\-rank Riemannian Optimization\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=i0zzO7Hslk)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1)\.
- \[36\]J\. Novikova, O\. Dušek, and V\. Rieser\(2017\)The E2E Dataset: New Challenges for End\-to\-End Generation\.InProceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue,Saarbrücken, Germany\.Note:arXiv:1706\.09254External Links:[Link](https://arxiv.org/abs/1706.09254)Cited by:[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px1.p1.1)\.
- \[37\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language Models are Unsupervised Multitask Learners\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§5\.2](https://arxiv.org/html/2609.21039#S5.SS2.p1.1)\.
- \[38\]S\. J\. Reddi, S\. Kale, and S\. Kumar\(2018\)On the Convergence of Adam and Beyond\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ryQu7f-RZ)Cited by:[Appendix B](https://arxiv.org/html/2609.21039#A2.p2.1.1),[Appendix B](https://arxiv.org/html/2609.21039#A2.p3.3.1),[Appendix B](https://arxiv.org/html/2609.21039#A2.p3.4.1),[§2](https://arxiv.org/html/2609.21039#S2.p2.1),[item \(H5\)](https://arxiv.org/html/2609.21039#S4.I1.ix5.p1.1),[§4\.1](https://arxiv.org/html/2609.21039#S4.SS1.p2.1)\.
- \[39\]W\. Ring and B\. Wirth\(2012\)Optimization Methods on Riemannian Manifolds and Their Application to Shape Space\.SIAM J\. Optim\.22\(2\),pp\. 596–627\.External Links:[Document](https://dx.doi.org/10.1137/11082885X),[Link](https://doi.org/10.1137/11082885X),https://doi\.org/10\.1137/11082885XCited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[40\]R\. Saha, V\. Srivastava, and M\. Pilanci\(2023\)Matrix Compression via Randomized Low Rank and Low Precision Factorization\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=rxsCTtkqA9)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[41\]H\. Sakai and H\. Iiduka\(2022\)Riemannian Adaptive Optimization Algorithm and its Application to Natural Language Processing\.IEEE Transactions on Cybernetics52\(8\),pp\. 7328–7339\.External Links:[Document](https://dx.doi.org/10.1109/TCYB.2021.3049845)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[42\]H\. Sakai and H\. Iiduka\(2025\)A general framework of Riemannian adaptive optimization methods with a convergence analysis\.Transactions on Machine Learning Research,pp\. n/a\.Note:Reproducibility CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=knv4lQFVoE)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p4.2),[§2](https://arxiv.org/html/2609.21039#S2.p2.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[43\]H\. Sato and T\. Iwai\(2015\)A new, globally convergent Riemannian conjugate gradient method\.Optimization64\(4\),pp\. 1011–1031\.External Links:[Document](https://dx.doi.org/10.1080/02331934.2013.836650),[Link](https://doi.org/10.1080/02331934.2013.836650)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[44\]H\. Sato\(2016\)A Dai–Yuan\-type Riemannian conjugate gradient method with the weak Wolfe conditions\.Comput\. Optim\. Appl\.64\(1\),pp\. 101–118\.External Links:[Document](https://dx.doi.org/10.1007/s10589-015-9801-1),ISSN 1573\-2894,[Link](https://doi.org/10.1007/s10589-015-9801-1)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[45\]H\. Sato\(2021\)Riemannian Optimization and Its Applications\.Springer International Publishing\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1007/978-3-030-62391-3),ISBN 978\-3\-030\-62389\-0Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[46\]H\. Sato\(2022\)Riemannian Conjugate Gradient Methods: General Framework and Specific Algorithms with Convergence Analyses\.SIAM J\. Optim\.32\(4\),pp\. 2690–2717\.External Links:[Document](https://dx.doi.org/10.1137/21M1464178),[Link](https://doi.org/10.1137/21M1464178),https://doi\.org/10\.1137/21M1464178Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[47\]S\. Schotthöfer, T\. Klein, and J\. Kusch\(2025\)A geometric framework for momentum\-based optimizers for low\-rank training\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=cCefuzQrjK)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§1](https://arxiv.org/html/2609.21039#S1.p4.2),[§2](https://arxiv.org/html/2609.21039#S2.p3.1),[§4\.2](https://arxiv.org/html/2609.21039#S4.SS2.p2.1),[Table 1](https://arxiv.org/html/2609.21039#S5.T1.6.1.6.1)\.
- \[48\]S\. Schotthöfer, E\. Zangrando, G\. Ceruti, F\. Tudisco, and J\. Kusch\(2025\)GeoLoRA: Geometric integration for parameter efficient fine\-tuning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=bsFWJ0Kget)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1),[§2](https://arxiv.org/html/2609.21039#S2.p4.1),[§5\.4](https://arxiv.org/html/2609.21039#S5.SS4.p1.1),[Table 2](https://arxiv.org/html/2609.21039#S5.T2.2.1.13.1.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[49\]S\. Schotthöfer, E\. Zangrando, J\. Kusch, G\. Ceruti, and F\. Tudisco\(2022\)Low\-rank lottery tickets: finding efficient low\-rank neural networks via matrix differential equations\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA,pp\. 20051–20063\.External Links:ISBN 9781713871088Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§1](https://arxiv.org/html/2609.21039#S1.p2.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[50\]J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu\(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.External Links:ISSN 0925\-2312,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.neucom.2023.127063),[Link](https://www.sciencedirect.com/science/article/pii/S0925231223011864)Cited by:[§5\.3](https://arxiv.org/html/2609.21039#S5.SS3.p1.1)\.
- \[51\]B\. Vandereycken and S\. Vandewalle\(2010\)A Riemannian Optimization Approach for Computing Low\-Rank Solutions of Lyapunov Equations\.SIAM Journal on Matrix Analysis and Applications31\(5\),pp\. 2553–2579\.External Links:[Document](https://dx.doi.org/10.1137/090764566),[Link](https://doi.org/10.1137/090764566),https://doi\.org/10\.1137/090764566Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[52\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is All You Need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[53\]T\. Vogels, S\. P\. Karimireddy, and M\. Jaggi\(2019\)PowerSGD: Practical Low\-Rank Gradient Compression for Distributed Optimization\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/d9fbed9da256e344c1fa46bb46c34c5f-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[54\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2019\)GLUE: A Multi\-Task Benchmark and Analysis Platform for Natural Language Understanding\.External Links:[Link](https://arxiv.org/abs/1804.07461),1804\.07461Cited by:[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px3.p1.1)\.
- \[55\]H\. Wang, S\. Agarwal, and D\. Papailiopoulos\(2021\)Pufferfish: Communication\-efficient Models At No Extra Cost\.InProceedings of Machine Learning and Systems,A\. Smola, A\. Dimakis, and I\. Stoica \(Eds\.\),Vol\.3,pp\. 365–386\.External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2021/file/94cb28874a503f34b3c4a41bddcea2bd-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1)\.
- \[56\]Z\. Wang, J\. Liang, R\. He, Z\. Wang, and T\. Tan\(2025\)LoRA\-Pro: Are Low\-Rank Adapters Properly Optimized?\.InInternational Conference on Representation Learning,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),pp\. 93787–93808\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/ea184f920a0f0f8d8030aa1bd7ac9fd4-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p3.1),[Table 1](https://arxiv.org/html/2609.21039#S5.T1.6.1.8.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[57\]Z\. Wen and W\. Yin\(2013\)A feasible method for optimization with orthogonality constraints\.Mathematical Programming142\(1\),pp\. 397–434\.External Links:[Document](https://dx.doi.org/10.1007/s10107-012-0584-1),ISSN 1436\-4646,[Link](https://doi.org/10.1007/s10107-012-0584-1)Cited by:[3rd item](https://arxiv.org/html/2609.21039#S3.I1.i3.p1.1)\.
- \[58\]A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Yang, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, X\. Liu, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Guo, and Z\. Fan\(2024\)Qwen2 technical report\.External Links:2407\.10671,[Link](https://arxiv.org/abs/2407.10671)Cited by:[§5\.3](https://arxiv.org/html/2609.21039#S5.SS3.p1.1)\.
- \[59\]Y\. Yang\(2007\)Globally Convergent Optimization Algorithms on Riemannian Manifolds: Uniform Framework for Unconstrained and Constrained Optimization\.Journal of Optimization Theory and Applications132\(2\),pp\. 245–265\.External Links:[Document](https://dx.doi.org/10.1007/s10957-006-9081-0),ISSN 1573\-2878,[Link](https://doi.org/10.1007/s10957-006-9081-0)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p1.1)\.
- \[60\]J\. Yen, S\. Si, Z\. Meng, F\. Yu, S\. S\. Duvvuri, I\. S\. Dhillon, C\. Hsieh, and S\. Kumar\(2025\)LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VpWki1v2P8)Cited by:[Table 1](https://arxiv.org/html/2609.21039#S5.T1.6.1.7.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[61\]E\. Zangrando, S\. Schotthöfer, J\. Kusch, G\. Ceruti, and F\. Tudisco\(2024\)Geometry\-aware training of factorized layers in tensor Tucker format\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p2.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.
- \[62\]F\. Zhang and M\. Pilanci\(2024\)Riemannian preconditioned LoRA for fine\-tuning foundation models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24,pp\. 59641–59669\.Cited by:[§D\.1](https://arxiv.org/html/2609.21039#A4.SS1.p1.1),[§1](https://arxiv.org/html/2609.21039#S1.p1.1),[§2](https://arxiv.org/html/2609.21039#S2.p3.1),[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2609.21039#S5.SS1.SSS0.Px3.p1.1),[§5\.4](https://arxiv.org/html/2609.21039#S5.SS4.p1.1),[Table 1](https://arxiv.org/html/2609.21039#S5.T1),[Table 1](https://arxiv.org/html/2609.21039#S5.T1.6.1.4.1),[Table 2](https://arxiv.org/html/2609.21039#S5.T2),[Table 2](https://arxiv.org/html/2609.21039#S5.T2.2.1.5.1.1),[Table 2](https://arxiv.org/html/2609.21039#S5.T2.2.1.9.1.1),[§5](https://arxiv.org/html/2609.21039#S5.p1.1)\.
- \[63\]H\. Zhang, S\. J\. Reddi, and S\. Sra\(2016\)Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds\.InAdvances in Neural Information Processing Systems,D\. Lee, M\. Sugiyama, U\. Luxburg, I\. Guyon, and R\. Garnett \(Eds\.\),Vol\.29\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2016/file/98e6f17209029f4ae6dc9d88ec8eac2c-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p2.1)\.
- \[64\]Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)AdaLoRA: Adaptive Budget Allocation for Parameter\-Efficient Fine\-Tuning\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lq62uWRJjiY)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1)\.
- \[65\]J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian\(2024\)GaLore: Memory\-Efficient LLM Training by Gradient Low\-Rank Projection\.In5th Workshop on practical ML for limited/low resource settings,External Links:[Link](https://openreview.net/forum?id=AzqPyO22zt)Cited by:[§1](https://arxiv.org/html/2609.21039#S1.p1.1)\.
- \[66\]X\. Zhu and H\. Sato\(2020\)Riemannian conjugate gradient methods with inverse retraction\.Computational Optimization and Applications77\(3\),pp\. 779–810\.External Links:[Document](https://dx.doi.org/10.1007/s10589-020-00219-6),ISSN 1573\-2894,[Link](https://doi.org/10.1007/s10589-020-00219-6)Cited by:[2nd item](https://arxiv.org/html/2609.21039#S3.I1.i2.p1.1)\.
- \[67\]Z\. Zhu, Y\. Wu, Q\. Gu, and V\. Cevher\(2024\)Imbalance\-Regularized LoRA: A Plug\-and\-Play Method for Improving Fine\-Tuning of Foundation Models\.InAdaptive Foundation Models: Evolving AI for Personalized and Efficient Learning,External Links:[Link](https://openreview.net/forum?id=hQfJ7nyDeK)Cited by:[§2](https://arxiv.org/html/2609.21039#S2.p3.1)\.

## Appendix AAppendices

### A\.1\. Geometry of the Stiefel Manifold

The \(column\-orthonormal\) Stiefel manifold is defined as

St⁡\(n,r\)≔\{X∈ℝn×r:X⊤​X=Ir\}\.\\mathrm\{St\}\(n,r\)\\coloneqq\\\{X\\in\\mathbb\{R\}^\{n\\times r\}\\colon X^\{\\top\}\\\!X=I\_\{r\}\\\}\.It is a smooth embedded submanifold ofℝn×r\\mathbb\{R\}^\{n\\times r\}of dimensionn​r−12​r​\(r\+1\)nr\-\\tfrac\{1\}\{2\}r\(r\+1\)\.

The tangent space at a pointX∈St⁡\(n,r\)X\\in\\mathrm\{St\}\(n,r\)is given by

TX​St​\(n,r\)=\{ξ∈ℝn×r:X⊤​ξ\+ξ⊤​X=0\}\.\\mathrm\{T\}\_\{X\}\\mathrm\{St\}\(n,r\)=\\left\\\{\\xi\\in\\mathbb\{R\}^\{n\\times r\}\\colon X^\{\\top\}\\\!\\xi\+\\xi^\{\\top\}\\\!X=0\\right\\\}\.Equivalently, any tangent vectorξ∈TX​St​\(n,r\)\\xi\\in\\mathrm\{T\}\_\{X\}\\mathrm\{St\}\(n,r\)can be decomposed as

ξ=X​Ω\+X⟂​K,\\xi=X\\varOmega\+X\_\{\\perp\}K,whereΩ∈ℝr×r\\varOmega\\in\\mathbb\{R\}^\{r\\times r\}is skew\-symmetric,X⟂∈ℝn×\(n−r\)X\_\{\\perp\}\\in\\mathbb\{R\}^\{n\\times\(n\-r\)\}satisfies\[X​X⟂\]∈O⁡\(n\)\[X\\;X\_\{\\perp\}\]\\in\\mathrm\{O\}\(n\),O⁡\(n\)\\mathrm\{O\}\(n\)being the orthogonal group, andK∈ℝ\(n−r\)×rK\\in\\mathbb\{R\}^\{\(n\-r\)\\times r\}\.

#### A\.1\.1\. Orthogonal Projection ontoTX​St​\(n,r\)\\mathrm\{T\}\_\{X\}\\mathrm\{St\}\(n,r\)

For any matrixξ∈ℝn×r\\xi\\in\\mathbb\{R\}^\{n\\times r\}, its orthogonal projection ontoTX​St​\(n,r\)\\mathrm\{T\}\_\{X\}\\mathrm\{St\}\(n,r\)with respect to the Euclidean inner product is given by

PX⁡\(ξ\)=X​skew​\(X⊤​ξ\)\+\(In−X​X⊤\)​ξ,\\Proj\_\{X\}\(\\xi\)=X\\,\\mathrm\{skew\}\(X^\{\\top\}\\\!\\xi\)\+\(I\_\{n\}\-XX^\{\\top\}\)\\,\\xi,\(A\.1\)whereskew⁡\(M\)≔12​\(M−M⊤\)\\mathrm\{skew\}\(M\)\\coloneqq\\tfrac\{1\}\{2\}\(M\-M^\{\\top\}\)denotes the skew\-symmetric part of a square matrix\.

Equation \([A\.1](https://arxiv.org/html/2609.21039#A1.E1)\) admits the equivalent and more compact expression

PX⁡\(ξ\)=ξ−X​sym​\(X⊤​ξ\),\\Proj\_\{X\}\(\\xi\)=\\xi\-X\\,\\mathrm\{sym\}\(X^\{\\top\}\\\!\\xi\),wheresym⁡\(M\)≔12​\(M\+M⊤\)\\mathrm\{sym\}\(M\)\\coloneqq\\tfrac\{1\}\{2\}\(M\+M^\{\\top\}\)denotes the symmetric part\. This formula is commonly used in Riemannian optimization on the Stiefel manifold; see, e\.g\.,\([Absil et al\., 2008](https://arxiv.org/html/2609.21039#bib.bib4), Prop\. 3\.6\.1\)\.

#### A\.1\.2\. Row\-Orthonormal Stiefel Manifold

In this work, we enforce a*row\-orthonormal*constraint on the factorA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}, namely,

This corresponds to working with the transpose variableX≔A⊤∈St⁡\(n,r\)X\\coloneqq A^\{\\top\}\\in\\mathrm\{St\}\(n,r\)\. All Riemannian operations \(projection, retraction, and gradient computation\) are therefore performed onXX, and the updated factor is recovered asA=X⊤A=X^\{\\top\}\.

### A\.2\. Cayley Transform and Retraction

A commonly used retraction on the Stiefel manifoldSt⁡\(n,r\)=\{X∈ℝn×r:X⊤​X=Ir\}\\mathrm\{St\}\(n,r\)=\\\{X\\in\\mathbb\{R\}^\{n\\times r\}\\colon X^\{\\top\}\\\!X=I\_\{r\}\\\}is based on the Cayley transform\. The Cayley transform generates a smooth curve on the Stiefel manifold by exponentiating a skew\-symmetric matrix in a rational form, thereby avoiding explicit matrix exponentials\.

The closed\-form Cayley retraction at a pointX∈St⁡\(n,r\)X\\in\\mathrm\{St\}\(n,r\)is defined as

Y⁡\(α\)=\(In−α2​Ω\)−1​\(In\+α2​Ω\)​X,Y\(\\alpha\)=\\left\(I\_\{n\}\-\\tfrac\{\\alpha\}\{2\}\\varOmega\\right\)^\{\-1\}\\left\(I\_\{n\}\+\\tfrac\{\\alpha\}\{2\}\\varOmega\\right\)X,\(A\.2\)whereΩ∈ℝn×n\\varOmega\\in\\mathbb\{R\}^\{n\\times n\}is a skew\-symmetric matrix andα≥0\\alpha\\geq 0is a step\-size parameter\. The curve satisfies

Y⁡\(0\)=X,dd​α​Y​\(α\)\|α=0=Ω​X,Y\(0\)=X,\\qquad\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\alpha\}Y\(\\alpha\)\\big\|\_\{\\alpha=0\}=\\varOmega X,and therefore defines a valid first\-order retraction on the Stiefel manifold\([Absil et al\., 2008](https://arxiv.org/html/2609.21039#bib.bib4)\)\.

Computing the closed\-form expression \([A\.2](https://arxiv.org/html/2609.21039#A1.E2)\) requires solving a linear system involving ann×nn\\times nmatrix, which can be computationally expensive for largenn\. A fixed\-point approximation of the Cayley transform is given by

Y⁡\(α\)=X\+α2​Ω​\(X\+Y⁡\(α\)\)\.Y\(\\alpha\)=X\+\\frac\{\\alpha\}\{2\}\\,\\varOmega\\bigl\(X\+Y\(\\alpha\)\\bigr\)\.\(A\.3\)Starting from the initializationY0=\(I\+α​Ω\)​XY\_\{0\}=\(I\+\\alpha\\varOmega\)X, this fixed\-point equation can be solved with a small number of iterations, each involving only matrix multiplications\. In practice, only a few iterations are sufficient to obtain an accurate approximation of the exact Cayley retraction\.

#### A\.2\.1\. Cayley Retraction in Our Setting

As mentioned in the main text, in the factorization of a weight matrix, we work with the row\-orthonormal factor of sizerr\-by\-nnsatisfyingA​A⊤=IrAA^\{\\top\}=I\_\{r\}\. For convenience, we switch to a column\-orthonormal representation so thatX≔A⊤∈St⁡\(n,r\)X\\coloneqq A^\{\\top\}\\in\\mathrm\{St\}\(n,r\)\.

Letξ∈TXt​St​\(n,r\)\\xi\\in\\mathrm\{T\}\_\{X\_\{t\}\}\\mathrm\{St\}\(n,r\)be a tangent vector atXt≔At⊤X\_\{t\}\\coloneqq A\_\{t\}^\{\\top\}\. Following\([Li et al\., 2020](https://arxiv.org/html/2609.21039#bib.bib29), Eq\. \(2\)\), the Cayley retraction used in our algorithm is defined as

Xt\+1=\(In−12​Ω\)−1​\(In\+12​Ω\)​Xt,At\+1≔Xt\+1⊤,X\_\{t\+1\}=\\left\(I\_\{n\}\-\\tfrac\{1\}\{2\}\\varOmega\\right\)^\{\-1\}\\left\(I\_\{n\}\+\\tfrac\{1\}\{2\}\\varOmega\\right\)X\_\{t\},\\qquad A\_\{t\+1\}\\coloneqq X\_\{t\+1\}^\{\\top\},where the skew\-symmetric matrixΩ\\varOmegais constructed as

Ω^=ξ​Xt⊤−12​Xt​\(Xt⊤​ξ​Xt⊤\),Ω=Ω^−Ω^⊤\.\\widehat\{\\varOmega\}=\\xi X\_\{t\}^\{\\top\}\-\\tfrac\{1\}\{2\}X\_\{t\}\(X\_\{t\}^\{\\top\}\\xi X\_\{t\}^\{\\top\}\),\\qquad\\varOmega=\\widehat\{\\varOmega\}\-\\widehat\{\\varOmega\}^\{\\top\}\.In practice, we compute the Cayley retraction using the fixed\-point iteration \([A\.3](https://arxiv.org/html/2609.21039#A1.E3)\)\. This yields an efficient and numerically stable retraction satisfying the standard first\-order retraction conditionsRetrx⁡\(0x\)=x\\Retraction\_\{x\}\(0\_\{x\}\)=xandD⁡Retrx⁡\(0x\)=Id\\D\\\!\\Retraction\_\{x\}\(\{0\}\_\{x\}\)=\\mathrm\{Id\}required for the convergence analysis of Riemannian optimization methods; see, e\.g\.,\([Absil et al\., 2008](https://arxiv.org/html/2609.21039#bib.bib4), §4\.1\)\.

#### A\.2\.2\. Fixed\-Point Approximation of the Cayley Retraction

In this section, we analyze the fixed\-point iteration used to approximate the Cayley retraction and show that it geometrically converges to the exact Cayley transform\. Most importantly, we further show that the resulting approximate mapping retains the retraction properties required by the convergence theory\.

Let\{Yi\}i≥0\\\{Y\_\{i\}\\\}\_\{i\\geq 0\}be the sequence of fixed\-point iterates defined by

Yi\+1=ℱ⁡\(Yi\),Y\_\{i\+1\}=\\mathcal\{F\}\(Y\_\{i\}\),\(A\.4\)whereℱ\{\\mathcal\{F\}\}is the fixed\-point map of \([A\.3](https://arxiv.org/html/2609.21039#A1.E3)\), namely,ℱ⁡\(Y\)≔X\+α2​Ω​\(X\+Y\)\{\\mathcal\{F\}\}\(Y\)\\coloneqq X\+\\frac\{\\alpha\}\{2\}\\varOmega\(X\+Y\), and the initialization isY0=\(I\+α​Ω\)​XY\_\{0\}=\(I\+\\alpha\\varOmega\)X\. We recall the contraction property of a fixed\-point map\.

###### Lemma A\.1\(Contraction of the fixed\-point map\)\.

Assume thatα​‖Ω‖2<2\\alpha\\\|\\varOmega\\\|\_\{2\}<2\(α≥0\\alpha\\geq 0\)\. Thenℱ\{\\mathcal\{F\}\}is a contraction mapping onℝn×r\\mathbb\{R\}^\{n\\times r\}with contraction factor

ρ≔α2​‖Ω‖2<1\.\\rho\\coloneqq\\frac\{\\alpha\}\{2\}\\\|\\varOmega\\\|\_\{2\}<1\.

###### Proof\.

For anyY1,Y2∈ℝn×rY\_\{1\},\\,Y\_\{2\}\\in\\mathbb\{R\}^\{n\\times r\},

‖ℱ⁡\(Y1\)−ℱ⁡\(Y2\)‖=‖α2​Ω​Y1−α2​Ω​Y2‖≤α2​‖Ω‖2​‖Y1−Y2‖<‖Y1−Y2‖,\\\|\{\\mathcal\{F\}\}\(Y\_\{1\}\)\-\{\\mathcal\{F\}\}\(Y\_\{2\}\)\\\|=\\left\\\|\\frac\{\\alpha\}\{2\}\\varOmega Y\_\{1\}\-\\frac\{\\alpha\}\{2\}\\varOmega Y\_\{2\}\\right\\\|\\leq\\frac\{\\alpha\}\{2\}\\\|\\varOmega\\\|\_\{2\}\\,\\\|Y\_\{1\}\-Y\_\{2\}\\\|<\\\|Y\_\{1\}\-Y\_\{2\}\\\|,which shows thatℱ\{\\mathcal\{F\}\}is a contraction mapping\. ∎

###### Theorem A\.2\(Geometric convergence to the Cayley retraction\)\.

LetY⋆≔RetrX⁡\(α​ξ\)Y^\{\\star\}\\coloneqq\\Retraction\_\{X\}\(\\alpha\\xi\)be the exact Cayley retraction ofα​ξ\\alpha\\xiatXX\. Under the assumptions of Lemma[A\.1](https://arxiv.org/html/2609.21039#A1.Thmtheorem1), the fixed\-point iterates defined by \([A\.4](https://arxiv.org/html/2609.21039#A1.E4)\) satisfy

∀i≥0,‖Yi−Y⋆‖≤ρi​‖Y0−Y⋆‖\.\\forall i\\geq 0,\\qquad\\\|Y\_\{i\}\-Y^\{\\star\}\\\|\\leq\\rho^\{i\}\\,\\\|Y\_\{0\}\-Y^\{\\star\}\\\|\.\(A\.5\)

###### Proof\.

Since by Lemma[A\.1](https://arxiv.org/html/2609.21039#A1.Thmtheorem1)ℱ\{\\mathcal\{F\}\}is a contraction mapping, the Banach fixed\-point theorem applies, with the exact retractionY⋆=RetrX⁡\(α​ξ\)Y^\{\\star\}=\\Retraction\_\{X\}\(\\alpha\\xi\)being the unique fixed point ofℱ\{\\mathcal\{F\}\}\. The theorem’s result \([A\.5](https://arxiv.org/html/2609.21039#A1.E5)\) follows directly\. ∎

We are now in the position to state the properties of the approximate retraction, which we denote by

Retr~X​\(α​ξ\)≔Ys,\\widetilde\{\\Retraction\}\_\{X\}\(\\alpha\\xi\)\\coloneqq Y\_\{s\},wheressis the total number of fixed\-point iterations performed\.

###### Lemma A\.3\(Second\-order accuracy\)\.

For sufficiently smallα\\alphaand any fixed number of iterationsss, the approximate Cayley retraction satisfies

Retr~X​\(α​ξ\)=X\+α​ξ\+𝒪⁡\(α2\)\+𝒪⁡\(ρs\),\\widetilde\{\\Retraction\}\_\{X\}\(\\alpha\\xi\)=X\+\\alpha\\xi\+\{\\mathcal\{O\}\}\(\\alpha^\{2\}\)\+\{\\mathcal\{O\}\}\(\\rho^\{s\}\),and converges to the exact Cayley retraction ass→∞s\\to\\infty\.

In particular, for fixedssand sufficiently small step sizeα\\alpha, the approximation error remains of higher order and does not dominate the first\-order behavior of the update\. This justifies using a finite number of fixed\-point iterations in practice\.

###### Proof\.

The exact Cayley transform \([A\.2](https://arxiv.org/html/2609.21039#A1.E2)\) admits the expansion

RetrX⁡\(α​ξ\)=X\+α​ξ\+𝒪⁡\(α2\)\.\\Retraction\_\{X\}\(\\alpha\\xi\)=X\+\\alpha\\xi\+\{\\mathcal\{O\}\}\(\\alpha^\{2\}\)\.\(A\.6\)The initialization of the fixed\-point iteration method is

Y0=\(I\+α​Ω\)​X=X\+α​Ω​X=X\+α​ξ,Y\_\{0\}=\(I\+\\alpha\\varOmega\)X=X\+\\alpha\\varOmega X=X\+\\alpha\\xi,which is clearly a first\-order approximation ofRetrX⁡\(α​ξ\)\\Retraction\_\{X\}\(\\alpha\\xi\)\. Therefore

‖Y0−Y⋆‖=𝒪⁡\(α2\)\.\\\|Y\_\{0\}\-Y^\{\\star\}\\\|=\{\\mathcal\{O\}\}\(\\alpha^\{2\}\)\.\(A\.7\)By Theorem[A\.2](https://arxiv.org/html/2609.21039#A1.Thmtheorem2), we have the contraction estimate \([A\.5](https://arxiv.org/html/2609.21039#A1.E5)\)\. Inserting \([A\.7](https://arxiv.org/html/2609.21039#A1.E7)\) into the \([A\.5](https://arxiv.org/html/2609.21039#A1.E5)\), withi=si=s, we obtain

‖Ys−Y⋆‖≤ρs​𝒪​\(α2\)\.\\\|Y\_\{s\}\-Y^\{\\star\}\\\|\\leq\\rho^\{s\}\\,\{\\mathcal\{O\}\}\(\\alpha^\{2\}\)\.Sinceα\\alphais fixed within one update, the factor𝒪⁡\(α2\)\{\\mathcal\{O\}\}\(\\alpha^\{2\}\)can be absorbed into the constant, i\.e\.,

‖Ys−Y⋆‖≤𝒪⁡\(ρs\)\.\\\|Y\_\{s\}\-Y^\{\\star\}\\\|\\leq\{\\mathcal\{O\}\}\(\\rho^\{s\}\)\.or, equivalently,Ys=RetrX⁡\(α​ξ\)\+𝒪⁡\(ρs\)Y\_\{s\}=\\Retraction\_\{X\}\(\\alpha\\xi\)\+\{\\mathcal\{O\}\}\(\\rho^\{s\}\)\. Combining this with the expansion of the exact retraction \([A\.6](https://arxiv.org/html/2609.21039#A1.E6)\), the result of the lemma follows immediately\. ∎

In particular, Lemma[A\.3](https://arxiv.org/html/2609.21039#A1.Thmtheorem3)shows that the approximate Cayley mapping satisfies the first\-order retraction conditions up to a controllable error\. Such inexact retractions preserve the convergence guarantees of Riemannian first\-order methods provided the approximation error is sufficiently small; see, e\.g\., standard analyses of inexact retraction schemes\.

Lemma[A\.3](https://arxiv.org/html/2609.21039#A1.Thmtheorem3)shows that for any fixedssand sufficiently smallα\\alpha, the approximate Cayley transform defined by the fixed\-point iteration still satisfies the first\-order retraction condition\. The term𝒪⁡\(ρs\)\{\\mathcal\{O\}\}\(\\rho^\{s\}\)is the numerical approximation error, controlled by the number of iterationsssof the fixed\-point approximation method\. In particular, for a fixedα\\alphaand large enoughss, the approximate Cayley transform converges to the exact Cayley map\.

## Appendix BProof of[Theorem4\.2](https://arxiv.org/html/2609.21039#S4.Thmtheorem2)

###### Proof\.

Consider the sequenceℒt\{\\mathcal\{L\}\}\_\{t\}of convex loss functions and define

R⁡\(T\)≔∑t=1Tℒt​\(Bt,At\)−ℒt​\(B∗,A∗\),R\(T\)\\coloneqq\\sum\_\{t=1\}^\{T\}\{\\mathcal\{L\}\}\_\{t\}\(B\_\{t\},A\_\{t\}\)\-\{\\mathcal\{L\}\}\_\{t\}\(B^\{\*\},A^\{\*\}\),where\(Bt,At\)\(B\_\{t\},A\_\{t\}\)is the iteration in[Algorithm1](https://arxiv.org/html/2609.21039#alg1), and\(B∗,A∗\)\(B^\{\*\},A^\{\*\}\)is a minimizer of∑t=1Tℒt​\(B,A\)\\sum\_\{t=1\}^\{T\}\{\\mathcal\{L\}\}\_\{t\}\(B,A\)\. Using the convexity ofℒt\{\\mathcal\{L\}\}\_\{t\}, and definingGtA,B=∇A,Bℒt​\(Bt,At\)G\_\{t\}^\{A,B\}=\\nabla\_\{A,B\}\{\\mathcal\{L\}\}\_\{t\}\(B\_\{t\},A\_\{t\}\), we get

R⁡\(T\)≤∑t=1T⟨Bt−B∗,GtB⟩\+∑t=1T⟨At−A∗,GtA⟩\.R\(T\)\\leq\\sum\_\{t=1\}^\{T\}\\langle B\_\{t\}\-B^\{\*\},G\_\{t\}^\{B\}\\rangle\+\\sum\_\{t=1\}^\{T\}\\langle A\_\{t\}\-A^\{\*\},G\_\{t\}^\{A\}\\rangle\.\(B\.1\)We now bound the first term on the right\-hand side of \([B\.1](https://arxiv.org/html/2609.21039#A2.E1)\), i\.e\., the Euclidean term⟨Bt−B∗,GtB⟩\\langle B\_\{t\}\-B^\{\*\},G\_\{t\}^\{B\}\\rangle\. LetHtB:ℝm×r→ℝm×rH\_\{t\}^\{B\}\\colon\\mathbb\{R\}^\{m\\times r\}\\to\\mathbb\{R\}^\{m\\times r\}the linear operator defined byHtB​\(X\)=\(VtB\+ε\)⊙XH\_\{t\}^\{B\}\(X\)=\(V\_\{t\}^\{B\}\+\\varepsilon\)\\,\\odot X\. We bound the norm‖Bt\+1−B∗‖\(HtB\)1/22\\\|B\_\{t\+1\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\(see[DefinitionB\.1](https://arxiv.org/html/2609.21039#A2.Thmtheorem1)\):

‖Bt\+1−B∗‖\(HtB\)1/22=\\displaystyle\\\|B\_\{t\+1\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}=∥Bt−ηt\(HtB\)−1/2MtB−B∗∥\(HtB\)1/22=∥Bt−B∗∥\(HtB\)1/22\+\\displaystyle\\\|B\_\{t\}\-\\eta\_\{t\}\(H\_\{t\}^\{B\}\)^\{\-1/2\}M\_\{t\}^\{B\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}=\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\+\(B\.2\)\+ηt2∥\(HtB\)−1/2MtB∥\(HtB\)1/22−2ηt⟨Bt−B∗,\(HtB\)−1/2MtB⟩\(HtB\)1/2\\displaystyle\+\\eta\_\{t\}^\{2\}\\\|\(H\_\{t\}^\{B\}\)^\{\-1/2\}M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\-2\\eta\_\{t\}\\langle B\_\{t\}\-B^\{\*\},\(H\_\{t\}^\{B\}\)^\{\-1/2\}M\_\{t\}^\{B\}\\rangle\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}=[LemmaB\.2](https://arxiv.org/html/2609.21039#A2.Thmtheorem2)\\displaystyle\\underset\{\\text\{\\lx@cref\{creftypecap~refnum\}\{lemma:operator\_induced\_norm\}\}\}\{=\}∥Bt−B∗∥\(HtB\)1/22\+ηt2∥MtB∥\(HtB\)−1/22−2ηt⟨Bt−B∗,MtB⟩,\\displaystyle\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\+\\eta\_\{t\}^\{2\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\-2\\eta\_\{t\}\\langle B\_\{t\}\-B^\{\*\},M\_\{t\}^\{B\}\\rangle,where the last inner product is in the Frobenius norm \(for clarity, we always omit the subscript\)\. By rearranging \([B\.2](https://arxiv.org/html/2609.21039#A2.E2)\) \(bringing the inner product on the left\-hand side and the norm on the right\-hand side\), we get:

2ηt⟨Bt−B∗,MtB⟩=‖Bt−B∗‖\(HtB\)1/22−‖Bt\+1−B∗‖\(HtB\)1/22⏟≕ΔtB\+ηt2∥MtB∥\(HtB\)−1/22\.\\displaystyle 2\\eta\_\{t\}\\langle B\_\{t\}\-B^\{\*\},M\_\{t\}^\{B\}\\rangle=\\underbrace\{\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\-\\\|B\_\{t\+1\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\}\_\{\\eqqcolon\\varDelta^\{B\}\_\{t\}\}\+\\eta\_\{t\}^\{2\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\.\(B\.3\)Using the definition ofMtB=β1​Mt−1B\+\(1−β1\)​GtBM\_\{t\}^\{B\}=\\beta\_\{1\}M\_\{t\-1\}^\{B\}\+\(1\-\\beta\_\{1\}\)G\_\{t\}^\{B\}in \([B\.3](https://arxiv.org/html/2609.21039#A2.E3)\), and by definingΔtB≔‖Bt−B∗‖\(HtB\)1/22−‖Bt\+1−B∗‖\(HtB\)1/22\\varDelta\_\{t\}^\{B\}\\coloneqq\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\-\\\|B\_\{t\+1\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}, we get

⟨Bt−B∗,GtB⟩=\\displaystyle\\langle B\_\{t\}\-B^\{\*\},G\_\{t\}^\{B\}\\rangle=12​\(1−β1\)​ηtΔtB\+ηt2​\(1−β1\)∥MtB∥\(HtB\)−1/22−β11−β1⟨Bt−B∗,Mt−1B⟩\\displaystyle\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\varDelta\_\{t\}^\{B\}\+\\frac\{\\eta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\-\\frac\{\\beta\_\{1\}\}\{1\-\\beta\_\{1\}\}\\langle B\_\{t\}\-B^\{\*\},M\_\{t\-1\}^\{B\}\\rangle\(B\.4\)≤\\displaystyle\\leq12​\(1−β1\)​ηtΔtB\+ηt2​\(1−β1\)∥MtB∥\(HtB\)−1/22\+β11−β1\|⟨Bt−B∗,Mt−1B⟩\|\\displaystyle\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\varDelta\_\{t\}^\{B\}\+\\frac\{\\eta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\+\\frac\{\\beta\_\{1\}\}\{1\-\\beta\_\{1\}\}\\left\\lvert\\langle B\_\{t\}\-B^\{\*\},M\_\{t\-1\}^\{B\}\\rangle\\right\\rvert≤[LemmaB\.3](https://arxiv.org/html/2609.21039#A2.Thmtheorem3)\\displaystyle\\underset\{\\text\{\\lx@cref\{creftypecap~refnum\}\{lemma:young\_inequality\}\}\}\{\\leq\}ΔtB2​\(1−β1\)​ηt\+ηt2​\(1−β1\)∥MtB∥\(HtB\)−1/22\+β12​\(1−β1\)​αt2∥Mt−1B∥\(HtB\)−1/22\\displaystyle\\frac\{\\varDelta\_\{t\}^\{B\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\+\\frac\{\\eta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\+\\frac\{\\beta\_\{1\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\+\\displaystyle\+β1​αt22​\(1−β1\)​‖Bt−B∗‖\(HtB\)1/22\.\\displaystyle\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\.Apart from the retraction, the term⟨At−A∗,GtA⟩\\langle A\_\{t\}\-A^\{\*\},G\_\{t\}^\{A\}\\rangleis similar to \([B\.4](https://arxiv.org/html/2609.21039#A2.E4)\), i\.e\.,

At\+1=RetrAt\(−ηtPAt\(\(HtA\)−1/2MtA\)\)\.A\_\{t\+1\}=\\Retraction\_\{A\_\{t\}\}\\Bigl\(\-\\eta\_\{t\}\\Proj\_\{A\_\{t\}\}\\\!\\left\(\(H\_\{t\}^\{A\}\)^\{\-1/2\}M\_\{t\}^\{A\}\\right\)\\Bigr\)\.We define

Et≔RetrAt\(−ηtPAt\(HtA\)−1/2MtA\)−\(At−ηtPAt\(HtA\)−1/2MtA\),E\_\{t\}\\coloneqq\\Retraction\_\{A\_\{t\}\}\(\-\\eta\_\{t\}\\Proj\_\{A\_\{t\}\}\(H\_\{t\}^\{A\}\)^\{\-1/2\}M\_\{t\}^\{A\}\)\-\(A\_\{t\}\-\\eta\_\{t\}\\Proj\_\{A\_\{t\}\}\(H\_\{t\}^\{A\}\)^\{\-1/2\}M\_\{t\}^\{A\}\),and the invertible linear operatorΓt=PAt\(HtA\)−1/2PAt:TAtSt\(n,r\)→TAtSt\(n,r\)\\Gamma\_\{t\}=\\Proj\_\{A\_\{t\}\}\(H\_\{t\}^\{A\}\)^\{\-1/2\}\\Proj\_\{A\_\{t\}\}\\colon\\mathrm\{T\}\_\{A\_\{t\}\}\\mathrm\{St\}\(n,r\)\\to\\mathrm\{T\}\_\{A\_\{t\}\}\\mathrm\{St\}\(n,r\)\. We defineΓ¯t\\bar\{\\Gamma\}\_\{t\}as an extension of the previous map on the whole space,Γ¯t=Γt\+γt​\(I−PAt\)\\bar\{\\Gamma\}\_\{t\}=\\Gamma\_\{t\}\+\\gamma\_\{t\}\(I\-\\Proj\_\{A\_\{t\}\}\), whereγt\>0\\gamma\_\{t\}\>0is a scalar\. With a small abuse of notation, we will still denote byΓt\\Gamma\_\{t\}the mapΓ¯t\\bar\{\\Gamma\}\_\{t\}when there is no risk of confusion\. LetDt≔PAt\(HtA\)−1/2MtA∈TAtSt\(n,r\)D\_\{t\}\\coloneqq\\Proj\_\{A\_\{t\}\}\(H\_\{t\}^\{A\}\)^\{\-1/2\}M\_\{t\}^\{A\}\\in\\mathrm\{T\}\_\{A\_\{t\}\}\\mathrm\{St\}\(n,r\), and consider the norm

‖At\+1−A∗‖Γt−12=‖At−ηt​Dt\+Et−A∗‖Γt−12=‖At−A∗‖Γt−12−2​ηt​⟨Γt−1​Dt,At−A∗⟩\+ηt2​‖Dt‖Γt−12−2​ηt​⟨Et,Γt−1​Dt⟩\+2​⟨Γt−1​Et,At−A∗⟩\+‖Et‖Γt−12≤⟨Et,At−A∗⟩Γt−1≤0‖At−A∗‖Γt−12−2​ηt​⟨Γt−1​Dt,At−A∗⟩\+ηt2​‖Dt‖Γt−12\+−2​ηt​⟨Et,Γt−1​Dt⟩\+‖Et‖Γt−12⏟≕δt\.\\displaystyle\\begin\{aligned\} \\\|A\_\{t\+1\}\-A^\{\*\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}&=\\\|A\_\{t\}\-\\eta\_\{t\}D\_\{t\}\+E\_\{t\}\-A^\{\*\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\\\\ &=\\\|A\_\{t\}\-A^\{\*\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\-2\\eta\_\{t\}\\langle\\Gamma\_\{t\}^\{\-1\}D\_\{t\},A\_\{t\}\-A^\{\*\}\\rangle\+\\eta\_\{t\}^\{2\}\\\|D\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\\\\ &\\quad\-2\\eta\_\{t\}\\langle E\_\{t\},\\Gamma\_\{t\}^\{\-1\}D\_\{t\}\\rangle\+2\\langle\\Gamma\_\{t\}^\{\-1\}E\_\{t\},A\_\{t\}\-A^\{\*\}\\rangle\+\\\|E\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\\\\ \\underset\{\\langle E\_\{t\},A\_\{t\}\-A^\{\*\}\\rangle\_\{\\Gamma\_\{t\}^\{\-1\}\}\\leq 0\}\{\\leq\}&\\\|A\_\{t\}\-A^\{\*\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\-2\\eta\_\{t\}\\langle\\Gamma\_\{t\}^\{\-1\}D\_\{t\},A\_\{t\}\-A^\{\*\}\\rangle\+\\eta\_\{t\}^\{2\}\\\|D\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\+\\underset\{\\eqqcolon\\delta\_\{t\}\}\{\\underbrace\{\-2\\eta\_\{t\}\\langle E\_\{t\},\\Gamma\_\{t\}^\{\-1\}D\_\{t\}\\rangle\+\\\|E\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\}\}\.\\end\{aligned\}namely,

‖At\+1−A∗‖Γt−12≤‖At−A∗‖Γt−12−2​ηt​⟨Γt−1​Dt,At−A∗⟩\+ηt2​‖Dt‖Γt−12\+δt\.\\\|A\_\{t\+1\}\-A^\{\\ast\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\\leq\\\|A\_\{t\}\-A^\{\\ast\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\-2\\eta\_\{t\}\\left\\langle\\Gamma\_\{t\}^\{\-1\}D\_\{t\},A\_\{t\}\-A^\{\\ast\}\\right\\rangle\+\\eta\_\{t\}^\{2\}\\\|D\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\+\\delta\_\{t\}\.\(B\.5\)Bringing the inner product in \([B\.5](https://arxiv.org/html/2609.21039#A2.E5)\) to the left\-hand side and the norm to the right\-hand side leads to

2​ηt​⟨Γt−1​Dt,At−A∗⟩≤‖At−A∗‖Γt−12−‖At\+1−A∗‖Γt−12\+ηt2​‖Dt‖Γt−12\+δt\.2\\eta\_\{t\}\\left\\langle\\Gamma\_\{t\}^\{\-1\}D\_\{t\},A\_\{t\}\-A^\{\\ast\}\\right\\rangle\\leq\\\|A\_\{t\}\-A^\{\\ast\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\-\\\|A\_\{t\+1\}\-A^\{\\ast\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\+\\eta\_\{t\}^\{2\}\\\|D\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\+\\delta\_\{t\}\.DefiningΔtA≔‖At−A∗‖Γt−12−‖At\+1−A∗‖Γt−12\\varDelta\_\{t\}^\{A\}\\coloneqq\\\|A\_\{t\}\-A^\{\*\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\-\\\|A\_\{t\+1\}\-A^\{\*\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}, and dividing by2​ηt2\\eta\_\{t\}, we can write

⟨Γt−1​Dt,At−A∗⟩≤12​ηt​ΔtA\+ηt2​‖Dt‖Γt−12\+12​ηt​δt\.\\langle\\Gamma\_\{t\}^\{\-1\}D\_\{t\},A\_\{t\}\-A^\{\*\}\\rangle\\leq\\frac\{1\}\{2\\eta\_\{t\}\}\\varDelta\_\{t\}^\{A\}\+\\frac\{\\eta\_\{t\}\}\{2\}\\\|D\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\+\\frac\{1\}\{2\\eta\_\{t\}\}\\delta\_\{t\}\.We now notice that, by definition ofΓt\\Gamma\_\{t\}, we haveΓt−1​Dt=MtA\\Gamma\_\{t\}^\{\-1\}D\_\{t\}=M\_\{t\}^\{A\}, and, by using again Young inequality \([LemmaB\.3](https://arxiv.org/html/2609.21039#A2.Thmtheorem3)\), we get

⟨GtA,At−A∗⟩≤12​\(1−β1\)​ηt​ΔtA\+ηt2​\(1−β1\)​‖Dt‖Γt−12\+12​\(1−β1\)​ηt​δt\+β1​αt22​\(1−β1\)​‖At−A∗‖F2\+β12​\(1−β1\)​αt2​‖Mt−1A‖F2=12​\(1−β1\)​ηtΔtA\+ηt2​\(1−β1\)‖\(HtA\)−1/4\(HtA\)1/4Γt1/2Γt−1Dt‖F2\+12​\(1−β1\)​ηtδt\+β1​αt22​\(1−β1\)‖\(HtA\)−1/4\(HtA\)1/4\(At−A∗\)‖F2\+β12​\(1−β1\)​αt2∥\(HtA\)−1/4\(HtA\)1/4Mt−1A∥F2=12​\(1−β1\)​ηtΔtA\+ηt2​\(1−β1\)∥\(HtA\)1/4Γt1/2MtA∥\(HtA\)−1/22\+12​\(1−β1\)​ηtδt\+β1​αt22​\(1−β1\)∥\(HtA\)1/4\(At−A∗\)∥\(HtA\)−1/22\+β12​\(1−β1\)​αt2∥\(HtA\)−1/4Mt−1A∥\(HtA\)1/22≤ΔtA2​\(1−β1\)​ηt\+ηt​‖\(HtA\)1/4​Γt1/2‖op22​\(1−β1\)∥MtA∥\(HtA\)−1/22\+δt2​\(1−β1\)​ηt\+β1​αt2​‖\(HtA\)1/4‖op22​\(1−β1\)∥At−A∗∥\(HtA\)−1/22\+β1∥\(HtA\)−1/4∥op22​\(1−β1\)​αt2∥Mt−1A∥\(HtA\)1/22\.\\displaystyle\\begin\{aligned\} \\langle G\_\{t\}^\{A\},A\_\{t\}\-A^\{\*\}\\rangle&\\leq\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\varDelta\_\{t\}^\{A\}\+\\frac\{\\eta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|D\_\{t\}\\\|\_\{\\Gamma\_\{t\}^\{\-1\}\}^\{2\}\+\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\delta\_\{t\}\\\\ &\\quad\+\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|A\_\{t\}\-A^\{\\ast\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\frac\{\\beta\_\{1\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\\\|M\_\{t\-1\}^\{A\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\\\ &=\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\varDelta\_\{t\}^\{A\}\+\\frac\{\\eta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\}\\left\\\|\(H\_\{t\}^\{A\}\)^\{\-1/4\}\(H\_\{t\}^\{A\}\)^\{1/4\}\\Gamma\_\{t\}^\{1/2\}\\Gamma\_\{t\}^\{\-1\}D\_\{t\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\delta\_\{t\}\\\\ &\\quad\+\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\left\\\|\(H\_\{t\}^\{A\}\)^\{\-1/4\}\(H\_\{t\}^\{A\}\)^\{1/4\}\(A\_\{t\}\-A^\{\\ast\}\)\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\frac\{\\beta\_\{1\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\\\|\(H\_\{t\}^\{A\}\)^\{\-1/4\}\(H\_\{t\}^\{A\}\)^\{1/4\}M\_\{t\-1\}^\{A\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\\\\ &=\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\varDelta\_\{t\}^\{A\}\+\\frac\{\\eta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|\(H\_\{t\}^\{A\}\)^\{1/4\}\\Gamma\_\{t\}^\{1/2\}M\_\{t\}^\{A\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}^\{2\}\+\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\delta\_\{t\}\\\\ &\\quad\+\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|\(H\_\{t\}^\{A\}\)^\{1/4\}\(A\_\{t\}\-A^\{\\ast\}\)\\\|\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}^\{2\}\+\\frac\{\\beta\_\{1\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\\\|\(H\_\{t\}^\{A\}\)^\{\-1/4\}M\_\{t\-1\}^\{A\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{1/2\}\}^\{2\}\\\\ &\\leq\\frac\{\\varDelta\_\{t\}^\{A\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\+\\frac\{\\eta\_\{t\}\\\|\(H\_\{t\}^\{A\}\)^\{1/4\}\\Gamma\_\{t\}^\{1/2\}\\\|\_\{\\mathrm\{op\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|M\_\{t\}^\{A\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}^\{2\}\+\\frac\{\\delta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\\\ &\\quad\+\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}\\\|\(H\_\{t\}^\{A\}\)^\{1/4\}\\\|\_\{\\mathrm\{op\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\\|A\_\{t\}\-A^\{\\ast\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}^\{2\}\+\\frac\{\\beta\_\{1\}\\\|\(H\_\{t\}^\{A\}\)^\{\-1/4\}\\\|\_\{\\mathrm\{op\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\\\|M\_\{t\-1\}^\{A\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{1/2\}\}^\{2\}\.\\end\{aligned\}

\(B\.6\)The termδt\\delta\_\{t\}is the only one structurally different from the ones in \([B\.4](https://arxiv.org/html/2609.21039#A2.E4)\)\. Thus, \([B\.6](https://arxiv.org/html/2609.21039#A2.E6)\) can be bounded using the definition ofEtE\_\{t\}\(retraction error\) with Lagrange remainder error

‖Et‖≤C​‖D2⁡RetrAt⁡\(ζ\)‖​‖ηt​Dt‖2≤C~​ηt2​‖Dt‖2,\\\|E\_\{t\}\\\|\\leq C\\\|\\D^\{2\}\\Retraction\_\{A\_\{t\}\}\(\\zeta\)\\\|\\\|\\eta\_\{t\}D\_\{t\}\\\|^\{2\}\\leq\\tilde\{C\}\\eta\_\{t\}^\{2\}\\\|D\_\{t\}\\\|^\{2\},as

δt2​\(1−β1\)​ηt\\displaystyle\\frac\{\\delta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}≤12​\(1−β1\)​ηt​\[2​ηt​‖Et‖​‖MtA‖\+‖Γt−1‖op​‖Et‖2\]\\displaystyle\\leq\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\Big\[2\\eta\_\{t\}\\\|E\_\{t\}\\\|\\\|M\_\{t\}^\{A\}\\\|\+\\\|\\Gamma\_\{t\}^\{\-1\}\\\|\_\{\\mathrm\{op\}\}\\\|E\_\{t\}\\\|^\{2\}\\Big\]≤12​\(1−β1\)​\[2​‖Et‖​‖MtA‖\+1ηt​‖Γt−1‖op​‖Et‖2\]\\displaystyle\\leq\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\}\\Big\[2\\\|E\_\{t\}\\\|\\\|M\_\{t\}^\{A\}\\\|\+\\frac\{1\}\{\\eta\_\{t\}\}\\\|\\Gamma\_\{t\}^\{\-1\}\\\|\_\{\\mathrm\{op\}\}\\\|E\_\{t\}\\\|^\{2\}\\Big\]≤12​\(1−β1\)​\[2​ηt2​‖Dt‖2​‖MtA‖\+ηt3​‖Γt−1‖op​‖Dt‖4\]\.\\displaystyle\\leq\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\}\\Big\[2\\eta\_\{t\}^\{2\}\\\|D\_\{t\}\\\|^\{2\}\\\|M\_\{t\}^\{A\}\\\|\+\\eta\_\{t\}^\{3\}\\\|\\Gamma\_\{t\}^\{\-1\}\\\|\_\{\\mathrm\{op\}\}\\\|D\_\{t\}\\\|^\{4\}\\Big\]\.Using the definition

‖MtA‖≔‖\(1−β1\)​∑s=1tβ1t−s​GsA‖≤G∞,\\\|M\_\{t\}^\{A\}\\\|\\coloneqq\\\|\(1\-\\beta\_\{1\}\)\\sum\_\{s=1\}^\{t\}\\beta\_\{1\}^\{t\-s\}G\_\{s\}^\{A\}\\\|\\leq G\_\{\\infty\},and the fact that‖Dt‖≲G∞\\\|D\_\{t\}\\\|\\lesssim G\_\{\\infty\},‖Γt−1‖op\\\|\\Gamma\_\{t\}^\{\-1\}\\\|\_\{\\mathrm\{op\}\}bounded, we get

∑t=1Tδt2​\(1−β1\)​ηt≲12​\(1−β1\)​\[G∞3​∑t=1Tηt2\+G∞4​∑t=1Tηt3\]\.\\sum\_\{t=1\}^\{T\}\\frac\{\\delta\_\{t\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\\lesssim\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\}\\Big\[G\_\{\\infty\}^\{3\}\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{2\}\+G\_\{\\infty\}^\{4\}\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{3\}\\Big\]\.\(B\.7\)By combining equations \([B\.1](https://arxiv.org/html/2609.21039#A2.E1)\), \([B\.4](https://arxiv.org/html/2609.21039#A2.E4)\), \([B\.6](https://arxiv.org/html/2609.21039#A2.E6)\) and \([B\.7](https://arxiv.org/html/2609.21039#A2.E7)\), we get

R⁡\(T\)≤∑t=1T\(ΔtB2​\(1−β1\)​ηt\+ηt∥MtB∥\(HtB\)−1/222​\(1−β1\)\+β1∥Mt−1B∥\(HtB\)−1/222​\(1−β1\)​αt2\+β1​αt2​‖Bt−B∗‖\(HtB\)1/222​\(1−β1\)\)\+∑t=1T\(ΔtA2​\(1−β1\)​ηt\+ηtK1∥MtA∥\(HtA\)−1/222​\(1−β1\)\+β1​K2​‖Mt−1A‖\(HtA\)1/222​\(1−β1\)​αt2\+β1αt2K3∥At−A∗∥\(HtA\)−1/222​\(1−β1\)\)\+12​\(1−β1\)​\[G∞3​∑t=1Tηt2\+G∞4​∑t=1Tηt3\],\\displaystyle\\begin\{aligned\} R\(T\)&\\leq\\sum\_\{t=1\}^\{T\}\\Biggl\(\\frac\{\\varDelta\_\{t\}^\{B\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\+\\frac\{\\eta\_\{t\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\+\\frac\{\\beta\_\{1\}\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\+\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{1/2\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\Biggr\)\\\\ &\\quad\+\\sum\_\{t=1\}^\{T\}\\Biggl\(\\frac\{\\varDelta\_\{t\}^\{A\}\}\{2\(1\-\\beta\_\{1\}\)\\eta\_\{t\}\}\+\\frac\{\\eta\_\{t\}K\_\{1\}\\\|M\_\{t\}^\{A\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\+\\frac\{\\beta\_\{1\}K\_\{2\}\\\|M\_\{t\-1\}^\{A\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{1/2\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\\alpha\_\{t\}^\{2\}\}\+\\frac\{\\beta\_\{1\}\\alpha\_\{t\}^\{2\}K\_\{3\}\\\|A\_\{t\}\-A^\{\*\}\\\|\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\Biggr\)\\\\ &\\quad\+\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\}\\Biggl\[G\_\{\\infty\}^\{3\}\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{2\}\+G\_\{\\infty\}^\{4\}\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{3\}\\Biggr\],\\end\{aligned\}

\(B\.8\)whereK1,K2,K3K\_\{1\},K\_\{2\},K\_\{3\}are three constants bounding the operator norms in \([B\.6](https://arxiv.org/html/2609.21039#A2.E6)\)\. Notice that up to these constants, the terms inAAandBBare similar to each other; therefore, we focus on bounding the terms inBB\(the ones inAAyield analogous bounds\)\.

Now, we observe that, thanks to\([Reddi et al\., 2018](https://arxiv.org/html/2609.21039#bib.bib11), Lemma 2\), forηt=η/t\\eta\_\{t\}=\\eta/\\sqrt\{t\}we obtain the following bounds

∑t=1Tηt∥MtB∥\(HtB\)−1/22≤η​‖GB‖L1​L2​\(\[0,T\]\)​1\+log⁡T\(1−β1\)​\(1−β1/β2\)​1−β2,\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\\\|M\_\{t\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\\leq\\frac\{\\eta\\\|G^\{B\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\\sqrt\{1\+\\log T\}\}\{\(1\-\\beta\_\{1\}\)\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\},where‖GB‖L1​L2​\(\[0,T\]\)\\\|G^\{B\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}is theL1−L2L^\{1\}\-L^\{2\}norm defined by

‖GB‖L1​L2​\(\[0,T\]\)≔∑i,j\(∑t=1T\|\(GtB\)i​j\|2\)1/2\.\\\|G^\{B\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\\coloneqq\\sum\_\{i,j\}\\left\(\\sum\_\{t=1\}^\{T\}\\left\|\(G\_\{t\}^\{B\}\)\_\{ij\}\\right\|^\{2\}\\right\)^\{1/2\}\.Moreover,

∑t=1Tηt∥MtA∥2\(HtA\)−1/2≤η​‖GA‖L1​L2​\(\[0,T\]\)​1\+log⁡T\(1−β1\)​\(1−β1/β2\)​1−β2,\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\\\|M\_\{t\}^\{A\}\\\|^\{2\}\_\{\(H\_\{t\}^\{A\}\)^\{\-1/2\}\}\\leq\\frac\{\\eta\\\|G^\{A\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\\sqrt\{1\+\\log T\}\}\{\(1\-\\beta\_\{1\}\)\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\},where

∑t=1Tηt2=η2∑t=1T1t≤η2\(1\+logT\),and∑t=1Tηt3=∑t=1Tηt3/2≤1\+∫1Tt−3/2dt=3−2T\.\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{2\}=\\eta^\{2\}\\sum\_\{t=1\}^\{T\}\\frac\{1\}\{t\}\\leq\\eta^\{2\}\(1\+\\log T\),\\hskip 9\.24994pt\\text\{and\}\\hskip 9\.24994pt\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{3\}=\\sum\_\{t=1\}^\{T\}\\frac\{\\eta\}\{t^\{3/2\}\}\\leq 1\+\\int\_\{1\}^\{T\}t^\{\-3/2\}\\,\\mathrm\{d\}t=3\-\\frac\{2\}\{\\sqrt\{T\}\}\.\(B\.9\)
Similarly, we can bound the third term on the right\-hand side of \([B\.8](https://arxiv.org/html/2609.21039#A2.E8)\), i\.e\.,

∥Mt−1B∥\(HtB\)−1/22\\displaystyle\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}=‖\(HtB\)−1/4\(Ht−1B\)1/4\(Ht−1B\)−1/4\(Mt−1B\)‖F2\\displaystyle=\\left\\\|\(H\_\{t\}^\{B\}\)^\{\-1/4\}\(H\_\{t\-1\}^\{B\}\)^\{1/4\}\(H\_\{t\-1\}^\{B\}\)^\{\-1/4\}\(M\_\{t\-1\}^\{B\}\)\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}≤‖\(HtB\)−1/4\(Ht−1B\)1/4‖2→22∥Mt−1B∥\(Ht−1B\)−1/22\\displaystyle\\leq\\left\\\|\(H\_\{t\}^\{B\}\)^\{\-1/4\}\(H\_\{t\-1\}^\{B\}\)^\{1/4\}\\right\\\|\_\{2\\to 2\}^\{2\}\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\-1\}^\{B\}\)^\{\-1/2\}\}^\{2\}≤\(1β2\)1/4∥Mt−1B∥\(Ht−1B\)−1/22,\\displaystyle\\leq\\Bigl\(\\frac\{1\}\{\\beta\_\{2\}\}\\Bigr\)^\{1/4\}\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\-1\}^\{B\}\)^\{\-1/2\}\}^\{2\},\(B\.10\)where the last inequality is given by the fact that

‖\(HtB\)−1/4\(Ht−1B\)1/4‖2→24=\\displaystyle\\left\\\|\(H\_\{t\}^\{B\}\)^\{\-1/4\}\(H\_\{t\-1\}^\{B\}\)^\{1/4\}\\right\\\|\_\{2\\to 2\}^\{4\}=‖\(Vt−1BVtB\)1/4‖L∞4=‖Vt−1Bβ2​Vt−1B\+\(1−β2\)​\(GtB\)∘2‖L∞\\displaystyle\\left\\\|\\Bigl\(\\frac\{V\_\{t\-1\}^\{B\}\}\{V\_\{t\}^\{B\}\}\\Bigr\)^\{1/4\}\\right\\\|\_\{L^\{\\infty\}\}^\{4\}=\\left\\\|\\frac\{V\_\{t\-1\}^\{B\}\}\{\\beta\_\{2\}V\_\{t\-1\}^\{B\}\+\(1\-\\beta\_\{2\}\)\\,\(G\_\{t\}^\{B\}\)^\{\\circ 2\}\}\\right\\\|\_\{L^\{\\infty\}\}=\\displaystyle=‖1\+\(1−β2\)​Vt−1Bβ2​Vt−1B\+\(1−β2\)​\(GtB\)∘2‖L∞≤1\+1−β2β2\.\\displaystyle\\left\\\|1\+\\frac\{\(1\-\\beta\_\{2\}\)V\_\{t\-1\}^\{B\}\}\{\\beta\_\{2\}V\_\{t\-1\}^\{B\}\+\(1\-\\beta\_\{2\}\)\\,\(G\_\{t\}^\{B\}\)^\{\\circ 2\}\}\\right\\\|\_\{L^\{\\infty\}\}\\leq 1\+\\frac\{1\-\\beta\_\{2\}\}\{\\beta\_\{2\}\}\.The result in \([B\.10](https://arxiv.org/html/2609.21039#A2.Ex20)\) allows to apply\([Reddi et al\., 2018](https://arxiv.org/html/2609.21039#bib.bib11), Lemma 2\)\(or, equivalently, using \([B\.9](https://arxiv.org/html/2609.21039#A2.E9)\)\) also on the time\-shifted term∥Mt−1B∥\(HtB\)−1/2\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}, which for1/αt2=α/t1/\\alpha\_\{t\}^\{2\}=\\alpha/\\sqrt\{t\}gives

∑t=1T∥Mt−1B∥\(HtB\)−1/22αt2\\displaystyle\\sum\_\{t=1\}^\{T\}\\frac\{\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\}^\{B\}\)^\{\-1/2\}\}^\{2\}\}\{\\alpha\_\{t\}^\{2\}\}≤∑t=1Tαt\(1β2\)1/4∥Mt−1B∥\(Ht−1B\)−1/22\\displaystyle\\leq\\sum\_\{t=1\}^\{T\}\\frac\{\\alpha\}\{\\sqrt\{t\}\}\\Bigl\(\\frac\{1\}\{\\beta\_\{2\}\}\\Bigr\)^\{1/4\}\\\|M\_\{t\-1\}^\{B\}\\\|\_\{\(H\_\{t\-1\}^\{B\}\)^\{\-1/2\}\}^\{2\}≤α​‖GB‖L1​L2​\(\[0,T\]\)​1\+log⁡Tβ21/4​\(1−β1\)​\(1−β1/β2\)​1−β2\.\\displaystyle\\leq\\frac\{\\alpha\\\|G^\{B\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\\sqrt\{1\+\\log T\}\}\{\\beta\_\{2\}^\{1/4\}\(1\-\\beta\_\{1\}\)\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\}\.\(B\.11\)Using assumptions \(H2\) thatβ1,t=β1​bt\\beta\_\{1,t\}=\\beta\_\{1\}b^\{t\}, and \(H3\) thatsupt‖Bt−B∗‖max≤D∞\\sup\_\{t\}\\\|B\_\{t\}\-B^\{\*\}\\\|\_\{\\max\}\\leq D\_\{\\infty\}\(the same quantity can be bounded by22for the term inAA, given that Stiefel is compact and has finite diameter\), together with the last bound from\([Reddi et al\., 2018](https://arxiv.org/html/2609.21039#bib.bib11), Theorem 4\), and combining the results from \([B\.9](https://arxiv.org/html/2609.21039#A2.E9)\), \([B\.11](https://arxiv.org/html/2609.21039#A2.Ex24)\),\([B\.8](https://arxiv.org/html/2609.21039#A2.E8)\) \(forAtA\_\{t\}they are similar\), we get

R⁡\(T\)≤D∞2​‖\(HTB\)1/4‖2→222​ηT​\(1−β1\)\+D∞22​\(1−β1\)​∑t=1Tβ1,t​αt2​‖\(HtB\)1/4‖2→22\+diam∞​\(St⁡\(n,r\)\)2​‖\(HTA\)1/4‖2→222​ηT​\(1−β1\)\+diam∞​\(St⁡\(n,r\)\)22​\(1−β1\)​∑t=1Tβ1,t​αt2​‖\(HtA\)1/4‖2→22\+\(η\+2​α​β13/4\)​1\+log⁡T2​\(1−β1\)2​\(1−β1/β2\)​1−β2​‖GA‖L1​L2​\(\[0,T\]\)\+\(η\+2​α​β13/4\)​1\+log⁡T2​\(1−β1\)2​\(1−β1/β2\)​1−β2​‖GB‖L1​L2​\(\[0,T\]\)\+12​\(1−β1\)​\[G∞3​η2​\(1\+log⁡T\)\+G∞4​η3​\(3−2T\)\]\.\\begin\{split\}R\(T\)&\\leq\\frac\{D\_\{\\infty\}^\{2\}\\bigl\\\|\(H\_\{T\}^\{B\}\)^\{1/4\}\\bigr\\\|\_\{2\\to 2\}^\{2\}\}\{2\\eta\_\{T\}\(1\-\\beta\_\{1\}\)\}\+\\frac\{D\_\{\\infty\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\sum\_\{t=1\}^\{T\}\\beta\_\{1,t\}\\alpha\_\{t\}^\{2\}\\bigl\\\|\(H\_\{t\}^\{B\}\)^\{1/4\}\\bigr\\\|\_\{2\\to 2\}^\{2\}\\\\ &\\quad\+\\frac\{\\mathrm\{diam\}\_\{\\infty\}\(\\mathrm\{St\}\(n,r\)\)^\{2\}\\bigl\\\|\(H\_\{T\}^\{A\}\)^\{1/4\}\\bigr\\\|\_\{2\\to 2\}^\{2\}\}\{2\\eta\_\{T\}\(1\-\\beta\_\{1\}\)\}\+\\frac\{\\mathrm\{diam\}\_\{\\infty\}\(\\mathrm\{St\}\(n,r\)\)^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\sum\_\{t=1\}^\{T\}\\beta\_\{1,t\}\\alpha\_\{t\}^\{2\}\\bigl\\\|\(H\_\{t\}^\{A\}\)^\{1/4\}\\bigr\\\|\_\{2\\to 2\}^\{2\}\\\\ &\\quad\+\\frac\{\(\\eta\+2\\alpha\\beta\_\{1\}^\{3/4\}\)\\sqrt\{1\+\\log T\}\}\{2\(1\-\\beta\_\{1\}\)^\{2\}\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\}\\bigl\\\|G^\{A\}\\bigr\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\\\\ &\\quad\+\\frac\{\(\\eta\+2\\alpha\\beta\_\{1\}^\{3/4\}\)\\sqrt\{1\+\\log T\}\}\{2\(1\-\\beta\_\{1\}\)^\{2\}\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\}\\bigl\\\|G^\{B\}\\bigr\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\\\\ &\\quad\+\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\}\\left\[G\_\{\\infty\}^\{3\}\\eta^\{2\}\(1\+\\log T\)\+G\_\{\\infty\}^\{4\}\\eta^\{3\}\\left\(3\-\\frac\{2\}\{\\sqrt\{T\}\}\\right\)\\right\]\.\\end\{split\}\(B\.12\)To conclude, we bound the two terms∑t=1Tβ1,t​αt2​‖\(HtB\)1/4‖2→22\\sum\_\{t=1\}^\{T\}\\beta\_\{1,t\}\\alpha\_\{t\}^\{2\}\\\|\(H\_\{t\}^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}uniformly intt, for the choiceβ1,t=β1​bt\\beta\_\{1,t\}=\\beta\_\{1\}b^\{t\}, andαt2=t/α\\alpha\_\{t\}^\{2\}=\\sqrt\{t\}/\{\\alpha\}, we get the upper bound

∑t=1Tβ1,t​αt2​‖\(HtB\)1/4‖2→22\\displaystyle\\sum\_\{t=1\}^\{T\}\\beta\_\{1,t\}\\alpha\_\{t\}^\{2\}\\\|\(H\_\{t\}^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}≤β1α\|‖\(HB\)1/4‖2→22\|∑t=1TLt∞⁡t​bt\\displaystyle\\leq\\frac\{\\beta\_\{1\}\}\{\\alpha\}\\\|\\\|\(H^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\\\|\_\{L^\{\\infty\}\_\{t\}\}\\sum\_\{t=1\}^\{T\}\\sqrt\{t\}b^\{t\}≤Cauchy–Schwarzt​bt/2,bt/2​β1α​‖‖\(HB\)1/4‖2→22‖Lt∞​\(∑t=1Tt​bt\)1/2​\(∑t=1Tbt/2\)1/2\\displaystyle\\underset\{\\begin\{subarray\}\{c\}\\text\{Cauchy\-\-Schwarz\}\\\\ \\sqrt\{t\}b^\{t/2\},~b^\{t/2\}\\end\{subarray\}\}\{\\leq\}\\frac\{\\beta\_\{1\}\}\{\\alpha\}\\\|\\\|\(H^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\\\|\_\{L^\{\\infty\}\_\{t\}\}\\left\(\\sum\_\{t=1\}^\{T\}tb^\{t\}\\right\)^\{1/2\}\\left\(\\sum\_\{t=1\}^\{T\}b^\{t/2\}\\right\)^\{1/2\}≤β1α​‖‖\(HB\)1/4‖2→22‖Lt∞​b\(1−b\)3/2\.\\displaystyle\\leq\\frac\{\\beta\_\{1\}\}\{\\alpha\}\\\|\\\|\(H^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\\\|\_\{L^\{\\infty\}\_\{t\}\}\\frac\{b\}\{\(1\-b\)^\{3/2\}\}\.This gives the final bound

R⁡\(T\)\\displaystyle R\(T\)≲D∞2​‖\(HTB\)1/4‖2→222​ηT​\(1−β1\)\+D∞22​\(1−β1\)​β1α​‖‖\(HB\)1/4‖2→22‖Lt∞​b\(1−b\)3/2\\displaystyle\\lesssim\\frac\{D\_\{\\infty\}^\{2\}\\\|\(H\_\{T\}^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\}\{2\\eta\_\{T\}\(1\-\\beta\_\{1\}\)\}\+\\frac\{D\_\{\\infty\}^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\frac\{\\beta\_\{1\}\}\{\\alpha\}\\\|\\\|\(H^\{B\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\\\|\_\{L^\{\\infty\}\_\{t\}\}\\frac\{b\}\{\(1\-b\)^\{3/2\}\}\+\(η\+2​α​β13/4\)​1\+log⁡T2​\(1−β1\)2​\(1−β1/β2\)​1−β2​‖GB‖L1​L2​\(\[0,T\]\)\+diam∞​\(St⁡\(n,r\)\)2​‖\(HTA\)1/4‖2→222​ηT​\(1−β1\)\\displaystyle\+\\frac\{\(\\eta\+2\\alpha\\beta\_\{1\}^\{3/4\}\)\\sqrt\{1\+\\log T\}\}\{2\(1\-\\beta\_\{1\}\)^\{2\}\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\}\\\|G^\{B\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\+\\frac\{\\mathrm\{diam\}\_\{\\infty\}\(\\mathrm\{St\}\(n,r\)\)^\{2\}\\\|\(H\_\{T\}^\{A\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\}\{2\\eta\_\{T\}\(1\-\\beta\_\{1\}\)\}\+diam∞​\(St⁡\(n,r\)\)22​\(1−β1\)​β1α​‖‖\(HA\)1/4‖2→22‖Lt∞​b\(1−b\)3/2\\displaystyle\+\\frac\{\\mathrm\{diam\}\_\{\\infty\}\(\\mathrm\{St\}\(n,r\)\)^\{2\}\}\{2\(1\-\\beta\_\{1\}\)\}\\frac\{\\beta\_\{1\}\}\{\\alpha\}\\\|\\\|\(H^\{A\}\)^\{1/4\}\\\|\_\{2\\to 2\}^\{2\}\\\|\_\{L^\{\\infty\}\_\{t\}\}\\frac\{b\}\{\(1\-b\)^\{3/2\}\}\+\(η\+2​α​β13/4\)​1\+log⁡T2​\(1−β1\)2​\(1−β1/β2\)​1−β2​‖GA‖L1​L2​\(\[0,T\]\)\\displaystyle\+\\frac\{\(\\eta\+2\\alpha\\beta\_\{1\}^\{3/4\}\)\\sqrt\{1\+\\log T\}\}\{2\(1\-\\beta\_\{1\}\)^\{2\}\(1\-\\beta\_\{1\}/\\sqrt\{\\beta\_\{2\}\}\)\\sqrt\{1\-\\beta\_\{2\}\}\}\\\|G^\{A\}\\\|\_\{L^\{1\}L^\{2\}\(\[0,T\]\)\}\+12​\(1−β1\)​\[G∞3​η2​\(1\+log⁡T\)\+G∞4​η3​\(3−2T\)\]\.\\displaystyle\+\\frac\{1\}\{2\(1\-\\beta\_\{1\}\)\}\\left\[G\_\{\\infty\}^\{3\}\\eta^\{2\}\(1\+\\log T\)\+G\_\{\\infty\}^\{4\}\\eta^\{3\}\\left\(3\-\\frac\{2\}\{\\sqrt\{T\}\}\\right\)\\right\]\.By using the fact thatηT−1\\eta\_\{T\}^\{\-1\}, we can collect the constantC1C\_\{1\}of the terms of order11, the constantC2C\_\{2\}of orderT\\sqrt\{T\}, the constantC3C\_\{3\}of the terms of order1\+log⁡T\\sqrt\{1\+\\log T\}, andC4C\_\{4\}of the terms of orderlog⁡T\\log Tand withC5C\_\{5\}the constant of the term of orderT−1/2T^\{\-1/2\}to get the final bound\. In particular, the order of the bound is

R\(T\)≤C1T\+C2\+C31\+log⁡T\+C4logT\+C5T−1/2\.R\(T\)\\leq C\_\{1\}\\sqrt\{T\}\+C\_\{2\}\+C\_\{3\}\\sqrt\{1\+\\log T\}\+C\_\{4\}\\log T\+C\_\{5\}\\,T^\{\-1/2\}\.∎

###### Definition B\.1\.

\(HH\-norm induced by a full\-rank operator\) Consider a self\-adjoint positive definite operatorH:V→VH\\colon V\\to Vdefined on a finite\-dimensional real Hilbert space\(V,g\)\(V,g\)\. We define theHH\-weighted inner product as

gH​\(v,w\)≔g⁡\(H1/2​v,H1/2​w\),g\_\{H\}\(v,w\)\\coloneqq g\(H^\{1/2\}v,H^\{1/2\}w\),and denote the corresponding norm as

‖x‖H=‖H1/2​x‖g\.\\\|x\\\|\_\{H\}=\\\|H^\{1/2\}x\\\|\_\{g\}\.

###### Lemma B\.2\.

\(Properties ofHH\-induced inner products\) LetHHandVVbe as in[DefinitionB\.1](https://arxiv.org/html/2609.21039#A2.Thmtheorem1)\. Then the following holds:

- •gHγ​\(Hα​v,Hβ​w\)=gHγ​\(v,Hα\+β​w\)=gHα\+β\+γ​\(v,w\)g\_\{H^\{\\gamma\}\}\(H^\{\\alpha\}v,H^\{\\beta\}w\)=g\_\{H^\{\\gamma\}\}\(v,H^\{\\alpha\+\\beta\}w\)=g\_\{H^\{\\alpha\+\\beta\+\\gamma\}\}\(v,w\),
- •‖Hα​x‖Hγ=‖Hα\+γ/2​x‖g=‖x‖H2​α\+γ\\\|H^\{\\alpha\}x\\\|\_\{H^\{\\gamma\}\}=\\\|H^\{\\alpha\+\\gamma/2\}x\\\|\_\{g\}=\\\|x\\\|\_\{H^\{2\\alpha\+\\gamma\}\}\.

###### Proof\.

The first point follows from the definition, self\-adjointnessg⁡\(H​v,w\)=g⁡\(v,H​w\)g\(Hv,w\)=g\(v,Hw\)\(which holds for any power\), and the fact that powers commute \(Hα​Hβ=Hβ​HαH^\{\\alpha\}H^\{\\beta\}=H^\{\\beta\}H^\{\\alpha\}for allα,β\\alpha,\\beta\):

gHγ​\(Hα​v,Hβ​w\)=\\displaystyle g\_\{H^\{\\gamma\}\}\(H^\{\\alpha\}v,H^\{\\beta\}w\)=g⁡\(Hα\+γ/2​v,Hβ\+γ/2​w\)=g⁡\(v,Hα\+β\+γ​w\)=g⁡\(H\(α\+β\+γ\)/2​v,H\(α\+β\+γ\)/2​w\)\\displaystyle g\(H^\{\\alpha\+\\gamma/2\}v,H^\{\\beta\+\\gamma/2\}w\)=g\(v,H^\{\\alpha\+\\beta\+\\gamma\}w\)=g\(H^\{\(\\alpha\+\\beta\+\\gamma\)/2\}v,H^\{\(\\alpha\+\\beta\+\\gamma\)/2\}w\)=\\displaystyle=gHα\+β\+γ​\(v,w\)\.\\displaystyle g\_\{H^\{\\alpha\+\\beta\+\\gamma\}\}\(v,w\)\.The first equality is similar

gHγ​\(Hα​v,Hβ​w\)=\\displaystyle g\_\{H^\{\\gamma\}\}\(H^\{\\alpha\}v,H^\{\\beta\}w\)=g⁡\(Hα​Hγ/2​v,Hβ​Hγ/2​w\)=g⁡\(Hγ/2​v,Hγ/2​Hα\+β​w\)=gHγ​\(v,Hα\+β​w\)\.\\displaystyle g\(H^\{\\alpha\}H^\{\\gamma/2\}v,H^\{\\beta\}H^\{\\gamma/2\}w\)=g\(H^\{\\gamma/2\}v,H^\{\\gamma/2\}H^\{\\alpha\+\\beta\}w\)=g\_\{H^\{\\gamma\}\}\(v,H^\{\\alpha\+\\beta\}w\)\.The norm equality follows immediately from the definition

‖Hα​x‖Hγ2=gHγ​\(Hα​x,Hα​x\)=g⁡\(Hα\+γ/2​x,Hα\+γ/2​x\)=‖Hα\+γ/2​x‖g2=‖x‖H2​α\+γ2\.\\\|H^\{\\alpha\}x\\\|\_\{H^\{\\gamma\}\}^\{2\}=g\_\{H^\{\\gamma\}\}\(H^\{\\alpha\}x,H^\{\\alpha\}x\)=g\(H^\{\\alpha\+\\gamma/2\}x,H^\{\\alpha\+\\gamma/2\}x\)=\\\|H^\{\\alpha\+\\gamma/2\}x\\\|\_\{g\}^\{2\}=\\\|x\\\|\_\{H^\{2\\alpha\+\\gamma\}\}^\{2\}\.∎

###### Lemma B\.3\.

\(Young inequality for dualHHnorms\) LetHH,VVbe as in[DefinitionB\.1](https://arxiv.org/html/2609.21039#A2.Thmtheorem1)and letu,v∈Vu,v\\in V,ζ≠0\\zeta\\neq 0, andα∈ℝ\\alpha\\in\\mathbb\{R\}\. Then, we have

g⁡\(u,v\)≤12​ζ2​‖u‖H−α2\+ζ22​‖v‖Hα2\.g\(u,v\)\\leq\\frac\{1\}\{2\\zeta^\{2\}\}\\\|u\\\|^\{2\}\_\{H^\{\-\\alpha\}\}\+\\frac\{\\zeta^\{2\}\}\{2\}\\\|v\\\|\_\{H^\{\\alpha\}\}^\{2\}\.

###### Proof\.

Consider the expansion

0≤‖u−v‖g2=g⁡\(u−v,u−v\)=‖u‖g2\+‖v‖g2−2​g​\(u,v\),0\\leq\\\|u\-v\\\|\_\{g\}^\{2\}=g\(u\-v,u\-v\)=\\\|u\\\|\_\{g\}^\{2\}\+\\\|v\\\|\_\{g\}^\{2\}\-2g\(u,v\),which implies

g⁡\(u,v\)≤12​‖u‖g2\+12​‖v‖g2\.g\(u,v\)\\leq\\frac\{1\}\{2\}\\\|u\\\|\_\{g\}^\{2\}\+\\frac\{1\}\{2\}\\\|v\\\|\_\{g\}^\{2\}\.Now, sinceHHis self\-adjoint and positive definite with respect to the inner productgg, we have thatζ​Hα\\zeta H^\{\\alpha\}is too, and therefore we get

g\(ζ−1H−α/2u,ζHα/2v\)≤12∥ζ−1H−α/2u∥g2\+12∥ζHα/2v∥g2=12​ζ2∥u∥H−α2\+ζ22∥v∥Hα2\.g\(\\zeta^\{\-1\}H^\{\-\\alpha/2\}u,\\zeta H^\{\\alpha/2\}v\)\\leq\\frac\{1\}\{2\}\\\|\\zeta^\{\-1\}H^\{\-\\alpha/2\}u\\\|\_\{g\}^\{2\}\+\\frac\{1\}\{2\}\\\|\\zeta H^\{\\alpha/2\}v\\\|\_\{g\}^\{2\}=\\frac\{1\}\{2\\zeta^\{2\}\}\\\|u\\\|\_\{H^\{\-\\alpha\}\}^\{2\}\+\\frac\{\\zeta^\{2\}\}\{2\}\\\|v\\\|\_\{H^\{\\alpha\}\}^\{2\}\.∎

### B\.1\. Additional Results on Different Numerical Retractions

In this section, we present numerical results comparing several possible retraction choices on the Stiefel manifold\. In[Figure2](https://arxiv.org/html/2609.21039#A2.F2), we compare matrix size against GPU wall\-clock time and final error\. In particular, we compare QR decomposition, polar decomposition, a direct solver for the Cayley linear system \(Cayley\-Direct\), an iterative method for the Cayley linear system that employs the Sherman–Morrison–Woodbury formula \(Cayley\-SMW\), the fixed\-point iteration \(Cayley\-FP\), and the Newton–Schulz iteration\.

Figure 2:Comparison of different numerical retraction methods\.

## Appendix CProof of[Proposition4\.3](https://arxiv.org/html/2609.21039#S4.Thmtheorem3)

Let\(B,A\)∈ℱ\(B,A\)\\in\\mathcal\{F\}and consider

‖∇\(ℒ∘Φ\)​\(t​B,t−1​A\)‖2\\displaystyle\\\|\\nabla\(\{\\mathcal\{L\}\}\\circ\\Phi\)\(tB,t^\{\-1\}A\)\\\|^\{2\}=∥DΦ\(tB,t−1A\)⊤∇ℒ\(W\)∥2\\displaystyle=\\\|\\D\\Phi\(tB,t^\{\-1\}A\)^\{\\top\}\\nabla\{\\mathcal\{L\}\}\(W\)\\\|^\{2\}=∥tB⊤∇ℒ\(W\)∥2\+∥∇ℒ\(W\)A⊤t−1∥2⟶t→0\+∞,\\displaystyle=\\\|tB^\{\\top\}\\nabla\{\\mathcal\{L\}\}\(W\)\\\|^\{2\}\+\\\|\\nabla\{\\mathcal\{L\}\}\(W\)A^\{\\top\}t^\{\-1\}\\\|^\{2\}\\underset\{\{t\\to 0\}\}\{\\longrightarrow\}\+\\infty,and therefore‖∇\(ℒ∘Φ\)‖L∞​\(ℱ\)=\+∞\\\|\\nabla\(\{\\mathcal\{L\}\}\\circ\\Phi\)\\\|\_\{L^\{\\infty\}\(\\mathcal\{F\}\)\}=\+\\infty\. For the second claim, fixAAandBBsuch thatΦ~​\(B,A\)=W\\tilde\{\\Phi\}\(B,A\)=W\. Then, we have

sup\(B′,A′\)∈ℱ~‖∇\(ℒ∘Φ~\)​\(B′,A′\)‖2=supO∈St⁡\(r,r\)‖∇\(ℒ∘Φ~\)​\(B​O,O⊤​A\)‖2<\+∞,\\sup\_\{\(B^\{\\prime\},A^\{\\prime\}\)\\in\\widetilde\{\\mathcal\{F\}\}\}\\\|\\nabla\(\{\\mathcal\{L\}\}\\circ\\widetilde\{\\Phi\}\)\(B^\{\\prime\},A^\{\\prime\}\)\\\|^\{2\}=\\sup\_\{O\\in\\mathrm\{St\}\(r,r\)\}\\\|\\nabla\(\{\\mathcal\{L\}\}\\circ\\widetilde\{\\Phi\}\)\(BO,O^\{\\top\}A\)\\\|^\{2\}<\+\\infty,because of compactness ofSt⁡\(r,r\)\\mathrm\{St\}\(r,r\)and continuity\.

## Appendix DAdditional Experimental Details

### D\.1\. GPT2 E2E Fine\-Tuning

We use hyperparameters tuned as in\([Zhang and Pilanci, 2024](https://arxiv.org/html/2609.21039#bib.bib61)\), as reported in[Table5](https://arxiv.org/html/2609.21039#A4.T5)\.

Table 5:Hyperparameters for GPT2 LoRA fine\-tuning on E2E\.