Ori：面向超低秩 LLM 训练的漂移感知流式 k-PCA

# Drift-Aware Streaming 𝑘-PCA for Very Low-Rank LLM Training
Source: [https://arxiv.org/html/2610.22794](https://arxiv.org/html/2610.22794)
Xinyu Luo1Jincheng Zhou1111Department of Computer Science, Purdue University\.Mengbo Wang1,2Zilin Shen1Nadia Atallah Lanman2,3Petros Drineas1Bruno Ribeiro1Brian Bullins1\{luo466, zhou791, wang4887, shen624, natallah,pdrineas, ribeiro, bbullins\}@purdue\.edu††thanks:Equal contribution\.

###### Abstract

A prominent family of low\-rank LLM optimizers, pioneered by GaLore, performs gradient\-subspace projection: at each step, they project the gradient into a low\-dimensional subspace and map the update back to the full model\. Most existing work either recomputes the gradient subspace at certain intervals, or performs post\-hoc adaptation to catch up to the moving subspace\. In this work, we explicitly identify in practice the slow, stable manner in which LLM gradient subspaces drift across time\. To this end, we propose Ori, a novel low\-rank optimizer for training LLMs that captures the shifting principal gradient directions over time to achieve state\-of\-the\-art accuracy under stringent low\-rank constraints\. Ori employs a streamingkk\-PCA algorithm inspired by Oja\+\+, whose low per\-step cost enables high\-frequency, on\-the\-fly subspace tracking\. In practice, Ori reduces the performance gap of the state\-of\-the\-art low\-rank optimizers to full\-rank optimizers by up to 69\.3% on a 1B LLaMA pretraining task, and achieves the best performance on 10 out of 16 task scenarios in Roberta\-Base fine\-tuning\. Together, these contributions allow Ori to further push the performance and efficiency frontiers of gradient\-projecting optimizers, narrowing the gap to full\-rank training for LLMs\. Code can be found at[https://anonymous\.4open\.science/r/Ori\-Submission\-2132](https://anonymous.4open.science/r/Ori-Submission-2132)\.

22footnotetext:Purdue Institute for Cancer Research, Purdue University\.33footnotetext:Department of Comparative Pathobiology, Purdue University\.## 1Introduction

Training large language models \(LLMs\) under strict memory budgets has motivated a variety of low\-rank optimization methods that restrict either the weights or the gradients to a low\-dimensional subspace\[[10](https://arxiv.org/html/2610.22794#bib.bib26),[39](https://arxiv.org/html/2610.22794#bib.bib33),[20](https://arxiv.org/html/2610.22794#bib.bib17)\]\. This broad field is primarily divided into two families\.Weight\-subspace methods, such as LoRA, adapt the model weights directly\[[10](https://arxiv.org/html/2610.22794#bib.bib26),[14](https://arxiv.org/html/2610.22794#bib.bib25)\]\. More recently,gradient\-subspace methods, pioneered by GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], project the gradients into a low\-rank subspace before the optimizer step\. The latter approach and its subsequent works have proven particularly effective, demonstrating superior memory savings and performance by focusing on the dynamics of the optimizer itself\[[39](https://arxiv.org/html/2610.22794#bib.bib33),[38](https://arxiv.org/html/2610.22794#bib.bib10),[14](https://arxiv.org/html/2610.22794#bib.bib25),[8](https://arxiv.org/html/2610.22794#bib.bib19),[5](https://arxiv.org/html/2610.22794#bib.bib24),[28](https://arxiv.org/html/2610.22794#bib.bib2)\]\.

Yet most gradient\-subspace methods in this promising family face a core technical challenge: that the principal gradient directions, instead of being static, evolve as the model weights are updated throughout training\. Consequently, the success of any such method hinges on how effectively it adapts to this changing subspace\. GaLore tackles this by periodically recomputing the entire subspace from scratch via an expensive SVD\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], a process that acts as a “cold restart” and discards prior subspace information\. More recent methods, such as Online Subspace Descent \(OSD\)\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\], move towards more continuous adaptation by using a secondary, non\-convex optimization loop to “catch up” to the moving subspace\.

Figure 1:Trade\-off between evaluation perplexity \(Y\-axis, lower is better\) and peak optimizer state memory \(X\-axis, lower is better\) for various low\-rank optimizers in different ranks on the 1B LLaMA pre\-training task\. Bottom\-left corner is the ideal region\.Ori \(⋆\\star\) forms a better Pareto frontier, achieving a superior performance\-to\-memory trade\-off compared to state\-of\-the\-art baselines like GaLore \(∘\\circ\) and OSD \(□\\square\)\.However, existing adaptation strategies may overlook a key empirical observation that we make explicit in this work: the gradient subspace does not jump erratically but, rather, drifts smoothly after warm\-up\. In other words, the projector at stepttis predictably close to that att−Δ​tt\-\\Delta tfor smallΔ​t\\Delta t\(e\.g\., 20\)\. This slow, structured drift suggests a different perspective\. Rather than recomputing the subspace \(as in GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\]\) or optimizing it with non\-convex gradient descent \(as in OSD\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\]\), we can track it with a streaming update that makes small, frequent rotations aligned with the most recent gradients, while also preserving and transferring optimizer statistics across these rotations\. In light of these observations, we arrive at a different design problem:

How do we continuously maintain a memory\-light, compute\-efficient, and stable projector that tracks a drifting gradient subspace?

We thus introduceOri, a streamingkk\-PCA\-basedOja\-style updated low\-rankiterative optimizer, designed to exploit the slow\-drift regime of LLM gradient subspaces\. Rather than treating each subspace estimation as an independent, “cold\-start” problem—as is the case with periodic SVD recomputation in GaLore—Ori treats subspace estimation as a continuous tracking problem\. The core innovation of Ori lies in its drift\-aware bootstrapping mechanism: we carry over \(transfer\) the subspace basis from previous steps as the initialization state for the current step\. Because of the slow\-drift phenomenon, the prior basis provides a suitable starting point that requires only a few lightweight, streamingkk\-PCA\-inspired updates to realign with the instantaneous gradient geometry\.

In practice, Ori instantiates this via the Oja\+\+ algorithm\[[24](https://arxiv.org/html/2610.22794#bib.bib3),[1](https://arxiv.org/html/2610.22794#bib.bib34)\], which enables high\-frequency basis rotations at a per\-step cost of only𝒪⁡\(d​k2\)\\mathcal\{O\}\(dk^\{2\}\)\. This approach allows for a warm\-start transfer of Adam\-style first and second moments between successive bases, ensuring stable optimization dynamics\. Beyond computational efficiency, we show that this formulation allows Ori to achieve provable sublinear dynamic regret bounds under drift, providing new theoretical tools and guarantees for tracking fidelity that static recomputation methods lack\.

Empirically, Ori narrowly tracks the evolving gradient subspace and converts this tracking fidelity into end\-to\-end wins where it matters most: very low ranks\. Under tight rank budgets, each dimension in the projector is precious, and stale misalignment is especially costly\. Empirical results show that Ori reduces the performance gap of state\-of\-the\-art low\-rank baselines to the full\-rank Adam optimizer byup to 69\.3%on the 1B LLaMA pre\-training task, where the optimizer state rank is 128 in contrast to the model embedding dimension of 2048\. And on Roberta\-Base fine\-tuning tasks, where the ranks are even lower \(8 or 4\), Ori achieves superior performance on10 out of 16scenarios\.

Moreover, Ori achieved this superior performance with one of the lowest computation budgets\. See Figure[1](https://arxiv.org/html/2610.22794#S1.F1)for a performance–memory overview on LLaMA\-1B pre\-training, where Ori consistently sits on the bottom\-left frontier\. Through ablation, we observe that \(i\) increasing the projector\-update period degrades performance—quantifying the cost of staleness, and \(ii\) injecting a small carry\-over \(via a drift coefficientρ≈0\.99\\rho\\approx 0\.99\) improves accuracy—confirming the value of continuity and statistic transfer\. Across pre\-training and fine\-tuning settings, Ori consistently improves the performance–memory trade\-off over LoRA, GaLore, and OSD in the low\-rank regime, thereby reducing the gap to full\-rank Adam\.Our primary contributions in this work are as follows:

- •We identify and quantify slow gradient subspace drift during LLM training and argue that exploiting its continuity – not merely reacting to it – matters for low\-rank optimization\.
- •We propose Ori, a drift\-aware low\-rank optimizer that equips gap\-free streamingkk\-PCA with moment transfer to achieve continuous, high\-frequency projector updates at negligible cost\.
- •We provide theoretical support via new dynamic\-regret bounds under drift, yielding sublinear regret in the noiseless setting with mild dependence on the consecutive drift parameter\.
- •We present empirical results showing Ori’s superior performance under stringent rank constraints, with ablations isolating the roles of update frequency and drift\-aware continuity\.

## 2Background and Related Work

We refer the reader to Appendix[B](https://arxiv.org/html/2610.22794#A2)for a more detailed discussion of related work\.

#### The Landscape: Low\-Rank Optimization\.

Low\-rank optimization in LLMs can be broadly divided into two categories\.Weight\-subspace methodsadapt the model weights by constraining them to a low\-rank manifold, a paradigm popularized by LoRA and its many variants\[[10](https://arxiv.org/html/2610.22794#bib.bib26),[14](https://arxiv.org/html/2610.22794#bib.bib25),[37](https://arxiv.org/html/2610.22794#bib.bib18),[20](https://arxiv.org/html/2610.22794#bib.bib17),[8](https://arxiv.org/html/2610.22794#bib.bib19)\]\. In contrast, Ori belongs to the second category,gradient\-subspace methods, which operate on a low\-rank projection of the gradients\. This line of work was pioneered by GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], which uses periodic Singular Value Decomposition \(SVD\) to identify and project gradients onto a low\-rank basis\. Many subsequent works have built upon this foundation by incorporating diverse strategies such as quantization\[[38](https://arxiv.org/html/2610.22794#bib.bib10)\], structured sparsity\[[22](https://arxiv.org/html/2610.22794#bib.bib30)\], random projections\[[9](https://arxiv.org/html/2610.22794#bib.bib28),[32](https://arxiv.org/html/2610.22794#bib.bib23)\], and importance sampling\[[36](https://arxiv.org/html/2610.22794#bib.bib29),[16](https://arxiv.org/html/2610.22794#bib.bib6)\]\. However, a common limitation underlines these approaches: by relying on static or infrequently updated projectors, they are inherently reactive to the evolving gradient landscape\. This can lead to a “staleness penalty,” where gradients are projected onto an outdated basis, losing critical information for optimization\. Ori is designed to directly counteract this penalty by replacing infrequent recomputation with a continuous, online tracking mechanism\.

#### The Mechanism: Online Subspace Tracking\.

The challenge of staleness in subspace projection has motivated online methods that update projectors frequently without relying on full SVD\. Recent work exploresauxiliary optimization techniquesfor dynamic subspace learning\[[15](https://arxiv.org/html/2610.22794#bib.bib32),[23](https://arxiv.org/html/2610.22794#bib.bib31),[28](https://arxiv.org/html/2610.22794#bib.bib2),[27](https://arxiv.org/html/2610.22794#bib.bib4)\]\.[Nguyen and Nguyen \[23\]](https://arxiv.org/html/2610.22794#bib.bib31)propose Subset\-Norm and Subspace\-Momentum, which decouples optimization by applying momentum in a low\-dimensional subspace while performing SGD in the orthogonal complement\.[Robert et al\. \[28\]](https://arxiv.org/html/2610.22794#bib.bib2)use block power iteration on a weighted gradient\-EMA combination to adapt optimizer states to evolving subspaces\.[Rajabi et al\. \[27\]](https://arxiv.org/html/2610.22794#bib.bib4)track subspaces on the Grassmannian manifold via geodesic updates, leveraging past estimates and error feedback for accuracy\.[Liang et al\. \[15\]](https://arxiv.org/html/2610.22794#bib.bib32)propose a robust procedure using a secondary optimizer to perform gradient descent directly on the subspace manifold, initialized from the previous projector\.

Ori adopts a fundamentally distinct strategy\. Rather than relying on a secondary optimizer, it leverages the streamingkk\-PCA algorithm,Oja\+\+\[[1](https://arxiv.org/html/2610.22794#bib.bib34)\], to update the projector\. This distinction carries significant implications\. First, gradient descent on the subspace manifold is inherently non\-convex and complex, but Oja\-style multiplicative updates come with strong theoretical assurances\. Specifically, Oja\+\+ achieves a provably “gap\-free” convergence rate, offering greater robustness in static settings compared to gradient\-based methods\. Second, as formalized in our theoretical contributions, our Oja\-based update exhibits desirable properties even in non\-stationary, drifting environments, yielding strong dynamic regret bounds\. Finally, the Oja\+\+ update is computationally cheaper than gradient descent on the manifold, enabling Ori to update the projector more frequently\. This higher frequency allows for improved tracking of the evolving gradient landscape\.

## 3Ori Optimizer

In this section, we further discuss the technical foundation of Ori\. First, we establish and quantify the problem of gradient subspace drift, highlighting the limitations of existing approaches\. Next, we introduce the core motivation of bootstrapping subspace updates, which forms the conceptual basis of our method\. We then detail the specific online algorithm, a modified Oja\+\+, used to implement this solution\. Finally, we present the complete Ori algorithm, synthesizing these components into a cohesive and efficient optimization procedure\.

### 3\.1The Phenomenon of Gradient Subspace Drift

#### Formalizing and Quantifying Drift\.

Low\-rank optimizers that rely on periodic updates, such as GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], implicitly assume that the principal gradient subspace remains relatively stable between updates\. In this work, we propose to explicitly characterize this evolution\. To do so, we adopt a formal metric from prior work on the changing geometry of optimization trajectories: the*canonical distance measure*\[[7](https://arxiv.org/html/2610.22794#bib.bib27)\]\. At any training steptt, the top\-kkprincipal gradient directionsUk\(t\)U^\{\(t\)\}\_\{k\}, which can be found via SVD of the gradient matrixGt=U\(t\)​Σ​V\(t\)⊤G\_\{t\}=U^\{\(t\)\}\\Sigma\{V^\{\(t\)\}\}^\{\\top\}, span a subspace represented by the orthogonal projectorPt=Uk\(t\)​Uk\(t\)⊤P\_\{t\}=U^\{\(t\)\}\_\{k\}\{U^\{\(t\)\}\_\{k\}\}^\{\\top\}\. Our notion of gradient subspace overlap, based on the canonical distance between the gradient subspace at stepttand a later stept\+Δ​tt\+\\Delta t, is then defined as

overlap​\(Uk\(t\),Uk\(t\+Δ​t\)\)≡Tr​\(Pt​Pt\+Δ​t\)Tr​\(Pt\)​Tr​\(Pt\+Δ​t\)=1−‖Pt−Pt\+Δ​t‖F22​k∝‖Pt−Pt\+Δ​t‖F2\.\\displaystyle\\text\{overlap\}\\left\(U^\{\(t\)\}\_\{k\},U^\{\(t\+\\Delta t\)\}\_\{k\}\\right\)\\equiv\\frac\{\\text\{Tr\}\(P\_\{t\}P\_\{t\+\\Delta t\}\)\}\{\\sqrt\{\\text\{Tr\}\(P\_\{t\}\)\\text\{Tr\}\(P\_\{t\+\\Delta t\}\)\}\}=1\-\\frac\{\\left\\\|P\_\{t\}\-P\_\{t\+\\Delta t\}\\right\\\|\_\{F\}^\{2\}\}\{2k\}\\propto\\left\\\|P\_\{t\}\-P\_\{t\+\\Delta t\}\\right\\\|\_\{F\}^\{2\}\.\(1\)This metric,444[Equation1](https://arxiv.org/html/2610.22794#S3.E1)is well\-defined because the projectorPPis idempotent \(P2=PP^\{2\}=P\) and its trace equals its rankTr​\(P\)=k\\text\{Tr\}\(P\)=k; we have‖Pt−Pt\+Δ​t‖F2=Tr​\(Pt\)\+Tr​\(Pt\+Δ​t\)−2​Tr​\(Pt​Pt\+Δ​t\)=2​k−2​Tr​\(Pt​Pt\+Δ​t\)\\left\\\|P\_\{t\}\-P\_\{t\+\\Delta t\}\\right\\\|\_\{F\}^\{2\}=\\text\{Tr\}\(P\_\{t\}\)\+\\text\{Tr\}\(P\_\{t\+\\Delta t\}\)\-2\\text\{Tr\}\(P\_\{t\}P\_\{t\+\\Delta t\}\)=2k\-2\\text\{Tr\}\(P\_\{t\}P\_\{t\+\\Delta t\}\)\.which ranges from 0 \(orthogonal\) to 1 \(identical\), is directly related to the geometric overlap between two subspaces\. Specifically,Tr​\(Pt​Pt\+Δ​t\)\\text\{Tr\}\(P\_\{t\}P\_\{t\+\\Delta t\}\)captures thekkprincipal angles,\{θ1,…​θk\}\\\{\\theta\_\{1\},\\dots\\theta\_\{k\}\\\}, between subspacesPtP\_\{t\}andPt\+Δ​tP\_\{t\+\\Delta t\}, in that

Tr​\(Pt​Pt\+Δ​t\)=∑i=1kcos2⁡θi≤k\.\\displaystyle\\text\{Tr\}\(P\_\{t\}P\_\{t\+\\Delta t\}\)=\\sum\_\{i=1\}^\{k\}\\cos^\{2\}\{\\theta\_\{i\}\}\\leq k\.Thus, it provides a principled tool for empirically measuring the rate and nature of subspace evolution\.

#### Empirical Evidence of Slow Drift\.

Figure 2:Average gradient subspace overlap during pre\-training\.We plot the overlap \([Equation1](https://arxiv.org/html/2610.22794#S3.E1)\) between subspaces at stepttandt\+Δ​tt\+\\Delta t\(withΔ​t=10\\Delta t=10\), averaged over a 60M LLaMA model\. The high, stable overlap, significantly above a random baseline, confirms that the subspace evolves gradually\.With the subspace overlap metric established, we empirically tested the stability assumption by measuring the overlap \(withΔ​t=10\\Delta t=10\) throughout the pre\-training of a 60M LLaMA model via the Adam optimizer on the C4 dataset \(further experiment details can be found in Appendix[E](https://arxiv.org/html/2610.22794#A5)\)\. The results, shown in Figure[2](https://arxiv.org/html/2610.22794#S3.F2), provide clear evidence for a “slow drift” hypothesis\. After an initial transient phase, the average gradient subspace overlap across all tested ranks \(k∈\{8,32,128\}k\\in\\\{8,32,128\\\}\) stabilizes at a high value between 0\.55 and 0\.75\. Crucially, this is significantly higher than the overlap observed for random subspaces of the same rank, confirming the existence of meaningful temporal structure\. This indicates: \(i\)The static subspace assumption is false, as the overlap is consistently below 1\.0, and the change over just 10 steps is substantial\. \(ii\)The evolution is not erratic, as the overlap remains remarkably stable, suggesting the subspace moves with a predictable “velocity” rather than teleporting randomly\. \(iii\)The phenomenon is fundamental, as the stable drift is present across a range of ranks, suggesting it is an intrinsic property of the optimization trajectory\. This temporal coherence is precisely the property that an online tracking algorithm can exploit\.

### 3\.2Online Subspace Tracking via Oja\+\+ with Moment Transfer

\{wrapfloat\}

algorithmR0\.52

Algorithm 1Ori Projector Update \(Oja\+\+\)0:

Q∈ℝd×rQ\\in\\mathbb\{R\}^\{d\\times r\}, gradient

G∈ℝd×nG\\in\\mathbb\{R\}^\{d\\times n\},

HojaH\_\{\\text\{oja\}\}:

Nupdates,ρ,NvecsN\_\{\\text\{updates\}\},\\rho,N\_\{\\text\{vecs\}\}, LR schedule for

η\\eta
1:FunctionUpdate\-Projector\(Q,G,Hoja\)\(Q,G,H\_\{\\text\{oja\}\}\)

2:

Qinit←ρ​Q\+\(1−ρ\)​Randn​\(d,r\)Q\_\{\\text\{init\}\}\\leftarrow\\rho Q\+\(1\-\\rho\)\\mathrm\{Randn\}\(d,r\)\#[Equation3](https://arxiv.org/html/2610.22794#S3.E3)

3:

Qnew←𝟎d×rQ\_\{\\text\{new\}\}\\leftarrow\\mathbf\{0\}\_\{d\\times r\}
4:

rcurr←0r\_\{\\text\{curr\}\}\\leftarrow 0
5:\# Oja\+\+ main update loop\[[1](https://arxiv.org/html/2610.22794#bib.bib34)\]

6:for

k=0k=0to

log2⁡\(r\)−1\\log\_\{2\}\(r\)\-1do

7:

b←r/2k\+1b\\leftarrow r/2^\{k\+1\}\# One block of Oja\+\+ updates

8:

rnext←rcurr\+br\_\{\\text\{next\}\}\\leftarrow r\_\{\\text\{curr\}\}\+b
9:

Qnew\(:,rcurr:rnext\)←Qinit\(:,rcurr:rnext\)Q\_\{\\text\{new\}\}\(:,r\_\{\\text\{curr\}\}\{:\}r\_\{\\text\{next\}\}\)\\leftarrow Q\_\{\\text\{init\}\}\(:,r\_\{\\text\{curr\}\}\{:\}r\_\{\\text\{next\}\}\)
10:

Qslice←Qnew\(:,0:rnext\)Q\_\{\\text\{slice\}\}\\leftarrow Q\_\{\\text\{new\}\}\(:,0\{:\}r\_\{\\text\{next\}\}\)\# Current active slice

11:

Qslice←Oja\-Step​\(Qslice,G,Nupdates,Nvecs\)Q\_\{\\text\{slice\}\}\\leftarrow\\textsc\{Oja\-Step\}\(Q\_\{\\text\{slice\}\},G,N\_\{\\text\{updates\}\},N\_\{\\text\{vecs\}\}\)
12:

Qnew\(:,0:rnext\)←QsliceQ\_\{\\text\{new\}\}\(:,0\{:\}r\_\{\\text\{next\}\}\)\\leftarrow Q\_\{\\text\{slice\}\}
13:

rcurr←rnextr\_\{\\text\{curr\}\}\\leftarrow r\_\{\\text\{next\}\}
14:endfor

15:return

QnewQ\_\{\\text\{new\}\}
16:End Function

17:\# Subroutine: Oja updates, main update loop

18:FunctionOja\-Step\(Qin,G,Nupdates,Nvecs\)\(Q\_\{\\text\{in\}\},G,N\_\{\\text\{updates\}\},N\_\{\\text\{vecs\}\}\)

19:for

j=1j=1to

NupdatesN\_\{\\text\{updates\}\}do

20:

η←GetCosineAnnealingLR​\(j\)\\eta\\leftarrow\\textsc\{GetCosineAnnealingLR\}\(j\)
21:

Gsub←SampleColumns​\(G,Nvecs\)G\_\{\\text\{sub\}\}\\leftarrow\\textsc\{SampleColumns\}\(G,N\_\{\\text\{vecs\}\}\)
22:

Qin←Qin\+η​Gsub​\(Gsub⊤​Qin\)Q\_\{\\text\{in\}\}\\leftarrow Q\_\{\\text\{in\}\}\+\\eta G\_\{\\text\{sub\}\}\(G\_\{\\text\{sub\}\}^\{\\top\}Q\_\{\\text\{in\}\}\)\#[Equation2](https://arxiv.org/html/2610.22794#S3.E2)

23:endfor

24:return

QinQ\_\{\\text\{in\}\}
25:End Function

The empirical validation of slow, structured subspace drift motivates a paradigm shift from periodic, expensiverecomputation or adaption, to continuous, lightweightonline tracking, leading naturally to the use of streamingkk\-PCA algorithms, for which Oja’s method provides a classic foundation\[[24](https://arxiv.org/html/2610.22794#bib.bib3)\]\. Its core is a simple multiplicative update that incrementally rotates the current basisQt−1Q\_\{t\-1\}with a new data samplextx\_\{t\}\(in our case, derived from the gradient matrixGtG\_\{t\}\) to better align with new information inxtx\_\{t\}, followed by a re\-orthogonalization step \(typically a QR decomposition\) to maintain the basis properties\. The fundamental update equation is:

Qt←QR​\(\(I\+ηt​xt​xt⊤\)​Qt−1\),Q\_\{t\}\\leftarrow\\text\{QR\}\\left\(\(I\+\\eta\_\{t\}x\_\{t\}x\_\{t\}^\{\\top\}\)Q\_\{t\-1\}\\right\),\(2\)whereηt\\eta\_\{t\}is a learning rate \(typically controlled by a cosine scheduler\) and QR denotes the QR decomposition used for re\-orthogonalization\. While standard Oja’s algorithm initializes the process with a random matrix, our key innovation is to explicitly leverage the temporal coherence of the subspace drift\. We introduce a bootstrapping mechanism that uses the final projector from the previous update step,Qt−1Q\_\{t\-1\}, as a warm start for the current one\. This is controlled by a drift coefficientρ\\rho, which interpolates between the previous subspace and a new random Gaussian matrixR∼𝒩⁡\(0,1\)R\\sim\{\\mathcal\{N\}\}\(0,1\):

Qinit=ρ​Qt−1\+\(1−ρ\)​R\.Q\_\{\\text\{init\}\}=\\rho Q\_\{t\-1\}\+\(1\-\\rho\)R\.\(3\)This matrixQinitQ\_\{\\text\{init\}\}then serves as the next starting point for the update rule in Equation[2](https://arxiv.org/html/2610.22794#S3.E2)\.

While[Equation2](https://arxiv.org/html/2610.22794#S3.E2)and[Equation3](https://arxiv.org/html/2610.22794#S3.E3)form the conceptual basis, we specifically employ a more robust variant, Oja\+\+\[[1](https://arxiv.org/html/2610.22794#bib.bib34)\], for its crucial theoretical advantage ofgap\-free convergence\. This means its convergence rate is independent of the eigengap \(the difference between consecutive singular values of the gradient covariance matrix\), a property vital for LLM training where such gaps can be small\. The bootstrapping technique applies directly to Oja\+\+ as well: the previous basisQt−1Q\_\{t\-1\}serves as the warm start for the multi\-stage Oja\+\+ update\. This combination of a theoretically sound, gap\-free algorithm with a drift\-aware bootstrapping strategy gives rise to the full form of our proposed optimizer, Ori, whose core projector update routine is detailed in[Algorithm1](https://arxiv.org/html/2610.22794#alg1)\. For the algorithm pseudocode of Ori with AdamW, we refer the readers to[Algorithm2](https://arxiv.org/html/2610.22794#alg2)in Appendix[C](https://arxiv.org/html/2610.22794#A3)\. In particular, we provide an amortized runtime complexity analysis of Ori in Appendix[C\.3](https://arxiv.org/html/2610.22794#A3.SS3)\.

#### Applying Ori on Other Optimizers\.

To demonstrate that the design principles behind Ori can be applied to specialized, state\-of\-the\-art low\-rank optimizers other than GaLore, we examine LDAdam\[[28](https://arxiv.org/html/2610.22794#bib.bib2)\]\. LDAdam enhances the standard Adam optimizer by employing a projection\-aware, momentum\-based update alongside a generalized error\-feedback mechanism\. This design aligns with findings from MicroAdam\[[21](https://arxiv.org/html/2610.22794#bib.bib1)\], which showed that applying error feedback at every iteration is essential for recovering Adam\-like convergence at million\- and billion\-parameter scales \(e\.g\., BERT/LLaMA\)\. These techniques significantly enhance language model pre\-training performance in extremely low\-rank settings \(e\.g\., rank 32 with an embedding size of 2048\)\. However, LDAdam faces a critical barrier to practical deployment: it requires updating the projector at every training step \(T=1T=1\) to maintain performance, incurring prohibitive computational costs\. As shown in our later empirical results \([Table2](https://arxiv.org/html/2610.22794#S5.T2)\), when the update interval is increased toT=200T=200– matching the standard schedule of GaLore – LDAdam’s performance degrades significantly\. We attribute this drop to LDAdam’s reliance on a simple single\-step power iteration method for subspace estimation, which we hypothesize is insufficient for tracking the drifting gradient subspace over large update intervals\. Inspired by Ori, we introduce a multi\-step stochastic power iteration mechanism into LDAdam’s update routine\. We refer to this improved optimizer as O\-LDAdam, reflecting its infusion of Ori’s online subspace tracking into LDAdam’s error\-correcting framework\. We provide further algorithmic details in Appendix[D](https://arxiv.org/html/2610.22794#A4)\.

## 4Dynamic Regret Analysis with Oja

Our theoretical analysis departs from the classical static view of streamingkk\-PCA: rather than asking whether an online method converges to a fixed principal subspace, we ask whether it can track a drifting low\-rank comparator under the notion of concept drift\[[3](https://arxiv.org/html/2610.22794#bib.bib35)\]\. To this end, we aim to establishdynamic regret bounds\.

To formalize this question, we consider a nonstationary environment segmented intoSSsteps, each comprisingTTsequential updates\. Letxζ∈ℝdx\_\{\\zeta\}\\in\\mathbb\{R\}^\{d\}denote the streaming data at unified time indexζ∈\[S​T\]\\zeta\\in\[ST\], whereζ=\(s−1\)​T\+t\\zeta=\(s\-1\)T\+trefers to updatet∈\[T\]t\\in\[T\]inside steps∈\[S\]s\\in\[S\]\. We assume eachxζx\_\{\\zeta\}is drawn i\.i\.d from an unknown underlying distribution𝒟\{\\mathcal\{D\}\}\. Let the step\-ssoptimalkk\-subspace be spanned byVs∈ℝd×kV\_\{s\}\\in\\mathbb\{R\}^\{d\\times k\}with orthonormal columns\. The dynamic comparator is a sequence of rank\-kkorthogonal projectors\{Ps\}s=1S\\\{P\_\{s\}\\\}\_\{s=1\}^\{S\}which isPs=Vs​Vs⊤P\_\{s\}=V\_\{s\}V\_\{s\}^\{\\top\}with*bounded drift*across consecutive steps:

‖Ps−Ps\+1‖F≤λ,s∈\[S−1\]\.\\left\\\|P\_\{s\}\-P\_\{s\+1\}\\right\\\|\_\{F\}\\;\\leq\\;\\lambda,\\qquad s\\in\[S\-1\]\.We now instantiate the analysis for multiplicative Oja updates on an explicit basis\. The learner maintains an orthonormal basisQ\(ζ\)∈ℝd×kQ^\{\(\\zeta\)\}\\in\\mathbb\{R\}^\{d\\times k\}and updates with a noisy perturbation followed by reorthonormalization:

Q\(ζ\+1\)=QR⁡\(\(I\+η​g^ζ\)​Q\(ζ\)\),g^ζ=x^ζ​x^ζ⊤,x^ζ=xζ\+yζ,\\displaystyle Q^\{\(\\zeta\+1\)\}\\;=\\;\\mathrm\{QR\}\\\!\\left\(\\big\(I\+\\eta\\,\\hat\{g\}\_\{\\zeta\}\\big\)\\,Q^\{\(\\zeta\)\}\\right\),\\qquad\\hat\{g\}\_\{\\zeta\}\\;=\\;\\hat\{x\}\_\{\\zeta\}\\hat\{x\}\_\{\\zeta\}^\{\\top\},\\qquad\\hat\{x\}\_\{\\zeta\}\\;=\\;x\_\{\\zeta\}\+y\_\{\\zeta\},\(4\)whereyζy\_\{\\zeta\}is an additive \(possibly adversarial\) noise term\. At eachζ\\zeta, the learner outputs a rank\-kkprojectorP\(ζ\)P^\{\(\\zeta\)\}, an orthonormal basisQ\(ζ\)∈ℝd×kQ^\{\(\\zeta\)\}\\in\\mathbb\{R\}^\{d\\times k\}withP\(ζ\)=Q\(ζ\)​\(Q\(ζ\)\)⊤P^\{\(\\zeta\)\}=Q^\{\(\\zeta\)\}\(Q^\{\(\\zeta\)\}\)^\{\\top\}\. As in\[[19](https://arxiv.org/html/2610.22794#bib.bib36)\], we measure performance by the \(variance\) regret against the stepwise comparator:

RS:=∑ζ=1S​T\(xζ⊤​Ps⁡\(ζ\)​xζ−xζ⊤​P\(ζ\)​xζ\)=∑ζ=1S​T⟨Ps⁡\(ζ\)−P\(ζ\),xζ​xζ⊤⟩,\\displaystyle R\_\{S\}\\;:=\\;\\sum\_\{\\zeta=1\}^\{ST\}\\\!\\Big\(x\_\{\\zeta\}^\{\\top\}P\_\{s\(\\zeta\)\}x\_\{\\zeta\}\-x\_\{\\zeta\}^\{\\top\}P^\{\(\\zeta\)\}x\_\{\\zeta\}\\Big\)\\;=\\;\\sum\_\{\\zeta=1\}^\{ST\}\\\!\\langle P\_\{s\(\\zeta\)\}\-P^\{\(\\zeta\)\},x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\\rangle,\(5\)wheres⁡\(ζ\)=⌈ζ/T⌉s\(\\zeta\)=\\lceil\\zeta/T\\rceil\. Since‖x‖22=‖x−P​x‖22\+x⊤​P​x\\left\\\|x\\right\\\|\_\{2\}^\{2\}=\\left\\\|x\-Px\\right\\\|\_\{2\}^\{2\}\+x^\{\\top\}Pxfor any projectorPP, maximizing captured variance is equivalent to minimizing squared residual—so[Equation5](https://arxiv.org/html/2610.22794#S4.E5)is also the residual regret\.

Establishing this measure of performance leads us to the following guarantee for bounding the residual regret in this setting, the proof of which is found in Appendix[A\.3](https://arxiv.org/html/2610.22794#A1.SS3)\.

###### Theorem 1\(Noisy Oja underλ\\lambda\-drift\)\.

Forζ∈\[S​T\]\\zeta\\in\[ST\], letQ\(ζ\)Q^\{\(\\zeta\)\}follow[Equation4](https://arxiv.org/html/2610.22794#S4.E4)withη=k2\+\(S−1\)​k​λk​d⋅S​T\\eta=\\sqrt\{\\frac\{\\frac\{k\}\{2\}\+\(S\-1\)\\sqrt\{k\}\\lambda\}\{kd\\cdot ST\}\}, letEtot:=∑ζ=1S​T‖Eζ‖2E\_\{\\mathrm\{tot\}\}:=\\sum\_\{\\zeta=1\}^\{ST\}\\\|E\_\{\\zeta\}\\\|\_\{2\}whereEζ:=xζ​yζ⊤\+yζ​xζ⊤\+yζ​yζ⊤E\_\{\\zeta\}:=x\_\{\\zeta\}y\_\{\\zeta\}^\{\\top\}\+y\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\+y\_\{\\zeta\}y\_\{\\zeta\}^\{\\top\}, and suppose‖xζ‖2≤1\\left\\\|x\_\{\\zeta\}\\right\\\|\_\{2\}\\leq 1\. Then we have

𝔼⁡\[RS\]≤2​k​d⋅S​T​\(k2\+\(S−1\)​k​λ\)\+2​k​Etot\.\\mathbb\{E\}\[R\_\{S\}\]\\leq 2\\sqrt\{kd\\cdot ST\\\!\\left\(\\tfrac\{k\}\{2\}\+\(S\-1\)\\sqrt\{k\}\\,\\lambda\\right\)\}\+2k\\,E\_\{\\mathrm\{tot\}\}\.\(6\)

We note in the noiseless setting, i\.e\.,Etot=0E\_\{\\mathrm\{tot\}\}=0, we achieve sublinear regret in terms of the totalS​TSTupdates\. Additionally, our results establish a natural balance in the dependence between the number of steps and sequential updates, with the dependence of the regret onSSdegrading asλ\\lambdaincreases\.

## 5Experiments

We aim to demonstrate that Ori: \(i\) achieves superior performance against existing low\-rank optimizers in both pre\-training and fine\-tuning, particularly under extremely low rank scenarios; \(ii\) improves memory and computational efficiency; and \(iii\) derives its benefits from its core design principles, as shown through ablation studies\.

### 5\.1Baselines and Experimental Setup

We evaluate Ori against a consistent set of state\-of\-the\-art optimizers, all built upon the Adam optimizer\. For each baseline, we adopt the hyperparameter configurations recommended in their respective original publications and official codebases\. Full experimental details are provided in Appendix[E](https://arxiv.org/html/2610.22794#A5)\. Our baselines include:Full\-Rank Adam, the standard optimizer serving as a practical upper bound on performance \(using defaultβ1=0\.9,β2=0\.999\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.999, andϵ=10−8\\epsilon=10^\{\-8\}\);Weight\-Subspace Methods, represented byLoRA\[[10](https://arxiv.org/html/2610.22794#bib.bib26)\]for fine\-tuning \(with a LoRA alpha of 32 and a dropout of 0\.05\) and its pre\-training variantReLoRA\[[14](https://arxiv.org/html/2610.22794#bib.bib25)\]\(with optimizer states reset every 1000 steps and no full\-rank warmup\); andGradient\-Subspace Methods\. This category includesGaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], which employs periodic SVD \(update frequencyT=200T=200; pre\-training scaleα=0\.25\\alpha=0\.25; fine\-tuning scaleα∈\{2,4\}\\alpha\\in\\\{2,4\\\}\), andOSD \(Online Subspace Descent\)\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\], another state\-of\-the\-art method that performs gradient descent directly on the subspace manifold \(similarly usingT=200T=200, pre\-trainingα=0\.25\\alpha=0\.25, and fine\-tuningα∈\{2,4\}\\alpha\\in\\\{2,4\\\}\)\. Finally,LDAdam\[[28](https://arxiv.org/html/2610.22794#bib.bib2)\], which update its projection matrix via power iteration, alongside orthogonal techniques such as an error feedback mechanism and momentum\-based state updates\. Although the original LDAdam was designed exclusively for most frequent per\-step updates \(T=1T=1\), for the purpose of this work, we adapted it also to the efficient delayed update setting \(T=200T=200\)\. For LDAdam’s other hyperparameters, we use the values suggested by the authors \(interpolation factorρ=0\.908\\rho=0\.908, first step projection matrix initialized via SVD\)\.

Table 1:Comparison with full\- and low\-rank optimizers on pre\-training various sizes of LLaMA models on C4 dataset\. Validation perplexity is reported \(lower↓\\downarrowis better\)\.kkis the rank for low\-rank optimizers, anddmodeld\_\{\\text\{model\}\}is the model embedding dimension\. We show the best scores in bold and second best ones with underlines\.Ori is on\-par with the best baseline on moderately high rank settings, and achieves superior performance when the rankkkis severely constrained\.Table 2:Comparison of O\-LDAdam versus LDAdam\[[28](https://arxiv.org/html/2610.22794#bib.bib2)\]with different subspace projection update frequencies \(T∈\{1,200\}T\\in\\\{1,200\\\}\) when pretraining LLaMA on C4 dataset\. O\-LDAdam improves LDAdam’s single\-step power iteration update with Ori\-inspired multi\-step, stochastic power iteration update\.In the very low\-rank regimes \(e\.g\.r=32r=32whendmodel=2048d\_\{\\text\{model\}\}=2048\), O\-LDAdam with update frequencyT=200T=200achieves on\-par or even superior performance to LDAdam with per\-step updates \(T=1T=1\)\.Table 3:Comparison with full\- and low\-rank optimizers on fine\-tuning Roberta\-Base on GLUE dataset\. Average accuracy score and standard deviation is reported \(lower↓\\downarrowis better\) for each task, optimizer, and rank combination over three random seeds\. We also report average accuracy over all tasks for each method\. We show the best scores in bold and second best ones with underlines\.Ori consistently achieves superior performance on 10 out of 16 scenarios\.
### 5\.2Main results: LLaMA pre\-training

We evaluate Ori’s pre\-training performance against the baselines on the C4 dataset\[[26](https://arxiv.org/html/2610.22794#bib.bib37)\]using LLaMA\-architecture models of four sizes \(60M, 130M, 350M, and 1B\), following similar experimental settings as prior work\[[14](https://arxiv.org/html/2610.22794#bib.bib25),[39](https://arxiv.org/html/2610.22794#bib.bib33)\]\. Departing from prior work that often tests a single rank, we systematically evaluate all low\-rank methods across three distinct ranks for each model \(e\.g\.,r∈\{512,128,32\}r\\in\\\{512,128,32\\\}for the 1B model\), enabling a rigorous assessment of optimizer performance under varying degrees of memory constraints\. To ensure a fair comparison, model hyperparameters were held constant for each model size, while each optimizer’s learning rate was tuned from\{0\.01,0\.005,0\.001,0\.0005,0\.0001\}\\\{0\.01,0\.005,0\.001,0\.0005,0\.0001\\\}\. We report validation perplexity after a fixed token budget, with full details provided in Appendix[E](https://arxiv.org/html/2610.22794#A5)\.

As shown in[Table1](https://arxiv.org/html/2610.22794#S5.T1), Ori’s performance is consistently strong, particularly in memory\-constrained scenarios\. At higher, less restrictive ranks \(e\.g\.,r=128r=128for the 60M model,r=256r=256for the 130M model\), Ori performs on par with GaLore, establishing it as a highly competitive method in common low\-rank settings\. However, Ori’s distinct advantage emerges when the rankrris severely constrained, where it demonstrates the most robust performance as its accuracy degrades the least under memory pressure\. This trend, where the cost of a stale subspace projection becomes most apparent, is consistent across all model scales\. For instance, when training the 130M model at an aggressively low rank of just 16, Ori achieves a validation perplexity of 43\.61, substantially outperforming GaLore \(60\.29\) and OSD \(54\.05\)\. This result empirically validates our central hypothesis: actively tracking the gradient subspace drift is most critical when the optimizer has the fewest dimensions to work with\. By excelling in these challenging low\-rank regimes, Ori consistently narrows the performance gap to full\-rank Adam, pushing the performance boundaries of memory\-efficient optimization\.

#### Generalizing to Error\-Feedback Optimizers \(O\-LDAdam\)\.

To test the generality of our subspace tracking approach, we benchmarked against LDAdam, a stronger baseline that utilizes error feedback and momentum buffering\. While LDAdam achieves impressive performance when updating the projector at every step \(T=1T=1\), this comes at a high computational cost\. As shown in Table 2, simply increasing the update interval toT=200T=200\(matching GaLore’s efficiency\) causes LDAdam’s performance to collapse \(e\.g\., perplexity spikes from 20\.27 to 45\.66 on LLaMA\-1B atr=32r=32\)\. However, by replacing LDAdam’s simple power iteration with our stochastic power iteration \(O\-LDAdam\), we effectively bridge this gap\. O\-LDAdam \(T=200T=200\) not only recovers the performance of the expensive LDAdam \(T=1T=1\) baseline but frequently surpasses it in extremely low\-rank regimes\. Notably, on the 1B model at rank 32, O\-LDAdam achieves a perplexity of 15\.66, significantly outperforming the vanilla LDAdam \(T=1T=1\) score of 20\.27, all while maintaining the computationally efficientT=200T=200schedule\. This confirms that the subspace tracking principles of Ori are orthogonal to error\-feedback mechanisms and can be composed to create state\-of\-the\-art low\-rank optimizers\.

### 5\.3Finetuning

To evaluate Ori’s effectiveness on adaptation tasks, we fine\-tuned a RoBERTa\-Base model on the GLUE benchmark\[[31](https://arxiv.org/html/2610.22794#bib.bib38)\]\. For each task, we report the mean and standard deviation of task\-specific metrics over three runs with different random seeds\. Further details on evaluation metrics are provided in Appendix[E](https://arxiv.org/html/2610.22794#A5)\.

The fine\-tuning results in Table 3 underscore Ori’s prominent performance\. Across both rank settings, Ori achieves the highest average score among all low\-rank methods\. At rank\-8, Ori’s average score of 86\.30 not only surpasses GaLore \(84\.34\) and LoRA \(79\.68\) but also exceeds the full\-rank Adam baseline \(85\.82\)\. This strong performance continues at rank\-4, where Ori \(85\.61\) again leads all low\-rank optimizers except for LDAdam withT=1T=1\. In particular, Ori achieves the top score in 10 out of the 16 total task/rank combinations\. Its particularly strong advantage on challenging tasks like CoLA \(63\.58 vs\. GaLore’s 58\.24 at r=8\) further validates its robustness and demonstrates that its drift\-aware optimization is highly effective for both pre\-training and adaptation tasks\.

### 5\.4Ablation Study

\(a\)Impact of Drift Coefficient \(ρ\\rho\) on final evaluation loss\. Lower↓\\downarrowis better\.\(b\)Impact of Update Frequency \(TperiodT\_\{\\text\{period\}\}\) on final evaluation loss\. Lower↓\\downarrowis better\.\(c\)Oja\+\+ vs\. Standard Oja \(Perplexity\)\.

Figure 3:Ablation Studies on the 60M LLaMA model pre\-trained on C4 dataset for 0\.44B tokens\. \([3\(a\)](https://arxiv.org/html/2610.22794#S5.F3.sf1)\)Varying the drift coefficientρ\\rhoshows peak performance aroundρ=0\.99\\rho=0\.99, validating the importance of transferring optimizer statistics\. \([3\(b\)](https://arxiv.org/html/2610.22794#S5.F3.sf2)\)Performance degrades as the update frequencyTperiodT\_\{\\text\{period\}\}increases, demonstrating the cost of a stale projector\. \([3\(c\)](https://arxiv.org/html/2610.22794#S5.F3.sf3)\)Oja\+\+ consistently outperforms the standard Oja algorithm across all ranks\.To isolate the contributions of Ori’s core design components, we conduct a series of ablation studies, performed via the 60M LLaMA model pre\-trained on the C4 dataset \([Section5\.2](https://arxiv.org/html/2610.22794#S5.SS2)\)\. Details about the ablation experiment settings can be found in Appendix[E\.7](https://arxiv.org/html/2610.22794#A5.SS7)\.

#### Impact of the Drift Coefficient \(ρ\\rho\)\.

We first analyze the effect of the drift coefficientρ\\rho, which controls how much of the previous step’s projector is carried over to initialize the current step’s projector\. As shown in[Figure3\(a\)](https://arxiv.org/html/2610.22794#S5.F3.sf1), we observe a clear and consistent trend across all ranks: a value ofρ=0\.99\\rho=0\.99yields the best performance\. This has two key implications\. First, the significant performance drop atρ=0\.0\\rho=0\.0\(a fully random restart\) corroborates our central claim that transferring information about the previous subspace is crucial for effective optimization\. Second, the slight performance dip atρ=1\.0\\rho=1\.0\(a direct carryover with no noise\) suggests that injecting a small amount of randomness is a beneficial regularization strategy\. Overall, we chooseρ=0\.99\\rho=0\.99for Ori in all pre\-training and fine\-tuning\.

#### Impact of Projector Update Frequency \(TT\)

We test our hypothesis that frequent, lightweight updates are superior to infrequent, heavy ones by varying the update period from 10 to 200 \(which matches GaLore’s recommended schedule\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\]\)\. As shown in[Figure3\(b\)](https://arxiv.org/html/2610.22794#S5.F3.sf2), more frequent updates consistently lead to lower loss \(better performance\)\. This phenomenon is especially prominent in the extremely low\-rank setting \(r=8r=8\), where the cost of using a stale projector is most severe\. This finding contrasts sharply with the ablation results from GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], in which the authors reported no benefit from more frequent GaLore SVD updates\. We hypothesize this is a unique advantage of Ori: because GaLore’s periodic SVD acts as a “cold restart” that does not carry over subspace statistics, frequent updates may not be as beneficial\. In contrast, since Ori is explicitly designed to track subspace drift, it can effectively leverage more frequent updates to maintain a higher\-fidelity representation of the gradient landscape\. For our main experiments, we setT=20T=20, as it strikes an effective balance between model performance and computational overhead\.

#### Oja\+\+ vs\. Standard Oja\.

To empirically justify choosing Oja\+\+, we compare the performance of Oja\+\+ against the standard Oja algorithm as the subroutine for updating the gradient projector\. The results in[Figure3\(c\)](https://arxiv.org/html/2610.22794#S5.F3.sf3)are unequivocal: Oja\+\+ consistently outperforms standard Oja across all tested ranks\. This aligns with prior theoretical work showing Oja\+\+ has a provably better convergence rate in static streamingkk\-PCA settings\[[1](https://arxiv.org/html/2610.22794#bib.bib34)\]\. Our results provide new empirical evidence that this advantage extends to the dynamic, non\-stationary environment of LLM training, where Oja\+\+’s superior tracking ability translates directly to better model performance\.

#### Memory and Computation Time\.

Finally, aside from the theoretical runtime complexity analysis in Appendix[C\.3](https://arxiv.org/html/2610.22794#A3.SS3), we provide an empirical memory and computation time comparison in Appendix[F\.1](https://arxiv.org/html/2610.22794#A6.SS1)\.

## 6Conclusion

We introduced Ori, a drift\-aware low\-rank optimizer that continuously tracks the gradient subspace with streamingkk\-PCA and preserves optimizer statistics across projector updates, mitigating staleness while keeping memory and compute low\. Across theory \(dynamic\-regret under drift\) and experiments, Ori consistently narrows the gap to full\-rank training in the extremely low\-rank regime while remaining faster and more memory\-efficient than prior gradient\-projection methods\. We discuss limitations of our work and directions for future research, including behavior under heterogeneous training data and explicit optimizer state rotation across basis changes, in Appendix[G](https://arxiv.org/html/2610.22794#A7)\.

## References

- \[1\]\(2017\)First efficient convergence for streaming k\-pca: a global, gap\-free, and near\-optimal rate\.In2017 IEEE 58th Annual Symposium on Foundations of Computer Science \(FOCS\),pp\. 487–492\.Cited by:[§A\.2](https://arxiv.org/html/2610.22794#A1.SS2.p2.4.1),[§1](https://arxiv.org/html/2610.22794#S1.p5.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px2.p2.1),[§3\.2](https://arxiv.org/html/2610.22794#S3.SS2.p4.1),[§5\.4](https://arxiv.org/html/2610.22794#S5.SS4.SSS0.Px3.p1.1),[5](https://arxiv.org/html/2610.22794#algx1.l5.4)\.
- \[2\]X\. Chen, K\. Feng, C\. Li, X\. Lai, X\. Yue, Y\. Yuan, and G\. Wang\(2024\)Fira: can we achieve full\-rank training of llms under low\-rank constraint?\.arXiv preprint arXiv:2410\.01623\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1)\.
- \[3\]K\. Crammer, Y\. Mansour, E\. Even\-Dar, and J\. W\. Vaughan\(2010\)Regret minimization with concept drift\.\.InCOLT,pp\. 168–180\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p1.1),[§B\.3](https://arxiv.org/html/2610.22794#A2.SS3.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.22794#S4.p1.1)\.
- \[4\]A\. Das\(2024\)Natural galore: accelerating galore for memory\-efficient llm training and fine\-tuning\.arXiv preprint arXiv:2410\.16029\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1)\.
- \[5\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)Qlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§1](https://arxiv.org/html/2610.22794#S1.p1.1)\.
- \[6\]R\. J\. George, D\. Pitt, J\. Zhao, J\. Kossaifi, C\. Luo, Y\. Tian, and A\. Anandkumar\(2024\)Tensor\-galore: memory\-efficient training via gradient tensor decomposition\.InOPT 2024: Optimization for Machine Learning,Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1)\.
- \[7\]G\. Gur\-Ari, D\. A\. Roberts, and E\. Dyer\(2018\)Gradient descent happens in a tiny subspace\.arXiv preprint arXiv:1812\.04754\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p1.1),[§B\.3](https://arxiv.org/html/2610.22794#A2.SS3.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.22794#S3.SS1.SSS0.Px1.p1.1)\.
- \[8\]A\. Han, J\. Li, W\. Huang, M\. Hong, A\. Takeda, P\. K\. Jawanpuria, and B\. Mishra\(2024\)SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining\.Advances in Neural Information Processing Systems37,pp\. 118267–118295\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]Y\. He, P\. Li, Y\. Hu, C\. Chen, and K\. Yuan\(2024\)Subspace optimization for large language models with convergence guarantees\.arXiv preprint arXiv:2410\.11289\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.22794#S5.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.22794#S5.T3.9.1.10.1),[Table 3](https://arxiv.org/html/2610.22794#S5.T3.9.1.3.1)\.
- \[11\]S\. R\. Kamalakara, A\. Locatelli, B\. Venkitesh, J\. Ba, Y\. Gal, and A\. N\. Gomez\(2022\)Exploring low rank training of deep neural networks\.arXiv preprint arXiv:2209\.13569\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1)\.
- \[12\]B\. W\. Larsen, S\. Fort, N\. Becker, and S\. Ganguli\(2021\)How many degrees of freedom do we need to train deep networks: a loss landscape perspective\.arXiv preprint arXiv:2107\.05802\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p1.1),[§B\.3](https://arxiv.org/html/2610.22794#A2.SS3.SSS0.Px1.p1.1)\.
- \[13\]Z\. Li, Y\. Deng, P\. Zhong, M\. Razaviyayn, and V\. Mirrokni\(2025\)Pike: adaptive data mixing for large\-scale multi\-task learning under low gradient conflicts\.arXiv preprint arXiv:2502\.06244\.Cited by:[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px1.p1.1)\.
- \[14\]V\. Lialin, N\. Shivagunde, S\. Muckatira, and A\. Rumshisky\(2023\)Relora: high\-rank training through low\-rank updates\.arXiv preprint arXiv:2307\.05695\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.11.1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.17.1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.5.1.1),[Table 9](https://arxiv.org/html/2610.22794#A6.T9.5.1.10.1.1),[Table 9](https://arxiv.org/html/2610.22794#A6.T9.5.1.4.1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.22794#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.22794#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.22794#S5.T1.11.1.6.1)\.
- \[15\]K\. Liang, B\. Liu, L\. Chen, and Q\. Liu\(2024\)Memory\-efficient llm training with online subspace descent\.Advances in Neural Information Processing Systems37,pp\. 64412–64432\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1),[§B\.2](https://arxiv.org/html/2610.22794#A2.SS2.SSS0.Px1.p1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.13.1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.19.1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.7.1.1),[Table 9](https://arxiv.org/html/2610.22794#A6.T9.5.1.12.1.1),[Table 9](https://arxiv.org/html/2610.22794#A6.T9.5.1.6.1.1),[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p2.1),[§1](https://arxiv.org/html/2610.22794#S1.p3.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.22794#S5.SS1.p1.1),[Table 1](https://arxiv.org/html/2610.22794#S5.T1.11.1.8.1),[Table 3](https://arxiv.org/html/2610.22794#S5.T3.9.1.12.1),[Table 3](https://arxiv.org/html/2610.22794#S5.T3.9.1.5.1)\.
- \[16\]X\. Liao, S\. Li, Y\. Xu, Z\. Li, Y\. Liu, and Y\. He\(2024\)GaLore\+\+: boosting low\-rank adaptation for llms with cross\-head projection\.arXiv preprint arXiv:2412\.19820\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]S\. Loeschcke, M\. Toftrup, M\. Kastoryano, S\. Belongie, and V\. Snæbjarnarson\(2024\)LoQT: low\-rank adapters for quantized pretraining\.Advances in Neural Information Processing Systems37,pp\. 115282–115308\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p2.1)\.
- \[18\]X\. Lv, N\. Ding, K\. Zhang, E\. Hua, G\. Cui, and B\. Zhou\(2024\)Scalable efficient training of large language models with low\-dimensional projected attention\.arXiv preprint arXiv:2411\.02063\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1)\.
- \[19\]T\. V\. Marinov, P\. Mianjy, and R\. Arora\(2018\)Streaming principal component analysis in noisy setting\.InInternational Conference on Machine Learning,pp\. 3413–3422\.Cited by:[§4](https://arxiv.org/html/2610.22794#S4.p2.3)\.
- \[20\]Z\. Mo, L\. Huang, and S\. J\. Pan\(2025\)Parameter and memory efficient pretraining via low\-rank riemannian optimization\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]I\. Modoranu, M\. Safaryan, G\. Malinovsky, E\. Kurtić, T\. Robert, P\. Richtárik, and D\. Alistarh\(2024\)Microadam: accurate adaptive optimization with low space overhead and provable convergence\.Advances in Neural Information Processing Systems37,pp\. 1–43\.Cited by:[§3\.2](https://arxiv.org/html/2610.22794#S3.SS2.SSS0.Px1.p1.1)\.
- \[22\]A\. Muhamed, O\. Li, D\. Woodruff, M\. Diab, and V\. Smith\(2024\)Grass: compute efficient low\-memory llm training with structured sparse gradients\.arXiv preprint arXiv:2406\.17660\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]T\. H\. Nguyen and H\. L\. Nguyen\(2024\)Lean and mean adaptive optimization via subset\-norm and subspace\-momentum with convergence guarantees\.arXiv preprint arXiv:2411\.07120\.Cited by:[§B\.2](https://arxiv.org/html/2610.22794#A2.SS2.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]E\. Oja\(1982\)Simplified neuron model as a principal component analyzer\.Journal of mathematical biology15\(3\),pp\. 267–273\.Cited by:[§1](https://arxiv.org/html/2610.22794#S1.p5.1),[§3\.2](https://arxiv.org/html/2610.22794#S3.SS2.p3.1)\.
- \[25\]S\. Qiu, A\. Potapczynski, M\. Finzi, M\. Goldblum, and A\. G\. Wilson\(2024\)Compute better spent: replacing dense layers with structured matrices\.arXiv preprint arXiv:2406\.06248\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1)\.
- \[26\]C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu\(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§E\.1](https://arxiv.org/html/2610.22794#A5.SS1.p1.1),[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2610.22794#S5.SS2.p1.1)\.
- \[27\]S\. Rajabi, N\. Nonta, and S\. Rambhatla\(2025\)SubTrack\+\+: gradient subspace tracking for scalable llm training\.arXiv preprint arXiv:2502\.01586\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]T\. Robert, M\. Safaryan, I\. Modoranu, and D\. Alistarh\(2024\)Ldadam: adaptive optimization from low\-dimensional gradient statistics\.arXiv preprint arXiv:2410\.16103\.Cited by:[Appendix D](https://arxiv.org/html/2610.22794#A4.p1.1),[§E\.4](https://arxiv.org/html/2610.22794#A5.SS4.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.22794#S3.SS2.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.22794#S5.SS1.p1.1),[Table 2](https://arxiv.org/html/2610.22794#S5.T2),[Table 2](https://arxiv.org/html/2610.22794#S5.T2.12.1.4.1)\.
- \[29\]S\. Schotthöfer, T\. Klein, and J\. Kusch\(2025\)A geometric framework for momentum\-based optimizers for low\-rank training\.arXiv preprint arXiv:2506\.17475\.Cited by:[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px2.p1.1)\.
- \[30\]D\. Su, A\. Gu, J\. Xu, Y\. Tian, and J\. Zhao\(2025\)Galore 2: large\-scale llm pre\-training by gradient low\-rank projection\.arXiv preprint arXiv:2504\.20437\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1)\.
- \[31\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.arXiv preprint arXiv:1804\.07461\.Cited by:[§E\.1](https://arxiv.org/html/2610.22794#A5.SS1.p3.1),[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px1.p1.1),[§5\.3](https://arxiv.org/html/2610.22794#S5.SS3.p1.1)\.
- \[32\]Z\. Wang, Z\. Shen, Y\. He, G\. Sun, H\. Wang, L\. Lyu, and A\. Li\(2024\)Flora: federated fine\-tuning large language models with heterogeneous low\-rank adaptations\.Advances in Neural Information Processing Systems37,pp\. 22513–22533\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[33\]J\. Xiao, S\. Sang, T\. Zhi, J\. Liu, Q\. Yan, L\. Luo, and B\. Yuan\(2025\)Coap: memory\-efficient training with correlation\-aware gradient projection\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 30116–30126\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1)\.
- \[34\]Z\. Yang, Z\. Liu, S\. Choudhary, X\. Xie, C\. Gao, S\. Kunzmann, and Z\. Zhang\(2024\)Comera: computing\-and memory\-efficient training via rank\-adaptive tensor optimization\.Advances in Neural Information Processing Systems37,pp\. 77200–77225\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1)\.
- \[35\]T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. Finn\(2020\)Gradient surgery for multi\-task learning\.Advances in neural information processing systems33,pp\. 5824–5836\.Cited by:[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px1.p1.1)\.
- \[36\]H\. Zhang, J\. Yin, G\. Wang, Z\. Liu, T\. Zhang, A\. Shrivastava, L\. Yang, and V\. Braverman\(2025\)I3s: importance sampling subspace selection for low\-rank optimization in llm pretraining\.arXiv preprint arXiv:2502\.05790\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px3.p2.1),[§B\.3](https://arxiv.org/html/2610.22794#A2.SS3.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[37\]Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)Adalora: adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1),[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[38\]Z\. Zhang, A\. Jaiswal, L\. Yin, S\. Liu, J\. Zhao, Y\. Tian, and Z\. Wang\(2024\)Q\-galore: quantized galore with int4 projection and layer\-adaptive low\-rank gradients\.arXiv preprint arXiv:2407\.08296\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1)\.
- \[39\]J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian\(2024\)Galore: memory\-efficient llm training by gradient low\-rank projection\.arXiv preprint arXiv:2403\.03507\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p1.1),[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2610.22794#A3.I1.i1.p1.1),[§C\.1](https://arxiv.org/html/2610.22794#A3.SS1.p1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.12.1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.18.1.1),[Table 11](https://arxiv.org/html/2610.22794#A6.T11.5.1.6.1.1),[Table 9](https://arxiv.org/html/2610.22794#A6.T9.5.1.11.1.1),[Table 9](https://arxiv.org/html/2610.22794#A6.T9.5.1.5.1.1),[Appendix G](https://arxiv.org/html/2610.22794#A7.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p1.1),[§1](https://arxiv.org/html/2610.22794#S1.p2.1),[§1](https://arxiv.org/html/2610.22794#S1.p3.1),[§2](https://arxiv.org/html/2610.22794#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.22794#S3.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.22794#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.22794#S5.SS2.p1.1),[§5\.4](https://arxiv.org/html/2610.22794#S5.SS4.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.22794#S5.T1.11.1.7.1),[Table 3](https://arxiv.org/html/2610.22794#S5.T3.9.1.11.1),[Table 3](https://arxiv.org/html/2610.22794#S5.T3.9.1.4.1)\.
- \[40\]K\. Zhou, S\. Wang, and J\. Xu\(2024\)SwitchLoRA: switched low\-rank adaptation can learn full\-rank information\.arXiv preprint arXiv:2406\.06564\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px1.p2.1)\.
- \[41\]P\. Zmushko, A\. Beznosikov, M\. Takáč, and S\. Horváth\(2024\)FRUGAL: memory\-efficient optimization by reducing state overhead for scalable training\.arXiv preprint arXiv:2411\.07837\.Cited by:[§B\.1](https://arxiv.org/html/2610.22794#A2.SS1.SSS0.Px2.p1.1)\.

## Appendix AProofs

A central tool is the identity relating projectors and principal angles\.

###### Definition 1\(Principal angles\)\.

For orthonormalQ1,Q2∈ℝd×kQ\_\{1\},Q\_\{2\}\\in\\mathbb\{R\}^\{d\\times k\}, letΘ⁡\(Q1,Q2\)=diag​\(θ1,…,θk\)\\Theta\(Q\_\{1\},Q\_\{2\}\)=\\textrm\{diag\}\(\\theta\_\{1\},\\ldots,\\theta\_\{k\}\)withcos⁡θi\\cos\\theta\_\{i\}the singular values ofQ1⊤​Q2Q\_\{1\}^\{\\top\}Q\_\{2\}\.

###### Lemma 2\.

\(Distance Identity\.\) LetP1=Q1​Q1⊤P\_\{1\}=Q\_\{1\}Q\_\{1\}^\{\\top\}andP2=Q2​Q2⊤P\_\{2\}=Q\_\{2\}Q\_\{2\}^\{\\top\}be two rank\-kkorthogonal projection matrices, whereQ1,Q2∈ℝd×kQ\_\{1\},Q\_\{2\}\\in\\mathbb\{R\}^\{d\\times k\}are matrices with orthonormal columns\. Then, we have the following:

‖P1−P2‖F2=2​‖sin⁡Θ⁡\(Q1,Q2\)‖F2\.\\displaystyle\\left\\\|P\_\{1\}\-P\_\{2\}\\right\\\|\_\{F\}^\{2\}=2\\left\\\|\\sin\{\\Theta\(Q\_\{1\},Q\_\{2\}\)\}\\right\\\|\_\{F\}^\{2\}\.

###### Proof\.

Provided in[SectionA\.1](https://arxiv.org/html/2610.22794#A1.SS1)\. ∎

[Lemma2](https://arxiv.org/html/2610.22794#Thmtheorem2)converts the drift assumption into an angle bound:

‖sin⁡Θ⁡\(Vs,Vs−1\)‖F2≤λ2/2\.\\left\\\|\\sin\\Theta\(V\_\{s\},V\_\{s\-1\}\)\\right\\\|\_\{F\}^\{2\}\\;\\leq\\;\\lambda^\{2\}/2\.\(7\)
#### Per\-step potential drop\.

Define the misalignment potentialdζ2:=‖sin⁡Θ⁡\(Q\(ζ\),Vs\)‖F2d\_\{\\zeta\}^\{2\}:=\\left\\\|\\sin\\Theta\(Q^\{\(\\zeta\)\},V\_\{s\}\)\\right\\\|\_\{F\}^\{2\}\. Analyses of noisy Oja yield a one\-step inequality that links potential drop to the instantaneous regret:

𝔼⁡\[dζ\+12\]−dζ2≤−2​η​⟨Ps−P\(ζ\),g^ζ⟩\+𝒪⁡\(η2\),P\(ζ\):=Q\(ζ\)​\(Q\(ζ\)\)⊤,\\mathbb\{E\}\\\!\\left\[d\_\{\\zeta\+1\}^\{2\}\\right\]\-d\_\{\\zeta\}^\{2\}\\;\\leq\\;\-\\,2\\eta\\,\\langle P\_\{s\}\-P^\{\(\\zeta\)\},\\hat\{g\}\_\{\\zeta\}\\rangle\\;\+\\;\{\\mathcal\{O\}\}\(\\eta^\{2\}\),\\qquad P^\{\(\\zeta\)\}:=Q^\{\(\\zeta\)\}\(Q^\{\(\\zeta\)\}\)^\{\\top\},\(8\)where the𝒪⁡\(η2\)\{\\mathcal\{O\}\}\(\\eta^\{2\}\)term represents the second\-order error from the Taylor expansion of the potential function\. As detailed in[SectionA\.2](https://arxiv.org/html/2610.22794#A1.SS2), the magnitude of this geometrically\-induced error is bounded by a constant proportional toη2​‖g^ζ‖F2\\eta^\{2\}\\left\\\|\\hat\{g\}\_\{\\zeta\}\\right\\\|\_\{F\}^\{2\}\. This quantity is, in turn, controlled by our assumption of bounded input data \(‖xζ‖2≤1\\left\\\|x\_\{\\zeta\}\\right\\\|\_\{2\}\\leq 1\), which ensures the stability of the analysis\.

Decomposingg^ζ=xζ​xζ⊤\+Eζ\\hat\{g\}\_\{\\zeta\}=x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\+E\_\{\\zeta\}with the error termEζ:=xζ​yζ⊤\+yζ​xζ⊤\+yζ​yζ⊤E\_\{\\zeta\}:=x\_\{\\zeta\}y\_\{\\zeta\}^\{\\top\}\+y\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\+y\_\{\\zeta\}y\_\{\\zeta\}^\{\\top\}, we get

⟨Ps−P\(ζ\),g^ζ⟩=⟨Ps−P\(ζ\),xζ​xζ⊤⟩\+⟨Ps−P\(ζ\),Eζ⟩\.\\langle P\_\{s\}\-P^\{\(\\zeta\)\},\\hat\{g\}\_\{\\zeta\}\\rangle\\;=\\;\\langle P\_\{s\}\-P^\{\(\\zeta\)\},x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\\rangle\\;\+\\;\\langle P\_\{s\}\-P^\{\(\\zeta\)\},E\_\{\\zeta\}\\rangle\.Hölder’s inequality and‖Ps‖∗=‖P\(ζ\)‖∗=k\\\|P\_\{s\}\\\|\_\{\*\}=\\\|P^\{\(\\zeta\)\}\\\|\_\{\*\}=kimply\|⟨Ps−P\(ζ\),Eζ⟩\|≤‖Ps−P\(ζ\)‖∗​‖Eζ‖2≤2​k​‖Eζ‖2\|\\langle P\_\{s\}\-P^\{\(\\zeta\)\},E\_\{\\zeta\}\\rangle\|\\leq\\\|P\_\{s\}\-P^\{\(\\zeta\)\}\\\|\_\{\*\}\\,\\\|E\_\{\\zeta\}\\\|\_\{2\}\\leq 2k\\,\\\|E\_\{\\zeta\}\\\|\_\{2\}\. Rearranging[Equation8](https://arxiv.org/html/2610.22794#A1.E8)and using𝒪⁡\(η2\)≤4​k​d⋅η2\{\\mathcal\{O\}\}\(\\eta^\{2\}\)\\leq 4kd\\cdot\\eta^\{2\}gives

⟨Ps−P\(ζ\),xζ​xζ⊤⟩≤dζ2−𝔼⁡\[dζ\+12\]2​η\+2​k​d​η\+2​k​‖Eζ‖2\.\\langle P\_\{s\}\-P^\{\(\\zeta\)\},x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\\rangle\\;\\leq\\;\\frac\{d\_\{\\zeta\}^\{2\}\-\\mathbb\{E\}\[d\_\{\\zeta\+1\}^\{2\}\]\}\{2\\eta\}\+2kd\\eta\+2k\\,\\\|E\_\{\\zeta\}\\\|\_\{2\}\.\(9\)Detailed proof for[Equation9](https://arxiv.org/html/2610.22794#A1.E9)is shown at[SectionA\.2](https://arxiv.org/html/2610.22794#A1.SS2)\. Summing[Equation9](https://arxiv.org/html/2610.22794#A1.E9)overζ∈\{\(s−1\)​T\+1,…,s​T\}\\zeta\\in\\\{\(s\-1\)T\+1,\\dots,sT\\\}yields the per\-step bound

Rs≤d\(s−1\)​T\+12−𝔼⁡\[ds​T\+12\]2​η\+2​k​d​η​T\+2​k​Es,Es:=∑ζ=\(s−1\)​T\+1s​T‖Eζ‖2\.R\_\{s\}\\;\\leq\\;\\frac\{d\_\{\(s\-1\)T\+1\}^\{2\}\-\\mathbb\{E\}\\\!\\left\[d\_\{sT\+1\}^\{2\}\\right\]\}\{2\\eta\}\\;\+\\;2kd\\eta T\\;\+\\;2k\\,E\_\{s\},\\qquad E\_\{s\}:=\\sum\_\{\\zeta=\(s\-1\)T\+1\}^\{sT\}\\\!\\\!\\\|E\_\{\\zeta\}\\\|\_\{2\}\.\(10\)

#### Linking steps via drift in angle space\.

By[Lemma2](https://arxiv.org/html/2610.22794#Thmtheorem2), we have that

ds​\(Vs\)2−ds​\(Vs−1\)2=12​\(‖PQ−Ps‖F2−‖PQ−Ps−1‖F2\)≤k​λ,d\_\{s\}\(V\_\{s\}\)^\{2\}\-d\_\{s\}\(V\_\{s\-1\}\)^\{2\}\\;=\\;\\tfrac\{1\}\{2\}\\\!\\left\(\\left\\\|P\_\{Q\}\-P\_\{s\}\\right\\\|\_\{F\}^\{2\}\-\\left\\\|P\_\{Q\}\-P\_\{s\-1\}\\right\\\|\_\{F\}^\{2\}\\right\)\\;\\leq\\;\\sqrt\{k\}\\,\\lambda,wherePQ:=Q\(\(s−1\)​T\+1\)​\(Q\(\(s−1\)​T\+1\)\)⊤P\_\{Q\}:=Q^\{\(\(s\-1\)T\+1\)\}\(Q^\{\(\(s\-1\)T\+1\)\}\)^\{\\top\}\. With any orthonormal initializationQ\(1\)Q^\{\(1\)\}, we haved1​\(V1\)2≤kd\_\{1\}\(V\_\{1\}\)^\{2\}\\leq k\.

### A\.1Proof of Lemma[2](https://arxiv.org/html/2610.22794#Thmtheorem2): Distance Identity

###### Proof\.

‖P1−P2‖F2=T​r​\(\(P1−P2\)⊤​\(P1−P2\)\)=T​r​\(P12−P1​P2−P2​P1\+P22\)\\displaystyle\\left\\\|P\_\{1\}\-P\_\{2\}\\right\\\|\_\{F\}^\{2\}=Tr\\left\(\(P\_\{1\}\-P\_\{2\}\)^\{\\top\}\(P\_\{1\}\-P\_\{2\}\)\\right\)=Tr\(P\_\{1\}^\{2\}\-P\_\{1\}P\_\{2\}\-P\_\{2\}P\_\{1\}\+P\_\{2\}^\{2\}\)SinceP1P\_\{1\}andP2P\_\{2\}are orthogonal projectors, they are symmetricPi⊤=PiP\_\{i\}^\{\\top\}=P\_\{i\}and idempotent \(Pi2=PiP\_\{i\}^\{2\}=P\_\{i\}\)\. The trace of a rank\-kkprojector is its rank, soT​r​\(P1\)=T​r​\(P2\)=kTr\(P\_\{1\}\)=Tr\(P\_\{2\}\)=k\. The expression simplifies to:

‖P1−P2‖F2=Tr​\(P1\)\+Tr​\(P2\)−2​Tr​\(P1​P2\)=2​k−2​Tr​\(P1​P2\)\.\\displaystyle\|\|P\_\{1\}\-P\_\{2\}\|\|\_\{F\}^\{2\}=\\text\{Tr\}\(P\_\{1\}\)\+\\text\{Tr\}\(P\_\{2\}\)\-2\\text\{Tr\}\(P\_\{1\}P\_\{2\}\)=2k\-2\\text\{Tr\}\(P\_\{1\}P\_\{2\}\)\.The trace term can be rewritten as:

Tr​\(P1​P2\)=Tr​\(Q1​Q1⊤​Q2​Q2⊤\)=Tr​\(Q2⊤​Q1​Q1⊤​Q2\)=‖Q1⊤​Q2‖F2\.\\displaystyle\\text\{Tr\}\(P\_\{1\}P\_\{2\}\)=\\text\{Tr\}\(Q\_\{1\}Q\_\{1\}^\{\\top\}Q\_\{2\}Q\_\{2\}^\{\\top\}\)=\\text\{Tr\}\(Q\_\{2\}^\{\\top\}Q\_\{1\}Q\_\{1\}^\{\\top\}Q\_\{2\}\)=\|\|Q\_\{1\}^\{\\top\}Q\_\{2\}\|\|\_\{F\}^\{2\}\.The singular values of the matrixQ1⊤​Q2Q\_\{1\}^\{\\top\}Q\_\{2\}are the cosines of the principal angles,\{cos⁡θi\}i=1k\\\{\\cos\\theta\_\{i\}\\\}\_\{i=1\}^\{k\}\. The squared Frobenius norm is the sum of the squares of the singular values:

‖Q1⊤​Q2‖F2=∑i=1kcos2⁡θi\.\\displaystyle\\left\\\|Q\_\{1\}^\{\\top\}Q\_\{2\}\\right\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{k\}\\cos^\{2\}\\theta\_\{i\}\.Substituting this back gives:

‖P1−P2‖F2=2​k−2​∑i=1kcos2⁡θi=2​∑i=1k\(1−cos2⁡θi\)=2​∑i=1ksin2⁡θi=2​‖sin⁡Θ⁡\(Q1,Q2\)‖F2\.\\displaystyle\\left\\\|P\_\{1\}\-P\_\{2\}\\right\\\|\_\{F\}^\{2\}=2k\-2\\sum\_\{i=1\}^\{k\}\\cos^\{2\}\\theta\_\{i\}=2\\sum\_\{i=1\}^\{k\}\(1\-\\cos^\{2\}\\theta\_\{i\}\)=2\\sum\_\{i=1\}^\{k\}\\sin^\{2\}\\theta\_\{i\}=2\\left\\\|\\sin\{\\Theta\(Q\_\{1\},Q\_\{2\}\)\}\\right\\\|\_\{F\}^\{2\}\.∎

### A\.2Detailed proof of equation[8](https://arxiv.org/html/2610.22794#A1.E8)and equation[9](https://arxiv.org/html/2610.22794#A1.E9)\(one\-step potential drop for Oja\)

RecallP\(ζ\)=Q\(ζ\)​\(Q\(ζ\)\)⊤P^\{\(\\zeta\)\}=Q^\{\(\\zeta\)\}\(Q^\{\(\\zeta\)\}\)^\{\\top\},Ps=Vs​Vs⊤P\_\{s\}=V\_\{s\}V\_\{s\}^\{\\top\}, and define the misalignment potential

dζ2:=‖sin⁡Θ⁡\(Q\(ζ\),Vs\)‖F2=12​‖P\(ζ\)−Ps‖F2\.d\_\{\\zeta\}^\{2\}\\;:=\\;\\left\\\|\\sin\\Theta\\\!\\big\(Q^\{\(\\zeta\)\},V\_\{s\}\\big\)\\right\\\|\_\{F\}^\{2\}\\;=\\;\\tfrac\{1\}\{2\}\\left\\\|P^\{\(\\zeta\)\}\-P\_\{s\}\\right\\\|\_\{F\}^\{2\}\.DenoteP:=P\(ζ\)P:=P^\{\(\\zeta\)\}andZ:=g^ζZ:=\\hat\{g\}\_\{\\zeta\}for brevity, whereZZis symmetric\. Oja’s update reads

Q\(ζ\+1\)=QR⁡\(\(I\+η​Z\)​Q\(ζ\)\),P\(ζ\+1\)=Q\(ζ\+1\)​\(Q\(ζ\+1\)\)⊤\.Q^\{\(\\zeta\+1\)\}\\;=\\;\\mathrm\{QR\}\\\!\\Big\(\(I\+\\eta Z\)\\,Q^\{\(\\zeta\)\}\\Big\),\\qquad P^\{\(\\zeta\+1\)\}\\;=\\;Q^\{\(\\zeta\+1\)\}\(Q^\{\(\\zeta\+1\)\}\)^\{\\top\}\.\(11\)Here is the proof of equation[8](https://arxiv.org/html/2610.22794#A1.E8)

𝔼⁡\[dζ\+12\]−dζ2≤−2​η​⟨Ps−P,Z⟩\+𝒪⁡\(η2\),\\mathbb\{E\}\\\!\\left\[d\_\{\\zeta\+1\}^\{2\}\\right\]\-d\_\{\\zeta\}^\{2\}\\;\\leq\\;\-\\,2\\eta\\,\\langle P\_\{s\}\-P,Z\\rangle\\;\+\\;\{\\mathcal\{O\}\}\(\\eta^\{2\}\),
###### Proof\.

Consider the pre\-retraction iterateQ~=\(I\+η​Z\)​Q\(ζ\)\\widetilde\{Q\}=\(I\+\\eta Z\)Q^\{\(\\zeta\)\}and its \(non\-idempotent\) projectorP~:=Q~​Q~⊤\\widetilde\{P\}:=\\widetilde\{Q\}\\widetilde\{Q\}^\{\\top\}:

P~\\displaystyle\\widetilde\{P\}=\(I\+η​Z\)​P2​\(I\+η​Z\)=\(I\+η​Z\)​P​\(I\+η​Z\)=P\+η⁡\(Z​P\+P​Z\)\+η2​Z​P​Z\.\\displaystyle=\(I\+\\eta Z\)\\,P^\{2\}\\,\(I\+\\eta Z\)=\(I\+\\eta Z\)\\,P\\,\(I\+\\eta Z\)=P\\;\+\\;\\eta\(ZP\+PZ\)\\;\+\\;\\eta^\{2\}ZPZ\.\(12\)Let𝖳P:=\{Δ=Δ⊤:PΔP=0,\(I−P\)Δ\(I−P\)=0\}\\mathsf\{T\}\_\{P\}:=\\\{\\Delta=\\Delta^\{\\top\}:\\ P\\Delta P=0,\\ \(I\-P\)\\Delta\(I\-P\)=0\\\}denote the tangent space of the Grassmannian atPP\. Projecting the first\-order term onto𝖳P\\mathsf\{T\}\_\{P\}gives the canonical tangent increment

ΔP:=Π𝖳P​\(Z​P\+P​Z\)=\(I−P\)​Z​P\+P​Z​\(I−P\)\.\\Delta\_\{P\}\\;:=\\;\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\\big\(ZP\+PZ\\big\)\\;=\\;\(I\-P\)\\,Z\\,P\\;\+\\;P\\,Z\\,\(I\-P\)\.\(13\)Standard properties of retractions \(here,QR\\mathrm\{QR\}\) imply that the reorthonormalized projector satisfies

P\(ζ\+1\)=P\+η​ΔP\+𝒪⁡\(η2\),P^\{\(\\zeta\+1\)\}\\;=\\;P\\;\+\\;\\eta\\,\\Delta\_\{P\}\\;\+\\;\{\\mathcal\{O\}\}\(\\eta^\{2\}\),\(14\)i\.e\., retraction preserves the first\-order tangent component and pushes all curvature effects into𝒪⁡\(η2\)\{\\mathcal\{O\}\}\(\\eta^\{2\}\); which is a similar "one\-step" analysis strategy of Appendix i\.B in[Allen\-Zhu and Li \[1\]](https://arxiv.org/html/2610.22794#bib.bib34)\.

Defineϕ⁡\(P\)=12​‖P−Ps‖F2\\phi\(P\)=\\tfrac\{1\}\{2\}\\left\\\|P\-P\_\{s\}\\right\\\|\_\{F\}^\{2\}, whose Euclidean gradient is∇ϕ​\(P\)=P−Ps\\nabla\\phi\(P\)=P\-P\_\{s\}\. By expanding, we have

dζ\+12−dζ2\\displaystyle d\_\{\\zeta\+1\}^\{2\}\-d\_\{\\zeta\}^\{2\}=ϕ⁡\(P\(ζ\+1\)\)−ϕ⁡\(P\)=⟨P−Ps,P\(ζ\+1\)−P⟩\+12​‖P\(ζ\+1\)−P‖F2\\displaystyle=\\phi\\\!\\big\(P^\{\(\\zeta\+1\)\}\\big\)\-\\phi\(P\)=\\langle P\-P\_\{s\},P^\{\(\\zeta\+1\)\}\-P\\rangle\\;\+\\;\\frac\{1\}\{2\}\\left\\\|P^\{\(\\zeta\+1\)\}\-P\\right\\\|\_\{F\}^\{2\}=η⁡⟨P−Ps,ΔP⟩\+η22​‖ΔP‖F2\.\\displaystyle=\\eta\\,\\langle P\-P\_\{s\},\\Delta\_\{P\}\\rangle\\;\+\\;\\frac\{\\eta^\{2\}\}\{2\}\\left\\\|\\Delta\_\{P\}\\right\\\|\_\{F\}^\{2\}\.\(15\)Write

ΔP=\(I−P\)​Z​P\+P​Z​\(I−P\)=Z​P\+P​Z−2​P​Z​P\.\\Delta\_\{P\}\\;=\\;\(I\-P\)ZP\+PZ\(I\-P\)\\;=\\;ZP\+PZ\-2PZP\.A direct computation \(e\.g\., in a block basis that diagonalizesPP\) yields the identity

⟨P−Ps,ΔP⟩=−2​⟨Π𝖳P​\(Ps−P\),Z⟩,Π𝖳P​\(Ps−P\)=\(I−P\)​Ps​P\+P​Ps​\(I−P\)\.\\langle P\-P\_\{s\},\\Delta\_\{P\}\\rangle\\;=\\;\-\\,2\\,\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangle,\\qquad\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\)\\;=\\;\(I\-P\)P\_\{s\}P\+PP\_\{s\}\(I\-P\)\.\(16\)*Proof of equation[16](https://arxiv.org/html/2610.22794#A1.E16)\(sketch\)\.*Work in the orthonormal basis\[QQ⟂\]\[Q\\ \\ Q\_\{\\perp\}\]withP=diag​\(Ik,0\)P=\\textrm\{diag\}\(I\_\{k\},0\)and decomposeZ=\(AB⊤BC\),Ps=\(XY⊤YZ0\)\.Z=\\begin\{pmatrix\}A&B^\{\\top\}\\\\ B&C\\end\{pmatrix\},\\;P\_\{s\}=\\begin\{pmatrix\}X&Y^\{\\top\}\\\\ Y&Z\_\{0\}\\end\{pmatrix\}\.ThenΔP=\(0B⊤B0\)\\Delta\_\{P\}=\\begin\{pmatrix\}0&B^\{\\top\}\\\\ B&0\\end\{pmatrix\}andΠ𝖳P​\(Ps−P\)=\(0Y⊤Y0\)\.\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\)=\\begin\{pmatrix\}0&Y^\{\\top\}\\\\ Y&0\\end\{pmatrix\}\.Hence⟨P−Ps,ΔP⟩=−2​Tr⁡\(Y⊤​B\)=−2​⟨Π𝖳P​\(Ps−P\),Z⟩\.\\langle P\-P\_\{s\},\\Delta\_\{P\}\\rangle=\-2\\,\\Tr\(Y^\{\\top\}B\)=\-\\,2\\,\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangle\.□\\square

Combining equation[15](https://arxiv.org/html/2610.22794#A1.E15)and equation[16](https://arxiv.org/html/2610.22794#A1.E16)we obtain

dζ\+12−dζ2=−2​η​⟨Π𝖳P​\(Ps−P\),Z⟩\+𝒪⁡\(η2\)\.d\_\{\\zeta\+1\}^\{2\}\-d\_\{\\zeta\}^\{2\}\\;=\\;\-\\,2\\eta\\,\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangle\\;\+\\;\{\\mathcal\{O\}\}\(\\eta^\{2\}\)\.\(17\)
We now relate⟨Π𝖳P​\(Ps−P\),Z⟩\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangleto the full inner product⟨Ps−P,Z⟩\\langle P\_\{s\}\-P,Z\\ranglethat appears in the instantaneous regret\. Observe that

⟨Ps−P,Z⟩\\displaystyle\\langle P\_\{s\}\-P,Z\\rangle=⟨Π𝖳P​\(Ps−P\),Z⟩\+⟨\(I−P\)​\(Ps−P\)​\(I−P\),Z⟩⏟≥0​if​Z⪰0\+⟨P⁡\(Ps−P\)​P,Z⟩⏟≤0​if​Z⪰0\\displaystyle=\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangle\\;\+\\;\\underbrace\{\\langle\(I\-P\)\(P\_\{s\}\-P\)\(I\-P\),Z\\rangle\}\_\{\\geq 0\\text\{ if \}Z\\succeq 0\}\\;\+\\;\\underbrace\{\\langle P\\,\(P\_\{s\}\-P\)\\,P,Z\\rangle\}\_\{\\leq 0\\text\{ if \}Z\\succeq 0\}=⟨Π𝖳P​\(Ps−P\),Z⟩\+⟨\(I−P\)​Ps​\(I−P\),Z⟩−⟨P−P​Ps​P,Z⟩\.\\displaystyle=\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangle\\;\+\\;\\langle\(I\-P\)P\_\{s\}\(I\-P\),Z\\rangle\\;\-\\;\\langle P\-PP\_\{s\}P,Z\\rangle\.\(18\)Since0⪯Ps⪯I0\\preceq P\_\{s\}\\preceq IandZ⪰0Z\\succeq 0, both\(I−P\)​Ps​\(I−P\)\(I\-P\)P\_\{s\}\(I\-P\)and\(P−P​Ps​P\)\(P\-PP\_\{s\}P\)are positive semidefinite, and hence the last line of equation[18](https://arxiv.org/html/2610.22794#A1.E18)shows

⟨Π𝖳P​\(Ps−P\),Z⟩≤⟨Ps−P,Z⟩whenever​Z⪰0\.\\langle\\Pi\_\{\\mathsf\{T\}\_\{P\}\}\(P\_\{s\}\-P\),Z\\rangle\\;\\leq\\;\\langle P\_\{s\}\-P,Z\\rangle\\qquad\\text\{whenever \}Z\\succeq 0\.\(19\)Plugging equation[19](https://arxiv.org/html/2610.22794#A1.E19)into equation[17](https://arxiv.org/html/2610.22794#A1.E17)yields

dζ\+12−dζ2≤−2​η​⟨Ps−P,Z⟩\+𝒪⁡\(η2\),\(Z⪰0\)\.d\_\{\\zeta\+1\}^\{2\}\-d\_\{\\zeta\}^\{2\}\\;\\leq\\;\-\\,2\\eta\\,\\langle P\_\{s\}\-P,Z\\rangle\\;\+\\;\{\\mathcal\{O\}\}\(\\eta^\{2\}\),\\qquad\(Z\\succeq 0\)\.\(20\)
To be explicit about the control of the𝒪⁡\(η2\)\{\\mathcal\{O\}\}\(\\eta^\{2\}\)higher\-order terms, we trace their origin\. The termP\(ζ\+1\)−PP^\{\(\\zeta\+1\)\}\-Pin the Taylor expansion represents the full update step on the manifold\. As established in equation[12](https://arxiv.org/html/2610.22794#A1.E12)and the subsequent discussion on retractions, this step can be decomposed into a first\-order tangent componentη​ΔP\\eta\\Delta\_\{P\}, and higher\-order components\. These higher\-order components, which we collect into the𝒪⁡\(η2\)\{\\mathcal\{O\}\}\(\\eta^\{2\}\)term, are a direct consequence of two factors: \(i\) the explicit algebraic termη2​Z​P​Z\\eta^\{2\}ZPZthat arises from the pre\-retraction update, and \(ii\) the intrinsic curvature of the manifold, which means the path defined by the QR retraction deviates from the straight\-line path of the tangent direction\.

Recalling the algebraic second\-order term, we have

P~=P\+η⁡\(Z​P\+P​Z\)\+η2​Z​P​Z,‖η2​Z​P​Z‖F≤η2​‖Z‖F​‖P‖F​‖Z‖F=η2​‖P‖F​‖Z‖F2\.\\displaystyle\\tilde\{P\}=P\+\\eta\(ZP\+PZ\)\+\\eta^\{2\}ZPZ,\\quad\\left\\\|\\eta^\{2\}ZPZ\\right\\\|\_\{F\}\\leq\\eta^\{2\}\\left\\\|Z\\right\\\|\_\{F\}\\left\\\|P\\right\\\|\_\{F\}\\left\\\|Z\\right\\\|\_\{F\}=\\eta^\{2\}\\left\\\|P\\right\\\|\_\{F\}\\left\\\|Z\\right\\\|\_\{F\}^\{2\}\.SincePPis a rank\-k orthogonal projector, its Frobenius norm is a constant:‖P‖F=T​r​\(P⊤​P\)=T​r​\(P\)=k\\left\\\|P\\right\\\|\_\{F\}=\\sqrt\{Tr\(P^\{\\top\}P\)\}=\\sqrt\{Tr\(P\)\}=\\sqrt\{k\}\. Therefore:

‖η2​Z​P​Z‖F≤k⋅η2​‖Z‖F2\.\\displaystyle\\left\\\|\\eta^\{2\}ZPZ\\right\\\|\_\{F\}\\leq\\sqrt\{k\}\\cdot\\eta^\{2\}\\left\\\|Z\\right\\\|\_\{F\}^\{2\}\.
For geometric second\-order term, we need to relate‖ΔP‖F2\\\|\\Delta\_\{P\}\\\|\_\{F\}^\{2\}to‖Z‖F2\\\|Z\\\|\_\{F\}^\{2\}\. From equation[13](https://arxiv.org/html/2610.22794#A1.E13), we haveΔP=\(I−P\)​Z​P\+P​Z​\(I−P\)\\Delta\_\{P\}=\(I\-P\)ZP\+PZ\(I\-P\)\. Using the triangle inequality and the sub\-multiplicativity of the Frobenius norm, we get:

‖ΔP‖F≤‖\(I−P\)​Z​P‖F\+‖P​Z​\(I−P\)‖F≤2​‖P‖F​‖I−P‖F​‖Z‖F\.\\displaystyle\\left\\\|\\Delta\_\{P\}\\right\\\|\_\{F\}\\leq\\\|\(I\-P\)ZP\\\|\_\{F\}\+\\\|PZ\(I\-P\)\\\|\_\{F\}\\leq 2\\\|P\\\|\_\{F\}\\\|I\-P\\\|\_\{F\}\\\|Z\\\|\_\{F\}\.
SincePPis a rank\-kkprojector andI−PI\-Pis a rank\-\(d−k\)\(d\-k\)projector, their norms are constants

‖P‖F=kand‖I−P‖F=d−k,\\displaystyle\\\|P\\\|\_\{F\}=\\sqrt\{k\}\\quad\\text\{and\}\\quad\\\|I\-P\\\|\_\{F\}=\\sqrt\{d\-k\},respectively\. This means‖ΔP‖F\\\|\\Delta\_\{P\}\\\|\_\{F\}is bounded by a constant times‖Z‖F\\\|Z\\\|\_\{F\}\. Consequently, the second\-order geometric errorη2​‖ΔP‖F2\\eta^\{2\}\\\|\\Delta\_\{P\}\\\|\_\{F\}^\{2\}is bounded by\(4​k​d−4​k2\)​η2​‖Z‖F2\(4kd\-4k^\{2\}\)\\eta^\{2\}\\\|Z\\\|\_\{F\}^\{2\}\. In total, we have𝒪⁡\(η2\)≤4​k​d⋅η​‖Z‖F2\{\\mathcal\{O\}\}\(\\eta^\{2\}\)\\leq 4kd\\cdot\\eta\\left\\\|Z\\right\\\|\_\{F\}^\{2\}\. The norm of these higher\-order terms is therefore fundamentally governed by the norm of the update matrixZ=g^ζZ=\\hat\{g\}\_\{\\zeta\}\. Under‖xζ‖2≤1\\left\\\|x\_\{\\zeta\}\\right\\\|\_\{2\}\\leq 1\(so‖Z‖F=‖xζ​xζ⊤‖F=‖xζ‖22≤1\\left\\\|Z\\right\\\|\_\{F\}=\\left\\\|x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\\right\\\|\_\{F\}=\\left\\\|x\_\{\\zeta\}\\right\\\|\_\{2\}^\{2\}\\leq 1\) we have𝒪⁡\(η2\)≤4​k​d⋅η2\{\\mathcal\{O\}\}\(\\eta^\{2\}\)\\leq 4kd\\cdot\\eta^\{2\}\. Finally, taking expectation over the randomness at timeζ\\zeta\(data and any internal noise\) yields

𝔼⁡\[dζ\+12\]−dζ2≤−2​η​⟨Ps−P,g^ζ⟩\+4​k​d⋅η2,\\mathbb\{E\}\\\!\\left\[d\_\{\\zeta\+1\}^\{2\}\\right\]\-d\_\{\\zeta\}^\{2\}\\;\\leq\\;\-\\,2\\eta\\,\\langle P\_\{s\}\-P,\\hat\{g\}\_\{\\zeta\}\\rangle\\;\+\\;4kd\\cdot\\eta^\{2\},which is Eq\. equation[8](https://arxiv.org/html/2610.22794#A1.E8)\. ∎

Ifg^ζ=xζ​xζ⊤\+Eζ\\hat\{g\}\_\{\\zeta\}=x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\+E\_\{\\zeta\}with an arbitrary symmetric perturbationEζE\_\{\\zeta\}, the argument above applies verbatim to the PSD partxζ​xζ⊤x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}; the additional term satisfies

\|⟨Ps−P,Eζ⟩\|≤‖Ps−P‖∗​‖Eζ‖2≤2​k​‖Eζ‖2,\\big\|\\langle P\_\{s\}\-P,E\_\{\\zeta\}\\rangle\\big\|\\;\\leq\\;\\left\\\|P\_\{s\}\-P\\right\\\|\_\{\*\}\\,\\left\\\|E\_\{\\zeta\}\\right\\\|\_\{2\}\\;\\leq\\;2k\\,\\left\\\|E\_\{\\zeta\}\\right\\\|\_\{2\},since‖Ps‖∗=‖P‖∗=k\\left\\\|P\_\{s\}\\right\\\|\_\{\*\}=\\left\\\|P\\right\\\|\_\{\*\}=k\. This gives the noise contribution used in the main text\. ∎

### A\.3Proof of Theorem[1](https://arxiv.org/html/2610.22794#Thmtheorem1)

###### Proof\.

Fix a steps∈\[S\]s\\in\[S\]and writeP\(ζ\)=Q\(ζ\)​\(Q\(ζ\)\)⊤P^\{\(\\zeta\)\}=Q^\{\(\\zeta\)\}\(Q^\{\(\\zeta\)\}\)^\{\\top\}andPs=Vs​Vs⊤P\_\{s\}=V\_\{s\}V\_\{s\}^\{\\top\}\. Define the misalignment potentialdζ2:=‖sin⁡Θ⁡\(Q\(ζ\),Vs\)‖F2=12​‖P\(ζ\)−Ps‖F2d\_\{\\zeta\}^\{2\}:=\\left\\\|\\sin\\Theta\(Q^\{\(\\zeta\)\},V\_\{s\}\)\\right\\\|\_\{F\}^\{2\}=\\tfrac\{1\}\{2\}\\left\\\|P^\{\(\\zeta\)\}\-P\_\{s\}\\right\\\|\_\{F\}^\{2\}\(Lemma[2](https://arxiv.org/html/2610.22794#Thmtheorem2)\)\. By the one\-step potential drop for Oja \(proved in Appendix[A\.2](https://arxiv.org/html/2610.22794#A1.SS2)\), for the noisy updateQ\(ζ\+1\)=Q​R​\(\(I\+η​g^ζ\)​Q\(ζ\)\)Q^\{\(\\zeta\+1\)\}=QR\\\!\\big\(\(I\+\\eta\\,\\hat\{g\}\_\{\\zeta\}\)Q^\{\(\\zeta\)\}\\big\)withg^ζ=x^ζ​x^ζ⊤\\hat\{g\}\_\{\\zeta\}=\\hat\{x\}\_\{\\zeta\}\\hat\{x\}\_\{\\zeta\}^\{\\top\}and‖xζ‖2≤1\\left\\\|x\_\{\\zeta\}\\right\\\|\_\{2\}\\leq 1, we have

𝔼⁡\[dζ\+12\]−dζ2≤−2​η​⟨Ps−P\(ζ\),g^ζ⟩\+4​k​d​η2\.\\mathbb\{E\}\\\!\\left\[d\_\{\\zeta\+1\}^\{2\}\\right\]\-d\_\{\\zeta\}^\{2\}\\;\\leq\\;\-\\,2\\eta\\,\\langle P\_\{s\}\-P^\{\(\\zeta\)\},\\hat\{g\}\_\{\\zeta\}\\rangle\\;\+\\;4kd\\eta^\{2\}\.\(21\)Decompose the noisy rank\-one matrix asg^ζ=xζ​xζ⊤\+Eζ,\\hat\{g\}\_\{\\zeta\}=x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\+E\_\{\\zeta\},whereEζ:=xζ​yζ⊤\+yζ​xζ⊤\+yζ​yζ⊤E\_\{\\zeta\}:=x\_\{\\zeta\}y\_\{\\zeta\}^\{\\top\}\+y\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\+y\_\{\\zeta\}y\_\{\\zeta\}^\{\\top\}\. Using Hölder and‖Ps‖∗=‖P\(ζ\)‖∗=k\\\|P\_\{s\}\\\|\_\{\*\}=\\\|P^\{\(\\zeta\)\}\\\|\_\{\*\}=k,

\|⟨Ps−P\(ζ\),Eζ⟩\|≤‖Ps−P\(ζ\)‖∗​‖Eζ‖2≤2​k​‖Eζ‖2\.\\big\|\\langle P\_\{s\}\-P^\{\(\\zeta\)\},E\_\{\\zeta\}\\rangle\\big\|\\;\\leq\\;\\\|P\_\{s\}\-P^\{\(\\zeta\)\}\\\|\_\{\*\}\\,\\\|E\_\{\\zeta\}\\\|\_\{2\}\\;\\leq\\;2k\\,\\\|E\_\{\\zeta\}\\\|\_\{2\}\.Rearranging equation[21](https://arxiv.org/html/2610.22794#A1.E21)gives the per\-iteration regret bound

⟨Ps−P\(ζ\),xζ​xζ⊤⟩≤dζ2−𝔼⁡\[dζ\+12\]2​η\+2​k​d​η\+2​k​‖Eζ‖2\.\\langle P\_\{s\}\-P^\{\(\\zeta\)\},x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\\rangle\\;\\leq\\;\\frac\{d\_\{\\zeta\}^\{2\}\-\\mathbb\{E\}\[d\_\{\\zeta\+1\}^\{2\}\]\}\{2\\eta\}\\;\+\\;2kd\\eta\\;\+\\;2k\\,\\\|E\_\{\\zeta\}\\\|\_\{2\}\.\(22\)

#### Within a step\.

Summing equation[22](https://arxiv.org/html/2610.22794#A1.E22)overζ=\(s−1\)​T\+1,…,s​T\\zeta=\(s\-1\)T\+1,\\dots,sTyields

Rs:=∑ζ=\(s−1\)​T\+1s​T⟨Ps−P\(ζ\),xζ​xζ⊤⟩≤d\(s−1\)​T\+12−𝔼⁡\[ds​T\+12\]2​η\+2​k​d​η\+2​k​Es,R\_\{s\}\\;:=\\;\\sum\_\{\\zeta=\(s\-1\)T\+1\}^\{sT\}\\\!\\langle P\_\{s\}\-P^\{\(\\zeta\)\},x\_\{\\zeta\}x\_\{\\zeta\}^\{\\top\}\\rangle\\;\\leq\\;\\frac\{d\_\{\(s\-1\)T\+1\}^\{2\}\-\\mathbb\{E\}\[d\_\{sT\+1\}^\{2\}\]\}\{2\\eta\}\\;\+\\;2kd\\eta\\;\+\\;2k\\,E\_\{s\},\(23\)whereEs:=∑ζ=\(s−1\)​T\+1s​T‖Eζ‖2E\_\{s\}:=\\sum\_\{\\zeta=\(s\-1\)T\+1\}^\{sT\}\\\|E\_\{\\zeta\}\\\|\_\{2\}\.

#### Across steps \(drift linking\)\.

Summing equation[23](https://arxiv.org/html/2610.22794#A1.E23)overs=1s=1toSSgives

𝔼⁡\[RS\]≤2​k​d⋅η​T\+12​η​∑s=1S\(d\(s−1\)​T\+12−𝔼⁡\[ds​T\+12\]\)\+2​k​Etot,Etot:=∑s=1SEs\.\\mathbb\{E\}\[R\_\{S\}\]\\;\\leq\\;2kd\\cdot\\eta T\\;\+\\;\\frac\{1\}\{2\\eta\}\\sum\_\{s=1\}^\{S\}\\\!\\Big\(d\_\{\(s\-1\)T\+1\}^\{2\}\-\\mathbb\{E\}\[d\_\{sT\+1\}^\{2\}\]\\Big\)\\;\+\\;2k\\,E\_\{\\rm tot\},\\qquad E\_\{\\rm tot\}:=\\sum\_\{s=1\}^\{S\}E\_\{s\}\.\(24\)The terminal terms are non\-positive and can be dropped:−∑s=1S𝔼\[ds​T\+12\]≤0\-\\sum\_\{s=1\}^\{S\}\\mathbb\{E\}\[d\_\{sT\+1\}^\{2\}\]\\leq 0\. It remains to upper bound∑s=1Sd\(s−1\)​T\+12\\sum\_\{s=1\}^\{S\}d\_\{\(s\-1\)T\+1\}^\{2\}where eachd\(s−1\)​T\+12d\_\{\(s\-1\)T\+1\}^\{2\}is measured against the*local*comparatorVsV\_\{s\}\. LetPQ:=Q\(\(s−1\)​T\+1\)​\(Q\(\(s−1\)​T\+1\)\)⊤P\_\{Q\}:=Q^\{\(\(s\-1\)T\+1\)\}\\\!\(Q^\{\(\(s\-1\)T\+1\)\}\)^\{\\top\}\. By Lemma[2](https://arxiv.org/html/2610.22794#Thmtheorem2)and Cauchy\-Schwarz inequality,

d\(s−1\)​T\+1​\(Vs\)2−d\(s−1\)​T\+1​\(Vs−1\)2\\displaystyle d\_\{\(s\-1\)T\+1\}\(V\_\{s\}\)^\{2\}\-d\_\{\(s\-1\)T\+1\}\(V\_\{s\-1\}\)^\{2\}=12​\(‖PQ−Ps‖F2−‖PQ−Ps−1‖F2\)\\displaystyle=\\;\\tfrac\{1\}\{2\}\\\!\\left\(\\\|P\_\{Q\}\-P\_\{s\}\\\|\_\{F\}^\{2\}\-\\\|P\_\{Q\}\-P\_\{s\-1\}\\\|\_\{F\}^\{2\}\\right\)=⟨PQ,Ps−1−Ps⟩≤k​λ,\\displaystyle=\\;\\langle P\_\{Q\},P\_\{s\-1\}\-P\_\{s\}\\rangle\\;\\leq\\;\\sqrt\{k\}\\,\\lambda,using‖PQ‖F=k\\\|P\_\{Q\}\\\|\_\{F\}=\\sqrt\{k\}and the drift assumption‖Ps−Ps−1‖F≤λ\\\|P\_\{s\}\-P\_\{s\-1\}\\\|\_\{F\}\\leq\\lambda\. With the initializationP\(1\)=0P^\{\(1\)\}=0\(henced1​\(V1\)2=12​‖P1‖F2=k2d\_\{1\}\(V\_\{1\}\)^\{2\}=\\tfrac\{1\}\{2\}\\\|P\_\{1\}\\\|\_\{F\}^\{2\}=\\tfrac\{k\}\{2\}\), we obtain

∑s=1Sd\(s−1\)​T\+12≤k2\+\(S−1\)​k​λ\.\\sum\_\{s=1\}^\{S\}d\_\{\(s\-1\)T\+1\}^\{2\}\\;\\leq\\;\\frac\{k\}\{2\}\\;\+\\;\(S\-1\)\\sqrt\{k\}\\,\\lambda\.\(25\)

#### Conclusion\.

Plugging equation[25](https://arxiv.org/html/2610.22794#A1.E25)into equation[24](https://arxiv.org/html/2610.22794#A1.E24)yields

𝔼⁡\[RS\]≤2​k​d⋅η​S​T\+k2\+\(S−1\)​k​λ2​η\+2​k​Etot,\\mathbb\{E\}\[R\_\{S\}\]\\;\\leq\\;2kd\\cdot\\eta ST\\;\+\\;\\frac\{\\frac\{k\}\{2\}\+\(S\-1\)\\sqrt\{k\}\\,\\lambda\}\{2\\eta\}\\;\+\\;2k\\,E\_\{\\rm tot\},which is \(7\)\. Optimizing the right\-hand side overη\>0\\eta\>0givesη⋆=k2\+\(S−1\)​k​λ2​k​d⋅S​T\\eta^\{\\star\}=\\sqrt\{\\frac\{\\frac\{k\}\{2\}\+\(S\-1\)\\sqrt\{k\}\\lambda\}\{2kd\\cdot ST\}\}and

𝔼⁡\[RS\]≤2​k​d​S​T​\(k2\+\(S−1\)​k​λ\)\+2​k​Etot,\\mathbb\{E\}\[R\_\{S\}\]\\;\\leq\\;\\sqrt\{2kdST\\\!\\left\(\\tfrac\{k\}\{2\}\+\(S\-1\)\\sqrt\{k\}\\,\\lambda\\right\)\}\\;\+\\;2k\\,E\_\{\\rm tot\},which is \(8\)\. ∎

#### Remark \(initialization\)\.

If one starts Oja from any orthonormalQ\(1\)Q^\{\(1\)\}\(soP\(1\)P^\{\(1\)\}is a rank\-kkprojector\), thend1​\(V1\)2≤kd\_\{1\}\(V\_\{1\}\)^\{2\}\\leq kand the bound above holds withk2\\tfrac\{k\}\{2\}replaced bykk; usingP\(1\)=0P^\{\(1\)\}=0in the analysis yields the slightly tighterk2\\tfrac\{k\}\{2\}constant used in the theorem\.

## Appendix BMore discussions on related work

### B\.1Low\-Rank Training Paradigms

#### Parameter weights subspace projection based descent\.

Recent research has explored learned subspace descent methods, where gradient updates are confined to a low\-dimensional subspace of the model’s weights that is itself learned during training\. A straightforward instance is to parameterize each weight matrix as a product of low\-rank factors \(e\.g\.W=B​AW=BAwithB∈ℝm×r,A∈ℝr×nB\\in\\mathbb\{R\}^\{m\\times r\},A\\in\\mathbb\{R\}^\{r\\times n\}\) and optimize those factors via gradient descent\[[10](https://arxiv.org/html/2610.22794#bib.bib26),[11](https://arxiv.org/html/2610.22794#bib.bib21),[18](https://arxiv.org/html/2610.22794#bib.bib20)\]\. While such plain low\-rank factorization can significantly reduce parameters, it often falls short of full\-rank performance\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], prompting more adaptive approaches\. ReLoRA\[[14](https://arxiv.org/html/2610.22794#bib.bib25)\]tackles this by periodically absorbing the learned low\-rank updates into the main weights and introducing new factor matrices, gradually increasing the effective rank ofWWover training to recover expressiveness\. SLTrain\[[8](https://arxiv.org/html/2610.22794#bib.bib19)\]takes a complementary route by representing each weight as the sum of a trainable low\-rank matrix and a sparse matrix \(W=B​A\+SW=BA\+S\) and jointly optimizing both; this hybrid decomposition boosts model capacity while still confining updates to a learned low\-dimensional \(and partially sparse\) subspace\. Other methods learn to adjust the subspace itself: AdaLoRA\[[37](https://arxiv.org/html/2610.22794#bib.bib18)\]dynamically adapts the rank of the low\-rank adapters based on an importance metric \(simulating SVD pruning of negligible directions\), effectively learning the most productive subspace basis during training\. Likewise, LORO\[[20](https://arxiv.org/html/2610.22794#bib.bib17)\]directly optimizes the model on a low\-rank manifold \(using Riemannian optimization\), ensuring all gradient steps stay within a trainable low\-dimensional parameter space\. Beyond pure low\-rank factors, structured subspace strategies have been proposed: BTT\[[25](https://arxiv.org/html/2610.22794#bib.bib16)\]introduces a block tensor train factorization \(via Kronecker\-product structured weight matrices\) with the factor components learned by gradient descent, and CoMERA\[[34](https://arxiv.org/html/2610.22794#bib.bib15)\]reshapes weight matrices into higher\-order tensors that are decomposed into smaller learned tensors, enforcing a compact learned subspace of parameters\. All of these approaches maintain that the projection basis \(e\.g\. the low\-rank factors or tensor components\) is updated by training, allowing the subspace to evolve to fit the data\.

In contrast, other methods restrict gradient descent to a fixed or partially fixed subspace of the weights, where the projection basis remains static \(or changes only via non\-gradient steps\) for some duration\. SwitchLoRA\[[40](https://arxiv.org/html/2610.22794#bib.bib14)\]exemplifies this paradigm by using LoRA\-style low\-rank adapters but periodically switching their basis vectors \(columns ofBBor rows ofAA\) with new ones during training, instead of continuously refining the same basis\. This schedule of refreshing the adapter subspace \(inspired by ReLoRA’s rank\-growth idea\) lets the model incrementally cover diverse low\-dimensional subspaces without explicitly learning the basis via backpropagation\. Similarly, LoQT\[[17](https://arxiv.org/html/2610.22794#bib.bib13)\]employs a fixed\-basis approach for efficiency: it initializes each weight with two low\-rank factorsW≈B​AW\\approx BA, then freezes one factor \(keeping it constant and even quantized to save memory\) and only trains the other factorBB\. Periodically, the productB​ABAis added into the full weight matrixWW\(and possiblyBBis reinitialized\), but the subspace defined by the fixed factorAAremains unchanged during those intervals\. By operating in a predetermined low\-rank subspace \(or a sequence of static subspaces\), SwitchLoRA and LoQT avoid the overhead of learning new basis directions, instead confining updates to coefficient adjustments within a fixed projector\. This ensures that gradient descent explores a low\-dimensional slice of the parameter space defined a priori \(or piecewise fixed\), in contrast to the above methods that co\-learn the subspace itself during training\.

#### Gradient subspace projection descent\.

Recent research has explored gradient subspace projection descent methods to reduce memory and computation by restricting updates to a low\-dimensional subspace\. GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\]pioneered this approach, using SVD to project each gradient onto a top\-rrsingular vector subspace\. This low\-rank update dramatically lowers memory usage for gradients and optimizer states\. Numerous extensions build on GaLore’s foundation\. Fira\[[2](https://arxiv.org/html/2610.22794#bib.bib12)\]adds an orthogonal complement term to reintroduce full\-rank gradient information, closing the performance gap to full Adam without extra memory\. Natural GaLore\[[4](https://arxiv.org/html/2610.22794#bib.bib11)\]integrates natural gradient preconditioning into GaLore’s update, while Q\-GaLore\[[38](https://arxiv.org/html/2610.22794#bib.bib10)\]combines low\-rank projection with low\-bit quantization for further memory savings\. Other variants modify how the subspace is selected\. COAP\[[33](https://arxiv.org/html/2610.22794#bib.bib9)\]learns a correlation\-aware projection matrix for improved efficiency, and GRASS\[[22](https://arxiv.org/html/2610.22794#bib.bib30)\]constructs sparse projection bases guided by gradient row norms to accelerate training\. FRUGAL\[[41](https://arxiv.org/html/2610.22794#bib.bib8)\]performs an Adam\-style update in the projected subspace and a simultaneous SGD step on the orthogonal complement, enabling updates across the full parameter space with minimal extra memory overhead\. Tensor\-GaLore\[[6](https://arxiv.org/html/2610.22794#bib.bib7)\]extends the low\-rank projection principle to higher\-order tensor parameters \(beyond flat matrices\), broadening applicability while preserving memory benefits\.

#### Theory\.

We frame the phenomenon of gradient subspace drift within the broader theoretical context of concept drift in online learning\. Foundational work in this area establishes regret guarantees for learners adapting to a slowly shifting target distribution\[[3](https://arxiv.org/html/2610.22794#bib.bib35)\]\. This theoretical lens is highly relevant to LLM training, where empirical evidence shows that gradients concentrate in a low\-dimensional, slowly evolving subspace\[[7](https://arxiv.org/html/2610.22794#bib.bib27),[12](https://arxiv.org/html/2610.22794#bib.bib22)\]\. While the idea of a shifting target is well\-established, our work is the first to formally analyze a low\-rank optimizer through the framework of dynamic regret under concept drift\. This provides a principled theoretical foundation for Ori’s performance and differentiates it from prior methods that focused primarily on algorithmic or empirical contributions\.

Meanwhile, some works address the computational cost of maintaining the subspace or explore dynamic subspace strategies\. Galore\+\[[16](https://arxiv.org/html/2610.22794#bib.bib6)\]reduces SVD overhead by performing the decomposition on only a randomly chosen component \(e\.g\. one attention head\) per update, significantly speeding up training at the cost of a slight approximation error\. SubTrack\-Grad\[[27](https://arxiv.org/html/2610.22794#bib.bib4)\]and Online Subspace Descent\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\]both dynamically update the projection basis: they solve auxiliary optimization problems \(e\.g\. least\-squares fits\) to adjust the subspace so that it continues to approximate the current gradient well over time\. In contrast, Flora\[[32](https://arxiv.org/html/2610.22794#bib.bib23)\]and GoLore\[[9](https://arxiv.org/html/2610.22794#bib.bib28)\]propose using fixed random projections of gradients, achieving memory savings with competitive performance to GaLore but without expensive SVD computations\. I3S\[[36](https://arxiv.org/html/2610.22794#bib.bib29)\]introduces an importance sampling subspace selection scheme, weighting basis directions by their singular values to adaptively refine GaLore’s subspace and further improve convergence\. Finally, the recent GaLore 2 framework\[[30](https://arxiv.org/html/2610.22794#bib.bib5)\]addresses practical challenges like SVD cost and parallelization integration \(e\.g\. with FSDP\), and demonstrates the scalability of subspace\-projected training by pre\-training a 7B\-parameter LLM with up to 500B tokens using significantly less memory\.

### B\.2Online Subspace Tracking

#### Online k\-PCA subspace descent related method\.

Recent optimizers exploit low\-dimensional subspaces to reduce memory cost in large\-scale training via onlinekk\-PCA\. For example, Subspace\-Momentum confines the momentum vector to a fixedkk\-dimensional subspace \(e\.g\., spanned by top singular directions\) while performing standard SGD in the orthogonal complement, substantially lowering optimizer\-state memory and admitting high\-probability convergence guarantees under standard conditions\[[23](https://arxiv.org/html/2610.22794#bib.bib31)\]\. The subspace can be chosen once or periodically refreshed via streaming singular vectors \(e\.g\., computing top\-kkcomponents everyGGsteps\) to adapt to evolving gradients\[[23](https://arxiv.org/html/2610.22794#bib.bib31)\]\. Similarly, Online Subspace Descent forgoes expensive SVD\-based updates by continuously adapting the projection matrix through an onlinekk\-PCA algorithm\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\]\. This approach provides a convergence guarantee for arbitrary subspace update rules in such low\-rank optimizers and maintains an up\-to\-date basis with minimal overhead, eliminating full SVD computations during training\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\]\. Applied to LLM pre\-training, the online subspace optimizer achieves lower perplexity than prior low\-rank methods and narrows the performance gap to full\-rank training on benchmarks\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\]\.

### B\.3Concept Drift in Online Learning

#### Subspace learning\.

A growing body of work indicates that SGD dynamics concentrate in a low\-dimensional, slowly evolving subspace, motivating algorithms that explicitly track or exploit subspace drift during training\. Theory shows: \(i\) gradient updates lie within a tiny active subspace, suggesting that learning can be captured by a small set of dominant directions\[[7](https://arxiv.org/html/2610.22794#bib.bib27)\]; \(ii\) online learning with concept drift admits regret guarantees against a slowly shifting comparator, formalizing performance when the target \(or optimal subspace\) moves over time\[[3](https://arxiv.org/html/2610.22794#bib.bib35)\]; and \(iii\) low\-dimensional training succeeds once the subspace dimension exceeds a threshold tied to the loss landscape’s Gaussian width, explaining why few degrees of freedom can suffice and how this threshold depends on initialization\[[12](https://arxiv.org/html/2610.22794#bib.bib22)\]\. On the practical side for large\-scale LLMs, recent methods instantiate drifting\-subspace training by refreshing projection bases during optimization; among them, I3S\[[36](https://arxiv.org/html/2610.22794#bib.bib29)\]samples singular directions by importance to diversify consecutive subspaces, alleviating “frozen” dominant bases while retaining convergence guarantees comparable to random\-projection methods and delivering stronger pretraining perplexity in practice\. Together, these results motivate modeling—and algorithmically tracking—gradient subspace drift when designing memory\-efficient, very low\-rank training procedures for LLMs\.

## Appendix COri algorithm details

### C\.1AdamW with Ori

[Algorithm2](https://arxiv.org/html/2610.22794#alg2)shows the full algorithm of an AdamW optimizer equipped with Ori low\-rank gradient projector updates\. We note that Ori is not intrinsically entangled with AdamW; rather similar to GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], the Ori projector update subroutine can be easily integrated with other optimizers\. We leave such exploration to future work\.

Algorithm 2AdamW with Ori \(per\-layer update\)0:Layer weights

W∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}
0:Learning rate

η\\eta, AdamW hyperparameters

β1,β2\\beta\_\{1\},\\beta\_\{2\}, numerical constant

ϵ\\epsilon
0:Rank

kk, scale factor

γ\\gamma
0:Ori hyperparameters

TwarmupT\_\{\\text\{warmup\}\},

TperiodT\_\{\\text\{period\}\}, Oja\+\+ settings

HojaH\_\{\\text\{oja\}\}
0:Updated weights

WW
1:Initialize

Q∈ℝm×kQ\\in\\mathbb\{R\}^\{m\\times k\}randomly

2:

P←QR⁡\(Q\)P\\leftarrow\\mathrm\{QR\}\(Q\)\{PPhas orthonormal columns\}

3:Initialize first moment

M←0∈ℝk×nM\\leftarrow 0\\in\\mathbb\{R\}^\{k\\times n\}
4:Initialize second moment

V←0∈ℝk×nV\\leftarrow 0\\in\\mathbb\{R\}^\{k\\times n\}
5:

t←0t\\leftarrow 0
6:repeat

7:

G←∇Wℒ​\(W\)G\\leftarrow\\nabla\_\{W\}\\mathcal\{L\}\(W\)\{Full gradient\}

8:if

t<Twarmupt<T\_\{\\text\{warmup\}\}or

tmodTperiod=0t\\bmod T\_\{\\text\{period\}\}=0then

9:

Q←UpdateProjector​\(Q,G,Hoja\)Q\\leftarrow\\textsc\{UpdateProjector\}\(Q,G,H\_\{\\text\{oja\}\}\)\{Ori/Oja\+\+ update\}

10:

P←QR⁡\(Q\)P\\leftarrow\\mathrm\{QR\}\(Q\)
11:endif

12:

R←P⊤​GR\\leftarrow P^\{\\top\}G\{Projected gradient inℝk×n\\mathbb\{R\}^\{k\\times n\}\}

13:

M←β1​M\+\(1−β1\)​RM\\leftarrow\\beta\_\{1\}M\+\(1\-\\beta\_\{1\}\)R
14:

V←β2​V\+\(1−β2\)​\(R⊙R\)V\\leftarrow\\beta\_\{2\}V\+\(1\-\\beta\_\{2\}\)\(R\\odot R\)\{⊙\\odot: elementwise square\}

15:

M^←M/\(1−β1t\+1\)\\hat\{M\}\\leftarrow M/\(1\-\\beta\_\{1\}^\{t\+1\}\)\{Bias correction\}

16:

V^←V/\(1−β2t\+1\)\\hat\{V\}\\leftarrow V/\(1\-\\beta\_\{2\}^\{t\+1\}\)
17:

N←M^⊘\(V^\+ϵ\)N\\leftarrow\\hat\{M\}\\oslash\(\\sqrt\{\\hat\{V\}\}\+\\epsilon\)\{⊘\\oslash: elementwise divide\}

18:

G~←γ⁡\(P​N\)\\tilde\{G\}\\leftarrow\\gamma\\,\(PN\)\{Lift back toℝm×n\\mathbb\{R\}^\{m\\times n\}\}

19:

W←W−η​G~W\\leftarrow W\-\\eta\\tilde\{G\}
20:

t←t\+1t\\leftarrow t\+1
21:untilconvergence criteria met

22:return

WW

### C\.2Ori Hyperparameters and Recommended Search Strategy

In total, Ori consists of the following 7 important hyperparameters, listed roughly in the order of the most to least influential to model performance as we observed during method development\.

- •Scaleγ\\gamma: Similar to GaLore’s scaleα\\alpha\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], this parameter effectively scales the actual learning rate for the model layers for which Ori project the gradients\. We suggest always search over this parameter as it highly correlates with model performance\. Empirically, we found that one can set a higherγ\\gammavalue in the extremely low rank settings, while smallerγ\\gammatend to yield more stable training at higher ranks\. We suggestγ∈\{0\.25,0\.5,0\.75,1\.0\}\\gamma\\in\\\{0\.25,0\.5,0\.75,1\.0\\\}for large model pre\-training, andγ∈\{2\.0,4\.0\}\\gamma\\in\\\{2\.0,4\.0\\\}for very low\-rank fine\-tuning\.
- •Drift coefficientρ\\rho: This parameter controls how much the last step’s projector is carried over to warm\-start the next step’s projector Oja updates\. Aρ=0\.0\\rho=0\.0means total random restart, and aρ=1\.0\\rho=1\.0means total carry\-over with no randomness injected\. As established in[Section5\.4](https://arxiv.org/html/2610.22794#S5.SS4), theρ=0\.99\\rho=0\.99tend to be the universally good setting\. We also found thatρ∈\{0\.8,0\.9\}\\rho\\in\\\{0\.8,0\.9\\\}can be good alternatives in a few scenarios\.
- •Projector update periodTperiodT\_\{\\text\{period\}\}: This parameter controls the frequency of Ori’s update\. As established in[Section5\.4](https://arxiv.org/html/2610.22794#S5.SS4), we suggestTperiod=20T\_\{\\text\{period\}\}=20as it strikes a perfect balance between model performance and computation budget\.
- •Ori warmup periodTwarmupT\_\{\\text\{warmup\}\}: This parameter controls for how many steps at the begging does Ori ignoreTperiodT\_\{\\text\{period\}\}and update its projector consecutively\. We recommand avoid setting too large a value as it may lead to overfitting\. Empirically, we observe thatTwarmup=20T\_\{\\text\{warmup\}\}=20tend to be stable\.
- •Initial Oja learning rateη\\eta: The initial learning rate for the Oja update \([Equation2](https://arxiv.org/html/2610.22794#S3.E2)\), which gets decreased by a cosine annealing scheduler, and then gets reset at the next model update step\. We observe that, as long asη\\etais not too small, Ori yields stable learning andη\\eta’s specific value do not have too much influence\. We recommend settingη=1\.0\\eta=1\.0for all scenarios\.
- •Number of inner Oja\-loop updatesNupdateN\_\{\\text\{update\}\}: This parameter controls the number of inner Oja updates \(see[Algorithm1](https://arxiv.org/html/2610.22794#alg1)\)\. Note that since Ori uses Oja\+\+ outer loop, the total number of inner Oja updates depends on the optimizer rank:\(log2⁡\(k\)\+1\)​Nupdate\(\\log\_\{2\}\(k\)\+1\)N\_\{\\text\{update\}\}\. In practice, we setNupdate=2N\_\{\\text\{update\}\}=2for all experiments\.
- •Number of gradient columns to sample for Oja updateNvecsN\_\{\\text\{vecs\}\}: In practice, we observe that setting it in a way that scales with the model’s embedding dimension is a good choice\. We suggestNvecs∈\{dmodel/2,dmodel,dmodel×2\}N\_\{\\text\{vecs\}\}\\in\\\{d\_\{\\text\{model\}\}/2,d\_\{\\text\{model\}\},d\_\{\\text\{model\}\}\\times 2\\\}\.

### C\.3Amortized Complexity Analysis

In this section, we analyze the amortized complexity of Ori update steps as a function of the hyperparameter knobs as described in[Algorithm2](https://arxiv.org/html/2610.22794#alg2)\.

#### Setup\.

Letd×nd\\times nbe the weight matrix dimensions,rrthe rank,TperiodT\_\{\\text\{period\}\}the update interval,NupdatesN\_\{\\text\{updates\}\}the inner Oja iterations per block stage,NvecsN\_\{\\text\{vecs\}\}the number of sampled gradient columns, andB=⌈log2⁡\(r\)⌉B=\\lceil\\log\_\{2\}\(r\)\\rceilthe number of Oja\+\+ block stages\.

#### Per\-step cost \(every step, lines 12–19 of[Algorithm2](https://arxiv.org/html/2610.22794#alg2)\)\.

Projecting the gradientR=P⊤​GR=P^\{\\top\}Gand lifting backG~=γ⁡\(P​N\)\\tilde\{G\}=\\gamma\(PN\)each costO⁡\(d​n​r\)O\(dnr\)\. The Adam update in the projected space costsO⁡\(r​n\)O\(rn\)\. Thus:

Cstep=O⁡\(d​n​r\)\.C\_\{\\text\{step\}\}=O\(dnr\)\.

#### Per\-update cost \(everyTperiodT\_\{\\text\{period\}\}steps,[Algorithm1](https://arxiv.org/html/2610.22794#alg1)\)\.

The Oja\+\+ outer loop runsBBblock stages with growing active slice sizes, each performingNupdatesN\_\{\\text\{updates\}\}inner iterations\. Each inner iteration costsO⁡\(d⋅Nvecs⋅rnext​\(k\)\)O\(d\\cdot N\_\{\\text\{vecs\}\}\\cdot r\_\{\\text\{next\}\}\(k\)\), wherernext​\(k\)r\_\{\\text\{next\}\}\(k\)is the active slice at stagekk\. QR re\-orthogonalization costsO⁡\(d​r2\)O\(dr^\{2\}\)per update\. Since∑k=0B−1rnext​\(k\)=O⁡\(r​B\)\\sum\_\{k=0\}^\{B\-1\}r\_\{\\text\{next\}\}\(k\)=O\(rB\):

Cupdate=O⁡\(B⋅Nupdates⋅d⋅Nvecs⋅r\+d​r2\)\.C\_\{\\text\{update\}\}=O\(B\\cdot N\_\{\\text\{updates\}\}\\cdot d\\cdot N\_\{\\text\{vecs\}\}\\cdot r\+dr^\{2\}\)\.

#### Amortized per\-training\-step cost\.

Camortized=O⁡\(d​n​r\)\+O⁡\(B⋅Nupdates⋅d⋅Nvecs⋅r\+d​r2Tperiod\)\.C\_\{\\text\{amortized\}\}=O\(dnr\)\+O\\left\(\\frac\{B\\cdot N\_\{\\text\{updates\}\}\\cdot d\\cdot N\_\{\\text\{vecs\}\}\\cdot r\+dr^\{2\}\}\{T\_\{\\text\{period\}\}\}\\right\)\.
In our default configuration \(Nupdates=2N\_\{\\text\{updates\}\}=2,Tperiod=20T\_\{\\text\{period\}\}=20\), the amortized update overhead scales asO⁡\(B⋅Nvecs⋅d​r10\)O\\left\(\\frac\{B\\cdot N\_\{\\text\{vecs\}\}\\cdot dr\}\{10\}\\right\), which is small relative toCstep=O⁡\(d​n​r\)C\_\{\\text\{step\}\}=O\(dnr\)whenB⋅Nvecs/10≪nB\\cdot N\_\{\\text\{vecs\}\}/10\\ll n, which is easily satisfied in practice \(e\.g\.,B=7B=7,Nvecs=512N\_\{\\text\{vecs\}\}=512,n=2048n=2048gives ratio≈0\.17\\approx 0\.17\)\.

#### Memory per layer\.

Ori stores the projectorP∈ℝd×rP\\in\\mathbb\{R\}^\{d\\times r\}and Adam momentsM,V∈ℝr×nM,V\\in\\mathbb\{R\}^\{r\\times n\}, totalingO⁡\(r⁡\(d\+2​n\)\)O\(r\(d\+2n\)\)\. Atr=128r=128,d=n=2048d=n=2048, this is 9\.4% of full\-rank Adam’sO⁡\(2​d​n\)O\(2dn\), as confirmed later by[Table9](https://arxiv.org/html/2610.22794#A6.T9)\(0\.25060\.2506GB for Ori vs2\.49422\.4942GB for Adam\)\.

Despite updating10×10\\timesmore frequently, Ori is the fastest gradient\-subspace method\. This pattern holds across all model sizes and ranks \(as shown in[Table11](https://arxiv.org/html/2610.22794#A6.T11)\)\.

## Appendix DStochastic Power Iteration for LDAdam

LDAdam\[[28](https://arxiv.org/html/2610.22794#bib.bib2)\]is a memory\-efficient variant of Adam for low\-rank LLM training that performs adaptive updates inside a low\-dimensional gradient subspace, while using projection\-aware rules to transfer the first/second\-moment states of Adam as the subspace changes over time\. It computes the subspace via the warm\-started block power iteration method \(warm\-started by SVD library call at the very first update step\) and adds a generalized error\-feedback mechanism to compensate for information lost when compressing both gradients and the first/second\-moment optimizer states\. As a result, it approaches Adam\-like behavior with much smaller optimizer memory in the low\-rank setting\.

A key practical limitation, however, is that LDAdam typically requires per\-step projector updates \(T=1\) to maintain accuracy, which can be computationally prohibitive\. On the other hand, naively increasing LDAdam’s subspace update interval toT=200T=200– matching the standard schedule used by GaLore – causes a significant performance drop\. This is shown in[Table2](https://arxiv.org/html/2610.22794#S5.T2), our main results for pretraining the Llama model on C4 dataset\. In particular, on the largest, 1b\-size model, we find that the training process under LDAdam withT=200T=200may fail to converge, leading to very large evaluation perplexity \(above 160 perplexity for both rankr=512r=512andr=128r=128\)\.

We attribute this sensitivity and instability to LDAdam’s reliance on a single\-step full\-block power\-iteration\-style refresh: when the gradient subspace drifts over long intervals \(e\.g\.,T=200T=200\), a one\-step update is often insufficient to re\-align the projector\.

Motivated by Ori’s online subspace tracking, we replace LDAdam’s single\-step, full\-block power iteration with a multi\-step stochastic procedure\. Concretely, at each steptt, given the input matrixMMfrom which we intend to compute the subspace basis, we sample a row or column vectorxtx\_\{t\}fromMM, then update the basis via

Qt←QR​\(\(ηt​xt​xt⊤\)​Qt−1\),Q\_\{t\}\\leftarrow\\text\{QR\}\\left\(\(\\eta\_\{t\}x\_\{t\}x\_\{t\}^\{\\top\}\)Q\_\{t\-1\}\\right\),\(26\)whereηt\\eta\_\{t\}is the update rate at steptt\. This update can be viewed as a special case of Ori’s Oja\-style rule with an explicit identity “carry\-over” term,

Qt←QR​\(\(α​I\+ηt​xt​xt⊤\)​Qt−1\),Q\_\{t\}\\leftarrow\\text\{QR\}\\left\(\(\\alpha I\+\\eta\_\{t\}x\_\{t\}x\_\{t\}^\{\\top\}\)Q\_\{t\-1\}\\right\),\(27\)where stochastic power iteration corresponds to settingα=0\\alpha=0\. We refer to the resulting optimizer as O\-LDAdam, reflecting the integration of Ori\-style online subspace tracking into LDAdam’s projection\-aware, error\-correcting framework\.[Algorithm3](https://arxiv.org/html/2610.22794#alg3)presents the algorithm pseudocode of O\-LDAdam\.

Algorithm 3O\-LDAdam1:forweightinmodel\.parameters\(\):

2:grad

←\\leftarrowweight\.grad

3:\# states: basis

UU, moments

\(m,v\)\(m,v\), error buffer

ξ\\xi
4:

A←A\\leftarrowgrad

\+ξ\+\\xi\{unload error buffer\}

5:

Uold←UU\_\{\\text\{old\}\}\\leftarrow U
6:for

i=1i=1to

NstepsN\_\{\\text\{steps\}\}:\{stochastic subspace tracking\}

7:

xi←SampleColumn​\(A\)x\_\{i\}\\leftarrow\\texttt\{SampleColumn\}\(A\)
8:

U←QR​\(\(α​I\+ηi​xi​xi⊤\)​U\)U\\leftarrow\\text\{QR\}\\left\(\(\\alpha I\+\\eta\_\{i\}x\_\{i\}x\_\{i\}^\{\\top\}\)U\\right\)\{α=0\\alpha=0for stochastic PI\}

9:end for

10:

\(m,v\)←transfer\_states\(Uold→U,m,v\)\(m,v\)\\leftarrow\\texttt\{transfer\\\_states\}\(U\_\{\\text\{old\}\}\\to U,m,v\)\{proj\-aware transfer\}

11:a

←U⊤​A\\leftarrow U^\{\\top\}A\{original→\\tocompact\}

12:

Δ​a,\(m,v\)←AdamUpdate​\(a,m,v\)\\Delta a,\(m,v\)\\leftarrow\\texttt\{AdamUpdate\}\(a,m,v\)\{update in compact space\}

13:

Δ←U​Δ​a\\Delta\\leftarrow U\\,\\Delta a\{compact→\\tooriginal\}

14:weight\.data

←\\leftarrowweight\.data

\+Δ\+\\Delta
15:

ξ←error\_feedback​\(A,Uold,U,m\)\\xi\\leftarrow\\texttt\{error\\\_feedback\}\(A,U\_\{\\text\{old\}\},U,m\)\{compensate compression\}

## Appendix EExperiment details

### E\.1Dataset and Training Details

We use the C4 dataset for the LLaMA model pre\-training task\. C4 dataset is a colossal, cleaned version of Common Crawl’s web crawl corpus, mainly intended for pre\-training language models and word representations, available under the Open Data Commons Attribution License \(ODC\-BY\) license\[[26](https://arxiv.org/html/2610.22794#bib.bib37)\]\. Similar to prior work’s experiment settings, we train the model without data repetition over a sufficiently large amount of data\. We report the final validation perplexity \(the lower the better\) for the main experiment table \([Table1](https://arxiv.org/html/2610.22794#S5.T1)\)\.

For all pre\-training experiments, we adjust the gradient accumulation steps accordingly, depending on the available GPUs, so that the effective total batch size for all runs are 512 \(for instance, an actual batch size of 64 with 8 gradient accumulation steps results in the effective batch size of 512\)\. We set the LLaMA model max length to 256, adopted a cosine learning rate scheduler with a 0\.1 minimum learning rate ratio, and trained with bfloat16 data type\.

We use the GLUE dataset for the Roberta\-Base model fine\-tuning task\. GLUE is a benchmark for evaluating the performance of NLP models on a variety of tasks, including question answering, sentiment analysis, and textual entailment\[[31](https://arxiv.org/html/2610.22794#bib.bib38)\]\. We report the final accuracy score \(the higher the better\) for the main experiment table \([Table3](https://arxiv.org/html/2610.22794#S5.T3)\)\. The metric of choice for each GLUE task is listed in[Table4](https://arxiv.org/html/2610.22794#A5.T4)\. Notably, different from many of the prior optimization works, we run each task \+ optimizer \+ rank combination for three runs with different random seeds to improve statistical confidence level\. Thus, we were able to report the error bar \(standard deviation\) in the table\.

For all the fine\-tuning experiments, similar to prior work’s experiment settings, we train all models for a total of 30 epochs, with 1 gradient accumulation step\. We set the model max length to 512, and adopted a linear learning rate scheduler\.

Table 4:Accuracy metrics we used for the Roberta\-Base fine\-tuning experiments of[Table3](https://arxiv.org/html/2610.22794#S5.T3)\.
### E\.2Compute resources

We used a variety of compute resources with different GPU hardware, including RTX 4090, RTX 4070 Ti Super, RTX A500, RTX A6000, 40GB A100, 80GB A100, H100, MI100, and MI210\. For the large model pre\-training, such as the 350M and 1B LLaMA model pre\-training experiments, we trained the model on nodes equipped with 4 H100 or 4 MI210\.

### E\.3Ori hyperparameters

We report the detailed Ori hyperparameters used in each experiment setting for reproducibility\.[Table5](https://arxiv.org/html/2610.22794#A5.T5)lists all hyperparameters used for the LLaMA\-C4 pre\-trianing tasks, and[Table6](https://arxiv.org/html/2610.22794#A5.T6)lists all hyperparameters used for the Roberta\-Base\-GLUE fine\-tuning tasks\.

Table 5:Ori hyperparameter used for the LLaMA pre\-training results on the C4 dataset \([Table1](https://arxiv.org/html/2610.22794#S5.T1)\)\.Table 6:Ori hyperparameter used for the Roberta\-Base fine\-tuning results on the GLUE dataset \([Table3](https://arxiv.org/html/2610.22794#S5.T3)\)
### E\.4O\-LDAdam hyperparameters

We report the subset of O\-LDAdam hyperparemeters pertinent to the Ori\-inspired stochastic power iteration update in each experiment setting for reproducibility\. The rest of O\-LDAdam hyperparameters are inherited from, and thus the same as, the baseline LDAdam\[[28](https://arxiv.org/html/2610.22794#bib.bib2)\]\.[Table7](https://arxiv.org/html/2610.22794#A5.T7)lists all hyperparameters used for the LLaMA\-C4 pretraining tasks, and[Table8](https://arxiv.org/html/2610.22794#A5.T8)lists all hyperparameters used for the Roberta\-Base\-GLUE fine\-tuning tasks\.

Table 7:O\-LDAdam hyperparameter related to the Ori\-inspired stochastic power iteration update used for the LLaMA pre\-training results on the C4 dataset \([Table2](https://arxiv.org/html/2610.22794#S5.T2)\)\.Table 8:O\-LDAdam hyperparameter related to the Ori\-inspired stochastic power iteration update used for the Roberta\-Base fine\-tuning results on the GLUE dataset \([Table3](https://arxiv.org/html/2610.22794#S5.T3)\)
### E\.5Baseline hyperparameters

Baseline hyperparameters are discussed in the main experiment section \(see[Section5\.1](https://arxiv.org/html/2610.22794#S5.SS1)\)\.

### E\.6Visualizing subspace drift

We reused the setting of 60M LLaMA model pre\-training on C4 dataset to visualize an actual gradient subspace drift as the model is being trained in real life, resulting in[Figure2](https://arxiv.org/html/2610.22794#S3.F2)\.

For this experiment, we trained the 60M LLaMA model with the Adam optimizer for a reduced 1000 training steps, totalling 100M training tokens\. We setΔ​t=10\\Delta t=10,k∈\{128,32,8\}k\\in\\\{128,32,8\\\}, and computedoverlap​\(Uk\(t\),Uk\(t\+Δ​t\)\)\\text\{overlap\}\\left\(U^\{\(t\)\}\_\{k\},U^\{\(t\+\\Delta t\)\}\_\{k\}\\right\)\([Equation1](https://arxiv.org/html/2610.22794#S3.E1)\) for every layers in the model\. We then report and plot the mean overlap, averaged over all layers, through the training trajectories\.

To further demonstrate that the overlap values shown in[Figure2](https://arxiv.org/html/2610.22794#S3.F2)are not ad\-hoc and it indeed signify that the gradient subspace has non\-trivial overlapping, we simulated three random baselines, one for each rankk∈\{128,32,8\}k\\in\\\{128,32,8\\\}\. The random baseline is conducted by sampling a random matrix with the same shape and Frobenius norm as the true gradient matrix, for each layer at each step\. We then pretend these random matrices were some “gradients”, and record their overlap values with the earlier random matrices\.

### E\.7Ablation study details

We reused the setting of 60M LLaMA model pre\-training on C4 dataset to perform the ablation study\. The ori optimizer in the ablation studies used the same hyperparameters as detailed in[Table5](https://arxiv.org/html/2610.22794#A5.T5)\.

For the ablation of Oja\+\+ vs\. Standard Oja, recall that the default choice of Oja\+\+ algorithm consists of an outer loop and an inner loop[Algorithm1](https://arxiv.org/html/2610.22794#alg1), and the total number of Oja inner updates in the default setting is\(log2⁡\(k\)\+1\)​Nupdate\(\\log\_\{2\}\(k\)\+1\)N\_\{\\text\{update\}\}\. Thus, to ensure fairness, we set this number to the Oja\-variant of Ori\. That is, since we useNupdate=2N\_\{\\text\{update\}\}=2for all experiments, in this ablation study, we set the number of Oja baseline udpates to 16 whenk=128k=128, 12 whenk=32k=32, and finally 8 whenk=8k=8\.

### E\.8Measuring optimizer state peak memory

Finally, we report our procedure for measuring the memory footprint of the optimizers evaluated in our experiments\. Our goal is to record the peak memory usage, incurred solely by the said optimizer’s update step\. To this end, we take advantage of PyTorch’s memory recording functionalities\.

The below code listing shows the pseudocode for recording peack memory usage incurred solely by the optimizer’sstep\(\)operations\. We use this procedure to obtain all the results in[Tables9](https://arxiv.org/html/2610.22794#A6.T9)and[11](https://arxiv.org/html/2610.22794#A6.T11)\.

Python code for profiling optimizer peak memory\.

importtorch

model=…

optimizer=…

forbatchinenumerate\(dataloader\):

loss=model\(\*\*batch\)\.loss

loss\.backward\(\)

mem\_after\_backward=torch\.cuda\.memory\_allocated\(\)

torch\.cuda\.reset\_peak\_memory\_stats\(\)

optimizer\.step\(\)

peak\_mem\_during\_step=torch\.cuda\.max\_memory\_allocated\(\)

optim\_peak\_mem=peak\_mem\_during\_step\-mem\_after\_backward

## Appendix FAdditional experiment results

### F\.1Memory and Computation Time

Table 9:Memory and Wallclock Time Comparison between Ori and non\-error\-feedback baselines \(Adam, ReLoRA, GaLore, OSD\)\. Analysis is conducted by pre\-training 1B LLaMA on C4 dataset\.Table 10:Memory and Wallclock Time Comparison between O\-LDAdam and an error\-feedback baseline, LDAdam\. Analysis is conducted by pre\-training 1B LLaMA on C4 dataset\.We profile the optimizer state peak memory and wall\-clock time during the pre\-training experiments from[Section5\.2](https://arxiv.org/html/2610.22794#S5.SS2)\. Details on the memory profiling procedure can be found in[SectionE\.8](https://arxiv.org/html/2610.22794#A5.SS8)\. As shown in[Table9](https://arxiv.org/html/2610.22794#A6.T9), Ori exhibits a superior efficiency profile among standard, non\-error\-feedback gradient\-subspace methods \(Adam, ReLoRA, GaLore, OSD\)\. In addition, as shown in[Table10](https://arxiv.org/html/2610.22794#A6.T10), O\-LDAdam demonstrates competitive and comparable efficiency profile to the error\-feedback baseline LDAdam under the large update frequency regime \(T=200T=200\)\. Full table on pre\-training of other model sizes can be found in[AppendixF](https://arxiv.org/html/2610.22794#A6)\.

Memory Usage:Compared to standard, non\-error\-feedback baselines, Ori consistently consumes the least optimizer peak memory across all ranks\. For instance, at rank 32, Ori requires only 0\.2487 GB, which is 12\.1% less than OSD \(0\.2832 GB\) and 26\.5% less than GaLore \(0\.3385 GB\)\. This confirms Ori’s tangible memory savings\.

On the other hand, compared to the error\-feedback methods, O\-LDAdam consistently yields the same memory footprint as LDAdam baselines\.

Training Speed:Despite updating its subspace projector 10x more frequently \(TOri=20T\_\{\\text\{Ori\}\}=20vs\.TGaLore/OSD=200T\_\{\\text\{GaLore/OSD\}\}=200\), Ori is still the fastest among all standard, non\-error\-feedback methods\. While the weight\-subspace method ReLoRA is faster at lower ranks due to its different update mechanism, its worse model performance \(see[Table1](https://arxiv.org/html/2610.22794#S5.T1)\) highlights a clear trade\-off\. In contrast, Ori achieves superior model performance without compromising on speed, making it a more effective and efficient choice overall\. The lightweight nature of the Oja\+\+ update allows Ori to be both more accurate and faster than its direct competitors, making it a better choice for efficient training\.

On the other hand, although O\-LDAdam yields a slightly longer average update time than LDAdam withT=200T=200due to the need to perform power iterations multiple times \([Algorithm3](https://arxiv.org/html/2610.22794#alg3)\), the additional cost is not prohibitively large\. Moreover, O\-LDAdam demonstrates a nontrivial speedup compared to the standard LDAdam setting with a per\-step update schedule \(T=1T=1\)\. Together with the model performance results shown in[Table2](https://arxiv.org/html/2610.22794#S5.T2), this demonstrates that O\-LDAdam strikes a good balance between model performance and computational cost in the extremely low\-rank regime\.

[Table11](https://arxiv.org/html/2610.22794#A6.T11)presents the full results of memory and Wallclock time comparison of all the optimizer methods on the LLaMA model pre\-training task on the C4 dataset\. As shown by the table, our method Ori achieves superior memory and computation time profile almost across the broad, demonstrating its general efficiency under various settings\.

Table 11:Memory and Wallclock Time Comparison of pre\-training various sized LLaMa model on the C4 dataset\.

## Appendix GLimitations

We discuss several limitations of this work that present opportunities for future research\.

#### Behavior under heterogeneous training data\.

Our empirical evaluation focuses on single\-domain pretraining \(the C4 dataset\[[26](https://arxiv.org/html/2610.22794#bib.bib37)\]\) and standard fine\-tuning benchmarks \(the GLUE dataset\[[31](https://arxiv.org/html/2610.22794#bib.bib38)\]\), following the established experimental protocol used by GaLore\[[39](https://arxiv.org/html/2610.22794#bib.bib33)\], OSD\[[15](https://arxiv.org/html/2610.22794#bib.bib32)\], and other gradient\-subspace methods\. An important open question, shared by the many works of low\-rank optimizer research, is whether global gradient subspace projection methods perform well when training on highly heterogeneous data mixtures spanning diverse domains \(e\.g\., code, mathematics, chat, and multilingual data simultaneously\)\. In such settings, it is conceivable that the top\-kkprincipal gradient directions could be dominated by the most prevalent domains, potentially suppressing rare but important domain\-specific gradient signals\. Recent work by[Li et al\. \[13\]](https://arxiv.org/html/2610.22794#bib.bib39)suggests that gradient conflicts are minimal during large\-scale LLM pretraining across multilingual and multi\-domain settings, which provides initial evidence that this concern may be less severe than expected\. Nevertheless, a rigorous investigation—including defining appropriate evaluation metrics for specialized capability retention, curating heterogeneous benchmarks, and systematically comparing low\-rank methods under diverse data compositions—would be a valuable contribution to the field\. Notably, Ori’s high\-frequency streaming updates and potential easy composability with standard multi\-task optimization methods \(e\.g\., PCGrad\[[35](https://arxiv.org/html/2610.22794#bib.bib40)\]\) position it well for such settings, but empirical validation remains future work\.

#### Optimizer state handling across basis changes\.

When Ori updates the gradient projectorPPvia Oja\+\+, the Adam optimizer momentsMMandVVare not explicitly rotated into the new basis\. Instead, Ori relies on the slow\-drift property \(ρ=0\.99\\rho=0\.99\) and Adam’s exponential moving average to naturally decay stale moment information\. While this is empirically effective \(as validated by the high subspace overlap in[Figure2](https://arxiv.org/html/2610.22794#S3.F2)and the strong end\-to\-end performance\), explicitly projecting optimizer states across basis changes—as explored by[Schotthöfer et al\. \[29\]](https://arxiv.org/html/2610.22794#bib.bib41)in the context of weight\-space low\-rank factorization—could yield further improvements, particularly in regimes with faster subspace drift or lowerρ\\rho\. We consider this a promising direction for future work\.

#### Adaptive rank\.

Ori operates at a fixed rankrrthroughout training, which must be specified in advance\. Combining Ori’s streaming subspace tracking with rank\-adaptive methods such as AdaLoRA\[[37](https://arxiv.org/html/2610.22794#bib.bib18)\], which dynamically allocates rank budgets across layers based on importance metrics, could enable more flexible and efficient low\-rank training\. Since Ori’s subspace tracking mechanism is orthogonal to rank selection, such a combination is a natural extension\.

## Appendix HBroader Social Impact

This paper proposes a novel low\-rank optimizer for training large language models, which is a foundational research area in machine learning\. As such, we do not foresee any immediate negative societal impacts\. However, we recognize that our work, like most AI research, could have broad, long\-term implications, including potential issues with energy consumption or, indirectly, in downstream applications\.