No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

# Non-Identifiability and Optimizer State in Low-Rank Training
Source: [https://arxiv.org/html/2607.05872](https://arxiv.org/html/2607.05872)
###### Abstract

Memory\-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank\-rrsubspace that is recomputed everyTTsteps, on the assumption that this subspace is a slowly drifting object that can be tracked\. We show that beyond a small reproducible core, there is no such object\. Two estimates of the top\-rrsubspace computed*at the same step*from disjoint minibatches disagree as much as estimates computedTTsteps apart \(0\.730\.73versus0\.740\.74of the maximal chordal distance2​r\\sqrt\{2r\}, measured at Pythia\-160M withr=128r\{=\}128\): the apparent rotation at every refresh is dominated by estimator noise\. This holds across four model families in three architecture classes and from 70M to 6\.9B parameters, strengthening with scale, and more weakly in a vision transformer\. Only∼39\{\\sim\}39of128128directions are reproducible across minibatches, and averaging cannot recover the rest: underNN\-fold averaging the gradient’s spectral tail shrinks asN−1/4N^\{\-1/4\}rather than theN−1/2N^\{\-1/2\}of pure noise, so no averaging budget makes the subspace well defined\. At Pythia\-1B, a basis averaged over ten thousand steps still moves by∼84%\{\\sim\}84\\%of the geometric maximum per refresh, and sweeping the averaging strength leaves perplexity essentially unchanged \(a0\.360\.36perplexity span\)\. What helps instead follows from treating each refresh as a change of coordinates for Adam’s state\. Carrying the second moment blindly is provably about\(r−k⋆\)/2\(r\{\-\}k^\{\\star\}\)/2worse than the best rotation\-blind estimator, while the first moment transports exactly through the rotation, the optimal linear map under isotropic gradients and the rule LDAdam uses\. In a three\-seed comparison at 1B and 40k steps, the full LDAdam update reaches18\.718\.7perplexity at the defaultβ2=0\.999\\beta\_\{2\}\{=\}0\.999, beating untransported GaLore after its bestβ2\\beta\_\{2\}fix \(19\.319\.3\)\. Shortening the second\-moment memory toβ2=0\.99\\beta\_\{2\}\{=\}0\.99helps the refreshing optimizers at their reported recipes, by1\.81\.8to2626perplexity, though for canonical GaLore the effect is small and recipe\-sensitive; a full\-rank control reverses the preference\. One measurable fact, subspace non\-identifiability, clarifies why GaLore works, which patches work, and what to check before trusting a low\-rank assumption: the reproducible rankk⋆k^\{\\star\}\.

## 1Introduction

Training large models is increasingly bottlenecked by optimizer memory: Adam stores two moments per parameter, doubling the footprint of the weights themselves\. Low\-rank gradient optimizers such as GaLore\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib4)\)sidestep this by projecting each gradientG∈ℝm×nG\\in\\mathbb\{R\}^\{m\\times n\}onto the top\-rrleft\-singular subspace ofGG, running Adam in the resultingrr\-dimensional space, and recomputing the subspace everyTTsteps\. This matches full\-rank Adam at a fraction of the memory, and a family of refinements has grown around it\(Robertet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib6); Rajabiet al\.,[2025b](https://arxiv.org/html/2607.05872#bib.bib7); Zmushkoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib62); Heet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib26); Zhuet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib19)\)\.

Underlying the whole approach is a geometric premise that is rarely stated and, to our knowledge, was never measured: that the gradient’s top\-rrsubspace is a meaningful, slowly varying object that the refresh tracks\. If the premise held, two consecutive estimates of the subspace would nearly coincide, and carrying optimizer state across a refresh would be a small correction\.

We measure the premise and find that beyond a small reproducible core, there is no such object\. The rotation between consecutive subspaces, measured as the chordal distance‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}, saturates its geometric maximum2​r\\sqrt\{2r\}to96\.796\.7–99\.6%99\.6\\%at every refresh and at every rankr∈\{32,…,256\}r\\in\\\{32,\\dots,256\\\}during training \(Pythia\-1B\(Bidermanet al\.,[2023](https://arxiv.org/html/2607.05872#bib.bib22)\); 160M behaves the same\): each refresh adopts a nearly orthogonal frame\. The decisive measurement is the one that removes time\. On a pretrained Pythia\-160M checkpoint, two estimates of the top\-rrsubspace computed*at the same step*from disjoint minibatches disagree by0\.725​2​r0\.725\\,\\sqrt\{2r\}, within a few percent of the0\.742​2​r0\.742\\,\\sqrt\{2r\}the subspace rotates across a full refresh interval of160160M training \(a gap the trajectory control of §[3\.2](https://arxiv.org/html/2607.05872#S3.SS2)confirms is small at every training state\)\. The apparent rotation is not motion\. It is estimator noise: of ther=128r\{=\}128directions GaLore extracts, onlyk⋆≈39k^\{\\star\}\\approx 39are reproducible across two halves of one batch, and the rest are re\-drawn with the sample\. \(Both checkpoint values sit below the training\-time saturation for a reason the paper quantifies: a pretrained gradient’s reproducible spike is stronger, and thek⋆k^\{\\star\}shared directions pull the two estimates together\.\)

The cause is spectral, and it is not curable by averaging\. The per\-matrix gradient spectrum has a small dominant spike, a few to∼20\{\\sim\}20directions consistent with the tiny gradient subspace documented for classification byGur\-Ariet al\.\([2018](https://arxiv.org/html/2607.05872#bib.bib21)\), followed by a smooth tail with no gap anywhere near the ranks GaLore uses: atr=128r\{=\}128the adjacent singular\-value ratio is≈1\.005\{\\approx\}1\.005, the effective rank straddles the cutoff, and the top\-rrblock holds only4646–68%68\\%of the gradient’s nuclear mass\. A rank cut inside a gapless bulk returns near\-degenerate directions that resample freely \(Davis–Kahan\)\. AveragingNNgradients does not open a gap: the deep spectral tail shrinks only as≈N−1/4\{\\approx\}N^\{\-1/4\}rather than theN−1/2N^\{\-1/2\}of pure noise, because it is itself signal with a power\-law spectrum, so no averaging budget makes the top\-rrframe well defined\. GaLore works anyway, and the reason is unglamorous: it captures a roughly constant fraction of gradient energy in a frame that is re\-drawn, not tracked \(§[4](https://arxiv.org/html/2607.05872#S4)\)\.

The redraw has a concrete cost, and the cost has remedies that we can now rank\. Adam’s second moment is a slow average with memoryτv=1/\(1−β2\)≈1000\\tau\_\{v\}=1/\(1\-\\beta\_\{2\}\)\\approx 1000steps at the defaultβ2=0\.999\\beta\_\{2\}\{=\}0\.999; a frame scrambled everyT≪τvT\\ll\\tau\_\{v\}steps leaves it chronically misaligned\. The obvious fix, stabilizing the basis by averaging the projection over time, fails exactly as the geometry predicts: at Pythia\-1B a basis averaged over ten thousand steps still moves by84%84\\%of maximal per refresh \(against97%97\\%with no averaging\), and final perplexity is essentially flat across four decades of averaging window \(a0\.360\.36perplexity span\)\. What works is moving the state, not the frame\. The first moment transports exactly through the rotationR=Unew⊤​UoldR=U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}; the second moment transports through the squared entries ofRR, and a carry that ignoresRRis provably a factor≈\(r−k⋆\)/2\{\\approx\}\(r\{\-\}k^\{\\star\}\)/2worse than the best blind alternative \(§[5](https://arxiv.org/html/2607.05872#S5), Appendix[B](https://arxiv.org/html/2607.05872#A2)\)\. In a three\-seed controlled comparison at 1B, transported state \(the LDAdam update, which combines transport with error feedback\) at the defaultβ2\\beta\_\{2\}reaches18\.7318\.73perplexity, better than untransported GaLore after its bestβ2\\beta\_\{2\}fix \(19\.2819\.28\)\. The cheaper remedy, shortening the memory toβ2=0\.99\\beta\_\{2\}\{=\}0\.99, helps the refreshing optimizers we test at their matched recipes, from1\.81\.8to2626perplexity depending on optimizer and recipe \(both the size and, for canonical GaLore, the sign of the effect are recipe\-dependent\), while a full\-rank control with no refreshes reverses the preference, isolating the redraw as the cause\.

#### Contributions\.

- •The object is not there\.A same\-step control showing that the GaLore subspace’s apparent per\-refresh rotation \(‖Δ​U‖F≈2​r\\\|\\Delta U\\\|\_\{F\}\\approx\\sqrt\{2r\}, saturated96\.796\.7–99\.6%99\.6\\%across ranks and scales\) equals the disagreement between two simultaneous estimates: the subspace is statistically non\-identifiable beyond a small reproducible core ofk⋆≈39k^\{\\star\}\\approx 39of128128directions \(§[3](https://arxiv.org/html/2607.05872#S3)\)\. The same holds across four model families \(three architecture classes\), from 70M to 6\.9B parameters, and in a vision transformer, strengthening with scale \(§[3\.6](https://arxiv.org/html/2607.05872#S3.SS6)\)\.
- •Why, and why averaging cannot fix it\.The gradient spectrum is gapless atrr, and its deep tail is power\-law signal that shrinks underNN\-fold averaging only as≈N−1/4\{\\approx\}N^\{\-1/4\};k⋆k^\{\\star\}is predictable from the spectrum \(within8%8\\%of the direct count, with the per\-layer ordering matching measured energy retention\), and no averaging budget opens a gap \(§[3](https://arxiv.org/html/2607.05872#S3)\)\. At 1B, ten\-thousand\-step basis averaging leaves both the realized rotation and the final perplexity essentially unchanged \(§[5](https://arxiv.org/html/2607.05872#S5)\)\.
- •What to do instead, with theory and a controlled test\.The optimizer state should be transported through each refresh: the first moment byRR, the second moment by the squared entries ofRR, with blind carry provably≈\(r−k⋆\)/2\{\\approx\}\(r\{\-\}k^\{\\star\}\)/2suboptimal \(Appendix[B](https://arxiv.org/html/2607.05872#A2)\)\. A three\-seed2×22\{\\times\}2at 1B/40K shows transport at the defaultβ2\\beta\_\{2\}beats theβ2\\beta\_\{2\}fix alone \(18\.7318\.73vs19\.2819\.28\), and the two remedies are partially independent \(§[5](https://arxiv.org/html/2607.05872#S5)\)\.
- •A prescription and a probe\.Lowerβ2\\beta\_\{2\}to0\.990\.99for any optimizer that refreshes a gradient subspace \(validated for canonical GaLore and LDAdam, with APOLLO\-Mini as a directional consistency check and a full\-rank control reversing the preference\), and measurek⋆k^\{\\star\}before trusting a rank: ifr\>k⋆r\>k^\{\\star\}, the excess directions are noise being refreshed \(§[5](https://arxiv.org/html/2607.05872#S5)\)\.

The object of study is the gradient subspace these methods compute, not a LoRA\-style weight parameterization \(§[2](https://arxiv.org/html/2607.05872#S2)\)\.

## 2Setup and notation

#### The GaLore\-family optimizer\.

We study optimizers that train a weight matrixW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}by running Adam inside a periodically\-recomputed low\-rank subspace of its gradient\. At stepttthe gradient isGt=∇Wℒ∈ℝm×nG\_\{t\}=\\nabla\_\{W\}\\mathcal\{L\}\\in\\mathbb\{R\}^\{m\\times n\}\. EveryTTsteps the optimizer computes the top\-rrleft singular subspace of the current gradient,Ut∈ℝm×rU\_\{t\}\\in\\mathbb\{R\}^\{m\\times r\}withUt⊤​Ut=IrU\_\{t\}^\{\\top\}U\_\{t\}=I\_\{r\}\(the columns are the leading left singular vectors ofGtG\_\{t\}\), and holds it fixed for the nextTTsteps\. Within an interval it projects each gradient into the subspace,G~=U⊤​G∈ℝr×n\\tilde\{G\}=U^\{\\top\}G\\in\\mathbb\{R\}^\{r\\times n\}, applies the Adam update in thatrr\-dimensional space, and maps the result back:W←W−η​U​Adam​\(G~\)W\\leftarrow W\-\\eta\\,U\\,\\mathrm\{Adam\}\(\\tilde\{G\}\)\. This is GaLore\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib4)\); the same template,*explicitly form a top\-rrgradient subspace, refresh it everyTTsteps*, underlies the family that has grown around it\(Robertet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib6); Rajabiet al\.,[2025b](https://arxiv.org/html/2607.05872#bib.bib7); Zmushkoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib62); Heet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib26); Zhuet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib19)\)\. Our claims are restricted to this family\. LoRA and other low\-rank*parameterizations*, which impose a low\-rank structure on the*update*and never compute a gradient subspace, are out of scope\.

#### Adam’s two clocks\.

Adam\(Kingma and Ba,[2015](https://arxiv.org/html/2607.05872#bib.bib25)\)carries two exponential moving averages of the projected gradient: the first momentmt=β1​mt−1\+\(1−β1\)​g~tm\_\{t\}=\\beta\_\{1\}m\_\{t\-1\}\+\(1\{\-\}\\beta\_\{1\}\)\\tilde\{g\}\_\{t\}and the second momentvt=β2​vt−1\+\(1−β2\)​g~t2v\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\{\-\}\\beta\_\{2\}\)\\tilde\{g\}\_\{t\}^\{\\,2\}\. Each is a low\-pass filter with a relaxation time set by its decay,

τm=11−β1≈10,τv=11−β2≈1000\(β1=0\.9,β2=0\.999\),\\tau\_\{m\}=\\frac\{1\}\{1\-\\beta\_\{1\}\}\\approx 10,\\qquad\\tau\_\{v\}=\\frac\{1\}\{1\-\\beta\_\{2\}\}\\approx 1000\\quad\(\\beta\_\{1\}\{=\}0\.9,\\ \\beta\_\{2\}\{=\}0\.999\),\(1\)so the second moment forgets the past100×100\\timesmore slowly than the first\. These two timescales are the constants against which the refresh intervalTTmust be compared, and the asymmetry between them is the source of the staleness we analyze in §[5](https://arxiv.org/html/2607.05872#S5)\.

#### Measuring subspace rotation\.

To quantify how much the basis changes at a refresh we compare the bases before and after,UoldU\_\{\\rm old\}andUnew∈ℝm×rU\_\{\\rm new\}\\in\\mathbb\{R\}^\{m\\times r\}, through theirrrprincipal anglesθ1,…,θr\\theta\_\{1\},\\dots,\\theta\_\{r\}\(the angles betweenrange​\(Uold\)\\mathrm\{range\}\(U\_\{\\rm old\}\)andrange​\(Unew\)\\mathrm\{range\}\(U\_\{\\rm new\}\)\)\. Singular vectors carry an arbitrary per\-column sign, so the raw Frobenius distance‖Unew−Uold‖F\\\|U\_\{\\rm new\}\-U\_\{\\rm old\}\\\|\_\{F\}is dominated by sign flips rather than rotation; we align signs first \(Appendix[A](https://arxiv.org/html/2607.05872#A1)\) and report the sign\-corrected distance

‖Δ​U‖F=\(2​r−2​∑i=1rcos⁡θi\)1/2,0≤‖Δ​U‖F≤2​r\.\\\|\\Delta U\\\|\_\{F\}\\;=\\;\\Bigl\(2r\-2\\textstyle\\sum\_\{i=1\}^\{r\}\\cos\\theta\_\{i\}\\Bigr\)^\{1/2\},\\qquad 0\\leq\\\|\\Delta U\\\|\_\{F\}\\leq\\sqrt\{2r\}\.\(2\)This is the chordal \(principal\-angle\) Frobenius distance between orthonormal frames\(Bendokatet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib16)\): it is0when the subspace is unchanged and attains its maximum2​r\\sqrt\{2r\}when the two bases are orthogonal \(θi=π/2\\theta\_\{i\}=\\pi/2for allii\), i\.e\. when the new subspace shares no direction with the old\. We also track the fraction of gradient energy the subspace captures,

γg=‖U​U⊤​G‖F2‖G‖F2∈\[0,1\],\\gamma\_\{g\}=\\frac\{\\\|UU^\{\\top\}G\\\|\_\{F\}^\{2\}\}\{\\\|G\\\|\_\{F\}^\{2\}\}\\;\\in\\;\[0,1\],\(3\)which is what the projection actually preserves and the quantity we return to in §[4](https://arxiv.org/html/2607.05872#S4)\.

## 3The subspace is not identifiable

The subspace GaLore projects onto is recomputed everyTTsteps, and it moves\. This section establishes what that motion is\. The measured per\-refresh basis change saturates the geometric maximum at every rank and scale \(§[3\.1](https://arxiv.org/html/2607.05872#S3.SS1)\)\. A same\-step control then locates the cause: two top\-rrestimates taken at the identical step from disjoint minibatches already disagree by as much as two estimates takenTTsteps apart \(§[3\.2](https://arxiv.org/html/2607.05872#S3.SS2)\)\. The refresh is therefore not the motion of a stable object\. The top\-rrsubspace of a minibatch gradient is statistically non\-identifiable beyond a small reproducible core, at the ranks these methods use\. The rest of the section shows why: the gradient spectrum has no gap atrr\(§[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\); the identifiable part is small but measurable and consistent \(§[3\.4](https://arxiv.org/html/2607.05872#S3.SS4)\); and averaging does not manufacture identifiability \(§[3\.5](https://arxiv.org/html/2607.05872#S3.SS5)\)\.

### 3\.1A scale\-invariant rotation law

We measure‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}at every refresh during training and find that the subspace rotates almost as far as it geometrically can\. Figure[1](https://arxiv.org/html/2607.05872#S3.F1)reports the measured rotation across an8×8\\timesrange of ranks at Pythia\-1B \(exact values in Appendix[A](https://arxiv.org/html/2607.05872#A1)\): in every case‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}reaches96\.796\.7–99\.6%99\.6\\%of its maximum2​r\\sqrt\{2r\}, the value attained when the new basis is orthogonal to the old\. The dependence on rank is‖Δ​U‖F≈2​r\\\|\\Delta U\\\|\_\{F\}\\approx\\sqrt\{2r\}: for instance‖Δ​U‖F​\(r=256\)/‖Δ​U‖F​\(r=32\)=22\.10/7\.93=2\.79\\\|\\Delta U\\\|\_\{F\}\(r\{=\}256\)/\\\|\\Delta U\\\|\_\{F\}\(r\{=\}32\)=22\.10/7\.93=2\.79, close to8=2\.83\\sqrt\{8\}=2\.83\. The same holds from Pythia\-160M to 1B\. At this saturation the mean cosine,1−‖Δ​U‖F2/2​r1\-\\\|\\Delta U\\\|\_\{F\}^\{2\}/2r, is below0\.070\.07: at each refresh the optimizer discards the frame it was using and adopts a nearly orthogonal one \(mean angle8686–90∘90^\{\\circ\}\)\. \(On a pretrained 160M checkpoint the same redraw sits lower, at0\.74​2​r0\.74\\,\\sqrt\{2r\}, a mean angle of≈63∘\{\\approx\}63^\{\\circ\}: a pretrained gradient’s reproducible spike is stronger, and itsk⋆≈39k^\{\\star\}\\\!\\approx\\\!39shared directions pull consecutive estimates together; §[3\.2](https://arxiv.org/html/2607.05872#S3.SS2)\.\) This is not slow drift\. It is turnover\.

![Refer to caption](https://arxiv.org/html/2607.05872v1/x1.png)Figure 1:Measured subspace rotation per refresh against rank \(Pythia\-1B,T=50T\{=\}50; log–log\)\. The points sit on the2​r\\sqrt\{2r\}ceiling, the maximum possible Frobenius distance between two rank\-rrframes, at every rank \(annotated ratios0\.9670\.967–0\.9960\.996\)\. Each refresh returns a nearly independent subspace\. Exact values and the sign correction: Appendix[A](https://arxiv.org/html/2607.05872#A1)\.
### 3\.2The rotation is estimator noise

A near\-maximal‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}across a refresh has an obvious reading: the subspace genuinely rotates by a large angle inTTsteps\. We test that reading with a control that removes time\. At a single step we split the batch into two disjoint halves, compute the top\-rrsubspace of each half’s gradient, and measure the distance between the two \(protocol in Appendix[F](https://arxiv.org/html/2607.05872#A6)\)\. Nothing has changed between the two estimates except the sample\. If the top\-rrsubspace were a well\-defined property of the gradient, the two same\-step estimates would agree\.

They do not\. Averaged over four layers at Pythia\-160M withr=128r\{=\}128, the same\-step split\-batch disagreement is0\.725​2​r0\.725\\,\\sqrt\{2r\}\. The across\-time rotation over a full refresh interval of training, at the same160160M scale, is0\.742​2​r0\.742\\,\\sqrt\{2r\}\(Figure[3](https://arxiv.org/html/2607.05872#S3.F3)a\), essentially the same\-step value: the across\-time rotation exceeds the same\-step floor by at most a few percent of2​r\\sqrt\{2r\}\. That floor is not an artifact of the pretrained checkpoint at which the control is run: repeated at five points along Pythia\-160M training \(steps11K to143143K\), the four\-layer same\-step disagreement stays in0\.640\.64–0\.71​2​r0\.71\\,\\sqrt\{2r\}\(Appendix[F](https://arxiv.org/html/2607.05872#A6)\), lowest early and rising toward the pretrained value, so at every training state the bulk of the across\-time rotation is already present with no time elapsed\. The distance a subspace travels inTTsteps is essentially the distance two estimates of it already sit apart at a single step, so the “rotation” carries little information about elapsed time: it is dominated by estimator noise, not by motion of a stable object\. A direct count confirms the same fact at the level of individual directions: averaged over the four layers, onlyk⋆≈39k^\{\\star\}\\\!\\approx\\\!39of ther=128r\{=\}128directions the top\-rrcut returns are reproducible across the split \(4848on the query\-key\-value matrix, with standard error below11over repetitions\), and the rest reshuffle with the sample\.

This also settles an apparent conflict with reports that the GaLore subspace is stable\. Those reports measure quantities that stay small while‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}is near\-maximal\. The rank\-rrsubspace hasrrprincipal angles, and many can remain small while the tail angles near90∘90^\{\\circ\}dominate the chordal distance, so an angle distribution that looks mostly stable, without conversion to‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}, still corresponds to a near\-maximal rotation\(Zmushkoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib62)\)\. An incremental tracker is built to move its estimate slowly, so its self\-consistency across steps is a property of the tracker, not of the subspace a fresh SVD would return\(Rajabiet al\.,[2025b](https://arxiv.org/html/2607.05872#bib.bib7)\)\. And a raw singular\-vector overlap conflates the arbitrary per\-column sign of the SVD with genuine rotation; once signs are aligned \(§[2](https://arxiv.org/html/2607.05872#S2)\), the rotation is the near\-maximal value of Figure[1](https://arxiv.org/html/2607.05872#S3.F1)\. None of these is the sign\-corrected frame distance‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}, which is the quantity that governs the carried optimizer state\.

### 3\.3Why: there is no spectral gap at rankrr

The recomputed top\-rrsubspace can be reproducible across samples only if the rank\-rrcut separates well\-determined directions from the rest, that is, if the spectrum has a gap at indexrr\. There is not\. Figure[2](https://arxiv.org/html/2607.05872#S3.F2)shows the per\-matrix gradient spectrum at Pythia\-160M and 1B: a small dominant spike, then a smooth, near\-isotropic tail that the rank\-128128cutoff slices straight through\. Across layers, steps, and both scales:

- •the dominant spike is small: a few to∼20\{\\sim\}20directions hold the first half of the gradient’s squared singular\-value \(Frobenius\) mass, consistent with the tiny gradient subspace documented for classification byGur\-Ariet al\.\([2018](https://arxiv.org/html/2607.05872#bib.bib21)\);
- •there is no gap at the cutoff: the adjacent singular\-value ratio atr=128r\{=\}128is≈1\.005\\approx 1\.005on average, and no jump anywhere in the bulk exceeds∼3%\{\\sim\}3\\%\(max\\maxratio1\.0111\.011–1\.0301\.030\);
- •the effective rank \(entropy of the normalized spectrum\) straddles the cutoff, ranging8989–358358across layers and both scales, sor=128r\{=\}128sits inside the flat tail, not at a boundary;
- •the top\-rrblock captures only4646–68%68\\%of the gradient’s singular\-value \(nuclear\) mass, leaving roughly a third to over half outside the subspace at all times\. \(The spike dominates the squared mass while the long flat tail dominates the nuclear mass; the contrast between the two norms is itself a signature of the tail’s flatness\.\)

This agrees with independent measurements:Gur\-Ariet al\.\([2018](https://arxiv.org/html/2607.05872#bib.bib21)\)establish the small dominant spike directly, and a concurrent large\-model analysis reports the same spike\-tail profile with the spike at∼1\.5%\{\\sim\}1\.5\\%of directions\(Huanget al\.,[2026](https://arxiv.org/html/2607.05872#bib.bib69)\), a separation at the spike rather than atr=128r\{=\}128\. The complementary evidence from the randomization side, that past the spike the SVD frame carries little more than a random one, is taken up in §[4](https://arxiv.org/html/2607.05872#S4)and §[6](https://arxiv.org/html/2607.05872#S6)\.

![Refer to caption](https://arxiv.org/html/2607.05872v1/x2.png)Figure 2:Per\-matrix gradient singular spectrum at Pythia\-160M \(left\) and 1B \(right\): the reason the subspace is not identifiable\. A small dominant spike \(a few to∼20\{\\sim\}20directions holding half the squared mass\) gives way to a smooth, near\-isotropic tail with no gap\. The GaLore cutoffr=128r\{=\}128\(dashed\) falls inside the flat region, where the adjacent singular\-value ratio is≈1\.005\\approx 1\.005and the top\-rrblock holds only4646–68%68\\%of the gradient’s nuclear mass\. The rank\-rrsubspace is not separated from its complement\.#### From no gap to Haar\-random redraw\.

When singular directions are near\-degenerate they are maximally sensitive to perturbation: the Davis–Kahan and Wedinsin⁡θ\\sin\\thetatheorems bound a subspace’s rotation by the perturbation size over the gap that isolates it\(Davis and Kahan,[1970](https://arxiv.org/html/2607.05872#bib.bib63); Wedin,[1972](https://arxiv.org/html/2607.05872#bib.bib64)\)\. As the gap atrrvanishes, that bound diverges, and the SVD is free to return any frame within the degenerate block\. Sub\-threshold directions therefore behave as a Haar\-random draw within the tail, whether the perturbation is a step of training or a change of minibatch\. This is why two same\-step estimates disagree as much as two across\-time estimates \(§[3\.2](https://arxiv.org/html/2607.05872#S3.SS2)\)\. The same model predicts the redraw distance approaches the2​r\\sqrt\{2r\}ceiling asr/d→0r/d\\to 0\(Appendix[A](https://arxiv.org/html/2607.05872#A1.SS0.SSS0.Px3)\)\. At 160M, wherer/d=1/6r/d=1/6, it stays below the ceiling, matching the measured0\.74​2​r0\.74\\,\\sqrt\{2r\}\. The Weingarten lower bound of §[5](https://arxiv.org/html/2607.05872#S5)follows for the carried second moment under the same Haar model \(Theorem[1](https://arxiv.org/html/2607.05872#Thmtheorem1), Appendix[B](https://arxiv.org/html/2607.05872#A2)\)\. The stable part of the geometry is real but small: the spike ofGur\-Ariet al\.\([2018](https://arxiv.org/html/2607.05872#bib.bib21)\)is genuine, and GaLore’sr=128r\{=\}128overshoots it by an order of magnitude, reaching past the determined directions into the gapless bulk that redraws\.

### 3\.4The identifiable part is small, measurable, and consistent

The same\-step count gives a direct measure of how many directions survive resampling: a four\-layer mean ofk⋆≈39k^\{\\star\}\\\!\\approx\\\!39of128128, ranging from4848on the query\-key\-value matrix down to3131on the attention output, with a standard error below11over repetitions, so the count is stable rather than noise\. Reading the same data a second way reproduces the pattern: inverting the measured spectral shape through a detectability threshold fixed a priori, with one calibrated noise scale per layer and no shape parameters, recovers the same layer ordering,qkv\>mlp\>embed\>dense\\text\{qkv\}\>\\text\{mlp\}\>\\text\{embed\}\>\\text\{dense\}\(inversion42,37,19,1742,37,19,17against the direct48,43,34,3148,43,34,31\)\. Both readings derive from the same measurement, so this is a consistency check between two estimators rather than an independent prediction; both place the reproducible core well belowr=128r\{=\}128\. The ordering matches the measured energy the top\-rrblock retains in each layer \(7373–81%81\\%for qkv and mlp versus4242–48%48\\%for embed and dense\): layers whose gradient concentrates more have more reproducible directions\.k⋆k^\{\\star\}is thus a measurable property, not an artifact of one estimator\.

### 3\.5Averaging does not rescue it

Identifiability might be recovered by averaging: with a large enough gradient sample the noisy tail would shrink and a gap could open atrr\. It does not shrink fast enough\. UnderNN\-fold gradient averaging, pure sampling noise falls asN−1/2N^\{\-1/2\}, so a noise singular value scaled byN\\sqrt\{N\}would stay flat\. The deep spectral tail instead shrinks only as≈N−1/4\\approx N^\{\-1/4\}: across all four probed layers the scaled tail value grows by2\.02\.0–3\.2×3\.2\\timesfromN=1N\{=\}1toN=64N\{=\}64\(Figure[3](https://arxiv.org/html/2607.05872#S3.F3)\)\. The tail is not noise being averaged away\. It is signal, with a de\-censored power\-law spectrum of exponentα=1\.03\\alpha=1\.03–1\.351\.35, obtained two independent ways \(a direct tail fit and the overlap\-versus\-NNcurve\)\. A shuffled\-document control, which breaks within\-document correlation, retains most of the anomaly \(1\.71\.7–2\.2×2\.2\\timesagainst the2\.02\.0–3\.2×3\.2\\timesof the true pool\), so the persistent tail is mostly genuine gradient signal, with only a minor part attributable to within\-document correlation\.

We state the limits of this claim plainly\. We do not claim a closed\-form law for how the subspace overlap grows withNN: the naive spiked\-covariance prediction fails out of sample \(mean absolute error0\.0730\.073\), so we report only the measured scaling, not a formula\. That scaling is enough for the conclusion\. Because the tail is signal that decays as a power law and shrinks under averaging only asN−1/4N^\{\-1/4\}, no finite averaging budget opens a gap atrr, and no rankrrlands on a gap that is not there\. The top\-rrsubspace GaLore uses is not well\-defined, and no averaging budget we test recovers it; this holds from 160M to 2\.8B and across GPU backends \(Appendix[F](https://arxiv.org/html/2607.05872#A6)\)\.

![Refer to caption](https://arxiv.org/html/2607.05872v1/x3.png)Figure 3:Averaging cannot make the top\-rrsubspace identifiable \(Pythia\-160M,r=128r\{=\}128, four layers\)\.\(a\)Same\-step split\-batch subspace disagreement‖Δ​U‖F/2​r\\\|\\Delta U\\\|\_\{F\}/\\sqrt\{2r\}against averaging foldNN\. AtN=1N\{=\}1it is0\.7250\.725, indistinguishable from the across\-time rotation0\.7420\.742\(dashed\), and it falls only slowly withNN\.\(b\)The deep tail singular value scaled byN\\sqrt\{N\}\. Pure noise would be flat; the measured tail grows2\.02\.0–3\.2×3\.2\\timesfromN=1N\{=\}1to6464, so it shrinks as≈N−1/4\\approx N^\{\-1/4\}, notN−1/2N^\{\-1/2\}\. The tail is signal with power\-law exponentα=1\.03\\alpha=1\.03–1\.351\.35, so no averaging budget opens a gap atrr\.
### 3\.6The non\-identifiability is general

The same\-step control is measured at Pythia\-160M, but the effect is not specific to Pythia, to language, or to GaLore\. We ran the same split\-batch probe on eleven pretrained checkpoints, all trained with full\-rank Adam: seven Pythia scales \(70M–6\.9B\), GPT\-2\-large, Qwen2\.5\-3B, Llama\-3\-8B, and a vision transformer \(ViT\-B\) probed on CIFAR\-10\. In every language model the same\-step disagreement is0\.670\.67–0\.830\.83of the maximum2​r\\sqrt\{2r\}, with no spectral gap atrr\(mean adjacent ratio1\.0041\.004–1\.0091\.009\); it rises with scale \(Figure[4](https://arxiv.org/html/2607.05872#S3.F4)\) as the reproducible core shrinks, fromk⋆≈50k^\{\\star\}\\\!\\approx\\\!50at 70M tok⋆≈21k^\{\\star\}\\\!\\approx\\\!21at 6\.9B\. The effect crosses architecture classes \(GPT\-NeoX, GPT\-2, and the Llama\-class Qwen2\.5 and Llama\-3\) and holds, more weakly, in the vision transformer \(0\.530\.53of maximal on CIFAR\)\. Because every model here is trained with full\-rank Adam, the non\-identifiability is a property of the gradient itself, not of the low\-rank projection: the top\-rrsubspace any method would extract is estimator noise past a small core, at every scale we measured\. Per\-model numbers and the protocol are in Appendix[G](https://arxiv.org/html/2607.05872#A7)\.

![Refer to caption](https://arxiv.org/html/2607.05872v1/x4.png)Figure 4:Same\-step subspace disagreement‖Δ​U‖F/2​r\\\|\\Delta U\\\|\_\{F\}/\\sqrt\{2r\}atN=1N\{=\}1\(split\-batch, pretrained checkpoints,r=128r\{=\}128, mean over four layer types\) across architectures, scales, and domains\. It rises with scale in Pythia \(70M–6\.9B\), sits in the same band for GPT\-2\-large, Qwen2\.5\-3B, and Llama\-3\-8B, and holds more weakly for a vision transformer on CIFAR\. Every model is full\-rank\-Adam\-pretrained, so the non\-identifiability is a property of the gradient, not of GaLore; no spectral gap atrrappears in any of them\.#### Forward pointer\.

This scaling is measured at Pythia\-160M, where the tail is already flat atr=128r\{=\}128\. At 1B, wherer=128r\{=\}128lies deeper in the tail of a wider gradient, it predicts that averaging successive bases across refreshes cannot stabilize the subspace either\. We verify this directly, after the energy interlude of §[4](https://arxiv.org/html/2607.05872#S4), with the second\-moment remedies at 1B in §[5](https://arxiv.org/html/2607.05872#S5): basis averaging leaves the per\-refresh rotation near its maximum, while the two remedies that do help act on the carried optimizer state rather than on the subspace\.

## 4Why GaLore still works

If the subspace it projects onto is replaced by a nearly orthogonal one everyTTsteps, why does GaLore train as well as it does? The answer is that the projection’s job is not to track a fixed subspace but to capture gradient energy, and how much it captures barely depends on*which*subspace it picks\. The descent contribution of a projection is governed by the fraction of gradient energy it retains,γg\\gamma\_\{g\}\(Eq\.[3](https://arxiv.org/html/2607.05872#S2.E3)\)\. For a subspaceUUspanningrrdirections of the gradient second\-moment, the expected coverage is the cumulative spectral mass it covers,

𝔼​\[γg\]=∑i=1rλi∑i=1nλi,\\mathbb\{E\}\[\\gamma\_\{g\}\]\\;=\\;\\frac\{\\sum\_\{i=1\}^\{r\}\\lambda\_\{i\}\}\{\\sum\_\{i=1\}^\{n\}\\lambda\_\{i\}\},\(4\)whereλ1≥⋯≥λn\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{n\}are the spectrum’s energies \(Appendix[D](https://arxiv.org/html/2607.05872#A4)\)\. This identity is an expectation over the gradient covariance; the per\-matrix SVD energy fractions reported below are the instantaneous quantity it matches in expectation\. In the presence of a sharp gap, only the gap\-defined subspace captures the dominant mass and a rotation away from it would be costly\. But §[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)showed the tail past the small spike is smooth and globally near\-isotropic, with no gap anywhere in the bulk rather than merely atrr; it is this global flatness, not the absence of a single gap atrr, that the argument needs\. In that regime Eq\.[4](https://arxiv.org/html/2607.05872#S4.E4)is nearly flat in the choice of subspace: anyrrdirections that include the spike and otherwise sample the tail capture a comparable fraction\. The freshly\-recomputed basis, near\-orthogonal to its predecessor, is therefore an essentially equivalent projector: measured during 1B training withr=128r\{=\}128, GaLore retains a roughly constantγg≈0\.67\\gamma\_\{g\}\\approx 0\.67–0\.730\.73of the squared gradient throughout, regardless of the redraw\. \(The nuclear fraction is lower,4646–68%68\\%, because the flat tail carries more of the nuclear mass; §[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\.\)Heet al\.\([2024](https://arxiv.org/html/2607.05872#bib.bib26)\)reach a compatible conclusion from the convergence side: once the true gradient shrinks relative to its noise, the SVD\-selected subspace loses its advantage over a random one, and a projection chosen “without any preference” suffices in its place\. Our diagnostic also reconciles a tension in more recent GaLore\-family work\.Heet al\.\([2024](https://arxiv.org/html/2607.05872#bib.bib26)\)and a randomized Grassmann variant\(Rajabiet al\.,[2025a](https://arxiv.org/html/2607.05872#bib.bib15)\)replace the exact SVD projection with a random one and report training at least as good as GaLore, the latter arguing that flat curvature makes the precise subspace unnecessary; GaLore\-2\(Suet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib5)\)instead finds that a purely random projection degrades quality and retains a periodic SVD in a hybrid schedule\. The reproducible rankk⋆k^\{\\star\}reconciles them: the exact SVD earns its cost only to the extent the subspace is identifiable, and is redundant oncek⋆≪rk^\{\\star\}\\ll r\. Because the subspace grows*less*identifiable as training proceeds \(the same\-step floor rises across the trajectory; Appendix[F](https://arxiv.org/html/2607.05872#A6)\), a random projection is safest late in training, while any residual advantage of the exact SVD is concentrated early\. This is a quantitative form of the flat\-curvature intuition, and of the boundary the hybrid schedule approximates\. Low\-rank projection is known to introduce a per\-step bias relative to full Adam\(Heet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib26); Panet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib13)\); our account is orthogonal, since we do not claim the projection is unbiased, only that the*choice*of whichrrdirections it selects barely changes the energy captured\.

This reframes what GaLore is\. It is not tracking a persistent low\-rank structure in the gradient; there is none to track at rankrr\. It is a rotating energy\-capture projector: a frame that turns over completely at every refresh while preserving the energy fraction that drives descent\. The descent direction survives the redraw\. What does not survive is the optimizer state carried across the rotation, and, as we show next, the second moment in particular pays for it\.

## 5What works and what does not

If the rank\-rrframe that GaLore projects onto is redrawn nearly orthogonally at every refresh \(§[3](https://arxiv.org/html/2607.05872#S3)\), the optimizer state carried across that refresh points at a target that no longer exists\. The first moment survives the move: it is a vector in the subspace and transports exactly by the rotationR=Unew⊤​UoldR=U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}\(Proposition[3](https://arxiv.org/html/2607.05872#Thmtheorem3)\), the change of basis that LDAdam\(Robertet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib6)\)and SUMO\(Refaelet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib68)\)apply\. The second moment does not\. It is a per\-coordinate variance whose optimal new\-basis value passes through the*squared*entries ofRR, and a carry blind toRRsits a factor≈\(r−k⋆\)/2\\approx\(r\{\-\}k^\{\\star\}\)/2above the best any blind estimator can reach \(Theorem[1](https://arxiv.org/html/2607.05872#Thmtheorem1)\)\. With memoryτv=1/\(1−β2\)≈1000\\tau\_\{v\}=1/\(1\-\\beta\_\{2\}\)\\approx 1000steps at the default and refreshes everyT≈50T\\approx 50–200200steps, the second moment averages statistics from many dead subspaces and is chronically misaligned; its lag grows withτv\\tau\_\{v\}on the same curve as full\-rank Adam \(Appendix[H](https://arxiv.org/html/2607.05872#A8)\)\. Three remedies present themselves: average the basis so it stops moving, transport the state through the rotation, or shorten the second moment’s memory\. We test all three at Pythia\-1B\. Averaging fails; transport and a shorter memory both work, and we measure why\.

### 5\.1Averaging the basis does not work

The direct fix for a subspace that moves too much is to stop it from moving\. We build the EMA\-basis variant: instead of the top\-rrsingular subspace of the current gradient, the basis is the top\-rreigenvectors of an exponential moving average ofG​G⊤GG^\{\\top\}with decayβ3\\beta\_\{3\}, so thatβ3=0\\beta\_\{3\}\{=\}0recovers GaLore exactly and largerβ3\\beta\_\{3\}averages the subspace over a growing window\. At Pythia\-1B trained from scratch on WikiText\(Merityet al\.,[2016](https://arxiv.org/html/2607.05872#bib.bib28)\)\(r=128r\{=\}128,T=160T\{=\}160, learning rate4×10−34\\times 10^\{\-3\}, warmup40004000,4040K steps, AMD MI210, one seed\), the final perplexity does not move withβ3\\beta\_\{3\}\. At the defaultβ2=0\.999\\beta\_\{2\}\{=\}0\.999the row is flat:\{21\.088,21\.025,21\.074,21\.026,21\.385\}\\\{21\.088,\\,21\.025,\\,21\.074,\\,21\.026,\\,21\.385\\\}forβ3=\{0,0\.9,0\.99,0\.999,0\.9999\}\\beta\_\{3\}=\\\{0,\\,0\.9,\\,0\.99,\\,0\.999,\\,0\.9999\\\}, spanning0\.360\.36across four decades of averaging window \(0\.060\.06excluding the longest,10410^\{4\}\-step window, which alone sits0\.360\.36above the minimum\)\. The flat row replicates on a second cluster \(NVIDIA L40S, same recipe\):20\.3920\.39atβ3=0\\beta\_\{3\}\{=\}0against20\.5220\.52atβ3=0\.99\\beta\_\{3\}\{=\}0\.99\(we compare only within a cluster, never across\)\.

The measurement of why is direct\. Averaging the covariance does slow the basis, but not enough to matter: late in training \(steps≥20\\geq 20K\) the mean realized rotation is‖Δ​U‖F=15\.48,14\.51,13\.42\\\|\\Delta U\\\|\_\{F\}=15\.48,\\,14\.51,\\,13\.42forβ3=0,0\.999,0\.9999\\beta\_\{3\}=0,\\,0\.999,\\,0\.9999, against the ceiling2​r=16\.0\\sqrt\{2r\}=16\.0\. Even a10,00010\{,\}000\-step average ofG​G⊤GG^\{\\top\}leaves the refresh\-to\-refresh basis change at84%84\\%of maximal \(against97%97\\%with no averaging\)\. Basis averaging cannot stabilize what is not identifiable: past the dominant spike the spectrum is gapless \(§[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\), so no amount of temporal averaging carves a stable rank\-128128frame out of the isotropic tail\.

### 5\.2Transporting the state works

If the basis will not hold still, move the state with it\. Table[1](https://arxiv.org/html/2607.05872#S5.T1)crosses the two arms that matter,\{\\\{carry, transport\}\\\}against\{β2=0\.999,β2=0\.99\}\\\{\\beta\_\{2\}\{=\}0\.999,\\ \\beta\_\{2\}\{=\}0\.99\\\}, three seeds per cell at Pythia\-1B/4040K\. This table uses warmup14401440; its carry\-β2=0\.999\\beta\_\{2\}\{=\}0\.999cell \(22\.0722\.07\) therefore sits about0\.80\.8–1\.01\.0PPL above the same nominal configuration in the averaging and gasket experiments \(warmup40004000:21\.0921\.09at one seed, §[5\.1](https://arxiv.org/html/2607.05872#S5.SS1), and21\.2721\.27over three seeds, §[5\.3](https://arxiv.org/html/2607.05872#S5.SS3)\), purely through the warmup difference\. Magnitudes shift between recipes; directions do not\. We compareβ2\\beta\_\{2\}and state policies only within a fixed recipe, never perplexities across recipes\. Read the table three ways, in order\. First, transporting the state at the defaultβ2=0\.999\\beta\_\{2\}\{=\}0\.999reaches18\.7318\.73, better than loweringβ2\\beta\_\{2\}without transport \(19\.2819\.28\): moving the state is worth more than the hyperparameter fix on its own\. Second, the second\-moment cost of the wrongβ2\\beta\_\{2\}under a plain carry is large,22\.0722\.07against19\.2819\.28: a2\.792\.79PPL gap, about twenty times the standard error of the difference \(two\-samplet≈19t\\approx 19on44degrees of freedom\)\. Third, that gap does not vanish once the state is transported: it shrinks to1\.811\.81but persists; the two remedies are therefore partially independent rather than substitutes\. \(The transport row is the LDAdam update, so this1\.811\.81is the same measurement that recurs below as LDAdam’s matched\-4040Kβ2\\beta\_\{2\}gap, §[5\.3](https://arxiv.org/html/2607.05872#S5.SS3)\.\) One caveat is mandatory: the transport arm is the full LDAdam update, which applies the change\-of\-basis transport of both moments together with error feedback, and this experiment does not separate the contribution of transport from that of error feedback\. The transport rule itself is LDAdam’s; our contribution here is the optimality statement behind it \(Proposition[3](https://arxiv.org/html/2607.05872#Thmtheorem3), Theorem[1](https://arxiv.org/html/2607.05872#Thmtheorem1)\) and the controlled2×22\\times 2comparison, not the rule\.

Table 1:Final validation perplexity at Pythia\-1B under the two independent remedies: transporting the optimizer state across each refresh and shortening the second moment’s memory\. Rows are the state policy at a refresh; columns areβ2\\beta\_\{2\}\. Each cell is mean±\\pmstandard deviation over three seeds\. Recipe \(identical across cells\): Pythia\-1B trained from scratch on WikiText, rankr=128r\{=\}128, refresh intervalT=160T\{=\}160, learning rate4×10−34\\times 10^\{\-3\}, warmup14401440,4040K steps\. “Carry” is canonical GaLore \(the second moment is carried unchanged across refreshes\); “transport” is the LDAdam update\(Robertet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib6)\)\(change\-of\-basis transport of both moments plus error feedback\)\. Transporting at the defaultβ2=0\.999\\beta\_\{2\}\{=\}0\.999\(18\.7318\.73\) already beats loweringβ2\\beta\_\{2\}alone \(19\.2819\.28\); the carryβ2\\beta\_\{2\}gap is2\.792\.79, about twenty times the standard error of the difference, and persists under transport \(1\.811\.81\)\. Transport here is the full LDAdam update; the table does not separate state transport from error feedback\.
### 5\.3A shorter memory works, with a recipe caveat

The second remedy, loweringβ2\\beta\_\{2\}from0\.9990\.999to0\.990\.99, is the cheaper of the two, and it helps the refreshing optimizers we test at their matched recipes\. For canonical GaLore the effect is small \(about22perplexity\) and hardware\-sensitive: it is three\-seed robust on our primary cluster, while a matched\-recipe run on a different GPU backend does not reproduce it, consistent with the projection’s periodic SVD being ill\-conditioned and resolving differently across backends\. We therefore present the shorter memory as the secondary, less robust remedy, behind state transport\. The forgetting\-factor reason is standard\(Guoet al\.,[1993](https://arxiv.org/html/2607.05872#bib.bib65)\): when the target is reset everyT≪τvT\\ll\\tau\_\{v\}steps, a long memory keeps integrating pre\-rotation statistics that are now bias, and the error\-minimizing forgetting factor drops below the no\-reset optimum\. A bias–variance argument specialized to a self\-reset target makes the direction precise \(Appendix[I](https://arxiv.org/html/2607.05872#A9)\): the minimizer is interior and moves down as the refresh interval shrinks\. Empirically theβ2\\beta\_\{2\}gain does not depend on the averaging window of the previous experiment: across theβ3\\beta\_\{3\}grid the gap betweenβ2=0\.999\\beta\_\{2\}\{=\}0\.999andβ2=0\.99\\beta\_\{2\}\{=\}0\.99is1\.94,2\.04,2\.16,2\.27,2\.131\.94,\\,2\.04,\\,2\.16,\\,2\.27,\\,2\.13PPL atβ3=0,0\.9,0\.99,0\.999,0\.9999\\beta\_\{3\}=0,\\,0\.9,\\,0\.99,\\,0\.999,\\,0\.9999\(seed 0\), so averaging neither substitutes for the memory fix nor interferes with it\. At the gasket recipe repeated over seeds \(same recipe, warmup40004000\), canonical GaLore gives21\.27±0\.1721\.27\\pm 0\.17atβ2=0\.999\\beta\_\{2\}\{=\}0\.999against19\.04±0\.1319\.04\\pm 0\.13atβ2=0\.99\\beta\_\{2\}\{=\}0\.99\(three seeds each\), a2\.232\.23PPL gap\.

#### Across the family\.

The direction holds for every optimizer we tried \(Figure[5](https://arxiv.org/html/2607.05872#S5.F5); exact values and configurations in Appendix[E](https://arxiv.org/html/2607.05872#A5), Table[3](https://arxiv.org/html/2607.05872#A5.T3)\)\. For LDAdam, whose error feedback makes the second moment’s calibration load\-bearing, loweringβ2\\beta\_\{2\}from0\.9990\.999to0\.990\.99improves perplexity from57\.69±0\.1157\.69\\pm 0\.11to31\.27±0\.3131\.27\\pm 0\.31over three seeds \(1010K steps, learning rate10−310^\{\-3\}\): a26\.426\.4PPL gap against per\-cell seed spreads of0\.10\.1–0\.30\.3, two orders of magnitude larger than the noise\. The length caveat below is specific to canonical GaLore’s small margin; the LDAdam gap is present at both horizons,26\.426\.4at its own1010K recipe and1\.811\.81at the common4040K recipe \(Table[1](https://arxiv.org/html/2607.05872#S5.T1)\), so it is not a short\-run artifact\. We do not rank optimizers by gap size: at the common recipe LDAdam’s gap \(1\.811\.81\) is smaller than canonical GaLore’s \(2\.792\.79\), and the26\.426\.4belongs to its own recipe only\. APOLLO\-Mini moves from97\.83/101\.7797\.83/101\.77atβ2=0\.999\\beta\_\{2\}\{=\}0\.999to96\.76/97\.2096\.76/97\.20atβ2=0\.99\\beta\_\{2\}\{=\}0\.99\(two seeds\); it resamples a random Gaussian projection everyTTrather than an SVD top\-rrsubspace, so two consecutive frames are near\-orthogonal by construction and the refresh geometry differs\. We read APOLLO as a directional consistency check only, not a third strong leg: two seeds, small margins, high absolute perplexity\. The direction is nonetheless the same\.

#### A length caveat, stated plainly\.

For canonical GaLore the margin is training\-length dependent\. It is present at4040K \(above\) but absent, even slightly reversed, in shorter1010–2020K runs: the staleness cost accrues over rotations, and the second\-moment bias\-correction transient occupies a larger fraction of a short run\. The effect is real but integrates with training; short\-horizon tuning understates it\.

#### The causal control\.

The check that ties theβ2\\beta\_\{2\}effect to the refresh, and not to these training runs in general, is full\-rank AdamW: the same model, data, and schedule with no subspace refresh\. There the preference reverses to the standardβ2=0\.999\\beta\_\{2\}\{=\}0\.999at both learning rates tested \(97\.997\.9against118\.0118\.0at learning rate6×10−46\\times 10^\{\-4\}, and158\.3158\.3against200\.9200\.9at10−310^\{\-3\};14\.414\.4K steps, one seed\), exactly as the forgetting\-factor argument predicts when the target is never reset\. The control is directional \(one seed, two learning rates, shorter horizon\) but the margins are far outside seed noise, and its absolute perplexities are not comparable to the4040K GaLore runs above\. The lower\-β2\\beta\_\{2\}advantage is therefore specific to subspace refreshing\. A second moment that a refresh periodically invalidates is what makes0\.9990\.999the wrong default\.

![Refer to caption](https://arxiv.org/html/2607.05872v1/x5.png)Figure 5:Loweringβ2\\beta\_\{2\}from the0\.9990\.999default to0\.990\.99at Pythia\-1B \(final validation perplexity, log scale\)\. The axis is logarithmic and each optimizer sits at its own training recipe \(steps, learning rate, and length differ across the three, and the full\-rank control\); the comparison is within\-optimizer, where eachβ2\\beta\_\{2\}pair shares its recipe, not across optimizers\. For each refreshing optimizer shown, the lower value is better: LDAdam by57\.69→31\.2757\.69\\to 31\.27at its own1010K recipe \(three seeds; at the common4040K recipe its gap is1\.811\.81, Table[1](https://arxiv.org/html/2607.05872#S5.T1)\), canonical GaLore by21\.27→19\.0421\.27\\to 19\.04\(three seeds\), APOLLO\-Mini directionally \(two seeds\)\. Gap sizes are not comparable across optimizers because recipes differ\. The full\-rank AdamW control, which runs with no subspace refresh, reverses the preference back to0\.9990\.999, directionally isolating the periodic redraw as the cause\. The canonical\-GaLore advantage is flat across the EMA\-basis averaging window \(β2\\beta\_\{2\}gap1\.941\.94–2\.272\.27PPL overβ3∈\{0,…,0\.9999\}\\beta\_\{3\}\\in\\\{0,\\dots,0\.9999\\\}, §[5\.1](https://arxiv.org/html/2607.05872#S5.SS1)\), so a shorter memory and basis averaging act independently\. Exact values, seed counts, and recipe: Appendix[E](https://arxiv.org/html/2607.05872#A5), Table[3](https://arxiv.org/html/2607.05872#A5.T3)\.

### 5\.4Prescription

The geometry gives a recipe\. Measure how many gradient directions your layer actually resolves, the small determined spike of §[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)rather than the nominal rank; call itk⋆k^\{\\star\}\. If the projection rankrrexceedsk⋆k^\{\\star\}, the directions beyondk⋆k^\{\\star\}sit in the gapless tail and are re\-drawn at each refresh, so the state carried across the refresh goes stale\. Then do two things\. Transport the optimizer state through the rotation: the first moment byRR\(Proposition[3](https://arxiv.org/html/2607.05872#Thmtheorem3)\), the second moment by the symmetric\-square rule of Theorem[1](https://arxiv.org/html/2607.05872#Thmtheorem1)\. And lowerβ2\\beta\_\{2\}toward0\.990\.99\. Both help, and their gains are partly independent \(Table[1](https://arxiv.org/html/2607.05872#S5.T1)\), so applying both is worth more than either alone\. Do not expect basis averaging to recover the difference: when the rank sits in the gapless tail there is no stable subspace to average toward \(§[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\), and four decades of averaging window leave both the perplexity and the realized rotation unchanged \(§[5\.1](https://arxiv.org/html/2607.05872#S5.SS1)\)\.

## 6Related work

#### The GaLore family and its premise\.

GaLore\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib4)\)and its refinements share one move: run an optimizer inside an explicitly recomputed top\-rrgradient subspace\. The variations are in what they carry across the refresh: LDAdam transports both moments and adds error feedback\(Robertet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib6)\), SubTrack\+\+ tracks the subspace incrementally\(Rajabiet al\.,[2025b](https://arxiv.org/html/2607.05872#bib.bib7)\), FRUGAL keeps a reduced state\(Zmushkoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib62)\), and GoLore\(Heet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib26)\)and APOLLO\(Zhuet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib19)\)alter the projection itself, while Q\-GaLore quantizes it and adapts the refresh rate per layer\(Zhanget al\.,[2024b](https://arxiv.org/html/2607.05872#bib.bib72)\)\. Memory\-efficient optimizers outside the projection family instead compress the second moment directly, by low\-rank factorization\(Shazeer and Stern,[2018](https://arxiv.org/html/2607.05872#bib.bib70)\)or block\-wise learning\-rate sharing\(Zhanget al\.,[2024a](https://arxiv.org/html/2607.05872#bib.bib71)\)\. The shared, usually unstated, premise is that this subspace is a meaningful object that persists across refreshes\. Our contribution is orthogonal to these methods: we do not propose another member of the family but measure the premise they rest on, find it false at the ranks they use, and show both why they nonetheless succeed \(energy capture, §[4](https://arxiv.org/html/2607.05872#S4)\) and what the redraw costs them \(second\-moment staleness, §[5](https://arxiv.org/html/2607.05872#S5)\)\.Robertet al\.\([2025](https://arxiv.org/html/2607.05872#bib.bib6)\)note from the theory side that GaLore requires a “strong stable\-rank assumption” but do not measure it; we measure it\. LDAdam also supplies the state\-transport rule we build on: it transports both moments by the change of basis and adds error feedback\. We do not prove that its second\-moment rule is optimal; we prove that first\-moment transport is the MMSE map and that a rotation\-blind second\-moment carry is a factor≈\(r−k⋆\)/2\\approx\(r\{\-\}k^\{\\star\}\)/2suboptimal \(Theorem[1](https://arxiv.org/html/2607.05872#Thmtheorem1), Appendix[B](https://arxiv.org/html/2607.05872#A2)\), a bound LDAdam escapes because it observes the rotation\. That LDAdam already defaults toβ2=0\.99\\beta\_\{2\}\{=\}0\.99is, on our account, not a tuning accident but a symptom of exactly the staleness we characterize\.

#### What prior “stability” measurements measured\.

Reports that the GaLore subspace is stable rest on quantities that remain small while the sign\-corrected chordal rotation is near\-maximal: a distribution of principal angles\(Zmushkoet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib62)\), which leaves the dominant tail contribution to chordal distance unmeasured, and incremental\-tracker self\-consistency\(Rajabiet al\.,[2025b](https://arxiv.org/html/2607.05872#bib.bib7)\), a property of the tracker rather than of a fresh SVD \(§[3\.2](https://arxiv.org/html/2607.05872#S3.SS2)\)\. None measured‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}, the quantity that determines what happens to carried state\.

#### Random and averaged subspaces\.

Two recent works reach compatible conclusions from the randomization side\.Rajabiet al\.\([2025a](https://arxiv.org/html/2607.05872#bib.bib15)\)argue that the gradient subspace landscape has nearly flat curvature and that much of the gradient energy lies outside any core subspace, so random low\-rank projections preserve the relevant geometry and match SVD\-based selection; they exploit this to navigate the Grassmannian with random steps\.Panet al\.\([2025](https://arxiv.org/html/2607.05872#bib.bib13)\)independently diagnose that noise\-dominated singular directions make the SVD projection unreliable, though they debias GaLore by probabilistically sampling full\-rank layer updates rather than by randomizing the projection\. Both locate an unreliability in the SVD\-selected frame past the spike that we trace to the same\-step non\-identifiability of the top\-rrcut\. A separate line averages the subspace rather than randomizing it:Gonget al\.\([2025](https://arxiv.org/html/2607.05872#bib.bib14)\)maintain an adaptively estimated low\-dimensional subspace instead of a fresh SVD at each refresh, closest in spirit to the averaged\-basis variant we test in §[5\.1](https://arxiv.org/html/2607.05872#S5.SS1)\. Our EMA\-basis measurement is our own implementation at Pythia\-1B withr=128r\{=\}128and speaks only to that setting; we make no claim about the performance reported for Alice\.

#### Gradient geometry and subspace perturbation\.

That optimization concentrates in a small gradient subspace is established\(Gur\-Ariet al\.,[2018](https://arxiv.org/html/2607.05872#bib.bib21)\), and gradient spectra serve directly as optimization diagnostics\(Liuet al\.,[2026](https://arxiv.org/html/2607.05872#bib.bib73)\); we reconcile it \(§[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\): the small determined subspace is the spike, andrrovershoots it into the gapless tail\. The link from a vanishing spectral gap to maximal subspace sensitivity is the Davis–Kahan and Wedinsin⁡θ\\sin\\thetatheory\(Davis and Kahan,[1970](https://arxiv.org/html/2607.05872#bib.bib63); Wedin,[1972](https://arxiv.org/html/2607.05872#bib.bib64)\); the maximal\-rotation value2​r\\sqrt\{2r\}is the diameter of the Stiefel manifold\(Bendokatet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib16); Meckes,[2019](https://arxiv.org/html/2607.05872#bib.bib51)\)\. The Weingarten calculus on the orthogonal group\(Collins and Matsumoto,[2009](https://arxiv.org/html/2607.05872#bib.bib39)\)enters separately, bounding the second\-moment error that the rotation induces \(§[5](https://arxiv.org/html/2607.05872#S5)\)\.

#### Forgetting factors and tracking\.

Treatingβ2\\beta\_\{2\}as a forgetting factor for a second moment that tracks an abruptly\-reset target places our staleness result in the adaptive\-filtering tradition: the bias–variance behavior of the forgetting\-factor RLS algorithm under parameter jumps is classical\(Guoet al\.,[1993](https://arxiv.org/html/2607.05872#bib.bib65)\), and our predictionβ2⋆<0\.999\\beta\_\{2\}^\{\\star\}<0\.999under a periodically re\-drawn subspace is its specialization to GaLore’s periodic refresh\. The distinctive element here is that the “jumps” are not external nonstationarity but are*imposed by the optimizer itself*at every refresh\. GaLore’s periodic SVD is likewise the batch counterpart to incremental subspace tracking \(Oja’s rule\(Oja,[1982](https://arxiv.org/html/2607.05872#bib.bib66)\), GROUSE\(Balzanoet al\.,[2010](https://arxiv.org/html/2607.05872#bib.bib67)\), PAST\(Yang,[1995](https://arxiv.org/html/2607.05872#bib.bib46)\)\); the same\-step result bounds all of these, since a subspace two simultaneous batches cannot agree on is recoverable by no tracker\. Unlike convergence analyses of low\-rank optimizers\(Heet al\.,[2024](https://arxiv.org/html/2607.05872#bib.bib26); Panet al\.,[2025](https://arxiv.org/html/2607.05872#bib.bib13)\), we characterize the identifiability of the projected subspace itself\.

## 7Limitations

Five boundaries of the evidence are worth stating plainly\. First, the full identifiability apparatus \(the four\-layer same\-step control and theN−1/4N^\{\-1/4\}averaging\-budget test\) is developed at Pythia\-160M; the same\-step disagreement andk⋆k^\{\\star\}are additionally measured on pretrained checkpoints up to 6\.9B \(§[3\.6](https://arxiv.org/html/2607.05872#S3.SS6)\), and at 1B we further verify that basis averaging neither stabilizes the subspace nor improves perplexity\. TheN−1/4N^\{\-1/4\}averaging\-budget sweep, developed at 160M, is confirmed at 410M, 1B, and 2\.8B and reproduced across GPU backends \(Appendix[F](https://arxiv.org/html/2607.05872#A6)\); only the closed\-form out\-of\-sample fit is reported solely at 160M\. Second, the basis\-averaging sweep is one seed per cell; we read it as a direction, supported by the measured rotation and a second\-cluster replication, not as a headline number\. Third, the transport arm of our2×22\{\\times\}2is the full LDAdam update, which combines state transport with error feedback; the comparison bounds their joint effect and does not separate them\. Fourth, perplexity results are single\-recipe per table \(recipes stated in each caption; magnitudes shift between recipes, directions do not\), on WikiText and Pythia only\. Fifth, the canonical\-GaLoreβ2\\beta\_\{2\}margin is training\-length dependent: present at 40K steps, absent at 10–20K, so short\-horizon tuning will not see it\. Our claims are scoped to optimizers that explicitly compute and refresh a gradient subspace; LoRA and other parameterizations make no such assumption, and our measurements say nothing about them\.

## 8Conclusion

The premise beneath GaLore\-family optimizers is that there is a low\-rank gradient subspace to track\. We measured the premise\. Beyond a reproducible core of roughly3939of128128directions, the object is not there: two estimates of the subspace taken at the same instant disagree as much as estimates taken a full refresh apart, the spectrum offers no gap for a rank cut to rest on, and the tail that a larger sample might average away turns out to be signal, shrinking four times too slowly for any budget to rescue\. What looked like a subspace rotating is a subspace being re\-drawn\.

Once that is said, the engineering sorts itself\. Stabilizing the basis is chasing the phantom: at 1B, four decades of averaging window change neither the realized rotation nor the loss\. The state, not the frame, is what must move: the first moment transports exactly through the rotation, the second through its squared entries, blind carry is provably a factor\(r−k⋆\)/2\(r\{\-\}k^\{\\star\}\)/2worse, and the transported optimizer \(the LDAdam update, transport plus error feedback\) beats the best untransported configuration in a controlled three\-seed comparison\. The cheaper fix, shortening the second moment’s memory toβ2=0\.99\\beta\_\{2\}\{=\}0\.99, helps the refreshing optimizers we tested at their reported recipes \(small and recipe\-sensitive for canonical GaLore\) and reverses in a full\-rank control, because a memory of a thousand steps is a liability precisely when the coordinates it averages over are re\-drawn every160160\.

The takeaway we want a reader to carry out of this paper is the probe, not the numbers\. Before building on a top\-rrgradient subspace, measure how much of it is real: split a batch, compute the subspace twice, and count the directions that agree\. If your rank exceeds that count, the excess is noise that will be re\-drawn at every refresh, and the right response is not a better tracker but transported state and a shorter memory\. Low\-rank training works, and it works for a reason; the reason is just not the one its geometry seemed to promise\.

## References

- J\. Baik, G\. Ben Arous, and S\. Péché \(2005\)Phase transition of the largest eigenvalue for non\-null complex sample covariance matrices\.Annals of Probability33\(5\),pp\. 1643–1697\.Cited by:[Appendix F](https://arxiv.org/html/2607.05872#A6.SS0.SSS0.Px4.p1.23)\.
- L\. Balzano, R\. Nowak, and B\. Recht \(2010\)Online identification and tracking of subspaces from highly incomplete information\.InAllerton Conference on Communication, Control, and Computing,Note:arXiv:1006\.4046Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px5.p1.2)\.
- T\. Bendokat, R\. Zimmermann, and P\.\-A\. Absil \(2024\)A Grassmann manifold handbook: basic geometry and computational aspects\.Advances in Computational Mathematics\.Note:arXiv:2011\.13699Cited by:[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px3.p1.11),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- S\. Biderman, H\. Schoelkopf, Q\. Anthony, H\. Bradley,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InICML,Cited by:[§1](https://arxiv.org/html/2607.05872#S1.p3.12)\.
- B\. Collins and S\. Matsumoto \(2009\)On some properties of orthogonal Weingarten functions\.Journal of Mathematical Physics50\(11\),pp\. 113516\.Cited by:[Appendix B](https://arxiv.org/html/2607.05872#A2.1.p1.8),[Appendix B](https://arxiv.org/html/2607.05872#A2.p1.1),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- C\. Davis and W\. M\. Kahan \(1970\)The rotation of eigenvectors by a perturbation\. III\.SIAM Journal on Numerical Analysis7\(1\),pp\. 1–46\.Cited by:[§3\.3](https://arxiv.org/html/2607.05872#S3.SS3.SSS0.Px1.p1.7),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- W\. Gong, M\. Scetbon, C\. Ma, and E\. Meeds \(2025\)Towards efficient optimizer design for LLM via structured Fisher approximation with a low\-rank extension\.arXiv:2502\.07752\.Note:arXiv:2502\.07752Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px3.p1.2)\.
- L\. Guo, L\. Ljung, and P\. Priouret \(1993\)Performance analysis of the forgetting factor RLS algorithm\.International Journal of Adaptive Control and Signal Processing7\(6\),pp\. 525–537\.Cited by:[Appendix I](https://arxiv.org/html/2607.05872#A9.p1.2),[§5\.3](https://arxiv.org/html/2607.05872#S5.SS3.p1.17),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px5.p1.2)\.
- G\. Gur\-Ari, D\. A\. Roberts, and E\. Dyer \(2018\)Gradient descent happens in a tiny subspace\.arXiv:1812\.04754\.Cited by:[§1](https://arxiv.org/html/2607.05872#S1.p4.10),[1st item](https://arxiv.org/html/2607.05872#S3.I1.i1.p1.1),[§3\.3](https://arxiv.org/html/2607.05872#S3.SS3.SSS0.Px1.p1.7),[§3\.3](https://arxiv.org/html/2607.05872#S3.SS3.p1.6),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- Y\. He, P\. Li, Y\. Hu, C\. Chen, and K\. Yuan \(2024\)Subspace optimization for large language models with convergence guarantees\.arXiv:2410\.11289\.Cited by:[§1](https://arxiv.org/html/2607.05872#S1.p1.5),[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px1.p1.14),[§4](https://arxiv.org/html/2607.05872#S4.p1.16),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px5.p1.2)\.
- Z\. Huang, H\. Cao, F\. Dong,et al\.\(2026\)Spectra: rethinking optimizers for LLMs under spectral anisotropy\.arXiv:2602\.11185\.Cited by:[§3\.3](https://arxiv.org/html/2607.05872#S3.SS3.p1.6)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.InICLR,Cited by:[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px2.p1.2)\.
- A\. Z\. Liu, E\. Paquette, and J\. Sous \(2026\)Spectral lens: activation and gradient spectra as diagnostics of LLM optimization\.arXiv:2605\.05683\.Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- E\. S\. Meckes \(2019\)The random matrix theory of the classical compact groups\.Cambridge Tracts in Mathematics, Vol\.218,Cambridge University Press\.External Links:ISBN 9781108419529Cited by:[Appendix A](https://arxiv.org/html/2607.05872#A1.SS0.SSS0.Px3.p1.13),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv:1609\.07843\.Cited by:[§5\.1](https://arxiv.org/html/2607.05872#S5.SS1.p1.23)\.
- E\. Oja \(1982\)Simplified neuron model as a principal component analyzer\.Journal of Mathematical Biology15\(3\),pp\. 267–273\.Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px5.p1.2)\.
- R\. Pan, Y\. Luo, Y\. Liu, Y\. You, and T\. Zhang \(2025\)Unbiased gradient low\-rank projection\.arXiv:2510\.17802\.Note:arXiv:2510\.17802Cited by:[§4](https://arxiv.org/html/2607.05872#S4.p1.16),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px3.p1.2),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px5.p1.2)\.
- S\. Rajabi, N\. Nonta, and S\. Rambhatla \(2025a\)Geometrically principled randomized optimization for efficient LLM training\.arXiv:2510\.01878\.Note:arXiv:2510\.01878Cited by:[§4](https://arxiv.org/html/2607.05872#S4.p1.16),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px3.p1.2)\.
- S\. Rajabi, N\. Nonta, and S\. Rambhatla \(2025b\)SubTrack\+\+: gradient subspace tracking for scalable LLM training\.arXiv:2502\.01586\.Cited by:[§1](https://arxiv.org/html/2607.05872#S1.p1.5),[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px1.p1.14),[§3\.2](https://arxiv.org/html/2607.05872#S3.SS2.p3.6),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px2.p1.1),[Remark 2](https://arxiv.org/html/2607.05872#Thmtheorem2.p1.18.18)\.
- Y\. Refael, G\. Smorodinsky, T\. Tirer, and O\. Lindenbaum \(2025\)SUMO: subspace\-aware moment\-orthogonalization for accelerating memory\-efficient LLM training\.arXiv:2505\.24749\.Cited by:[§5](https://arxiv.org/html/2607.05872#S5.p1.9)\.
- T\. Robert, M\. Safaryan, I\. Modoranu, and D\. Alistarh \(2025\)LDAdam: adaptive optimization from low\-dimensional gradient statistics\.InICLR,Cited by:[Table 3](https://arxiv.org/html/2607.05872#A5.T3.16.6.6.4),[§1](https://arxiv.org/html/2607.05872#S1.p1.5),[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px1.p1.14),[Table 1](https://arxiv.org/html/2607.05872#S5.T1),[§5](https://arxiv.org/html/2607.05872#S5.p1.9),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3),[Remark 2](https://arxiv.org/html/2607.05872#Thmtheorem2.p1.18.18)\.
- N\. Shazeer and M\. Stern \(2018\)Adafactor: adaptive learning rates with sublinear memory cost\.arXiv:1804\.04235\.Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3)\.
- D\. Su, A\. Gu, J\. Xu, Y\. Tian, and J\. Zhao \(2025\)GaLore 2: large\-scale LLM pre\-training by gradient low\-rank projection\.arXiv:2504\.20437\.Cited by:[§4](https://arxiv.org/html/2607.05872#S4.p1.16)\.
- P\. Wedin \(1972\)Perturbation bounds in connection with singular value decomposition\.BIT Numerical Mathematics12\(1\),pp\. 99–111\.Cited by:[§3\.3](https://arxiv.org/html/2607.05872#S3.SS3.SSS0.Px1.p1.7),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px4.p1.3)\.
- B\. Yang \(1995\)Projection approximation subspace tracking\.IEEE Transactions on Signal Processing43\(1\),pp\. 95–107\.Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px5.p1.2)\.
- Y\. Zhang, C\. Chen, Z\. Li, T\. Ding, C\. Wu, D\. P\. Kingma, Y\. Ye, Z\. Luo, and R\. Sun \(2024a\)Adam\-mini: use fewer learning rates to gain more\.arXiv:2406\.16793\.Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3)\.
- Z\. Zhang, A\. Jaiswal, L\. Yin, S\. Liu, J\. Zhao, Y\. Tian, and Z\. Wang \(2024b\)Q\-GaLore: quantized GaLore with INT4 projection and layer\-adaptive low\-rank gradients\.arXiv:2407\.08296\.Cited by:[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)GaLore: memory\-efficient LLM training by gradient low\-rank projection\.InICML,Cited by:[Table 3](https://arxiv.org/html/2607.05872#A5.T3.19.9.9.4),[§1](https://arxiv.org/html/2607.05872#S1.p1.5),[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px1.p1.14),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3)\.
- H\. Zhu, Z\. Zhang, W\. Cong, X\. Liu, S\. Park,et al\.\(2024\)APOLLO: SGD\-like memory, AdamW\-level performance\.arXiv:2412\.05270\.Cited by:[Table 3](https://arxiv.org/html/2607.05872#A5.T3.22.12.12.4),[§1](https://arxiv.org/html/2607.05872#S1.p1.5),[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px1.p1.14),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3)\.
- P\. Zmushko, A\. Beznosikov, M\. Takac, and S\. Horvath \(2024\)FRUGAL: memory\-efficient optimization by reducing state overhead for scalable training\.arXiv:2411\.07837\.Cited by:[§1](https://arxiv.org/html/2607.05872#S1.p1.5),[§2](https://arxiv.org/html/2607.05872#S2.SS0.SSS0.Px1.p1.14),[§3\.2](https://arxiv.org/html/2607.05872#S3.SS2.p3.6),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px1.p1.3),[§6](https://arxiv.org/html/2607.05872#S6.SS0.SSS0.Px2.p1.1)\.

## Appendix AThe sign\-corrected rotation metric and the2​r\\sqrt\{2r\}scaling

#### Sign artifact in the raw distance\.

Singular vectors carry an arbitrary per\-column sign: theii\-th column ofUUmay be negated without changing the subspace\. Computed directly,‖Unew−Uold‖F\\\|U\_\{\\rm new\}\-U\_\{\\rm old\}\\\|\_\{F\}is therefore dominated by sign inconsistencies rather than genuine rotation\. For twom×rm\\\!\\times\\\!rorthonormal matrices with independent random column signs,

𝔼​\[‖Unew−Uold‖F2\]=2​r−2​𝔼​\[tr​\(Unew⊤​Uold\)\]≈2​r,\\mathbb\{E\}\\bigl\[\\\|U\_\{\\rm new\}\-U\_\{\\rm old\}\\\|\_\{F\}^\{2\}\\bigr\]=2r\-2\\,\\mathbb\{E\}\\bigl\[\\mathrm\{tr\}\(U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}\)\\bigr\]\\approx 2r,since therrdiagonal entries ofR=Unew⊤​UoldR=U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}have independent random signs and mean zero\. The raw metric thus returns≈2​r\\approx\\sqrt\{2r\}regardless ofTT, a sign artifact rather than a measurement of rotation\.

#### Corrected metric\.

The quantity we compute and report is the principal\-angle \(chordal\) form‖Δ​U‖F=\(2​r−2​∑icos⁡θi\)1/2\\\|\\Delta U\\\|\_\{F\}=\(2r\-2\\sum\_\{i\}\\cos\\theta\_\{i\}\)^\{1/2\}withcos⁡θi=σi​\(Unew⊤​Uold\)\\cos\\theta\_\{i\}=\\sigma\_\{i\}\(U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}\)the singular values of the cross\-Gram matrix \(Eq\.[2](https://arxiv.org/html/2607.05872#S2.E2)\), which is sign\-invariant by construction and measures genuine rotation rather than the arbitrary column sign\. \(A per\-column sign alignmentsi=sign​\(ei⊤​R​ei\)s\_\{i\}=\\mathrm\{sign\}\(e\_\{i\}^\{\\top\}R\\,e\_\{i\}\)followed by a Frobenius difference upper\-bounds this quantity, with equality when the columns coincide with the principal vectors; we do not rely on that route\.\) We use‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}for the singular\-value form throughout\.

#### The2​r\\sqrt\{2r\}scaling\.

Table 2:Subspace rotation per refresh, Pythia\-1B,T=50T\{=\}50, averaged over refresh events \(the data plotted in Figure[1](https://arxiv.org/html/2607.05872#S3.F1)\)\.‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}reaches96\.796\.7–99\.6%99\.6\\%of its frame maximum2​r\\sqrt\{2r\}at every rank; the implied mean angle is8686–90∘90^\{\\circ\}\.Table[2](https://arxiv.org/html/2607.05872#A1.T2)reports‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}measured at refresh events for Pythia\-1B across ranks; the mean\-angle column isarccos⁡\(1−‖Δ​U‖F2/2​r\)\\arccos\(1\-\\\|\\Delta U\\\|\_\{F\}^\{2\}/2r\), the angle implied by the mean cosine\. The ratio‖Δ​U‖F​\(r=256\)/‖Δ​U‖F​\(r=32\)=22\.10/7\.93≈2\.79\\\|\\Delta U\\\|\_\{F\}\(r\{=\}256\)/\\\|\\Delta U\\\|\_\{F\}\(r\{=\}32\)=22\.10/7\.93\\approx 2\.79matches256/32=8≈2\.83\\sqrt\{256/32\}=\\sqrt\{8\}\\approx 2\.83to1\.5%1\.5\\%, confirming‖Δ​U‖F∝r\\\|\\Delta U\\\|\_\{F\}\\propto\\sqrt\{r\}\. All ranks reach96\.796\.7–99\.6%99\.6\\%of2​r\\sqrt\{2r\}\. The2​r\\sqrt\{2r\}ceiling is the maximum Frobenius distance between twom×rm\\\!\\times\\\!rorthonormal matrices, attained when allrrprincipal angles equalπ/2\\pi/2, the independent\-Haar\-sampling upper bound\(Meckes,[2019](https://arxiv.org/html/2607.05872#bib.bib51)\)\. That correlated adjacent SVD bases nonetheless operate near this ceiling at every refresh is the non\-trivial empirical fact established in §[3](https://arxiv.org/html/2607.05872#S3)\.

## Appendix BBlind\-estimator lower bound for the second moment \(Weingarten\)

This bound formalizes why carrying the second moment unchanged across a refresh is structurally suboptimal when the estimator does not observe the rotation, using the Weingarten calculus for Haar\-random orthogonal matrices\(Collins and Matsumoto,[2009](https://arxiv.org/html/2607.05872#bib.bib39)\)\.

###### Theorem 1\(Blind\-estimator lower bound\)\.

Letvold=λ=\(λ1,…,λr\)v\_\{\\mathrm\{old\}\}=\\lambda=\(\\lambda\_\{1\},\\dots,\\lambda\_\{r\}\)be the true per\-coordinate second moments in the current subspaceUU\(λi=𝔼​\[gi2\]\\lambda\_\{i\}=\\mathbb\{E\}\[g\_\{i\}^\{2\}\]forg=U⊤​∇ℒg=U^\{\\top\}\\nabla\\mathcal\{L\}\)\. Let the new subspace beU′=U​RU^\{\\prime\}=URwithR∼Haar​\(O​\(r\)\)R\\sim\\mathrm\{Haar\}\(O\(r\)\), and letv^\\hat\{v\}be any estimator measurable with respect toσ​\(λ\)\\sigma\(\\lambda\)only \(i\.e\. blind toRR\)\. Then the optimal new\-basis values arev∗​\(U′\)i=∑jλj​Rj​i2v^\{\*\}\(U^\{\\prime\}\)\_\{i\}=\\sum\_\{j\}\\lambda\_\{j\}R\_\{ji\}^\{2\}, and

𝔼R​\[‖v^−v∗‖2\]=‖v^−λ¯​𝟏‖2\+2​r​Var​\(λ\)r\+2≥2​r​Var​\(λ\)r\+2,\\mathbb\{E\}\_\{R\}\\bigl\[\\\|\\hat\{v\}\-v^\{\*\}\\\|^\{2\}\\bigr\]=\\\|\\hat\{v\}\-\\bar\{\\lambda\}\\mathbf\{1\}\\\|^\{2\}\+\\frac\{2r\\,\\mathrm\{Var\}\(\\lambda\)\}\{r\+2\}\\;\\geq\\;\\frac\{2r\\,\\mathrm\{Var\}\(\\lambda\)\}\{r\+2\},\(5\)with equality iffv^=λ¯​𝟏\\hat\{v\}=\\bar\{\\lambda\}\\mathbf\{1\}\(the mean predictor\)\. Canonical carryv^=λ\\hat\{v\}=\\lambdaincurs𝔼R​\[‖λ−v∗‖2\]=r​\(r\+4\)​Var​\(λ\)/\(r\+2\)\\mathbb\{E\}\_\{R\}\[\\\|\\lambda\-v^\{\*\}\\\|^\{2\}\]=r\(r\+4\)\\,\\mathrm\{Var\}\(\\lambda\)/\(r\+2\), a factor\(r\+4\)/2≈r/2\(r\+4\)/2\\approx r/2above the floor\.

###### Proof\.

By the Weingarten calculus\(Collins and Matsumoto,[2009](https://arxiv.org/html/2607.05872#bib.bib39)\),𝔼R​\[v∗​\(U′\)i\]=λ¯\\mathbb\{E\}\_\{R\}\[v^\{\*\}\(U^\{\\prime\}\)\_\{i\}\]=\\bar\{\\lambda\}andVarR​\[v∗​\(U′\)i\]=2​Var​\(λ\)/\(r\+2\)\\mathrm\{Var\}\_\{R\}\[v^\{\*\}\(U^\{\\prime\}\)\_\{i\}\]=2\\,\\mathrm\{Var\}\(\\lambda\)/\(r\+2\)\. Sincev^\\hat\{v\}is independent ofRR, the bias–variance decomposition gives𝔼R​\[\(v^i−vi∗\)2\]=\(v^i−λ¯\)2\+2​Var​\(λ\)/\(r\+2\)\\mathbb\{E\}\_\{R\}\[\(\\hat\{v\}\_\{i\}\-v^\{\*\}\_\{i\}\)^\{2\}\]=\(\\hat\{v\}\_\{i\}\-\\bar\{\\lambda\}\)^\{2\}\+2\\,\\mathrm\{Var\}\(\\lambda\)/\(r\+2\)\. Summing overiiand minimizing overv^\\hat\{v\}yields \([5](https://arxiv.org/html/2607.05872#A2.E5)\); substitutingv^=λ\\hat\{v\}=\\lambdagives the canonical\-carry cost\. ∎

#### Worked moments\.

The two moments the proof invokes follow from the second\- and fourth\-order statistics of a single Haar\-random frame\. A columnR⋅iR\_\{\\cdot i\}ofR∼Haar​\(O​\(r\)\)R\\sim\\mathrm\{Haar\}\(O\(r\)\)is a uniform point on the unit sphereSr−1S^\{r\-1\}, so its entriesxj:=Rj​ix\_\{j\}:=R\_\{ji\}satisfyxj2∼Beta​\(12,r−12\)x\_\{j\}^\{2\}\\sim\\mathrm\{Beta\}\\\!\\bigl\(\\tfrac\{1\}\{2\},\\tfrac\{r\-1\}\{2\}\\bigr\)and, forj≠kj\\neq k,

𝔼​\[xj2\]=1r,𝔼​\[xj4\]=3r​\(r\+2\),𝔼​\[xj2​xk2\]=1r​\(r\+2\)\.\\mathbb\{E\}\[x\_\{j\}^\{2\}\]=\\frac\{1\}\{r\},\\qquad\\mathbb\{E\}\[x\_\{j\}^\{4\}\]=\\frac\{3\}\{r\(r\+2\)\},\\qquad\\mathbb\{E\}\[x\_\{j\}^\{2\}x\_\{k\}^\{2\}\]=\\frac\{1\}\{r\(r\+2\)\}\.\(6\)These are consistent with∑jxj2=1\\sum\_\{j\}x\_\{j\}^\{2\}=1: summing gives∑j𝔼​\[xj4\]\+∑j≠k𝔼​\[xj2​xk2\]=3r\+2\+r−1r\+2=1\\sum\_\{j\}\\mathbb\{E\}\[x\_\{j\}^\{4\}\]\+\\sum\_\{j\\neq k\}\\mathbb\{E\}\[x\_\{j\}^\{2\}x\_\{k\}^\{2\}\]=\\tfrac\{3\}\{r\+2\}\+\\tfrac\{r\-1\}\{r\+2\}=1\. Writingv∗​\(U′\)i=∑jλj​Rj​i2v^\{\*\}\(U^\{\\prime\}\)\_\{i\}=\\sum\_\{j\}\\lambda\_\{j\}R\_\{ji\}^\{2\}andS1=∑jλjS\_\{1\}=\\sum\_\{j\}\\lambda\_\{j\},S2=∑jλj2S\_\{2\}=\\sum\_\{j\}\\lambda\_\{j\}^\{2\}, the first moment is𝔼R​\[v∗​\(U′\)i\]=S1/r=λ¯\\mathbb\{E\}\_\{R\}\[v^\{\*\}\(U^\{\\prime\}\)\_\{i\}\]=S\_\{1\}/r=\\bar\{\\lambda\}, and

𝔼R​\[v∗​\(U′\)i2\]\\displaystyle\\mathbb\{E\}\_\{R\}\\bigl\[v^\{\*\}\(U^\{\\prime\}\)\_\{i\}^\{2\}\\bigr\]=∑jλj2​𝔼​\[Rj​i4\]\+∑j≠kλj​λk​𝔼​\[Rj​i2​Rk​i2\]=3​S2\+\(S12−S2\)r​\(r\+2\)=2​S2\+S12r​\(r\+2\),\\displaystyle=\\sum\_\{j\}\\lambda\_\{j\}^\{2\}\\,\\mathbb\{E\}\[R\_\{ji\}^\{4\}\]\+\\sum\_\{j\\neq k\}\\lambda\_\{j\}\\lambda\_\{k\}\\,\\mathbb\{E\}\[R\_\{ji\}^\{2\}R\_\{ki\}^\{2\}\]=\\frac\{3S\_\{2\}\+\(S\_\{1\}^\{2\}\-S\_\{2\}\)\}\{r\(r\+2\)\}=\\frac\{2S\_\{2\}\+S\_\{1\}^\{2\}\}\{r\(r\+2\)\},VarR​\[v∗​\(U′\)i\]\\displaystyle\\mathrm\{Var\}\_\{R\}\\bigl\[v^\{\*\}\(U^\{\\prime\}\)\_\{i\}\\bigr\]=2​S2\+S12r​\(r\+2\)−S12r2=2r​\(r\+2\)​\(S2−S12r\)=2​Var​\(λ\)r\+2,\\displaystyle=\\frac\{2S\_\{2\}\+S\_\{1\}^\{2\}\}\{r\(r\+2\)\}\-\\frac\{S\_\{1\}^\{2\}\}\{r^\{2\}\}=\\frac\{2\}\{r\(r\+2\)\}\\Bigl\(S\_\{2\}\-\\frac\{S\_\{1\}^\{2\}\}\{r\}\\Bigr\)=\\frac\{2\\,\\mathrm\{Var\}\(\\lambda\)\}\{r\+2\},\(7\)usingr​Var​\(λ\)=S2−S12/rr\\,\\mathrm\{Var\}\(\\lambda\)=S\_\{2\}\-S\_\{1\}^\{2\}/r\. The canonical carryv^=λ\\hat\{v\}=\\lambdathen costs𝔼R​\[‖λ−v∗‖2\]=∑i\[\(λi−λ¯\)2\+VarR​\(vi∗\)\]=r​Var​\(λ\)\+2​r​Var​\(λ\)r\+2=r​\(r\+4\)r\+2​Var​\(λ\)\\mathbb\{E\}\_\{R\}\[\\\|\\lambda\-v^\{\*\}\\\|^\{2\}\]=\\sum\_\{i\}\\bigl\[\(\\lambda\_\{i\}\-\\bar\{\\lambda\}\)^\{2\}\+\\mathrm\{Var\}\_\{R\}\(v^\{\*\}\_\{i\}\)\\bigr\]=r\\,\\mathrm\{Var\}\(\\lambda\)\+\\tfrac\{2r\\,\\mathrm\{Var\}\(\\lambda\)\}\{r\+2\}=\\tfrac\{r\(r\+4\)\}\{r\+2\}\\mathrm\{Var\}\(\\lambda\), whereas the mean predictorv^=λ¯​𝟏\\hat\{v\}=\\bar\{\\lambda\}\\mathbf\{1\}attains the floor2​rr\+2​Var​\(λ\)\\tfrac\{2r\}\{r\+2\}\\mathrm\{Var\}\(\\lambda\); their ratio is\(r\+4\)/2\(r\+4\)/2, the factor quoted in Theorem[1](https://arxiv.org/html/2607.05872#Thmtheorem1)\.

## Appendix CFirst\-moment transport is the MMSE linear map

###### Proposition 3\(MMSE first\-moment transport\)\.

Let𝐦=Uold⊤​g\\mathbf\{m\}=U\_\{\\rm old\}^\{\\top\}gwithg∼𝒩​\(0,σ2​I\)g\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\), and consider estimatingm^=Unew⊤​g\\hat\{m\}=U\_\{\\rm new\}^\{\\top\}gfrom𝐦\\mathbf\{m\}\. Among linear estimatorsA​𝐦A\\mathbf\{m\}, the minimum\-mean\-squared\-error choice isA=Unew⊤​Uold=RA=U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}=R, i\.e\.m^=R​𝐦\\hat\{m\}=R\\,\\mathbf\{m\}\.

###### Proof\.

Writeg=Uold​𝐦\+\(I−Uold​Uold⊤\)​gg=U\_\{\\rm old\}\\mathbf\{m\}\+\(I\-U\_\{\\rm old\}U\_\{\\rm old\}^\{\\top\}\)g, som^=R​𝐦\+Unew⊤​\(I−Uold​Uold⊤\)​g\\hat\{m\}=R\\mathbf\{m\}\+U\_\{\\rm new\}^\{\\top\}\(I\-U\_\{\\rm old\}U\_\{\\rm old\}^\{\\top\}\)g\. The second term has zero mean and is uncorrelated with𝐦\\mathbf\{m\}under isotropy, so𝔼​‖m^−A​𝐦‖2=𝔼​‖R​𝐦−A​𝐦‖2\+σ2​‖Unew⊤​\(I−Uold​Uold⊤\)‖F2\\mathbb\{E\}\\\|\\hat\{m\}\-A\\mathbf\{m\}\\\|^\{2\}=\\mathbb\{E\}\\\|R\\mathbf\{m\}\-A\\mathbf\{m\}\\\|^\{2\}\+\\sigma^\{2\}\\\|U\_\{\\rm new\}^\{\\top\}\(I\-U\_\{\\rm old\}U\_\{\\rm old\}^\{\\top\}\)\\\|\_\{F\}^\{2\}\. The second term is irreducible; the first is minimized uniquely atA=RA=R\. ∎

The isotropy assumption is exact only in the near\-degenerate tail of the gradient spectrum, which is precisely the part of the subspace that redraws \(§[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\); the determined spike is anisotropic but stable and does not rotate, soRRis the optimal transport over exactly the directions that move\.

## Appendix DCoverage equals cumulative spectral mass

###### Proposition 4\(Coverage growth\)\.

If the gradient second\-momentΣg\\Sigma\_\{g\}has eigenvaluesλ1≥⋯≥λn\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{n\}andUUspans the top\-rreigenvectors, then𝔼g​\[γg\]=∑i=1rλi/∑i=1nλi\\mathbb\{E\}\_\{g\}\[\\gamma\_\{g\}\]=\\sum\_\{i=1\}^\{r\}\\lambda\_\{i\}/\\sum\_\{i=1\}^\{n\}\\lambda\_\{i\}\.

###### Proof\.

𝔼​\[‖U​U⊤​g‖2\]=tr​\(U​U⊤​Σg​U​U⊤\)=∑i=1rλi\\mathbb\{E\}\[\\\|UU^\{\\top\}g\\\|^\{2\}\]=\\mathrm\{tr\}\(UU^\{\\top\}\\Sigma\_\{g\}UU^\{\\top\}\)=\\sum\_\{i=1\}^\{r\}\\lambda\_\{i\}and𝔼​\[‖g‖2\]=tr​\(Σg\)=∑i=1nλi\\mathbb\{E\}\[\\\|g\\\|^\{2\}\]=\\mathrm\{tr\}\(\\Sigma\_\{g\}\)=\\sum\_\{i=1\}^\{n\}\\lambda\_\{i\}\. ∎

When a few eigenvalues dominate \(the spike of §[3\.3](https://arxiv.org/html/2607.05872#S3.SS3)\),𝔼​\[γg\]\\mathbb\{E\}\[\\gamma\_\{g\}\]rises steeply forrrup to the spike width and then flattens: past the spike, adding tail directions of nearly equal energy raises coverage only marginally and almost independently of*which*tail directions are chosen, the property that makes a re\-drawn frame an essentially equivalent projector \(§[4](https://arxiv.org/html/2607.05872#S4)\)\.

## Appendix EExperimental configurations and reproducibility

#### Models and corpora\.

Spectrum and drift measurements use Pythia\-160M; theβ2\\beta\_\{2\}validation and the‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}rotation law use Pythia\-1B\. Language\-model training is on WikiText; the drift\-rate invariance check additionally uses C4 \(below\)\. Runs were executed on two independent clusters, NVIDIA L40S and AMD MI210, which we use to cross\-check the staleness measurements\.

#### β2\\beta\_\{2\}runs \(Figure[5](https://arxiv.org/html/2607.05872#S5.F5)\)\.

Table[3](https://arxiv.org/html/2607.05872#A5.T3)gives the exact values; each row has its own recipe, so perplexities are comparable within rows only \(LDAdam’s57\.6957\.69here and its18\.7318\.73in Table[1](https://arxiv.org/html/2607.05872#S5.T1)are the same optimizer at1010K/lr10−310^\{\-3\}and4040K/lr4×10−34\\times 10^\{\-3\}respectively\)\. Canonical GaLore uses the configurationr=128r\{=\}128, refreshT=160T\{=\}160, learning rate4×10−34\\times 10^\{\-3\}, projection scale0\.250\.25, cosine schedule with warmup40004000,4040K steps, three seeds perβ2\\beta\_\{2\}; LDAdam uses the faithful official implementation atr=128r\{=\}128, learning rate10−310^\{\-3\},1010K steps, three seeds perβ2\\beta\_\{2\}; APOLLO\-Mini uses two seeds\. The full\-rank AdamW control runs the identical data and schedule with no projection\.

Table 3:Loweringβ2\\beta\_\{2\}from the0\.9990\.999default to0\.990\.99at Pythia\-1B \(the data of Figure[5](https://arxiv.org/html/2607.05872#S5.F5)\)\. For each refreshing optimizer shown, the lower value is better; for full\-rank AdamW the preference reverses\. LDAdam’s gap is largest at its own1010K recipe, but gap sizes are not comparable across recipes and we do not read it as a larger staleness effect \(§[5\.3](https://arxiv.org/html/2607.05872#S5.SS3)\)\. APOLLO\-Mini \(below the second rule\) refreshes a random Gaussian projection rather than an SVD top\-rrsubspace and enters as a directional check only\. The canonical\-GaLore margin is training\-length dependent \(§[5\.3](https://arxiv.org/html/2607.05872#S5.SS3)\)\.
#### Spectrum probe \(Figure[2](https://arxiv.org/html/2607.05872#S3.F2)\)\.

At Pythia\-160M and 1B we take the per\-matrix gradientGGfor the embedding, attention query/key/value, attention output, and MLP weight matrices, compute its singular values, and report: the adjacent\-ratioσr/σr\+1\\sigma\_\{r\}/\\sigma\_\{r\+1\}atr=128r\{=\}128; the maximum adjacent ratio anywhere in the bulk \(maxi⁡σi/σi\+1\\max\_\{i\}\\sigma\_\{i\}/\\sigma\_\{i\+1\}over the smooth region, “max bulk gap”\); the effective rankexp⁡\(H\)\\exp\(H\)withHHthe Shannon entropy of the normalized squared spectrum; and the top\-rrnuclear fraction∑i≤rσi/∑iσi\\sum\_\{i\\leq r\}\\sigma\_\{i\}/\\sum\_\{i\}\\sigma\_\{i\}\. The two scales agree closely: meanσr/σr\+1\\sigma\_\{r\}/\\sigma\_\{r\+1\}atr=128r\{=\}128is≈1\.005\\approx 1\.005at both scales \(1\.0011\.001–1\.0101\.010at160160M,1\.0011\.001–1\.0171\.017at11B\); max bulk gap stays in1\.0111\.011–1\.0221\.022\(160160M\) and1\.0111\.011–1\.0301\.030\(11B\); effective rank spans8989–280280\(160160M\) and9797–358358\(11B\), straddlingrrat both; and the top\-rrnuclear fraction is0\.520\.52–0\.680\.68\(160160M\) and0\.460\.46–0\.650\.65\(11B\)\. Measurements are taken over four layer types at steps80/100/12080/100/120\.

#### Rotation probe \(‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\},αdrift\\alpha\_\{\\mathrm\{drift\}\}\)\.

Both quantities use the same primitive, the mean principal angle between two orthonormal bases,θ¯=arccos⁡\(svdvals​\(Unew⊤​Uold\)\)\\overline\{\\theta\}=\\arccos\\bigl\(\\mathrm\{svdvals\}\(U\_\{\\rm new\}^\{\\top\}U\_\{\\rm old\}\)\\bigr\)averaged over columns\.‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}\(Eq\.[2](https://arxiv.org/html/2607.05872#S2.E2)\) applies it across a refresh of the imposed GaLore subspace;αdrift\\alpha\_\{\\mathrm\{drift\}\}\(Appendix[J](https://arxiv.org/html/2607.05872#A10)\) applies it per step to the intrinsic top\-rrgradient\-covariance subspace\.

## Appendix FIdentifiability probe protocols

The same\-step control \(§[3\.2](https://arxiv.org/html/2607.05872#S3.SS2)\), the reproducible countk⋆k^\{\\star\}\(§[3\.4](https://arxiv.org/html/2607.05872#S3.SS4)\), and the averaging\-budget scaling \(§[3\.5](https://arxiv.org/html/2607.05872#S3.SS5)\) all read from one construction, described here so they reproduce without the code\. All three run on a pretrained Pythia\-160M infloat32, evaluation mode \(no dropout; gradients still flow\),r=128r\{=\}128, on the four layer types \{embed\_in, layer\-0 attentionquery\_key\_value, attentiondense, MLPdense\_h\_to\_4h\}\.

#### Gradient pool\.

We drawK=128K\{=\}128minibatches, eachB=8B\{=\}8sequences of length512512taken from disjoint, non\-overlapping slices of WikiText\-103, and for each compute the per\-matrix gradient of the next\-token cross\-entropy loss\. For a gradientGGwe work on the smaller side,M=GM=Gifrows≤cols\\mathrm\{rows\}\\leq\\mathrm\{cols\}elseG⊤G^\{\\top\}, and define its top\-rrsubspace as the top\-rreigenvectors ofM​M⊤MM^\{\\top\}\(equivalently the top\-rrleft singular vectors\)\. Because every minibatch is evaluated at the*same*checkpoint, two subspaces built from two pools differ only in the sample, never in elapsed training\.

#### Same\-step disagreement andk⋆k^\{\\star\}\(Figure[3](https://arxiv.org/html/2607.05872#S3.F3)a\)\.

For an averaging foldN∈\{1,2,4,8,16,32,64\}N\\in\\\{1,2,4,8,16,32,64\\\}and44repetitions, we draw2​N2Ndistinct pool indices, split them into two halves ofNN, average each half intoG¯A,G¯B\\bar\{G\}\_\{A\},\\bar\{G\}\_\{B\}, and take their top\-rrsubspacesU1,U2U\_\{1\},U\_\{2\}\. We report‖Δ​U‖F=\(2​r−2​∑iσi​\(U1⊤​U2\)\)1/2\\\|\\Delta U\\\|\_\{F\}=\(2r\-2\\sum\_\{i\}\\sigma\_\{i\}\(U\_\{1\}^\{\\top\}U\_\{2\}\)\)^\{1/2\}\(Eq\.[2](https://arxiv.org/html/2607.05872#S2.E2)\), the subspace overlapov=∑iσi​\(U1⊤​U2\)2/r∈\[0,1\]\\mathrm\{ov\}=\\sum\_\{i\}\\sigma\_\{i\}\(U\_\{1\}^\{\\top\}U\_\{2\}\)^\{2\}/r\\in\[0,1\], andk⋆​\(N\)=r⋅ovk^\{\\star\}\(N\)=r\\cdot\\mathrm\{ov\}, the effective number of shared directions\. AtN=1N\{=\}1this is the same\-step split\-batch disagreement \(0\.725​2​r0\.725\\,\\sqrt\{2r\}over four layers; a four\-layer\-meank⋆≈39k^\{\\star\}\\\!\\approx\\\!39,4848on the query\-key\-value matrix, per\-layer standard error below11over the1212repetitions of the trajectory run below\)\. The across\-time reference0\.742​2​r0\.742\\,\\sqrt\{2r\}is a separate measurement: the sign\-corrected rotation‖Δ​U‖F\\\|\\Delta U\\\|\_\{F\}across one refresh interval of160160M training, not on the frozen pool\. The frozen same\-step value is the noise floor; the across\-time rotation exceeds it only marginally\.

#### The floor is not specific to the pretrained checkpoint\.

A natural objection is that the same\-step floor is read from a pretrained checkpoint while the across\-time rotation is read during training, so the two sit at different optimization states\. We therefore reran theN=1N\{=\}1probe unchanged \(same four layers,K=128K\{=\}128pool,r=128r\{=\}128,1212repetitions\) at five points along the released Pythia\-160M training trajectory \(revisions step10001000, step1000010000, step4000040000, step8000080000, step143000143000\)\. The four\-layer same\-step disagreement is\{0\.648,0\.668,0\.700,0\.708,0\.710\}​2​r\\\{0\.648,\\,0\.668,\\,0\.700,\\,0\.708,\\,0\.710\\\}\\,\\sqrt\{2r\}: lowest early in training and rising toward the pretrained value, but never below0\.64​2​r0\.64\\,\\sqrt\{2r\}\. The reproducible core is correspondingly larger early \(query\-key\-valuek⋆k^\{\\star\}50\.9,58\.5,52\.8,51\.4,51\.650\.9,58\.5,52\.8,51\.4,51\.6across the same checkpoints\)\. At every training state the across\-time rotation \(0\.7420\.742\) therefore exceeds the same\-step floor by at most a few percent of2​r\\sqrt\{2r\}, so the rotation is dominated by non\-identifiability rather than by motion of a stable subspace, and the conclusion does not depend on the state at which the floor is measured\.

#### NN\-averaging spectra and the budget test \(Figure[3](https://arxiv.org/html/2607.05872#S3.F3)b\)\.

The same run additionally logs the singular spectrum of each averaged matrixG¯A\\bar\{G\}\_\{A\}\(densely over the top300300ranks, log\-spaced in the tail\)\. Three analyses follow\. \(i\)*Deep\-tail scaling*: a tail singular value at fixed rank, scaled byN\\sqrt\{N\}, is flat if the tail is pure sampling noise \(σ∝N−1/2\\sigma\\propto N^\{\-1/2\}\); the measured value instead rises2\.02\.0–3\.2×3\.2\\timesfromN=1N\{=\}1to6464, i\.e\.σ∝N−1/4\\sigma\\propto N^\{\-1/4\}\. \(ii\)*Out\-of\-sample prediction*: a spiked\-covariance \(BBP\) model\(Baiket al\.,[2005](https://arxiv.org/html/2607.05872#bib.bib58)\)with a single noise scaleν1\\nu\_\{1\}per layer, calibrated only to reproduce the measured overlap atN=1N\{=\}1, predictsov​\(4​N\)\\mathrm\{ov\}\(4N\)from the de\-censored spectrum ofG¯N\\bar\{G\}\_\{N\}\(setting the noise floor toν1/N\\nu\_\{1\}/\\sqrt\{N\}\); over2020held\-out points the mean absolute error is0\.0730\.073, above the pre\-registered pass threshold of0\.030\.03, which is why we report a measured scaling and not a closed\-form law\. \(iii\)*De\-censored exponent*: the log–log slope of ranks33–6060of theN=64N\{=\}64spectrum gives a signal power law with exponentα=1\.03\\alpha=1\.03–1\.351\.35, agreeing with the exponent implied by the overlap\-versus\-NNcurve\.

#### The averaging\-budget result holds across scale and GPU backend\.

We reran the same\-step andNN\-averaging probe unchanged on pretrained checkpoints at 410M, 1B, and 2\.8B\. At every scale, averaging toN=64N\{=\}64\(64×64\\timesthe gradient sample\) removes only2424–29%29\\%of the single\-batch same\-step disagreement \(Table[4](https://arxiv.org/html/2607.05872#A6.T4)\), the same weak decay measured at 160M, so no averaging budget restores identifiability at any scale tested\. As a numerics control, rerunning the 160M probe on a second GPU backend \(AMD MI210, ROCm\) reproduces the reference measurement \(NVIDIA L40S, CUDA\) to within0\.3%0\.3\\%\(0\.7090\.709versus0\.7070\.707atN=1N\{=\}1\), confirming the persistent tail is a property of the gradient, not of the linear\-algebra backend\.

Table 4:Same\-step disagreement decay underNN\-fold averaging, across scale \(pretrained checkpoints,r=128r\{=\}128, four\-layer mean\)\. Entries are theN=64N\{=\}64disagreement as a fraction of theN=1N\{=\}1disagreement:1\.01\.0would mean averaging changed nothing and0would mean it fully resolved the subspace\. Averaging removes only about a quarter at every scale; the reproducible corek⋆k^\{\\star\}separately shrinks with scale \(Table[5](https://arxiv.org/html/2607.05872#A7.T5)\)\.

## Appendix GGenerality of same\-step non\-identifiability

Table[5](https://arxiv.org/html/2607.05872#A7.T5)gives the per\-model same\-step disagreement \(§[3\.6](https://arxiv.org/html/2607.05872#S3.SS6), Figure[4](https://arxiv.org/html/2607.05872#S3.F4)\)\. The probe is the same\-step protocol of Appendix[F](https://arxiv.org/html/2607.05872#A6)run unchanged on pretrained checkpoints \(all trained with full\-rank Adam\),r=128r\{=\}128,K=32K\{=\}32gradient matrices ofB=4B\{=\}4sequences, four layer types \(embedding, attention input, attention output, MLP\), four repetitions; the language models use WikiText, the vision transformer uses CIFAR\-10 images\. We report the mean over layers of theN=1N\{=\}1disagreement‖Δ​U‖F/2​r\\\|\\Delta U\\\|\_\{F\}/\\sqrt\{2r\}, the mean adjacent singular\-value ratio atrr, and the mean reproducible countk⋆k^\{\\star\}\. The disagreement rises with scale andk⋆k^\{\\star\}falls; no model shows a spectral gap atrr\.

Table 5:Same\-step subspace disagreement across architectures, scales, and domains \(pretrained, full\-rank\-Adam checkpoints;r=128r\{=\}128; mean over four layer types\)\. The Pythia\-160M row reproduces the main\-text control \(0\.730\.73here atB=4,K=32B\{=\}4,K\{=\}32versus0\.7250\.725atB=8,K=128B\{=\}8,K\{=\}128;k⋆k^\{\\star\}shifts by one under the same change of settings,3838here versus the four\-layer mean3939of the main text\)\.#### Rank versusk⋆k^\{\\star\}: transport, do not truncate\.

A natural reading ofk⋆k^\{\\star\}is that the rank should be reduced toward it\. It should not\. In a single\-seed sweep at Pythia\-1B \(β2=0\.99\\beta\_\{2\}\{=\}0\.99,T=160T\{=\}160,1414K steps\), canonical GaLore reaches27\.927\.9perplexity atr=128r\{=\}128against29\.829\.8atr=64r\{=\}64, and worse below \(this recipe is not comparable to the4040K runs elsewhere; we compare only within it\)\. The result is directional but consistent with the coverage argument \(§[4](https://arxiv.org/html/2607.05872#S4)\): the directions beyondk⋆k^\{\\star\}are individually non\-identifiable yet still capture useful gradient energy, so the excess rank is worth keeping\. The fix for the stale state it carries is to transport it \(§[5\.2](https://arxiv.org/html/2607.05872#S5.SS2)\), not to truncate the rank\.

## Appendix HThe second moment lags on the same curve as full\-rank Adam

The mechanism of §[5](https://arxiv.org/html/2607.05872#S5)predicts that the second moment’s staleness should be governed by its memoryτv=1/\(1−β2\)\\tau\_\{v\}=1/\(1\-\\beta\_\{2\}\), and that a GaLore\-family optimizer at a givenβ2\\beta\_\{2\}should be no more anomalous than full\-rank Adam at the sameβ2\\beta\_\{2\}\. We confirm both\. We define the per\-cycle*staleness lag*as the mean distance between the carried second moment and its post\-rotation optimum over a refresh cycle\. Lag rises monotonically withτv\\tau\_\{v\}: for full\-rank Adam at Pythia\-160M \(AMD MI210,44K steps, two seeds\), lag=0\.11,0\.14,0\.19,0\.25,0\.26,0\.29=0\.11,0\.14,0\.19,0\.25,0\.26,0\.29atβ2=0\.9,0\.95,0\.97,0\.99,0\.995,0\.999\\beta\_\{2\}=0\.9,0\.95,0\.97,0\.99,0\.995,0\.999\(τv=10,20,33,100,200,1000\\tau\_\{v\}=10,20,33,100,200,1000\), with seeds agreeing to∼0\.01\{\\sim\}0\.01\. On a second cluster \(NVIDIA L40S\), GaLore at matchedβ2\\beta\_\{2\}lies on the same curve as full\-rank Adam measured there:0\.130\.13against0\.160\.16atβ2=0\.95\\beta\_\{2\}\{=\}0\.95, and0\.300\.30against0\.360\.36atβ2=0\.99\\beta\_\{2\}\{=\}0\.99\(within\-cluster comparisons only\)\. The lag is also flat across the refresh interval \(0\.270\.27–0\.310\.31over an8×8\\timesrange ofTT\), as expected for a quantity set byτv\\tau\_\{v\}rather than byTT\. This replicates across both clusters \(NVIDIA L40S and AMD MI210, different framework versions and data pipelines\)\.

We restrict this overlay claim toβ2≤0\.99\\beta\_\{2\}\\leq 0\.99\. Atβ2=0\.999\\beta\_\{2\}=0\.999the instantaneous lag shows a large transient that does not persist to steady state; we therefore do not read it as a steady\-state divergence, and the loss consequence of high\-β2\\beta\_\{2\}staleness is the integrated effect documented in §[5\.3](https://arxiv.org/html/2607.05872#S5.SS3), not an instantaneous lag explosion\.

## Appendix IA forgetting\-factor bias–variance argument forβ2\\beta\_\{2\}

This appendix supports the qualitative claim of §[5\.3](https://arxiv.org/html/2607.05872#S5.SS3)that a subspace re\-drawn everyT≪τvT\\ll\\tau\_\{v\}steps pushes the error\-minimizing second\-moment decay below the no\-reset optimum\. It is a specialization of the classical forgetting\-factor bias–variance tradeoff\(Guoet al\.,[1993](https://arxiv.org/html/2607.05872#bib.bib65)\)to a target that the optimizer resets itself\. We do not derive an optimalβ2\\beta\_\{2\}; we show the minimizer is interior and moves down as the reset becomes more frequent, which is all the prescription rests on\.

#### Model\.

Fix one projected coordinate\. The second moment follows the exponential moving averagevt=β2​vt−1\+\(1−β2\)​xtv\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)x\_\{t\}, wherext=g~t2x\_\{t\}=\\tilde\{g\}\_\{t\}^\{\\,2\}is the instantaneous squared projected gradient, an unbiased noisy reading of the true per\-coordinate second moment with𝔼​\[xt\]=λt∗\\mathbb\{E\}\[x\_\{t\}\]=\\lambda\_\{t\}^\{\\ast\}andVar​\(xt\)=s2\\mathrm\{Var\}\(x\_\{t\}\)=s^\{2\}\. The targetλt∗\\lambda\_\{t\}^\{\\ast\}is piecewise constant, holding for theTTsteps of a refresh interval and jumping at each refresh when the basis is re\-drawn\. Under the Haar model of Appendix[B](https://arxiv.org/html/2607.05872#A2)the coordinate’s target after a refresh has the same meanλ¯\\bar\{\\lambda\}and a mean\-squared changeΔ2¯:=𝔼​\[\(λnew∗−λold∗\)2\]=Θ​\(Var​\(λ\)\)\\overline\{\\Delta^\{2\}\}:=\\mathbb\{E\}\[\(\\lambda^\{\\ast\}\_\{\\mathrm\{new\}\}\-\\lambda^\{\\ast\}\_\{\\mathrm\{old\}\}\)^\{2\}\]=\\Theta\(\\mathrm\{Var\}\(\\lambda\)\)across the jump\.

#### The two error terms\.

Averaged over a refresh interval, the steady\-state per\-coordinate mean\-squared error𝔼​\[\(vt−λt∗\)2\]\\mathbb\{E\}\[\(v\_\{t\}\-\\lambda\_\{t\}^\{\\ast\}\)^\{2\}\]splits into a tracking\-noise term and a lag term:

MSE​\(β2\)=1−β21\+β2​s2⏟noise\+Δ2¯​1−β22​TT​\(1−β22\)⏟lag\.\\mathrm\{MSE\}\(\\beta\_\{2\}\)\\;=\\;\\underbrace\{\\frac\{1\-\\beta\_\{2\}\}\{1\+\\beta\_\{2\}\}\\,s^\{2\}\}\_\{\\text\{noise\}\}\\;\+\\;\\underbrace\{\\overline\{\\Delta^\{2\}\}\\,\\frac\{1\-\\beta\_\{2\}^\{2T\}\}\{T\\,\(1\-\\beta\_\{2\}^\{2\}\)\}\}\_\{\\text\{lag\}\}\.\(8\)The noise term is the stationary variance of an EMA of i\.i\.d\. readings,Var​\(v∞\)=\(1−β2\)​s2/\(1\+β2\)\\mathrm\{Var\}\(v\_\{\\infty\}\)=\(1\-\\beta\_\{2\}\)s^\{2\}/\(1\+\\beta\_\{2\}\); it*decreases*inβ2\\beta\_\{2\}, so a longer memory averages away more sampling noise\. The lag term is the mean\-squared residual bias after a target jump: right after a refresh the bias decays asβ2k\\beta\_\{2\}^\{k\}over thekksteps since the jump, and averagingΔ2¯​β22​k\\overline\{\\Delta^\{2\}\}\\beta\_\{2\}^\{2k\}overk=0,…,T−1k=0,\\dots,T\-1givesΔ2¯​\(1−β22​T\)/\(T​\(1−β22\)\)\\overline\{\\Delta^\{2\}\}\(1\-\\beta\_\{2\}^\{2T\}\)/\\bigl\(T\(1\-\\beta\_\{2\}^\{2\}\)\\bigr\); it*increases*inβ2\\beta\_\{2\}\.

#### Why the optimum moves down under refresh\.

Writeτv=1/\(1−β2\)\\tau\_\{v\}=1/\(1\-\\beta\_\{2\}\)\. Forβ2\\beta\_\{2\}near11,1−β22≈2/τv1\-\\beta\_\{2\}^\{2\}\\approx 2/\\tau\_\{v\}andβ22​T≈e−2​T/τv\\beta\_\{2\}^\{2T\}\\approx e^\{\-2T/\\tau\_\{v\}\}, so the lag term is12​Δ2¯​\(τv/T\)​\(1−e−2​T/τv\)\\tfrac\{1\}\{2\}\\overline\{\\Delta^\{2\}\}\\,\(\\tau\_\{v\}/T\)\\bigl\(1\-e^\{\-2T/\\tau\_\{v\}\}\\bigr\), the sameτv​\(1−e−2​T/τv\)/T\\tau\_\{v\}\(1\-e^\{\-2T/\\tau\_\{v\}\}\)/Tshape that governs any EMA tracking a reset target\. Two limits fix its behavior\. When resets are rare relative to the memory,T≫τvT\\gg\\tau\_\{v\}, the lag term isO​\(τv/T\)→0O\(\\tau\_\{v\}/T\)\\to 0andMSE\\mathrm\{MSE\}is monotone decreasing inτv\\tau\_\{v\}: the minimizer isβ2→1\\beta\_\{2\}\\to 1, the standard no\-reset preference\. When resets are frequent,T≪τvT\\ll\\tau\_\{v\},1−e−2​T/τv≈2​T/τv1\-e^\{\-2T/\\tau\_\{v\}\}\\approx 2T/\\tau\_\{v\}and the lag term*saturates*atΔ2¯\\overline\{\\Delta^\{2\}\}, its ceiling, onceτv≳T\\tau\_\{v\}\\gtrsim T\. Past that point raisingβ2\\beta\_\{2\}buys only the residualO​\(s2/τv\)O\(s^\{2\}/\\tau\_\{v\}\)of noise reduction while holding near\-maximal lag, so the minimizerβ2⋆\\beta\_\{2\}^\{\\star\}is interior and decreases asTTshrinks\. This is the direction §[5\.3](https://arxiv.org/html/2607.05872#S5.SS3)confirms empirically: the second moment, whose target GaLore re\-draws everyT≪τvT\\ll\\tau\_\{v\}steps, prefersβ2=0\.99\\beta\_\{2\}\{=\}0\.99over0\.9990\.999, while the full\-rank control, whose target is never reset \(T=∞T=\\infty\), keeps the0\.9990\.999preference\. We do not claim0\.990\.99is the derived optimum; the argument fixes only the sign of the shift\.

## Appendix JThe eigenbasis drift rate is architecture\-intrinsic

The rotation law‖Δ​U‖F≈2​r\\\|\\Delta U\\\|\_\{F\}\\approx\\sqrt\{2r\}concerns the*imposed*GaLore subspace\. For completeness we also measure the drift rate of the model’s*intrinsic*gradient geometry: the per\-step mean principal\-angle rotation of the top\-rrgradient\-covariance eigenbasis,αdrift\\alpha\_\{\\mathrm\{drift\}\}\. At Pythia\-160M this isαdrift≈0\.05\\alpha\_\{\\mathrm\{drift\}\}\\approx 0\.05rad/step, and it is strikingly invariant:0\.0490\.049–0\.0540\.054across allβ2\\beta\_\{2\},0\.0570\.057\(random sampling\) versus0\.0540\.054\(sequential\), and0\.05580\.0558\(WikiText\) versus0\.05560\.0556\(C4\), a difference below0\.5%0\.5\\%\. The drift rate is thus a property of the architecture, not of the optimizer setting, the sampling order, or the corpus\.