Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

# Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
Source: [https://arxiv.org/html/2605.26484](https://arxiv.org/html/2605.26484)
###### Abstract

Model merging has emerged as a lightweight paradigm for enhancing Large Language Models \(LLMs\), yet its underlying mechanisms remain poorly understood\. In this work, we analyze late\-stage pre\-training trajectories and uncover aRank\-1 Subspacephenomenon: while raw optimization steps oscillate violently, consecutive*merged*checkpoints collapse onto a stable, approximately one\-dimensional linear manifold\. We theoretically ground this observation in a*river\-valley*landscape analysis: averaging acts as a geometric low\-pass filter that dampens high\-curvature noise to reveal the optimal descent direction\. Capitalizing on this insight, we proposeExtra\-Merge, a training\-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates\. Extensive experiments across GPT\-2 and LLaMA families \(124M to 2B\) demonstrate that Extra\-Merge consistently outperforms standard merging baselines\. Notably, it yields consistent zero\-shot accuracy gains on Pythia\-12B downstream tasks and generalizes effectively to the Muon optimizer\(Jordanet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib28)\)\.

Machine Learning, ICML

## 1Introduction

Pre\-training Large Language Models \(LLMs\) is characterized by rigid optimization schedules and immense computational costs\(OpenAIet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib47); Comaniciet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib37)\)\. Modern pre\-training runs consume vast GPU hours\(Kaplanet al\.,[2020](https://arxiv.org/html/2605.26484#bib.bib12); Hoffmannet al\.,[2022](https://arxiv.org/html/2605.26484#bib.bib13);hägele2024scalinglawscomputeoptimaltraining\), with the optimization trajectory strictly dictated by learning rate schedulers—such as Warmup\-Stable\-Decay \(WSD\)\(Huet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib15)\)or Cosine annealing\. Consequently, the final model capability is often path\-dependent, realized only during the terminal phase of the learning rate decay\. Given these constraints, there is a critical demand for*post\-hoc*mechanisms that can exploit the latent information within the optimization trajectory to maximize performance, without incurring the prohibitive cost of additional gradient steps\.

In response, model merging has emerged as a potent, training\-free paradigm to address this efficiency bottleneck\. Rooted in classical Polyak averaging\(Polyak and Juditsky,[1992](https://arxiv.org/html/2605.26484#bib.bib16)\)and revitalized by Stochastic Weight Averaging \(SWA\)\(Izmailovet al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib17)\), this approach posits that averaging points along the optimization path approximates the center of a flat local minimum, thereby enhancing generalization\. Recently, frameworks like Latest Weight Averaging \(LAWA\)\(Kaddour,[2022](https://arxiv.org/html/2605.26484#bib.bib4)\)and Pre\-trained Model Averaging \(PMA\)\(Liet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib1)\)have successfully adapted this paradigm to LLM pre\-training\. By aggregating late\-stage checkpoints via uniform or exponential moving averages, these methods consistently reduce validation loss and bolster performance on downstream reasoning tasks, offering a ”free lunch” improvement over the final converged model\.

Despite model merging’s widespread adoption, the geometric principles underpinning model merging in the non\-convex landscape of LLMs remain obscure\. While existing literature attributes these gains to the static property of*local flatness*\(Izmailovet al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib17); Chenet al\.,[2017](https://arxiv.org/html/2605.26484#bib.bib19); Liet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib1)\), this perspective overlooks the dynamic geometry of the optimization trajectory itself\. It remains unclear how the merging process transforms the chaotic oscillation of raw gradients into a coherent path of descent\. We aims to bridge this gap by shifting the focus from the static loss landscape to the*trajectory geometry*, addressing a fundamental question:

What are the geometric properties of the manifold formed by merged checkpoints, and can we exploit this structure for futher enhancement?

Our contributions are summarized as follows:

- •Rank\-1 subspace phenomenon in merged trajectories\.We empirically show that late\-stage*merged*checkpoints concentrate on an approximately one\-dimensional linear manifold\.Concretely, while interpolation between adjacent*raw*checkpoints exhibits a convex\-basin profile \(midpoint loss lower than both endpoints\), interpolation between adjacent*merged*checkpoints becomes nearly monotone, indicating a smoother path of progress\. Through extensive PCA analysis, we demonstrate that thisRank\-1 Subspacecaptures the dominant direction of descent \(explaining\>94%\>94\\%of variance\), this contrasts sharply with raw trajectories,whose variance disperses over multiple components and whose projections are non\-monotone\.
- •Extra\-Merge Strategy\.Capitalizing on this geometric insight, we propose Extra\-Merge, a training\-free algorithm that estimates the subspace tangent to extrapolate progress without additional gradient updates\. We provide a rigorous theoretical justification under the*river\-valley*loss framework\(Wenet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib18)\), proving that averaging acts as a geometric low\-pass filter that aligns the trajectory with the optimal descent direction, thereby guaranteeing loss reduction via extrapolation\.
- •Universality across Scales and Optimizers\.Finally, We validate Extra\-Merge across a wide range of model scales \(GPT\-2/LLaMA, 124M to 2B\) and learning schedulers\. Crucially, our method yields consistent zero\-shot accuracy gains on Pythia\-12B\(Bidermanet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib67)\)downstream benchmarks and generalizes effectively to the orthogonal update regime of theMuon optimizer, demonstrating that the Rank\-1 Subspace is a robust, optimizer\-agnostic property of LLM training\.

#### Conflict of Interest Disclosure\.

The authors declare no financial conflicts of interest related to this work\.

## 2Related Works

#### Model Merging in Pre\-training\.

Model merging is a classic topic in machine learning\(Utans,[1996](https://arxiv.org/html/2605.26484#bib.bib20); Chenet al\.,[2017](https://arxiv.org/html/2605.26484#bib.bib19); Wortsmanet al\.,[2022](https://arxiv.org/html/2605.26484#bib.bib21); Yanget al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib44)\)\. In this paper, we specifically employ checkpoints saved during pre\-training and average these checkpoint parameters to improve performance without requiring substantial resources\. The modern resurgence of this paradigm in deep learning is largely attributed toStochastic Weight Averaging \(SWA\)\(Izmailovet al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib17)\)\. SWA demonstrated that averaging weights traversed by SGD with a cyclical or high constant learning rate leads to wider optima and better generalization than standard training\. More recently, these techniques have been adapted specifically for Large Language Models \(LLMs\)\.Latest Weight Averaging \(LAWA\)\(Kaddour,[2022](https://arxiv.org/html/2605.26484#bib.bib4); Sanyalet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib7)\)and Pre\-trained Model Averaging \(PMA\)\(Liet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib1)\)investigate the benefits of averaging late\-stage checkpoints in pre\-training\. Similarly, recent studies\(Tianet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib3); Liuet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib43); Luoet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib46)\)have integrated merging directly into the pre\-training loop\. Unlike prior works that view checkpoints as static points for interpolation, we interpret them as a dynamic sequence, allowing us to extrapolate beyond the convex hull of observed weights\. To ensure a comprehensive evaluation of novelty and distinctiveness, we provide a detailed taxonomic analysis of the model merging landscape in Appendix[A](https://arxiv.org/html/2605.26484#A1)\.

#### Geometry of the Loss Landscape\.

The geometry of the loss landscape is fundamental to understanding the success of deep neural networks\(Fort and Ganguli,[2019](https://arxiv.org/html/2605.26484#bib.bib40); Gur\-Ariet al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib41); Liet al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib42)\)\. Closely related to merging is the phenomenon ofLinear Mode Connectivity \(LMC\)\(Frankleet al\.,[2020](https://arxiv.org/html/2605.26484#bib.bib23); Gotmareet al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib34); Goodfellowet al\.,[2014](https://arxiv.org/html/2605.26484#bib.bib35); Vlaar and Frankle,[2022](https://arxiv.org/html/2605.26484#bib.bib38); Lucaset al\.,[2021](https://arxiv.org/html/2605.26484#bib.bib39)\), which establishes that multiple models trained independently on the same task can often be connected via a low\-loss linear path\. While LMC typically addresses the connectivity between converged and independent models, our work investigates the local geometry of the trajectoryduringthe late stages of a single pre\-training run\. Most relevant to our proposed Extra\-Merge is theRiver\-Valley landscapeperspective on optimization dynamics\(Cohenet al\.,[2021](https://arxiv.org/html/2605.26484#bib.bib31); Wenet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib18); Songet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib30); Daviset al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib72)\)\. These works characterize the neural network loss landscape as being composed of flat directions \(“rivers”\) and sharp directions \(“mountains”\), corresponding to Hessian eigendirections with distinct curvatures\. Furthermore, recent studies have leveraged this intuition to propose effective acceleration strategies for LLM pre\-training\(Wanget al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib11); Zhouet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib9); Wanget al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib32)\), providing empirical validation for the correctness of this perspective\.

## 3Preliminaries

#### Problem Setup\.

We consider the pre\-training of a Large Language Model \(LLM\) parameterized byθ∈ℝd\\theta\\in\\mathbb\{R\}^\{d\}\. The optimization process generates a trajectory of parameters\{θ0,θ1,…,θT\}\\\{\\theta\_\{0\},\\theta\_\{1\},\\dots,\\theta\_\{T\}\\\}by minimizing a loss functionℒ​\(θ\)\\mathcal\{L\}\(\\theta\)over a dataset, typically using optimizers such as AdamW\. We denote the checkpoint saved at training stepttasθt\\theta\_\{t\}\.

![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/val_loss_multi_curve.png)Figure 1:Comparison of merging strategies\.We compare the validation loss of the raw optimization trajectory against PMA \(Uniform\) and EMA\-based merging schemes \(γ∈\{0\.1,0\.2\}\\gamma\\in\\\{0\.1,0\.2\\\}\)\. Experiments are conducted on GPT\-2 Small \(124M\) trained on FineWeb \(τ=500,n=8\\tau=500,n=8\)\.![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/interp_small_per500.png)Figure 2:Geometric shift from Convexity to Monotonicity\.We visualize the loss landscape along the linear interpolation pathθ​\(α\)=\(1−α\)​θt\+α​θt\+τ\\theta\(\\alpha\)=\(1\-\\alpha\)\\theta\_\{t\}\+\\alpha\\theta\_\{t\+\\tau\}between consecutive checkpoints \(τ=500\\tau=500\)\.Left \(Raw\):The trajectory exhibits aconvex basinprofile, where the midpoint loss is significantly lower than the endpoints\.Right \(Merged\):The landscape transforms into amonotonic descent, under the merging strategy of \(τ=500,n=8\\tau=500,n=8\)\.
#### The LAWA Framework\.

Current research on model merging in pre\-training is mainly captured by theLatest Weight Averaging\(LAWA\) framework\(Kaddour,[2022](https://arxiv.org/html/2605.26484#bib.bib4); Sanyalet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib7)\)\. LAWA aggregates a sequence ofnncheckpoints sampled with a fixed intervalτ\\tau, computing a weighted average:

θavg\(t\)=∑i=0n−1wi⋅θt−i​τ,s\.t\.∑i=0n−1wi=1,\\theta\_\{\\text\{avg\}\}^\{\(t\)\}=\\sum\_\{i=0\}^\{n\-1\}w\_\{i\}\\cdot\\theta\_\{t\-i\\tau\},\\quad\\text\{s\.t\.\}\\quad\\sum\_\{i=0\}^\{n\-1\}w\_\{i\}=1,\(1\)wherewi=0n−1\{w\}\_\{i=0\}^\{n\-1\}are scalar weights determining the contribution of each checkpoint\. This formulation allows for various weighting schemes, such as Exponential Moving Average \(EMA\), whereθavg\(t\)=γ​θt\+\(1−γ\)​θavg\(t−τ\)\\theta\_\{\\text\{avg\}\}^\{\(t\)\}=\\gamma\\theta\_\{t\}\+\(1\-\\gamma\)\\theta\_\{\\text\{avg\}\}^\{\(t\-\\tau\)\}\.

#### Baseline: Pre\-trained Model Averaging \(PMA\)\.

While EMA is standard in convex optimization, recent work onPre\-trained Model Averaging\(PMA\)\(Liet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib1)\)provides compelling evidence that simpleUniform Averaging\(i\.e\.,wi=1/nw\_\{i\}=1/n\) is superior for LLM pre\-training\. PMA posits that in the high\-dimensional, non\-convex landscape of LLMs, uniform weights effectively approximate the centroid of the local loss basin, offering better stability than EMA\.

To rigorously establish our baseline, we validate this observation in Figure[1](https://arxiv.org/html/2605.26484#S3.F1)\. We compare the raw trajectory against LAWA instantiated with both Uniform weights \(PMA\) and EMA weights \(γ∈0\.1,0\.2\\gamma\\in\{0\.1,0\.2\}\) on GPT\-2 Small\. Consistent withLiet al\.\([2025](https://arxiv.org/html/2605.26484#bib.bib1)\), Uniform Averaging yields the lowest and most stable validation loss\. Consequently, we adopt PMA \(Uniform Averaging\) as the standard baseline throughout this work, denoted asθavg\(t\)=1n​∑i=0n−1θt−i​τ\\theta\_\{\\text\{avg\}\}^\{\(t\)\}=\\frac\{1\}\{n\}\\sum\_\{i=0\}^\{n\-1\}\\theta\_\{t\-i\\tau\}\.

## 4Main Findings

### 4\.1Warmup: Pairwise Connectivity

To unravel the geometric mechanism of model merging, we begin by probing the connectivity between*adjacent*checkpoints\. This serves as a fundamental test: if the optimization trajectory is smooth, linear interpolation between steps should yield monotonic improvement\. Conversely, non\-monotonicity implies curvature or oscillation\.

We analyze the validation lossℒ​\(θ​\(α\)\)\\mathcal\{L\}\(\\theta\(\\alpha\)\)along the linear pathθ​\(α\)=\(1−α\)​θt\+α​θt\+τ\\theta\(\\alpha\)=\(1\-\\alpha\)\\theta\_\{t\}\+\\alpha\\theta\_\{t\+\\tau\}forα∈\[0,1\]\\alpha\\in\[0,1\]\. Figure[2](https://arxiv.org/html/2605.26484#S3.F2)contrasts the landscape of the raw trajectory \(θraw​t→θraw​t\+τ\\theta^\{\\text\{raw\}\}t\\to\\theta^\{\\text\{raw\}\}\{t\+\\tau\}\) against the merged trajectory \(θavg​t→θavg​t\+τ\\theta^\{\\text\{avg\}\}t\\to\\theta^\{\\text\{avg\}\}\{t\+\\tau\}\) on GPT\-2 Small under the same setting of Figure[1](https://arxiv.org/html/2605.26484#S3.F1)\. Two distinct geometric regimes emerge:

- •Raw checkpoints exhibit Convex Basins\.As shown in Figure[2](https://arxiv.org/html/2605.26484#S3.F2)\(Left\), the interpolation between raw checkpoints consistently forms a U\-shaped convex basin\. Notably, the midpoint loss \(α≈0\.5\\alpha\\approx 0\.5\) is significantly lower than both endpoints\. This phenomenon suggests that consecutive raw steps are not moving strictly ”downhill” along the valley floor\. Instead, they oscillate across the high\-curvature walls of the loss valley, effectively ”bouncing” around the optimal path\.
- •Merged checkpoints reveal Monotonic Descent\.In sharp contrast, Figure[2](https://arxiv.org/html/2605.26484#S3.F2)\(Right\) reveals that the path between merged checkpoints is strictly monotonic\. The loss decreases smoothly fromtttot\+τt\+\\tauwithout the barrier or basin structure observed in the raw trajectory\. This suggests that the averaging operation acts as a geometric filter, effectively canceling out the orthogonal oscillations\.

![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/pca_explained_pc1_coord_grid.png)Figure 3:PCA analysis of raw and merged trajectories\.We perform PCA on sequences ofK=5K=5consecutive checkpoints for GPT\-2 and LLaMA models \(From 124M–2B\)\.Top \(Spectral Concentration\):Merged trajectories \(pink\) exhibit a dominant Rank\-1 structure, with the first principal component \(PC1\) explaining\>94%\>94\\%of the variance\. In contrast, raw trajectories \(blue\) disperse energy across multiple components\.Bottom \(Trajectory Rectification\):Projecting checkpoints onto PC1 reveals that merging rectifies the optimization path into a strictly monotonic flow, whereas raw steps exhibit non\-monotonic\. This confirms that merging collapses the trajectory onto a stable 1D linear manifold\.Implication\.The shift from local convexity to monotonicity indicates that merged checkpoints do not merely reside in a lower\-loss region; they lie on a geometrically distinct, smoother manifold\. This implies that while the raw trajectory is chaotic, the*merged*trajectory aligns with the monotonic direction, hinting at the existence of a stable, linear structure\. In the following section, we extend this pairwise analysis to multiple checkpoints to uncover the vaster geometry of this trajectory

### 4\.2Rank\-1 Subspace of Merged Models

To rigorously verify whether the local smoothness observed in Section[4\.1](https://arxiv.org/html/2605.26484#S4.SS1)translates to a global low\-dimensional manifold, we analyze the spectral geometry of trajectory segments\. Specifically, we collect sequences ofKKconsecutive checkpoints, form the centered parameter matrixΘ∈ℝd×K\\Theta\\in\\mathbb\{R\}^\{d\\times K\}, and perform Principal Component Analysis \(PCA\)\. We quantify the geometric structure using two metrics: the Explained Variance Ratio \(EVR\) of thekk\-th principal component \(Rk=σk2/∑jσj2R\_\{k\}=\\sigma\_\{k\}^\{2\}/\\sum\_\{j\}\\sigma\_\{j\}^\{2\}\) to measure linearity, and the Projected Coordinates along principal axes to assess monotonicity\. We conduct this analysis across GPT\-2 \(Small/Medium\) and LLaMA \(0\.5B/2B\) families, comparing raw trajectoriesθtraw\{\\theta^\{\\text\{raw\}\}\_\{t\}\}against merged onesθtavg​​\(τ=500,n=8\)\{\\theta^\{\\text\{avg\}\}\_\{t\}\\text\{ \}\(\\tau=500,n=8\)\}with a default segment length ofK=5K=5\.

Phenomenon 1: Spectral Concentration\.As shown in Figure[3](https://arxiv.org/html/2605.26484#S4.F3)\(Top\), merged checkpoints exhibit a dominant Rank\-1 structure, consistently concentrating\>94%\>94\\%of variance in the first principal component \(R1R\_\{1\}\) across all model scales\. In contrast, raw trajectories display a dispersed spectrum with significant energy distributed across other components\. This phenomenon indicates that averaging effectively suppresses orthogonal variance, isolating a single dominant axis of variation\.

Phenomenon 2: Trajectory Rectification\.Figure[3](https://arxiv.org/html/2605.26484#S4.F3)\(Bottom\) visualizes the evolution of normalized coordinates projected onto the first principal directionu1u\_\{1\}\. While raw checkpoints exhibit non\-monotonic oscillation, merged checkpoints evolve in a strictly monotonic, quasi\-linear fashion\. which implies that the extracted principal direction is not an artifact of noise but aligns with a coherent, directed path of descent\.

![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/PC1_EVR_vs_k.png)Figure 4:Robustness of the Rank\-1 Subspace\.We analyze the persistence of linearity \(PC1 Explained Variance Ratio\) as a function of the trajectory lengthKK\(number of checkpoints included in PCA\), varying the merging window sizennin GPT\-2 Small experiment\. Merged trajectories with sufficient window sizes \(n\>4n\>4\) maintain a robust linear structure \(E​V​R\>0\.8EVR\>0\.8\) even over long horizons in the late\-stage of training \(K=16K=16\)\.![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/final.png)\(a\)Geometric Intuition
![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/left_only.png)\(b\)Empirical Validation

Figure 5:\(a\)A visual illustration of the loss landscape\. While raw optimization steps \(blue trajectory\) oscillate violently across the high\-curvature valley walls, the merged checkpoints \(pink nodes\) filter out these oscillations, collapsing onto the stable valley floor \(the Rank\-1 Subspace\)\. Extra\-Merge \(dashed arrow\) identifies the tangent of this subspace to extrapolate towards the ”undiscovered” low\-loss region\.\(b\)Validation loss profile on GPT\-2 Small\. The green zone represents interpolation between merged checkpoints, while the gray zone denotes the extrapolation phase\. By extending along the principal direction with an adaptive step size, Extra\-Merge locates a minimum \(star\) significantly lower than the final merged checkpoint\.Phenomenon 3: Persistence over Horizons\.We further assess the stability of this structure by extending the trajectory lengthKK\. Figure[4](https://arxiv.org/html/2605.26484#S4.F4)reveals that while the linearity of raw trajectories \(n=1n=1\) decays rapidly withKK, merged trajectories with sufficient window sizes \(n\>4n\>4\) maintain robust linearity \(R1\>0\.8R\_\{1\}\>0\.8\) even over long horizons\. This suggests that with sufficient smoothing, the Rank\-1 Subspace acts as a stable, global backbone of the late\-stage training dynamics\.

Above findings highlight a fundamental geometric shift: model merging effectively suppresses the high\-frequency oscillations of the training trajectory, collapsing the optimization path onto aRank\-1 Subspace: a quasi\-linear 1D manifold that aligns with the primary direction of loss descent\.

Is Linearity Trivial under Smoothing?One might hypothesize that the observed Rank\-1 structure is a trivial artifact of the sliding window, which introduces temporal autocorrelation via data overlap\. However, smoothing does not inherently induce rank collapse\. Consider a null hypothesis where the trajectory is an isotropic random walk inℝd\\mathbb\{R\}^\{d\}: sliding window averaging acts as a scalar contraction on the covariance eigenvalues but preserves the effective rank \(i\.e\., the trajectory ”cloud” shrinks but remains spherical\)\. In our experiment Figure[3](https://arxiv.org/html/2605.26484#S4.F3), the emergence of a dominant principal component \(R1\>94%R\_\{1\}\>94\\%\) implies a latent linear drift\. We provide a formal justification for this phenomenon under theRiver\-Valleyloss landscape framework in Section[5](https://arxiv.org/html/2605.26484#S5)\.

## 5Extra\-Merge

Leveraging the geometric insight from Section[4\.1](https://arxiv.org/html/2605.26484#S4.SS1)where merged checkpoints concentrate on a linear manifold, we proposeExtra\-Merge, a training\-free algorithm designed to exploit this geometric stability\. Unlike standard merging which interpolateswithinthe convex hull of past checkpoints, Extra\-Merge extrapolates the optimization trajectory along the estimatedRank\-1 Subspaceto minimize loss\. Given a sequence of merged checkpointsθiavg\{\\theta^\{\\text\{avg\}\}\_\{i\}\}generated by a fixed PMA schedule \(intervalτ\\tau, windownn\), the procedure operates in two phases:

Step 1\. Extrapolation Direction Estimation\.We apply PCA to a sliding window of theKKmost recent merged checkpoints, denoted asΘK=\[θavg​t−K\+1,…,θavg​t\]\\Theta\_\{K\}=\[\\theta^\{\\text\{avg\}\}\{t\-K\+1\},\\dots,\\theta^\{\\text\{avg\}\}\{t\}\]\. We extract the first principal componentu1∈ℝdu\_\{1\}\\in\\mathbb\{R\}^\{d\}and orientu1u\_\{1\}to align with the temporal evolution of training\. We define the extrapolation directionv^=sgn​\(⟨θavg​t−θavg​t−1,u1⟩\)⋅u1\\hat\{v\}=\\text\{sgn\}\(\\langle\\theta^\{\\text\{avg\}\}\{t\}\-\\theta^\{\\text\{avg\}\}\{t\-1\},u\_\{1\}\\rangle\)\\cdot u\_\{1\}, ensuring the direction points towards descent\.

Step 2\. Adaptive Greedy Line Search\.Starting from the latest merged checkpointθavg​t\\theta^\{\\text\{avg\}\}\{t\}, we perform a line search alongv^\\hat\{v\}\. To avoid manual tuning of the search scale, we adopt anadaptive step sizestrategy based on the trajectory’s local velocity\. Letzi=θavg​i⋅v^z\_\{i\}=\\theta^\{\\text\{avg\}\}i\\cdot\\hat\{v\}be the projection coordinate\. We define the base strideδ\\deltaas a fractionα\\alphaof the coordinate displacement between the last two merged states:δ=α⋅\|zt−z​t−1\|\\delta=\\alpha\\cdot\|z\_\{t\}\-z\{t\-1\}\|, withα=0\.1\\alpha=0\.1as a robust default\. The extrapolation is defined iteratively:

θextra\(k\)=θtavg\+k⋅δ⋅v^,k=1,2,…\\theta\_\{\\text\{extra\}\}^\{\(k\)\}=\\theta^\{\\text\{avg\}\}\_\{t\}\+k\\cdot\\delta\\cdot\\hat\{v\},\\quad k=1,2,\\dots\(2\)We evaluate the validation lossℒ​\(θextra\(k\)\)\\mathcal\{L\}\(\\theta\_\{\\text\{extra\}\}^\{\(k\)\}\)at each step and terminate the search immediately when the loss fails to improve relative to the anchor point \(i\.e\.,ℒ​\(θextra\(k\)\)\>ℒ​\(θtavg\)\\mathcal\{L\}\(\\theta\_\{\\text\{extra\}\}^\{\(k\)\}\)\>\\mathcal\{L\}\(\\theta^\{\\text\{avg\}\}\_\{t\}\)\), returning the candidate with the minimal loss\.

Figure[5](https://arxiv.org/html/2605.26484#S4.F5)provides both a geometric interpretation and empirical verification of this procedure\. For Figure[5\(a\)](https://arxiv.org/html/2605.26484#S4.F5.sf1), while raw optimization steps oscillate across the sharp curvature of the loss landscape, merged checkpoints effectively collapse onto the valley floor, forming a stable Rank\-1 Subspace\. Extra\-Merge exploits this stability to extend the trajectory into the “undiscovered” region\. Empirically \(Figure[5\(b\)](https://arxiv.org/html/2605.26484#S4.F5.sf2)\), we validate this behavior on GPT\-2 Small \(τ=500,n=8\\tau=500,n=8\) and estimating the subspace tangent from the lastK=4K=4merged checkpoints\. The resulting loss curve exhibits a smooth continuation from the interpolation phase \(green zone\) to the extrapolation phase \(gray zone\), successfully locating a significantly lower\-loss solution without additional gradient updates\.

### 5\.1Theoretical Insight

In this section, we formalize the geometric intuition behind theRank\-1 Subspaceand provide a theoretical justification for Extra\-Merge\. We demonstrate the underlying mechanism within the*River\-Valley*framework\(Wenet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib18)\), begin by defining the local geometry of late\-stage pre\-training\. Throughout this section,NNcorresponds to the merging window sizennin PMA/LAWA, andTTcorresponds to the checkpoint intervalτ\\tauused in practice\. “River” refers to the \(approximately\) flattest direction, and “Mountains” refer to the remaining high\-curvature subspace\.

###### Assumption 5\.1\(River\-Valley Landscape \(Assumption 2 in\(Wenet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib18)\)\); Informal version of Assumptions[C\.1](https://arxiv.org/html/2605.26484#A3.Thmtheorem1)and[C\.4](https://arxiv.org/html/2605.26484#A3.Thmtheorem4)\)\.

Consider a local neighborhoodUUaround the optimization path\. We assume the lossℒ​\(θ\)\\mathcal\{L\}\(\\theta\)decomposes into a 1D*flat*direction \(the “river”\) and a\(d−1\)\(d\-1\)\-dimensional*sharp*subspace \(the “mountains”\)\. Formally, letv∈ℝdv\\in\\mathbb\{R\}^\{d\}be the unit vector along the river, with associated projection matricesPF=v​v⊤P\_\{F\}=vv^\{\\top\}andPS=I−v​v⊤P\_\{S\}=I\-vv^\{\\top\}\. For anyθ∈U\\theta\\in U, the loss is approximated as:

ℒ​\(θ\)≈ℓ​\(v⊤​\(θ−θ⋆\)\)\+12​‖H1/2​PS​\(θ−θ⋆\)‖22,\\mathcal\{L\}\(\\theta\)\\approx\\ell\\\!\\left\(v^\{\\top\}\(\\theta\-\\theta\_\{\\star\}\)\\right\)\+\\frac\{1\}\{2\}\\,\\\|H^\{1/2\}P\_\{S\}\(\\theta\-\\theta\_\{\\star\}\)\\\|\_\{2\}^\{2\},\(3\)whereθ⋆\\theta\_\{\\star\}is a reference point on the river,ℓ​\(⋅\)\\ell\(\\cdot\)is a slowly varying function alongvv, andHHis a positive semi\-definite Hessian matrix supported on the mountain subspace \(i\.e\.,H=PS​H​PSH=P\_\{S\}HP\_\{S\}\) and satisfiesz⊤​H​z∈\[μ​‖z‖22,LS​‖z‖22\]z^\{\\top\}Hz\\in\[\\mu\\\|z\\\|\_\{2\}^\{2\},\\,L\_\{S\}\\\|z\\\|\_\{2\}^\{2\}\]for allz∈range​\(PS\)z\\in\\mathrm\{range\}\(P\_\{S\}\)\.

Intuitively, Assumption[5\.1](https://arxiv.org/html/2605.26484#S5.Thmtheorem1)posits that the loss landscape resembles a narrow valley\. The gradient aligns with the river directionvvonly on the valley floor \(the manifoldMM\); elsewhere, it is dominated by steep componentsH​PS​\(θ−θ⋆\)HP\_\{S\}\(\\theta\-\\theta\_\{\\star\}\)\. Figure[5\(a\)](https://arxiv.org/html/2605.26484#S4.F5.sf1)is a 3\-D illustration of this landscape\. Next we model the late\-stage behavior of Stochastic Gradient Descent \(SGD\) within this landscape\.

###### Assumption 5\.2\(Decoupled late\-stage dynamics; Informal version of Assumptions[C\.3](https://arxiv.org/html/2605.26484#A3.Thmtheorem3),[C\.4](https://arxiv.org/html/2605.26484#A3.Thmtheorem4)\)\.

Letv∈ℝdv\\in\\mathbb\{R\}^\{d\}be the \(locally constant\) river direction, Write the river and mountain coordinates astk:=v⊤​\(θk−θ⋆\)t\_\{k\}:=v^\{\\top\}\(\\theta\_\{k\}\-\\theta\_\{\\star\}\)andzk:=PS​\(θk−θ⋆\)z\_\{k\}:=P\_\{S\}\(\\theta\_\{k\}\-\\theta\_\{\\star\}\)\. Assume the late\-stage dynamics on a neighborhoodUUcan be approximated by

tk\+1=tk−η​\(ℓ′​\(tk\)\+μF\),zk\+1=\(I−η​H\)​zk\+η​gk,t\_\{k\+1\}=t\_\{k\}\-\\eta\\big\(\\ell^\{\\prime\}\(t\_\{k\}\)\+\\mu\_\{F\}\\big\),\\quad z\_\{k\+1\}=\(I\-\\eta H\)z\_\{k\}\+\\eta g\_\{k\},whereη∈\(0,1/LS\)\\eta\\in\(0,1/L\_\{S\}\),H⪰0H\\succeq 0is supported onrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)with eigenvalues in\[μ,LS\]\[\\mu,L\_\{S\}\], andgkg\_\{k\}are i\.i\.d\. zero\-mean noise supported onrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)withCov​\(gk\)=σ2​PS\\mathrm\{Cov\}\(g\_\{k\}\)=\\sigma^\{2\}P\_\{S\}\(e\.g\.,gk∼𝒩​\(0,σ2​PS\)g\_\{k\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}P\_\{S\}\)\)\.

We now analyze the properties of merged checkpoints under these dynamics\. Letθavg=1N​∑m=0N−1θt0\+m​T\\theta^\{\\text\{avg\}\}=\\frac\{1\}\{N\}\\sum\_\{m=0\}^\{N\-1\}\\theta\_\{t\_\{0\}\+mT\}be the merged parameter obtained by averagingNNcheckpoints sampled everyTTsteps\. Our first result characterizes how this averaging operation interacts with the landscape geometry\.

![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/GPT2small-Result.png)
![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/GPT2medium-Result.png)
![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/GPT2XL-Result.png)
![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/05Bllama-Result.png)
![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/2Bllama-Result.png)

Figure 6:Validation loss trajectories across model scales and schedules\.We compare Raw, PMA , and Extra\-Merge on \(First row\) GPT\-2 family trained with WSD scheduler, and \(Last row\) LLaMA family trained with Cosine scheduler\. While PMA gradually aligns with the raw baseline as the learning rate decays, Extra\-Merge achieve consistently lower loss\.Table 1:Zero\-shot performance on Pythia\-12B downstream tasks\.We compare the raw checkpoint \(at step 140k\) against various merging strategies using checkpoints from 130k to 140k\. Extra\-Merge consistently outperforms both the raw baseline and standard averaging methods \(PMA/EMA\) across all four benchmarks\.###### Theorem 5\.3\(Averaging reduces expected distance to the river\)\.

Let checkpoints be saved everyTTsteps and letθavg\\theta^\{\\text\{avg\}\}be the uniform average ofNNlate\-stage checkpoints\. Define the distance to the river asD​\(θ\):=‖PS​\(θ−θ⋆\)‖2D\(\\theta\):=\\\|P\_\{S\}\(\\theta\-\\theta\_\{\\star\}\)\\\|\_\{2\}\. Then

𝔼​D​\(θ¯\)2≤1N​\(1\+2​ε1−ε\)​∑j=1d−1η​σ22​λj−η​λj2,\\mathbb\{E\}\\,D\(\\bar\{\\theta\}\)^\{2\}\\;\\leq\\;\\frac\{1\}\{N\}\\Big\(1\+\\frac\{2\\varepsilon\}\{1\-\\varepsilon\}\\Big\)\\sum\_\{j=1\}^\{d\-1\}\\frac\{\\eta\\sigma^\{2\}\}\{2\\lambda\_\{j\}\-\\eta\\lambda\_\{j\}^\{2\}\},whereε:=maxj⁡\|1−η​λj\|T\\varepsilon:=\\max\_\{j\}\|1\-\\eta\\lambda\_\{j\}\|^\{T\},\{λj\}j=1d−1\\\{\\lambda\_\{j\}\\\}\_\{j=1\}^\{d\-1\}are eigenvalues ofHHonrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)\. In particular, whenTTis large enough such thatε≪1\\varepsilon\\ll 1, the mountain deviation decays as𝒪~​\(1/N\)\\tilde\{\\mathcal\{O\}\}\(1/N\)\.

Proof in Appendix[C\.2](https://arxiv.org/html/2605.26484#A3.SS2)\. Theorem[5\.3](https://arxiv.org/html/2605.26484#S5.Thmtheorem3)provides the justification for the ”Rank\-1 Subspace” phenomenon observed in Figure[3](https://arxiv.org/html/2605.26484#S4.F3)\. Individual checkpointsθt\\theta\_\{t\}are displaced from the valley floor by a constant noise floor proportional toη​σ2/λmin\\eta\\sigma^\{2\}/\\lambda\_\{\\min\}\. In contrast, the averaging process suppresses this orthogonal oscillation by a factor of1/N1/N\. AsNNincreases,θavg\\theta^\{\\text\{avg\}\}collapses onto the river manifoldMM\.

Next, we address the validity of Extra\-Merge\. We analyze the sliding\-window merging process, where we collect a sequence ofKKmerged checkpoints\{θsavg\}s=1K\\\{\\theta^\{\\text\{avg\}\}\_\{s\}\\\}\_\{s=1\}^\{K\}\. We perform PCA on these centered states to derive the extrapolation directionu^1\\hat\{u\}\_\{1\}\.

###### Theorem 5\.4\(PCA Recovers the Descent Direction\)\.

Letu^1\\hat\{u\}\_\{1\}be the first principal component of a sequence ofKKsliding\-window merged checkpoints\. Define the Signal\-to\-Noise Ratio \(SNR\) of the trajectory asρ:=σdrift2/σresid2\\rho:=\\sigma\_\{\\text\{drift\}\}^\{2\}/\\sigma\_\{\\text\{resid\}\}^\{2\}, whereσdrift2\\sigma\_\{\\text\{drift\}\}^\{2\}is the variance due to river descent andσresid2\\sigma\_\{\\text\{resid\}\}^\{2\}is the residual mountain noise after averaging\. Ifρ\>1\\rho\>1, thenu^1\\hat\{u\}\_\{1\}aligns with the true river directionvv:

sin⁡\(∠​\(u^1,v\)\)≤Cρ−1,\\sin\(\\angle\(\\hat\{u\}\_\{1\},v\)\)\\leq\\frac\{C\}\{\\rho\-1\},\(4\)whereCCis a constant determined by the residual noise level and standard PCA estimation stability\.

Proof in Appendix[C\.3](https://arxiv.org/html/2605.26484#A3.SS3)\. Theorem[5\.4](https://arxiv.org/html/2605.26484#S5.Thmtheorem4)is pivotal for Extra\-Merge\. It ensures that the PCA direction extracted from merged checkpoints is not a noise artifact: it recovers the tangent of the low\-curvature subspace \(the river directionvv\) once the drift\-to\-noise ratioρ\\rhois above11\. Therefore, performing a 1D line search alongu^1\\hat\{u\}\_\{1\}acts as a training\-free descent step on the rectified \(merged\) trajectory\.

#### Theoretical Implications for Hyperparameters\.

To make the conditionρ\>1\\rho\>1concrete, we derive explicit bounds for the signal and noise terms in Lemmas[C\.9](https://arxiv.org/html/2605.26484#A3.Thmtheorem9)and[C\.8](https://arxiv.org/html/2605.26484#A3.Thmtheorem8)\(due to space limit, we defer the theoretical result in Appendix[C](https://arxiv.org/html/2605.26484#A3)\)\. In particular, the residual variance decreases roughly as1/N1/N, while the drift signal grows with the temporal spanK​τK\\tau\. As a result, the effective SNR scales \(up to constants and mild correlation effects inτ\\tau\) asρ∝N⋅\(K​τ\)2\\rho\\;\\propto\\;N\\cdot\(K\\tau\)^\{2\}\. This gives the following guidance:

- •Window SizeNN\(Noise Suppression\):largerNNlowers the residual mountain noise floor \(approximately1/N1/N\), making PCA less likely to lock onto noise, increasingNNcan be helpful once the model is in a late training stage\.
- •PCA WindowKK\(Signal Amplification\):increasingKKamplifies the cumulative drift along the river, improving identifiability of the subspace direction\.
- •Intervalτ\\tau\(Span & Decorrelation\):a largerτ\\taureduces temporal correlation in the mountain dynamics, improving the effectiveness of averaging \(Theorem[5\.3](https://arxiv.org/html/2605.26484#S5.Thmtheorem3)\)\.
- •Trade\-off:KK\(andK​τK\\tau\) should remain moderate so the local “straight\-river” approximation stays valid; then we useK=4K=4in Figure[5\(b\)](https://arxiv.org/html/2605.26484#S4.F5.sf2)for balance\.

## 6Experiments

### 6\.1Experiment Settings

Models and Datasets\.We conduct pre\-training experiments on two widely adopted decoder\-only families:

- •GPT\-2 on FineWeb\.We train GPT\-2 Small \(124M\), Medium \(355M\), and XL \(1\.55B\) on the FineWeb dataset\. This setup follows the highly optimized training recipe used in the nano\-gpt benchmark111[https://github\.com/KellerJordan/modded\-nanogpt](https://github.com/KellerJordan/modded-nanogpt), representing a rigorous standard for small\-to\-medium scale pre\-training\.
- •LLaMA on C4\.To verify scalability, we train LLaMA architectures with 0\.5B and 2B parameters on the Colossal Clean Crawled Corpus \(C4\) dataset\.

Training Protocol\.All models are trained using the AdamW optimizer with hyperparametersβ1=0\.9,β2=0\.95\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.95, and weight decayλ=0\.1\\lambda=0\.1\. To ensure our method is agnostic to the learning rate \(LR\) schedule, we employ two distinct strategies: the WSD \(Warmup\-Stable\-Decay\) scheduler for GPT\-2, and Cosine decay for LLaMA\. This diversity confirms that the observed improvements are intrinsic to the geometry of merging\. Detailed hyperparameters for all runs are provided in Appendix[B\.3](https://arxiv.org/html/2605.26484#A2.SS3)\.

Merging Baselines and Extra\-Merge Configuration\.We adopt averagePre\-trained Model Averaging\(PMA\) as our primary baseline\. To ensure a competitive comparison, we tune the merging intervalτ\\tauand window sizenn, settingτ=500,n=8\\tau=500,n=8for GPT\-2 Small/Medium, andτ=1000,n=10\\tau=1000,n=10for the larger GPT\-2 XL and LLaMA models, details of the merge settings can be found in Appendix[B](https://arxiv.org/html/2605.26484#A2)\. We report results on the latter half of the training trajectory, where merging becomes effective\. For Extra\-Merge, we apply PCA over a sliding window ofK=4K=4consecutive checkpoints\.

### 6\.2Main Results: Pre\-training Validation Loss

We report the validation loss trajectories for both GPT\-2 and LLaMA families in Figure[1](https://arxiv.org/html/2605.26484#S5.T1), covering model scales from 124M to 2B\. As illustrated, while PMA yields lower loss than the raw model during the high\-learning\-rate phase, its performancegradually aligns withthe raw baseline as the learning rate decays\. In contrast,Extra\-Mergeachieves a sustained loss reduction across all settings\. By extrapolating along the optimization trajectory, our method consistently reaches lower loss values at equivalent training steps, effectively accelerating convergence without additional gradient updates\. This represents afree\-lunchgain, improving upon the raw baseline compared to PMA\.

### 6\.3Scaling to Downstream Tasks: Pythia\-12B

To assess the scalability and generalization capability of our method on larger\-scale models and real\-world tasks, we extend our evaluation to thePythia\-12Bmodel\. We utilize the official intermediate checkpoints222[https://huggingface\.co/EleutherAI/pythia\-12b](https://huggingface.co/EleutherAI/pythia-12b)ranging from step 130k to 140k, with a saving interval of 1k steps\.

We compare Extra\-Merge against the Raw baseline and PMA variants \(Uniform and EMA\) on four standard reasoning benchmarks: ARC\-Challenge, ARC\-Easy, HellaSwag, and PIQA\. For Extra\-Merge, we apply extrapolation along the principal direction extracted from theK=4K=4merged checkpoints\(τ=1000,N=10\\tau=1000,N=10\), and using average score as the indicator\.We report the accuracy in Table[1](https://arxiv.org/html/2605.26484#S5.T1)\.

Results\.As detailed in Table[1](https://arxiv.org/html/2605.26484#S5.T1), we observe that among standard merging strategies, PMA with uniform averaging generally yields the most robust improvements, outperforming EMA variants\. Extra\-Merge achieves an average accuracy gain of\+0\.59%over the raw baseline and\+0\.40%over the best PMA setting, confirming its efficacy on downstream tasks beyond simple perplexity reduction\.

### 6\.4Extra\-Merge on Muon Optimizer

To demonstrate the universality of Extra\-Merge across different optimizers, we extend our evaluation to theMuonoptimizer\(Jordanet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib28)\)\. Unlike AdamW, Muon is characterized by its use of orthogonal updates, which induces a fundamentally different trajectory structure during training and are now widely adopted\.

![Refer to caption](https://arxiv.org/html/2605.26484v1/figs/GPT2small-Result-Muon.png)Figure 7:Validation loss trajectory on FineWeb using the Muon optimizer\.We compare the Raw baseline, PMA, and Extra\-Merge on GPT\-2 Small\. Extra\-Merge consistently outperforms PMA\.We follow the experiment setting of the standardmodded nano\-gptconfiguration\. We maintain the identical merging protocol used in the AdamW experiments to ensure consistency\. Detailed hyperparameters and experimental settings are provided in Appendix[B\.4](https://arxiv.org/html/2605.26484#A2.SS4)\. As illustrated in Figure[7](https://arxiv.org/html/2605.26484#S6.F7), our method remains highly effective under Muon optimization\. This confirms that the acceleration benefits of our method are robust to the underlying optimizer\.

## 7Conclusion and Future Works

In this work, we identified theRank\-1 Subspace, a phenomenon where chaotic training trajectories collapse onto a stable 1D manifold upon averaging\. Leveraging this, we proposedExtra\-Merge, a training\-free extrapolation strategy that consistently improves baseline merging performance across diverse architectures \(GPT\-2, LLaMA, Pythia\) and optimizers \(AdamW, Muon\)\.

This work offers a novel geometric perspective on the linearity of LLM loss landscape\. Future research could explore two key directions: \(1\) developing subspace\-aware optimization strategies that actively align updates with this latent manifold to accelerate convergence, and \(2\) analyzing how this subspace reorients under different data distributions, which could unlock geometric approaches for efficient domain adaptation and transfer learning\.

## Impact Statement

This paper contributes to advancing the field of deep learning, with a specific focus on understanding the training dynamics of Large Language Models \(LLMs\) and improving optimization efficiency through model merging techniques\. Our research utilized a computational setup consisting of 8 NVIDIA RTX 4090 GPUs and 2 NVIDIA H100 GPUs, involving approximately 100 hours of training and inference time\. By detailing these experiments, we hope to provide the community with new insights into the geometry of loss landscapes and the efficacy of merging strategies\. As this work focuses on fundamental optimization methodologies, we do not foresee immediate negative societal impacts; rather, we aim to facilitate the development of more efficient and accessible AI systems\.

## Acknowledgements

This work was funded by the Strategic Priority Research Program of the Chinese Academy of Sciences \(Grant No\. XDB0680101\), CAS Project for Young Scientists in Basic Research under Grant No\. YSBR\-034, the National Key Research and Development Program of China under Grants No\. 2023YFA1011602, and Xiaomi Young Talents Program\.

## References

- T\. Akiba, M\. Shing, Y\. Tang, Q\. Sun, and D\. Ha \(2025\)Evolutionary optimization of model merging recipes\.Nature Machine Intelligence7\(2\),pp\. 195–204\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational Conference on Machine Learning,pp\. 2397–2430\.Cited by:[§B\.3](https://arxiv.org/html/2605.26484#A2.SS3.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2605.26484#S1.I1.i3.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§B\.3](https://arxiv.org/html/2605.26484#A2.SS3.SSS0.Px3.p1.1)\.
- H\. Chen, S\. Lundberg, and S\. Lee \(2017\)Checkpoint ensembles: ensemble methods from a single training process\.arXiv preprint arXiv:1710\.03282\.Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p3.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§B\.3](https://arxiv.org/html/2605.26484#A2.SS3.SSS0.Px3.p1.1)\.
- J\. M\. Cohen, S\. Kaur, Y\. Li, J\. Z\. Kolter, and A\. Talwalkar \(2021\)Gradient descent on neural networks typically occurs at the edge of stability\.arXiv preprint arXiv:2103\.00065\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p1.1)\.
- D\. Davis, D\. Drusvyatskiy, and L\. Jiang \(2025\)Gradient descent with adaptive stepsize converges \(nearly\) linearly under fourth\-order growth\.External Links:2409\.19791,[Link](https://arxiv.org/abs/2409.19791)Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Fort and S\. Ganguli \(2019\)Emergent properties of the local geometry of neural loss landscapes\.arXiv preprint arXiv:1910\.05929\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Frankle, G\. K\. Dziugaite, D\. Roy, and M\. Carbin \(2020\)Linear mode connectivity and the lottery ticket hypothesis\.InInternational Conference on Machine Learning,pp\. 3259–3269\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2023\)A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§B\.3](https://arxiv.org/html/2605.26484#A2.SS3.SSS0.Px3.p1.1)\.
- I\. J\. Goodfellow, O\. Vinyals, and A\. M\. Saxe \(2014\)Qualitatively characterizing neural network optimization problems\.arXiv preprint arXiv:1412\.6544\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Gotmare, N\. S\. Keskar, C\. Xiong, and R\. Socher \(2018\)Using mode connectivity for loss landscape analysis\.External Links:1806\.06977,[Link](https://arxiv.org/abs/1806.06977)Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Gur\-Ari, D\. A\. Roberts, and E\. Dyer \(2018\)Gradient descent happens in a tiny subspace\.arXiv preprint arXiv:1812\.04754\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p1.1)\.
- S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui, X\. Long, Z\. Zheng, Y\. Fang, Y\. Huang, W\. Zhao, X\. Zhang, Z\. L\. Thai, K\. Zhang, C\. Wang, Y\. Yao, C\. Zhao, J\. Zhou, J\. Cai, Z\. Zhai, N\. Ding, C\. Jia, G\. Zeng, D\. Li, Z\. Liu, and M\. Sun \(2024\)MiniCPM: unveiling the potential of small language models with scalable training strategies\.External Links:2404\.06395,[Link](https://arxiv.org/abs/2404.06395)Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p1.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- P\. Izmailov, D\. Podoprikhin, T\. Garipov, D\. Vetrov, and A\. G\. Wilson \(2018\)Averaging weights leads to wider optima and better generalization\.arXiv preprint arXiv:1803\.05407\.Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p2.1),[§1](https://arxiv.org/html/2605.26484#S1.p3.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Jin, X\. Ren, D\. Preotiuc\-Pietro, and P\. Cheng \(2022\)Dataless knowledge fusion by merging weights of language models\.arXiv preprint arXiv:2212\.09849\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024\)Muon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§B\.4](https://arxiv.org/html/2605.26484#A2.SS4.SSS0.Px1.p1.1),[§6\.4](https://arxiv.org/html/2605.26484#S6.SS4.p1.1)\.
- J\. Kaddour \(2022\)Stop wasting my time\! saving days of imagenet and bert training with latest weight averaging\.arXiv preprint arXiv:2209\.14981\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p2.1),[§1](https://arxiv.org/html/2605.26484#S1.p2.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.26484#S3.SS0.SSS0.Px2.p1.2)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p1.1)\.
- C\. Li, H\. Farkhoor, R\. Liu, and J\. Yosinski \(2018\)Measuring the intrinsic dimension of objective landscapes\.arXiv preprint arXiv:1804\.08838\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, Y\. Ma, S\. Yan, C\. Zhang, J\. Liu, J\. Lu, Z\. Xu, M\. Chen, M\. Wang, S\. Zhan,et al\.\(2025\)Model merging in pre\-training of large language models\.arXiv preprint arXiv:2505\.12082\.Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p2.1),[§1](https://arxiv.org/html/2605.26484#S1.p3.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.26484#S3.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2605.26484#S3.SS0.SSS0.Px3.p2.2)\.
- D\. Liu, Z\. Wang, B\. Wang, W\. Chen, C\. Li, Z\. Tu, D\. Chu, B\. Li, and D\. Sui \(2024\)Checkpoint merging via bayesian optimization in llm pretraining\.arXiv preprint arXiv:2403\.19390\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p2.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.\(2025\)Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§B\.4](https://arxiv.org/html/2605.26484#A2.SS4.SSS0.Px1.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2017\)Decoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§B\.1](https://arxiv.org/html/2605.26484#A2.SS1.SSS0.Px4.p1.6)\.
- J\. Lucas, J\. Bae, M\. R\. Zhang, S\. Fort, R\. Zemel, and R\. Grosse \(2021\)Analyzing monotonic linear interpolation in neural network loss landscapes\.arXiv preprint arXiv:2104\.110441\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Luo, Z\. Sun, H\. Wen, X\. Shi, J\. Cui, C\. Dang, K\. Lyu, and W\. Chen \(2025\)How learning rate decay wastes your best data in curriculum\-based llm pretraining\.arXiv preprint arXiv:2511\.18903\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- M\. S\. Matena and C\. A\. Raffel \(2022\)Merging models with fisher\-weighted averaging\.Advances in Neural Information Processing Systems35,pp\. 17703–17716\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, J\. Jang, A\. Jiang, R\. Jiang, H\. Jin, D\. Jin, S\. Jomoto, B\. Jonn, H\. Jun, T\. Kaftan, Ł\. Kaiser, A\. Kamali, I\. Kanitscheider, N\. S\. Keskar, T\. Khan, L\. Kilpatrick, J\. W\. Kim, C\. Kim, Y\. Kim, J\. H\. Kirchner, J\. Kiros, M\. Knight, D\. Kokotajlo, Ł\. Kondraciuk, A\. Kondrich, A\. Konstantinidis, K\. Kosic, G\. Krueger, V\. Kuo, M\. Lampe, I\. Lan, T\. Lee, J\. Leike, J\. Leung, D\. Levy, C\. M\. Li, R\. Lim, M\. Lin, S\. Lin, M\. Litwin, T\. Lopez, R\. Lowe, P\. Lue, A\. Makanju, K\. Malfacini, S\. Manning, T\. Markov, Y\. Markovski, B\. Martin, K\. Mayer, A\. Mayne, B\. McGrew, S\. M\. McKinney, C\. McLeavey, P\. McMillan, J\. McNeil, D\. Medina, A\. Mehta, J\. Menick, L\. Metz, A\. Mishchenko, P\. Mishkin, V\. Monaco, E\. Morikawa, D\. Mossing, T\. Mu, M\. Murati, O\. Murk, D\. Mély, A\. Nair, R\. Nakano, R\. Nayak, A\. Neelakantan, R\. Ngo, H\. Noh, L\. Ouyang, C\. O’Keefe, J\. Pachocki, A\. Paino, J\. Palermo, A\. Pantuliano, G\. Parascandolo, J\. Parish, E\. Parparita, A\. Passos, M\. Pavlov, A\. Peng, A\. Perelman, F\. de Avila Belbute Peres, M\. Petrov, H\. P\. de Oliveira Pinto, Michael, Pokorny, M\. Pokrass, V\. H\. Pong, T\. Powell, A\. Power, B\. Power, E\. Proehl, R\. Puri, A\. Radford, J\. Rae, A\. Ramesh, C\. Raymond, F\. Real, K\. Rimbach, C\. Ross, B\. Rotsted, H\. Roussez, N\. Ryder, M\. Saltarelli, T\. Sanders, S\. Santurkar, G\. Sastry, H\. Schmidt, D\. Schnurr, J\. Schulman, D\. Selsam, K\. Sheppard, T\. Sherbakov, J\. Shieh, S\. Shoker, P\. Shyam, S\. Sidor, E\. Sigler, M\. Simens, J\. Sitkin, K\. Slama, I\. Sohl, B\. Sokolowsky, Y\. Song, N\. Staudacher, F\. P\. Such, N\. Summers, I\. Sutskever, J\. Tang, N\. Tezak, M\. B\. Thompson, P\. Tillet, A\. Tootoonchian, E\. Tseng, P\. Tuggle, N\. Turley, J\. Tworek, J\. F\. C\. Uribe, A\. Vallone, A\. Vijayvergiya, C\. Voss, C\. Wainwright, J\. J\. Wang, A\. Wang, B\. Wang, J\. Ward, J\. Wei, C\. Weinmann, A\. Welihinda, P\. Welinder, J\. Weng, L\. Weng, M\. Wiethoff, D\. Willner, C\. Winter, S\. Wolrich, H\. Wong, L\. Workman, S\. Wu, J\. Wu, M\. Wu, K\. Xiao, T\. Xu, S\. Yoo, K\. Yu, Q\. Yuan, W\. Zaremba, R\. Zellers, C\. Zhang, M\. Zhang, S\. Zhao, T\. Zheng, J\. Zhuang, W\. Zhuk, and B\. Zoph \(2024\)GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2605.26484#S1.p1.1)\.
- G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. A\. Raffel, L\. Von Werra, T\. Wolf,et al\.\(2024\)The fineweb datasets: decanting the web for the finest text data at scale\.Advances in Neural Information Processing Systems37,pp\. 30811–30849\.Cited by:[§B\.1](https://arxiv.org/html/2605.26484#A2.SS1.SSS0.Px2.p1.1)\.
- B\. T\. Polyak and A\. B\. Juditsky \(1992\)Acceleration of stochastic approximation by averaging\.SIAM journal on control and optimization30\(4\),pp\. 838–855\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p2.1),[§1](https://arxiv.org/html/2605.26484#S1.p2.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.OpenAI\.Note:Accessed: 2024\-11\-15External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§B\.1](https://arxiv.org/html/2605.26484#A2.SS1.SSS0.Px1.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§B\.2](https://arxiv.org/html/2605.26484#A2.SS2.SSS0.Px2.p1.2)\.
- A\. Ramé, K\. Ahuja, J\. Zhang, M\. Cord, L\. Bottou, and D\. Lopez\-Paz \(2023\)Model ratatouille: recycling diverse models for out\-of\-distribution generalization\.InInternational Conference on Machine Learning,pp\. 28656–28679\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- S\. Sanyal, A\. Neerkaje, J\. Kaddour, A\. Kumar, and S\. Sanghavi \(2023\)Early weight averaging meets high learning rates for llm pre\-training\.arXiv preprint arXiv:2306\.03241\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p2.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.26484#S3.SS0.SSS0.Px2.p1.2)\.
- M\. Song, K\. Ahn, and C\. Yun \(2024\)Does sgd really happen in tiny subspaces?\.arXiv preprint arXiv:2405\.16002\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu \(2023\)RoFormer: enhanced transformer with rotary position embedding\.External Links:2104\.09864,[Link](https://arxiv.org/abs/2104.09864)Cited by:[§B\.2](https://arxiv.org/html/2605.26484#A2.SS2.SSS0.Px1.p1.1)\.
- C\. Tian, J\. Wang, Q\. Zhao, K\. Chen, J\. Liu, Z\. Liu, J\. Mao, W\. X\. Zhao, Z\. Zhang, and J\. Zhou \(2025\)WSM: decay\-free learning rate schedule via checkpoint merging for llm pre\-training\.arXiv preprint arXiv:2507\.17634\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p2.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§B\.2](https://arxiv.org/html/2605.26484#A2.SS2.SSS0.Px1.p1.1)\.
- J\. Utans \(1996\)Weight averaging for neural networks and local resampling schemes\.InProc\. AAAI\-96 Workshop on Integrating Multiple Learned Models\. AAAI Press,pp\. 133–138\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§B\.1](https://arxiv.org/html/2605.26484#A2.SS1.SSS0.Px1.p1.1)\.
- T\. J\. Vlaar and J\. Frankle \(2022\)What can linear interpolation of neural network loss landscapes tell us?\.InInternational Conference on Machine Learning,pp\. 22325–22341\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Wang, M\. Wang, Z\. Zhou, J\. Yan, L\. Wu,et al\.\(2025\)The sharpness disparity principle in transformers for accelerating language model pre\-training\.arXiv preprint arXiv:2502\.19002\.Cited by:[§B\.2](https://arxiv.org/html/2605.26484#A2.SS2.SSS0.Px3.p1.5),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Wang, J\. Wang, H\. He, Z\. Wang, G\. Huang, F\. Xiong, L\. Wu,et al\.\(2024\)Improving generalization and convergence by enhancing implicit regularization\.Advances in Neural Information Processing Systems37,pp\. 118701–118744\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Wen, Z\. Li, J\. Wang, D\. Hall, P\. Liang, and T\. Ma \(2024\)Understanding warmup\-stable\-decay learning rates: a river valley loss landscape perspective\.arXiv preprint arXiv:2410\.05192\.Cited by:[§C\.1](https://arxiv.org/html/2605.26484#A3.SS1.p1.1),[Assumption C\.1](https://arxiv.org/html/2605.26484#A3.Thmtheorem1),[2nd item](https://arxiv.org/html/2605.26484#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2605.26484#S5.SS1.p1.4),[Assumption 5\.1](https://arxiv.org/html/2605.26484#S5.Thmtheorem1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz,et al\.\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations,pp\. 38–45\.Cited by:[§B\.2](https://arxiv.org/html/2605.26484#A2.SS2.SSS0.Px1.p1.1)\.
- M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith,et al\.\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational conference on machine learning,pp\. 23965–23998\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1),[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Yadav, D\. Tam, L\. Choshen, C\. A\. Raffel, and M\. Bansal \(2023\)Ties\-merging: resolving interference when merging models\.Advances in Neural Information Processing Systems36,pp\. 7093–7115\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- A\. Yang, B\. Xiao, B\. Wang, B\. Zhang, C\. Bian, C\. Yin, C\. Lv, D\. Pan, D\. Wang, D\. Yan,et al\.\(2023a\)Baichuan 2: open large\-scale language models\.arXiv preprint arXiv:2309\.10305\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p2.1)\.
- E\. Yang, L\. Shen, G\. Guo, X\. Wang, X\. Cao, J\. Zhang, and D\. Tao \(2024\)Model merging in llms, mllms, and beyond: methods, theories, applications, and opportunities\.ACM Computing Surveys\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Yang, Z\. Wang, L\. Shen, S\. Liu, G\. Guo, X\. Wang, and D\. Tao \(2023b\)Adamerging: adaptive model merging for multi\-task learning\.arXiv preprint arXiv:2310\.02575\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. Li \(2024\)Language models are super mario: absorbing abilities from homologous models as a free lunch\.InForty\-first International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)Hellaswag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§B\.3](https://arxiv.org/html/2605.26484#A2.SS3.SSS0.Px3.p1.1)\.
- J\. Zhang, J\. Liu, J\. He,et al\.\(2023\)Composing parameter\-efficient modules with arithmetic operation\.Advances in Neural Information Processing Systems36,pp\. 12589–12610\.Cited by:[Appendix A](https://arxiv.org/html/2605.26484#A1.p1.1)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)Galore: memory\-efficient llm training by gradient low\-rank projection\.arXiv preprint arXiv:2403\.03507\.Cited by:[§B\.2](https://arxiv.org/html/2605.26484#A2.SS2.SSS0.Px3.p1.5)\.
- W\. Zhou, B\. Wang, W\. Chen, and X\. Cheng \(2025\)BSFA: leveraging the subspace dichotomy to accelerate neural network training\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 18845–18860\.Cited by:[§2](https://arxiv.org/html/2605.26484#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AA Comprehensive Review of Model Merging Methods

Post\-Fine\-Tuning Merging\. This topic focuses on integrating models fine\-tuned on diverse tasks into a unified entity without incurring additional inference costs\. Early heuristic approaches established the baseline: Model Soups demonstrates that averaging weights from hyperparameter sweeps enhances robustness and accuracy\(Wortsmanet al\.,[2022](https://arxiv.org/html/2605.26484#bib.bib21)\), while Fisher\-Weighted Averaging refines this by weighting parameters based on their Fisher information\(Matena and Raffel,[2022](https://arxiv.org/html/2605.26484#bib.bib48)\)\. Task Arithmetic interprets task\-specific updates as vectors, enabling capability composition via linear operations\(Ilharcoet al\.,[2022](https://arxiv.org/html/2605.26484#bib.bib49)\)\. Addressing interference, TIES\-Merging introduces trimming and sign\-resolution steps to mitigate parameter conflicts\(Yadavet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib50)\), and RegMean formulates merging as a linear regression problem with a closed\-form solution\(Jinet al\.,[2022](https://arxiv.org/html/2605.26484#bib.bib51)\)\. Moving towards adaptive methods, AdaMerging autonomously learns merging coefficients by minimizing the distance to task models on unlabelled test data\(Yanget al\.,[2023b](https://arxiv.org/html/2605.26484#bib.bib52)\), whereas MetaGPT frames the objective as minimizing the average loss across tasks\(Zhanget al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib53)\)\. For automated discovery, Evolutionary Model Merge employs evolutionary algorithms to search for optimal merging recipes in large model spaces\(Akibaet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib54)\)\. Finally, specialized techniques like DARE facilitate the merging of adapters by randomly dropping and rescaling parameters\(Yuet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib55)\), and PAPA balances ensemble generalization with the efficiency of weight averaging\(Raméet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib56)\)\.

Pre\-training Model Merging\. In contrast to the above, this topic aims to accelerate convergence and improve generalization by exploiting the trajectory of intermediate model states\. Fundamental to this approach, LAWA establishes that averaging recent checkpoints significantly reduces the training time required to reach target performance\(Kaddour,[2022](https://arxiv.org/html/2605.26484#bib.bib4)\)\. Subsequent analysis in Early Weight Averaging reveals that models trained with high learning rates benefit disproportionately from averaging due to increased trajectory exploration\(Sanyalet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib7)\)\. Empirical studies on Baichuan 2 validate these gains across different pre\-training stages, highlighting the importance of checkpoint selection\(Yanget al\.,[2023a](https://arxiv.org/html/2605.26484#bib.bib57)\)\. Theoretically, the WSM framework formalizes the relationship between learning rate schedules and model merging, proposing a ”warmup\-stable\-merge” protocol\(Tianet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib3)\)\. To optimize combination strategies, Dynamic Model Merging suggests weighting checkpoints based on generation quality metrics like perplexity, while other approaches utilize Bayesian Optimization to systematically search for the most effective weighting configurations\(Liuet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib43)\)\. While large\-scale model merging during pre\-training is a nascent frontier, our comprehensive review identifies that simple averaging strategies exemplified by Pre\-trained Model Averaging \(PMA\) constitute the optimal baseline practice\. This approach is theoretically grounded in Polyak\-Ruppert averaging\(Polyak and Juditsky,[1992](https://arxiv.org/html/2605.26484#bib.bib16)\), thereby rendering our adoption of PMA as the primary baseline both sufficient and rigorous\.

## Appendix BExperiment Details

#### Overview\.

We evaluate Extra\-Merge on \(i\) pre\-training loss trajectories for GPT\-2 on FineWeb and LLaMA on C4, and \(ii\) downstream accuracy for Pythia\-12B using official intermediate checkpoints\. Unless otherwise stated, we report the validation metrics of the*second half*of the training trajectory, where weight averaging becomes effective\.

#### Compute & implementation\.

All experiments are run on 8×\\timesRTX4090\-24G and 2×\\timesH100 with PyTorch 2\.5\.1, CUDA 12\.2\. Gradient clipping is set to1\.01\.0unless specified\.

#### Merging methods \(common definitions\)\.

Letθt\\theta\_\{t\}be the raw checkpoint saved everyτ\\tauoptimizer steps\. We define:

- •Raw:useθt\\theta\_\{t\}directly\.
- •PMA \(Uniform\):θtPMA=1N​∑i=0N−1θt−i​τ\\theta^\{\\text\{PMA\}\}\_\{t\}=\\frac\{1\}\{N\}\\sum\_\{i=0\}^\{N\-1\}\\theta\_\{t\-i\\tau\}\.
- •PMA \(EMA\):θtEMA=∑i=0N−1wi​θt−i​τ\\theta^\{\\text\{EMA\}\}\_\{t\}=\\sum\_\{i=0\}^\{N\-1\}w\_\{i\}\\theta\_\{t\-i\\tau\}, wherewi∝γiw\_\{i\}\\propto\\gamma^\{i\}and∑iwi=1\\sum\_\{i\}w\_\{i\}=1\.
- •Extra\-Merge:\(1\) compute merged checkpointsθtPMA\\theta^\{\\text\{PMA\}\}\_\{t\}as above; \(2\) run PCA on a sliding window ofKKconsecutive merged checkpoints to obtainu^1\\hat\{u\}\_\{1\}; \(3\) extrapolateθtEM=θtPMA\+α​u^1\\theta^\{\\text\{EM\}\}\_\{t\}=\\theta^\{\\text\{PMA\}\}\_\{t\}\+\\alpha\\hat\{u\}\_\{1\}\.

We select the extrapolation coefficientα\\alphaby a 1D line search\] on the validation set using candidate valuesα∈\{\\alpha\\in\\\{\[0,0\.2,0\.4,0\.6,0\.8,1,1\.2,1\.4,1\.6,1\.8,2\.0\]\}\\\}, and report results using the bestα\\alphaonce the validation loss \(indicator\) raise \(unless stated otherwise\)\.

### B\.1GPT\-2 on FineWeb

#### Models\.

We train GPT\-2 Small \(124M\), Medium \(355M\), and XL \(1\.55B\) using the standard decoder\-only Transformer architecture\(Radfordet al\.,[2019](https://arxiv.org/html/2605.26484#bib.bib58); Vaswaniet al\.,[2017](https://arxiv.org/html/2605.26484#bib.bib59)\)\. To optimize computational efficiency, we follow the hyperparameter setting in NanoGPT speedrun benchmark\.333[https://github\.com/KellerJordan/modded\-nanogpt](https://github.com/KellerJordan/modded-nanogpt), which adopt specific settings for learning rates, weight decay, and scheduler phases for each model\. The architectural details and corresponding hyperparameters are summarized in Table[2](https://arxiv.org/html/2605.26484#A2.T2)\.

Table 2:GPT\-2 architectures and hyperparameters used in our experiments\. We decouple the learning rates for the language model head \(L​RheadLR\_\{\\text\{head\}\}\) and the transformer body \(L​RbodyLR\_\{\\text\{body\}\}\)\.TwarmupT\_\{\\text\{warmup\}\}andTdecayT\_\{\\text\{decay\}\}denote the number of steps for the warmup and decay phases of the WSD scheduler, respectively\.
#### Dataset\.

We pre\-train on the FineWeb dataset\(Penedoet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib60)\), specifically utilizing the 10B token subset consistent with the NanoGPT speedrun benchmark\. Tokenization utilizes the GPT\-2 BPE tokenizer with a sequence length ofL=1024L=1024\. We report validation loss on the standard FineWeb validation split \(10M tokens\)\.

#### Training Protocol\.

We train all models for a total of 20,000 steps\. The global batch size is set to 512 sequences \(approx\. 0\.52 million tokens per step\), resulting in a total training budget of approximately 10 billion tokens\. We utilize mixed\-precision training withbfloat16\.

#### Optimization\.

We use the AdamW optimizer\(Loshchilov and Hutter,[2017](https://arxiv.org/html/2605.26484#bib.bib61)\)withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, andϵ=10−8\\epsilon=10^\{\-8\}\. We employ a split optimization strategy where the embeddings and language model head are trained with a higher learning rate \(L​RheadLR\_\{\\text\{head\}\}\) compared to the transformer backbone \(L​RbodyLR\_\{\\text\{body\}\}\), specifically settingL​Rbody=0\.5×L​RheadLR\_\{\\text\{body\}\}=0\.5\\times LR\_\{\\text\{head\}\}\. The learning rate follows a Warmup–Stable–Decay \(WSD\) schedule:

- •Warmup:Linear warmup forTwarmupT\_\{\\text\{warmup\}\}steps \(see Table[2](https://arxiv.org/html/2605.26484#A2.T2)\)\.
- •Stable:Constant learning rate at the peak value\.
- •Decay:Linear decay for the finalTdecayT\_\{\\text\{decay\}\}steps, reaching a minimum learning rate ofl​rmin=0\.05×l​rmaxlr\_\{\\min\}=0\.05\\times lr\_\{\\max\}\.

Weight decay is applied only to the transformer body parameters for GPT\-2 Medium \(λ=0\.125\\lambda=0\.125\), while it is set to 0 for Small and XL configurations based on empirical performance\.

#### Merge setting\.

We save checkpoints everyτ\\tausteps and apply merging on the latter half of training:

- •GPT\-2 Small/Medium:τ=500\\tau=500,N=8N=8\.
- •GPT\-2 XL:τ=1000\\tau=1000,N=10N=10\.
- •Extra\-Merge: PCA windowK=4K=4consecutive merged checkpoints\.

### B\.2LLaMA on C4

#### Models\.

We train LLaMA\(Touvronet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib62)\)incorporating Rotary Positional Embeddings \(RoPE\)\(Suet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib63)\), RMSNorm, and SwiGLU activations\. We evaluate two specific scales: 0\.5B and 2B\. Using the code from HuggingFace Transformer Library\(Wolfet al\.,[2020](https://arxiv.org/html/2605.26484#bib.bib64)\)\. The architectural configurations are detailed in Table[3](https://arxiv.org/html/2605.26484#A2.T3)\.

Table 3:LLaMA architectures and hyperparameters used in our experiments\.L​RmaxLR\_\{\\max\}denotes the peak learning rate used in the cosine schedule\.
#### Dataset\.

We pre\-train on the C4 \(Colossal Clean Crawled Corpus\)\(Raffelet al\.,[2020](https://arxiv.org/html/2605.26484#bib.bib65)\)dataset\. Tokenization utilizes a SentencePiece tokenizer with a vocabulary size ofV=32,100V=32,100\. The sequence length is set toL=2048L=2048\. We follow the standard C4 validation protocol, reporting validation loss and perplexity on the official validation split\.

#### Training Protocol\.

We follow the training setup ofWanget al\.\([2025](https://arxiv.org/html/2605.26484#bib.bib11)\); Zhaoet al\.\([2024](https://arxiv.org/html/2605.26484#bib.bib66)\), utilizing the AdamW optimizer withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decayλ=0\.1\\lambda=0\.1, andϵ=10−8\\epsilon=10^\{\-8\}\. Gradient clipping is set to1\.01\.0\. The learning rate follows a cosine decay schedule:

- •Warmup:Linear warmup for 1,000 steps\.
- •Peak LR:See Table[3](https://arxiv.org/html/2605.26484#A2.T3)for model\-specific values\.
- •Decay:Cosine decay to a terminal learning rate ofl​rmin=0\.1×l​rmaxlr\_\{\\min\}=0\.1\\times lr\_\{\\max\}\.

The global batch size is set to 512 sequences\. We train for a total of 100,000 steps, amounting to approximately 105 billion tokens\.

#### Implementation Details\.

Models are trained using PyTorch Fully Sharded Data Parallel \(FSDP\)\. We employ aHYBRID\_SHARDstrategy withbfloat16mixed precision, where parameters and buffers are maintained in FP32 while gradients and communication reductions occur inbfloat16\. We do not utilize CPU offloading\.

#### Merge setting\.

For LLaMA models we useτ=1000\\tau=1000,N=10N=10, and Extra\-Merge usesK=4K=4merged checkpoints\.

### B\.3Pythia\-12B Downstream Evaluation

#### Model and checkpoints\.

We evaluateEleutherAI/Pythia\-12B\(GPT\-NeoX style; 36 layers, model dimension 5120, 40 heads\)\(Bidermanet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib67)\)using the*official intermediate checkpoints*hosted on Hugging Face\.444[https://huggingface\.co/EleutherAI/pythia\-12b](https://huggingface.co/EleutherAI/pythia-12b)We use checkpoints from step 130k to 140k \(inclusive\) with a 1k\-step interval, i\.e\., 11 checkpoints in total\.

#### Merging protocol on checkpoints\.

We treat each 1k training step as one unit for checkpoint indexing and apply merging with:

τ=1000​\(steps\),N=10,K=4\.\\tau=1000\\ \\text\{\(steps\)\},\\quad N=10,\\quad K=4\.We compare Raw, PMA \(Uniform/EMA\), and Extra\-Merge\. For each method, we evaluate downstream tasks on the resulting merged/extrapolated checkpoint\.

#### Benchmarks and metric\.

We evaluate on ARC\-Challenge\(Clarket al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib69)\), ARC\-Easy\(Clarket al\.,[2018](https://arxiv.org/html/2605.26484#bib.bib69)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2605.26484#bib.bib70)\), and PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2605.26484#bib.bib71)\), reporting accuracy \(%\)\. We compute theaverage scoreacross tasks as the primary indicator\. Evaluation is implemented with the LM Evaluation Harness\(Gaoet al\.,[2023](https://arxiv.org/html/2605.26484#bib.bib68)\), using:

- •Few\-shot setting: 0\-shot
- •Max sequence length: 2048
- •Decoding: greedy \(since these are multiple\-choice log\-likelihood tasks\)
- •Dtype:float16, batch size: 16

### B\.4Extra\-Merge with Muon Optimizer

#### Setting\.

We additionally evaluate Extra\-Merge under theMuonoptimizer\(Jordanet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib28); Liuet al\.,[2025](https://arxiv.org/html/2605.26484#bib.bib27)\), following the NanoGPT speedrun codebase configuration\.555[https://github\.com/KellerJordan/modded\-nanogpt](https://github.com/KellerJordan/modded-nanogpt)Muon differs from AdamW by employing orthogonalized updates via Newton\-Schulz iterations, which substantially changes the geometry of the training trajectory\.

#### Protocol\.

We run GPT\-2 Small on FineWeb with the same data pipeline and model configuration as in Section[B\.1](https://arxiv.org/html/2605.26484#A2.SS1), replacing AdamW with Muon\. All merging hyperparameters are kept identical to the AdamW setting \(τ=500\\tau=500,N=8N=8,K=4K=4\) to isolate the effect of the optimizer\. Other Muon\-specific hyperparameters follow the default recipe in the benchmark codebase, including a split optimization strategy:

- •Transformer Body:Optimized via Muon with learning rateη=0\.02\\eta=0\.02and momentumμ=0\.95\\mu=0\.95\.
- •LM Head / Embeddings:Optimized via AdamW with learning rateη=0\.0036\\eta=0\.0036\.

### B\.5Large\-scale PCA via Gram Matrix \(Implementation Detail\)

Directly computing PCA in parameter space is infeasible since the parameter dimensionddis extremely large while the number of checkpoints is small \(K≪dK\\ll d\)\. We therefore compute the top principal component using theK×KK\\times KGram matrix\.

Let\{xi\}i=1K\\\{x\_\{i\}\\\}\_\{i=1\}^\{K\}be the centered checkpoints, wherexi∈ℝdx\_\{i\}\\in\\mathbb\{R\}^\{d\}andxi=θi−1K​∑j=1Kθjx\_\{i\}=\\theta\_\{i\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\theta\_\{j\}\. Define the Gram matrixG∈ℝK×KG\\in\\mathbb\{R\}^\{K\\times K\}as:

Gi​j=xi⊤​xj\.G\_\{ij\}=x\_\{i\}^\{\\top\}x\_\{j\}\.Letq1q\_\{1\}be the top eigenvector ofGG\. The corresponding principal direction in parameter space is recovered by:

u^1=∑i=1K\(q1\)i​xi‖∑i=1K\(q1\)i​xi‖2\.\\hat\{u\}\_\{1\}=\\frac\{\\sum\_\{i=1\}^\{K\}\(q\_\{1\}\)\_\{i\}x\_\{i\}\}\{\\left\\\|\\sum\_\{i=1\}^\{K\}\(q\_\{1\}\)\_\{i\}x\_\{i\}\\right\\\|\_\{2\}\}\.This procedure is mathematically equivalent to PCA on thed\(≫K\)d\(\\gg K\)\-dimensional centered matrix but costs only𝒪​\(K2\)\\mathcal\{O\}\(K^\{2\}\)inner products plus𝒪​\(K3\)\\mathcal\{O\}\(K^\{3\}\)eigendecomposition, which is negligible forK=4K=4\.

## Appendix CTheoretical Insight: Model Merging as River Projection and Extra\-Merge as Subspace Extrapolation

### C\.1Problem setting

Our setting follows the river\-valley landscape formalism ofWenet al\.\([2024](https://arxiv.org/html/2605.26484#bib.bib18)\)\. We restate a self\-contained version with a few additional assumptions tailored to checkpoint averaging and PCA\.

#### Notation\.

Letw∈ℝdw\\in\\mathbb\{R\}^\{d\}denote parameters andL:ℝd→ℝL:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}the loss\. Let∇L​\(w\)\\nabla L\(w\)and∇2L​\(w\)\\nabla^\{2\}L\(w\)be the gradient and Hessian\. Letλ1​\(H\)≥⋯≥λd​\(H\)\\lambda\_\{1\}\(H\)\\geq\\cdots\\geq\\lambda\_\{d\}\(H\)andv1​\(H\),…,vd​\(H\)v\_\{1\}\(H\),\\dots,v\_\{d\}\(H\)denote eigenvalues/eigenvectors of a symmetric matrixHHin*descending*order; thusvd​\(∇2L​\(w\)\)v\_\{d\}\(\\nabla^\{2\}L\(w\)\)is the eigenvector of the*smallest*eigenvalue and is called the*flattest direction*\. We writev​\(w\):=vd​\(∇2L​\(w\)\)v\(w\):=v\_\{d\}\(\\nabla^\{2\}L\(w\)\)and define the orthogonal projectionsPF​\(w\):=v​\(w\)​v​\(w\)⊤P\_\{F\}\(w\):=v\(w\)v\(w\)^\{\\top\}andPS​\(w\):=I−PF​\(w\)P\_\{S\}\(w\):=I\-P\_\{F\}\(w\), whereFFdenotes the \(one\-dimensional\)*river direction*andSSdenotes the\(d−1\)\(d\-1\)\-dimensional*mountain subspace*\.

#### River\-valley geometry\.

We assume a one\-dimensional “river” manifoldMMsuch that at points onMMthe gradient aligns with the flattest direction\.

###### Assumption C\.1\(River manifold and regularity, Assumption 2 in\(Wenet al\.,[2024](https://arxiv.org/html/2605.26484#bib.bib18)\)\)\.

There exists a11\-dimensionalC2C^\{2\}manifoldM⊂ℝdM\\subset\\mathbb\{R\}^\{d\}and an open neighborhoodU⊃MU\\supset Msuch that:

1. 1\.\(Alignment on the river\)For allw∈Mw\\in M,∇L​\(w\)\\nabla L\(w\)is parallel tov​\(w\)v\(w\)\.
2. 2\.\(Analyticity\)LLis analytic onUU\.
3. 3\.\(Bounded Hessian\)There existsβmax\>0\\beta\_\{\\max\}\>0such that‖∇2L​\(w\)‖op≤βmax\\\|\\nabla^\{2\}L\(w\)\\\|\_\{\\mathrm\{op\}\}\\leq\\beta\_\{\\max\}for allw∈Uw\\in U\.
4. 4\.\(Eigengap\)There existβ\>0\\beta\>0andβ♭\>0\\beta\_\{\\flat\}\>0such that for allw∈Uw\\in U,\|λd​\(∇2L​\(w\)\)\|<β♭\|\\lambda\_\{d\}\(\\nabla^\{2\}L\(w\)\)\|<\\beta\_\{\\flat\}andλd−1​\(∇2L​\(w\)\)\>β\+4​β♭\\lambda\_\{d\-1\}\(\\nabla^\{2\}L\(w\)\)\>\\beta\+4\\beta\_\{\\flat\}\.
5. 5\.\(Slow spinning\)There existρ∈\[0,0\.01\)\\rho\\in\[0,0\.01\)and0<νmin≤ν0<\\nu\_\{\\min\}\\leq\\nusuch that for allw∈Uw\\in U,νmin<‖∇L​\(w\)‖22≤ν\\nu\_\{\\min\}<\\\|\\nabla L\(w\)\\\|\_\{2\}^\{2\}\\leq\\nuand‖∇v​\(w\)‖op≤ρ⋅β2​ν\\\|\\nabla v\(w\)\\\|\_\{\\mathrm\{op\}\}\\leq\\rho\\cdot\\frac\{\\beta\}\{2\\nu\}\.

To obtain closed\-form\(N,T\)\(N,T\)\-dependent bounds, we adopt the standard “straight river” simplification\.

###### Assumption C\.2\(Straight river onUU\)\.

OnUU,v​\(w\)≡vv\(w\)\\equiv vis constant, hencePF​\(w\)≡PF:=v​v⊤P\_\{F\}\(w\)\\equiv P\_\{F\}:=vv^\{\\top\}andPS​\(w\)≡PS:=I−v​v⊤P\_\{S\}\(w\)\\equiv P\_\{S\}:=I\-vv^\{\\top\}are constant\. The river is the affine lineM=\{w⋆\+α​v:α∈ℝ\}M=\\\{w\_\{\\star\}\+\\alpha v:\\alpha\\in\\mathbb\{R\}\\\}for some reference pointw⋆∈Mw\_\{\\star\}\\in M\.

We model SGD with stochasticity only in mountain directions and a persistent, slow drift along the river\. This drift accounts for the variance in the PCA signal\.

###### Assumption C\.3\(SGD with drift and mountain\-only noise\)\.

Letη∈\(0,2/LS\)\\eta\\in\(0,2/L\_\{S\}\)be a constant learning rate\. The SGD iterates satisfywk\+1=wk−η​∇L​\(wk\)−η​μF​v\+η​gkw\_\{k\+1\}=w\_\{k\}\-\\eta\\nabla L\(w\_\{k\}\)\-\\eta\\mu\_\{F\}v\+\\eta g\_\{k\}, where\(gk\)k≥0\(g\_\{k\}\)\_\{k\\geq 0\}are i\.i\.d\. withgk∼𝒩​\(0,σ2​PS\)g\_\{k\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}P\_\{S\}\)\. The term−η​μF​v\-\\eta\\mu\_\{F\}vrepresents a small, constant drift along the river direction, withμF\>0\\mu\_\{F\}\>0\. This captures the effect of a persistent, non\-zero expected gradient component along the river, typical in late\-stage training\. We assume iterates remain inUU\.

###### Assumption C\.4\(Quadratic mountain component onUU\)\.

OnUU, the loss decomposes asL​\(w\)=ℓ​\(PF​\(w−w⋆\)\)\+12​‖H1/2​PS​\(w−w⋆\)‖22L\(w\)=\\ell\(P\_\{F\}\(w\-w\_\{\\star\}\)\)\+\\frac\{1\}\{2\}\\\|H^\{1/2\}P\_\{S\}\(w\-w\_\{\\star\}\)\\\|\_\{2\}^\{2\}, whereℓ:ℝ→ℝ\\ell:\\mathbb\{R\}\\to\\mathbb\{R\}isC2C^\{2\}andHHis a symmetric matrix satisfyingH​PF=0HP\_\{F\}=0,H​PS=PS​H=HHP\_\{S\}=P\_\{S\}H=H, and its spectrum onrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)is contained in\[μ,LS\]\[\\mu,L\_\{S\}\]for some0<μ≤LS0<\\mu\\leq L\_\{S\}\.

#### Decoupled dynamics\.

The straight\-river and quadratic\-mountain assumptions imply that the SGD dynamics separate into a one\-dimensional river recursion and a\(d−1\)\(d\-1\)\-dimensional mountain recursion\. In particular, the river coordinatetkt\_\{k\}evolves without depending on the mountain statezkz\_\{k\}, and the mountain recursion depends only onzkz\_\{k\}and the projected noise\. Under Assumptions[C\.2](https://arxiv.org/html/2605.26484#A3.Thmtheorem2)–[C\.4](https://arxiv.org/html/2605.26484#A3.Thmtheorem4), writingtk:=v⊤​\(wk−w⋆\)t\_\{k\}:=v^\{\\top\}\(w\_\{k\}\-w\_\{\\star\}\)andzk:=PS​\(wk−w⋆\)z\_\{k\}:=P\_\{S\}\(w\_\{k\}\-w\_\{\\star\}\), the updates satisfy

tk\+1=tk−η​ℓ′​\(tk\)−η​μF,zk\+1=\(I−η​H\)​zk\+η​gk,t\_\{k\+1\}=t\_\{k\}\-\\eta\\,\\ell^\{\\prime\}\(t\_\{k\}\)\-\\eta\\mu\_\{F\},\\qquad z\_\{k\+1\}=\(I\-\\eta H\)z\_\{k\}\+\\eta g\_\{k\},withgk∼𝒩​\(0,σ2​PS\)g\_\{k\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}P\_\{S\}\)\.

#### Checkpoint protocol\.

We save checkpoints everyTTsteps:w\(m\):=wk0\+m​Tw^\{\(m\)\}:=w\_\{k\_\{0\}\+mT\}form=0,1,2,…m=0,1,2,\\dots, starting afterk0k\_\{0\}\.

### C\.2Theorem 1: Averaging Reduces Mountain Oscillations

We first formalize the distance to the river manifold\. Under Assumption[C\.2](https://arxiv.org/html/2605.26484#A3.Thmtheorem2), this distance is simply the norm of the projection onto the mountain subspace\.

###### Definition C\.5\(River deviation\)\.

Under Assumption[C\.2](https://arxiv.org/html/2605.26484#A3.Thmtheorem2), the river deviation of a pointwwisD​\(w\):=‖PS​\(w−w⋆\)‖2=dist​\(w,M\)D\(w\):=\\\|P\_\{S\}\(w\-w\_\{\\star\}\)\\\|\_\{2\}=\\mathrm\{dist\}\(w,M\)\.

For the sake of readability, we restate Theorem[5\.3](https://arxiv.org/html/2605.26484#S5.Thmtheorem3)as below\.

###### Theorem C\.6\(Averaging reduces expected distance to the river\)\.

Assume Assumptions[C\.2](https://arxiv.org/html/2605.26484#A3.Thmtheorem2)–[C\.4](https://arxiv.org/html/2605.26484#A3.Thmtheorem4)\. Letη∈\(0,2/LS\)\\eta\\in\(0,2/L\_\{S\}\)and let checkpoints\{w\(m\)\}m=0N−1\\\{w^\{\(m\)\}\\\}\_\{m=0\}^\{N\-1\}be taken after a burn\-in periodk0k\_\{0\}such that the mountain dynamics are stationary\. Letw¯=1N​∑m=0N−1w\(m\)\\bar\{w\}=\\frac\{1\}\{N\}\\sum\_\{m=0\}^\{N\-1\}w^\{\(m\)\}be the merged checkpoint\. Then the expected squared river deviation ofw¯\\bar\{w\}is

𝔼​D​\(w¯\)2=∑j=1d−1\(η​σ22​λj−η​λj2\)⋅1N2​\[N\+2​∑r=1N−1\(N−r\)​\(1−η​λj\)r​T\],\\mathbb\{E\}\\,D\(\\bar\{w\}\)^\{2\}\\;=\\;\\sum\_\{j=1\}^\{d\-1\}\\left\(\\frac\{\\eta\\sigma^\{2\}\}\{2\\lambda\_\{j\}\-\\eta\\lambda\_\{j\}^\{2\}\}\\right\)\\cdot\\frac\{1\}\{N^\{2\}\}\\left\[N\+2\\sum\_\{r=1\}^\{N\-1\}\(N\-r\)\\big\(1\-\\eta\\lambda\_\{j\}\\big\)^\{rT\}\\right\],\(5\)where\{λj\}j=1d−1\\\{\\lambda\_\{j\}\\\}\_\{j=1\}^\{d\-1\}are the eigenvalues ofHHonrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)\. In particular, if checkpoints are weakly correlated such that\|1−η​λj\|T≤ε\|1\-\\eta\\lambda\_\{j\}\|^\{T\}\\leq\\varepsilonfor alljj, then

𝔼​D​\(w¯\)2≤1N​\(1\+2​ε1−ε\)​∑j=1d−1η​σ22​λj−η​λj2\.\\mathbb\{E\}\\,D\(\\bar\{w\}\)^\{2\}\\;\\leq\\;\\frac\{1\}\{N\}\\left\(1\+\\frac\{2\\varepsilon\}\{1\-\\varepsilon\}\\right\)\\sum\_\{j=1\}^\{d\-1\}\\frac\{\\eta\\sigma^\{2\}\}\{2\\lambda\_\{j\}\-\\eta\\lambda\_\{j\}^\{2\}\}\.\(6\)

###### Proof\.

Define the mountain coordinatezk:=PS​\(wk−w⋆\)z\_\{k\}:=P\_\{S\}\(w\_\{k\}\-w\_\{\\star\}\)\. We project the SGD update from Assumption[C\.3](https://arxiv.org/html/2605.26484#A3.Thmtheorem3)onto the mountain subspacerange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\):

zk\+1\\displaystyle z\_\{k\+1\}=PS​\(wk\+1−w⋆\)\\displaystyle=P\_\{S\}\(w\_\{k\+1\}\-w\_\{\\star\}\)=PS​\(\(wk−w⋆\)−η​∇L​\(wk\)−η​μF​v\+η​gk\)\\displaystyle=P\_\{S\}\\left\(\(w\_\{k\}\-w\_\{\\star\}\)\-\\eta\\nabla L\(w\_\{k\}\)\-\\eta\\mu\_\{F\}v\+\\eta g\_\{k\}\\right\)=PS​\(wk−w⋆\)−η​PS​\(∇L​\(wk\)\)−η​μF​PS​\(v\)\+η​PS​\(gk\)\.\\displaystyle=P\_\{S\}\(w\_\{k\}\-w\_\{\\star\}\)\-\\eta P\_\{S\}\(\\nabla L\(w\_\{k\}\)\)\-\\eta\\mu\_\{F\}P\_\{S\}\(v\)\+\\eta P\_\{S\}\(g\_\{k\}\)\.Under our assumptions:

- •From Assumption[C\.4](https://arxiv.org/html/2605.26484#A3.Thmtheorem4),∇L​\(w\)=ℓ′​\(PF​\(w−w⋆\)\)​v\+H​PS​\(w−w⋆\)\\nabla L\(w\)=\\ell^\{\\prime\}\(P\_\{F\}\(w\-w\_\{\\star\}\)\)v\+HP\_\{S\}\(w\-w\_\{\\star\}\)\. Thus,PS​\(∇L​\(wk\)\)=PS​\(ℓ′​\(⋅\)​v\)\+PS​\(H​zk\)=0\+H​zk=H​zkP\_\{S\}\(\\nabla L\(w\_\{k\}\)\)=P\_\{S\}\(\\ell^\{\\prime\}\(\\cdot\)v\)\+P\_\{S\}\(Hz\_\{k\}\)=0\+Hz\_\{k\}=Hz\_\{k\}\.
- •From Assumption[C\.2](https://arxiv.org/html/2605.26484#A3.Thmtheorem2),PS​v=0P\_\{S\}v=0\.
- •From Assumption[C\.3](https://arxiv.org/html/2605.26484#A3.Thmtheorem3),gkg\_\{k\}is supported onrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\), soPS​gk=gkP\_\{S\}g\_\{k\}=g\_\{k\}\.

Substituting these into the update ofzkz\_\{k\}:

zk\+1=zk−η​H​zk\+η​gk=\(I−η​H\)​zk\+η​gk\.z\_\{k\+1\}=z\_\{k\}\-\\eta Hz\_\{k\}\+\\eta g\_\{k\}=\(I\-\\eta H\)z\_\{k\}\+\\eta g\_\{k\}\.LetA:=I−η​HA:=I\-\\eta H\. The conditionη∈\(0,2/LS\)\\eta\\in\(0,2/L\_\{S\}\)ensures that the eigenvalues ofAAonrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)are in\(−1,1\)\(\-1,1\)\.

Let\{qj\}j=1d−1\\\{q\_\{j\}\\\}\_\{j=1\}^\{d\-1\}be an orthonormal eigenbasis forHHonrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)with eigenvaluesλj\\lambda\_\{j\}\. Letuk\(j\):=qj⊤​zku\_\{k\}^\{\(j\)\}:=q\_\{j\}^\{\\top\}z\_\{k\}be the coordinate ofzkz\_\{k\}alongqjq\_\{j\}\. Projecting the dynamics ofzkz\_\{k\}ontoqjq\_\{j\}:

uk\+1\(j\)\\displaystyle u\_\{k\+1\}^\{\(j\)\}=qj⊤​zk\+1=qj⊤​\(\(I−η​H\)​zk\+η​gk\)\\displaystyle=q\_\{j\}^\{\\top\}z\_\{k\+1\}=q\_\{j\}^\{\\top\}\\left\(\(I\-\\eta H\)z\_\{k\}\+\\eta g\_\{k\}\\right\)=qj⊤​\(I−η​H\)​zk\+η​qj⊤​gk\\displaystyle=q\_\{j\}^\{\\top\}\(I\-\\eta H\)z\_\{k\}\+\\eta q\_\{j\}^\{\\top\}g\_\{k\}=\(1−η​λj\)​qj⊤​zk\+η​qj⊤​gk\\displaystyle=\(1\-\\eta\\lambda\_\{j\}\)q\_\{j\}^\{\\top\}z\_\{k\}\+\\eta q\_\{j\}^\{\\top\}g\_\{k\}=\(1−η​λj\)​uk\(j\)\+η​ξk\(j\),\\displaystyle=\(1\-\\eta\\lambda\_\{j\}\)u\_\{k\}^\{\(j\)\}\+\\eta\\xi\_\{k\}^\{\(j\)\},whereξk\(j\):=qj⊤​gk\\xi\_\{k\}^\{\(j\)\}:=q\_\{j\}^\{\\top\}g\_\{k\}\. Sincegk∼𝒩​\(0,σ2​PS\)g\_\{k\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}P\_\{S\}\)and\{qj\}\\\{q\_\{j\}\\\}is an orthonormal basis forrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\), the variablesξk\(j\)\\xi\_\{k\}^\{\(j\)\}are i\.i\.d\.𝒩​\(0,σ2\)\\mathcal\{N\}\(0,\\sigma^\{2\}\)\. This is an AR\(1\) process666AutoRegressive model of order 1with coefficientaj:=1−η​λj∈\(−1,1\)a\_\{j\}:=1\-\\eta\\lambda\_\{j\}\\in\(\-1,1\)\.

For an AR\(1\) processxt\+1=a​xt\+ϵtx\_\{t\+1\}=ax\_\{t\}\+\\epsilon\_\{t\}with i\.i\.d\. noiseϵt\\epsilon\_\{t\}where𝔼​\[ϵt\]=0,Var​\(ϵt\)=σϵ2\\mathbb\{E\}\[\\epsilon\_\{t\}\]=0,\\mathrm\{Var\}\(\\epsilon\_\{t\}\)=\\sigma\_\{\\epsilon\}^\{2\}, the stationary variance isVar​\(x\)=σϵ2/\(1−a2\)\\mathrm\{Var\}\(x\)=\\sigma\_\{\\epsilon\}^\{2\}/\(1\-a^\{2\}\)\. For our processuk\(j\)u\_\{k\}^\{\(j\)\}, the noise term isη​ξk\(j\)\\eta\\xi\_\{k\}^\{\(j\)\}, so its variance isVar​\(η​ξk\(j\)\)=η2​σ2\\mathrm\{Var\}\(\\eta\\xi\_\{k\}^\{\(j\)\}\)=\\eta^\{2\}\\sigma^\{2\}\.

Var​\(uk\(j\)\)=η2​σ21−aj2=η2​σ21−\(1−η​λj\)2=η2​σ21−\(1−2​η​λj\+η2​λj2\)=η​σ22​λj−η​λj2\.\\mathrm\{Var\}\(u\_\{k\}^\{\(j\)\}\)=\\frac\{\\eta^\{2\}\\sigma^\{2\}\}\{1\-a\_\{j\}^\{2\}\}=\\frac\{\\eta^\{2\}\\sigma^\{2\}\}\{1\-\(1\-\\eta\\lambda\_\{j\}\)^\{2\}\}=\\frac\{\\eta^\{2\}\\sigma^\{2\}\}\{1\-\(1\-2\\eta\\lambda\_\{j\}\+\\eta^\{2\}\\lambda\_\{j\}^\{2\}\)\}=\\frac\{\\eta\\sigma^\{2\}\}\{2\\lambda\_\{j\}\-\\eta\\lambda\_\{j\}^\{2\}\}\.The autocovariance at lagΔ≥0\\Delta\\geq 0isCov​\(uk\(j\),uk\+Δ\(j\)\)=ajΔ​Var​\(uk\(j\)\)\\mathrm\{Cov\}\(u\_\{k\}^\{\(j\)\},u\_\{k\+\\Delta\}^\{\(j\)\}\)=a\_\{j\}^\{\\Delta\}\\mathrm\{Var\}\(u\_\{k\}^\{\(j\)\}\)\.

Now we formalize the variance of the averaged checkpoint\. Letz¯=1N​∑m=0N−1z\(m\)\\bar\{z\}=\\frac\{1\}\{N\}\\sum\_\{m=0\}^\{N\-1\}z^\{\(m\)\}, wherez\(m\)=zk0\+m​Tz^\{\(m\)\}=z\_\{k\_\{0\}\+mT\}\. The expected squared deviation is𝔼​D​\(w¯\)2=𝔼​‖z¯‖22\\mathbb\{E\}D\(\\bar\{w\}\)^\{2\}=\\mathbb\{E\}\\\|\\bar\{z\}\\\|\_\{2\}^\{2\}\. By orthogonality of the basis\{qj\}\\\{q\_\{j\}\\\}, this is∑j=1d−1𝔼​\[\(u¯\(j\)\)2\]\\sum\_\{j=1\}^\{d\-1\}\\mathbb\{E\}\[\(\\bar\{u\}^\{\(j\)\}\)^\{2\}\]\. Since the process is stationary and zero\-mean,𝔼​\[\(u¯\(j\)\)2\]=Var​\(u¯\(j\)\)\\mathbb\{E\}\[\(\\bar\{u\}^\{\(j\)\}\)^\{2\}\]=\\mathrm\{Var\}\(\\bar\{u\}^\{\(j\)\}\)\.

Var​\(u¯\(j\)\)\\displaystyle\\mathrm\{Var\}\(\\bar\{u\}^\{\(j\)\}\)=Var​\(1N​∑m=0N−1uk0\+m​T\(j\)\)=1N2​∑m=0N−1∑n=0N−1Cov​\(uk0\+m​T\(j\),uk0\+n​T\(j\)\)\\displaystyle=\\mathrm\{Var\}\\left\(\\frac\{1\}\{N\}\\sum\_\{m=0\}^\{N\-1\}u^\{\(j\)\}\_\{k\_\{0\}\+mT\}\\right\)=\\frac\{1\}\{N^\{2\}\}\\sum\_\{m=0\}^\{N\-1\}\\sum\_\{n=0\}^\{N\-1\}\\mathrm\{Cov\}\(u^\{\(j\)\}\_\{k\_\{0\}\+mT\},u^\{\(j\)\}\_\{k\_\{0\}\+nT\}\)=1N2​∑m=0N−1∑n=0N−1aj\|m−n\|​T​Var​\(u\(j\)\)\\displaystyle=\\frac\{1\}\{N^\{2\}\}\\sum\_\{m=0\}^\{N\-1\}\\sum\_\{n=0\}^\{N\-1\}a\_\{j\}^\{\|m\-n\|T\}\\mathrm\{Var\}\(u^\{\(j\)\}\)=Var​\(u\(j\)\)N2​\(∑m=naj0⋅T\+∑m≠naj\|m−n\|​T\)\.\\displaystyle=\\frac\{\\mathrm\{Var\}\(u^\{\(j\)\}\)\}\{N^\{2\}\}\\left\(\\sum\_\{m=n\}a\_\{j\}^\{0\\cdot T\}\+\\sum\_\{m\\neq n\}a\_\{j\}^\{\|m\-n\|T\}\\right\)\.The double summation can be re\-indexed by the lagr=\|m−n\|r=\|m\-n\|\.

- •For lagr=0r=0\(diagonal terms,m=nm=n\), there areNNterms, each equal toaj0=1a\_\{j\}^\{0\}=1\.
- •For lagr∈\{1,…,N−1\}r\\in\\\{1,\\dots,N\-1\\\}, there are2​\(N−r\)2\(N\-r\)pairs\(m,n\)\(m,n\)such that\|m−n\|=r\|m\-n\|=r\.

Thus, the sum becomes:

Var​\(u¯\(j\)\)=Var​\(u\(j\)\)N2​\(N\+2​∑r=1N−1\(N−r\)​ajr​T\)\.\\mathrm\{Var\}\(\\bar\{u\}^\{\(j\)\}\)=\\frac\{\\mathrm\{Var\}\(u^\{\(j\)\}\)\}\{N^\{2\}\}\\left\(N\+2\\sum\_\{r=1\}^\{N\-1\}\(N\-r\)a\_\{j\}^\{rT\}\\right\)\.Summing overjjand substituting the expression forVar​\(u\(j\)\)\\mathrm\{Var\}\(u^\{\(j\)\}\)gives the first result\. Then, we use the triangle inequality and the sum of a geometric series\. Note that\|aj\|=\|1−η​λj\|<1\|a\_\{j\}\|=\|1\-\\eta\\lambda\_\{j\}\|<1\.

1N2​\|N\+2​∑r=1N−1\(N−r\)​ajr​T\|\\displaystyle\\frac\{1\}\{N^\{2\}\}\\left\|N\+2\\sum\_\{r=1\}^\{N\-1\}\(N\-r\)a\_\{j\}^\{rT\}\\right\|≤1N2​\(N\+2​∑r=1N−1\(N−r\)​\|aj\|r​T\)\\displaystyle\\leq\\frac\{1\}\{N^\{2\}\}\\left\(N\+2\\sum\_\{r=1\}^\{N\-1\}\(N\-r\)\|a\_\{j\}\|^\{rT\}\\right\)≤1N2​\(N\+2​\(N−1\)​∑r=1N−1\(\|aj\|T\)r\)\\displaystyle\\leq\\frac\{1\}\{N^\{2\}\}\\left\(N\+2\(N\-1\)\\sum\_\{r=1\}^\{N\-1\}\(\|a\_\{j\}\|^\{T\}\)^\{r\}\\right\)≤1N2​\(N\+2​\(N−1\)​∑r=1∞\(\|aj\|T\)r\)\\displaystyle\\leq\\frac\{1\}\{N^\{2\}\}\\left\(N\+2\(N\-1\)\\sum\_\{r=1\}^\{\\infty\}\(\|a\_\{j\}\|^\{T\}\)^\{r\}\\right\)=1N2​\(N\+2​\(N−1\)​\|aj\|T1−\|aj\|T\)\.\\displaystyle=\\frac\{1\}\{N^\{2\}\}\\left\(N\+2\(N\-1\)\\frac\{\|a\_\{j\}\|^\{T\}\}\{1\-\|a\_\{j\}\|^\{T\}\}\\right\)\.Using the condition\|aj\|T≤ε\|a\_\{j\}\|^\{T\}\\leq\\varepsilon:

Var​\(u¯\(j\)\)\\displaystyle\\mathrm\{Var\}\(\\bar\{u\}^\{\(j\)\}\)≤Var​\(u\(j\)\)N2​\(N\+2​\(N−1\)​ε1−ε\)\\displaystyle\\leq\\frac\{\\mathrm\{Var\}\(u^\{\(j\)\}\)\}\{N^\{2\}\}\\left\(N\+2\(N\-1\)\\frac\{\\varepsilon\}\{1\-\\varepsilon\}\\right\)=Var​\(u\(j\)\)​\(1N\+2​\(N−1\)N2​ε1−ε\)\.\\displaystyle=\\mathrm\{Var\}\(u^\{\(j\)\}\)\\left\(\\frac\{1\}\{N\}\+\\frac\{2\(N\-1\)\}\{N^\{2\}\}\\frac\{\\varepsilon\}\{1\-\\varepsilon\}\\right\)\.≤1N​\(1\+2​ε1−ε\)​Var​\(u\(j\)\)\.\\displaystyle\\leq\\frac\{1\}\{N\}\\left\(1\+\\frac\{2\\varepsilon\}\{1\-\\varepsilon\}\\right\)\\mathrm\{Var\}\(u^\{\(j\)\}\)\.Summing overjjyields the second result\. ∎

### C\.3Theorem 2: PCA on Sliding\-Window Merges Recovers the River Direction

We analyze a sequence ofKKmerged checkpoints from sliding windows:w¯\(s\):=1N​∑i=0N−1w\(s\+i\)\\bar\{w\}^\{\(s\)\}:=\\frac\{1\}\{N\}\\sum\_\{i=0\}^\{N\-1\}w^\{\(s\+i\)\}\. We perform PCA on the centered sequencew~\(s\):=w¯\(s\)−1K​∑r=0K−1w¯\(r\)\\tilde\{w\}^\{\(s\)\}:=\\bar\{w\}^\{\(s\)\}\-\\frac\{1\}\{K\}\\sum\_\{r=0\}^\{K\-1\}\\bar\{w\}^\{\(r\)\}\. For the sake of readability, we restate Theorem[5\.4](https://arxiv.org/html/2605.26484#S5.Thmtheorem4)in a slightly more detailed form as below\.

###### Theorem C\.7\(PCA direction aligns with the flattest eigenvector\)\.

Assume Assumptions[C\.2](https://arxiv.org/html/2605.26484#A3.Thmtheorem2)–[C\.4](https://arxiv.org/html/2605.26484#A3.Thmtheorem4)\. Decompose each merged state asw¯\(s\)=w⋆\+ts​v\+εs\\bar\{w\}^\{\(s\)\}=w\_\{\\star\}\+t\_\{s\}v\+\\varepsilon\_\{s\}\. LetΣ=𝔼​\[w~\(s\)​w~\(s\)⊤\]\\Sigma=\\mathbb\{E\}\[\\tilde\{w\}^\{\(s\)\}\\tilde\{w\}^\{\(s\)\\top\}\]be the population covariance\. Letσsig2:=Var​\(t~s\)\\sigma\_\{\\mathrm\{sig\}\}^\{2\}:=\\mathrm\{Var\}\(\\tilde\{t\}\_\{s\}\)andσnoise2:=λmax​\(𝔼​\[ε~s​ε~s⊤\]\)\\sigma\_\{\\mathrm\{noise\}\}^\{2\}:=\\lambda\_\{\\max\}\(\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]\)\. If the signal\-to\-noise gapδΣ:=σsig2−σnoise2\>0\\delta\_\{\\Sigma\}:=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}\-\\sigma\_\{\\mathrm\{noise\}\}^\{2\}\>0, thenvvis the unique top eigenvector ofΣ\\Sigma\.

Furthermore, under a standard concentration assumption for the sample covarianceΣ^\\hat\{\\Sigma\}, with probability at least1−α1\-\\alpha:

\|u^1⊤​v\|2≥1−\(‖Σ^−Σ‖opδΣ\)2,\|\\hat\{u\}\_\{1\}^\{\\top\}v\|^\{2\}\\geq 1\-\\left\(\\frac\{\\\|\\hat\{\\Sigma\}\-\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\delta\_\{\\Sigma\}\}\\right\)^\{2\},\(7\)whereu^1\\hat\{u\}\_\{1\}is the top eigenvector ofΣ^\\hat\{\\Sigma\}\.

###### Proof\.

The centered merged state isw~\(s\)=t~s​v\+ε~s\\tilde\{w\}^\{\(s\)\}=\\tilde\{t\}\_\{s\}v\+\\tilde\{\\varepsilon\}\_\{s\}, wheret~s\\tilde\{t\}\_\{s\}andε~s\\tilde\{\\varepsilon\}\_\{s\}are the centered versions oftst\_\{s\}andεs\\varepsilon\_\{s\}\. The population covariance matrix is:

Σ\\displaystyle\\Sigma=𝔼​\[w~\(s\)​\(w~\(s\)\)⊤\]=𝔼​\[\(t~s​v\+ε~s\)​\(t~s​v\+ε~s\)⊤\]\\displaystyle=\\mathbb\{E\}\[\\tilde\{w\}^\{\(s\)\}\(\\tilde\{w\}^\{\(s\)\}\)^\{\\top\}\]=\\mathbb\{E\}\\left\[\(\\tilde\{t\}\_\{s\}v\+\\tilde\{\\varepsilon\}\_\{s\}\)\(\\tilde\{t\}\_\{s\}v\+\\tilde\{\\varepsilon\}\_\{s\}\)^\{\\top\}\\right\]=𝔼​\[t~s2​v​v⊤\+t~s​v​ε~s⊤\+ε~s​t~s​v⊤\+ε~s​ε~s⊤\]\\displaystyle=\\mathbb\{E\}\\left\[\\tilde\{t\}\_\{s\}^\{2\}vv^\{\\top\}\+\\tilde\{t\}\_\{s\}v\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\+\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{t\}\_\{s\}v^\{\\top\}\+\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\\right\]=𝔼​\[t~s2\]​v​v⊤\+𝔼​\[t~s​v​ε~s⊤\]\+𝔼​\[ε~s​t~s​v⊤\]\+𝔼​\[ε~s​ε~s⊤\]\.\\displaystyle=\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}^\{2\}\]vv^\{\\top\}\+\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}v\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]\+\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{t\}\_\{s\}v^\{\\top\}\]\+\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]\.By construction,εs:=PS​\(w¯\(s\)−w⋆\)\\varepsilon\_\{s\}:=P\_\{S\}\(\\bar\{w\}^\{\(s\)\}\-w\_\{\\star\}\)lies in the mountain subspacerange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\), hence it is orthogonal tovv, i\.e\.,v⊤​εs=0v^\{\\top\}\\varepsilon\_\{s\}=0\. Centering is a linear operation, soε~s\\tilde\{\\varepsilon\}\_\{s\}also lies inrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)and satisfiesv⊤​ε~s=0v^\{\\top\}\\tilde\{\\varepsilon\}\_\{s\}=0\. Therefore, sincet~s\\tilde\{t\}\_\{s\}andε~s\\tilde\{\\varepsilon\}\_\{s\}are independent:

𝔼​\[t~s​v​ε~s⊤\]=v​𝔼​\[t~s​ε~s⊤\]=v​𝔼​\[t~s\]​𝔼​\[ε~s⊤\]=v⋅0⋅𝟎⊤=𝟎\.\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}v\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]=v\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]=v\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}\]\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]=v\\cdot 0\\cdot\\mathbf\{0\}^\{\\top\}=\\mathbf\{0\}\.
Then the covariance matrix simplifies to a block\-diagonal structure:

Σ=𝔼​\[t~s2\]​v​v⊤\+𝔼​\[ε~s​ε~s⊤\]\.\\Sigma=\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}^\{2\}\]vv^\{\\top\}\+\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]\.Since𝔼​\[t~s\]=0\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}\]=0, we have𝔼\[t~s2\]=Var\(t~s\)=:σsig2\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}^\{2\}\]=\\mathrm\{Var\}\(\\tilde\{t\}\_\{s\}\)=:\\sigma\_\{\\mathrm\{sig\}\}^\{2\}\. LetΣε,cent:=𝔼​\[ε~s​ε~s⊤\]\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}:=\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]\. The covariance matrix is:

Σ=σsig2​v​v⊤\+Σε,cent\.\\Sigma=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}vv^\{\\top\}\+\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\.
The noise componentεs=PS​\(w¯\(s\)−w⋆\)\\varepsilon\_\{s\}=P\_\{S\}\(\\bar\{w\}^\{\(s\)\}\-w\_\{\\star\}\)lies in the mountain subspacerange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\), which is orthogonal tovv\. The centered noiseε~s\\tilde\{\\varepsilon\}\_\{s\}also lies in this subspace\. Therefore, the covarianceΣε,cent\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}is supported onrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\), which impliesΣε,cent​v=𝟎\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}v=\\mathbf\{0\}\. Let’s testvvas an eigenvector ofΣ\\Sigma:

Σ​v=\(σsig2​v​v⊤\+Σε,cent\)​v=σsig2​v​\(v⊤​v\)\+Σε,cent​v=σsig2​v​\(1\)\+𝟎=σsig2​v\.\\Sigma v=\(\\sigma\_\{\\mathrm\{sig\}\}^\{2\}vv^\{\\top\}\+\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)v=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}v\(v^\{\\top\}v\)\+\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}v=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}v\(1\)\+\\mathbf\{0\}=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}v\.So,vvis an eigenvector with eigenvalueσsig2\\sigma\_\{\\mathrm\{sig\}\}^\{2\}\. Now consider any unit vectoru⟂vu\\perp v\. Such a vectoruulies inrange​\(PS\)\\mathrm\{range\}\(P\_\{S\}\)\.

Σ​u=\(σsig2​v​v⊤\+Σε,cent\)​u=σsig2​v​\(v⊤​u\)\+Σε,cent​u=σsig2​v​\(0\)\+Σε,cent​u=Σε,cent​u\.\\Sigma u=\(\\sigma\_\{\\mathrm\{sig\}\}^\{2\}vv^\{\\top\}\+\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)u=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}v\(v^\{\\top\}u\)\+\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}u=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}v\(0\)\+\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}u=\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}u\.This shows that the action ofΣ\\Sigmaon the subspace orthogonal tovvis identical to the action ofΣε,cent\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\. The eigenvalues ofΣ\\Sigmaare therefore\{σsig2\}∪\{eigenvalues of​Σε,cent\}\\\{\\sigma\_\{\\mathrm\{sig\}\}^\{2\}\\\}\\cup\\\{\\text\{eigenvalues of \}\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\\\}\. The largest eigenvalue ofΣε,cent\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}isλmax\(Σε,cent\)=:σnoise2\\lambda\_\{\\max\}\(\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)=:\\sigma\_\{\\mathrm\{noise\}\}^\{2\}\. Thus, the eigenvalues ofΣ\\Sigmaareλ1​\(Σ\)=σsig2\\lambda\_\{1\}\(\\Sigma\)=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}andλ2​\(Σ\)=σnoise2\\lambda\_\{2\}\(\\Sigma\)=\\sigma\_\{\\mathrm\{noise\}\}^\{2\}\(assuming no other eigenvalue ofΣε,cent\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}is larger\)\. The conditionδΣ=σsig2−σnoise2\>0\\delta\_\{\\Sigma\}=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}\-\\sigma\_\{\\mathrm\{noise\}\}^\{2\}\>0is precisely the condition thatλ1​\(Σ\)\>λ2​\(Σ\)\\lambda\_\{1\}\(\\Sigma\)\>\\lambda\_\{2\}\(\\Sigma\)\. By the properties of symmetric matrices, this makes the top eigenvectorvvunique\.Then using Davis\-Kahansin⁡Θ\\sin\\Thetatheorem, we have:

sin⁡∠​\(u^1,v\)≤‖Σ^−Σ‖opλ1​\(Σ\)−λ2​\(Σ\)\.\\sin\\angle\(\\hat\{u\}\_\{1\},v\)\\leq\\frac\{\\\|\\hat\{\\Sigma\}\-\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\lambda\_\{1\}\(\\Sigma\)\-\\lambda\_\{2\}\(\\Sigma\)\}\.In our notation, the eigengap isλ1​\(Σ\)−λ2​\(Σ\)=σsig2−σnoise2=δΣ\\lambda\_\{1\}\(\\Sigma\)\-\\lambda\_\{2\}\(\\Sigma\)=\\sigma\_\{\\mathrm\{sig\}\}^\{2\}\-\\sigma\_\{\\mathrm\{noise\}\}^\{2\}=\\delta\_\{\\Sigma\}\. So,

sin⁡∠​\(u^1,v\)≤‖Σ^−Σ‖opδΣ\.\\sin\\angle\(\\hat\{u\}\_\{1\},v\)\\leq\\frac\{\\\|\\hat\{\\Sigma\}\-\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\delta\_\{\\Sigma\}\}\.The squared cosine of the angle is what we need:

\|u^1⊤​v\|2=cos2⁡∠​\(u^1,v\)=1−sin2⁡∠​\(u^1,v\)≥1−\(‖Σ^−Σ‖opδΣ\)2\.\|\\hat\{u\}\_\{1\}^\{\\top\}v\|^\{2\}=\\cos^\{2\}\\angle\(\\hat\{u\}\_\{1\},v\)=1\-\\sin^\{2\}\\angle\(\\hat\{u\}\_\{1\},v\)\\geq 1\-\\left\(\\frac\{\\\|\\hat\{\\Sigma\}\-\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\delta\_\{\\Sigma\}\}\\right\)^\{2\}\.This completes the proof\. ∎

#### Analysis of the Signal\-to\-Noise Gap\.

The conditionδΣ\>0\\delta\_\{\\Sigma\}\>0is crucial\. We now connectσsig2\\sigma\_\{\\mathrm\{sig\}\}^\{2\}andσnoise2\\sigma\_\{\\mathrm\{noise\}\}^\{2\}to the hyperparameters\(N,T,K\)\(N,T,K\)\.

###### Lemma C\.8\(Residual mountain energy\)\.

Under the conditions of Theorem[C\.6](https://arxiv.org/html/2605.26484#A3.Thmtheorem6), the residual noise variance is bounded by:σnoise2=λmax​\(Σε,cent\)≤tr​\(Σε,cent\)≤𝔼​‖εs‖22=𝔼​D​\(w¯\(s\)\)2\.\\sigma\_\{\\mathrm\{noise\}\}^\{2\}=\\lambda\_\{\\max\}\(\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)\\leq\\mathrm\{tr\}\(\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)\\leq\\mathbb\{E\}\\\|\\varepsilon\_\{s\}\\\|\_\{2\}^\{2\}=\\mathbb\{E\}D\(\\bar\{w\}^\{\(s\)\}\)^\{2\}\.The RHS is given by the expression in Theorem[C\.6](https://arxiv.org/html/2605.26484#A3.Thmtheorem6)\.

###### Proof\.

For any positive semidefinite matrixAA, its largest eigenvalue \(operator norm\) is bounded by its trace:λmax​\(A\)≤tr​\(A\)\\lambda\_\{\\max\}\(A\)\\leq\\mathrm\{tr\}\(A\)\. Applying this toΣε,cent\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}:

σnoise2=λmax​\(Σε,cent\)≤tr​\(Σε,cent\)\.\\sigma\_\{\\mathrm\{noise\}\}^\{2\}=\\lambda\_\{\\max\}\(\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)\\leq\\mathrm\{tr\}\(\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)\.The trace is the sum of diagonal elements, which equals the expected squared Frobenius norm:

tr​\(Σε,cent\)=tr​\(𝔼​\[ε~s​ε~s⊤\]\)=𝔼​\[tr​\(ε~s​ε~s⊤\)\]=𝔼​\[‖ε~s‖22\]\.\\mathrm\{tr\}\(\\Sigma\_\{\\varepsilon,\\mathrm\{cent\}\}\)=\\mathrm\{tr\}\(\\mathbb\{E\}\[\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\]\)=\\mathbb\{E\}\[\\mathrm\{tr\}\(\\tilde\{\\varepsilon\}\_\{s\}\\tilde\{\\varepsilon\}\_\{s\}^\{\\top\}\)\]=\\mathbb\{E\}\[\\\|\\tilde\{\\varepsilon\}\_\{s\}\\\|\_\{2\}^\{2\}\]\.The centered vectorε~s=εs−1K​∑r=0K−1εr\\tilde\{\\varepsilon\}\_\{s\}=\\varepsilon\_\{s\}\-\\frac\{1\}\{K\}\\sum\_\{r=0\}^\{K\-1\}\\varepsilon\_\{r\}\. The sample meanε¯=1K​∑rεr\\bar\{\\varepsilon\}=\\frac\{1\}\{K\}\\sum\_\{r\}\\varepsilon\_\{r\}is the vector that minimizes the sum of squared distances∑s‖εs−c‖22\\sum\_\{s\}\\\|\\varepsilon\_\{s\}\-c\\\|\_\{2\}^\{2\}\. Therefore,𝔼​\[‖εs−ε¯‖22\]≤𝔼​\[‖εs−c‖22\]\\mathbb\{E\}\[\\\|\\varepsilon\_\{s\}\-\\bar\{\\varepsilon\}\\\|\_\{2\}^\{2\}\]\\leq\\mathbb\{E\}\[\\\|\\varepsilon\_\{s\}\-c\\\|\_\{2\}^\{2\}\]for any constant vectorcc\. Since the mountain process is stationary and zero\-mean,𝔼​\[εs\]=0\\mathbb\{E\}\[\\varepsilon\_\{s\}\]=0\. Choosingc=0c=0gives:

𝔼​\[‖ε~s‖22\]≤𝔼​\[‖εs‖22\]\.\\mathbb\{E\}\[\\\|\\tilde\{\\varepsilon\}\_\{s\}\\\|\_\{2\}^\{2\}\]\\leq\\mathbb\{E\}\[\\\|\\varepsilon\_\{s\}\\\|\_\{2\}^\{2\}\]\.Finally, by definition,εs=PS​\(w¯\(s\)−w⋆\)\\varepsilon\_\{s\}=P\_\{S\}\(\\bar\{w\}^\{\(s\)\}\-w\_\{\\star\}\), so its squared norm is the squared river deviation:

𝔼​\[‖εs‖22\]=𝔼​\[‖PS​\(w¯\(s\)−w⋆\)‖22\]=𝔼​D​\(w¯\(s\)\)2\.\\mathbb\{E\}\[\\\|\\varepsilon\_\{s\}\\\|\_\{2\}^\{2\}\]=\\mathbb\{E\}\[\\\|P\_\{S\}\(\\bar\{w\}^\{\(s\)\}\-w\_\{\\star\}\)\\\|\_\{2\}^\{2\}\]=\\mathbb\{E\}D\(\\bar\{w\}^\{\(s\)\}\)^\{2\}\.Combining these inequalities gives the desired result\. ∎

###### Lemma C\.9\(Signal variance from river drift\)\.

Under Assumption[C\.3](https://arxiv.org/html/2605.26484#A3.Thmtheorem3), letℓ′​\(t\)≈ℓ0′\\ell^\{\\prime\}\(t\)\\approx\\ell^\{\\prime\}\_\{0\}be the nearly constant river\-gradient component inUU\. The expected progress of a merged checkpoint per sliding step is approximately𝔼​\[ts\+1−ts\]≈−T​η​\(ℓ0′\+μF\)\\mathbb\{E\}\[t\_\{s\+1\}\-t\_\{s\}\]\\approx\-T\\eta\(\\ell^\{\\prime\}\_\{0\}\+\\mu\_\{F\}\)\. We define the magnitude of this step asΔt:=T​η​\(ℓ0′\+μF\)\\Delta\_\{t\}:=T\\eta\(\\ell^\{\\prime\}\_\{0\}\+\\mu\_\{F\}\)\. The signal variance is then lower\-bounded by:σsig2=Var​\(t~s\)≥Δt2⋅K2−112\.\\sigma\_\{\\mathrm\{sig\}\}^\{2\}=\\mathrm\{Var\}\(\\tilde\{t\}\_\{s\}\)\\geq\\Delta\_\{t\}^\{2\}\\cdot\\frac\{K^\{2\}\-1\}\{12\}\.

###### Proof\.

The river dynamics aretk\+1=tk−η​ℓ′​\(tk\)−η​μFt\_\{k\+1\}=t\_\{k\}\-\\eta\\ell^\{\\prime\}\(t\_\{k\}\)\-\\eta\\mu\_\{F\}\. With the approximationℓ′​\(tk\)≈ℓ0′\\ell^\{\\prime\}\(t\_\{k\}\)\\approx\\ell^\{\\prime\}\_\{0\}, this becomes an arithmetic progression:tk\+1≈tk−η​\(ℓ0′\+μF\)t\_\{k\+1\}\\approx t\_\{k\}\-\\eta\(\\ell^\{\\prime\}\_\{0\}\+\\mu\_\{F\}\)\. The position of thess\-th merged checkpoint ists=1N​∑i=0N−1tk0\+\(s\+i\)​Tt\_\{s\}=\\frac\{1\}\{N\}\\sum\_\{i=0\}^\{N\-1\}t\_\{k\_\{0\}\+\(s\+i\)T\}\. Its expectation is:

𝔼​\[ts\]≈1N​∑i=0N−1\(𝔼​\[tk0\]−\(s\+i\)​T​η​\(ℓ0′\+μF\)\)=C−s⋅T​η​\(ℓ0′\+μF\)=C−s​Δt\.\\mathbb\{E\}\[t\_\{s\}\]\\approx\\frac\{1\}\{N\}\\sum\_\{i=0\}^\{N\-1\}\\left\(\\mathbb\{E\}\[t\_\{k\_\{0\}\}\]\-\(s\+i\)T\\eta\(\\ell^\{\\prime\}\_\{0\}\+\\mu\_\{F\}\)\\right\)=C\-s\\cdot T\\eta\(\\ell^\{\\prime\}\_\{0\}\+\\mu\_\{F\}\)=C\-s\\Delta\_\{t\}\.The sequence of expectations\{𝔼​\[ts\]\}s=0K−1\\\{\\mathbb\{E\}\[t\_\{s\}\]\\\}\_\{s=0\}^\{K\-1\}is an arithmetic progression\. The variance of the centered process,σsig2=Var​\(t~s\)\\sigma\_\{\\mathrm\{sig\}\}^\{2\}=\\mathrm\{Var\}\(\\tilde\{t\}\_\{s\}\), is lower\-bounded by the variance of the centered means \(by the Law of Total Variance\):

σsig2=Var​\(t~s\)≥Var​\(𝔼​\[t~s\]\)=Var​\(𝔼​\[ts\]−1K​∑r𝔼​\[tr\]\)\.\\sigma\_\{\\mathrm\{sig\}\}^\{2\}=\\mathrm\{Var\}\(\\tilde\{t\}\_\{s\}\)\\geq\\mathrm\{Var\}\(\\mathbb\{E\}\[\\tilde\{t\}\_\{s\}\]\)=\\mathrm\{Var\}\(\\mathbb\{E\}\[t\_\{s\}\]\-\\frac\{1\}\{K\}\\sum\_\{r\}\\mathbb\{E\}\[t\_\{r\}\]\)\.This is the sample variance of the deterministic sequence\{C−s​Δt\}s=0K−1\\\{C\-s\\Delta\_\{t\}\\\}\_\{s=0\}^\{K\-1\}\. The constantCCdrops out, and we compute the variance of\{−s​Δt\}s=0K−1\\\{\-s\\Delta\_\{t\}\\\}\_\{s=0\}^\{K\-1\}:

Var​\(−s​Δt\)\\displaystyle\\mathrm\{Var\}\(\-s\\Delta\_\{t\}\)=Δt2​Var​\(s\)=Δt2​\(1K​∑s=0K−1s2−\(1K​∑s=0K−1s\)2\)\\displaystyle=\\Delta\_\{t\}^\{2\}\\mathrm\{Var\}\(s\)=\\Delta\_\{t\}^\{2\}\\left\(\\frac\{1\}\{K\}\\sum\_\{s=0\}^\{K\-1\}s^\{2\}\-\\left\(\\frac\{1\}\{K\}\\sum\_\{s=0\}^\{K\-1\}s\\right\)^\{2\}\\right\)=Δt2​\(\(K−1\)​\(2​K−1\)6−\(K−12\)2\)\\displaystyle=\\Delta\_\{t\}^\{2\}\\left\(\\frac\{\(K\-1\)\(2K\-1\)\}\{6\}\-\\left\(\\frac\{K\-1\}\{2\}\\right\)^\{2\}\\right\)=Δt2​\(2​\(K−1\)​\(2​K−1\)−3​\(K−1\)212\)\\displaystyle=\\Delta\_\{t\}^\{2\}\\left\(\\frac\{2\(K\-1\)\(2K\-1\)\-3\(K\-1\)^\{2\}\}\{12\}\\right\)=Δt2​\(K−1\)​\(4​K−2−3​K\+3\)12\\displaystyle=\\Delta\_\{t\}^\{2\}\\frac\{\(K\-1\)\(4K\-2\-3K\+3\)\}\{12\}=Δt2​\(K−1\)​\(K\+1\)12=Δt2​K2−112\.\\displaystyle=\\Delta\_\{t\}^\{2\}\\frac\{\(K\-1\)\(K\+1\)\}\{12\}=\\Delta\_\{t\}^\{2\}\\frac\{K^\{2\}\-1\}\{12\}\.This provides the lower bound for the signal variance\. ∎

Combining these lemmas, a sufficient condition forδΣ\>0\\delta\_\{\\Sigma\}\>0is:

\(T​η​\(ℓ0′\+μF\)\)2⋅K2−112⏟Signal \(LHS\)\>∑j=1d−1\(η​σ22​λj−η​λj2\)⋅1N2​\[N\+2​∑r=1N−1\(N−r\)​\(1−η​λj\)r​T\]⏟Noise \(RHS\)\.\\underbrace\{\\left\(T\\eta\(\\ell^\{\\prime\}\_\{0\}\+\\mu\_\{F\}\)\\right\)^\{2\}\\cdot\\frac\{K^\{2\}\-1\}\{12\}\}\_\{\\text\{Signal \(LHS\)\}\}\\;\>\\;\\underbrace\{\\sum\_\{j=1\}^\{d\-1\}\\left\(\\frac\{\\eta\\sigma^\{2\}\}\{2\\lambda\_\{j\}\-\\eta\\lambda\_\{j\}^\{2\}\}\\right\)\\cdot\\frac\{1\}\{N^\{2\}\}\\left\[N\+2\\sum\_\{r=1\}^\{N\-1\}\(N\-r\)\\big\(1\-\\eta\\lambda\_\{j\}\\big\)^\{rT\}\\right\]\}\_\{\\text\{Noise \(RHS\)\}\}\.This inequality clearly shows how the hyperparameters control the signal\-to\-noise gap:

- •IncreasingNNorTTsuppresses the noise term on the RHS\.
- •IncreasingKKor the drift\-per\-stepΔt\\Delta\_\{t\}\(viaTT\) amplifies the signal term on the LHS\.