不要过早停止长上下文持续预训练：一条经验缩放定律

# Don’t Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
Source: [https://arxiv.org/html/2610.29352](https://arxiv.org/html/2610.29352)
Ziqing Qiao††thanks:Equal contribution\.Xiaoyue Xu11footnotemark:1Affiliation:OpenBMBqzq24@mails\.tsinghua\.edu\.cn,\{xcj,han\-xu\}@tsinghua\.edu\.cnSikui ZhangAffiliation:OpenBMBqzq24@mails\.tsinghua\.edu\.cn,\{xcj,han\-xu\}@tsinghua\.edu\.cnYinuo Xu Bingxiang HeAffiliation:Department of Computer Science and Technology, Tsinghua University, Beijing, ChinaChaojun Xiao††thanks:Corresponding authors\.Affiliation:Department of Computer Science and Technology, Tsinghua University, Beijing, ChinaXu Han22footnotemark:2Affiliation:Department of Computer Science and Technology, Tsinghua University, Beijing, China

###### Abstract

Long\-context continual pre\-training \(LCCP\) is widely used to improve long\-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear\. We investigate this question through modeling, monitoring, and validating the benefits of scaling LCCP\. First, we establish an empirical additive power law relating long\-context performance to model size and the respective token budgets of short\-context pre\-training and LCCP\. Experiments across five model scales and more than one hundred short\- and long\-context budget configurations show that the fitted law reproduces the observed trends and extrapolates well to held\-out model scales\. Second, we introduceConstant\-Last4, a training and observation protocol that averages the four most recent consecutive checkpoints from a constant\-learning\-rate run to enable stable monitoring along a single extensible training trajectory\. Under this protocol, the same scaling\-law form fits well and accurately predicts the LCCP performance trajectory of a 1\.22B\-parameter model trained on approximately 100B LCCP tokens\. Finally, downstream evaluations along this trajectory show continued improvements in long\-context performance, with no clear sign of saturation within the evaluated range\. These gains persist after both 128K context extension and supervised fine\-tuning, with benefits extending to long\-context understanding and agentic tasks\. Together, these results argue for continuing to scale LCCP rather than stopping early: its gains are stably observable along an extensible trajectory, quantitatively predictable, and transferable to downstream tasks\.111The code is released at[https://github\.com/thunlp/dont\-stop\-lccp](https://github.com/thunlp/dont-stop-lccp)\.

## 1Introduction

Long\-context capability is fundamental to language models, supporting complex long\-horizon tasks and agentic workflows\([Kimi Team et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib52);[Zeng et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib51);[Liu et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib42)\)\. However, training on long sequences from the beginning of pre\-training incurs substantial computational costs\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib12)\)\. Long\-context continual pre\-training \(LCCP\) is therefore widely adopted, often as a brief stage following short\-context pre\-training\. It uses longer sequences and a long\-context\-oriented data mixture to extend the context window and improve long\-context performance\([Xiong et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib18);[Gao et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib17)\)\. A range of studies and models adopt relatively limited additional training to extend context windows\([Liu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib5);[Fu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib43);[Hu et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib59)\)\.

As applications increasingly demand long\-context understanding and sustained agentic interaction, some training recipes devote substantial token budgets to long\-context pre\-training\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib12);[Zeng et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib51)\)\. Yet how long\-context performance scales with the LCCP training budget remains insufficiently characterized\. We therefore ask:*Does continued LCCP yield sustained improvements in long\-context capability, and can these gains be quantitatively predicted?*

In this work, we address this question bymodeling, monitoring, and validatingthe benefits of scaling up LCCP\. As illustrated in Figure[1](https://arxiv.org/html/2610.29352#S1.F1), we conduct a systematic empirical study of how long\-context performance evolves with increasing LCCP budgets, spanning short\-context pre\-training, 32K LCCP, subsequent 128K context extension, and supervised fine\-tuning\(SFT\)\. We first establish an empirical scaling law formodelinglong\-context performance during LCCP, jointly accounting for model size, short\-context pre\-training tokens, and long\-context training tokens\. We then introduce a stable observation protocol for extensible LCCP, enabling performancemonitoringalong a constant\-learning\-rate trajectory and, together with the scaling law, prediction from early checkpoints\. Finally, we conduct downstream evaluations on long\-context and agentic tasks, therebyvalidatingwhether gains from additional LCCP translate into downstream improvements and persist through subsequent training stages\.

![Refer to caption](https://arxiv.org/html/2610.29352v1/overview.png)Figure 1:Overview of our framework for modeling, monitoring, and validating long\-context performance during LCCP scaling\. We develop an empirical scaling law for long\-context performance, introduceConstant\-Last4as an extensible training and observation protocol, and evaluate downstream performance gains after context extension and supervised fine\-tuning\.Modeling LCCP Scaling\.Unlike pre\-training from scratch, LCCP starts from a model that has already acquired basic capabilities, meaning that its long\-context performance may depend on the preceding short\-context pre\-training stage\([Wu et al\., 2025b](https://arxiv.org/html/2610.29352#bib.bib11)\)\. We therefore jointly model the non\-embedding parameter countNN, short\-context pre\-training tokensDsD\_\{s\}, and long\-context training tokensDlD\_\{l\}\. Using LongPPL\([Fang et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib2)\)as a proxy for long\-context performance, we evaluate 109 training configurations across five model scales under a linear\-decay learning\-rate schedule\. We find that an additive power law with an offset accurately captures the observed performance trends and extrapolates to held\-out model scales under the evaluated budget settings, providing a quantitative basis for predicting long\-context performance under different training budgets\.

Monitoring LCCP Scaling\.To study continued LCCP scaling, we need a training and observation protocol that enables consistent performance monitoring along a single extensible trajectory\. Although linear\-decay schedules provide stable endpoint performance, they require a predefined training budget and are therefore inconvenient for open\-ended training\. In contrast, a constant learning rate avoids this endpoint dependence, allowing cumulative long\-context training tokensDlD\_\{l\}to serve as a common coordinate for monitoring training progress\.

Raw constant\-learning\-rate checkpoints, however, exhibit substantial performance fluctuations and underperform budget\-matched linear\-decay models\. Motivated by prior work on training dynamics and checkpoint merging\([Wen et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib6);[Tian et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib7)\), we introduce theConstant\-Last4protocol, which averages the four most recent checkpoints for evaluation\. Across evaluated configurations, this protocol provides more stable observations and closely matches linear\-decay performance\. Under this protocol, the same scaling\-law form fits well and extrapolates from smaller models to the approximately 100B\-token LCCP trajectory of a 1\.22B model\. Moreover, when model size and the short\-context training budget are fixed, the law simplifies to a power law inDlD\_\{l\}that predicts later LongPPL from a fit to early LCCP checkpoints\. Together,Constant\-Last4and the empirical scaling law support stable monitoring and performance prediction when scaling up LCCP\.

Validating Capability Gains\.The practical value of the scaling law depends on whether the performance improvements it captures translate into gains in downstream long\-context and agentic tasks\. We therefore validate the scaling trend along the 32K LCCP trajectory using RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib33)\), where aggregate performance improves as the long\-context training budget increases, with no clear sign of saturation within the evaluated range\. Since LCCP is typically not the final training stage, we further evaluate checkpoints obtained at different LCCP budgets after 128K context extension and SFT for long\-context and agentic tasks, using identical training procedures within each setting\. The benefits of LCCP scaling remain observable after both subsequent training stages and across multiple long\-context and agentic benchmarks\. Together, these findings provide a quantitative basis for assessing the downstream benefits of continued LCCP\.

## 2Related Work

### 2\.1Scaling Laws for Large Language Models

Scaling laws relate pre\-training loss to model size and training data and characterize their trade\-offs under fixed compute budgets\([Kaplan et al\., 2020](https://arxiv.org/html/2610.29352#bib.bib35);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.29352#bib.bib4)\)\. Subsequent work has extended scaling\-law analysis to continual pre\-training, examining the effects of model size, additional training data, and data\-mixture ratios\([Que et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib23);[Gu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib24)\), and characterizing learning dynamics\([Wang et al\., 2025a](https://arxiv.org/html/2610.29352#bib.bib25)\), as well as to downstream performance prediction\([Ivgi et al\., 2022](https://arxiv.org/html/2610.29352#bib.bib21);[Hu et al\., 2024a](https://arxiv.org/html/2610.29352#bib.bib22);[Chen et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib27)\)\. However, scaling\-law analyses of long\-context performance remain comparatively limited\. Prior work has studied long\-context scaling in hybrid architectures under single\-stage pre\-training\([Qiao et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib1)\)\. We instead study LCCP, jointly modeling long\-context performance using model size and separate short\- and long\-context training budgets\.

### 2\.2Long\-Context Continual Pre\-Training

LCCP typically starts from a short\-context pre\-trained model and extends its context window through positional adjustments and continued training on longer sequences\([Xiong et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib18);[Chen et al\., 2023](https://arxiv.org/html/2610.29352#bib.bib20)\), using a long\-context\-oriented data mixture to improve performance\([Gao et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib17);[Fu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib43)\)\. Some methods emphasize token\-efficient context extension\([Wu et al\., 2025a](https://arxiv.org/html/2610.29352#bib.bib36);[Peng et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib26)\), whereas some leading large language models use long\-context training budgets on the order of a trillion tokens\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib12);[Liu et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib42)\)\. These differing budget choices highlight the importance of understanding the benefits of continued LCCP and assessing whether training is sufficient\.

Assessing the benefits of continued LCCP requires appropriate evaluation metrics\. LongPPL\([Fang et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib2)\)uses a reference model to identify tokens sensitive to distant context, providing a probabilistic measure of long\-context performance\. Recent work analyzes LCCP learning dynamics at the behavioral, probabilistic, and mechanistic levels and introduces a retrieval\-head\-based metric\([Liang et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib19)\)\. The findings suggest that behavioral scores can plateau while probabilistic and mechanistic metrics may continue to capture changes during training\. Complementing these analyses of learning dynamics, we focus on stable observation and quantitative prediction during continued LCCP\. Drawing on studies of learning\-rate dynamics in pre\-training\([Wen et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib6);[Hägele et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib41)\)and checkpoint averaging\([Tian et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib7);[Sanyal et al\., 2023](https://arxiv.org/html/2610.29352#bib.bib40)\), we developConstant\-Last4, an extensible protocol for LCCP training and LongPPL evaluation, and examine whether capability gains from additional LCCP persist after subsequent training stages\.

## 3An Empirical Scaling Law for LCCP

To quantify how long\-context performance scales during LCCP, we develop an empirical scaling law that jointly accounts for model size and the token budgets of the two training stages\. We evaluate both its fit to observed configurations and its predictive accuracy beyond the fitting range, across model scales and training budgets\.

### 3\.1Experimental Setup and Evaluation Metric

Long\-context performance after LCCP may depend not only on the amount of long\-context training, but also on the model’s short\-context pre\-training history\([Wu et al\., 2025b](https://arxiv.org/html/2610.29352#bib.bib11)\)\. This motivates treating the two stages separately rather than representing them by a single total token budget\. Specifically, we use three predictors: the non\-embedding parameter countNN, the number of short\-context pre\-training tokensDsD\_\{s\}, and the number of long\-context training tokensDlD\_\{l\}\.

Table 1:Model configurations\.To study LCCP scaling, we train models at five scales, S1–S5, withNNranging from 31M to 821M, as shown in Table[1](https://arxiv.org/html/2610.29352#S3.T1)\. As for the training budget, for S1–S4, we use

Ds/N\\displaystyle D\_\{s\}/N∈\{90,180,270,360,450\}\\displaystyle\\in\\\{90,180,270,360,450\\\}Dl/N\\displaystyle D\_\{l\}/N∈\{10,20,30,40,50\}\\displaystyle\\in\\\{10,20,30,40,50\\\}yielding 100 configurations\. For larger\-scale extrapolation, we construct five S5 configurations along the diagonalDs=9​DlD\_\{s\}=9D\_\{l\}, with\(Ds/N,Dl/N\)∈\{\(90,10\),\(180,20\),\(270,30\),\(360,40\),\(450,50\)\}\(D\_\{s\}/N,D\_\{l\}/N\)\\in\\\{\(90,10\),\(180,20\),\(270,30\),\(360,40\),\(450,50\)\\\}, and four off\-diagonal configurations,\(Ds/N,Dl/N\)∈\{\(90,50\),\(180,40\),\(360,20\),\(450,10\)\}\(D\_\{s\}/N,D\_\{l\}/N\)\\in\\\{\(90,50\),\(180,40\),\(360,20\),\(450,10\)\\\}, to provide broad coverage of the configuration grid\.

For the training setup, the short\-context stage uses a sequence length of 4K and the WSD learning\-rate schedule\([Hu et al\., 2024b](https://arxiv.org/html/2610.29352#bib.bib3)\)\. The ratio between the number of tokens in the warmup\+stable and decay phases is 8:1, andDsD\_\{s\}includes tokens from all WSD phases\. The LCCP stage uses a sequence length of 32K and a long\-context data mixture\. Following practices reported for open\-source models\([Olmo Team et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib9);[Bakouch et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib28)\), we use a linear\-decay learning\-rate schedule during LCCP to obtain stable model performance\. The learning rate starts from the value at the end of the short\-context stage and decays to one tenth of that value\. More details of training are provided in Appendix[A\.3](https://arxiv.org/html/2610.29352#A1.SS3)\.

For the fitting target, we use LongPPL\([Fang et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib2)\)as a proxy for long\-context performance\. Following the original protocol, we use Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib12)\)as the reference model and set the local chunk length to 4K tokens\. For each tokenxtx\_\{t\}, letℓreffull​\(xt\)\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)andℓrefchunk​\(xt\)\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{chunk\}\}\(x\_\{t\}\)denote the negative log\-likelihood under the full context and the local chunk, respectively\. We select tokens that are well predicted under the full context and depend substantially on distant context:

Δt=ℓrefchunk​\(xt\)−ℓreffull​\(xt\),𝒦long=\{t:ℓreffull​\(xt\)​<2,Δt\>​2\}\.\\Delta\_\{t\}=\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{chunk\}\}\(x\_\{t\}\)\-\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\),\\qquad\\mathcal\{K\}\_\{\\mathrm\{long\}\}=\\left\\\{t:\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)<2,\\;\\Delta\_\{t\}\>2\\right\\\}\.For an evaluated modelMM, LongPPL is computed from its full\-context loss only on𝒦long\\mathcal\{K\}\_\{\\mathrm\{long\}\}:

LongPPL⁡\(M\)=exp⁡\(1\|𝒦long\|​∑t∈𝒦longℓMfull​\(xt\)\)\.\\mathrm\{LongPPL\}\(M\)=\\exp\\left\(\\frac\{1\}\{\|\\mathcal\{K\}\_\{\\mathrm\{long\}\}\|\}\\sum\_\{t\\in\\mathcal\{K\}\_\{\\mathrm\{long\}\}\}\\ell\_\{M\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)\\right\)\.We use the same𝒦long\\mathcal\{K\}\_\{\\mathrm\{long\}\}set for all models\. The evaluation sequences contain 32K tokens and are constructed from GovReport\([Huang et al\., 2021](https://arxiv.org/html/2610.29352#bib.bib10)\)\. We compute LongPPL for all 109 training configurations and uselog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)as the target for subsequent scaling\-law fitting, consistent with previous work\([Qiao et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib1)\)\. Further details of the evaluation data are provided in Appendix[A\.1](https://arxiv.org/html/2610.29352#A1.SS1)\.

### 3\.2An Additive Two\-Stage Scaling Law

Figure 2:log⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)versusDsD\_\{s\}andDlD\_\{l\}, varying one budget while holding model size and the other budget fixed\. Both trends are consistent with power\-law decay with an offset\.Observation\.To guide the choice of functional form, we examine howlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)changes withDsD\_\{s\}orDlD\_\{l\}while holding the other two predictors fixed\. As shown in Figure[2](https://arxiv.org/html/2610.29352#S3.F2), the observed trends are consistent with power\-law decay with an offset across the tested model scales and budgets\. Guided by these trends and the additive structure of the Chinchilla scaling law\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.29352#bib.bib4)\), we consider the following formulation

log⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\+C,\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C,\(1\)whereAA,BsB\_\{s\}, andBlB\_\{l\}are scale coefficients,α\\alpha,βs\\beta\_\{s\}, andβl\\beta\_\{l\}are scaling exponents, andCCis an offset\. All seven parameters are fitted empirically\.

Fitting and scale extrapolation\.We evaluate the fitting and scale\-extrapolation performance of Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)using two held\-out\-scale experiments\. First, we fit the scaling law using the 75 configurations from S1–S3 and evaluate it on the 25 configurations from S4\. Second, we fit it using the 100 configurations from S1–S4 and evaluate it on the 9 configurations from S5\. In each experiment, we exclude configurations at the target model scale from fitting\. As shown in Figure[3](https://arxiv.org/html/2610.29352#S3.F3), Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)fits the observed data well and extrapolates to both held\-out model scales, achievingR2\>0\.95R^\{2\}\>0\.95and mean absolute percentage error \(MAPE\) below2%2\\%inlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)space\. Precise fitted parameter values are reported in Appendix[A\.5](https://arxiv.org/html/2610.29352#A1.SS5)\.

Figure 3:Scale\-extrapolation parity plots for fits from S1–S3 to S4 \(left\) and S1–S4 to S5 \(right\), with the fitted laws shown above each panel\. Dashed lines denote ideal predictions \(y=xy=x\)\.R2R^\{2\}and MAPE are computed inlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)space on held\-out configurations\.Notably, the fitted offset satisfiesC<0C<0\. Sincelog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)is a mean negative log\-likelihood and is nonnegative, we interpret Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)as a finite\-range empirical law rather than a valid asymptotic law\. Nevertheless, our extrapolation tests extend substantially beyond the fitting range: S5 has more than three times the parameter count of S4, the largest model used for fitting, with token budgets scaling by the same factor at matchedDs/ND\_\{s\}/NandDl/ND\_\{l\}/N\.

We also compare several alternative formulations, including variants with no offset or a nonnegative offset, formulations using effective data or the combined budgetD=Ds\+DlD=D\_\{s\}\+D\_\{l\}, and formulations with interaction terms among the three predictors\. Among these formulations, Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)attains the best overall extrapolation performance; details are provided in Appendix[A\.2](https://arxiv.org/html/2610.29352#A1.SS2)\. These comparisons provide additional empirical support for Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)as a predictive model of long\-context performance during LCCP within the evaluated range\.

## 4An Extensible Training and Observation Protocol

Under the linear\-decay setup, we successfully fit an additive power\-law model forlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\), which suggests that increasing the LCCP training budget can yield continued and predictable benefits within the studied range\. Although the linear\-decay schedule provides stable model performance, the learning\-rate schedule is tied to a prespecified training endpoint, making it less convenient for open\-ended LCCP scaling\. In this section, we therefore explore a training and observation protocol that supports both extensible training and stable performance evaluation, facilitating the application of the scaling law to LCCP scaling in practice\.

### 4\.1Constant\-Last4: Constant Learning Rate with Model Merging

We use a constant learning rate to support open\-ended LCCP without prespecifying a final training budget\([Liu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib5);[Blakeman et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib14)\)\. The cumulative number of long\-context training tokensDlD\_\{l\}then serves as a natural coordinate along a single training trajectory, which brings two practical benefits: additional long\-context data can be incorporated by resuming from the latest checkpoint, and model performance can be evaluated at any training budget without running a separate experiment for each budget\. To mitigate the elevated and fluctuating evaluation loss that can arise under constant\-learning\-rate training\([Wen et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib6)\), we consider checkpoint merging without additional training\([Tian et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib7)\)\. We treat merging as an*observation*mechanism: evaluation uses merged checkpoints, while the training trajectory remains unchanged\.

Observation rules and metrics\.We take the raw constant\-learning\-rate checkpoint as the simplest observation rule and compare it with a family of checkpoint\-merging methods\. As the reference for all rules, we use theLinear\-Decaycheckpoint trained with the same\(N,Ds,Dl\)\(N,D\_\{s\},D\_\{l\}\)\.

For judging observation rules, we retain LongPPL as the long\-context metric and introduce ShortPPL as a complementary diagnostic\. Reusing the same reference model and token\-level statistics as in LongPPL, we select tokens for which the local chunk yields a lower loss than the full context and compute perplexity over these positions:

𝒦short=\{t:Δt<0,ℓreffull\(xt\)<2\},ShortPPL\(M\)=exp\(1\|𝒦short\|∑t∈𝒦shortℓMfull\(xt\)\)\.\\mathcal\{K\}\_\{\\mathrm\{short\}\}=\\left\\\{t:\\Delta\_\{t\}<0,\\;\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)<2\\right\\\},\\qquad\\mathrm\{ShortPPL\}\(M\)=\\exp\\left\(\\frac\{1\}\{\|\\mathcal\{K\}\_\{\\mathrm\{short\}\}\|\}\\sum\_\{t\\in\\mathcal\{K\}\_\{\\mathrm\{short\}\}\}\\ell\_\{M\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)\\right\)\.Since the selected tokens do not benefit from distant context, ShortPPL primarily reflects general language\-modeling performance, and thus serves as a complementary metric\.

\(a\)Observation rules\(b\)Training and observation protocols
Figure 4:Comparison of training and observation protocols using ShortPPL and LongPPL\. \(a\) Different observation rules on S4 withDs=450​ND\_\{s\}=450NandDl=50​ND\_\{l\}=50N\. \(b\)Constant\-Raw,Constant\-Last4, and matchedLinear\-Decayacross 25 S4 configurations\.Constant\-Rawyields higher and noisier values for both metrics, whereasConstant\-Last4closely matchesLinear\-Decay\. CCC denotes Lin’s concordance correlation coefficient\([Lin, 1989](https://arxiv.org/html/2610.29352#bib.bib55)\)\.Comparison setup\.Starting from the S4 short\-context checkpoint withDs=450​ND\_\{s\}=450N, we continue constant\-learning\-rate training toDl=50​ND\_\{l\}=50Nand denote the resulting raw model byConstant\-Raw\. Along the trajectory, we save checkpoints at intervals of2​N2NinDlD\_\{l\}, yielding a pool of 25 checkpoints\. Within this pool we evaluate several representative merging methods:LastMM, which assigns equal weight to the finalMMconsecutive checkpoints\([Sanyal et al\., 2023](https://arxiv.org/html/2610.29352#bib.bib40)\), the recency\-weighted profiles1\-Sqrt\([Tian et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib7)\)andEMA, andDARE\([Yu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib13)\), a task\-vector method\. Additional implementation details are provided in Appendix[B](https://arxiv.org/html/2610.29352#A2)\.

Observation rule selection\.As shown in Figure[4\(a\)](https://arxiv.org/html/2610.29352#S4.F4.sf1),Constant\-Rawhas substantially higher LongPPL and ShortPPL than the budget\-matchedLinear\-Decaycheckpoint, whereas checkpoint merging recovers most of this gap on both metrics\. Considering performance, simplicity, and storage requirements, we selectLast4as the default observation rule: it averages only the four most recent consecutive checkpoints while preserving the latest raw checkpoint for continued training\.

To further assess the stability ofLast4merging, we compareConstant\-RawandConstant\-Last4withLinear\-Decayacross all S4 configurations in Figure[4\(b\)](https://arxiv.org/html/2610.29352#S4.F4.sf2)\. Across training budgets,Constant\-Last4closely matches the correspondingLinear\-Decaycheckpoint on both ShortPPL and LongPPL\. In summary,Constant\-Last4combines the extensibility of constant\-learning\-rate training with the stable performance of a decayed endpoint in our setting, providing a practical training and observation protocol for LCCP scaling\.

### 4\.2Scaling\-Law Validation under Constant\-Last4

For the newConstant\-Last4protocol, we need to test whether the empirical scaling law Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)remains valid under it\. We adopt a\(Ds/N,Dl/N\)\(D\_\{s\}/N,D\_\{l\}/N\)budget grid for S1–S4 similar to the one in Section[3](https://arxiv.org/html/2610.29352#S3)\. The realized token counts and details are provided in Appendix[A\.3](https://arxiv.org/html/2610.29352#A1.SS3.SSS0.Px3)\. Because a single constant\-learning\-rate trajectory provides checkpoints at multipleDlD\_\{l\}values, one run covers several long\-context training budgets and reduces the experimental cost relative toLinear\-Decay\. We run each LCCP trajectory independently three times from the same short\-context checkpoint and use the mean LongPPL across runs for S1–S4 models\.

We follow the scale\-extrapolation setup used in Section[3](https://arxiv.org/html/2610.29352#S3)\. First, we fit the scaling law on S1–S3 and evaluate it on S4\. We then fit it on S1–S4 and extrapolate to the S5 trajectory\. Whereas theLinear\-Decayexperiments used nine S5 configurations, here we extend pointwise scale extrapolation to prediction along an entire training trajectory\. Specifically, we fix the short\-context budget atDs=450​ND\_\{s\}=450Nand continue constant\-learning\-rate training toDl=120​ND\_\{l\}=120N\(approximately 100B tokens\)\. This setting provides dense observations oflog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)as a function ofDlD\_\{l\}at a fixed model scale and short\-context budget\. For S5, we save checkpoints at1​N1Nintervals and applyLast4to the four most recent checkpoints every5​N5N, yielding 24 merged evaluation points\. Appendix[B\.1](https://arxiv.org/html/2610.29352#A2.SS1)shows that the results are insensitive to checkpoint spacing within the evaluated range\.

As shown in Figure[5](https://arxiv.org/html/2610.29352#S4.F5), Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)fits theConstant\-Last4configurations well and extrapolates accurately to an unseen model scale\. Notably, the S5 trajectory reaches approximately 100B long\-context training tokens, more than eight times the largest absolute long\-context budget in the fitting set, yet the scaling law accurately predictslog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)across the entire trajectory\. Thus, the scaling law works under bothLinear\-DecayandConstant\-Last4within the tested range\.

Figure 5:Scaling\-law fitting and extrapolation underConstant\-Last4: held\-out S4 predictions from an S1–S3 fit \(left and middle\) and held\-out S5 trajectory predictions from an S1–S4 fit \(right\)\. In the middle panel, dot colors indicate the absolute percentage error between predicted and measuredlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)\. Precise fitted parameter values are reported in Appendix[A\.5](https://arxiv.org/html/2610.29352#A1.SS5)\.
### 4\.3A Simplified Law for Fixed\-Model LCCP

Figure 6:Prefix fitting oflog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)along the S5Constant\-Last4run\.In practical LCCP, one often starts from a fixed model that has undergone substantial short\-context pre\-training and continues training it with additional long\-context data\. In this setting, the main question is how performance evolves withDlD\_\{l\}and whether early observations can predict the benefits of further training\. WhenNNandDsD\_\{s\}are fixed, their contributions in Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)can be absorbed into a single offset, yielding

log⁡\(LongPPL\)=BlDlβl\+C′,\\log\(\\mathrm\{LongPPL\}\)=\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C^\{\\prime\},\(2\)whereC′=A/Nα\+Bs/Dsβs\+CC^\{\\prime\}=A/N^\{\\alpha\}\+B\_\{s\}/D\_\{s\}^\{\\beta\_\{s\}\}\+C\.We re\-estimateBlB\_\{l\},βl\\beta\_\{l\}andC′C^\{\\prime\}jointly from the given trajectory rather than taking them from the joint fit\. This lower\-dimensional form is practical for fixed\-model LCCP because it can be fitted using a small number of early checkpoints from a single training trajectory, avoiding the costly multi\-scale, multi\-budget training grid required for fitting the joint law\.

We evaluate the S5Constant\-Last4trajectory withDsD\_\{s\}fixed at450​N450NandDlD\_\{l\}extending to120​N120N, evaluating one merged checkpoint every5​N5N\. We fit Eq\.[2](https://arxiv.org/html/2610.29352#S4.E2)using Huber loss on prefixes up to20​N20N,40​N40N, and60​N60N, respectively, and evaluate each fit on all subsequent held\-out checkpoints through120​N120N\. As shown in Figure[6](https://arxiv.org/html/2610.29352#S4.F6), even a short prefix provides a reasonable estimate of the remaining trajectory, suggesting that early observations can help estimate later LongPPL for a fixed\-model LCCP trajectory\. We further fit the same power\-law form to ShortPPL and compare it with that of LongPPL to assess LongPPL’s usefulness as a long\-context proxy; details are provided in Appendix[D\.1](https://arxiv.org/html/2610.29352#A4.SS1)\.

## 5Downstream Validation of LCCP Scaling

For bothLinear\-DecayandConstant\-Last4, LongPPL decreases predictably as the LCCP budget increases\. To test whether this improvement translates into long\-context capability, we evaluate from three complementary perspectives: capability gains during 32K LCCP, their transfer to 128K context extension, and their preservation after SFT\. We conduct these evaluations using checkpoints sampled from the S5 32KConstant\-Last4trajectory introduced in Section[4\.2](https://arxiv.org/html/2610.29352#S4.SS2)\. Additional training and evaluation details for this section are provided in Appendix[C](https://arxiv.org/html/2610.29352#A3)\.

### 5\.1Capability Gains during 32K LCCP

Figure 7:RULER\-32K scores along the 32K LCCP trajectory of the S5 model\.To test whether the decrease in LongPPL translates into genuine long\-context capability, we evaluate the LCCP trajectory of the S5 model using RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib33)\)\. We construct 500 samples for each subtask and evaluate bothConstant\-RawandConstant\-Last4checkpoints every5​N5Nalong the trajectory\.

As shown in Figure[7](https://arxiv.org/html/2610.29352#S5.F7), the RULER scores ofConstant\-Last4exhibit an overall upward trend throughout the observed LCCP trajectory, with no clear sign of saturation\.Constant\-Rawfollows a similar trend but shows larger fluctuations and generally lower scores at comparable training budgets\. Detailed per\-subtask results in Appendix[C\.1](https://arxiv.org/html/2610.29352#A3.SS1)show that the aggregate gains are driven primarily by intermediate\-difficulty retrieval subtasks\. These results link continued LCCP scaling to improved long\-context benchmark performance within the evaluated range, withConstant\-Last4providing a more stable capability signal under a constant learning rate\.

### 5\.2Capability Gains after 128K Context Extension

Multi\-stage LCCP is common in practical training pipelines, where a model is first trained extensively at a moderate context length and then extended to a longer context with a smaller additional budget\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib12);[The Microsoft AI Team, 2026](https://arxiv.org/html/2610.29352#bib.bib34)\)\. To test whether gains from 32K LCCP transfer to a longer context, we select checkpoints atDl/N∈\{10,20,40,60,80,100,120\}D\_\{l\}/N\\in\\\{10,20,40,60,80,100,120\\\}from the previous S5 32K LCCP trajectory and apply the same 128K context\-extension procedure to bothConstant\-RawandConstant\-Last4models\. Specifically, we use ABF\([Xiong et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib18)\)to increase the RoPE base frequency from500,000500\{,\}000to5,000,0005\{,\}000\{,\}000, use a 128K\-stage data mixture with a higher proportion of long\-context sequences than the 32K LCCP mixture, and continue training for 2\.10B tokens at a sequence length of 128K\. The learning rate starts from the constant rate used during 32K LCCP and follows a linear decay to one hundredth of its initial value\.

For evaluation, we construct 100 128K\-length samples per RULER subtask\. As shown in Figure[8](https://arxiv.org/html/2610.29352#S5.F8)\(a\), the performance of bothConstant\-RawandConstant\-Last4models generally improves with increasing 32K budget, especially on NIAH\-related tasks\. These results suggest that the benefits of a larger 32K LCCP budget carry over to the longer\-context regime after context extension\.

Figure 8:Downstream performance after 128K extension and SFT at varying 32K LCCP budgets\. Shaded regions in \(b\) show the range across 3 training seeds\. Across these subsequent training stages, larger 32K LCCP budgets generally lead to improved downstream performance\.
### 5\.3Capability Gains after Supervised Fine\-Tuning

The goal of scaling LCCP is to improve downstream model performance\. Since models typically undergo SFT to adapt their pretrained capabilities to practical use, we evaluate whether the benefits of larger LCCP budgets persist after SFT\. We consider two settings, long\-context SFT and agentic SFT, using the same checkpoints of the 32K LCCP trajectory of the S5 model as in Section[5\.2](https://arxiv.org/html/2610.29352#S5.SS2)\. Within each setting, we apply the same SFT procedure to checkpoints obtained at different LCCP budgets\. More training details are provided in Appendices[A\.3](https://arxiv.org/html/2610.29352#A1.SS3)–[A\.4](https://arxiv.org/html/2610.29352#A1.SS4); evaluation protocols and supplementary results are provided in Appendix[C\.3](https://arxiv.org/html/2610.29352#A3.SS3)\.

Long\-context SFT\.We apply the same 2\.10B\-token SFT recipe to each checkpoint, using the non\-thinking response format\. We evaluate the SFT models on four long\-context benchmarks: LongBench\([Bai et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib29)\), GraphWalks\([OpenAI, 2025](https://arxiv.org/html/2610.29352#bib.bib32)\), LongReason\([Ling et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib30)\), and BABILong\([Kuratov et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib31)\), and report their equal\-weight average\. As shown in Figure[8](https://arxiv.org/html/2610.29352#S5.F8)\(b\), downstream performance generally improves with the LCCP training budget under the same SFT procedure\. Each point reports the mean over three independent SFT runs with different data seeds, with the shaded region indicating the minimum\-to\-maximum range across the three runs\.

Agentic SFT\.We further examine whether the benefits extend to agentic scenarios\. We consider two task categories: search agents, evaluated on GAIA\([Mialon et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib54)\)and BFCL\-WebSearch\([Patil et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib58)\), and tool\-use agents, evaluated onτ2\\tau^\{2\}\-telecom\([Barres et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib53)\)\. Each checkpoint undergoes the same 4\.20B\-token agentic SFT recipe with thinking mode enabled during training\. Each benchmark score is averaged over three evaluation runs, and the two search benchmark scores are further averaged with equal weights\. As shown in Figure[8](https://arxiv.org/html/2610.29352#S5.F8)\(c\), performance in both categories generally improves with larger LCCP budgets, despite fluctuations across checkpoints\.

Taken together, these evaluations show that, for the 1B\-scale S5 model, the benefits of scaling 32K LCCP to approximately 100B tokens remain evident after 128K context extension and after SFT for both long\-context and agentic tasks\. These results support further LCCP scaling as a promising direction for improving downstream capabilities when sufficient long\-context data are available\.

## 6Conclusion

We establish an empirical scaling law for LCCP that relates long\-context performance to model size and separate token budgets for short\-context pre\-training and LCCP, while supporting extrapolation to held\-out model scales and token budgets\. We introduceConstant\-Last4, enabling stable observation along an extensible trajectory\. In our 1B\-scale experiments, we observe continued improvements in long\-context performance along the evaluated 32K LCCP trajectory, up to≈100​B\\approx 100Btokens\. These gains remain after 128K context extension and SFT, including downstream long\-context and agentic tasks\. Taken together, these findings establish continued LCCP as a predictable and practically effective path toward improving long\-context and agentic capabilities\.

## AI use statement

We used generative AI tools to assist with translation, correct grammatical and stylistic issues, improve the clarity and academic presentation of the manuscript, refine scientific visualizations, and format LaTeX tables and figures\. We did not use generative AI to generate data, formulate scientific claims or hypotheses, design experiments or develop methods, conduct a literature review or perform qualitative analysis, or generate citations\. All AI\-assisted work was reviewed and verified by the authors\. The authors independently determined all interpretations and conclusions based on the original experimental results and take full responsibility for the final content of this work\.

## Ethics Statement

This work studies the scaling behavior of long\-context continual pre\-training\. Improved long\-context and agentic capabilities may amplify existing language\-model risks, including harmful outputs, bias, privacy leakage, and misuse of automated tools\. The use of large\-scale training corpora also warrants attention to data provenance, licensing, and potentially sensitive content\. Our capability evaluations do not establish deployment safety; downstream applications require task\-specific safety evaluation and appropriate safeguards\. Further scaling should weigh capability gains against these risks and the computational and environmental costs of training\.

## Reproducibility Statement

Sections[3](https://arxiv.org/html/2610.29352#S3)–[5](https://arxiv.org/html/2610.29352#S5)describe our experimental design, scaling\-law formulation, and evaluation protocols\. Appendix[A](https://arxiv.org/html/2610.29352#A1)documents metric computation and evaluation\-set construction, model and training configurations, realized token budgets, hardware, and training\-data composition and processing\. It also specifies the scaling\-law fitting procedure, held\-out evaluation splits, and fitted parameters\. Appendix[B](https://arxiv.org/html/2610.29352#A2)details the checkpoint\-merging methods and their comparison protocol\. Appendix[C](https://arxiv.org/html/2610.29352#A3)provides supplementary downstream results and evaluation settings, including the use of multiple data seeds or repeated evaluation runs where applicable\. These details support replication of our experimental procedures and results\.

## References

- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.Longbench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 3119–3137\.Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p2.1)\.
- Bakouchet al\.\(2025\)E\. Bakouch, L\. Ben Allal, A\. Lozhkov, N\. Tazi, L\. Tunstall, C\. M\. Patiño, E\. Beeching, A\. Roucher, A\. J\. Reedi, Q\. Gallouédec, K\. Rasul, N\. Habib, C\. Fourrier, H\. Kydlicek, G\. Penedo, H\. Larcher, M\. Morlon, V\. Srivastav, J\. Lochner, X\. Nguyen, C\. Raffel, L\. von Werra, and T\. WolfSmolLM3: smol, multilingual, long\-context reasoner\.Note:[https://huggingface\.co/blog/smollm3](https://huggingface.co/blog/smollm3)Cited by:[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p3.1)\.
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p3.1)\.
- Bercovichet al\.\(2025\)A\. Bercovich, I\. Levy, I\. Golan, M\. Dabbah, R\. El\-Yaniv, O\. Puny, I\. Galil, Z\. Moshe, T\. Ronen, N\. Nabwani,et al\.Llama\-nemotron: efficient reasoning models\.arXiv preprint arXiv:2505\.00949\.Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Blakemanet al\.\(2026\)A\. Blakeman, A\. Thomas, A\. Jhunjhunwala, A\. Gupta, A\. Khattar, A\. Rajfer, A\. Renduchintala, A\. Asif, A\. Vavre, A\. F\. Miranda,et al\.Nemotron 3 ultra: open, efficient mixture\-of\-experts hybrid mamba\-transformer model for agentic reasoning\.arXiv preprint arXiv:2606\.15007\.Cited by:[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p1.1)\.
- Chenet al\.\(2023\)S\. Chen, S\. Wong, L\. Chen, and Y\. TianExtending context window of large language models via positional interpolation\.arXiv preprint arXiv:2306\.15595\.Cited by:[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1)\.
- Chenet al\.\(2024\)Y\. Chen, B\. Huang, Y\. Gao, Z\. Wang, J\. Yang, and H\. JiScaling laws for predicting downstream performance in llms\.arXiv preprint arXiv:2410\.08527\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- De Gibertet al\.\(2024\)O\. De Gibert, G\. Nail, N\. Arefyev, M\. Bañón, J\. Van Der Linde, S\. Ji, J\. Zaragoza\-Bernabeu, M\. Aulamo, G\. Ramírez\-Sánchez, A\. Kutuzov,et al\.A new massive multilingual dataset for high\-performance language technologies\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 1116–1128\.Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Fanget al\.\(2025\)L\. Fang, Y\. Wang, Z\. Liu, C\. Zhang, S\. Jegelka, J\. Gao, B\. Ding, and Y\. WangWhat is wrong with perplexity for long\-context language modeling?\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 94541–94563\.Cited by:[§D\.1](https://arxiv.org/html/2610.29352#A4.SS1.p1.1),[§1](https://arxiv.org/html/2610.29352#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p4.1)\.
- Fuet al\.\(2024\)Y\. Fu, R\. Panda, X\. Niu, X\. Yue, H\. Hajishirzi, Y\. Kim, and H\. PengData engineering for scaling language models to 128k context\.arXiv preprint arXiv:2402\.10171\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1)\.
- Gaoet al\.\(2025\)T\. Gao, A\. Wettig, H\. Yen, and D\. ChenHow to train long\-context language models \(effectively\)\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7376–7399\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§1](https://arxiv.org/html/2610.29352#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p4.1),[§5\.2](https://arxiv.org/html/2610.29352#S5.SS2.p1.1)\.
- Guet al\.\(2024\)J\. Gu, Z\. Yang, C\. Ding, R\. Zhao, and F\. TanCmr scaling law: predicting critical mixture ratios for continual pre\-training of language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 16143–16162\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- Hägeleet al\.\(2024\)A\. Hägele, E\. Bakouch, A\. Kosson, L\. B\. Allal, L\. Von Werra, and M\. JaggiScaling laws and compute\-optimal training beyond fixed training durations\.Advances in Neural Information Processing Systems37,pp\. 76232–76264\.Cited by:[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p2.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.29352#S3.SS2.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p7.1),[§5\.1](https://arxiv.org/html/2610.29352#S5.SS1.p1.1)\.
- Huet al\.\(2024a\)S\. Hu, X\. Liu, X\. Han, X\. Zhang, C\. He, W\. Zhao, Y\. Lin, N\. Ding, Z\. Ou, G\. Zeng,et al\.Predicting emergent abilities with infinite resolution evaluation\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 18027–18047\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- Huet al\.\(2024b\)S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui, X\. Long, Z\. Zheng, Y\. Fang, Y\. Huang, W\. Zhao,et al\.Minicpm: unveiling the potential of small language models with scalable training strategies\.arXiv preprint arXiv:2404\.06395\.Cited by:[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p3.1)\.
- Huet al\.\(2025\)Z\. Hu, Y\. Liu, J\. Zhao, S\. Wang, Y\. Wang, W\. Shen, Q\. Gu, L\. A\. Tuan, S\. K\. Ng, Z\. Jiang,et al\.Longrecipe: recipe for efficient long context generalization in large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11857–11870\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1)\.
- Huanget al\.\(2021\)L\. Huang, S\. Cao, N\. Parulian, H\. Ji, and L\. WangEfficient attentions for long document summarization\.InProceedings of the 2021 conference of the north American chapter of the association for computational linguistics: Human language technologies,pp\. 1419–1436\.Cited by:[§A\.1](https://arxiv.org/html/2610.29352#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p4.3)\.
- Huanget al\.\(2025\)S\. Huang, T\. Cheng, J\. K\. Liu, W\. Xu, J\. Hao, L\. Song, Y\. Xu, J\. Yang, J\. Liu, C\. Zhang,et al\.Opencoder: the open cookbook for top\-tier code large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 33167–33193\.Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Ivgiet al\.\(2022\)M\. Ivgi, Y\. Carmon, and J\. BerantScaling laws under the microscope: predicting transformer performance from small scale experiments\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 7354–7371\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- Kimi Teamet al\.\(2026\)Kimi Team, T\. Bai, Y\. Bai, Y\. Bao, J\. Cai, X\. Cai, P\. Cao, Y\. Cao, Z\. Chai, Y\. Charles,et al\.Kimi k3: open frontier intelligence\.arXiv preprint arXiv:2607\.24653\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1)\.
- Kuratovet al\.\(2024\)Y\. Kuratov, A\. Bulatov, P\. Anokhin, I\. Rodkin, D\. Sorokin, A\. Sorokin, and M\. BurtsevBabilong: testing the limits of llms with long context reasoning\-in\-a\-haystack\.Advances in Neural Information Processing Systems37,pp\. 106519–106554\.Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p2.1)\.
- Lianget al\.\(2026\)Y\. Liang, S\. Chen, G\. Zhang, S\. Wang, and S\. ZhengRevealing the learning dynamics of long\-context continual pre\-training\.arXiv preprint arXiv:2604\.02650\.Cited by:[§D\.3](https://arxiv.org/html/2610.29352#A4.SS3.p1.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p2.1)\.
- Lin \(1989\)L\. I\. LinA concordance correlation coefficient to evaluate reproducibility\.\.Biometrics45 1,pp\. 255–68\.External Links:[Link](https://api.semanticscholar.org/CorpusID:32656801)Cited by:[Figure 4](https://arxiv.org/html/2610.29352#S4.F4)\.
- Linget al\.\(2025\)Z\. Ling, K\. Liu, K\. Yan, Y\. Yang, W\. Lin, T\. Fan, L\. Shen, Z\. Du, and J\. ChenLongreason: a synthetic long\-context reasoning benchmark via context expansion\.arXiv preprint arXiv:2501\.15089\.Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p2.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p1.1)\.
- Liuet al\.\(2025\)A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.Deepseek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1)\.
- Lozhkovet al\.\(2024\)A\. Lozhkov, R\. Li, L\. B\. Allal, F\. Cassano, J\. Lamy\-Poirier, N\. Tazi, A\. Tang, D\. Pykhtar, J\. Liu, Y\. Wei,et al\.Starcoder 2 and the stack v2: the next generation\.arXiv preprint arXiv:2402\.19173\.Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p3.1)\.
- MiroMind Teamet al\.\(2025\)MiroMind Team, S\. Bai, L\. Bing, C\. Chen, G\. Chen, Y\. Chen, Z\. Chen, Z\. Chen, J\. Dai, X\. Dong,et al\.Mirothinker: pushing the performance boundaries of open\-source research agents via model, context, and interactive scaling\.arXiv preprint arXiv:2511\.11793\.Cited by:[§C\.3](https://arxiv.org/html/2610.29352#A3.SS3.SSS0.Px2.p2.1)\.
- Nathawaniet al\.\(2025\)Nemotron\-Post\-Training\-Dataset\-v1External Links:[Link](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v1)Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Olmo Teamet al\.\(2025\)Olmo Team, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p3.1)\.
- OpenAI \(2025\)OpenAIGraphWalks: a multi hop reasoning long context benchmark\.Note:[https://huggingface\.co/datasets/openai/graphwalks](https://huggingface.co/datasets/openai/graphwalks)Released alongside GPT\-4\.1 \(April 2025\); see[https://openai\.com/index/gpt\-4\-1/](https://openai.com/index/gpt-4-1/)Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p2.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, C\. Cheng\-Jie Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§5\.3](https://arxiv.org/html/2610.29352#S5.SS3.p3.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, T\. Wolf,et al\.The fineweb datasets: decanting the web for the finest text data at scale\.Advances in Neural Information Processing Systems37,pp\. 30811–30849\.Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Penget al\.\(2024\)B\. Peng, J\. Quesnelle, H\. Fan, and E\. ShippoleYarn: efficient context window extension of large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 31932–31951\.Cited by:[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1)\.
- Qiaoet al\.\(2026\)Z\. Qiao, Y\. Xu, C\. Xiao, Z\. Su, Z\. Zhou, Y\. Chen, X\. Xu, X\. Han, and Z\. LiuRethinking the role of efficient attention in hybrid architectures\.arXiv preprint arXiv:2606\.15378\.Cited by:[§A\.2](https://arxiv.org/html/2610.29352#A1.SS2.SSS0.Px1.p1.1),[§D\.1](https://arxiv.org/html/2610.29352#A4.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p4.3)\.
- Queet al\.\(2024\)H\. Que, J\. Liu, G\. Zhang, C\. Zhang, X\. Qu, Y\. Ma, F\. Duan, Z\. Bai, J\. Wang, Y\. Zhang,et al\.D\-cpt law: domain\-specific continual pre\-training scaling law for large language models\.Advances in Neural Information Processing Systems37,pp\. 90318–90354\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§C\.3](https://arxiv.org/html/2610.29352#A3.SS3.SSS0.Px2.p4.1)\.
- Sanyalet al\.\(2023\)S\. Sanyal, A\. Neerkaje, J\. Kaddour, A\. Kumar, and S\. SanghaviEarly weight averaging meets high learning rates for llm pre\-training\.arXiv preprint arXiv:2306\.03241\.Cited by:[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p4.1)\.
- Songet al\.\(2026\)Y\. Song, J\. Kai, L\. Lu, K\. Qiu, and Z\. LinTowards compressive and scalable recurrent memory\.arXiv preprint arXiv:2602\.11212\.Cited by:[§D\.1](https://arxiv.org/html/2610.29352#A4.SS1.p1.1)\.
- The Microsoft AI Team \(2026\)The Microsoft AI TeamMAI\-thinking\-1: building a hill\-climbing machine\.Technical reportMicrosoft AI\.External Links:[Link](https://microsoft.ai/pdf/mai-thinking-1.pdf)Cited by:[§5\.2](https://arxiv.org/html/2610.29352#S5.SS2.p1.1)\.
- Tianet al\.\(2026\)C\. Tian, J\. Wang, Q\. Zhao, K\. Chen, J\. Liu, Z\. Liu, J\. Mao, X\. Zhao, Z\. Zhang, and J\. ZhouWSM: decay\-free learning rate schedule via checkpoint merging for llm pre\-training\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 134767–134795\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p6.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p4.1)\.
- Videauet al\.\(2026\)M\. Videau, B\. Youbi\-Idrissi, D\. Lopez\-Paz, and K\. AhujaSkaling: chinchilla’s exponents meet kaplan’s coupling\.External Links:2608\.07222,[Link](https://arxiv.org/abs/2608.07222)Cited by:[§A\.2](https://arxiv.org/html/2610.29352#A1.SS2.SSS0.Px3.p1.1)\.
- Wanget al\.\(2025a\)X\. Wang, H\. Tissue, L\. Wang, L\. Li, and D\. D\. ZengLearning dynamics in continual pre\-training for large language models\.arXiv preprint arXiv:2505\.07796\.Cited by:[§2\.1](https://arxiv.org/html/2610.29352#S2.SS1.p1.1)\.
- Wanget al\.\(2025b\)Y\. Wang, Z\. Fu, J\. Cai, P\. Tang, H\. Lyu, Y\. Fang, Z\. Zheng, J\. Zhou, G\. Zeng, C\. Xiao,et al\.Ultra\-fineweb: efficient data filtering and verification for high\-quality llm training data\.arXiv preprint arXiv:2505\.05427\.Cited by:[§A\.4](https://arxiv.org/html/2610.29352#A1.SS4.SSS0.Px1.p1.1)\.
- Wenet al\.\(2024\)K\. Wen, Z\. Li, J\. Wang, D\. Hall, P\. Liang, and T\. MaUnderstanding warmup\-stable\-decay learning rates: a river valley loss landscape perspective\.arXiv preprint arXiv:2410\.05192\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p6.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p1.1)\.
- Willetteet al\.\(2026\)J\. Willette, H\. Lee, and S\. J\. HwangDelta attention: fast and accurate sparse attention inference by delta correction\.Advances in Neural Information Processing Systems38,pp\. 12052–12080\.Cited by:[§D\.1](https://arxiv.org/html/2610.29352#A4.SS1.p1.1)\.
- Wuet al\.\(2025a\)L\. Wu, D\. Zhu, G\. Zhao, Z\. Yu, J\. Ran, X\. Wong, L\. Sun, and S\. LiLongattn: selecting long\-context training data via token\-level attention\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 19367–19380\.Cited by:[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1)\.
- Wuet al\.\(2025b\)W\. Wu, Y\. Wang, G\. Xiao, H\. Peng, and Y\. FuRetrieval head mechanistically explains long\-context factuality\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 62143–62156\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.29352#S3.SS1.p1.1)\.
- Xionget al\.\(2024\)W\. Xiong, J\. Liu, I\. Molybog, H\. Zhang, P\. Bhargava, R\. Hou, L\. Martin, R\. Rungta, K\. A\. Sankararaman, B\. Oguz,et al\.Effective long\-context scaling of foundation models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 4643–4663\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.29352#S2.SS2.p1.1),[§5\.2](https://arxiv.org/html/2610.29352#S5.SS2.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, Z\. Huang, W\. Zhu, Z\. Qiu, F\. Yuan, J\. Pan, and I\. TitovA controllable examination for long\-context language models\.Advances in Neural Information Processing Systems38\.Cited by:[§D\.3](https://arxiv.org/html/2610.29352#A4.SS3.p1.1)\.
- Yanoet al\.\(2026\)K\. Yano, S\. Kiyono, S\. Kobayashi, S\. Takase, and J\. SuzukiPre\-training llm without learning rate decay enhances supervised fine\-tuning\.arXiv preprint arXiv:2603\.16127\.Cited by:[§D\.2](https://arxiv.org/html/2610.29352#A4.SS2.p2.1)\.
- Yuet al\.\(2024\)L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. LiLanguage models are super mario: absorbing abilities from homologous models as a free lunch\.\.InIcml,Vol\.2,pp\. 21\.Cited by:[Appendix B](https://arxiv.org/html/2610.29352#A2.SS0.SSS0.Px2.p6.1),[§4\.1](https://arxiv.org/html/2610.29352#S4.SS1.p4.1)\.
- Zenget al\.\(2026\)A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie,et al\.Glm\-5: from vibe coding to agentic engineering\.arXiv preprint arXiv:2602\.15763\.Cited by:[§1](https://arxiv.org/html/2610.29352#S1.p1.1),[§1](https://arxiv.org/html/2610.29352#S1.p2.1)\.
- Zhenget al\.\(2024\)W\. Zheng, W\. Pan, X\. Xu, L\. Qin, L\. Yue, and M\. ZhouBreaking language barriers: cross\-lingual continual pre\-training at scale\.External Links:2407\.02118,[Link](https://arxiv.org/abs/2407.02118)Cited by:[§A\.2](https://arxiv.org/html/2610.29352#A1.SS2.SSS0.Px3.p2.1)\.

## Appendix AExperimental Setup and Reproducibility

### A\.1Details about LongPPL and ShortPPL

This section supplements the metric definitions in the main text and records statistics of the test set used for all PPL evaluations\. LongPPL and ShortPPL use the same GovReport documents\([Huang et al\., 2021](https://arxiv.org/html/2610.29352#bib.bib10)\), reference model, and full\-context forward pass; they differ only in the token positions over which losses are aggregated\.

#### Token selection\.

We precompute token annotations with Llama\-3\.1\-8B and its tokenizer and keep them fixed for every evaluated model\. As in the main text, let

Δt=ℓrefchunk​\(xt\)−ℓreffull​\(xt\),\\Delta\_\{t\}=\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{chunk\}\}\(x\_\{t\}\)\-\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\),
where the local context contains 4,096 tokens\. LongPPL selects positions that are easy for the reference model under full context but become substantially harder without distant context, whereas ShortPPL selects a complementary set of positions that are also easy under full context but have lower loss with only local context:

𝒦long\\displaystyle\\mathcal\{K\}\_\{\\mathrm\{long\}\}=\{t:ℓreffull​\(xt\)​<2,Δt\>​2\},\\displaystyle=\\left\\\{t:\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)<2,\\;\\Delta\_\{t\}\>2\\right\\\},𝒦short\\displaystyle\\mathcal\{K\}\_\{\\mathrm\{short\}\}=\{t:ℓreffull\(xt\)<2,Δt<0\}\.\\displaystyle=\\left\\\{t:\\ell\_\{\\mathrm\{ref\}\}^\{\\mathrm\{full\}\}\(x\_\{t\}\)<2,\\;\\Delta\_\{t\}<0\\right\\\}\.
The two sets are therefore disjoint\. At evaluation time, each document is decoded to text, retokenized with the evaluated model’s tokenizer, and the fixed character intervals are mapped to that model’s token positions\. This permits the same text spans to be used when the evaluated tokenizer differs from the reference tokenizer\.

#### Test\-set construction\.

The test set contains 10,000 GovReport documents\. Candidate documents contain 32,768–39,999 tokens under the Llama\-3\.1 tokenizer; each document is then retokenized with the evaluated model’s tokenizer and truncated to at most 32,768 tokens\. Because the tokenizers have different granularities, a small fraction of the retokenized sequences are shorter than this cap\. In the final evaluation inputs, 9,434 sequences contain exactly 32,768 tokens; across all 10,000 sequences, the mean and median lengths are 32,751 and 32,768 tokens, respectively\.

Table 2:Statistics of the GovReport PPL test set\. Document length is measured with the Llama\-3\.1 tokenizer; the remaining rows use the tokenizer shared by our evaluated models after truncation at 32,768 tokens\.As shown in Table[2](https://arxiv.org/html/2610.29352#A1.T2), Long key tokens account for 1\.42% of all evaluated tokens, compared with 25\.20% for Short key tokens\. Because the number of selected tokens varies substantially across documents, we compute both metrics by averaging negative log\-likelihoods over all selected tokens in the test set and then exponentiating, rather than weighting documents equally\.

### A\.2Alternative Scaling\-Law Formulations

We compare Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)with alternative offset constraints, data\-aggregation assumptions, and interaction terms on both theLinear\-DecayandConstant\-Last4data\.

Within each learning\-rate setting, every log\-space variant uses the same data, held\-out\-scale splits, sample weighting, exponent regularization, and multi\-start optimization as the corresponding main result: S4 is predicted from S1–S3, while S5 is predicted from S1–S4\.

For scaling\-law evaluation, letyi=log⁡\(LongPPLi\)y\_\{i\}=\\log\(\\mathrm\{LongPPL\}\_\{i\}\)andy^i\\hat\{y\}\_\{i\}denote its prediction\. We reportR2=1−∑i\(y^i−yi\)2/∑i\(yi−y¯\)2R^\{2\}=1\-\\sum\_\{i\}\(\\hat\{y\}\_\{i\}\-y\_\{i\}\)^\{2\}/\\sum\_\{i\}\(y\_\{i\}\-\\bar\{y\}\)^\{2\}andMAPE=100n​∑i\|y^i−yi\|/\|yi\|\\mathrm\{MAPE\}=\\frac\{100\}\{n\}\\sum\_\{i\}\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|/\|y\_\{i\}\|, with MAPE expressed as a percentage\. For numerical conditioning, defineN~=N/108\\widetilde\{N\}=N/10^\{8\},D~s=Ds/109\\widetilde\{D\}\_\{s\}=D\_\{s\}/10^\{9\}, andD~l=Dl/109\\widetilde\{D\}\_\{l\}=D\_\{l\}/10^\{9\}\. These normalizations are absorbed into the fitted amplitudes and do not change the function class of any formula\. For compactness, Tables[3](https://arxiv.org/html/2610.29352#A1.T3)and[4](https://arxiv.org/html/2610.29352#A1.T4)report only the held\-out scale\-extrapolation metrics\. As a diagnostic, across both training splits and all formulas, the unweighted training\-setR2R^\{2\}ranges from0\.96330\.9633to0\.99430\.9943on theLinear\-Decayand from0\.97400\.9740to0\.99540\.9954on theConstant\-Last4data\. The lower training\-fit values arise mainly from the combined\-budget and effective\-data variants\. These results show that training fit alone does not adequately distinguish the predictive performance of different formulations\. We therefore emphasize held\-out scale extrapolation when evaluating the proposed law and its alternatives\.

Table 3:Scale\-extrapolation results for formula ablations on theLinear\-Decaydata\. BothR2R^\{2\}and MAPE are computed inlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)space\. S5 contains five diagonal and four off\-diagonal budget configurations\. For the three powered\-pair variants,kkis allowed to vary over0<k≤20<k\\leq 2\.VariantFull formulaS4S5R2R^\{2\}MAPER2R^\{2\}MAPENo offsetlog⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}0\.79953\.62%\-0\.10319\.05%Nonnegative offsetlog⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\+C,C≥0\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C,\\ C\\geq 00\.79953\.62%\-0\.10319\.05%Free offset \(ours\)log⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C0\.96631\.43%0\.95161\.71%Combined budgetlog⁡\(LongPPL\)=ANα\+B\(Ds\+Dl\)β\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{\(D\_\{s\}\+D\_\{l\}\)^\{\\beta\}\}\+C0\.89352\.53%0\.84942\.40%Effective datalog⁡\(LongPPL\)=ANα\+B\(Dl\+K​Ds\)β\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{\(D\_\{l\}\+KD\_\{s\}\)^\{\\beta\}\}\+C0\.90792\.14%0\.82802\.94%Scale\-dependent effective datalog⁡\(LongPPL\)=ANα\+B\(Dl\+K​Nν​Ds\)β\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{\(D\_\{l\}\+KN^\{\\nu\}D\_\{s\}\)^\{\\beta\}\}\+C0\.09807\.68%0\.83463\.19%Powered pair \(NN–DsD\_\{s\}\)log⁡\(LongPPL\)=\(ANα\+BsDsβs\)k\+BlDlβl\+C,0<k≤2\\log\(\\mathrm\{LongPPL\}\)=\\left\(\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\\right\)^\{k\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C,\\ 0<k\\leq 20\.53115\.44%0\.79902\.82%Powered pair \(NN–DlD\_\{l\}\)log⁡\(LongPPL\)=\(ANα\+BlDlβl\)k\+BsDsβs\+C,0<k≤2\\log\(\\mathrm\{LongPPL\}\)=\\left\(\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\\right\)^\{k\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+C,\\ 0<k\\leq 20\.43706\.37%0\.45305\.22%Powered pair \(DsD\_\{s\}–DlD\_\{l\}\)log⁡\(LongPPL\)=ANα\+\(BsDsβs\+BlDlβl\)k\+C,0<k≤2\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\left\(\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\\right\)^\{k\}\+C,\\ 0<k\\leq 20\.79843\.58%0\.94881\.57%CPT interactionlog⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl​Nη\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}N^\{\\eta\}\}\+C0\.75574\.14%0\.71633\.50%

Table 4:The same formula ablations on theConstant\-Last4data, with bothR2R^\{2\}and MAPE computed inlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)space\. S4 contains a5×55\\times 5grid, whereas S5 contains 24 points that holdNNandDsD\_\{s\}fixed and vary onlyDlD\_\{l\}\. For the three powered\-pair variants,kkis allowed to vary over0<k≤20<k\\leq 2\.VariantFull formulaS4S5R2R^\{2\}MAPER2R^\{2\}MAPENo offsetlog⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}0\.93541\.89%\-4\.091711\.01%Nonnegative offsetlog⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\+C,C≥0\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C,\\ C\\geq 00\.93541\.89%\-4\.091711\.01%Free offset \(ours\)log⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C0\.97471\.08%0\.97180\.66%Combined budgetlog⁡\(LongPPL\)=ANα\+B\(Ds\+Dl\)β\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{\(D\_\{s\}\+D\_\{l\}\)^\{\\beta\}\}\+C0\.89011\.95%0\.13884\.02%Effective datalog⁡\(LongPPL\)=ANα\+B\(Dl\+K​Ds\)β\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{\(D\_\{l\}\+KD\_\{s\}\)^\{\\beta\}\}\+C0\.91551\.46%0\.79431\.49%Scale\-dependent effective datalog⁡\(LongPPL\)=ANα\+B\(Dl\+K​Nν​Ds\)β\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{\(D\_\{l\}\+KN^\{\\nu\}D\_\{s\}\)^\{\\beta\}\}\+C0\.47275\.45%\-0\.78786\.20%Powered pair \(NN–DsD\_\{s\}\)log⁡\(LongPPL\)=\(ANα\+BsDsβs\)k\+BlDlβl\+C,0<k≤2\\log\(\\mathrm\{LongPPL\}\)=\\left\(\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\\right\)^\{k\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\+C,\\ 0<k\\leq 20\.81903\.20%\-0\.30865\.49%Powered pair \(NN–DlD\_\{l\}\)log⁡\(LongPPL\)=\(ANα\+BlDlβl\)k\+BsDsβs\+C,0<k≤2\\log\(\\mathrm\{LongPPL\}\)=\\left\(\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\\right\)^\{k\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+C,\\ 0<k\\leq 20\.72684\.19%\-2\.32598\.86%Powered pair \(DsD\_\{s\}–DlD\_\{l\}\)log⁡\(LongPPL\)=ANα\+\(BsDsβs\+BlDlβl\)k\+C,0<k≤2\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\left\(\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}\}\\right\)^\{k\}\+C,\\ 0<k\\leq 20\.94541\.61%0\.08054\.58%CPT interactionlog⁡\(LongPPL\)=ANα\+BsDsβs\+BlDlβl​Nη\+C\\log\(\\mathrm\{LongPPL\}\)=\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\_\{s\}\}\{D\_\{s\}^\{\\beta\_\{s\}\}\}\+\\frac\{B\_\{l\}\}\{D\_\{l\}^\{\\beta\_\{l\}\}N^\{\\eta\}\}\+C0\.95371\.59%\-0\.28005\.34%

#### Offset constraints\.

A zero\-offset additive law, as used in some prior scaling analyses\([Qiao et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib1)\), has a plausible asymptotic interpretation: loss may approach zero as model and data scales grow, especially for the chosen predictable key tokens in LongPPL\. It is nevertheless substantially less accurate over the finite experimental range under both learning\-rate settings\. For both settings, constrainingC≥0C\\geq 0gives the same result as the zero\-offset law because the optimizer drivesCCto numerical zero\. The fitted values are1\.6×10−481\.6\\times 10^\{\-48\}and4\.6×10−374\.6\\times 10^\{\-37\}for theLinear\-DecayS1–S3 and S1–S4 fits, respectively, and2\.5×10−492\.5\\times 10^\{\-49\}and2\.0×10−362\.0\\times 10^\{\-36\}for theirConstant\-Last4counterparts\. AllowingCCto be free instead yieldsC=−0\.60C=\-0\.60andC=−0\.82C=\-0\.82for theLinear\-DecayS1–S3 and S1–S4 fits, respectively, andC=−0\.71C=\-0\.71andC=−0\.61C=\-0\.61for the correspondingConstant\-Last4fits\.

Importantly, a negative offset alone does not imply negative predictions within the relevant scaling regime, since the model\- and data\-dependent terms remain positive\. To stress\-test this behavior, Figure[9](https://arxiv.org/html/2610.29352#A1.F9)plots the zero contours of the S1–S4 fits at several extrapolated model sizes\. Finite zero contours emerge only at larger model scales and extreme data budgets\. For example, atN=10N=10B, settingDs=DlD\_\{s\}=D\_\{l\}gives a zero crossing at approximately11\.011\.0T tokens per stage forLinear\-Decayand29\.529\.5T tokens per stage forConstant\-Last4\. These budgets are orders of magnitude beyond the S1–S4 fitting range, whose maxima are approximately0\.110\.11T short\-context tokens and0\.0120\.012T long\-context tokens\. Predictions below zero implyLongPPL<1\\mathrm\{LongPPL\}<1, indicating that the fitted law has been extrapolated beyond its valid range\. The negative offset should therefore be interpreted as a finite\-range correction rather than an asymptotic loss floor\.

Figure 9:Zero contours of the free\-offset additive scaling law fitted on S1–S4\. The left and right panels show theLinear\-DecayandConstant\-Last4fits, respectively\. Each curve contains the\(Ds,Dl\)\(D\_\{s\},D\_\{l\}\)combinations for which the fittedlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)equals zero at the indicated non\-embedding parameter countNN\.
#### Combining the two training stages\.

We test whether short\-context pre\-training and LCCP data can be collapsed into a single data variable using three formulations\. The combined\-budget baseline,Ds\+DlD\_\{s\}\+D\_\{l\}, assumes that the two types of tokens are interchangeable\. The effective\-data formulation,Dl\+K​DsD\_\{l\}\+KD\_\{s\}, learns a fixed conversion factor between them, while the scale\-dependent formulation,Dl\+K​N~ν​DsD\_\{l\}\+K\\widetilde\{N\}^\{\\nu\}D\_\{s\}, allows this conversion factor to vary with model scale\.

All three formulations are less accurate than our additive law, which modelsDsD\_\{s\}andDlD\_\{l\}separately\. On the linear\-decay data, our law achievesRlog2=0\.9663R^\{2\}\_\{\\log\}=0\.9663on S4 and0\.95160\.9516on S5, compared with0\.89350\.8935and0\.84940\.8494for the combined\-budget baseline\. The effective\-data formulation improves some results but remains inferior overall, while the scale\-dependent formulation is unstable across extrapolation settings\. The same conclusion holds for the constant\-learning\-rate data, where our law achievesRlog2=0\.9747R^\{2\}\_\{\\log\}=0\.9747on S4 and0\.97180\.9718on S5 and outperforms all three alternatives\.

#### Interaction terms\.

We test two types of interaction terms to examine whether the additive law overlooks coupling among the scaling variables\. First, motivated by a previously proposed coupled form\([Videau et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib37)\), we replace each pairwise sum of the terms associated withNN,DsD\_\{s\}, andDlD\_\{l\}by\(xi\+xj\)k\(x\_\{i\}\+x\_\{j\}\)^\{k\}in turn, while leaving the third term additive\. The three resulting formulas are given explicitly in Tables[3](https://arxiv.org/html/2610.29352#A1.T3)and[4](https://arxiv.org/html/2610.29352#A1.T4)\. We constrain the coupling exponent to0<k≤20<k\\leq 2\. Across both datasets, both held\-out scales, and all three pairings, the fitted value reaches the upper bound to numerical precision\. The tested constrained fits provide no consistent extrapolation advantage over the additive law\.

Second, following the motivation of prior work on continued pretraining\([Zheng et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib38)\), we consider the interactionC\+A/Nα\+Bs/Dsβs\+Bl/\(Dlβl​Nη\)C\+A/N^\{\\alpha\}\+B\_\{s\}/D\_\{s\}^\{\\beta\_\{s\}\}\+B\_\{l\}/\(D\_\{l\}^\{\\beta\_\{l\}\}N^\{\\eta\}\)\. This formulation allows model size to modulate the contribution of LCCP data to long\-context learning\. This interaction is less accurate than the free\-offset additive law in all four extrapolation tests\. On the linear\-decay data, it achievesRlog2=0\.7557R^\{2\}\_\{\\log\}=0\.7557on S4 and0\.71630\.7163on S5\. On the constant\-learning\-rate data, it remains relatively accurate on S4, withRlog2=0\.9537R^\{2\}\_\{\\log\}=0\.9537, but deteriorates toRlog2=−0\.2800R^\{2\}\_\{\\log\}=\-0\.2800on S5\.

Overall, the tested interaction parameterizations provide no consistent extrapolation benefit\. This does not rule out interactions among the scaling variables, but indicates that these particular forms do not outperform the simpler additive law over the measured range\.

### A\.3Training Details

This section records the complete training procedure used for the S1–S5 scaling experiments and for the 128K extension and supervised fine\-tuning experiments in Sections[5\.2](https://arxiv.org/html/2610.29352#S5.SS2)and[5\.3](https://arxiv.org/html/2610.29352#S5.SS3)\. We report realized training token counts, rather than only their nominal multiples ofNN, since training budgets must be rounded to optimizer steps\. Below,NNdenotes the non\-embedding parameter count in Table[1](https://arxiv.org/html/2610.29352#S3.T1), and GBS/MBS denote the global and per\-device micro batch sizes in sequences\.

#### Common settings\.

All stages use the Muon optimizer, bfloat16 training, weight decay0\.10\.1, and gradient clipping at1\.01\.0\. Attention and hidden\-state dropout are both zero\. The 4K, 32K, and 128K stages use RoPE base frequencies of10,00010\{,\}000,500,000500\{,\}000, and5,000,0005\{,\}000\{,\}000, respectively, and the 128K stage uses full activation recomputation\. All runs were performed on two nodes of eight NVIDIA H800 GPUs each \(16 GPUs in total\)\.

#### 4K short\-context pre\-training\.

For every model scale, we train five WSD runs with realized budgets close to90​N90N,180​N180N,270​N270N,360​N360N, and450​N450N\. Each run consists of a linear warmup, a constant\-learning\-rate stable phase, and a linear decay phase; the warmup\-plus\-stable portion and the decay portion contain tokens in an8:18\{:\}1ratio, andDsD\_\{s\}counts both\. The decay phase reduces the learning rate linearly fromηstable\\eta\_\{\\mathrm\{stable\}\}toηend=ηstable/10\\eta\_\{\\mathrm\{end\}\}=\\eta\_\{\\mathrm\{stable\}\}/10; this terminal value is also the starting learning rate of the subsequent 32K stage\. Warmup lasts 1,000 optimizer steps at every scale except S1, where we use 500: at S1 a 1,000\-step warmup would consume0\.520\.52B tokens, or18\.5%18\.5\\%of the smallest \(90​N90N\) budget, whereas 500 steps gives a warmup fraction of9\.3%9\.3\\%, comparable to the9\.6%9\.6\\%at S2\. Table[5](https://arxiv.org/html/2610.29352#A1.T5)lists the realized budgets\.

Table 5:Short\-context pre\-training configurations\. All runs use a 4,096\-token sequence length and RoPE base10,00010\{,\}000\. Token counts are in billions and correspond, from left to right, to the nominal90​N90N,180​N180N,270​N270N,360​N360N, and450​N450Nbudgets\.
#### 32K long\-context continual pre\-training\.

When increasing the sequence length from 4K to 32K, we reduce the global batch size by a factor of eight so that the number of tokens per optimizer step is unchanged\. Every 32K run uses MBS 1 and a RoPE base500,000500\{,\}000\. TheLinear\-Decayruns have no warmup, begin at the terminal learning rate of 4K pre\-training, and linearly decay to one tenth of that rate\. For S1–S4, we train the full5×55\\times 5grid of short\- and long\-context budgets; for S5, we train the nine diagonal and off\-diagonal configurations described in Section[3](https://arxiv.org/html/2610.29352#S3)\.

TheConstant\-Rawtrajectories use the same batch sizes, data mixture, and starting learning rates as theLinear\-Decayruns, but keep the learning rate constant\. For S1, S2, S4, and S5, the trajectories are initialized from the same short\-context checkpoints as their correspondingLinear\-Decayruns\. For S3, we retrained the short\-context models because the original checkpoints were no longer available; the replacement runs have slightly different realized short\-context training budgets, as reported in Table[7](https://arxiv.org/html/2610.29352#A1.T7)\. S1–S4 are continued through approximately50​N50Nof long\-context data\. The S5 trajectory fixes the nominal short\-context budget atDs=450​ND\_\{s\}=450Nand continues LCCP throughDl=120​ND\_\{l\}=120N\. We evaluate this trajectory every5​N5N, starting atDl=5​ND\_\{l\}=5N, with adjacent checkpoints for merging spaced by1​N1N\. The realized budgets are summarized in Table[7](https://arxiv.org/html/2610.29352#A1.T7)\. All scaling\-law fits use the realized model sizes and token counts of the corresponding runs\.

Table 6:Linear\-Decay32K LCCP configurations\. All runs use a 32,768\-token sequence length, RoPE base500,000500\{,\}000, and no warmup\. Token counts are in billions and correspond, from left to right, to the nominal10​N10N,20​N20N,30​N30N,40​N40N, and50​N50Nbudgets\.Table 7:Constant\-Raw/Constant\-Last432K LCCP configurations\. Each row reports the realized short\- and long\-context token counts\. All runs use a 32,768\-token sequence length, RoPE base500,000500\{,\}000, and no warmup\. The identical start and end learning rates indicate a constant schedule\. Token counts are in billions;\\\\backslashdenotes an unavailable configuration\.ScaleNN\(M\)RealizedDsD\_\{s\}\(B\)ηstart\\eta\_\{\\mathrm\{start\}\}ηend\\eta\_\{\\mathrm\{end\}\}GBSMBSWarmupRealizedDlD\_\{l\}\(B\)10​N10N20​N20N30​N30N40​N40N50​N50N120​N120NS131\.4702\.83×10−59\.766\\\!\\times\\\!10^\{\-5\}×10−59\.766\\\!\\times\\\!10^\{\-5\}16100\.310\.630\.941\.261\.57\\\\backslash5\.660\.310\.630\.941\.261\.57\\\\backslash8\.490\.310\.630\.941\.261\.57\\\\backslash11\.320\.310\.630\.941\.261\.57\\\\backslash14\.160\.310\.630\.941\.261\.57\\\\backslashS260\.8455\.48×10−59\.542\\\!\\times\\\!10^\{\-5\}×10−59\.542\\\!\\times\\\!10^\{\-5\}16100\.601\.211\.822\.433\.04\\\\backslash10\.960\.601\.211\.822\.423\.04\\\\backslash16\.430\.591\.201\.812\.423\.04\\\\backslash21\.910\.591\.201\.812\.413\.04\\\\backslash27\.390\.581\.191\.802\.413\.04\\\\backslashS3104\.4279\.89×10−59\.766\\\!\\times\\\!10^\{\-5\}×10−59\.766\\\!\\times\\\!10^\{\-5\}32101\.122\.223\.324\.425\.49\\\\backslash19\.781\.142\.243\.344\.445\.49\\\\backslash29\.671\.162\.263\.364\.465\.49\\\\backslash39\.561\.182\.283\.384\.485\.49\\\\backslash49\.451\.202\.303\.404\.505\.49\\\\backslashS4245\.41722\.08×10−59\.766\\\!\\times\\\!10^\{\-5\}×10−59\.766\\\!\\times\\\!10^\{\-5\}32102\.454\.917\.369\.8112\.27\\\\backslash44\.172\.454\.917\.369\.8112\.27\\\\backslash66\.252\.454\.917\.369\.8112\.27\\\\backslash88\.332\.454\.917\.369\.8112\.27\\\\backslash110\.422\.454\.917\.369\.8112\.27\\\\backslashS5821\.147369\.94×10−59\.766\\\!\\times\\\!10^\{\-5\}×10−59\.766\\\!\\times\\\!10^\{\-5\}64108\.2216\.4424\.6632\.8841\.1098\.65

#### 128K context extension and supervised fine\-tuning\.

For the experiment in Section[5\.2](https://arxiv.org/html/2610.29352#S5.SS2), we apply the same 128K extension recipe to the selected S5Constant\-RawandConstant\-Last4checkpoints\. We increase the RoPE base to5,000,0005\{,\}000\{,\}000and use a data mixture with a larger fraction of sequences longer than 64K\. For long\-context SFT in Section[5\.3](https://arxiv.org/html/2610.29352#S5.SS3), every selected checkpoint undergoes the same 2\.10B\-token training stage on a mixture of general instruction\-following examples and high\-quality long\-context question\-answering data\. Because agentic SFT requires a larger capability shift from the preceding training stages, we double the training budget to 4\.20B tokens while retaining the same linear\-decay learning\-rate schedule\. All three stages use span\-based attention to isolate packed documents\. Table[8](https://arxiv.org/html/2610.29352#A1.T8)provides the remaining settings\.

Table 8:Training configurations applied after 32K LCCP\. The token counts are the realized values obtained from the listed sequence length, GBS, and number of optimizer steps\.

### A\.4Training Data

#### LCCP data\.

The LCCP corpus combines publicly available and internally curated data, primarily covering Chinese and English\. Public sources include filtered web text, such as FineWeb\- and HPLT\-derived collections\([Penedo et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib44);[Wang et al\., 2025b](https://arxiv.org/html/2610.29352#bib.bib45);[De Gibert et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib48)\); books and long\-form documents; academic papers and PDF text; legal, financial, and patent documents; source code and technical documentation, such as data derived from RefineCode and Stack v2\([Huang et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib47);[Lozhkov et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib46)\); and publicly released synthetic and instruction data from the Nemotron family\([Bercovich et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib50);[Nathawani et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib49)\)\. The same 32K mixture is used across all model scales and training\-budget configurations\.

#### Length\-aware LCCP mixtures\.

Documents are grouped by their tokenized source\-document lengths before sequence construction\. Table[9](https://arxiv.org/html/2610.29352#A1.T9)reports the configured sampling mass of each length bucket\. These values describe the source\-document lengths, rather than the lengths of the final training sequences, which are fixed at 32,768 and 131,072 tokens for the 32K and 128K stages, respectively\.

In the 32K mixture, documents shorter than 16K tokens account for approximately 52\.57% of the sampling mass, while documents shorter than 32K tokens account for approximately 65\.11%\. Documents of at least 32K tokens account for approximately 34\.89%\. For the 128K extension, the same broad source pool is retained while more sampling mass is assigned to longer documents\. The share of documents shorter than 16K tokens decreases to approximately 40\.60%, and the share of documents shorter than 32K tokens decreases to 50\.00%\. Accordingly, the share of documents of at least 32K tokens increases to 50\.00%\. More specifically, the share of documents in the 32K–128K range increases from 19\.80% to 28\.38%, while the share of documents of at least 128K tokens increases from 15\.09% to 21\.62%\.

Table 9:Aggregate LCCP sampling weights by tokenized source\-document length\. Values denote percentages of the configured sampling mass\.
#### LCCP data processing\.

Documents longer than the target sequence length are split into consecutive segments of the target length, starting from the beginning of each document, with a one\-token overlap between adjacent segments\. All resulting segments are retained rather than selecting a random window\. Instruction\-form examples that exceed the target length are filtered instead of segmented to preserve their supervision structure\. The 32K LCCP stage uses standard causal attention without document\-boundary masking, whereas the 128K extension uses span\-based attention to isolate packed documents\. We perform exact document\-level matching between the LCCP training corpus and the GovReport documents used for LongPPL evaluation, and find no exact matches\. This check is limited to exact document overlap\.

#### Supervised fine\-tuning data\.

The SFT mixture is fixed across all evaluated initializations and training budgets, including checkpoints obtained from different 32K LCCP budgets and the two checkpoint\-selection strategies\. For long\-context SFT, we use the non\-thinking response\-format variant\. Table[10](https://arxiv.org/html/2610.29352#A1.T10)reports the aggregate task\-family composition\. The long\-context component consists of history\-based question answering \(11\.52%\), multi\-hop reasoning \(2\.24%\), structured retrieval tasks \(4\.84%\), and additional long\-context reasoning data \(21\.40%\)\. For agentic SFT, we use the thinking response\-format variant\. The mixture consists of 60% search\-agent trajectories, in which the model interleaves reasoning with calls to a search engine, a web\-page extractor, and a Python sandbox, and 40% function and tool calling trajectories, ranging from single\-step tool selection to long\-horizon multi\-step tool chaining\. The loss is computed only on assistant turns\. SFT examples are packed into sequences of 32,768 tokens, with span\-based attention preventing attention across packed\-example boundaries\.

Table 10:Task\-level composition of the SFT mixture\. Percentages denote configured sampling mass\.

### A\.5Scaling\-Law Fitting Details

Eq\.[1](https://arxiv.org/html/2610.29352#S3.E1)contains seven parameters to estimate\. Although S1–S4 provide 100 configurations in total, a direct seven\-parameter nonlinear fit can be ill\-conditioned or nearly non\-identifiable: distinct parameter combinations can yield similar predictions over the observed range, making coefficients and exponents sensitive to noise and extrapolation unreliable\. We therefore use the experimental\-grid structure to stabilize estimation\.

At each model scale, the grid varies the short\- and long\-context budgets separately\. Every fixed\(N,Ds\)\(N,D\_\{s\}\)pair has five observations at distinctDlD\_\{l\}values, and every fixed\(N,Dl\)\(N,D\_\{l\}\)pair has five observations at distinctDsD\_\{s\}values\. WithNNandDsD\_\{s\}fixed, the remaining terms and offset are constant on theDlD\_\{l\}slice, so we fity=bl​Dl−βl\+cy=b\_\{l\}D\_\{l\}^\{\-\\beta\_\{l\}\}\+cto obtain a local estimate ofβl\\beta\_\{l\}\. We repeat this over all available slices in the training split and use the median as a training\-data\-driven reference; the analogous procedure yields the reference forβs\\beta\_\{s\}\. Taking the median limits the effect of a small number of anomalous slices\. These slice fits provide initialization and empirical thresholds for the exponent regularization; they do not replace the final joint fit\.

The joint fit uses Huber loss inlog⁡\(LongPPL\)\\log\(\\mathrm\{LongPPL\}\)space, with equal weight assigned to each training configuration\. We setf\_scale=0\.05by pilot fits on the training set before held\-out evaluation\. We apply the one\-sided exponent regularizer

λs​\[max⁡\(βs−β¯s,0\)\]2\+λl​\[max⁡\(βl−β¯l,0\)\]2,λs=λl=1\.\\lambda\_\{s\}\\bigl\[\\max\(\\beta\_\{s\}\-\\bar\{\\beta\}\_\{s\},0\)\\bigr\]^\{2\}\+\\lambda\_\{l\}\\bigl\[\\max\(\\beta\_\{l\}\-\\bar\{\\beta\}\_\{l\},0\)\\bigr\]^\{2\},\\qquad\\lambda\_\{s\}=\\lambda\_\{l\}=1\.The thresholdsβ¯s\\bar\{\\beta\}\_\{s\}andβ¯l\\bar\{\\beta\}\_\{l\}are obtained from the slice fits for the corresponding training split\. The regularizer penalizes above\-threshold exponents without fixing them or directly constraining the fitted coefficients and offsetCC\.

For each held\-out\-scale experiment, slice\-based exponent estimates, initialization settings, and regularization hyperparameters are determined exclusively from the training split\. Specifically, the S4 extrapolation uses only S1–S3 for these choices, whereas the S5 extrapolation uses only S1–S4\. The held\-out scale is used only for evaluation\.

Additionally, the one\-variable experiments in Section[4\.3](https://arxiv.org/html/2610.29352#S4.SS3)show that an axis exponent can be estimated reliably when the other two variables are fixed and sufficient observations are available along that axis\. For future fits of the same scaling law, the multipleDlD\_\{l\}observations on a singleConstant\-Last4trajectory can first define a reasonable range forβl\\beta\_\{l\}, followed by joint fitting\.

Table[11](https://arxiv.org/html/2610.29352#A1.T11)reports the parameters oflog⁡\(LongPPL\)=A​N−α\+Bs​Ds−βs\+Bl​Dl−βl\+C\\log\(\\mathrm\{LongPPL\}\)=AN^\{\-\\alpha\}\+B\_\{s\}D\_\{s\}^\{\-\\beta\_\{s\}\}\+B\_\{l\}D\_\{l\}^\{\-\\beta\_\{l\}\}\+C, whereNNis the non\-embedding parameter count andDs,DlD\_\{s\},D\_\{l\}are token counts, all expressed as absolute counts\. Figures and evaluation metrics use the unrounded parameters\.

Table 11:Fitted parameters of the joint scaling law\. LD denotesLinear\-Decay, and CL4 denotesConstant\-Last4\. Training scales are shown in parentheses\. Fits on S1–S3 predict S4; fits on S1–S4 predict S5\. Parameters are rounded to six decimal places\.

## Appendix BModel Merging

We provide implementation details for the checkpoint\-merging methods considered in Section[4\.1](https://arxiv.org/html/2610.29352#S4.SS1)\. For the observation\-rule comparison in Figure[4](https://arxiv.org/html/2610.29352#S4.F4)\(a\), all merging methods use the same pool of 25 checkpoints from the S4 constant\-learning\-rate trajectory withDs=450​ND\_\{s\}=450Nand a target budget ofDl=50​ND\_\{l\}=50N, ordered from the oldest checkpointθ1\\theta\_\{1\}to the newest checkpointθK\\theta\_\{K\}\. All methods in this comparison are evaluated on the same first 1,000 evaluation samples\.

#### Agreement with linear decay\.

For Figure[4](https://arxiv.org/html/2610.29352#S4.F4)\(b\), we measure agreement using Lin’s concordance correlation coefficient \(CCC\):

CCC=2​sx​ysx2\+sy2\+\(x¯−y¯\)2,\\mathrm\{CCC\}=\\frac\{2s\_\{xy\}\}\{s\_\{x\}^\{2\}\+s\_\{y\}^\{2\}\+\(\\bar\{x\}\-\\bar\{y\}\)^\{2\}\},wherexix\_\{i\}andyiy\_\{i\}are the perplexities of the matchedLinear\-Decayand constant\-learning\-rate models, respectively, across then=25n=25S4 configurations\. Here,x¯\\bar\{x\}andy¯\\bar\{y\}are sample means;sx2s\_\{x\}^\{2\},sy2s\_\{y\}^\{2\}, andsx​ys\_\{xy\}are the variances and covariance computed with denominatornn\. CCC is computed separately for ShortPPL and LongPPL, and forConstant\-RawandConstant\-Last4\.

#### Model merging methods\.

For a weighted parameter average, we write

θmerge=∑i=1Kwi​θi,wi≥0,∑i=1Kwi=1\.\\theta\_\{\\mathrm\{merge\}\}=\\sum\_\{i=1\}^\{K\}w\_\{i\}\\theta\_\{i\},\\qquad w\_\{i\}\\geq 0,\\qquad\\sum\_\{i=1\}^\{K\}w\_\{i\}=1\.\(3\)The representative deterministic weighting rules are illustrated in Figure[10](https://arxiv.org/html/2610.29352#A2.F10)\.

LastMM\.LastMMassigns equal weight to the finalMMconsecutive checkpoints and zero weight to all earlier checkpoints:

wi=\{1/M,i\>K−M,0,i≤K−M\.w\_\{i\}=\\begin\{cases\}1/M,&i\>K\-M,\\\\ 0,&i\\leq K\-M\.\\end\{cases\}\(4\)We evaluate several window sizes and useLast4, i\.e\., four weights of0\.250\.25, as the default method in the paper\. This rule performs a local trajectory average: it suppresses checkpoint\-level fluctuations while avoiding the stale parameters of much earlier checkpoints\.

1\-Sqrt\.Lettit\_\{i\}denote the training iteration of checkpointii, and define its normalized age relative to the newest checkpoint as

ai=tK−titK−t1\.a\_\{i\}=\\frac\{t\_\{K\}\-t\_\{i\}\}\{t\_\{K\}\-t\_\{1\}\}\.\(5\)The unnormalized and normalized1\-Sqrtweights are

w~i=1−ai,wi=w~i∑j=1Kw~j\.\\widetilde\{w\}\_\{i\}=1\-\\sqrt\{a\_\{i\}\},\\qquad w\_\{i\}=\\frac\{\\widetilde\{w\}\_\{i\}\}\{\\sum\_\{j=1\}^\{K\}\\widetilde\{w\}\_\{j\}\}\.\(6\)Thus, the oldest checkpoint receives zero weight, and the weight rises increasingly quickly near the end of the trajectory\. Unlike a rank\-only rule, our implementation uses the actual iteration gaps throughtit\_\{i\}\.

EMA\.We implement post\-hoc exponential moving averaging over all available checkpoints\. With one nominal checkpoint interval equal toΔ\\Deltaoptimizer iterations, its weights are

w~i=β\(tK−ti\)/Δ,wi=w~i∑j=1Kw~j\.\\widetilde\{w\}\_\{i\}=\\beta^\{\(t\_\{K\}\-t\_\{i\}\)/\\Delta\},\\qquad w\_\{i\}=\\frac\{\\widetilde\{w\}\_\{i\}\}\{\\sum\_\{j=1\}^\{K\}\\widetilde\{w\}\_\{j\}\}\.\(7\)We sweepβ∈\{0\.1,0\.5,0\.9\}\\beta\\in\\\{0\.1,0\.5,0\.9\\\}; Figure[10](https://arxiv.org/html/2610.29352#A2.F10)shows the representativeβ=0\.9\\beta=0\.9setting used in the main comparison\. A smallerβ\\betaconcentrates more mass on recent checkpoints, whereas a largerβ\\betaapproaches a more uniform average over the trajectory\.

Figure 10:Normalized checkpoint\-weight profiles on a common 25\-position axis, includingLast4,Last8,Last12,Last25,1\-Sqrt, andEMAwithβ=0\.9\\beta=0\.9\. Checkpoints are ordered from oldest to newest\.Beyond weighted parameter averaging, we evaluate two representative task\-vector merging methods,DAREandTIES\. For both methods, we use the oldest selected checkpoint as the referenceθ1\\theta\_\{1\}and define the task vector for each source checkpoint asδi=θi−θ1\\delta\_\{i\}=\\theta\_\{i\}\-\\theta\_\{1\}\.

DARE\.We evaluate theDARE\-linearvariant\([Yu et al\., 2024](https://arxiv.org/html/2610.29352#bib.bib13)\)\. For each coordinate of each task vector,DAREsamples an independent Bernoulli mask with retention probabilityρ\\rho, sets dropped coordinates to zero, and rescales retained coordinates by1/ρ1/\\rho\. We then average the sparsified task vectors and add the resulting update toθ1\\theta\_\{1\}\. We use random seed 42 and evaluateρ∈\{0\.1,0\.5\}\\rho\\in\\\{0\.1,0\.5\\\}\.

TIES\.For each task vector and parameter tensor,TIESretains the fractionρ\\rhoof coordinates with the largest absolute magnitudes and sets the remaining coordinates to zero\. It then elects a sign for each coordinate based on the sum of the trimmed task vectors, averages the nonzero updates whose signs agree with the elected sign, and adds the resulting update toθ1\\theta\_\{1\}\. We evaluateρ∈\{0\.1,0\.5\}\\rho\\in\\\{0\.1,0\.5\\\}\.

For visual clarity, Figure[4\(a\)](https://arxiv.org/html/2610.29352#S4.F4.sf1)focuses on the better\- performing merging methods\. We reportTIESseparately because its best\-performing configuration underperforms bothDAREand the finalConstant\-Rawcheckpoint\.

#### Interpreting the Advantage of Simple Averaging\.

TIES and DARE were developed primarily to control interference when combining task vectors from models with different fine\-tuning objectives or acquired capabilities\. Our setting is structurally different: all checkpoints lie on one continuous training trajectory, so their updates are strongly correlated and temporally nested rather than representing independent capability directions\. In this regime, local or recency\-weighted parameter averaging directly smooths trajectory noise while preserving the many small, consistently accumulated updates\. Magnitude trimming inTIESand random dropping inDAREcan remove part of this shared trajectory signal, while their use of the full checkpoint range also assigns influence to relatively stale states\. Consistent with this interpretation, the best testedTIESsetting \(ρ=0\.5\\rho=0\.5\) obtains a LongPPL of3\.79503\.7950, which is worse than the raw final checkpoint at3\.62333\.6233, while the best tested DARE setting \(ρ=0\.5\\rho=0\.5\) reaches3\.52173\.5217but remains behind1\-Sqrtat3\.47243\.4724andEMAwithβ=0\.9\\beta=0\.9at3\.47203\.4720\.

These results do not imply thatTIESorDAREis generally inferior\. Rather, they suggest that the mechanisms designed to resolve interference are less well suited to our setting, where checkpoints are highly correlated and lie along a single training trajectory\. In this regime, simple weighted averaging is more effective, stable, and inexpensive\.

### B\.1Sensitivity of Last4 to Checkpoint Spacing

The S5Constant\-Last4protocol in Section[4\.2](https://arxiv.org/html/2610.29352#S4.SS2)averages the four most recent checkpoints with equal weight and uses a1​N1Ngap between adjacent source checkpoints\. This choice balances evaluation stability against checkpoint storage, but it leaves open whetherLast4LongPPL depends on that1​N1Nspacing\.

We therefore run a spacing ablation on the same S5 32K constant\-learning\-rate trajectory, withDs=450​ND\_\{s\}=450Nand targets every10​N10NfromDl=20​ND\_\{l\}=20NtoDl=120​ND\_\{l\}=120N\. For each target budgetTTand spacings∈\{1​N,2​N,3​N,4​N,5​N\}s\\in\\\{1N,2N,3N,4N,5N\\\}, Last4\(ss\) uniformly averages the checkpoints\{T−3​s,T−2​s,T−s,T\}\\\{T\-3s,\\,T\-2s,\\,T\-s,\\,T\\\}\. For example, the1​N1Nwindow at20​N20Nuses17​N/18​N/19​N/20​N17N/18N/19N/20N, whereas the5​N5Nwindow uses5​N/10​N/15​N/20​N5N/10N/15N/20N\. As a control, we report the unmergedConstant\-Rawcheckpoint at the same target\. All protocols are evaluated on the same first1,0001\{,\}00032K samples for LongPPL, ShortPPL, and PPL, so differences among spacings are not due to a changing evaluation set\.

Figure 11:LAST4 window\-spacing ablation on the S5 32KConstanttrajectory\. \(a\) LongPPL versus long\-context budget\. \(b\) Relative difference from the defaultLast4\(1N\)window, computed as100×\(LongPPL−LongPPL1​N\)/LongPPL1​N100\\times\(\\mathrm\{LongPPL\}\-\\mathrm\{LongPPL\}\_\{1N\}\)/\\mathrm\{LongPPL\}\_\{1N\}\(%\)\.Table 12:Mean LongPPL and ShortPPL over the eleven target budgets\.Δ\\Deltais the original\-space difference against Last4\(1N\)\.Figure[11](https://arxiv.org/html/2610.29352#A2.F11)\(a\) and Table[12](https://arxiv.org/html/2610.29352#A2.T12)give the same conclusion\. Merging improves over Raw at every one of the eleven budgets: mean LongPPL falls from2\.2842\.284to about2\.202\.20\. By contrast, stretching the source spacing from1​N1Nto5​N5Nchanges the merged result only slightly\. The2​N2N–5​N5Nwindows have mean LongPPL0\.0080\.008to0\.0120\.012below the default1​N1Nwindow\. ShortPPL follows the same pattern: the improvement over Raw is about an order of magnitude larger than the differences among spacings\.

Thus, the1​N1NLast4 protocol used for S5 is not a fragile choice\. Within the tested1​N1N–5​N5Nrange, Last4 is insensitive to checkpoint spacing, and the stabilization gain of merging over Raw dominates residual effects of the window geometry\. Wider checkpoint spacing yields slightly lower perplexity on average, but the differences are small relative to the benefit of merging\.

The ablation covers only this S5 trajectory and perplexity metrics, so we read it as a robustness check of the main\-text1​N1NLast4 choice rather than as a universal guarantee for arbitrary spacings or downstream tasks\.

## Appendix CDownstream Evaluation Details

### C\.1RULER\-32K Results by Subtask

Section[5\.1](https://arxiv.org/html/2610.29352#S5.SS1)reports the aggregate RULER\-32K score along the S5 32K LCCP trajectory\. Figure[12](https://arxiv.org/html/2610.29352#A3.F12)shows the corresponding scores on all 13 subtasks for bothConstant\-RawandConstant\-Last4\.

Figure 12:Per\-subtask RULER\-32K performance along the S5 32K LCCP trajectory\. Scores are computed from 500 samples per subtask\.Figure 13:Per\-subtask RULER\-128K performance after the fixed 128K context\-extension stage\. The horizontal axis denotes the amount of 32K LCCP completed before context extension\. All results use 100 fixed samples per subtask\.The easiest retrieval tasks saturate early:niah\_single\_1remains at ceiling throughout training, whileniah\_single\_2andniah\_multikey\_1are already close to ceiling near the beginning of the trajectory\. The clearest gains instead occur on intermediate\-difficulty retrieval tasks\. UnderConstant\-Last4, the scores ofniah\_multikey\_2,niah\_multivalue, andniah\_multiqueryincrease from51\.2051\.20,10\.2010\.20, and18\.7518\.75at5​N5Nto71\.2071\.20,28\.7528\.75, and45\.0045\.00at120​N120N, respectively, despite checkpoint\-level fluctuations\. The correspondingConstant\-Rawcurves are less stable and show smaller endpoint gains\.

The improvement is not uniform across all capabilities\. The hardest multi\-key retrieval setting,niah\_multikey\_3, and the more compositional Variable Tracking and Common Words tasks remain near the bottom of their respective scales\. Scores on the QA tasks and Frequent Words also fluctuate rather than improve monotonically\. Despite this uneven progress, gains on several intermediate\-difficulty retrieval subtasks indicate continued benefits from additional LCCP\.

### C\.2RULER\-128K Results by Subtask

Similarly, Section[5\.2](https://arxiv.org/html/2610.29352#S5.SS2)reports aggregate RULER\-128K performance after applying the same 2\.10B\-token 128K context\-extension stage to checkpoints selected from the S5 32K LCCP trajectory\. We present the corresponding per\-subtask scores in Figure[13](https://arxiv.org/html/2610.29352#A3.F13)\.

The 128K results exhibit a pattern similar to the 32K evaluation in the previous section\. The score onniah\_single\_1remains at the ceiling of100100, while scores onniah\_single\_2andniah\_multikey\_1remain high across all checkpoints\. The most visible transfer gains occur onniah\_multikey\_2andniah\_multiquery\. As the preceding 32K LCCP budget increases from10​N10Nto120​N120N,niah\_multikey\_2rises from55to2020forConstant\-Rawand from88to2525forConstant\-Last4, reaching3333at the100​N100NLast4 checkpoint\.niah\_multiquerylikewise rises from23\.0023\.00to37\.5037\.50for Raw and from26\.7526\.75to36\.2536\.25for Last4, with a Last4 peak of45\.2545\.25at80​N80N\.

Other tasks remain substantially harder at 128K\. In particular,niah\_multikey\_3, Variable Tracking, and Common Words stay close to the floor, while Frequent Words and the two QA tasks vary across checkpoints without a reliable upward trend\. The benefits of additional 32K LCCP therefore vary across subtasks and are not consistently monotonic, but gains in multi\-key and multi\-query retrieval contribute to the improvement in the aggregate 128K score\.

### C\.3Evaluation after Long\-Context and Agentic SFT

#### Long\-context SFT\.

The experiment in Section[5\.3](https://arxiv.org/html/2610.29352#S5.SS3)applies a 2\.10B\-token SFT stage to checkpoints selected along the S5 32K LCCP trajectory\. For each starting checkpoint and checkpoint\-selection method, we train three SFT models using different data seeds\. We then evaluate each model on LongBench, GraphWalks, LongReason, and BABILong\. Figure[14](https://arxiv.org/html/2610.29352#A3.F14)reports the mean score over the three seeds for each benchmark separately, complementing the equal\-weight four\-task averages reported in the main text\.

Figure 14:Per\-task performance after 2\.10B tokens of SFT from checkpoints selected along the S5 32K LCCP trajectory for bothConstant\-Last4andConstant\-Raw\. Each point is the mean over three SFT runs trained with different data seeds\.To test whether this benefit persists with a larger SFT budget, we extend SFT to 4\.20B tokens \(4,000 steps\) from the sameConstant\-Last4andConstant\-Rawstarting checkpoints\. Apart from doubling the SFT duration, we keep the training and evaluation protocols consistent with the 2\.10B\-token experiment\. We evaluate LongBench, GraphWalks, LongReason, and BABILong and report both the individual scores and their equal\-weight average\. Due to the additional training cost, these 4,000\-step experiments use only one data seed and should therefore be interpreted as supporting evidence rather than a seed\-robust comparison\.

Figure 15:Performance after 4\.20B\-token SFT from S5 checkpoints at varying 32K LCCP budgets\. Individual benchmark scores and their equal\-weight average follow the 2\.10B\-token evaluation protocol\. These single\-data\-seed results serve as a supplementary robustness check\.Figure 16:GAIA and BFCL\-WebSearch performance after 4\.20B\-token agentic SFT along the S5 32K LCCP trajectory\. Each point averages three evaluation runs\.Figure[15](https://arxiv.org/html/2610.29352#A3.F15)provides preliminary evidence that the main conclusion persists after doubling the SFT budget\. ForConstant\-Last4, the four\-benchmark average initially decreases from26\.4926\.49at10​N10Nto25\.7425\.74at20​N20N, but then rises consistently to29\.9729\.97at120​N120N\. ForConstant\-Raw, the average fluctuates more across intermediate checkpoints but still increases from26\.4026\.40to29\.4929\.49over the same range\. Individual benchmarks remain noisy, so the gain is not uniformly monotonic on every task\. Nevertheless, the overall trend is consistent with the hypothesis that the benefits of additional LCCP persist when the SFT budget is doubled, while the single\-seed setting limits the strength of this conclusion\.

#### Agentic SFT\.

The agentic SFT experiment in Section[5\.3](https://arxiv.org/html/2610.29352#S5.SS3)applies a 4\.20B\-token SFT stage to checkpoints selected along the S5 32K LCCP trajectory\. We evaluate each model on three long\-horizon agentic benchmarks, covering two task types: search agent \(GAIA and the web search category of BFCL v4\(hereafter, BFCL\-WebSearch\)\) and multi\-turn tool calling \(τ2\\tau^\{2\}\-telecom\)\. In all three, the context accumulates over interactions and reaches tens of thousands of tokens \(Table[13](https://arxiv.org/html/2610.29352#A3.T13)\), exercising long\-context capability\. Inference uses the thinking response format with temperature 0\.6 and a maximum context length of 32,768 tokens, and we report the average score over three independent runs \(avg@3\)\.

For GAIA, we use the text\-only subset of the validation set, totaling 103 examples, and evaluate with the MiroThinker agent framework\([MiroMind Team et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib57)\)\. The agent runs a tool\-calling loop of up to 200 turns with web search, web\-page reading, and Python\-interpreter tools\. To fit long trajectories into the context budget, only the five most recent tool results are kept in the prompt while earlier ones are replaced by a placeholder\. Final answers are judged by GPT\-4\.1 against the reference\.

For BFCL\-WebSearch, the benchmark consists of 100 multi\-hop questions whose answers must be retrieved from the live web, and we follow the official protocol for evaluation\. The agent runs a native function\-calling loop of up to 20 steps with a web search tool and a web\-page reading tool\. We report the mean accuracy over its two settings \(base, where search results include text snippets, andno\-snippet, where the model must open web pages instead\)\.

Forτ2\\tau^\{2\}\-telecom, we evaluate on the telecom domain ofτ2\\tau^\{2\}\-bench, totaling 114 tasks\. The user is simulated by Qwen3\-235B\-A22B\-Instruct\-2507\([Qwen Team, 2025](https://arxiv.org/html/2610.29352#bib.bib56)\)with temperature 0, and each dialogue is capped at 100 steps\. A trial scores 1 only if all task\-specified checks on the final environment state and required tool calls pass, and 0 otherwise\.

Table[13](https://arxiv.org/html/2610.29352#A3.T13)summarizes the context lengths observed during evaluation\. For GAIA, we report both the final context \(the input of the last LLM call\) and the peak context \(the largest input over the course\)\. BFCL\-WebSearch andτ2\\tau^\{2\}\-telecom keep the full history in context, so the final context equals the peak\. BFCL trajectories exceeding the context limit are scored 0 and excluded from the table\.

Table 13:Context lengths \(tokens\) of agentic evaluation trajectories\.

## Appendix DAdditional Analyses

### D\.1Comparing LongPPL and ShortPPL Improvement Rates

Although LongPPL is widely used as a measure of long\-context performance\([Willette et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib16);[Song et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib15)\)and correlates well with a range of long\-context downstream tasks\([Qiao et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib1);[Fang et al\., 2025](https://arxiv.org/html/2610.29352#bib.bib2)\), it is still a perplexity\-based metric\. A lower LongPPL need not result exclusively from better use of distant context: it can also reflect broad next\-token prediction gains, adaptation to the evaluation domain, or changes in local token statistics\. We therefore compare changes in LongPPL and ShortPPL to examine whether tokens with different dependence on distant context improve at different rates during LCCP\.

We use ShortPPL as a matched control metric\. It is computed on tokens whose prediction has limited dependence on distant context, and thus provides a proxy for perplexity changes that do not require long\-range information\. ShortPPL is not a pure measure of general language\-modeling ability, but it helps capture broad improvements under the same model, data, and evaluation procedure\. Along the S5Constant\-Last4trajectory, we fit the same saturating power\-law form independently to both metrics over 24 checkpoints, fromDl=5​ND\_\{l\}=5NtoDl=120​ND\_\{l\}=120N\(approximately100100B tokens\):

log⁡M⁡\(Dl\)=EM\+AM​Dl−αM,M∈\{LongPPL,ShortPPL\}\.\\log M\(D\_\{l\}\)=E\_\{M\}\+A\_\{M\}D\_\{l\}^\{\-\\alpha\_\{M\}\},\\qquad M\\in\\\{\\mathrm\{LongPPL\},\\mathrm\{ShortPPL\}\\\}\.
For metricMM, we define the log\-scale elasticity

eM​\(Dl\)=\|∂log⁡M∂log⁡Dl\|\.e\_\{M\}\(D\_\{l\}\)=\\left\|\\frac\{\\partial\\log M\}\{\\partial\\log D\_\{l\}\}\\right\|\.For the fitted form, this iseM​\(Dl\)=αM​AM​Dl−αMe\_\{M\}\(D\_\{l\}\)=\\alpha\_\{M\}A\_\{M\}D\_\{l\}^\{\-\\alpha\_\{M\}\}, so it measures the relative rate of improvement at a given training budget\. We then compare the two metrics using

R⁡\(Dl\)=eLong​\(Dl\)eShort​\(Dl\)\.R\(D\_\{l\}\)=\\frac\{e\_\{\\mathrm\{Long\}\}\(D\_\{l\}\)\}\{e\_\{\\mathrm\{Short\}\}\(D\_\{l\}\)\}\.This ratio is a descriptive comparison: it indicates whether LongPPL is changing faster than the matched ShortPPL control, but it is not an exact causal decomposition of the total perplexity gain\.

Figure 17:Ratio of the log\-scale elasticities of LongPPL and ShortPPL along the S5Constant\-Last4LCCP trajectory\. Markers show measurements through \(Dl=120​ND\_\{l\}=120N\)\. Solid curves show fits over the measured range; dashed curves extend these fits to \(Dl=240​ND\_\{l\}=240N\)\.As shown in Figure[17](https://arxiv.org/html/2610.29352#A4.F17),R⁡\(Dl\)R\(D\_\{l\}\)is largest at the beginning of the trajectory and gradually decreases as LCCP proceeds\. Numerically, it declines from approximately3\.353\.35at5​N5Nto2\.222\.22at120​N120N, but remains above22throughout the observed range\. Thus, LongPPL improves substantially faster than ShortPPL, although the difference in the relative improvement rates of LongPPL and ShortPPL narrows as training proceeds\. In conjunction with the downstream evaluations in Section[5](https://arxiv.org/html/2610.29352#S5), the result supports using LongPPL as a useful proxy for long\-context performance during extended LCCP\.

### D\.2Effects of Checkpoint Merging across Training Stages

Table 14:Mean performance ofConstant\-RawandConstant\-Last4over comparable evaluation points\. The training stage identifies where each metric is evaluated\. For all rows,Δ=Last4−Raw\\Delta=\\textsc\{Last4\}\-\\textsc\{Raw\}; negative values indicate improvement for LongPPL, whereas positive values indicate improvement for other task scores\.During extended LCCP,Constant\-Last4lowers LongPPL and improves average RULER\-32K performance relative toConstant\-Raw\. Table[14](https://arxiv.org/html/2610.29352#A4.T14)summarizes these differences and compares performance after subsequent training initialized from the corresponding raw and merged checkpoints\. After 128K context extension, models initialized fromConstant\-Last4outperform those initialized fromConstant\-Rawby0\.600\.60points on average on RULER\-128K\. Similarly, after 2\.10B tokens of SFT, the four\-benchmark average is0\.510\.51points higher forConstant\-Last4\. However, with a larger SFT budget of 4\.20B tokens, this difference narrows and slightly favorsConstant\-Rawby0\.160\.16points\. In agentic SFT,Constant\-Last4retains a small0\.150\.15\-point advantage overConstant\-Raw\. Thus, the advantage of checkpoint merging observed during LCCP does not consistently persist through subsequent training\.

A related finding is that learning\-rate decay can improve pre\-training metrics while yielding worse performance after SFT than training without decay\([Yano et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib8)\)\. Although this concerns learning\-rate schedules rather than checkpoint merging, it similarly highlights that better performance before SFT need not imply better performance afterward\. Our limited experiments do not establish when merging advantages persist\. One tentative hypothesis is that these advantages may survive a modest amount of additional training or with a similar objective, as in context extension, but diminish or reverse with longer training or larger shifts in training data and objectives, as in SFT\. Further controlled experiments are needed to distinguish these factors\.

We therefore primarily view checkpoint merging as a stable*observation*mechanism for assessing long\-context performance along an extensible constant\-learning\-rate trajectory, rather than as a strategy for consistently improving subsequent training outcomes\. In this protocol, merging is used for evaluation, while continued LCCP proceeds from the latest raw checkpoint\.

### D\.3Dynamics of the Retrieval\-Head during LCCP

Previous work suggests that retrieval\-head metrics can serve as mechanistic probes of model development over the course of LCCP\([Liang et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib19)\)\. Following the definition in that work, we measure the copy\-retrieval behavior of individual attention heads\. For each teacher\-forced answer token, a head records a retrieval hit when its maximum\-attended input position lies within the reference sentence and contains the same token\_id as the current target token\. The per\-instance score of a head is the fraction of answer tokens producing such hits, and its checkpoint\-level retrieval score is obtained by averaging the per\-instance scores over our 2,400 LongBio single\-needle instances\([Yang et al\., 2026](https://arxiv.org/html/2610.29352#bib.bib39)\)\. A head is classified as a retrieval head if its checkpoint\-level score exceeds0\.10\.1\. For longitudinal comparison, we identify 10 retrieval heads at the raw5​N5Ncheckpoint and track the mean score of this fixed set at all subsequent checkpoints\. The retrieval\-head count, by contrast, is recomputed independently at each checkpoint as the number of heads whose checkpoint\-level score exceeds0\.10\.1\.

Figure 18:Retrieval\-head metrics and RULER\-32K performance along the S5Constant\-Last4trajectory\. The three square panels show the fixed\-head retrieval score, retrieval\-head count, and RULER\-32K score\.As shown in Figure[18](https://arxiv.org/html/2610.29352#A4.F18), the retrieval\-head metrics capture substantial learning during the early stages of training\. The mean score of the fixed retrieval heads rises rapidly, while the number of heads above the threshold increases from 10 to 13–14\. After aroundDl=30​ND\_\{l\}=30N, however, the fixed\-head score reaches a noisy plateau and exhibits no sustained upward trend\. This plateau does not coincide with saturation in behavioral performance\. The RULER\-32K score continues to improve up toDl=120​ND\_\{l\}=120N, with additional gains on several multi\-key and multi\-query tasks\. These results suggest that the selected retrieval\-head probes do not capture all behavioral improvements that occur later in LCCP\. Retrieval\-head metrics are therefore useful as early\-stage diagnostic signals, but they should be considered alongside LongPPL and behavioral evaluations rather than used alone to determine whether LCCP training has saturated\.