When to Review: Spaced Repetition for Continual Pre-Training of Language Models

# When to Review: Spaced Repetition for Continual Pre-Training of Language Models
Source: [https://arxiv.org/html/2608.17530](https://arxiv.org/html/2608.17530)
Devesh BatraYoages Kumar MantriGeremy BantugGreig A CowanRaad KhraishiAffiliation:University College London\[0\.8em\] NatWest AI Research

###### Abstract

Continual pre\-training of large language models must acquire new information without erasing old knowledge\. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten\. We formulate continual pre\-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step\. We introduce Spaced Repetition Training \(SRT\), a continual learning framework inspired by cognitive science, which schedules sample\-rehearsal using the SuperMemo\-2 \(SM\-2\) algorithm\. SRT maintains per\-example review state, maps per\-example perplexity to a recall\-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged\. On temporally separated Wikipedia and code corpora, SRT improves the stability\-plasticity trade\-off, recovering 5 to 37 percentage points of old\-knowledge accuracy lost by naive continual pre\-training across model scales while preserving or improving new\-knowledge acquisition\. At larger scale, SRT preserves broad benchmark performance that naive continual pre\-training and uniform replay substantially degrade\. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal\.

## 1Introduction

Large language models are increasingly updated after pre\-training as facts change, software ecosystems evolve, and new domain corpora appear\([11](https://arxiv.org/html/2608.17530#bib.bib7);[15](https://arxiv.org/html/2608.17530#bib.bib8)\)\. Continual pre\-training \(CPT\) is cheaper than training from scratch, but updates on new data can erase previously learned knowledge: catastrophic forgetting\([22](https://arxiv.org/html/2608.17530#bib.bib1);[17](https://arxiv.org/html/2608.17530#bib.bib2)\)\. In LLMs, this forgetting can also degrade broad capabilities such as multitask reasoning and factual recall\([20](https://arxiv.org/html/2608.17530#bib.bib24);[15](https://arxiv.org/html/2608.17530#bib.bib8)\)\.

Replay is an attractive mitigation strategy because it is architecture\-agnostic and scales naturally to LLM training\([26](https://arxiv.org/html/2608.17530#bib.bib6);[4](https://arxiv.org/html/2608.17530#bib.bib9);[21](https://arxiv.org/html/2608.17530#bib.bib10)\)\. Most replay methods, however, operate at the mixture level: they choose how much historical data to include and then sample old examples uniformly or by a fixed curriculum\. This ignores heterogeneity in retention difficulty\. Some examples remain stable after one review, while others drift quickly under interference from new learning\. Uniform replay can therefore spend budget on already\-retained examples while missing fragile ones\.

We instead treat CPT as an adaptive review\-scheduling problem\. Drawing on spaced repetition and difficulty\-adaptive memory consolidation\([7](https://arxiv.org/html/2608.17530#bib.bib13);[3](https://arxiv.org/html/2608.17530#bib.bib14)\), we ask which examples should be reviewed at the current training step, not only how much old data should be mixed in\. Algorithms such as SuperMemo\-2 already operationalize this idea for human memory through per\-item review state\([29](https://arxiv.org/html/2608.17530#bib.bib15)\); we adapt the same principle to language model training\.

We introduceSpaced Repetition Training\(SRT\), a drop\-in scheduler for continual pre\-training\. Each replayable example maintains an ease factor, review count, interval, and due step\. At review time, per\-example perplexity is converted into a recall\-quality score that updates the next interval: difficult examples return sooner, while confidently retained examples receive longer gaps\. The scheduler applies symmetrically to historical examples for retention and new examples for consolidation, and it leaves the model architecture, objective, and optimizer unchanged\.

#### Our contributions are as follows:

- •We formulate replay\-based LLM continual pre\-training as adaptive review scheduling, motivated by spaced repetition and heterogeneous retention difficulty\.
- •We propose SRT, a sample\-level SuperMemo\-2 scheduler that converts per\-example perplexity into recall quality and schedules both old and new examples without changing the model, loss, or optimizer\.
- •We show that SRT mitigates catastrophic forgetting more effectively than naive CPT and uniform replay while maintaining new\-knowledge acquisition on temporally separated Wikipedia and code corpora, and that it preserves broad benchmark performance that both baselines degrade\. Additional vision and tabular experiments suggest that the scheduling principle generalises beyond language\.

## 2Related Work

#### Catastrophic forgetting\.

Sequential training can overwrite representations needed for earlier tasks\([22](https://arxiv.org/html/2608.17530#bib.bib1)\)\. Standard mitigation strategies include regularization, which protects important parameters\([17](https://arxiv.org/html/2608.17530#bib.bib2);[31](https://arxiv.org/html/2608.17530#bib.bib25)\); architectural isolation or expansion\([27](https://arxiv.org/html/2608.17530#bib.bib5);[14](https://arxiv.org/html/2608.17530#bib.bib26)\); and replay, which interleaves previous examples with new data\([26](https://arxiv.org/html/2608.17530#bib.bib6);[25](https://arxiv.org/html/2608.17530#bib.bib3)\)\. For LLM CPT, replay is especially practical because task boundaries are weak, parameter growth is undesirable, and gradient\-level regularization is expensive\. SRT remains within the replay family but shifts the design question from mixture size to per\-example review timing\.

#### Continual pre\-training of LLMs\.

Domain\-adaptive pre\-training can improve target\-domain performance while shifting models away from prior capabilities\([11](https://arxiv.org/html/2608.17530#bib.bib7)\)\. Recent CPT work studies learning\-rate rewarming and re\-decay, data mixtures, replay ratios, curricula, and temporal benchmarks\([15](https://arxiv.org/html/2608.17530#bib.bib8);[4](https://arxiv.org/html/2608.17530#bib.bib9);[21](https://arxiv.org/html/2608.17530#bib.bib10)\)\. Other studies show that continual updates can degrade established benchmarks such as multitask language understanding and factual recall\([20](https://arxiv.org/html/2608.17530#bib.bib24)\)\. These methods primarily tune optimization or mixture\-level replay\. SRT is complementary: given a replay budget, it decides which stored examples should be reviewed now\.

#### Human\-inspired replay\.

SuperMemo\-2 maps recall quality to expanding inter\-review intervals through per\-item state\([29](https://arxiv.org/html/2608.17530#bib.bib15)\)\. Recent neural methods import related ideas via self\-synthesized rehearsal, Leitner\-style queues, or forgetting\-curve replay\([13](https://arxiv.org/html/2608.17530#bib.bib27);[24](https://arxiv.org/html/2608.17530#bib.bib16);[8](https://arxiv.org/html/2608.17530#bib.bib17)\)\. SRT differs by maintaining SuperMemo\-2 state for individual training examples and deriving recall quality directly from per\-example perplexity computed in the standard forward pass, requiring no synthesis pipeline or held\-out probe set\.

## 3Method

### 3\.1Problem Setup

Letpθp\_\{\\theta\}denote a language model with parametersθ\\thetainitialized from a pre\-trained checkpoint\. We consider a continual pre\-training setting in which the model is updated on an incoming corpus𝒟new=\{xjnew\}\\mathcal\{D\}\_\{\\mathrm\{new\}\}=\\\{x\_\{j\}^\{\\mathrm\{new\}\}\\\}while also retaining access to a historical corpus𝒟old=\{xiold\}\\mathcal\{D\}\_\{\\mathrm\{old\}\}=\\\{x\_\{i\}^\{\\mathrm\{old\}\}\\\}representing knowledge acquired before the update\. The goal of continual pre\-training is not simply to minimize next\-token loss on𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}, but to acquire new information while preserving performance on𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}\. We express this as a weighted combination of evaluation risks,

minθ′⁡λ​ℛold​\(θ′\)\+\(1−λ\)​ℛnew​\(θ′\),\\min\_\{\\theta^\{\\prime\}\}\\;\\lambda\\,\\mathcal\{R\}\_\{\\mathrm\{old\}\}\(\\theta^\{\\prime\}\)\+\(1\-\\lambda\)\\,\\mathcal\{R\}\_\{\\mathrm\{new\}\}\(\\theta^\{\\prime\}\),\(1\)whereℛold\\mathcal\{R\}\_\{\\mathrm\{old\}\}andℛnew\\mathcal\{R\}\_\{\\mathrm\{new\}\}are evaluation risks on old and new data respectively, andλ∈\[0,1\]\\lambda\\in\[0,1\]controls the stability\-plasticity trade\-off\.

In practice, training operates under a finite budget of optimization steps and tokens\. At steptt, a mini\-batchℬt\\mathcal\{B\}\_\{t\}of sizeBBmust allocate exposure between old and new examples,

ℬt=ℬtold∪ℬtnew,\|ℬtold\|\+\|ℬtnew\|≤B\.\\mathcal\{B\}\_\{t\}=\\mathcal\{B\}^\{\\mathrm\{old\}\}\_\{t\}\\cup\\mathcal\{B\}^\{\\mathrm\{new\}\}\_\{t\},\\qquad\|\\mathcal\{B\}^\{\\mathrm\{old\}\}\_\{t\}\|\+\|\\mathcal\{B\}^\{\\mathrm\{new\}\}\_\{t\}\|\\leq B\.\(2\)Naive continual pre\-training sets\|ℬtold\|=0\|\\mathcal\{B\}^\{\\mathrm\{old\}\}\_\{t\}\|=0and spends the entire batch on𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}\. Replay\-based methods instead allocate a fraction of each batch to examples drawn from𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}\. The central design question is therefore not only how much of each batch to allocate to old data, but which specific examples should be drawn from each pool at steptt, given that retention difficulty varies across both corpora\.

### 3\.2Spaced Repetition Training

SRT addresses this question by maintaining review state and scheduling using the SuperMemo\-2 algorithm\([29](https://arxiv.org/html/2608.17530#bib.bib15)\)\. Each examplexi∈𝒟old∪𝒟newx\_\{i\}\\in\\mathcal\{D\}\_\{\\mathrm\{old\}\}\\cup\\mathcal\{D\}\_\{\\mathrm\{new\}\}is associated with a review state

si​\(t\)=\(Ei​\(t\),ni​\(t\),Ii​\(t\),di​\(t\)\),s\_\{i\}\(t\)=\\bigl\(E\_\{i\}\(t\),n\_\{i\}\(t\),I\_\{i\}\(t\),d\_\{i\}\(t\)\\bigr\),\(3\)whereEiE\_\{i\}is the ease factor controlling interval growth,nin\_\{i\}is the number of consecutive successful reviews,IiI\_\{i\}is the current inter\-review interval measured in training steps, anddid\_\{i\}is the next due step\. Following standard SuperMemo\-2 initialization, we setEi=2\.5E\_\{i\}=2\.5,ni=0n\_\{i\}=0, andIi=1I\_\{i\}=1, with initial due times staggered uniformly over the first few training steps to avoid reviewing all items simultaneously\. State is maintained symmetrically for examples in𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}and𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}, so SRT applies the same scheduling logic to historical examples \(where it determines replay timing\) and to incoming examples \(where it determines consolidation timing\)\.

#### Due\-set sampling\.

At training steptt, we define the sets of examples in𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}and𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}whose next review is due,

ℳtold\\displaystyle\\mathcal\{M\}\_\{t\}^\{\\mathrm\{old\}\}=\{xi∈𝒟old:di≤t\},\\displaystyle=\\\{x\_\{i\}\\in\\mathcal\{D\}\_\{\\mathrm\{old\}\}:d\_\{i\}\\leq t\\\},\(4\)ℳtnew\\displaystyle\\mathcal\{M\}\_\{t\}^\{\\mathrm\{new\}\}=\{xj∈𝒟new:dj≤t\}\.\\displaystyle=\\\{x\_\{j\}\\in\\mathcal\{D\}\_\{\\mathrm\{new\}\}:d\_\{j\}\\leq t\\\}\.\(5\)The training batch at stepttallocates⌊ρ​B⌋\\lfloor\\rho B\\rfloorslots to due old examples sampled fromℳtold\\mathcal\{M\}\_\{t\}^\{\\mathrm\{old\}\}and the remaining slots to due new examples sampled fromℳtnew\\mathcal\{M\}\_\{t\}^\{\\mathrm\{new\}\}, whereρ∈\[0,1\]\\rho\\in\[0,1\]is the target old\-exposure fraction\. Both old and new examples are therefore selected by the scheduler, not only those drawn from𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}\. If either due set is empty at steptt, the unused budget is released to the other stream, ensuring that scheduling never blocks training progress\.

#### Perplexity\-derived recall quality\.

When a selected examplexi=\(xi,1,…,xi,Ti\)x\_\{i\}=\(x\_\{i,1\},\\ldots,x\_\{i,T\_\{i\}\}\)is reviewed, we compute its token\-average negative log\-likelihood under the current model,

Li\(t\)=−1Ti∑k=1Tilogpθt\(xi,k∣xi,<k\),L\_\{i\}\(t\)=\-\\frac\{1\}\{T\_\{i\}\}\\sum\_\{k=1\}^\{T\_\{i\}\}\\log p\_\{\\theta\_\{t\}\}\(x\_\{i,k\}\\mid x\_\{i,<k\}\),\(6\)and its perplexityPPLi​\(t\)=exp⁡\(Li​\(t\)\)\\mathrm\{PPL\}\_\{i\}\(t\)=\\exp\(L\_\{i\}\(t\)\)\. Perplexity is computed during the standard forward pass and requires no auxiliary evaluation\. We convert perplexity into the discrete recall quality scoreqi​\(t\)∈\{1,2,3,4,5\}q\_\{i\}\(t\)\\in\\\{1,2,3,4,5\\\}used by SuperMemo\-2 via monotonically increasing thresholds0<τ5<τ4<τ3<τ2<τ10<\\tau\_\{5\}<\\tau\_\{4\}<\\tau\_\{3\}<\\tau\_\{2\}<\\tau\_\{1\}that partition model performance into quality levels,

qi\(t\)=max\(0,5−∑k=15\[PPLi\(t\)≥τk\]\)\.q\_\{i\}\(t\)=\\max\\\!\\left\(0,\\;5\-\\sum\_\{k=1\}^\{5\}\\mathbf\{1\}\\\!\\left\[\\mathrm\{PPL\}\_\{i\}\(t\)\\geq\\tau\_\{k\}\\right\]\\right\)\.\(7\)Lower perplexity yields higher recall quality, withq=5q=5corresponding to fluent retention andq=0q=0to essentially no model on the example\. Threshold values were selected empirically; Section[6](https://arxiv.org/html/2608.17530#S6)analyses sensitivity to their scaling\.

The recall qualityqi​\(t\)q\_\{i\}\(t\)is computed from the training\-time loss at the current parametersθt\\theta\_\{t\}, that is, from the same forward pass used to obtain the gradient, before the optimizer step that producesθt\+1\\theta\_\{t\+1\}\. It therefore reflects how well the model retainedxix\_\{i\}prior to the current update, not how well it fitsxix\_\{i\}after being trained on it\.

For non\-generative models where perplexity is not defined, the same scheduler can be applied by replacingPPLi​\(t\)\\mathrm\{PPL\}\_\{i\}\(t\)with an alternative confidence\-based signal\. Details of this adaptation are provided in Appendix[C\.2](https://arxiv.org/html/2608.17530#A3.SS2)\.

#### Review state update\.

Given recall qualityqiq\_\{i\}, the ease factor is updated according to the SuperMemo\-2 rule,

Δ⁡\(qi\)\\displaystyle\\Delta\(q\_\{i\}\)=0\.1−\(5−qi\)​\(0\.08\+0\.02​\(5−qi\)\),\\displaystyle=0\.1\-\(5\-q\_\{i\}\)\\bigl\(0\.08\+0\.02\(5\-q\_\{i\}\)\\bigr\),\(8\)Ei\\displaystyle E\_\{i\}←max⁡\(1\.3,Ei\+Δ⁡\(qi\)\)\.\\displaystyle\\leftarrow\\max\\bigl\(1\.3,\\,E\_\{i\}\+\\Delta\(q\_\{i\}\)\\bigr\)\.\(9\)Ifqi<3q\_\{i\}<3, the example is treated as forgotten and we resetni←0n\_\{i\}\\leftarrow 0andIi←1I\_\{i\}\\leftarrow 1, scheduling immediate re\-review\. Otherwise we incrementnin\_\{i\}and update the interval according to

Ii←\{1,ni=1,6,ni=2,⌈Ii​Ei⌉,ni\>2\.I\_\{i\}\\leftarrow\\begin\{cases\}1,&n\_\{i\}=1,\\\\ 6,&n\_\{i\}=2,\\\\ \\lceil I\_\{i\}E\_\{i\}\\rceil,&n\_\{i\}\>2\.\\end\{cases\}\(10\)The next review is scheduled by settingdi←t\+Iid\_\{i\}\\leftarrow t\+I\_\{i\}\. The model parametersθ\\thetaare updated using standard next\-token cross\-entropy loss onℬt\\mathcal\{B\}\_\{t\}, identical to the underlying language modelling objective\. SRT therefore introduces no additional loss terms, no architectural changes, and no learned parameters beyond those already present inpθp\_\{\\theta\}\. The full procedure is summarized in Algorithm[1](https://arxiv.org/html/2608.17530#alg1)\.

Algorithm 1One SRT training step\.1:corpora

𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\},

𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}, old\-exposure cap

ρ\\rho, batch size

BB, step

tt
2:

ℳtold←\{xi∈𝒟old:di≤t\}\\mathcal\{M\}\_\{t\}^\{\\mathrm\{old\}\}\\leftarrow\\\{x\_\{i\}\\in\\mathcal\{D\}\_\{\\mathrm\{old\}\}:d\_\{i\}\\leq t\\\}⊳\\trianglerightdue old examples

3:

ℳtnew←\{xj∈𝒟new:dj≤t\}\\mathcal\{M\}\_\{t\}^\{\\mathrm\{new\}\}\\leftarrow\\\{x\_\{j\}\\in\\mathcal\{D\}\_\{\\mathrm\{new\}\}:d\_\{j\}\\leq t\\\}⊳\\trianglerightdue new examples

4:

bold←min⁡\(\|ℳtold\|,⌊ρ​B⌋\)b^\{\\mathrm\{old\}\}\\leftarrow\\min\\bigl\(\|\\mathcal\{M\}\_\{t\}^\{\\mathrm\{old\}\}\|,\\,\\lfloor\\rho B\\rfloor\\bigr\)
5:sample

ℬtold⊆ℳtold\\mathcal\{B\}\_\{t\}^\{\\mathrm\{old\}\}\\subseteq\\mathcal\{M\}\_\{t\}^\{\\mathrm\{old\}\}with

\|ℬtold\|=bold\|\\mathcal\{B\}\_\{t\}^\{\\mathrm\{old\}\}\|=b^\{\\mathrm\{old\}\}
6:sample

ℬtnew⊆ℳtnew\\mathcal\{B\}\_\{t\}^\{\\mathrm\{new\}\}\\subseteq\\mathcal\{M\}\_\{t\}^\{\\mathrm\{new\}\}to fill remaining slots

7:

ℬt←ℬtold∪ℬtnew\\mathcal\{B\}\_\{t\}\\leftarrow\\mathcal\{B\}\_\{t\}^\{\\mathrm\{old\}\}\\cup\\mathcal\{B\}\_\{t\}^\{\\mathrm\{new\}\}
8:for

xi∈ℬtx\_\{i\}\\in\\mathcal\{B\}\_\{t\}do⊳\\trianglerightscored atθt\\theta\_\{t\}, before the update

9:compute

PPLi​\(t\)\\mathrm\{PPL\}\_\{i\}\(t\)via Eq\.[6](https://arxiv.org/html/2608.17530#S3.E6)and

qi​\(t\)q\_\{i\}\(t\)via Eq\.[7](https://arxiv.org/html/2608.17530#S3.E7)

10:endfor

11:update

θ\\thetaon

ℬt\\mathcal\{B\}\_\{t\}using next\-token loss⊳\\trianglerightθt→θt\+1\\theta\_\{t\}\\rightarrow\\theta\_\{t\+1\}

12:for

xi∈ℬtx\_\{i\}\\in\\mathcal\{B\}\_\{t\}do⊳\\trianglerightupdate review state only

13:update

EiE\_\{i\},

nin\_\{i\},

IiI\_\{i\}via Eqs\.[9](https://arxiv.org/html/2608.17530#S3.E9)–[10](https://arxiv.org/html/2608.17530#S3.E10)

14:set

di←t\+Iid\_\{i\}\\leftarrow t\+I\_\{i\}
15:endfor

## 4Experimental Setup

### 4\.1Corpora

We construct temporally grounded corpora in two domains where knowledge evolves measurably over time: encyclopedic text and source code\. Both domains are split into an*old*corpus𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}representing information available before a temporal cutoff and a*new*corpus𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}representing post\-cutoff information, enabling evaluation that distinguishes retained knowledge from acquired knowledge\.

#### Wikipedia\.

For encyclopedic text, we align timestamped English Wikipedia snapshots by normalized article title, producing aligned old\-new pairs corresponding to the same entity at different points in time\. This formulation follows temporal knowledge benchmarks that evaluate whether language models can acquire updated information while retaining prior knowledge\([16](https://arxiv.org/html/2608.17530#bib.bib11);[5](https://arxiv.org/html/2608.17530#bib.bib12);[21](https://arxiv.org/html/2608.17530#bib.bib10)\)\. The new\-data corpus is extracted from sentence\-level diffs between aligned pairs, capturing inserted spans and the updated side of modified spans, which preserves localized factual updates while filtering unchanged text\. Full snapshot selection, topic stratification, and diff\-extraction details are provided in Appendix[A\.1](https://arxiv.org/html/2608.17530#A1.SS1)\.

#### Code\.

For source code, we construct temporally separated corpora from GitHub repositories selected by creation and commit timestamps\. Old examples are drawn from repositories that existed before January 2022, and new examples from repositories created after July 2024, with verification that𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}repositories did not exist before the cutoff window\. Repositories span thirteen widely used programming languages\. Unlike the Wikipedia corpus, old and new code examples are not aligned revisions of the same artifact; they represent temporally distinct samples from evolving software distributions, reflecting how software ecosystems shift through new libraries, repositories, and conventions rather than incremental edits to existing artifacts\. Full repository selection criteria and language coverage are provided in Appendix[A\.2](https://arxiv.org/html/2608.17530#A1.SS2)\.

#### Corpus statistics\.

Corpus sizes for both domains are summarized in Appendix[A](https://arxiv.org/html/2608.17530#A1)\(Table[8](https://arxiv.org/html/2608.17530#A1.T8)\)\. For Wikipedia, the old and new training corpora each contain31,72931\{,\}729aligned examples\. For code, the old corpus contains22,05522\{,\}055examples and the new corpus contains70,64370\{,\}643examples\.

### 4\.2Models and Training

We evaluate SRT across two model scales\. Smaller\-scale experiments use TinyLlama\-1\.1B\-Chat\([32](https://arxiv.org/html/2608.17530#bib.bib18)\), which provides a fast iteration platform for ablations and main comparisons\. Larger\-scale experiments use Llama\-3\.2\-3B\-Instruct\([10](https://arxiv.org/html/2608.17530#bib.bib19);[23](https://arxiv.org/html/2608.17530#bib.bib4)\), a 3\-billion\-parameter instruction\-tuned model released by Meta with a knowledge cutoff of December 2023\. We continually pre\-train from the instruction\-tuned checkpoint rather than a base checkpoint because updating already\-deployed instruct models on fresh data is the realistic practitioner scenario our method targets\.

All conditions use the same continual pre\-training recipe, following established practice for replay\-based LLM continual pre\-training\([15](https://arxiv.org/html/2608.17530#bib.bib8)\)\. We use causal language modeling with the next\-token cross\-entropy objective, AdamW optimization, linear learning\-rate scheduling with warmup and decay, gradient clipping, and bfloat16 precision where supported\. Hyperparameters were tuned on a held\-out development split and held fixed across all baselines and SRT conditions to ensure that performance differences reflect the replay strategy rather than optimization differences\. Complete hyperparameter values are provided in Appendix[B](https://arxiv.org/html/2608.17530#A2)\.

### 4\.3Baselines

We compare four conditions against the original pre\-trained checkpoint\.Baseis the pre\-trained model before continual update\. It establishes the starting point for both old\-knowledge retention and new\-knowledge acquisition\.CPTis naive continual pre\-training, where each training batch is drawn entirely from𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}\. This represents the standard no\-replay baseline and exhibits the canonical forgetting failure mode\.Uniform Replaydraws each batch using a nominal 20/80 old/new exposure ratio, with old examples sampled uniformly at random from𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}and new examples sampled uniformly at random from𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}\. This baseline shares the same nominal replay budget and pool structure as SRT but uses no per\-example scheduling for either pool, isolating the contribution of adaptive review timing\.PPL\-Prioritisedis a difficulty\-aware replay baseline that isolates whether SRT’s gains come from spaced\-repetition scheduling or simply from attending to hard examples\. It uses the same 20/80 old/new exposure ratio as Uniform Replay and SRT, but within each pool it selects the highest\-perplexity examples rather than sampling uniformly, without maintaining any SM\-2 review state or interval scheduling\. It therefore replays the currently hardest examples at every step\. Because its role is to separate difficulty\-prioritisation from interval\-based scheduling rather than to serve as a scale\-general baseline, we evaluate it on both TinyLlama settings \(Wikipedia and code\) as a diagnostic control\.SRTuses the same nominal 20/80 old/new exposure cap and pool structure as Uniform Replay, but selects examples in each batch using SuperMemo\-2 due times and perplexity\-derived recall quality as described in Section[3\.2](https://arxiv.org/html/2608.17530#S3.SS2)\. Both old and new examples are selected by the scheduler\. The old\-exposure fraction is set toρ=0\.2\\rho=0\.2, matching Uniform Replay so that the only difference between the two conditions is whether within\-pool selection is scheduled or uniform\.

### 4\.4Evaluation

We evaluate along two complementary axes\. Source\-grounded question answering measures retention and acquisition on the temporally split corpora\. Standard capability benchmarks measure whether continual updates degrade broad model abilities beyond the updated domains\.

#### Source\-grounded QA\.

For each of the Wikipedia and code domains, we generate multiple\-choice QA benchmarks in MMLU\-style format directly from source passages in𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}and𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}, producing matched old and new evaluation splits of500500questions each\. Questions are generated from source content and manually checked for answerability and label support\. We report accuracy as the mean across questions together with the bootstrap standard deviation computed from10,00010\{,\}000resamples\.

#### Broad capability benchmarks\.

We evaluate on four widely used benchmarks: MMLU\([12](https://arxiv.org/html/2608.17530#bib.bib20)\)for multitask knowledge and reasoning, BBH\([28](https://arxiv.org/html/2608.17530#bib.bib21)\)for hard reasoning tasks, GSM8K\([6](https://arxiv.org/html/2608.17530#bib.bib22)\)for grade\-school mathematical reasoning, and PIQA\([2](https://arxiv.org/html/2608.17530#bib.bib23)\)for physical commonsense\. All broad capability benchmarks are evaluated using the EleutherAI LM Evaluation Harness\([9](https://arxiv.org/html/2608.17530#bib.bib28)\), which provides standardized few\-shot prompts, scoring procedures, and task implementations across models\. Using a single evaluation framework ensures that benchmark scores are directly comparable across baselines and SRT, and that our reported numbers are reproducible by other researchers\. These benchmarks are not optimization targets for any condition and serve to detect whether continual updates damage capabilities unrelated to the updated domains\.

## 5Results

We evaluate SRT along two axes: source\-grounded temporal QA \(Section[5\.1](https://arxiv.org/html/2608.17530#S5.SS1)\) and broad capability benchmarks \(Section[5\.2](https://arxiv.org/html/2608.17530#S5.SS2)\)\.

### 5\.1Temporal QA: Retention and Acquisition

Table[1](https://arxiv.org/html/2608.17530#S5.T1)reports source\-grounded QA accuracy across both models\. The pattern is consistent at both scales: naive CPT improves new\-knowledge accuracy at the cost of retention, uniform replay partially mitigates the drop, andSRTrecovers retention while preserving or improving new\-knowledge accuracy\.

#### Naive CPT degrades retention\.

On TinyLlama,CPTraises new accuracy from13\.1%13\.1\\%to17\.0%17\.0\\%on Wikipedia data but old accuracy collapses from54\.3%54\.3\\%to11\.7%11\.7\\%\. The same pattern holds on Llama\-3\.2\-3B\-Instruct, where old Wikipedia accuracy drops from50\.2%50\.2\\%to43\.0%43\.0\\%\.

#### Scheduling, not exposure, drives SRT’s gains\.

Uniform Replayreaches25\.2%25\.2\\%and46\.2%46\.2\\%old Wikipedia accuracy on TinyLlama and Llama\-3\.2\-3B\-Instruct respectively, in both cases substantially aboveCPTbut belowSRT\(49\.0%49\.0\\%and51\.6%51\.6\\%\)\. BecauseUniform ReplayandSRTshare the same20/8020/80exposure cap and pool structure and differ only in within\-pool selection, this gap is attributable to scheduling rather than exposure\. The gap is larger at the smaller model scale \(23\.823\.8percentage points on TinyLlama versus5\.45\.4on Llama\-3\.2\-3B\-Instruct\), suggesting that smaller models benefit more from prioritized review while larger models retain more knowledge under uniform replay alone\.

#### Scheduling versus difficulty alone\.

To separate interval\-based scheduling from simple difficulty\-prioritisation, we compareSRTagainst a PPL\-Prioritised baseline that replays the highest\-perplexity old examples without SM\-2 review state \(Table[1](https://arxiv.org/html/2608.17530#S5.T1)\)\. Difficulty\-awareness alone is clearly beneficial: on Wikipedia, PPL\-Prioritised improves old\-knowledge retention overUniform Replay\(27\.4%27\.4\\%vs\.25\.2%25\.2\\%\) and achieves the highest new\-knowledge accuracy of any method \(38\.8%38\.8\\%\)\. However, the two difficulty\-aware methods occupy different points on the stability\-plasticity spectrum\. PPL\-Prioritised is markedly more plastic, acquiring new knowledge aggressively but recovering far less old knowledge thanSRT\(27\.4%27\.4\\%vs\.49\.0%49\.0\\%on Wikipedia, a21\.621\.6\-point retention gap\)\. We attribute this to its greedy focus on the currently hardest examples, which neglects intermediate\-difficulty examples thatSRT’s interval scheduling continues to revisit along their forgetting trajectory\. On code,SRToutperforms PPL\-Prioritised on all three metrics\. Overall,SRTachieves the best old\-knowledge retention on both domains and the best combined score, while PPL\-Prioritised attains stronger new\-knowledge accuracy on Wikipedia; interval\-based scheduling thus provides a substantially better retention\-acquisition balance than one\-shot difficulty ranking, though the two methods prioritise that balance differently\.

#### SRT improves both retention and acquisition\.

SRTproduces the best combined score on every model\-domain pair in Table[1](https://arxiv.org/html/2608.17530#S5.T1)\. On Wikipedia, it restores old accuracy to or above the base\-model level at both scales while matching or exceedingCPTon new accuracy\. On code QA,SRTis the only method that simultaneously improves both old and new accuracy relative toCPTon both models\.Uniform Replayis competitive withCPTon code at both scales but trailsSRTconsistently, confirming that scheduling provides value across model scales even when uniform replay alone is adequate\. To corroborate these findings on a benchmark we did not construct, we additionally evaluate on TemporalWiki\([16](https://arxiv.org/html/2608.17530#bib.bib11)\), an external factual\-probe set\.SRTsubstantially limits the perplexity degradation thatCPTincurs on this benchmark, consistent with the retention advantage measured on our source\-grounded QA\. Full results and caveats are provided in Appendix[D](https://arxiv.org/html/2608.17530#A4)\.

Wikipedia QACode QAModelMethodOldNewCombinedOldNewCombinedTinyLlamaBase54\.3±2\.154\.3\\pm 2\.113\.1±1\.513\.1\\pm 1\.533\.7±1\.533\.7\\pm 1\.519\.5±1\.819\.5\\pm 1\.811\.6±1\.411\.6\\pm 1\.415\.6±1\.115\.6\\pm 1\.1CPT11\.7±1\.411\.7\\pm 1\.417\.0±1\.417\.0\\pm 1\.414\.4±1\.114\.4\\pm 1\.117\.4±1\.717\.4\\pm 1\.714\.8±1\.614\.8\\pm 1\.616\.6±1\.216\.6\\pm 1\.2Uniform Replay25\.2±1\.925\.2\\pm 1\.913\.6±1\.513\.6\\pm 1\.519\.4±1\.419\.4\\pm 1\.414\.0±1\.614\.0\\pm 1\.615\.4±1\.615\.4\\pm 1\.614\.7±1\.114\.7\\pm 1\.1PPL\-Prior\.27\.4±2\.027\.4\\pm 2\.038\.8±2\.238\.8\\pm 2\.233\.1±1\.533\.1\\pm 1\.520\.8±3\.620\.8\\pm 3\.613\.6±3\.013\.6\\pm 3\.017\.2±2\.317\.2\\pm 2\.3SRT49\.0±2\.149\.0\\pm 2\.120\.0±1\.720\.0\\pm 1\.734\.6±1\.534\.6\\pm 1\.524\.4±1\.924\.4\\pm 1\.920\.7±1\.820\.7\\pm 1\.822\.6±1\.322\.6\\pm 1\.3Llama\-3\.2\-3B\-Inst\.Base50\.2±2\.250\.2\\pm 2\.239\.8±2\.139\.8\\pm 2\.145\.0±1\.545\.0\\pm 1\.555\.0±2\.255\.0\\pm 2\.252\.6±2\.252\.6\\pm 2\.253\.8±1\.553\.8\\pm 1\.5CPT43\.0±2\.243\.0\\pm 2\.242\.4±2\.242\.4\\pm 2\.242\.7±1\.542\.7\\pm 1\.554\.6±2\.254\.6\\pm 2\.253\.2±2\.253\.2\\pm 2\.253\.9±1\.553\.9\\pm 1\.5Uniform Replay46\.2±2\.246\.2\\pm 2\.239\.8±2\.139\.8\\pm 2\.143\.0±1\.543\.0\\pm 1\.556\.0±2\.256\.0\\pm 2\.257\.2±2\.257\.2\\pm 2\.256\.6±1\.656\.6\\pm 1\.6SRT51\.6±2\.251\.6\\pm 2\.242\.4±2\.242\.4\\pm 2\.247\.0±1\.547\.0\\pm 1\.560\.8±2\.260\.8\\pm 2\.257\.8±2\.257\.8\\pm 2\.259\.3±1\.559\.3\\pm 1\.5Table 1:Source\-grounded QA accuracy \(mean±\\pmbootstrap standard deviation, %\)\. Each old/new split has500500held\-out questions\.Uniform ReplayandSRTshare the same20/8020/80exposure cap and pool structure, differing only in whether within\-pool selection is uniform or scheduled\.

### 5\.2Broad Capability Preservation

Table[2](https://arxiv.org/html/2608.17530#S5.T2)reports accuracy on MMLU, BBH, GSM8K, and PIQA\. On TinyLlama, all methods remain within standard\-deviation bounds of the base model, reflecting that the base model’s MMLU and GSM8K scores are near random and leave little signal to disturb\.

The Llama\-3\.2\-3B\-Instruct results reveal a stark pattern\.CPTsubstantially degrades all four benchmarks, with the largest drop on GSM8K \(77\.6%77\.6\\%to38\.8%38\.8\\%\): a38\.838\.8\-point decline despite no obvious relationship between the temporally localized training data and grade\-school math\.Uniform Replayperforms even worse on reasoning, dropping BBH to8\.4%8\.4\\%and GSM8K to6\.8%6\.8\\%\. Re\-exposing the model to old data uniformly therefore damages reasoning capabilities more severely than no replay at all, suggesting that uniform replay introduces interference patterns that disrupt reasoning circuits rather than mitigating forgetting\.SRTlargely preserves base\-model performance across all four benchmarks \(57\.8%57\.8\\%,52\.6%52\.6\\%,76\.7%76\.7\\%,74\.3%74\.3\\%\), and as shown in Table[1](https://arxiv.org/html/2608.17530#S5.T1), this preservation does not come at the cost of adaptation\.

ModelMethodMMLUBBHGSM8KPIQATinyLlamaBase24\.9±3\.624\.9\\pm 3\.627\.1±0\.127\.1\\pm 0\.12\.4±0\.02\.4\\pm 0\.073\.4±0\.073\.4\\pm 0\.0CPT24\.9±0\.024\.9\\pm 0\.026\.9±0\.026\.9\\pm 0\.02\.0±0\.02\.0\\pm 0\.073\.3±1\.073\.3\\pm 1\.0Uniform Replay24\.4±0\.024\.4\\pm 0\.026\.8±0\.026\.8\\pm 0\.03\.3±0\.03\.3\\pm 0\.073\.2±0\.073\.2\\pm 0\.0SRT24\.4±0\.024\.4\\pm 0\.025\.8±0\.025\.8\\pm 0\.02\.5±0\.02\.5\\pm 0\.072\.2±0\.072\.2\\pm 0\.0Llama\-3\.2\-3B\-Inst\.Base57\.5±0\.057\.5\\pm 0\.053\.9±0\.053\.9\\pm 0\.077\.6±1\.277\.6\\pm 1\.275\.1±1\.075\.1\\pm 1\.0CPT51\.4±0\.451\.4\\pm 0\.444\.3±0\.644\.3\\pm 0\.638\.8±1\.338\.8\\pm 1\.369\.0±0\.769\.0\\pm 0\.7Uniform Replay50\.0±0\.050\.0\\pm 0\.08\.4±0\.08\.4\\pm 0\.06\.8±0\.16\.8\\pm 0\.174\.9±1\.074\.9\\pm 1\.0SRT57\.8±0\.157\.8\\pm 0\.152\.6±0\.152\.6\\pm 0\.176\.7±1\.276\.7\\pm 1\.274\.3±1\.074\.3\\pm 1\.0Table 2:Broad capability benchmark accuracy \(mean±\\pmstandard deviation, %\)\. At the Llama\-3\.2\-3B\-Instruct scale, bothCPTandUniform Replaydegrade reasoning benchmarks \(BBH, GSM8K\) substantially;SRTpreserves base\-model performance\.
### 5\.3Auxiliary Non\-Language Evidence

To test whether the scheduler is tied to language perplexity, we also summarize class\-incremental vision and tabular experiments, with details in Appendix[C](https://arxiv.org/html/2608.17530#A3)\. Replacing perplexity with predicted\-class confidence gives the same qualitative pattern: on MNIST, Fashion\-MNIST, CIFAR\-10, and Wine, SRT obtains the highest overall accuracy \(91\.6%91\.6\\%,65\.8%65\.8\\%,53\.1%53\.1\\%, and53\.3%53\.3\\%, respectively\)\. We treat these experiments as supporting evidence that adaptive review scheduling is modality\-agnostic when paired with an appropriate recall signal; the central claim remains LLM continual pre\-training\.

### 5\.4Computational Overhead

SRT adds a forward pass per reviewed example beyond standard continual pre\-training to compute the perplexity\-based recall quality score, which introduces some computational cost\. To quantify this overhead, we measure wall\-clock time, per\-step time, and throughput forCPT,Uniform Replay, andSRTunder matched configurations on the same hardware\. Table[3](https://arxiv.org/html/2608.17530#S5.T3)reports the results\.

The overhead ofSRTrelative toCPTis14\.7%14\.7\\%in wall\-clock time, corresponding to a14\.6%14\.6\\%reduction in tokens per second\.Uniform Replayalso incurs overhead relative toCPT\(10\.8%10\.8\\%wall\-clock,13\.0%13\.0\\%throughput reduction\), reflecting the cost of sampling and combining old and new examples into each batch\. The additional cost ofSRTrelative toUniform Replayis therefore only3\.5%3\.5\\%in wall\-clock time, attributable to the recall quality computation and SuperMemo\-2 state updates\. Given the substantial retention gains reported in Section[5\.1](https://arxiv.org/html/2608.17530#S5.SS1), this scheduling overhead is a favourable trade\-off for practitioners who care about old\-knowledge preservation\.

MetricCPTUniform ReplaySRTWall\-clock time \(s\)960\.07960\.071063\.871063\.871101\.361101\.36Time per step \(s\)9\.199\.1910\.3310\.3310\.7510\.75Tokens per second455\.83455\.83396\.51396\.51380\.90380\.90Tokens per step409640964096409640964096Overhead vs\.CPT—\+10\.8%\+10\.8\\%\+14\.7%\+14\.7\\%Overhead vs\.Uniform Replay——\+3\.5%\+3\.5\\%Table 3:Computational overhead ofUniform ReplayandSRTrelative toCPTunder matched configurations\.SRTadds approximately15%15\\%wall\-clock overhead relative to no\-replay training, of which most is shared withUniform Replaydue to batch construction cost; the additional scheduling\-specific overhead is only3\.5%3\.5\\%\.

## 6Ablations

We conduct three ablation studies to evaluate the sensitivity of SRT to its main design choices: the replay budget, the perplexity\-to\-quality threshold scaling, and the initial ease factor\. The replay\-ratio ablation uses TinyLlama Wikipedia QA to probe sensitivity to old\-new exposure allocation, while the threshold and ease\-factor ablations use TinyLlama code QA as a representative domain\.

#### Replay budget\.

Table[4](https://arxiv.org/html/2608.17530#S6.T4)sweeps the old/new exposure ratioρ\\rhoon TinyLlama Wikipedia\. The best configuration is20/8020/80, reaching49\.6%49\.6\\%old and20\.0%20\.0\\%new accuracy\. Performance is sensitive to the budget at both extremes\. Old\-heavy schedules \(50/5050/50and beyond\) reduce exposure to new data and fail to recover old\-knowledge accuracy because the model has less budget for both consolidation and adaptation\. New\-heavy schedules \(5/955/95\) behave closer toCPTand lose old knowledge for the same reason\. The20/8020/80split aligns with the matched\-budget design used in the main comparison and provides enough old\-data exposure for SRT’s scheduler to operate while preserving sufficient adaptation capacity\.

Old/NewOldNewCombined5/955/9523\.6±1\.823\.6\\pm 1\.813\.3±1\.513\.3\\pm 1\.518\.4±1\.318\.4\\pm 1\.320/8020/8049\.6±2\.149\.6\\pm 2\.120\.0±1\.720\.0\\pm 1\.734\.6±1\.534\.6\\pm 1\.535/6535/6523\.4±1\.923\.4\\pm 1\.915\.4±1\.615\.4\\pm 1\.619\.4±1\.219\.4\\pm 1\.250/5050/5021\.8±1\.921\.8\\pm 1\.913\.8±1\.513\.8\\pm 1\.517\.8±1\.117\.8\\pm 1\.165/3565/3520\.6±1\.820\.6\\pm 1\.812\.2±1\.412\.2\\pm 1\.416\.4±1\.216\.4\\pm 1\.280/2080/2023\.5±1\.923\.5\\pm 1\.914\.2±1\.614\.2\\pm 1\.618\.8±1\.218\.8\\pm 1\.295/595/527\.4±1\.927\.4\\pm 1\.919\.7±1\.719\.7\\pm 1\.723\.6±1\.323\.6\\pm 1\.3Table 4:TinyLlama Wikipedia QA accuracy as a function of the SRT old/new exposure ratio\. Accuracy is mean±\\pmbootstrap standard deviation, %\.
#### Perplexity\-to\-quality threshold scaling\.

We test sensitivity to the threshold values used in Eq\.[7](https://arxiv.org/html/2608.17530#S3.E7)\. A scalar multiplierα\\alphais applied uniformly to all five thresholds\(τ1,…,τ5\)\(\\tau\_\{1\},\\ldots,\\tau\_\{5\}\), soα<1\\alpha<1produces stricter scoring \(examples need lower perplexity to be considered retained\) andα\>1\\alpha\>1produces relaxed scoring\. Table[5](https://arxiv.org/html/2608.17530#S6.T5)reports TinyLlama code QA accuracy acrossα∈\{0\.5,1\.0,1\.5\}\\alpha\\in\\\{0\.5,1\.0,1\.5\\\}\. The defaultα=1\.0\\alpha=1\.0is best, reaching22\.6%22\.6\\%combined accuracy\. Both stricter \(α=0\.5\\alpha=0\.5\) and more relaxed \(α=1\.5\\alpha=1\.5\) thresholds degrade performance\. The effect is asymmetric: relaxation hurts more than equivalent strictness, because over\-relaxed thresholds assign high quality to poorly retained examples and stop scheduling them for review\.

α\\alphaCode OldCode NewCombined0\.50\.513\.2±1\.413\.2\\pm 1\.48\.6±1\.28\.6\\pm 1\.210\.9±1\.010\.9\\pm 1\.01\.01\.024\.4±1\.924\.4\\pm 1\.920\.7±1\.820\.7\\pm 1\.822\.6±1\.322\.6\\pm 1\.31\.51\.54\.8±1\.14\.8\\pm 1\.15\.4±1\.05\.4\\pm 1\.05\.1±0\.95\.1\\pm 0\.9Table 5:Sensitivity to perplexity threshold scaling on TinyLlama code QA\. The scalar multiplierα\\alphais applied uniformly to all five thresholds in Eq\.[7](https://arxiv.org/html/2608.17530#S3.E7)\.
#### Initial ease factor\.

We vary the initial ease factorE0E\_\{0\}, which controls how quickly review intervals expand after early successful reviews\. Table[6](https://arxiv.org/html/2608.17530#S6.T6)reports TinyLlama code QA acrossE0∈\{1\.5,2\.5,3\.5\}E\_\{0\}\\in\\\{1\.5,2\.5,3\.5\\\}\. The default SM\-2 valueE0=2\.5E\_\{0\}=2\.5is best, again reaching22\.6%22\.6\\%combined accuracy\. Smaller values keep intervals short and over\-schedule examples that have already been retained, reducing exposure for examples that genuinely need review\. Larger values inflate intervals too quickly and let fragile examples drift out of the review schedule before they are stably retained\. The default value chosen by the original SM\-2 algorithm\([29](https://arxiv.org/html/2608.17530#bib.bib15)\)is therefore a reasonable starting point for LLM continual pre\-training as well\.

E0E\_\{0\}Code OldCode NewCombined1\.51\.512\.2±1\.512\.2\\pm 1\.514\.2±1\.514\.2\\pm 1\.513\.2±1\.113\.2\\pm 1\.12\.52\.524\.4±1\.924\.4\\pm 1\.920\.7±1\.820\.7\\pm 1\.822\.6±1\.322\.6\\pm 1\.33\.53\.512\.0±1\.412\.0\\pm 1\.49\.0±1\.29\.0\\pm 1\.210\.5±1\.010\.5\\pm 1\.0Table 6:Sensitivity to the initial ease factorE0E\_\{0\}on TinyLlama code QA\. The default SM\-2 valueE0=2\.5E\_\{0\}=2\.5achieves the best combined accuracy\.

## 7Discussion

The results support a specific claim: SRT improves continual pre\-training when old and new knowledge must share a finite training budget and when retention difficulty is heterogeneous across examples\. We discuss three implications of this finding\.

#### Scheduling is the active ingredient, not exposure\.

Naive continual pre\-training is the headline practical comparison because it captures the realistic no\-review update regime, but it does not isolate the mechanism behind SRT’s gains\. Uniform replay serves this role\. Because Uniform Replay and SRT share the same20/8020/80exposure cap and pool structure, the only experimental variable separating them is whether within\-pool selection is uniform or scheduled\. On TinyLlama Wikipedia, this single variable is responsible for a23\.823\.8percentage\-point gap in old\-knowledge accuracy\. The same ordering,CPT<Uniform Replay<SRT\\textsc\{CPT\}\{\}<\\textsc\{Uniform Replay\}\{\}<\\textsc\{SRT\}\{\}, holds on Llama\-3\.2\-3B\-Instruct Wikipedia\. These results indicate that scheduling, not mere re\-exposure, is what produces SRT’s retention advantage\. Replay budgets that look adequate in aggregate can still fail when allocated uniformly across examples with heterogeneous retention difficulty\.

#### Uniform replay can damage reasoning capabilities\.

A more surprising finding is that uniform replay does not merely fall short of SRT on broad capabilities; it can be substantially worse than naive CPT itself\. On Llama\-3\.2\-3B\-Instruct, Uniform Replay drops BBH from a CPT baseline of44\.3%44\.3\\%to8\.4%8\.4\\%and GSM8K from38\.8%38\.8\\%to6\.8%6\.8\\%, while SRT preserves both close to the original model\. We do not claim direct evidence for the underlying mechanism; one possible explanation is that re\-exposing the model to old data without scheduling introduces interference dynamics that disrupt reasoning\-relevant computation more severely than no replay at all, but verifying this would require representational analysis beyond the scope of this work\. Regardless of mechanism, the result has an immediate practical implication: practitioners considering replay as a simple add\-on to continual updates should be cautious, because a poorly designed replay schedule can degrade capabilities the no\-replay baseline preserves\.

#### SRT is a scheduling layer, not a replacement\.

SRT addresses a question that mixture\-level replay does not: among reviewable examples, which ones should return now? It does so without modifying the architecture, the training objective, or the choice of replay budget\. SRT can therefore be combined with learning\-rate rewarming\([15](https://arxiv.org/html/2608.17530#bib.bib8)\), data\-mixture selection\([4](https://arxiv.org/html/2608.17530#bib.bib9)\), parameter\-efficient tuning, or synthetic rehearsal\([13](https://arxiv.org/html/2608.17530#bib.bib27)\)\. Its contribution is orthogonal to these methods, addressing the within\-pool selection problem they leave unspecified\. We see scheduling as a complementary layer that practitioners can adopt alongside existing CPT recipes rather than as an alternative to them\. The SRT scheduler is also architecture\-agnostic: auxiliary experiments on vision and tabular classification benchmarks \(Appendix[C](https://arxiv.org/html/2608.17530#A3)\) show that the same scheduling, with predicted\-class confidence in place of perplexity as the recall signal, mitigates class\-incremental forgetting more effectively than naive continual training and Elastic Weight Consolidation\.

## 8Conclusion

We introduced Spaced Repetition Training, a cognitively inspired continual pre\-training method that schedules per\-example review using SuperMemo\-2 and perplexity\-derived recall quality\. The central insight is that continual pre\-training is not solely a mixture problem but a scheduling problem: among reviewable examples, the model should revisit those it is currently struggling to retain, not those it has already learned\. On temporally grounded Wikipedia and code QA benchmarks across two model scales, SRT recovers old\-knowledge accuracy that naive continual pre\-training discards, while matching or exceeding new\-knowledge acquisition\. At the larger model scale, SRT preserves broad capability benchmark performance that both naive CPT and uniform replay degrade, in some cases substantially\. Together these results suggest that adaptive review timing is a simple, architecture\-agnostic, and effective mechanism for improving the stability\-plasticity trade\-off in continual pre\-training, and that established principles from human memory research transfer productively to language model training\.

## Limitations

Our language experiments use TinyLlama\-1\.1B and Llama\-3\.2\-3B\-Instruct, both from the Llama family, and evaluation is restricted to English; behaviour at larger scales, across different model families, and in multilingual settings remains open, even though our cross\-modality evidence demonstrates the effectiveness of SRT\. The temporal QA benchmarks are constructed for this work and should be interpreted as in\-domain measurements of retention and acquisition on our specific corpora rather than as community\-standard tests\. The broad\-capability results in Table[2](https://arxiv.org/html/2608.17530#S5.T2)are obtained from a single training run per condition, evaluated over multiple evaluation seeds; the reported variance therefore reflects evaluation noise rather than training\-run variability\. While the reasoning\-benchmark degradation under uniform replay is consistent across two independent benchmarks \(BBH and GSM8K\) and is not exhibited bySRTunder identical conditions, which makes a single\-run artifact less likely, confirming its stability would require multiple independent training runs\. We leave this to future work\. Finally, the perplexity\-to\-quality mapping uses fixed thresholds that may need recalibration for substantially different model scales or domains, and SRT introduces a forward pass per reviewed example beyond the normal training pass; a learned quality function and empirical overhead measurement at larger scales are natural extensions for future work\.

## Ethics and Reproducibility

Continual updates can introduce factual errors, preserve outdated information, or reinforce unsafe code patterns; source\-grounded multiple\-choice accuracy is not deployment reliability\. Replay assumes access to historical examples, so buffers and SRT state must respect retention limits, deletion requests, and data\-governance obligations\.

Exact benchmark artifacts, code, logs, and QA items are proprietary and are not released\. Replication is supported by the algorithm, recall\-quality mapping, corpus criteria, hyperparameters, evaluation protocol, and reported results, which are sufficient to recreate comparable public\-data pipelines\. Benchmark generation used LLM assistance with manual checks \(Appendix[A\.3](https://arxiv.org/html/2608.17530#A1.SS3)\)\.

## References

- Aeberhardet al\.\(1994\)S\. Aeberhard, D\. Coomans, and O\. De VelComparative analysis of statistical pattern recognition methods in high dimensional settings\.Pattern Recognition27\(8\),pp\. 1065–1077\.Cited by:[§C\.1](https://arxiv.org/html/2608.17530#A3.SS1.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.4](https://arxiv.org/html/2608.17530#S4.SS4.SSS0.Px2.p1.1)\.
- Cepedaet al\.\(2006\)N\. J\. Cepeda, H\. Pashler, E\. Vul, J\. T\. Wixted, and D\. RohrerDistributed practice in verbal recall tasks: a review and quantitative synthesis\.Psychological Bulletin132\(3\),pp\. 354–380\.Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p3.1)\.
- Chenet al\.\(2025\)J\. Chen, Z\. Chen, J\. Wang, K\. Zhou, Y\. Zhu, J\. Jiang, Y\. Min, W\. X\. Zhao, Z\. Dou, J\. Mao, Y\. Lin, R\. Song, J\. Xu, X\. Chen, R\. Yan, Z\. Wei, D\. Hu, W\. Huang, and J\. WenTowards effective and efficient continual pre\-training of large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 5779–5795\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.289)Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p2.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2608.17530#S7.SS0.SSS0.Px3.p1.1)\.
- Chenget al\.\(2024\)J\. Cheng, M\. Marone, O\. Weller, D\. Lawrie, D\. Khashabi, and B\. Van DurmeDated data: tracing knowledge cutoffs in large language models\.External Links:2403\.12958Cited by:[§4\.1](https://arxiv.org/html/2608.17530#S4.SS1.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.InarXiv preprint arXiv:2110\.14168,Cited by:[§4\.4](https://arxiv.org/html/2608.17530#S4.SS4.SSS0.Px2.p1.1)\.
- Ebbinghaus \(1885\)H\. EbbinghausMemory: a contribution to experimental psychology\.Note:English translation reprinted in Annals of Neurosciences, 2013Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p3.1)\.
- Fenget al\.\(2026\)Y\. Feng, H\. Wang, J\. Li, X\. Chu, Z\. Kang, Y\. Liu, Y\. Wang, P\. S\. Yu, and X\. WuFOREVER: forgetting curve\-inspired memory replay for language model continual learning\.External Links:2601\.03938Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouA framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§4\.4](https://arxiv.org/html/2608.17530#S4.SS4.SSS0.Px2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.2](https://arxiv.org/html/2608.17530#S4.SS2.p1.1)\.
- Gururanganet al\.\(2020\)S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. SmithDon’t stop pretraining: adapt language models to domains and tasks\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8342–8360\.Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p1.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations\.Cited by:[§A\.3](https://arxiv.org/html/2608.17530#A1.SS3.p1.1),[§4\.4](https://arxiv.org/html/2608.17530#S4.SS4.SSS0.Px2.p1.1)\.
- Huanget al\.\(2024\)J\. Huang, L\. Cui, A\. Wang, C\. Yang, X\. Liao, L\. Song, J\. Yao, and J\. SuMitigating catastrophic forgetting in large language models with self\-synthesized rehearsal\.arXiv preprint arXiv:2403\.01244\.Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2608.17530#S7.SS0.SSS0.Px3.p1.1)\.
- Hunget al\.\(2019\)C\. Hung, C\. Tu, C\. Wu, C\. Chen, Y\. Chan, and C\. ChenCompacting, picking and growing for unforgetting continual learning\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Ibrahimet al\.\(2024\)A\. Ibrahim, B\. Thérien, K\. Gupta, M\. L\. Richter, Q\. Anthony, T\. Lesort, E\. Belilovsky, and I\. RishSimple and scalable strategies to continually pre\-train large language models\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p1.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.17530#S4.SS2.p2.1),[§7](https://arxiv.org/html/2608.17530#S7.SS0.SSS0.Px3.p1.1)\.
- Janget al\.\(2022\)J\. Jang, S\. Ye, C\. Lee, S\. Yang, J\. Shin, J\. Han, G\. Kim, and M\. SeoTemporalwiki: a lifelong benchmark for training and evaluating ever\-evolving language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 6237–6250\.Cited by:[Table 12](https://arxiv.org/html/2608.17530#A4.T12),[Appendix D](https://arxiv.org/html/2608.17530#A4.p1.1),[§4\.1](https://arxiv.org/html/2608.17530#S4.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.17530#S5.SS1.SSS0.Px4.p1.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§C\.3](https://arxiv.org/html/2608.17530#A3.SS3.p1.1),[§1](https://arxiv.org/html/2608.17530#S1.p1.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Krizhevskyet al\.\(2009\)A\. Krizhevsky G\. Hintonet al\.Learning multiple layers of features from tiny images\.Technical reportUniversity of Toronto\.Cited by:[§C\.1](https://arxiv.org/html/2608.17530#A3.SS1.p1.1)\.
- LeCun \(1998\)Y\. LeCunThe MNIST database of handwritten digits\.External Links:[Link](http://yann.lecun.com/exdb/mnist/)Cited by:[§C\.1](https://arxiv.org/html/2608.17530#A3.SS1.p1.1)\.
- Li and Lee \(2024\)C\. Li and H\. LeeExamining forgetting in continual pre\-training of aligned large language models\.arXiv preprint arXiv:2401\.03129\.Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p1.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)J\. Li, M\. Armandpour, S\. I\. Mirzadeh, S\. Mehta, V\. Shankar, R\. Vemulapalli, S\. Bengio, O\. Tuzel, M\. Farajtabar, H\. Pouransari, and F\. FaghriTiC\-LM: a web\-scale benchmark for time\-continual LLM pretraining\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 32231–32273\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1551)Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p2.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.17530#S4.SS1.SSS0.Px1.p1.1)\.
- McCloskey and Cohen \(1989\)M\. McCloskey and N\. J\. CohenCatastrophic interference in connectionist networks: the sequential learning problem\.Psychology of Learning and Motivation24,pp\. 109–165\.Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p1.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Meta AI \(2024\)Meta AILlama 3\.2: Revolutionizing Edge AI and Vision with Open, Customizable Models\.External Links:[Link](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)Cited by:[§4\.2](https://arxiv.org/html/2608.17530#S4.SS2.p1.1)\.
- M’hamdi and May \(2024\)M\. M’hamdi and J\. MayLeitner\-guided memory replay for cross\-lingual continual learning\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 7808–7821\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.432)Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px3.p1.1)\.
- Parisiet al\.\(2019\)G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. WermterContinual lifelong learning with neural networks: a review\.Neural Networks113,pp\. 54–71\.Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Rolnicket al\.\(2019\)D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. WayneExperience replay for continual learning\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2608.17530#S1.p2.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Rusuet al\.\(2016\)A\. A\. Rusu, N\. C\. Rabinowitz, G\. Desjardins, H\. Soyer, J\. Kirkpatrick, K\. Kavukcuoglu, R\. Pascanu, and R\. HadsellProgressive neural networks\.InarXiv preprint arXiv:1606\.04671,Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. WeiChallenging big\-bench tasks and whether chain\-of\-thought can solve them\.Findings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[§4\.4](https://arxiv.org/html/2608.17530#S4.SS4.SSS0.Px2.p1.1)\.
- Wozniak \(1998\)P\. A\. WozniakSuperMemo 2 algorithm\.Note:Technical noteCited by:[§1](https://arxiv.org/html/2608.17530#S1.p3.1),[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.17530#S3.SS2.p1.1),[§6](https://arxiv.org/html/2608.17530#S6.SS0.SSS0.Px3.p1.1)\.
- Xiaoet al\.\(2017\)H\. Xiao, K\. Rasul, and R\. VollgrafFashion\-MNIST: a novel image dataset for benchmarking machine learning algorithms\.arXiv preprint arXiv:1708\.07747\.Cited by:[§C\.1](https://arxiv.org/html/2608.17530#A3.SS1.p1.1)\.
- Zenkeet al\.\(2017\)F\. Zenke, B\. Poole, and S\. GanguliContinual learning through synaptic intelligence\.InInternational Conference on Machine Learning,pp\. 3987–3995\.Cited by:[§2](https://arxiv.org/html/2608.17530#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024\)P\. Zhang, G\. Zeng, T\. Wang, and W\. LuTinyLlama: an open\-source small language model\.External Links:2401\.02385Cited by:[§4\.2](https://arxiv.org/html/2608.17530#S4.SS2.p1.1)\.

CorpusConstructionRowsOld median wordsNew median wordsMedian sent\. diffTinyLlama reference diffAdded/modified update text31,729–––Llama\-3 title\-aligned reconstruction2023\-11→\\rightarrow2024\-0128,9465435460Llama\-3 topic\-stratified corpus2023\-11→\\rightarrow2024\-01, topic/change stratified31,7295936143

Table 7:Wikipedia corpora used for language\-model continual pre\-training\.## Appendix ACorpora Construction Details

This appendix provides extended construction details for the Wikipedia and code corpora described in Section[4\.1](https://arxiv.org/html/2608.17530#S4.SS1), together with the question\-answering evaluation sets derived from each domain\.

Domain𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}Train𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}Train𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}QA𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}QAWikipedia text31,72931\{,\}72931,72931\{,\}729500500500500Source code22,05522\{,\}05570,64370\{,\}643500500500500Table 8:Continual pre\-training corpus sizes\. Old examples represent pre\-cutoff knowledge to be retained, and new examples represent post\-cutoff information to be acquired\. QA sets are held out exclusively for evaluation\.### A\.1Wikipedia Corpora

#### Snapshot selection\.

We construct Wikipedia corpora from timestamped English Wikipedia snapshots\. For TinyLlama\-1\.1B\-Chat experiments, snapshot dates are chosen to bracket the model’s pre\-training data assembly window in mid\-2023, with the old snapshot near that window and the new snapshot drawn from a sufficiently later date that revisions reflect post\-cutoff information\. For Llama\-3\.2\-3B\-Instruct experiments, we use November 2023 and January 2024 snapshots\. The short two\-month gap creates a challenging setting in which the model must integrate recent factual updates while retaining closely related prior knowledge\.

#### Diff extraction\.

For each aligned article pair, we segment both versions into sentences and compute sequence\-level diffs to identify inserted, deleted, and modified spans\. The𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}training corpus consists of inserted sentences together with the updated side of modified spans\. This procedure reduces unchanged text while preserving localized factual updates relevant for continual pre\-training, and ensures that𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}training examples reflect content not present in the corresponding𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}articles\.

#### Topic stratification\.

For Llama\-3\.2\-3B\-Instruct experiments, candidate articles are stratified by topic and magnitude of change\. Topics span politics, current affairs, computer science and artificial intelligence, business and economics, science and medicine, law and policy, and sports and entertainment\. Stratification ensures balanced coverage across domains rather than over\-representation of any single topic area\.

### A\.2Code Corpora

#### Repository selection\.

We collect repositories from GitHub using creation and commit timestamps as the temporal axis\. Old examples come from repositories that existed before January 2022, while new examples come from repositories created after July 2024\. We explicitly verify that repositories selected for𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}did not exist before the cutoff window to prevent temporal leakage between splits\.

#### Language coverage\.

Repositories span thirteen widely used programming languages: Python, JavaScript, TypeScript, Java, C, C\+\+, Go, Rust, C\#, Ruby, PHP, Swift, Kotlin, and Scala\. Each training example contains structured code text including the source file path and file content\. Unlike the Wikipedia corpus, old and new code examples are not aligned revisions of the same artifact, reflecting that real software ecosystems evolve through new repositories and libraries rather than incremental revisions to existing files\.

### A\.3Evaluation Question\-Answering Sets

For each domain, we construct held\-out multiple\-choice question\-answering benchmarks with500500questions per split, disjoint from the training corpora\. Wikipedia questions are generated from held\-out articles in an MMLU\-style format\([12](https://arxiv.org/html/2608.17530#bib.bib20)\), with four answer choices per question and a single correct answer grounded in the source passage\. Code\-domain questions are generated analogously from held\-out repositories and target repository\-specific functionality, API usage, configuration semantics, and implementation details\. All questions are manually checked for answerability and source support\. The𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}QA sets measure retention of pre\-cutoff knowledge, while the𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}QA sets measure acquisition of post\-cutoff information\.

## Appendix BTraining Details

This appendix provides the full training configuration used in all continual pre\-training experiments\. The same hyperparameters are used for both TinyLlama\-1\.1B\-Chat and Llama\-3\.2\-3B\-Instruct, and across all conditions \(CPT, Uniform Replay, and SRT\), so that performance differences reflect the replay strategy rather than optimization differences\. Hyperparameters are summarized in Table[9](https://arxiv.org/html/2608.17530#A2.T9)\.

HyperparameterValueOptimizationOptimizerAdamW \(fused\)β1\\beta\_\{1\},β2\\beta\_\{2\}0\.90\.9,0\.9990\.999ϵ\\epsilon1​e−81\\mathrm\{e\}\{\-8\}Weight decay0\.0010\.001Gradient clipping \(max norm\)1\.01\.0Learning rate scheduleLearning rate1​e−41\\mathrm\{e\}\{\-4\}SchedulerLinearWarmup steps200200Batch and sequencePer\-device batch size3232Gradient accumulation steps1616Effective batch size512512Sequence length2,0482\{,\}048Precision and memoryMixed precisionbfloat16Gradient checkpointingYesSRT\-specificOld\-exposure ratioρ\\rho0\.20\.2Initial ease factorE0E\_\{0\}2\.52\.5Initial repetitionsn0n\_\{0\}00Initial intervalI0I\_\{0\}11Minimum ease factor1\.31\.3EvaluationQA accuracy bootstrap resamples10,00010\{,\}000Table 9:Training and evaluation hyperparameters\. The same configuration is used across all model scales and all replay conditions\.#### Hardware and training duration\.

All experiments are conducted using bfloat16 mixed\-precision training with gradient checkpointing enabled to reduce memory consumption\. The effective batch size of512512\(per\-device batch size32×32\\timesgradient accumulation1616\) is held constant across model scales\. Training is run for sufficient steps to consume a single pass over𝒟new\\mathcal\{D\}\_\{\\mathrm\{new\}\}at the configured replay ratio; exact step counts depend on corpus size and are listed in the experiment logs released with the code\.

#### SRT thresholds\.

The perplexity thresholds in Table[9](https://arxiv.org/html/2608.17530#A2.T9)correspond to theτk\\tau\_\{k\}values defined in Eq\.[7](https://arxiv.org/html/2608.17530#S3.E7)\. Lower perplexity yields higher recall quality, withq=5q=5assigned whenPPL<τ5=50\\mathrm\{PPL\}<\\tau\_\{5\}=50andq=0q=0assigned whenPPL≥τ1=5000\\mathrm\{PPL\}\\geq\\tau\_\{1\}=5000\. Sensitivity to a uniform scaling of these thresholds is analysed in Section[6](https://arxiv.org/html/2608.17530#S6)\.

## Appendix CAuxiliary Vision and Tabular Experiments

Although the central contribution of this paper concerns language model continual pre\-training, the underlying SRT scheduler is architecture\-agnostic and can be applied to other modalities\. In an earlier version of this work, we evaluated SRT on small vision and tabular classification benchmarks under class\-incremental learning, providing a sanity check that the scheduling mechanism generalizes beyond text\. We summarize these auxiliary experiments here\. The results are consistent with the main language\-model findings: naive continual training rapidly forgets old classes, regularization\-based methods provide partial mitigation, and SRT retains substantially more old\-class accuracy while continuing to learn new classes\.

### C\.1Datasets and Class\-Incremental Setup

We use four standard publicly available benchmarks spanning vision and tabular modalities: MNIST handwritten digits\([19](https://arxiv.org/html/2608.17530#bib.bib29)\), Fashion\-MNIST clothing images\([30](https://arxiv.org/html/2608.17530#bib.bib30)\), CIFAR\-10 natural images\([18](https://arxiv.org/html/2608.17530#bib.bib31)\), and the UCI Wine dataset\([1](https://arxiv.org/html/2608.17530#bib.bib32)\)\. For each dataset, classes are partitioned into an*old*subset \(used to train the base model\) and a*new*subset \(introduced sequentially in the continual update phase\)\. For the image datasets, the partition uses semantically confusable classes to induce stronger interference between old and new representations\. Table[10](https://arxiv.org/html/2608.17530#A3.T10)reports the exact class assignments\.

DatasetClass assignmentMNISTOld:\{0,1,2,3,4,5,6\}\\\{0,1,2,3,4,5,6\\\}New:\{7,8,9\}\\\{7,8,9\\\}FashionOld: T\-shirt, Trouser, Dress, Coat,Sandal, Bag, BootNew: Pullover, Shirt, SneakerCIFAR\-10Old: Airplane, Automobile, Bird,Cat, Deer, ShipNew: Truck, Frog, Dog, HorseWineOld: classes\{1,2\}\\\{1,2\\\}New: class\{3\}\\\{3\\\}Table 10:Class\-incremental splits used in the auxiliary vision and tabular experiments\. Vision datasets use semantically confusable class assignments to induce stronger interference\.
### C\.2Recall Quality for Non\-Generative Models

For language models, SRT derives the SuperMemo\-2 recall quality from per\-example perplexity\. For classification models, perplexity is not defined, so we use the model’s predicted\-class confidence in place of perplexity\. Specifically, for an examplexix\_\{i\}with true labelyiy\_\{i\}, the recall quality is computed from the predicted probabilitypθ​\(yi∣xi\)p\_\{\\theta\}\(y\_\{i\}\\mid x\_\{i\}\): high confidence in the correct label corresponds to high recall quality \(longer next interval\), while low confidence corresponds to low recall quality \(shorter next interval\)\. The remaining SM\-2 update logic is identical to the language\-model formulation in Section[3\.2](https://arxiv.org/html/2608.17530#S3.SS2)\.

### C\.3Baseline: Elastic Weight Consolidation

For the vision and tabular experiments, we additionally compare against Elastic Weight Consolidation \(EWC\)\([17](https://arxiv.org/html/2608.17530#bib.bib2)\), a regularization\-based continual learning method that penalizes updates to parameters important for previously learned tasks\. EWC estimates per\-parameter importance using the diagonal of the Fisher Information Matrix computed on the old data and adds a quadratic penalty

ℒEWC​\(θ\)=ℒnew​\(θ\)\+λ2​∑iFi​\(θi−θi∗\)2,\\mathcal\{L\}\_\{\\mathrm\{EWC\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{new\}\}\(\\theta\)\+\\frac\{\\lambda\}\{2\}\\sum\_\{i\}F\_\{i\}\(\\theta\_\{i\}\-\\theta\_\{i\}^\{\*\}\)^\{2\},\(11\)whereθ∗\\theta^\{\*\}denotes the parameters at the end of training on the old data,FiF\_\{i\}is the Fisher importance of parameterθi\\theta\_\{i\}, andλ\\lambdacontrols the regularization strength\. EWC is a natural comparison for the vision setting because the small fully\-connected and convolutional models used here permit Fisher computation at low overhead\. In language\-model continual pre\-training, gradient\-level regularization of this kind is significantly more expensive and is not the focus of our main experiments\.

### C\.4Model Architectures

All vision and tabular models are simple feed\-forward or convolutional networks implemented in PyTorch with ReLU activations\. The MNIST and Fashion\-MNIST models use two convolution\-pooling blocks followed by two fully connected layers\. The CIFAR\-10 model uses the same convolutional structure with dropout in the classifier head\. The Wine model is a three\-layer multilayer perceptron operating on the 13\-dimensional UCI feature vector\. All models are trained with cross\-entropy loss and AdamW\.

### C\.5Results

Table[11](https://arxiv.org/html/2608.17530#A3.T11)reports accuracy across all four datasets for the Base model, naive CPT, EWC, and SRT, averaged over 10 random seeds\. The pattern is consistent across modalities\. Naive CPT loses essentially all old\-class accuracy on vision datasets while reaching high new\-class accuracy, mirroring the catastrophic\-forgetting failure mode observed at language scale\. EWC partially mitigates forgetting on MNIST and Wine but performs poorly on Fashion\-MNIST and CIFAR\-10, where semantic similarity between old and new classes makes parameter\-importance estimates less informative\. SRT retains substantially more old\-class accuracy than both baselines across all four datasets while maintaining competitive or superior new\-class accuracy, achieving the highest overall accuracy in every setting\.

DatasetMethodOldNewOverallMNISTBase98\.9±0\.698\.9\\pm 0\.60\.0±0\.00\.0\\pm 0\.069\.5±0\.469\.5\\pm 0\.4CPT0\.0±0\.00\.0\\pm 0\.098\.3±0\.798\.3\\pm 0\.729\.2±0\.229\.2\\pm 0\.2EWC67\.3±5\.467\.3\\pm 5\.457\.3±7\.157\.3\\pm 7\.129\.5±0\.229\.5\\pm 0\.2SRT98\.5±1\.198\.5\\pm 1\.175\.1±9\.475\.1\\pm 9\.491\.6±2\.691\.6\\pm 2\.6FashionBase97\.0±0\.497\.0\\pm 0\.40\.0±0\.00\.0\\pm 0\.067\.9±0\.067\.9\\pm 0\.0CPT0\.0±0\.10\.0\\pm 0\.194\.4±0\.594\.4\\pm 0\.528\.4±0\.228\.4\\pm 0\.2EWC29\.1±0\.229\.1\\pm 0\.295\.3±0\.295\.3\\pm 0\.228\.6±0\.028\.6\\pm 0\.0SRT53\.7±4\.353\.7\\pm 4\.394\.2±0\.594\.2\\pm 0\.565\.8±3\.165\.8\\pm 3\.1CIFAR\-10Base83\.6±0\.383\.6\\pm 0\.30\.0±0\.00\.0\\pm 0\.050\.1±0\.050\.1\\pm 0\.0CPT0\.0±0\.10\.0\\pm 0\.143\.7±27\.243\.7\\pm 27\.217\.5±10\.917\.5\\pm 10\.9EWC20\.4±12\.820\.4\\pm 12\.857\.7±31\.257\.7\\pm 31\.223\.1±12\.523\.1\\pm 12\.5SRT43\.5±10\.743\.5\\pm 10\.767\.6±21\.667\.6\\pm 21\.653\.1±14\.053\.1\\pm 14\.0WineBase97\.1±0\.297\.1\\pm 0\.20\.0±0\.00\.0\\pm 0\.070\.2±0\.070\.2\\pm 0\.0CPT4\.1±0\.14\.1\\pm 0\.1100\.0±0\.0100\.0\\pm 0\.030\.0±0\.030\.0\\pm 0\.0EWC8\.2±0\.28\.2\\pm 0\.2100\.0±0\.0100\.0\\pm 0\.033\.9±0\.133\.9\\pm 0\.1SRT42\.1±0\.142\.1\\pm 0\.182\.7±0\.282\.7\\pm 0\.253\.3±0\.153\.3\\pm 0\.1Table 11:Vision and tabular results under class\-incremental learning\. Accuracy is reported as mean±\\pmstandard deviation over 10 random seeds \(%\)\.SRTachieves the highest overall accuracy on all four datasets\.These auxiliary experiments are not intended as central evidence for the central claim of this paper, which concerns language model continual pre\-training\. They do, however, suggest that the scheduling mechanism underlying SRT generalizes beyond text to non\-generative classification settings, where the recall quality signal is derived from predicted\-class confidence rather than perplexity\.

## Appendix DExternal Validation on TemporalWiki

To provide external corroboration beyond our source\-grounded QA, we evaluate on TemporalWiki\([16](https://arxiv.org/html/2608.17530#bib.bib11)\), an established benchmark that measures factual knowledge via perplexity on subject\-relation\-object probes\. Probes are categorised as*Changed*\(facts that differ between consecutive Wikipedia snapshots\) and*Unchanged*\(facts that persist\)\. Lower perplexity indicates better retention of the probed fact\.

We score each probe by the perplexity the model assigns to the object tokens conditioned on the subject and relation, and report the mean over each category\. Table[12](https://arxiv.org/html/2608.17530#A4.T12)reports results for Base,CPT, andSRTon Llama\-3\.2\-3B\-Instruct\.

ModelChanged PPLUnchanged PPLAvg\. PPLBase196119612263226321122112CPT571357136706670662096209SRT𝟐𝟓𝟏𝟗\\mathbf\{2519\}𝟑𝟎𝟓𝟐\\mathbf\{3052\}𝟐𝟕𝟖𝟓\\mathbf\{2785\}Table 12:TemporalWiki\([16](https://arxiv.org/html/2608.17530#bib.bib11)\)object\-perplexity on Llama\-3\.2\-3B\-Instruct \(lower is better\)\.CPTnearly triples perplexity relative to Base on both categories, indicating catastrophic forgetting, whileSRTlimits the degradation to roughly30%30\\%above Base\.CPTnearly triples perplexity on both categories relative to the base model, a clear signature of catastrophic forgetting\.SRTlimits the degradation to roughly30%30\\%above base \(27852785vs\.21122112average\), corroborating on an external benchmark the retention advantage observed on our source\-grounded QA\.

Two caveats apply to this evaluation\. First, absolute perplexity values are not comparable to those in the original TemporalWiki paper, which used GPT\-2\-family models; perplexity is tokenizer\-dependent, so only the relative ordering across our conditions is meaningful\. Second, the publicly released TemporalWiki snapshots predate the knowledge cutoff of Llama\-3\.2\-3B\-Instruct, so both the Changed and Unchanged categories represent pre\-cutoff knowledge for our models\. This evaluation therefore serves as an external*retention*check rather than a test of new\-knowledge acquisition; constructing date\-aligned probes that postdate the model cutoff is left to future work\.