Rethinking Transfer in Continual Learning: A Replay-Based Realisation

# Rethinking Transfer in Continual Learning: A Replay-Based Realisation
Source: [https://arxiv.org/html/2607.15587](https://arxiv.org/html/2607.15587)
Yang Meng, Zhenya Liu, Zhuokai Zhao†, Yuxin Chen† Department of Computer Science, University of Chicago ymeng3@uchicago\.edu †Co\-last authors

###### Abstract

Continual learning studies how deployed language models can continually acquire new tasks without expensive retraining from scratch\. Existing methods, whether rehearsal\-based \(replaying stored past data\) or rehearsal\-free \(regularising or isolating parameters\), overwhelmingly target one objective: preventing catastrophic forgetting\. Forward transfer — the past helping the future — has meanwhile been pursued almost exclusively through parameter reuse, with no explicit account of when transfer should be expected at all\. We begin one step earlier: before designing a transfer mechanism, we ask when transfer should exist at all\. We answer with a framework of three measurable conditions: the target task must leave room for improvement beyond its own limited supervision, transferable information must survive continued optimisation, and replay must come from compatible previous tasks\. We instantiate this view as Transfer\-Selective Replay \(TSR\), which selects replay data predicted to benefit the incoming task rather than replaying past examples indiscriminately\. Selection is guided by a zero\-training task signature, while distillation preserves stability on previous tasks\. Under the standard continual\-learning protocol in the low\-budget regime, TSR consistently improves forward transfer while maintaining stability, outperforming existing replay baselines across heterogeneous and homogeneous task streams\. More broadly, the results argue for treating transfer as a first\-class objective of continual learning, to be understood before it is engineered\.

††footnotetext:Code is available at[https://github\.com/ymeng3/transfer\-selective\-replay](https://github.com/ymeng3/transfer-selective-replay)\.## 1Introduction

Large language models are trained once, at enormous cost, and then deployed into a world that does not stand still: new domains, new formats, and new skills keep arriving after release\(Wanget al\.,[2023b](https://arxiv.org/html/2607.15587#bib.bib25);[2024](https://arxiv.org/html/2607.15587#bib.bib19); Tankalaet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib47); Asawaet al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib46)\)\. Retraining from scratch for every addition is prohibitive, so a deployed model must learn tasks*sequentially*— the setting of continual learning\(De Langeet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib18); Ke and Liu,[2022](https://arxiv.org/html/2607.15587#bib.bib63); Momeniet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib48); Chenet al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib35)\)\. Its defining failure mode has been known for decades:*catastrophic forgetting*, where optimising for the new task overwrites what earlier tasks built\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib32); Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2607.15587#bib.bib54); Zhenget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib50)\)\. The field frames the resulting tension as a*stability–plasticity*trade\-off: stability preserves the past, plasticity lets the model keep learning, and improving either naively costs the other\(De Langeet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib18); Momeni and Liu,[2025](https://arxiv.org/html/2607.15587#bib.bib52)\)\.

Current continual learning methods can be broadly divided into two families\.*Rehearsal\-based*methods keep a small buffer of past examples and use it during new\-task training — replayed directly\(Chaudhryet al\.,[2019b](https://arxiv.org/html/2607.15587#bib.bib12); Rebuffiet al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib14)\), with logit anchoring\(Buzzegaet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib13)\), selected for retention\(Aljundiet al\.,[2019a](https://arxiv.org/html/2607.15587#bib.bib15);[b](https://arxiv.org/html/2607.15587#bib.bib16)\), or as gradient constraints\(Chaudhryet al\.,[2019a](https://arxiv.org/html/2607.15587#bib.bib33)\)\.*Rehearsal\-free*methods keep no data: they regularise parameters toward past solutions\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib32); Li and Hoiem,[2017](https://arxiv.org/html/2607.15587#bib.bib17)\)or isolate tasks in separate parameter subsets — which in the LLM era usually means separate LoRA adapters\(Huet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib1)\), kept orthogonal\(Wanget al\.,[2023a](https://arxiv.org/html/2607.15587#bib.bib3)\), gated\(Lianget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib7)\), or organised hierarchically\(Qianet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib36)\)\. The families differ in machinery but agree on the objective: both answer the stability question,*how do we keep the past from being lost?*

Continual LearningTransfer“let the past help the new”Stability“preserve the past”Where is it possible?audit headroomWhat carries it?a persistent carrierWhom to draw on?route to the right sourceReplayconventional role:protect old tasks \(§[2](https://arxiv.org/html/2607.15587#S2)\)Distillation \(KD\)the stability termParameter reuseextensively explored \(§[2](https://arxiv.org/html/2607.15587#S2)\)Datathis work: select*which past**data*the new task trains withthis work: point replay at the*new*task as wellTransfer\-Selective Replay \(one instantiation\)audited headroom⋅\\cdotselected replay⋅\\cdotdistillation for stability

Figure 1:A framework for continual transfer, and one instantiation\.Continual learning serves two objectives\. On the transfer side we organise the design space around three questions — where transfer is possible \(headroom\), what carries it \(parameters vs\. data\), and whom to draw on \(source selection\); prior transfer mechanisms occupy the parameter branch \(§[2](https://arxiv.org/html/2607.15587#S2)\), and this work takes the data branch, turning transfer into a data\-selection problem\. On the stability side, replay’s conventional role is protecting old tasks; we point the same machinery at the*new*task as well \(dashed arrow\) and assign the stability term to distillation \(§[5\.3](https://arxiv.org/html/2607.15587#S5.SS3)\)\. Transfer\-Selective Replay instantiates all three answers \(§[5](https://arxiv.org/html/2607.15587#S5)\)\.The complementary question — whether past tasks can help learning future ones — has received far less attention\.*Forward transfer*, the improvement in learning a new task due to knowledge acquired from previous tasks, is the promised upside of continual learning; it has been measured since GEM\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2607.15587#bib.bib54)\)and designed for since Progressive Networks\(Rusuet al\.,[2016](https://arxiv.org/html/2607.15587#bib.bib53)\)\. Yet on modern LLM benchmarks it is routinely reported as absent or negative\. Existing attempts to promote forward transfer have relied almost exclusively on*parameter reuse*: lateral connections and module reuse\(Rusuet al\.,[2016](https://arxiv.org/html/2607.15587#bib.bib53); Veniatet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib61)\), task masks and shared adapters\(Keet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib58);[2021a](https://arxiv.org/html/2607.15587#bib.bib60)\), prompt reuse\(Qin and Joty,[2022](https://arxiv.org/html/2607.15587#bib.bib62)\), and, for LLMs, routing, initialising from, or merging past adapters\(Huanget al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib8); Ilharcoet al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib9); Qianet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib36)\)\. The assumption behind this line is so uniform that it is rarely stated:*past knowledge lives in past parameters*\. Past\-task*data*, meanwhile, has been used almost exclusively to prevent forgetting rather than to improve learning on new tasks\. Replay has been developed to preserve old\-task performance, while forward transfer has relied on parameter reuse; the two directions have remained largely separate\. And although negative transfer is increasingly documented\(Choi and Kim,[2026](https://arxiv.org/html/2607.15587#bib.bib34); Chenet al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib35)\), the field has no explicit formulation of*when*transfer should be expected at all\.

This paper starts one step earlier:*before designing a transfer mechanism, we ask what makes transfer possible\.*We study the question in*low\-budget*continual learning — each task arrives with a fixed, small training sample\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib2)\)\. This is the continual\-learning regime where forward transfer becomes operationally valuable: with plentiful target data a task saturates on its own and leaves nothing for the past to supply \(§[4\.1](https://arxiv.org/html/2607.15587#S4.SS1)\)\. Under a fixed budget, the value of transfer is not to replace target supervision but to improve the performance attainable from that budget by exploiting previously acquired experience\.

We identify three measurable conditions that determine whether transfer is possible \(§[4](https://arxiv.org/html/2607.15587#S4)\):

- •*Where can transfer happen?*Only where the target task leaves headroom beyond what its own limited supervision can achieve — an auditable property before any transfer mechanism is introduced \(§[4\.1](https://arxiv.org/html/2607.15587#S4.SS1)\)\.
- •*What carries transfer?*Only a persistent carrier\. Replay data repeatedly reinforces the same optimisation direction throughout training, whereas one\-shot parameter reuse is progressively washed out \(§[4\.2](https://arxiv.org/html/2607.15587#S4.SS2)\)\.
- •*Whom should the new task draw on?*Source quality varies sharply, but compatible sources can be identified at zero training cost using a label\-aware*task signature*\(§[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)\)\.

We organise the design space of continual transfer around these three questions \(Figure[1](https://arxiv.org/html/2607.15587#S1.F1)\)\.

Transfer\-Selective Replay\(TSR\) is one*instantiation*of this framework\. Choosing replay data as the carrier turns forward transfer into a*data\-selection*problem: the central question is no longer whether to replay, but which previous experience should be replayed for the incoming task\. Concretely, TSR maintains a task memory holding one record per past task — a task signature, a small data subsample, and the adapter snapshot from when that task ended\. When a new task arrives, a few forward–backward passes at a fixed shared initialisation produce its signature at zero training cost; a softmax over signature similarities then routes every replay batch to the past data most aligned with the new task, and each replayed batch is distilled against its own era’s snapshot\.Distillationcomplements replay by preserving stability, allowing replay to focus on transfer — a role our analysis later identifies as optimisation regularisation rather than knowledge preservation \(§[5\.3](https://arxiv.org/html/2607.15587#S5.SS3)\)\.

We evaluate TSR under the standard continual\-learning protocol on TRACE\-8\(Wanget al\.,[2023b](https://arxiv.org/html/2607.15587#bib.bib25)\)and NumGLUE\-8\(Mishraet al\.,[2022](https://arxiv.org/html/2607.15587#bib.bib65)\), across three backbone families \(Qwen2\.5\(Qwen Team,[2024](https://arxiv.org/html/2607.15587#bib.bib27)\), Gemma 3\(Gemma Team,[2025](https://arxiv.org/html/2607.15587#bib.bib29)\), and Llama 3\.2\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib28)\)\) and thirteen state\-of\-the\-art continual\-learning methods spanning both rehearsal\-based and rehearsal\-free approaches\. TSR consistently outperforms existing replay baselines by improving forward transfer while maintaining positive backward transfer\.

##### Contributions\.

1. 1\.A framework for continual transfer\.Three questions — where \(headroom\), what \(carrier\), whom \(source\) — each made measurable: benchmark headroom audits with oracle ceilings before mechanism design, a same\-benchmark carrier dissociation, and the first pairwise data\-transfer matrix in continual learning with LLMs, including its negative cells \(§[4](https://arxiv.org/html/2607.15587#S4)\)\.
2. 2\.An instantiation that redefines the role of replay\.The field’s most reliable stability tool serves both objectives, and had so far been assigned only one: TSR points replay at the*current*task, routed by a zero\-training, label\-aware signature over a task memory of \(signature, data, snapshot\) records\. Its selection matches the hindsight\-oracle ceiling, and it is the strongest replay policy under the standard protocol \(§[5](https://arxiv.org/html/2607.15587#S5)–[6](https://arxiv.org/html/2607.15587#S6)\)\.
3. 3\.A clarification of the role of distillation in continual LLM learning\.Through a systematic teacher audit — identity, content, loss shaping, timing, snapshot source — we show that distillation contributes through optimisation*anchoring*rather than transferable teacher knowledge: its teachers matter as anchors, not as knowledge sources\. This dissociates the field’s knowledge\-preservation narrative from the operational mechanism, and motivates assigning distillation exclusively to the stability objective \(§[5\.3](https://arxiv.org/html/2607.15587#S5.SS3), Appendix[B\.4](https://arxiv.org/html/2607.15587#A2.SS4)\)\.

## 2Related Work and Positioning

##### Forward transfer\.

Transfer as an explicit goal of continual learning is not new:Lopez\-Paz and Ranzato \([2017](https://arxiv.org/html/2607.15587#bib.bib54)\)formalised forward and backward transfer metrics, and designing for forward transfer dates to Progressive Networks\(Rusuet al\.,[2016](https://arxiv.org/html/2607.15587#bib.bib53)\)\. GEM’s forward transfer is a zero\-shot quantity \(test before training\); the training\-time sense studied here — earlier tasks improving the*learning*of the current one — is the promised upside of sequential learning, and reusing source data for it is well studied outside CL as intermediate\-task training \(STILTs;Phanget al\.,[2018](https://arxiv.org/html/2607.15587#bib.bib23); Vuet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib44)\) without ever being instantiated inside a continual stream\. What is missing is therefore not the objective but a*formulation*: an account of when transfer should exist, and how to measure it, before a mechanism is designed \(§[4](https://arxiv.org/html/2607.15587#S4)\)\.

##### Parameter\-side transfer\.

The mechanisms proposed to realise transfer during training are almost exclusively parameter\-side: module reuse\(Veniatet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib61)\), masks and adapters over a shared backbone \(CAT/B\-CL/CTR;Keet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib58);[2021b](https://arxiv.org/html/2607.15587#bib.bib59);[2021a](https://arxiv.org/html/2607.15587#bib.bib60)\), meta\-learned representations\(Javed and White,[2019](https://arxiv.org/html/2607.15587#bib.bib56)\), and soft\-prompt reuse\(Qin and Joty,[2022](https://arxiv.org/html/2607.15587#bib.bib62)\)— a framing the NLP\-CL survey ofKe and Liu \([2022](https://arxiv.org/html/2607.15587#bib.bib63)\)makes explicit by listing all transfer mechanisms as architectural\. In the LLM era the same assumption drives inference\-time composition \(LoRAHub, task arithmetic, AdapterSoup;Huanget al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib8); Ilharcoet al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib9); Chronopoulouet al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib10)\) and continual\-training constraints: orthogonal adapters \(O\-LoRA/N\-LoRA;Wanget al\.,[2023a](https://arxiv.org/html/2607.15587#bib.bib3); Yanget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib4)\), single\-adapter merging\(Qiao and Mahdavi,[2025](https://arxiv.org/html/2607.15587#bib.bib5)\), and gated or routed per\-task adapters\(Zhanget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib6); Lianget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib7); Ostapenkoet al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib11)\)\. These works answer the design question differently but share one assumption: parameters are the transferable object\. This paper investigates the complementary data\-side formulation, and §[4\.2](https://arxiv.org/html/2607.15587#S4.SS2)measures where each carrier survives — inference\-time composition is never trained against, and nothing erases it\.

##### Data replay\.

Replay is classically a stability mechanism \(ER, DER\+\+, iCaRL;Chaudhryet al\.,[2019b](https://arxiv.org/html/2607.15587#bib.bib12); Buzzegaet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib13); Rebuffiet al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib14)\), and the replay\-selection literature optimises the same objective: MIR\(Aljundiet al\.,[2019a](https://arxiv.org/html/2607.15587#bib.bib15)\)picks maximally interfered samples, GSS\(Aljundiet al\.,[2019b](https://arxiv.org/html/2607.15587#bib.bib16)\)gradient\-diverse ones\. The structurally closest works remain retention\-directed: MER\(Riemeret al\.,[2019](https://arxiv.org/html/2607.15587#bib.bib55)\)places a generic buffer inside a transfer\-maximising meta\-objective without selecting*which*past task helps; Adaptive Memory Replay\(Smith and others,[2024](https://arxiv.org/html/2607.15587#bib.bib37)\)selects past\-data clusters conditioned on the current task, but its bandit reward is a forgetting metric; InsCL\(Wang and others,[2024](https://arxiv.org/html/2607.15587#bib.bib38)\)modulates replay by task similarity in the*inverse*direction \(dissimilar tasks get more replay, to protect them\); and learned replay controllers\(Shihabet al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib39)\)frame replay as data selection, again for retention\. Replay has been used to preserve the past, not to accelerate the future — the objective this paper inverts \(§[5](https://arxiv.org/html/2607.15587#S5)\)\.

Pointing replay at the new task then requires knowing which past task will help — a task\-similarity question\. Fisher embeddings \(Task2Vec;Achilleet al\.,[2019](https://arxiv.org/html/2607.15587#bib.bib20); Vuet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib44)\), optimal\-transport distances \(OTDD;Alvarez\-Melis and Fusi,[2020](https://arxiv.org/html/2607.15587#bib.bib21); Nguyenet al\.,[2025a](https://arxiv.org/html/2607.15587#bib.bib43)\), gradient\-based task groupings\(Fiftyet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib22)\), and example\-level gradient selection \(LESS;Xiaet al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib42)\) all predict task relationships; concurrent work predicts transfer from dataset traits\(Krishnaet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib40)\)or selects task orderings\(Nguyenet al\.,[2025b](https://arxiv.org/html/2607.15587#bib.bib41)\)\. Closest inside CL, TreeLoRA\(Qianet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib36)\)routes new tasks to adapter groups by gradient similarity at*evolving*weights — the same class of signal, coupled to a parameter carrier; the carrier dissociation of §[4\.2](https://arxiv.org/html/2607.15587#S4.SS2)predicts where that coupling holds\. The measured pairwise transfer matrix of §[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)provides the prediction target these representations have lacked: signatures are validated against measured transfer rather than a routing heuristic\.

##### Distillation\.

In continual learning, distillation is narrated as knowledge preservation — from LwF and iCaRL\(Li and Hoiem,[2017](https://arxiv.org/html/2607.15587#bib.bib17); Rebuffiet al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib14)\)to DER\+\+’s stored logits\(Buzzegaet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib13)\)and recent self\-distillation\(Yanget al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib66)\)\. Adjacent fields already read KL\-anchoring as regularisation: reference\-policy penalties in RL\(Tehet al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib68)\)and label\-smoothing accounts of supervised distillation\(Yuanet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib69); Mobahiet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib70)\)\. Missing is the continual\-LLM dissociation — whether the teacher’s*knowledge*, or merely its existence as an*anchor*, produces the gains; §[5\.3](https://arxiv.org/html/2607.15587#S5.SS3)answers by audit\.

##### Negative transfer\.

MERIT\(Choi and Kim,[2026](https://arxiv.org/html/2607.15587#bib.bib34)\)detects gradient\-level conflict in*parallel*multi\-task tuning; OPR\(Chenet al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib35)\)selects replayed rollouts by quality rather than source\-task identity; surveys note that replay can induce negative forward transfer in fragile domains\(Wanget al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib19)\)\. These works document the phenomenon\. Rather than documenting it, this paper predicts and avoids it: the negative cells of the transfer matrix are exactly what source selection routes around \(§[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)\)\.

## 3Problem Formulation

### 3\.1Continual learning

We consider a task\-incremental continual learning setting: a learner faces a stream of tasks𝒯=\(T1,…,TK\)\\mathcal\{T\}=\(T\_\{1\},\\dots,T\_\{K\}\), arriving one at a time, where each taskTtT\_\{t\}is a distributionDtD\_\{t\}over input–output pairs and provides a training and a test sample from it\. We study continual fine\-tuning with a single accumulating parameter state: the learner maintains one set of trainable parametersθ\\thetathroughout the stream; no per\-task parameters are added and no task identity \(i\.e\., which task a test example belongs to\) is available at inference\. When taskTtT\_\{t\}arrives, sequential learning solves

θt=arg⁡minθ⁡𝔼\(x,y\)∼Dt​\[ℓ​\(fθ​\(x\),y\)\],\\theta\_\{t\}\\;=\\;\\arg\\min\_\{\\theta\}\\;\\mathbb\{E\}\_\{\(x,y\)\\sim D\_\{t\}\}\\\!\\left\[\\,\\ell\\\!\\left\(f\_\{\\theta\}\(x\),\\,y\\right\)\\right\],\(1\)optimised from the previous stateθt−1\\theta\_\{t\-1\}; earlier tasks are accessible only through bounded experience retained from the stream \(possibly none\)\(De Langeet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib18)\)\. Solving Eq\. \([1](https://arxiv.org/html/2607.15587#S3.E1)\) naively is known to degrade earlier tasks — catastrophic forgetting — and, less discussed, to forgo any help earlier tasks could give the current one; continual\-learning methods differ in how they approximate Eq\. \([1](https://arxiv.org/html/2607.15587#S3.E1)\) under this constraint \(§[5](https://arxiv.org/html/2607.15587#S5)\)\. Throughout, we study the*low\-budget*regime: each incoming task provides only a limited number of labeled examples, while experience from earlier tasks is already in hand\. The concrete budget and the instantiation of the parameter state are described in §[6\.1](https://arxiv.org/html/2607.15587#S6.SS1)\.

### 3\.2Transfer

Throughout this paper,*forward transfer*refers to the contribution of previously acquired experience to the learning of the current task, under the fixed target\-data budget\. Conceptually, we decompose current\-task learning into two contributions:

current\-task learning=target\-data contribution⏟what the task teaches itself\+transfer contribution⏟what the past supplies\.\\text\{current\-task learning\}\\;=\\;\\underbrace\{\\text\{target\-data contribution\}\}\_\{\\text\{what the task teaches itself\}\}\\;\+\\;\\underbrace\{\\text\{transfer contribution\}\}\_\{\\text\{what the past supplies\}\}\.The transfer contribution is the object of this paper\. It is distinct from GEM’s zero\-shot forward\-transfer metric \(test before training\)\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2607.15587#bib.bib54)\)and from generalisation to unseen tasks; and it is*latent*— no standard continual\-learning metric reads it off directly\. Since the contribution is not directly observable, the remainder of the paper introduces measurable surrogates: §[4](https://arxiv.org/html/2607.15587#S4)constructs its measurement and asks under what conditions it exists at all\.

## 4Three Conditions for Realized Transfer

We organize the investigation around a simple decomposition of when forward transfer is realized in a continual stream:

transfer​\(j→t\)\>0​requires​headroom​\(t\)⏟Where?∧a persistent carrier⏟What?∧the right source​j⏟Whom?,\\text\{transfer\}\(j\\to t\)\>0\\;\\;\\text\{requires\}\\;\\;\\underbrace\{\\text\{headroom\}\(t\)\}\_\{\\textit\{Where?\}\}\\;\\wedge\\;\\underbrace\{\\text\{a persistent carrier\}\}\_\{\\textit\{What?\}\}\\;\\wedge\\;\\underbrace\{\\text\{the right source \}j\}\_\{\\textit\{Whom?\}\},\(2\)We examine each condition in turn; §[5](https://arxiv.org/html/2607.15587#S5)then derives the method from them\.

### 4\.1Where: headroom as transfer opportunity

Before asking how transfer can be realised, we must first ask whether any transfer opportunity exists at all\.

###### Condition 1

Without benchmark headroom, transfer is unmeasurable by any mechanism: realised transfer is bounded by what additional in\-domain data buys\.

##### Headroom\.

The transfer contribution of §[3](https://arxiv.org/html/2607.15587#S3)is latent — training on mixed data yields one outcome in which target learning and transfer are entangled — so we first need a diagnostic for*where*it could appear at all\. Three training runs under identical budgets — the*probe*protocol used throughout — define every quantity in the paper:target\-data\(theNNtarget examples alone\),target\-full\(10​N10Ntarget examples\), andmixed\(j\)\(j\)\(a50/5050/50mixture of source taskjj’s data and theNNtarget examples\)\. WritingSt​\(N\)S\_\{t\}\(N\)for end\-of\-training test accuracy after training onNNof tasktt’s own examples,

headroomt​\(N\)=St​\(Nmax\)−St​\(N\),\\mathrm\{headroom\}\_\{t\}\(N\)\\;=\\;S\_\{t\}\(N\_\{\\max\}\)\\;\-\\;S\_\{t\}\(N\),\(3\)withNmaxN\_\{\\max\}a fixed well\-resourced reference \(ten times the standard budget in our protocol\): the gain that additional*in\-domain*supervision — the most valuable supervision a task can receive — would still buy at budgetNN\.

##### Interpretation\.

Headroom measures how much improvement remains unavailable from the task’s own budget —*task saturation*, and through it the*transfer opportunity*\. Figure[2](https://arxiv.org/html/2607.15587#S4.F2)\(a\) is the evidence: the target\-data curve rises from\.397\.397atN=10N\{=\}10to\.588\.588atN=500N\{=\}500, so headroom is not a fixed property of a benchmark but a function of the target\-data budget — large exactly where labels are scarce\.

101050505005000\.40\.40\.50\.50\.60\.6transfer opportunitytarget\-dataSt​\(N\)S\_\{t\}\(N\)Target examplesNNTarget accuracy\(a\) Opportunity10105050500500−0\.05\-0\.0500\.050\.05Target examplesNNGain over target\-data\(b\) Transfer gainTSRERParam\-init1001003003005005000\.40\.40\.50\.50\.60\.6Training step \(atN=10N\{=\}10\)Target accuracy\(c\) Persistent gainTarget\-dataTSRERTarget\-fullFigure 2:Transfer opportunity shrinks with the target budget, and the gain lives where opportunity is large\(flagship target numglue\-cm, the matrix’s strongest pair;n=5n\{=\}5seeds per point; shading±1\\pm 1s\.e\.m\.\)\.*\(a\)*headroom vs\. budget;*\(b\)*paired transfer gain over target\-data \(TSR denotes selected replay, ER uniform replay\);*\(c\)*atN=10N\{=\}10the gain holds from the first evaluation to the last step\. Per\-target numbers: Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2)\.
##### Quantification\.

Since the contribution is latent, we report its practical effect as*equivalent target supervision*\(a sample\-efficiency reading\): a measured gain at budgetNNis converted, through theSt​\(N\)S\_\{t\}\(N\)curve, into the number of additional target labels that would buy the same improvement\. This is the long\-standing convention in transfer measurement — samples\-to\-threshold in RL transfer\(Taylor and Stone,[2009](https://arxiv.org/html/2607.15587#bib.bib74)\), source ranking under limited target supervision in taskonomy\(Zamiret al\.,[2018](https://arxiv.org/html/2607.15587#bib.bib75)\), and*effective data transferred*in neural scaling laws\(Hernandezet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib76)\)\. Figure[2](https://arxiv.org/html/2607.15587#S4.F2)\(b\): atN=10N\{=\}10, selected mixing gains\+\.066\+\.066— the performance of≈20\{\\approx\}20target examples, a2×2\\timeslabel efficiency,35%35\\%of the remaining opportunity — and the gain decays as saturation closes, never crossing zero on this target; across all five classification targets the same pattern holds with smaller amplitude, turning negative once headroom is spent\. Panel \(c\) shows the gain is present from the first evaluation and sustained for all500500steps — not an endpoint artefact \(per\-target numbers are in Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2)\)\. The two other arms preview the remaining conditions: uniform replay \(ER\) captures less than half the low\-budget gain and sits below selection at every budget \(§[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)\); the parameter\-init arm never separates from zero \(§[4\.2](https://arxiv.org/html/2607.15587#S4.SS2)\)\.

##### Implication\.

A benchmark with zero headroom cannot reveal transfer, whatever the mechanism\. On the canonical 15\-task suite ofRazdaibiedinaet al\.\([2023](https://arxiv.org/html/2607.15587#bib.bib24)\)\(PP15;Wanget al\.,[2023a](https://arxiv.org/html/2607.15587#bib.bib3); Qin and Joty,[2022](https://arxiv.org/html/2607.15587#bib.bib62)\), an instruction\-tuned 0\.5B base leaves essentially none: ten times more target data buys\+0\.02\+0\.02on average, and an audit of fourteen parameter\-reuse configurations with an oracle\-selected source lands within one seed standard deviation of the plain baseline \(Appendix[A\.1](https://arxiv.org/html/2607.15587#A1.SS1)\)\. TRACE\-8\(Wanget al\.,[2023b](https://arxiv.org/html/2607.15587#bib.bib25)\), by contrast, retains real headroom at the same scale — hence our benchmark choice \(§[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)draws the benchmark\-design corollary\)\.Wanget al\.\([2023b](https://arxiv.org/html/2607.15587#bib.bib25)\)argued qualitatively that classic suites lack challenge for aligned LLMs, andAkhtaret al\.\([2026](https://arxiv.org/html/2607.15587#bib.bib45)\)recommend reporting saturation statistics; the headroom audit is the actionable, budget\-conditioned version\. Our merge\-protocol artefact — an apparent transfer signal35×35\\timesits protocol\-clean value — joins the documented genus of evaluation artefacts\(Konget al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib49); Zhenget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib50); Sunet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib51)\)\.

### 4\.2What: the carrier must persist

Headroom tells us whether transfer is*possible*; it does not tell us how the transferred signal reaches the learner\. This motivates the next question:*what carries transfer?*A carrier can take many forms — parameters, data, prompts, external memory — and the stream constrains it only to be something retained from past tasks\. Whatever its form, the carrier’s signal must survive the continual optimisation\.

###### Condition 2

Realised transfer requires a carrier that preserves its signal throughout optimisation; a transient carrier is erased regardless of headroom or source quality\.

Figure[3](https://arxiv.org/html/2607.15587#S4.F3)tests the requirement across candidate carriers on TRACE\-8: identical budgets, the same source, different carriers\. A transient carrier — one\-shot initialisation from the source\-trained adapter — captures none of the mixing gain; its early advantage is real but erased by continued training \(*wash\-out*\)\. A persistent*parameter*carrier — an L2 anchor toward the same source adapter, kept in the loss at every step — captures roughly two\-thirds \(Appendix[B\.2](https://arxiv.org/html/2607.15587#A2.SS2)\)\. Persistent*data*— the source’s examples mixed into every batch — captures all of it\. The transfer gain thus rises with persistence, not with whether the carrier is parameters or data\. The same axis organises the literature: frozen columns\(Rusuet al\.,[2016](https://arxiv.org/html/2607.15587#bib.bib53)\)and inference\-time composition\(Huanget al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib8)\)succeed and are persistent by construction — the source signal is never optimised away — whereas merge\-then\-train pipelines re\-enter at the transient end; the literature appears contradictory only because it mixes transient and persistent parameter mechanisms \(Appendix[B\.1](https://arxiv.org/html/2607.15587#A2.SS1)\)\.

Among the carriers we evaluate, replay data is the simplest persistent one: no parameter growth, no architectural change, no task identity at inference\. Data is therefore not privileged because it is data, but because it is the simplest carrier whose signal is reintroduced throughout optimisation — the premise from which §[5](https://arxiv.org/html/2607.15587#S5)derives the method, and which §[6](https://arxiv.org/html/2607.15587#S6)stress\-tests at benchmark scale\.

TransientPersistentOne\-shotparameter reusePersistentparameter anchorReplay dataFrozenparametersInference\-timecompositionTransfercaptured\(measured\)persistent byconstruction \(literature\)Figure 3:Candidate carriers, ordered by the persistence of their optimization signal\.The transfer gain follows persistence rather than whether the carrier is parameters or data; bars show the measured share of the data\-mixing accuracy gain each carrier captures \(Appendix[B\.2](https://arxiv.org/html/2607.15587#A2.SS2)\)\. The literature examples shown here are persistent by construction\.
### 4\.3Whom: source selection

Knowing that transfer is possible, and knowing what carries it, still leaves one question open: which previous task should the learner draw on?

###### Condition 3

Given headroom and a persistent carrier, realised transfer is determined by source choice — including its sign; and the choice is predictable from label\-aware learning directions at the shared initialisation\.

#### 4\.3\.1The transfer matrix

We measure the full transfer matrix over all ordered pairs of TRACE\-8 \(Table[1](https://arxiv.org/html/2607.15587#S4.T1)\)\.

Table 1:Transfer matrix on TRACE\-8 \(pairwise joint training of sourcejjmixed50/5050/50withN=50N\{=\}50target examples vs\. target\-data; mean of55seeds; bold = column best\)\.Green= source helps the target \(darker = larger loss reduction\);red= source*hurts*\.Three structural findings\.\(1\)*Semantic pairs are mutual best sources*: fomc↔\\leftrightarrowcstance are each other’s best source; NumGLUE\-cm is NumGLUE\-ds’s best at−0\.88\-0\.88, the largest cell\. \(2\)*Negative transfer is real and target\-specific*: cstance, scienceqa, and numglue\-cm are fragile targets where the wrong source actively hurts \(worst cells\+0\.27\+0\.27,\+0\.26\+0\.26\) — the empirical basis for §[5](https://arxiv.org/html/2607.15587#S5)’s headline\. \(3\)*Generation targets absorb anything*\(−0\.2\-0\.2to−0\.85\-0\.85from nearly all sources\): pair\-specific selection is a classification\-task phenomenon here\.

##### Benchmark\-design corollary\.

Contrasting the two benchmarks:*a CL benchmark should report its data headroom \(at a stated data budget\) and transfer structure as part of its specification*\. PP15\-class suites evaluated with instruction\-tuned bases have neither, and any transfer comparison run on them inherits a ceiling of zero regardless of method \(Appendix[A\.1](https://arxiv.org/html/2607.15587#A1.SS1)\) — which may explain why routing and reuse comparisons in this literature so often report method\-insensitive results\.

#### 4\.3\.2Task signatures

The matrix defines what a selector must predict; measuring it requires training every ordered source–target pair — feasible for analysis, impractical in continual learning, where the source must be chosen the moment a new task arrives\. We therefore seek a*task signature*: a training\-free representation of each task from which measured transfer can be predicted\. The candidates we evaluate span three increasingly informative families — input similarity, dataset geometry, and learning direction \(Table[2](https://arxiv.org/html/2607.15587#S4.T2)\)\. Only the*gradient signature*consistently identifies the best source:

gt=1m∑b=1m∇BℒCE\(b\)\|\(A0,B0\),score\(j,t\)=cos\(gj,gt\),g\_\{t\}\\;=\\;\\frac\{1\}\{m\}\\sum\_\{b=1\}^\{m\}\\nabla\_\{B\}\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(b\)\\Big\\rvert\_\{\(A\_\{0\},B\_\{0\}\)\},\\qquad\\mathrm\{score\}\(j,t\)=\\cos\(g\_\{j\},g\_\{t\}\),\(4\)Heregtg\_\{t\}is the mean gradient over themmprobe batches, taken at the fixed shared initialisation\(A0,B0\)\(A\_\{0\},B\_\{0\}\)and flattened across layers;BBis the zero\-initialised LoRA factor, sogtg\_\{t\}is the adapter’s entire first\-order learning direction, and sources are scored by the cosine between directions\.

##### Mechanism\.

The signature is not a semantic representation: it captures the*optimisation bias*a task induces at the shared initialisation — the direction in which the task asks the adapter to change — so it reflects what a task asks the model to*learn*, not what its data looks like\. This is also why alignment should predict transfer under replay\. Replayed batches act on the same accumulating adapter as the target’s own batches: gradient steps from an aligned source reinforce the trajectory the target demands — the replay supplies optimisation bias rather than additional supervision — whereas steps from a conflicting source compete with the target update for the same parameters\. If this interpretation is correct, aligned source updates should produce more compatible descent on the target than conflicting ones\.

##### Local intervention\.

We test this optimisation hypothesis by intervention\. Holding the adapter at the shared initialisation — the state where signatures are taken — we apply one target update, add an equal\-norm gradient perturbation from a past task — so that only the replay direction, not its magnitude, differs across sources — and measure the loss change on an independent held\-out target batch, isolating its effect from additional target supervision or long\-horizon optimisation \(protocol in Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2.SSS0.Px3)\)\. Sources grouped by the stored signature produce strictly ordered marginal descent \(Table[2](https://arxiv.org/html/2607.15587#S4.T2), right\), demonstrating that replay directions selected by the signature provide progressively more compatible optimisation steps\. Current\-gradient alignment predicts the immediate reduction within target \(Spearmanρ=\.87\\rho=\.87\), and the stored signature preserves this geometry \(ρ=\.85\\rho=\.85against current\-gradient cosines\) — routing on the stored signature therefore predicts compatible descent rather than input similarity; §[6](https://arxiv.org/html/2607.15587#S6)tests whether this local effect survives repeated optimisation\.

Table 2:Task signatures evaluated against the measured transfer matrix of Table[1](https://arxiv.org/html/2607.15587#S4.T1)\.Deployment selects one source, so we score by best\-source top\-1 hit rate; robustness checks are in Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2.SSS0.Px2)\. JointOT is our OTDD\-style label\-aware distributional baseline\(Alvarez\-Melis and Fusi,[2020](https://arxiv.org/html/2607.15587#bib.bib21); Nguyenet al\.,[2025a](https://arxiv.org/html/2607.15587#bib.bib43)\)\. Right: local intervention at the shared initialisation \(three classification targets; protocol in Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2.SSS0.Px3)\)\.
##### Takeaways\.

\(i\)The gradient signature is the only reliable argmax predictor: it identifies the oracle\-best source for4/84/8targets overall and4/54/5classification targets; its one miss selects the matrix’s second\-best source\. \(ii\)Direction, not magnitude, carries the signal: squaring gradients \(Fisher/task2vec\-style\) destroys it; normalising per\-batch directions reproduces it exactly\. \(iii\)Functional beats distributional: input\-similarity and distribution\-geometry signatures find the best source for at most2/82/8targets, and adding labels does not fix it \(JointOT: top\-10/80/8despite ranking well\) — the failures mirror the mechanism: distributional signatures ignore what tasks ask the model to*do*, and gradient magnitudes ignore*where*the demands point\. Further analyses are reported in Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2.SSS0.Px2)\.

With measurable answers to*where*,*what*, and*whom*, we now construct one algorithmic instantiation\.

## 5Transfer\-Selective Replay

TSR is one instantiation of the §[4](https://arxiv.org/html/2607.15587#S4)framework\. The three conditions enter at two different levels: headroom admits no algorithm — it is a property of the benchmark, base, and budget, settled by the audit before any mechanism is designed \(§[4\.1](https://arxiv.org/html/2607.15587#S4.SS1)\) — so the algorithm instantiates the remaining two, carrying transfer with persistent data \(§[4\.2](https://arxiv.org/html/2607.15587#S4.SS2)\) and choosing sources by task signature \(§[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)\), while distillation takes over the stability objective that replay no longer serves \(§[5\.3](https://arxiv.org/html/2607.15587#S5.SS3)\)\. We present the general form first \(Algorithm[1](https://arxiv.org/html/2607.15587#alg1)\), then the instantiation evaluated in this paper \(Algorithm[2](https://arxiv.org/html/2607.15587#alg2), Figure[4](https://arxiv.org/html/2607.15587#S5.F4)\)\.

### 5\.1The general algorithm

Algorithm[1](https://arxiv.org/html/2607.15587#alg1)is what the framework requires of*any*instantiation: standard sequential training of an adapter, plus three replaceable modules —Probe\(represent the incoming task before training\),Route\(weigh past tasks by predicted transfer\), andReference\(anchor each batch for stability\) — with transfer carried by routed replay in every optimisation step\. The modules are intentionally unspecified here; §[5\.2](https://arxiv.org/html/2607.15587#S5.SS2)fixes them\.

##### Notation\.

Both algorithms share the following symbols\.DtD\_\{t\}is tasktt’s training distribution, andbba minibatch;θ\\thetais the trainable adapter,θ0\\theta\_\{0\}the shared initialisation,θt\\theta\_\{t\}the snapshot frozen when taskttends\. Each past taskjjleaves one record\(gj,ℳj,θj\)\(g\_\{j\},\\,\\mathcal\{M\}\_\{j\},\\,\\theta\_\{j\}\)— signature, stored data subsample, snapshot — in the task memory𝒫\\mathcal\{P\};wwis the routing distribution over these records, andθref\\theta^\{\\mathrm\{ref\}\}the frozen reference \(teacher\) of the stability term\.ℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}is the task loss of Eq\. \([1](https://arxiv.org/html/2607.15587#S3.E1)\),ℒKD\\mathcal\{L\}\_\{\\mathrm\{KD\}\}the stability regulariser \(§[5\.3](https://arxiv.org/html/2607.15587#S5.SS3)\),∇B\\nabla\_\{B\}the gradient w\.r\.t\. the adapter’s zero\-initialised low\-rank factorBB\(Eq\.[4](https://arxiv.org/html/2607.15587#S4.E4)\), andfθf\_\{\\theta\}the model’s token\-level predictive distribution\. Hyperparameters: replay ratioα\\alpha, stability weightλ\\lambda, routing temperatureτ\\tau, probe batchesmm, learning rateη\\eta\.

Algorithm 1General TSR1:task stream

T1,…,TKT\_\{1\},\\dots,T\_\{K\}with training data

D1,…,DKD\_\{1\},\\dots,D\_\{K\}; initial parameters

θ0\\theta\_\{0\}; modulesProbe,Route,Reference; hyperparameters

α\\alpha,

λ\\lambda,

η\\eta
2:

𝒫←\{\}\\mathcal\{P\}\\leftarrow\\\{\\\};

θ←θ0\\theta\\leftarrow\\theta\_\{0\}
3:for

t=1,…,Kt=1,\\dots,Kdo

4:gt←Probe​\(Dt\)g\_\{t\}\\leftarrow\\textsc\{Probe\}\(D\_\{t\}\)

5:w←Route​\(gt,\{gj\}j<t\)w\\leftarrow\\textsc\{Route\}\\big\(g\_\{t\},\\ \\\{g\_\{j\}\\\}\_\{j<t\}\\big\)

6:foreach training stepdo

7:with prob\.

1−α1\{\-\}\\alpha:

b∼Dtb\\sim D\_\{t\}; otherwise:

j∼wj\\sim w,

b∼ℳjb\\sim\\mathcal\{M\}\_\{j\}
8:θref←Reference​\(b,𝒫\)\\theta^\{\\mathrm\{ref\}\}\\leftarrow\\textsc\{Reference\}\(b,\\ \\mathcal\{P\}\)

9:

θ←θ−η​∇θ\[ℒCE​\(θ;b\)\+λ​ℒKD​\(θ;θref,b\)\]\\theta\\leftarrow\\theta\-\\eta\\,\\nabla\_\{\\theta\}\\big\[\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(\\theta;b\)\+\\lambda\\,\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\(\\theta;\\,\\theta^\{\\mathrm\{ref\}\},b\)\\,\\big\]
10:endfor

11:

ℳt←\\mathcal\{M\}\_\{t\}\\leftarrowsubsample of

DtD\_\{t\};

θt←θ\\theta\_\{t\}\\leftarrow\\theta;

𝒫←𝒫∪\{\(gt,ℳt,θt\)\}\\mathcal\{P\}\\leftarrow\\mathcal\{P\}\\cup\\\{\(g\_\{t\},\\,\\mathcal\{M\}\_\{t\},\\,\\theta\_\{t\}\)\\\}
12:endfor

The organising object is the*task memory*𝒫=\{\(gj,ℳj,θj\)\}j<t\\mathcal\{P\}=\\\{\(g\_\{j\},\\,\\mathcal\{M\}\_\{j\},\\,\\theta\_\{j\}\)\\\}\_\{j<t\}: one record per past task, written once when the task ends and never updated\. Its three fields answer an incoming task’s three needs — the signature says*whether*jjshould be drawn on, the data subsample is*what*gets replayed, and the era snapshot is*who*supervises that replay\. Because the fields are bound in one record, a single routing decision returns coherent \(data, teacher\) pairs: replayed knowledge always arrives with the reference model that knew it best\.

### 5\.2The evaluated instantiation

Algorithm[2](https://arxiv.org/html/2607.15587#alg2)— the*routed triple*— fixes the three modules; the evaluated hyperparameters are listed with the experimental setup \(§[6\.1](https://arxiv.org/html/2607.15587#S6.SS1)\)\. Figure[4](https://arxiv.org/html/2607.15587#S5.F4)traces one task through the resulting pipeline; each choice carries its rationale from §[4](https://arxiv.org/html/2607.15587#S4)\.

Algorithm 2Routed\-triple TSR1:task stream

T1,…,TKT\_\{1\},\\dots,T\_\{K\}with training data

D1,…,DKD\_\{1\},\\dots,D\_\{K\}; shared initialisation

θ0\\theta\_\{0\}; hyperparameters

mm,

τ\\tau,

α\\alpha,

λ\\lambda,

η\\eta
2:

𝒫←\{\}\\mathcal\{P\}\\leftarrow\\\{\\\};

θ←θ0\\theta\\leftarrow\\theta\_\{0\}
3:for

t=1,…,Kt=1,\\dots,Kdo

4:gt←1m​∑i=1m∇BℒCE​\(θ0;bi\)g\_\{t\}\\leftarrow\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\nabla\_\{B\}\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(\\theta\_\{0\};b\_\{i\}\),bi∼Dtb\_\{i\}\\sim D\_\{t\}⊳\\trianglerightProbe

5:wj←softmaxj<t​\(cos⁡\(gj,gt\)/τ\)w\_\{j\}\\leftarrow\\mathrm\{softmax\}\_\{j<t\}\\\!\\big\(\\cos\(g\_\{j\},\\,g\_\{t\}\)/\\tau\\big\)⊳\\trianglerightRoute

6:

j∗←arg⁡maxj<t⁡wjj^\{\\ast\}\\leftarrow\\arg\\max\_\{j<t\}w\_\{j\}
7:foreach training stepdo

8:draw

u∼Uniform​\(0,1\)u\\sim\\mathrm\{Uniform\}\(0,1\)
9:if

u\>αu\>\\alphathen

10:sample minibatch

b∼Dtb\\sim D\_\{t\};θref←θj∗\\theta^\{\\mathrm\{ref\}\}\\leftarrow\\theta\_\{j^\{\\ast\}\}⊳\\trianglerightcurrent\-task batch

11:else

12:draw one source

j∼Categorical​\(w\)j\\sim\\mathrm\{Categorical\}\(w\)⊳\\trianglerightpast taskjjchosen with prob\.wjw\_\{j\}

13:sample minibatch

bbuniformly from

ℳj\\mathcal\{M\}\_\{j\}
14:θref←θj\\theta^\{\\mathrm\{ref\}\}\\leftarrow\\theta\_\{j\}⊳\\trianglerightReference

15:endif

16:

θ←θ−η​∇θ\[ℒCE​\(θ;b\)\+λ​KL​\(fθref​\(b\)∥fθ​\(b\)\)\]\\theta\\leftarrow\\theta\-\\eta\\,\\nabla\_\{\\theta\}\\big\[\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(\\theta;b\)\+\\lambda\\,\\mathrm\{KL\}\\big\(f\_\{\\theta^\{\\mathrm\{ref\}\}\}\(b\)\\,\\\|\\,f\_\{\\theta\}\(b\)\\big\)\\,\\big\]
17:endfor

18:

ℳt←\\mathcal\{M\}\_\{t\}\\leftarrowsubsample of

DtD\_\{t\};

θt←θ\\theta\_\{t\}\\leftarrow\\theta;

𝒫←𝒫∪\{\(gt,ℳt,θt\)\}\\mathcal\{P\}\\leftarrow\\mathcal\{P\}\\cup\\\{\(g\_\{t\},\\,\\mathcal\{M\}\_\{t\},\\,\\theta\_\{t\}\)\\\}
19:endfor

*Probe*— the mean probe\-batch gradient at the fixed shared initialisation \(Eq\.[4](https://arxiv.org/html/2607.15587#S4.E4); Figure[4](https://arxiv.org/html/2607.15587#S5.F4)a\): the only tested signature that reliably finds oracle\-best sources, label\-aware where input\-similarity signatures are not \(§[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)\); ten forward–backward passes, no training\.

*Route*— softmax over signature cosines at temperatureτ\\tau\(Figure[4](https://arxiv.org/html/2607.15587#S5.F4)b\)\. The temperature makes the routing knob explicit:τ→0\\tau\{\\to\}0recovers argmax selection,τ→∞\\tau\{\\to\}\\inftyrecovers uniform pooling \(==ER\), and the evaluated setting sits between the two extremes compared in Appendix[B\.3](https://arxiv.org/html/2607.15587#A2.SS3)\.

*Replay*— routed source data mixed into training at ratioα\\alpha, batch by batch \(Figure[4](https://arxiv.org/html/2607.15587#S5.F4)c\): data is the persistent carrier of §[4\.2](https://arxiv.org/html/2607.15587#S4.SS2), so the transferred signal re\-enters at every optimisation step rather than only at initialisation\.

*Reference*— each batch is distilled against the routed record’s era snapshot:θj\\theta\_\{j\}on batches replayed fromℳj\\mathcal\{M\}\_\{j\}, the top\-routedθj∗\\theta\_\{j^\{\\ast\}\}on current\-task batches \(Figure[4](https://arxiv.org/html/2607.15587#S5.F4)c\)\. Why distillation owns stability — and why the teacher is an era snapshot — is the subject of §[5\.3](https://arxiv.org/html/2607.15587#S5.SS3)\.

Other instantiations of the three modules are configurations of the same skeleton \(Appendix[B\.3](https://arxiv.org/html/2607.15587#A2.SS3)\)\.

T1T\_\{1\}T2T\_\{2\}⋯\\cdotsTtT\_\{t\}Task stream→\\rightarrowA few probe batchesat the shared initialisationθ0\\theta\_\{0\}Task signaturegtg\_\{t\}Task memory𝒫\\mathcal\{P\}: one record per past tasksignaturegjg\_\{j\}• replay dataℳj\\mathcal\{M\}\_\{j\}• snapshotθj\\theta\_\{j\}Match signaturescos⁡\(gj,gt\)→w\\cos\(g\_\{j\},g\_\{t\}\)\\rightarrow wpicks data*and*teacherReplayb∼ℳjb\\sim\\mathcal\{M\}\_\{j\}: routed source examplesmixed into every training batchTeacherθref=θj\\theta^\{\\mathrm\{ref\}\}=\\theta\_\{j\}:the routed record’s snapshotTrain:ℒCE\+λ​ℒKD\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\lambda\\,\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\(a\) Probe\(b\) Route\(c\) TrainNew taskreplay dataℳj\\mathcal\{M\}\_\{j\}teacherθj\\theta\_\{j\}Figure 4:The Transfer\-Selective Replay pipeline\.
### 5\.3Knowledge distillation for stability

Once replay is reassigned to transfer, the stability term of the continual objective is unowned; TSR assigns it to distillation\. Each training batchbbis anchored by

ℒKD​\(θ;b\)=KL​\(fθref​\(b\)∥fθ​\(b\)\),\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\(\\theta;\\,b\)\\;=\\;\\mathrm\{KL\}\\big\(f\_\{\\theta^\{\\mathrm\{ref\}\}\}\(b\)\\,\\big\\\|\\,f\_\{\\theta\}\(b\)\\big\),\(5\)added to the task loss with weightλ\\lambda\(Algorithm[2](https://arxiv.org/html/2607.15587#alg2)\): the KL is taken between the token\-level predictive distributions of the frozen reference and the student on the same batch, and the reference switches with the batch’s origin —θj\\theta\_\{j\}on batches replayed fromℳj\\mathcal\{M\}\_\{j\}, the top\-routedθj∗\\theta\_\{j^\{\\ast\}\}on current\-task batches\.

The novelty is not the loss\. It lies, first, in the*role assignment*: replay serves transfer, while distillation alone owns stability\. It lies, second, in the*binding*: teachers are queryable era snapshots, routed jointly with the replay data through the task memory, so one routing decision returns both what to replay and whom to distil against, and the reference can answer on any batch, including current\-task ones\. To our knowledge, distillation as an explicitly routed stability module has no counterpart in recent continual learning with LLMs\.

##### What the teacher contributes\.

A systematic teacher audit dissociates teacher*knowledge*from optimisation*anchoring*: teacher identity is second\-order, the teacher’s value lives in its base\-shared component, and every attempt to make distillation carry transfer reduces both transfer*and*stability\. In this regime distillation functions as a stability regulariser — adjacent to reference\-KL in RL\(Tehet al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib68)\)and to label\-smoothing accounts of supervised distillation\(Yuanet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib69); Mobahiet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib70)\), though the dissociation in continual LLM learning is, to our knowledge, new — and we assign it exclusively to stability\. The one teacher property that consistently matters in long streams is*drift*: the most recent snapshot is increasingly biased toward the latest task, which is what motivates the era\-matched reference\. The full audit is reported in Appendix[B\.4](https://arxiv.org/html/2607.15587#A2.SS4)\.

## 6Experiments

Sections[4](https://arxiv.org/html/2607.15587#S4)–[5](https://arxiv.org/html/2607.15587#S5)measured*why*transfer should work and derived TSR as one instantiation\. This section validates the framework under the standard continual\-learning protocol, in three questions: does transfer improve low\-budget continual learning \(§[6\.2](https://arxiv.org/html/2607.15587#S6.SS2)\); what determines the size and shape of the gains \(§[6\.3](https://arxiv.org/html/2607.15587#S6.SS3)\); and how much do the design choices matter \(§[6\.4](https://arxiv.org/html/2607.15587#S6.SS4)\)\.

### 6\.1Setup

Models and streams\.Qwen2\.5\-0\.5B/7B\-Instruct with a single LoRA adapter\(Huet al\.,[2021](https://arxiv.org/html/2607.15587#bib.bib1)\)trained continually — the dominant regime for continual learning with LLMs\. Two eight\-task streams: TRACE\-8\(Wanget al\.,[2023b](https://arxiv.org/html/2607.15587#bib.bib25)\)\(heterogeneous: stance, finance, summarisation, code, science QA, arithmetic\) in its canonical order, and NumGLUE\-8\(Mishraet al\.,[2022](https://arxiv.org/html/2607.15587#bib.bib65)\)\(one numerical\-reasoning core in eight formats\) in type order\.

Protocol\.We follow the standard sequential protocol of the systems we compare against\(Wanget al\.,[2023a](https://arxiv.org/html/2607.15587#bib.bib3); Qianet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib36); Buzzegaet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib13)\): one accumulating adapter trains through all eight tasks; after each task, all seen tasks are evaluated;n=5n\{=\}5seeds per cell\. We evaluate in the low\-budget regime —N=50N\{=\}50examples per task — where transfer is operationally interesting \(§[4\.1](https://arxiv.org/html/2607.15587#S4.SS1)\); the protocol structure is the literature’s, the budget is the studied setting\. All remaining configuration \(adapter, optimiser, training budgets, and TSR’s module settings\) is detailed in Appendix[C\.1](https://arxiv.org/html/2607.15587#A3.SS1), together with cost accounting and stream construction; order\-robustness checks are in Appendix[C\.3](https://arxiv.org/html/2607.15587#A3.SS3)\.

Metrics\.Training uses token\-level cross\-entropy\. Evaluation reports exact\-match accuracy on classification tasks and evaluation loss on generation tasks, where exact match is degenerate\. LetAk,iA\_\{k,i\}denote accuracy onTiT\_\{i\}’s test set after training on taskTkT\_\{k\}\. We report the three standard sequential metrics\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2607.15587#bib.bib54); Chaudhryet al\.,[2019a](https://arxiv.org/html/2607.15587#bib.bib33)\):

Overall=1K​∑i=1KAK,i,\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}A\_\{K,i\},\(6\)Plas=1K​∑i=1KAi,i,\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}A\_\{i,i\},\(7\)BWT=1K−1​∑i=1K−1\(AK,i−Ai,i\)\.\\displaystyle=\\frac\{1\}\{K\-1\}\\sum\_\{i=1\}^\{K\-1\}\\left\(A\_\{K,i\}\-A\_\{i,i\}\\right\)\.\(8\)Overall is the final average accuracy; plasticity \(Plas\) is accuracy immediately after learning each task; and BWT is backward transfer, where0indicates perfect retention, positive values indicate improvement on earlier tasks, and negative values indicate forgetting\.

Baselines\.Naive sequential fine\-tuning \(SeqFT\)\.*Rehearsal\-free*: regularisation and distillation \(EWCKirkpatricket al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib32); LwFLi and Hoiem,[2017](https://arxiv.org/html/2607.15587#bib.bib17); SDFTYanget al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib66)\); parameter isolation \(O\-LoRAWanget al\.,[2023a](https://arxiv.org/html/2607.15587#bib.bib3)\); and two NeurIPS’25 methods — GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib7)\), gated per\-task LoRA branches with projection\-constrained gates, and DEAL\(Hanet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib67)\), wavelet\-filtered retention on a continually edited adapter \(port details in Appendix[C\.1](https://arxiv.org/html/2607.15587#A3.SS1)\)\.*Rehearsal\-based*: gradient constraints from memory samples \(A\-GEMChaudhryet al\.,[2019a](https://arxiv.org/html/2607.15587#bib.bib33)\); DER\+\+Buzzegaet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib13); and ERChaudhryet al\.,[2019b](https://arxiv.org/html/2607.15587#bib.bib12)— uniform replay over all past tasks — at a generous50%50\\%replay ratio, with a10%10\\%variant marking the budget spectrum\. Zero\-shot exact match is0under the strict answer format and is therefore omitted from Table[3](https://arxiv.org/html/2607.15587#S6.T3)\.

### 6\.2Does transfer improve low\-budget continual learning?

We now test whether the transfer contribution of §[3](https://arxiv.org/html/2607.15587#S3)improves continual learning under a fixed target\-data budget \(Table[3](https://arxiv.org/html/2607.15587#S6.T3)\)\.

Plasticity\.In the decomposition of §[3](https://arxiv.org/html/2607.15587#S3), plasticity is where the transfer contribution should appear first, so we read Plas first\. On TRACE, TSR posts the best plasticity at both scales \(\.439\.439at 0\.5B,\.670\.670at 7B\): signature\-weighted replay beats ER on the transfer axis, the controlled probe’s finding carried into the stream\. On NumGLUE, methods that only chase the current task can edge Plas \(SDFT\.334\.334at 0\.5B, SeqFT\.587\.587at 7B\) — but they pay with the worst forgetting in their columns; plasticity bought by abandoning the past is not transfer\.

Overall accuracy\.Overall folds retention in, and TSR is best everywhere\. The reading we care about is the budget one: with the same5050target examples per task, selected replay reaches final performance that ER does not — a gain that would otherwise require additional target supervision, and that Figure[2](https://arxiv.org/html/2607.15587#S4.F2)\(b\) converts into equivalent target labels on the probe \(§[4\.1](https://arxiv.org/html/2607.15587#S4.SS1)\)\. Stream\-level margins are smaller than probe\-level ones for a predicted reason: the stream itself gradually consumes pair\-specific headroom — by the time a target arrives, its best source has often already been trained into the adapter\.

Stability\.The plasticity gains do not come at the expense of forgetting: TSR is the only method in Table[3](https://arxiv.org/html/2607.15587#S6.T3)with*positive*BWT in all four settings\. Replay incidentally protects the sources it draws on, and distillation anchors every batch to an undrifted era reference\.Under a fixed target budget, realised transfer buys higher plasticity and higher final accuracy at no retention cost\.

Backbone robustness\.The same qualitative behaviour is observed on two additional backbone families \(Llama\-3\.2\-1B and Gemma\-3\-1B; Appendix[C\.3](https://arxiv.org/html/2607.15587#A3.SS3)\): replay\-based methods remain strongest, and TSR again attains the best overall accuracy, plasticity, and backward transfer — the framework’s predictions are not specific to Qwen models\.

Table 3:Continual learning under the standard sequential protocol\(TRACE\-8 / NumGLUE\-8 at 0\.5B / 7B\)\. Mean±\\pmstd over55seeds; bold = best per column\. Overall and Plas are*absolute*accuracies; BWT is the change from post\-task to end\-of\-stream accuracy; no row is normalised by the zero\-shot reference \(omitted: exact match is0under the strict answer format\)\. Rules separate the no\-mechanism baseline, the rehearsal\-free family, the rehearsal\-based family, and ours\. Overall, Plas and BWT as in Eqs\. \([6](https://arxiv.org/html/2607.15587#S6.E6)\)–\([8](https://arxiv.org/html/2607.15587#S6.E8)\); baselines and protocol in §[6\.1](https://arxiv.org/html/2607.15587#S6.SS1); routing\-policy variants in Appendix[B\.3](https://arxiv.org/html/2607.15587#A2.SS3); per\-task results in Appendix[C\.2](https://arxiv.org/html/2607.15587#A3.SS2)\.
### 6\.3What determines transfer?

The framework predicts that realised transfer depends jointly on transfer opportunity and task structure — and indeed the same method, protocol, and budget produce gains of very different size and shape across the four settings of Table[3](https://arxiv.org/html/2607.15587#S6.T3)\. Our evaluation scope is deliberatelytwo\-dimensional — target\-data budget and source\-task structure— the same axes along which the reuse of previously collected data is evaluated in online RL\(Ballet al\.,[2023](https://arxiv.org/html/2607.15587#bib.bib73)\)\. The differences follow the framework, not the method: two observations and two boundaries\.

##### Transfer follows task structure\.

TRACE is heterogeneous: sources differ sharply and several hurt \(Table[1](https://arxiv.org/html/2607.15587#S4.T1)\), so*selection*is what pays — routing beats pooling on plasticity, and argmax selection is strongest on the pure\-transfer probe \(Appendix[A\.2](https://arxiv.org/html/2607.15587#A1.SS2)\)\. NumGLUE is the opposite regime — one arithmetic core in eight formats: headroom is large and dense \(Appendix[A\.4\.1](https://arxiv.org/html/2607.15587#A1.SS4.SSS1)\), every source carries the core, and within such homogeneous families the framework predicts*pooling*rather than single\-source replay, which the controlled probe confirms\. The signature’s resolution also drops inside a family \(its cosines compress; Appendix[A\.4\.2](https://arxiv.org/html/2607.15587#A1.SS4.SSS2)\) — a bound on the current instantiation, not on the framework, since the selection module is the replaceable one \(§[5\.1](https://arxiv.org/html/2607.15587#S5.SS1)\)\. The routed triple stays best in the stream in both regimes because its soft routing spans them: sharp enough to select on TRACE, broad enough to keep coverage on NumGLUE\.

##### Transfer follows headroom\.

Figure[5](https://arxiv.org/html/2607.15587#S6.F5)repeats the headroom audit across Qwen2\.5 scales on TRACE: as the base grows, the target\-data arm saturates on its own and the best\-source gain decays monotonically to zero; at 7B the oracle buys nothing\. This is a probe\-level statement about opportunity, not a method comparison, and it agrees with Table[3](https://arxiv.org/html/2607.15587#S6.T3): every TRACE Plas column flattens at 7B — no policy can transfer where no opportunity remains — while Overall still separates through retention\.

0\.5B1\.5B3B7B0\.40\.40\.50\.50\.60\.60\.70\.7Base\-model scaleTarget\-data accuracy00\.010\.01Best\-source gainFigure 5:Transfer follows headroom across model scale\(TRACE headroom audit per Qwen2\.5 scale; error bands omitted for clarity\)\.NumGLUE shows the graded version of the same law: its in\-domain headroom persists at 7B, but the fraction other tasks can supply compresses \(\+0\.033→\+0\.003\+0\.033\\to\+0\.003\) — scale erodes what tasks share before it erodes what each still individually lacks\. Notably, signature picks are scale\-stable \(7B agrees with 0\.5B on7/87/8targets\), so signatures can be computed once on a small proxy model\.

##### Boundaries\.

The framework predicts its own two edges, and both are observed:*saturation*— on benchmarks without headroom, transfer is unmeasurable by any mechanism; and*generic transfer*— the regime where most sources provide similar benefit, so source identity matters little: on six Super\-NaturalInstructions pairs headroom exists and source data helps, but sources are near\-interchangeable and the value of selection is small \(Appendix[A\.3](https://arxiv.org/html/2607.15587#A1.SS3)\)\.Thus, task structure determines how transfer should be realised, whereas headroom determines whether it can be realised at all\.

### 6\.4Design choices

The framework specifies*what*must be decided, not*how*: Algorithm[2](https://arxiv.org/html/2607.15587#alg2)leaves implementation choices open, and we ablate these — how replay is routed, and whether task order still matters once routing is available\.

##### Routing\.

The routing module decides whether replay comes from a single selected source or a weighted combination; the framework is agnostic, so we compare argmax selection, soft routing, and uniform pooling\. In the stream, soft routing — the routed triple — beats both extremes and its single\-component variants \(Table[3](https://arxiv.org/html/2607.15587#S6.T3); Appendix[B\.3](https://arxiv.org/html/2607.15587#A2.SS3)\); on the controlled probe, even a softmax\-weighted top\-3 mixture*lowers*mean accuracy against the single selected source\. The reason is the shape of the matrix: on heterogeneous streams it is sparse — the second\- and third\-best sources are often weak or harmful, and averaging them dilutes transfer\.Sparse transfer matrices favour selection over averaging\.

##### Task order\.

If the matrix is usable prospectively, ordering tasks so strong sources precede their targets should help\. We compare the canonical order, the greedy best\-source curriculum \(fixed before evaluation\), its reversal, and arbitrary reshuffles, at both scales \(Appendix[C\.3](https://arxiv.org/html/2607.15587#A3.SS3)\)\. The replay family moves by at most\.03\.03under any of them, on both streams and both scales: replay already delivers the relevant source at training time, so a curriculum has little left to contribute\.Selection replaces curriculum\.

## 7Discussion

### 7\.1Implications

This work does not propose another replay algorithm; it changes the questions asked before one is designed\. Three reversals follow\.*Replay*: long treated as a forgetting\-mitigation tool, replay is the cheapest persistent carrier of transfer — pointing it at the new task turns a stability mechanism into a plasticity mechanism, and distillation absorbs the stability role it leaves behind\.*Transfer*: what looked like a parameter\-reuse problem is a data\-selection problem — the transferred signal must survive optimisation, and among persistent carriers data is the cheapest — so the productive question is not how to merge parameters but which examples to select\.*Benchmarks*: a continual\-learning benchmark measures more than accuracy — its headroom and transfer structure decide whether transfer is measurable on it at all, and both belong in its specification, alongside whether the task order is canonical, arbitrary, or optimised; selection makes replay\-based learners order\-insensitive while order dominates learners without data access \(§[6\.4](https://arxiv.org/html/2607.15587#S6.SS4)\), so curricula and selection are substitute uses of the same measured matrix\.

### 7\.2Limitations and open questions

The framework was validated in the low\-budget regime, where transfer opportunity is measurable; whether it yields equally useful mechanisms under richer supervision is an empirical question\. Within this regime, each limitation pairs with the question it opens\.

*Storage\.*Replay assumes access to past task data\. The buffer is small \(∼100\{\\sim\}100examples per task,∼3\{\\sim\}3MB for1515tasks, favourable versus per\-task adapter libraries\), but privacy\-constrained deployments cannot store raw examples — opening the question of whether transfer can be*carried differently*: synthetic or distilled replay and retrieval are candidates, judged by the same persistence criterion \(§[4\.2](https://arxiv.org/html/2607.15587#S4.SS2)\)\.

*The current instantiation\.*The implementation instantiates the framework with one task signature and one replay policy; neither is claimed optimal\. Signature resolution drops within homogeneous families \(§[6\.3](https://arxiv.org/html/2607.15587#S6.SS3)\) — opening the question of whether transfer can be*predicted better*: task signatures that retain within\-family resolution, and signatures for modalities beyond text\.

*Evaluation\.*Existing benchmarks are designed to expose forgetting, not transfer — headroom had to be audited here before any mechanism could be tested — opening the question of whether transfer can be*benchmarked directly*: streams with reported headroom, a published pairwise matrix, and known negative cells, so that transfer is evaluated as directly as forgetting is today\.

Beyond language, the conditions are stated at the level of tasks and data: vision and multimodal streams, and budget\-allocation problems outside supervised language modelling — e\.g\., allocating simulation budget for scientific surrogates\(Menget al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib71)\)— pose the same fixed\-budget transfer question\.

### 7\.3Conclusion

*Transfer in continual learning is a data\-selection problem rather than a parameter\-reuse problem\.*The two carrier studies point at one law: continual optimisation preserves persistent signals and suppresses transient, task\-specific ones\. TSR is one instantiation; the framework is intended to outlive it\. The particular signature, replay policy, and stability mechanism are all replaceable — what we hope remains is the discipline: before proposing a transfer mechanism, first ask whether transfer is possible, what can carry it, and whom it should come from\. We hope continual learning comes to evaluate transfer as systematically as it has evaluated forgetting\.

## 8acknowledgments

We gratefully acknowledge the support of NSF IIS\-2313131, NSF IIS\-2332475 and NSF IIS\-2543755, the NSF\-Simons AI\-Institute for the Sky \(SkAI\) via grants NSF AST\-2421845 and Simons Foundation MPS\-AI00010513, and the University of Chicago’s Research Computing Center for their support of this work\.

## References

- A\. Achille, M\. Lam, R\. Tewari, A\. Ravichandran, S\. Maji, C\. C\. Fowlkes, S\. Soatto, and P\. Perona \(2019\)Task2vec: task embedding for meta\-learning\.InICCV,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1)\.
- When AI benchmarks plateau: a systematic study of benchmark saturation\.arXiv preprint arXiv:2602\.16763\.Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2)\.
- R\. Aljundi, E\. Belilovsky, T\. Tuytelaars, L\. Charlin, M\. Caccia, M\. Lin, and L\. Page\-Caccia \(2019a\)Online continual learning with maximal interfered retrieval\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Aljundi, M\. Lin, B\. Goujaud, and Y\. Bengio \(2019b\)Gradient based sample selection for online continual learning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Alvarez\-Melis and N\. Fusi \(2020\)Geometric dataset distances via optimal transport\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1),[Table 2](https://arxiv.org/html/2607.15587#S4.T2)\.
- P\. Asawa, C\. M\. Glaze, G\. Orlanski,et al\.\(2026\)Continual learning bench: evaluating frontier AI systems in real\-world stateful environments\.arXiv preprint arXiv:2606\.05661\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1)\.
- P\. J\. Ball, L\. Smith, I\. Kostrikov, and S\. Levine \(2023\)Efficient online reinforcement learning with offline data\.InInternational Conference on Machine Learning,Cited by:[§6\.3](https://arxiv.org/html/2607.15587#S6.SS3.p1.1)\.
- P\. Buzzega, M\. Boschini, A\. Porrello, D\. Abati, and S\. Calderara \(2020\)Dark experience for general continual learning: a strong, simple baseline\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.4\.7](https://arxiv.org/html/2607.15587#A2.SS4.SSS7.Px1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p2.2),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- A\. Chaudhry, M\. Ranzato, M\. Rohrbach, and M\. Elhoseiny \(2019a\)Efficient lifelong learning with A\-GEM\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p3.3),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- A\. Chaudhry, M\. Rohrbach, M\. Elhoseiny, T\. Ajanthan, P\. K\. Dokania, P\. H\. Torr, and M\. Ranzato \(2019b\)On tiny episodic memories in continual learning\.arXiv preprint arXiv:1902\.10486\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- Y\. Chen, T\. Zhu, M\. Zhang, X\. Chen, J\. Huang, D\. Xu, and Y\. Wang \(2026\)On\-policy replay for continual supervised fine\-tuning\.arXiv preprint arXiv:2605\.29495\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px5.p1.1)\.
- M\. Choi and G\. Kim \(2026\)Decentralized instruction tuning: conflict\-aware splitting and weight merging\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Chronopoulou, M\. E\. Peters, A\. Fraser, and J\. Dodge \(2023\)Adaptersoup: weight averaging to improve generalization of pretrained language models\.InFindings of EACL,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- M\. De Lange, R\. Aljundi, M\. Masana, S\. Parisot, X\. Jia, A\. Leonardis, G\. Slabaugh, and T\. Tuytelaars \(2021\)A continual learning survey: defying forgetting in classification tasks\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.15587#S3.SS1.p1.6)\.
- C\. Fifty, E\. Amid, Z\. Zhao, T\. Yu, R\. Anil, and C\. Finn \(2021\)Efficiently identifying task groupings for multi\-task learning\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1)\.
- Gemma Team \(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[§C\.3](https://arxiv.org/html/2607.15587#A3.SS3.SSS0.Px2.p1.7),[§C\.4](https://arxiv.org/html/2607.15587#A3.SS4.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p7.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§C\.3](https://arxiv.org/html/2607.15587#A3.SS3.SSS0.Px2.p1.7),[§C\.4](https://arxiv.org/html/2607.15587#A3.SS4.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p7.1)\.
- X\. Han, Z\. Zhao, W\. Wang, M\. Wang, Z\. Liu, Y\. Chang, and X\. Zhao \(2025\)Data efficient adaptation in large language models via continuous low\-rank fine\-tuning\.InAdvances in Neural Information Processing Systems,Cited by:[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- D\. Hernandez, J\. Kaplan, T\. Henighan, and S\. McCandlish \(2021\)Scaling laws for transfer\.arXiv preprint arXiv:2102\.01293\.Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px3.p1.8)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p1.1)\.
- C\. Huang, Q\. Liu, B\. Y\. Lin, T\. Pang, C\. Du, and M\. Lin \(2023\)Lorahub: efficient cross\-task generalization via dynamic LoRA composition\.arXiv preprint arXiv:2307\.13269\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2607.15587#S4.SS2.p2.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2023\)Editing models with task arithmetic\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Javed and M\. White \(2019\)Meta\-learning representations for continual learning\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ke, B\. Liu, and X\. Huang \(2020\)Continual learning of a mixed sequence of similar and dissimilar tasks\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ke, B\. Liu, N\. Ma, H\. Xu, and L\. Shu \(2021a\)Achieving forgetting prevention and knowledge transfer in continual learning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ke and B\. Liu \(2022\)Continual learning of natural language processing tasks: a survey\.arXiv preprint arXiv:2211\.12701\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Ke, H\. Xu, and B\. Liu \(2021b\)Adapting BERT for continual learning of a sequence of aspect sentiment classification tasks\.InNAACL,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.InProceedings of the National Academy of Sciences,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- F\. Kong, R\. Zhang, Z\. Nie,et al\.\(2024\)MOMA: masked orthogonal matrix alignment for zero\-additional\-parameter model merging\.arXiv preprint arXiv:2412\.13526\.Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2)\.
- S\. Krishna, A\. Naik, C\. Agarwal, S\. Govindan, T\. Lee, and H\. Chang \(2025\)Latent traits and cross\-task transfer: deconstructing dataset interactions in LLM fine\-tuning\.arXiv preprint arXiv:2509\.13624\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1)\.
- J\. Li, Y\. Meng, D\. Simchi\-Levi, and C\. Wang \(2026\)Personalized incentive alignment: correcting utility\-driven selection bias in a/b tests\.InProceedings of the 29th International Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§C\.4](https://arxiv.org/html/2607.15587#A3.SS4.p1.1)\.
- Z\. Li and D\. Hoiem \(2017\)Learning without forgetting\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§B\.4\.7](https://arxiv.org/html/2607.15587#A2.SS4.SSS7.Px1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- Y\. Liang, J\. Chen, and W\. Li \(2025\)Gated integration of low\-rank adaptation for continual learning of large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2607.15587#S3.SS2.p1.2),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p3.3)\.
- Y\. Meng, R\. Jiang, Z\. Zhao, C\. Liu, R\. Willett, and Y\. Chen \(2026\)Accelerating PDE surrogates via RL\-guided mesh optimization\.InThe 29th International Conference on Artificial Intelligence and Statistics,Cited by:[§7\.2](https://arxiv.org/html/2607.15587#S7.SS2.p5.1)\.
- S\. Mishra, A\. Mitra, N\. Varshney, B\. Sachdeva, P\. Clark, C\. Baral, and A\. Kalyan \(2022\)NumGLUE: a suite of fundamental yet challenging mathematical reasoning tasks\.InProceedings of ACL,Cited by:[§A\.4\.1](https://arxiv.org/html/2607.15587#A1.SS4.SSS1.p1.8),[§C\.1](https://arxiv.org/html/2607.15587#A3.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p7.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p1.1)\.
- H\. Mobahi, M\. Farajtabar, and P\. Bartlett \(2020\)Self\-distillation amplifies regularization in Hilbert space\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2607.15587#S5.SS3.SSS0.Px1.p1.1)\.
- S\. Momeni and B\. Liu \(2025\)Achieving upper bound accuracy of joint training in continual learning\.arXiv preprint arXiv:2502\.12388\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1)\.
- S\. Momeni, S\. Mazumder, Z\. Ke, and B\. Liu \(2025\)In\-context continual learning assisted by an external continual learner\.InCOLING,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1)\.
- K\. Nguyen, H\. Nguyen, T\. Pham, and N\. Ho \(2025a\)Lightspeed geometric dataset distance via sliced optimal transport\.arXiv preprint arXiv:2501\.18901\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1),[Table 2](https://arxiv.org/html/2607.15587#S4.T2)\.
- T\. Nguyen, C\. N\. Nguyen, Q\. Pham, B\. T\. Nguyen, S\. Ramasamy, X\. Li, and C\. V\. Nguyen \(2025b\)Sequence transferability and task order selection in continual learning\.arXiv preprint arXiv:2502\.06544\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1)\.
- O\. Ostapenko, Z\. Su, E\. M\. Ponti, L\. Charlin, N\. Le Roux, M\. Pereira, L\. Caccia, and A\. Sordoni \(2024\)Towards modular LLMs by building and reusing a library of LoRAs\.arXiv preprint arXiv:2405\.11157\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Phang, T\. Févry, and S\. R\. Bowman \(2018\)Sentence encoders on STILTs: supplementary training on intermediate labeled\-data tasks\.arXiv preprint arXiv:1811\.01088\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Qian, Y\. Xu, Z\. Zhang, P\. Zhao, and Z\. Zhou \(2025\)TreeLoRA: efficient continual learning via layer\-wise LoRAs guided by a hierarchical gradient\-similarity tree\.InICML,Cited by:[§A\.2](https://arxiv.org/html/2607.15587#A1.SS2.SSS0.Px2.p1.5),[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p2.2)\.
- F\. Qiao and M\. Mahdavi \(2025\)Merge before forget: a single LoRA continual learning via continual merging\.arXiv preprint arXiv:2512\.23017\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Qin and S\. Joty \(2022\)LFPT5: a unified framework for lifelong few\-shot language learning based on prompt tuning of T5\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2)\.
- Qwen Team \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§C\.4](https://arxiv.org/html/2607.15587#A3.SS4.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p7.1)\.
- A\. Razdaibiedina, Y\. Mao, R\. Hou, M\. Khabsa, M\. Lewis, and A\. Almahairi \(2023\)Progressive prompts: continual learning for language models\.InInternational Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2)\.
- S\. Rebuffi, A\. Kolesnikov, G\. Sperl, and C\. H\. Lampert \(2017\)ICaRL: incremental classifier and representation learning\.InCVPR,Cited by:[§B\.4\.7](https://arxiv.org/html/2607.15587#A2.SS4.SSS7.Px1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Riemer, I\. Cases, R\. Ajemian, M\. Liu, I\. Rish, Y\. Tu, and G\. Tesauro \(2019\)Learning to learn without forgetting by maximizing transfer and minimizing interference\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1)\.
- A\. A\. Rusu, N\. C\. Rabinowitz, G\. Desjardins, H\. Soyer, J\. Kirkpatrick, K\. Kavukcuoglu, R\. Pascanu, and R\. Hadsell \(2016\)Progressive neural networks\.arXiv preprint arXiv:1606\.04671\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2607.15587#S4.SS2.p2.1)\.
- I\. F\. Shihab, F\. Afrin, and A\. Sharma \(2026\)Dynamic proxy\-mixing: transferring replay controllers from small to large models for continual instruction tuning\.arXiv preprint arXiv:2606\.00400\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1)\.
- J\. S\. Smithet al\.\(2024\)Adaptive memory replay for continual learning\.InCVPR Workshops,Note:arXiv:2404\.12526Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Sun, Y\. Yang, Y\. Li, J\. Li, and Y\. Gao \(2025\)Unveiling and addressing pseudo forgetting in large language models\.InFindings of ACL,Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2)\.
- P\. K\. Tankala, S\. Mishra, S\. Lokam, and N\. Goyal \(2025\)CurLL: a developmental framework to evaluate continual learning in language models\.InBabyLM Workshop,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1)\.
- M\. E\. Taylor and P\. Stone \(2009\)Transfer learning for reinforcement learning domains: a survey\.Journal of Machine Learning Research10,pp\. 1633–1685\.Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px3.p1.8)\.
- Y\. W\. Teh, V\. Bapst, W\. M\. Czarnecki, J\. Quan, J\. Kirkpatrick, R\. Hadsell, N\. Heess, and R\. Pascanu \(2017\)Distral: robust multitask reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2607.15587#S5.SS3.SSS0.Px1.p1.1)\.
- T\. Veniat, L\. Denoyer, and M\. Ranzato \(2021\)Efficient continual learning with modular networks and task\-driven priors\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p3.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Vu, T\. Wang, T\. Munkhdalai, A\. Sordoni, A\. Trischler, A\. Mattarella\-Micke, S\. Maji, and M\. Iyyer \(2020\)Exploring and predicting transferability across NLP tasks\.InEMNLP,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024\)A comprehensive survey of continual learning: theory, method and application\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px5.p1.1)\.
- X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. Huang \(2023a\)Orthogonal subspace learning for language model continual learning\.InFindings of EMNLP,Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p2.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p2.2),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- X\. Wang, Y\. Zhang, T\. Chen, S\. Gao, S\. Jin, X\. Yang, Z\. Xi, R\. Zheng, Y\. Zou, T\. Gui,et al\.\(2023b\)TRACE: a comprehensive benchmark for continual learning in large language models\.arXiv preprint arXiv:2310\.06762\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§1](https://arxiv.org/html/2607.15587#S1.p7.1),[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p1.1)\.
- Y\. Wanget al\.\(2024\)InsCL: a data\-efficient continual learning paradigm for fine\-tuning large language models with instructions\.InNAACL,Note:arXiv:2403\.11435Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Xia, S\. Malladi, S\. Gururangan, S\. Arora, and D\. Chen \(2024\)LESS: selecting influential data for targeted instruction tuning\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px3.p2.1)\.
- S\. Yang, K\. Ning, Y\. Liu,et al\.\(2025\)Is parameter collision hindering continual learning in LLMs?\.InCOLING,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Yang, T\. Pang, H\. Feng, H\. Wang, W\. Chen, M\. Zhu, and Q\. Liu \(2024\)Self\-distillation bridges distribution gap in language model fine\-tuning\.InACL,Cited by:[§B\.4\.7](https://arxiv.org/html/2607.15587#A2.SS4.SSS7.Px1.p1.1),[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2607.15587#S6.SS1.p4.3)\.
- L\. Yuan, F\. E\. Tay, G\. Li, T\. Wang, and J\. Feng \(2020\)Revisiting knowledge distillation via label smoothing regularization\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2607.15587#S5.SS3.SSS0.Px1.p1.1)\.
- A\. R\. Zamir, A\. Sax, W\. Shen, L\. J\. Guibas, J\. Malik, and S\. Savarese \(2018\)Taskonomy: disentangling task transfer learning\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px3.p1.8)\.
- X\. Zhang, L\. Bai, X\. Yang, and J\. Liang \(2025\)C\-LoRA: continual low\-rank adaptation for pre\-trained models\.arXiv preprint arXiv:2502\.17920\.Cited by:[§2](https://arxiv.org/html/2607.15587#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhao, Y\. Jiang, and Y\. Chen \(2024\)Direct acquisition optimization for low\-budget active learning\.arXiv preprint arXiv:2402\.06045\.Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p4.1)\.
- J\. Zheng, X\. Cai, S\. Qiu, and Q\. Ma \(2025\)Spurious forgetting in continual learning of language models\.InInternational Conference on Learning Representations,Note:arXiv:2501\.13453Cited by:[§1](https://arxiv.org/html/2607.15587#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px4.p1.2)\.

## Appendix

Appendix guide\.Appendix[A](https://arxiv.org/html/2607.15587#A1)collects the benchmark\-level evidence for the framework of §[4](https://arxiv.org/html/2607.15587#S4): the four regimes — PP15 \(no headroom\), TRACE \(controlled transfer measurement\), SuperNI \(generic transfer\), and NumGLUE \(homogeneous transfer\)\. Appendix[B](https://arxiv.org/html/2607.15587#A2)collects the implementation evidence behind §[5](https://arxiv.org/html/2607.15587#S5): carrier, replay, routing, and distillation\. Appendix[C](https://arxiv.org/html/2607.15587#A3)is the experimental record behind §[6](https://arxiv.org/html/2607.15587#S6): setup, per\-task results, robustness, and reproducibility\. Unless stated otherwise, appendix results follow the probe protocol of §[4\.1](https://arxiv.org/html/2607.15587#S4.SS1)and the stream protocol of §[6\.1](https://arxiv.org/html/2607.15587#S6.SS1)\.

## Appendix ABenchmark Regimes

### A\.1PP15: the zero\-headroom regime

Under the audit of §[4\.1](https://arxiv.org/html/2607.15587#S4.SS1), PP15 with an instruction\-tuned base leaves essentially no transfer opportunity: ten times more target supervision improves accuracy by only\+0\.02\+0\.02on average\. Consistently, the best of fourteen parameter\-reuse configurations \(oracle\-selected source; initialisation, perturbation, moment, and projection families;n=5n\{=\}5\) improves the no\-transfer baseline by\+0\.005\+0\.005— within one seed standard deviation — and the mean across configurations is−0\.008\-0\.008\. No transient advantage survives past∼50\{\\sim\}50steps under early\-budget checks\.

##### Protocol caution\.

An earlier oracle sweep under an asymmetric merge protocol reported peer\-pair gains up to\+0\.96\+0\.96; controlled re\-measurement attributed∼97%\{\\sim\}97\\%of that signal to an eval\-path artefact of the merge rule — oracle definitions must be protocol\-clean\.

### A\.2TRACE\-8: controlled transfer measurement

This appendix records the controlled protocol behind the transfer claims of §[4](https://arxiv.org/html/2607.15587#S4): the five probe arms \(target\-data, target\-full, mixed with a chosen source, random and hindsight\-oracle selection\) run per target on a fresh adapter\.

##### Budget sweep\.

The sweep behind Figure[2](https://arxiv.org/html/2607.15587#S4.F2)runs every arm atN∈\{10,20,50,100,200,500\}N\\in\\\{10,20,50,100,200,500\\\}on the five TRACE classification targets \(sources always contribute500500examples; mixing50/5050/50\)\. Three regularities hold across targets and confirm the flagship reading: the opportunity shrinks monotonically asNNgrows; the selected\-replay gain tracks it and concentrates on matrix\-strong targets — numglue\-cm captures35%35\\%of its remaining opportunity atN=10N\{=\}10\(\+\.066\+\.066, a2×2\\timeslabel efficiency\) — while matrix\-weak targets sit near zero and pay for the forced mixture once saturated; and the parameter\-init arm never separates from zero at any budget\.

##### Task\-signature robustness\.

Each signature uses1010probe batches and is scored by best\-source top\-1 hit rate against the measured matrix \(chance≈14%\{\\approx\}14\\%, i\.e\.1/71/7\)\. Two observations complement Table[2](https://arxiv.org/html/2607.15587#S4.T2)\.*The signal is distributed across depth*: restricting the gradient signature to the last four blocks \(GradB\_last4\) drops top\-1 from4/84/8to3/83/8, so no single depth carries the alignment\.*The fixed shared initialisation matters*: every task’s gradient is taken at the same reference point, so the signatures form a stable, training\-free pairwise object — unlike gradient similarities computed at evolving weights during CL \(e\.g\., for adapter routing\(Qianet al\.,[2025](https://arxiv.org/html/2607.15587#bib.bib36)\)\)\.

##### Local intervention\.

*Protocol\.*Three classification targets spanning the structure of Table[1](https://arxiv.org/html/2607.15587#S4.T1)\(C\-STANCE: weak, mixed structure; ScienceQA: negative sources present; NumGLUE\-ds: strong positive transfer\)\. Each repetition draws three independent batches — one for the target gradient, one for a source gradient, one held\-out target batch for evaluation — and applies pure parameter perturbations \(no optimiser state\), all normalised to the same normϵ\\epsilon: the*source\-only*effect isLt​\(θ−ϵ​g^s\)−Lt​\(θ\)L\_\{t\}\(\\theta\-\\epsilon\\hat\{g\}\_\{s\}\)\-L\_\{t\}\(\\theta\), and the*marginal*effect adds the source step after one equal\-norm target stepθT\\theta\_\{T\}, i\.e\.Lt​\(θT−ϵ​g^s\)−Lt​\(θT\)L\_\{t\}\(\\theta\_\{T\}\-\\epsilon\\hat\{g\}\_\{s\}\)\-L\_\{t\}\(\\theta\_\{T\}\); a second, independent target batch added toθT\\theta\_\{T\}gives the equal\-compute reference\. To first order the source\-only effect is−ϵ​∇Lt​\(θ\)⊤​g^s\-\\epsilon\\,\\nabla L\_\{t\}\(\\theta\)^\{\\\!\\top\}\\hat\{g\}\_\{s\}, so alignment should control immediate descent\. We useϵ∈\{0\.25,0\.5,1\}×\\epsilon\\in\\\{0\.25,0\.5,1\\\}\\timesthe measured one\-step parameter displacement and3030repetitions per state\.*Results\.*At the shared initialisation, the within\-target rank correlation between gradient cosine and immediate loss reduction isρ=\.93\\rho=\.93\(source\-only\) and\.87\.87\(marginal\) over9090repetitions\. The group ordering of Table[2](https://arxiv.org/html/2607.15587#S4.T2)is strict at all three scales, with helpful\-step rates of100%/96%/67%100\\%/96\\%/67\\%for top/middle/bottom sources and the bottom pair turning harmful at the largest scale\. The stored signature preserves the current gradient geometry \(Spearman\.85\.85\), so the quantity TSR routes on identifies the sources whose gradients provide compatible descent; the top\-aligned pair recovers roughly28%28\\%of the descent an additional target batch would provide, without additional target labels\.*Scope\.*The intervention is diagnostic at the shared initialisation, where TSR measures its signatures\. Later checkpoints were also examined; the one\-step probe loses power once the5050\-example target budget is largely fitted, including for target gradients themselves\. We therefore scope the claim to the shared initialisation; long\-horizon effects are what the stream experiments measure\.

### A\.3SuperNI: the generic\-transfer regime

SuperNI illustrates the generic\-transfer boundary predicted by the framework \(§[6\.3](https://arxiv.org/html/2607.15587#S6.SS3)\)\. On six held\-out Super\-NaturalInstructions pairs \(n=5n\{=\}5\), headroom exists: additional in\-domain supervision improves loss on6/66/6pairs — PP15’s zero\-headroom regime, not TRACE’s, is the anomaly\. Source data helps: mixed training improves target loss on5/65/6pairs\. But selection has almost no value: related and unrelated sources help nearly equally \(\+0\.04\+0\.04loss difference\)\. The selection premium collapses exactly where source\-quality variance does; we report this boundary rather than average over it\.

### A\.4NumGLUE\-8: the homogeneous regime

#### A\.4\.1Headroom audit

NumGLUE\-8 casts the eight task types of the NumGLUE suite\(Mishraet al\.,[2022](https://arxiv.org/html/2607.15587#bib.bib65)\)as the homogeneous regime: one numerical\-reasoning core in eight formats \(types 4/5 are TRACE’s numglue\-cm/ds\)\. The §[4\.1](https://arxiv.org/html/2607.15587#S4.SS1.SSS0.Px1)audit gives three regularities\. Headroom exists and persists across scale:7/87/8types retain accuracy headroom at 0\.5B and at 7B \(n=5n\{=\}5\)\. Pooling beats single\-source selection on the probe at both scales \(0\.3480\.348vs\.0\.3420\.342at 0\.5B,0\.6000\.600vs\.0\.5850\.585at 7B\) — the pooling prediction of §[4\.3](https://arxiv.org/html/2607.15587#S4.SS3)\. And the within\-family oracle ceiling compresses from\+0\.033\+0\.033to\+0\.003\+0\.003across the same scales: in\-domain headroom persists while the share other tasks can supply shrinks\.

#### A\.4\.2Pairwise transfer matrix

Full transfer matrix \(Δ\\Deltaacc\) behind §[6\.3](https://arxiv.org/html/2607.15587#S6.SS3)\(Δ\\Deltaacc of50/5050/50joint training vs\. target\-data;n=5n\{=\}5; green = helps, red = hurts, bold = column best; T4/T5 are TRACE’s numglue\-cm/ds\)\.

Table 4:Pairwise transfer matrix on NumGLUE\-8\.

## Appendix BImplementation Evidence

### B\.1The carrier

Our evidence rules out parameter reuse*as transient optimiser state*: initialising from, perturbing toward, or injecting the moments of a past adapter into a module that then continues training — the form parameter reuse takes in dominant LLM\-CL practice\. It does*not*contradict the two families of parameter\-side successes in the literature, and in fact explains both\. Inference\-time composition is never trained against — nothing erases it\. And architectural sharing in the Progressive\-Networks / CAT / CTR lineage keeps past parameters*frozen and present at every forward pass*— a persistent signal, like data, rather than a one\-shot initial condition\. The unifying property is persistence:*signals that participate in every training step survive; one\-shot parameter states are washed out*\. The property grades, rather than merely splits, the mechanism space: an L2 anchor toward the source adapter — a*persistent parameter*signal — captures roughly two\-thirds of data mixing’s gain at the better anchor strength where one\-shot initialisation captures none \(mean0\.45260\.4526vs\.0\.44720\.4472target\-data and0\.45470\.4547data; Appendix[B\.2](https://arxiv.org/html/2607.15587#A2.SS2)\)\. The three carriers therefore occupy the persistence axis exactly as predicted\.

##### A structural intuition\.

LoRA’s updateB​ABAis a*residual*on the shared base: it encodes what remains of a task after the base has accounted for what the tasks share\. Two related tasks can have near\-orthogonal residuals \(the shared part was subtracted\), which is why parameter similarity under\-determines transfer\. What related tasks actually share — the capability the base lacks — lives in their data, and is re\-suppliable at every step\.

### B\.2Replay and anchor ablations

##### Persistent parameter anchoring \(L2\-toward\-source\)\.

Training the target withL=CE\+μ​∥θ−θsource∥2L=\\mathrm\{CE\}\+\\mu\\lVert\\theta\-\\theta\_\{\\mathrm\{source\}\}\\rVert^\{2\}keeps the source adapter present at every step — a persistent signal whose carrier is*parameters*\. On the five oracle pairs \(n=5n\{=\}5\): mean accuracy0\.45260\.4526atμ=0\.1\\mu\{=\}0\.1and0\.44100\.4410atμ=1\.0\\mu\{=\}1\.0, vs\.0\.44720\.4472target\-data and0\.45470\.4547for data mixing\. Persistence is necessary \(the one\-shot init captures nothing\) but not sufficient: the data carrier remains stronger at everyμ\\muwe probed\.

### B\.3Routing\-policy variants

The routed triple has two knobs: the routing temperatureτ\\tau\(argmax atτ→0\\tau\{\\to\}0, uniform pooling atτ→∞\\tau\{\\to\}\\infty\) and the teacher binding \(recent snapshot vs\. the routed batch’s era snapshot\)\. All single\-component variants, same protocol and budgets as Table[3](https://arxiv.org/html/2607.15587#S6.T3):

Table 5:Routing\-policy variants of TSR under the standard stream protocol\.Each ablated variant loses on the axis its missing component serves \(argmax loses coverage, pooling loses selection, recent teachers lose the undrifted reference\)\.

### B\.4Distillation: the full ablation record

This appendix records the distillation audit referenced in §[6\.3](https://arxiv.org/html/2607.15587#S6.SS3)\. All experiments: PP15, sequential CL with a single shared LoRA,q,k,v,oq,k,v,oprojections,500500steps/task,n=5n\{=\}5seeds, final mean accuracy \(AVG\) unless noted\. The cumulative conclusion: KD in this setting is a trajectory regulariser anchored to the shared base — a stability mechanism — and cannot be converted into a transfer channel by teacher choice, loss reshaping, timing, weighting, or snapshot source\.

#### B\.4\.1Teacher identity is irrelevant \(routing ceiling\)

If KD transferred teacher\-specific knowledge, teacher choice should matter\. It does not:

Table 6:Teacher\-identity ablation for the distillation audit\.Even the per\-batch oracle — the tightest upper bound on routing benefit — adds nothing over a random teacher\. The gain over no distillation \(\+0\.13\+0\.13\) is therefore attributable to having*an*anchor, not to choosing the right one\.

#### B\.4\.2Six loss\-reshaping interventions all fail

Each intervention was designed to amplify or isolate teacher\-specific signal; every one hurts:

Table 7:Loss\-reshaping interventions on the distillation term\.Removing the base\-shared component or repelling the student from the base is catastrophic — direct evidence that the value of KD lies in the base\-shared anchor, not in task\-specific teacher content\.

#### B\.4\.3Timing and weight: regulariser signatures

KD interleaved with CE beats KD applied as a post\-hoc phase \(endpoint correction\), across two initialisation modes:

Table 8:Distillation timing: interleaved versus post\-hoc\.The weight curve is shallow over an order of magnitude \(AVG0\.6730\.673–0\.6830\.683forλ∈\[0\.1,2\]\\lambda\\in\[0\.1,2\]\) — no narrow optimum, no collapse at either end: the canonical signature of a soft regulariser\.

#### B\.4\.4Snapshot source is irrelevant

Anchoring to an accumulated merged snapshot versus a routed per\-task snapshot makes no measurable difference \(QR init:0\.6730\.673vs\.0\.6730\.673; routed\-mix init:0\.6270\.627vs\.0\.6070\.607, within1​σ1\\sigma\): both pull the student toward the same base\-shared manifold\.

#### B\.4\.5Transfer channels through distillation: local signal exists, none survives

Beyond loss reshaping, we tested whether snapshots carry extractable transfer along three channels, plus a vision\-imported learnable bridge\. A zero\-step diagnostic first confirmed a real*local*signal: perturbing parameters along a snapshot’s update direction predicts loss change with Pearsonr=0\.93r=0\.93at small step sizes\. None of it survives training:

Table 9:Transfer channels through distillation\.A control worth recording: under the projected\-bridge framework, a*random*teacher scores\+0\.014\+0\.014*above*the real teacher — the spectral\-regularisation signature, confirming the bridge’s gain \(where any\) is regularisation, not transfer\.

#### B\.4\.6Teacher choice in the stream

Three\-task stream numglue\_cm→\\tocstance→\\tonumglue\_ds \(target,N=50N\{=\}50,50/5050/50selected replay of cm;n=5n\{=\}5\)\. Arms differ only in the KD teacher\. “Forgetting” is accuracy change from the pre\-target reference\.

Table 10:Teacher choice in the three\-task stream\.The replayed task \(cm\) is protected in every arm — replay data itself does that work; in this*short*stream the recent snapshot is best or tied\.In the full eight\-task stream the ranking reverses: with seven candidate snapshots and substantial drift accumulated in the recent one, matched\-era teachers win \(TRACE\-0\.5B,n=5n\{=\}5: matched0\.458±\.0280\.458\{\\scriptstyle\\pm\.028\}\>\>random0\.445±\.0240\.445\{\\scriptstyle\\pm\.024\}\>\>recent0\.439±\.0330\.439\{\\scriptstyle\\pm\.033\}\)\. The regime dependence is mechanistic: the recent snapshot is the most complete teacher but also the most drifted; as the stream lengthens, drift costs more than completeness buys\. This motivates the routed\-triple design of §[5](https://arxiv.org/html/2607.15587#S5)\.

#### B\.4\.7Lineage, novelty, and mechanism

##### Lineage\.

Distillation for stability in continual learning descends from LwF\(Li and Hoiem,[2017](https://arxiv.org/html/2607.15587#bib.bib17)\)\(previous model as teacher, matched on new\-task inputs; CNN\-era task\-incremental classification\) and iCaRL\(Rebuffiet al\.,[2017](https://arxiv.org/html/2607.15587#bib.bib14)\)\(exemplars plus old\-class distillation\)\. DER\+\+\(Buzzegaet al\.,[2020](https://arxiv.org/html/2607.15587#bib.bib13)\)is the closest relative: it stores each buffered sample’s logits at insertion time and anchors to them on replay — a frozen, per\-sample form of era anchoring\. Our era snapshots generalise this in three ways: the anchor is a queryable model rather than stored values \(it can answer on any batch, including current\-task batches, and is taken at task end rather than at insertion\); the teacher is bound to the replay data in one routable record, so a single decision returns both; and the role assignment is inverted — DER\+\+ points replay at stability, we point replay at transfer and give stability to distillation alone\. Under our protocol the two are also empirically separated \(Table[3](https://arxiv.org/html/2607.15587#S6.T3)\)\. In recent continual learning with LLMs, distillation appears mainly as self\-distillation\(Yanget al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib66)\); we are not aware of a system using routed, era\-matched distillation as the explicit stability module\.

##### Mechanism: why anchoring, not knowledge, in this regime\.

Three properties of the regime jointly explain the audit\. \(i\) Ground\-truth supervision is present on every batch, so teacher logits carry little marginal label information — unlike compression distillation, where the teacher is the information source\. \(ii\) All teachers are small low\-rank perturbations of one frozen base \(r=16r\{=\}16,500500steps,N=50N\{=\}50\), so every anchor’s pull is dominated by the shared base direction — which is why identity is second\-order \(E1, E4\) and why deleting the shared component is catastrophic \(E2\)\. \(iii\) The teacher\-specific component is real but*transient*: it is locally measurable \(E5,r=0\.93r\{=\}0\.93\) yet, like the one\-shot parameter states of §[4\.2](https://arxiv.org/html/2607.15587#S4.SS2), it does not survive continued optimisation — what persists is what is re\-injected identically at every step: the data, and the shared anchor\. There is also a structural asymmetry: replay changes the*input*distribution, while distillation only tilts the target distribution on inputs already present — and a teacher’s source\-task knowledge barely manifests on other tasks’ inputs\. Cross\-task distillation does carry knowledge where a jointly trained shared encoder makes the teacher signal persistent and large \(multi\-task vision and recommendation systems\); in the carrier taxonomy of Appendix[B\.1](https://arxiv.org/html/2607.15587#A2.SS1)those systems occupy the persistent\-parameter cell\. Our negative result is scoped to the sequential, frozen\-base, low\-rank\-snapshot regime, where we recommend assigning distillation to stability and spending the complexity budget on data selection instead\.

## Appendix CExperimental Details

### C\.1Full\-stream setup

##### Configuration\.

All methods train a single LoRA adapter \(r=16r\{=\}16,α=32\\alpha\{=\}32on theq,k,v,oq,k,v,oprojections; AdamW, lr10−410^\{\-4\}, batch size44at 0\.5B /22at 7B, max length384384\) for500500steps per task on theN=50N\{=\}50target examples, with a replay buffer of100100examples per completed task \(Nmax=500N\_\{\\max\}\{=\}500in the §[4](https://arxiv.org/html/2607.15587#S4)audits\)\. TSR:m=10m\{=\}10probe batches, routing temperatureτ=0\.1\\tau\{=\}0\.1, replay ratioα=0\.5\\alpha\{=\}0\.5, distillation weightλ=0\.5\\lambda\{=\}0\.5at temperature22\. The scale audit of §[6\.3](https://arxiv.org/html/2607.15587#S6.SS3)usesn=5n\{=\}5seeds per scale \(per\-point seed s\.e\.m\.\.002\.002–\.010\.010\)\.

##### Evaluation\.

Evaluation uses500500\-example per\-task test sets; classification tasks report exact\-match accuracy, generation tasks evaluation loss\.

##### Stream construction\.

NumGLUE\-8 arranges the eight task types of the established NumGLUE suite\(Mishraet al\.,[2022](https://arxiv.org/html/2607.15587#bib.bib65)\)as a continual stream — the same construction by which Split\-CIFAR derives from CIFAR and TRACE itself from eight existing datasets; two of its types \(numglue\-cm/ds\) already appear inside TRACE\.

##### Cost of TSR\.

Signatures arem=10m\{=\}10forward\-backward passes per task \(all eight of a benchmark’s signatures take∼30\{\\sim\}30seconds at 0\.5B, and picks are scale\-stable, so a small proxy model suffices\); routing is an inner product; training cost matches standard replay; the buffer stores100100examples per task \(∼3\{\\sim\}3MB for1515tasks, versus∼130\{\\sim\}130MB for per\-task adapter libraries\)\. We regard this routable task record — signature, data subsample, snapshot — rather than the example\-level buffer of rehearsal methods or the adapter\-only library of parameter\-isolation methods, as the right unit of continual memory\.

##### New\-baseline ports\.

GainLoRA and DEAL \(NeurIPS’25\) are ported from their official releases into the stream protocol, mechanism\-specific components kept as released \(GainLoRA’s projection\-constrained model\-level gate; DEAL’s Haar\-DWT retention filter\) and only the adapter footprint unified for parity \(q,k,v,oq,k,v,oatr=16r\{=\}16,α=32\\alpha\{=\}32, lr10−410^\{\-4\}\)\. Two adaptations: DEAL’s per\-task re\-initialisation depends on a checkpoint\-namespace quirk that cannot restore the adapter from task 3 on, so we adopt the continuous reading consistent with the paper’s framing; and at DEAL’s native lr10−510^\{\-5\}the500500\-step budget under\-trains in all four settings \(TRACE\-0\.5B overall\.068\.068vs\.\.372\.372\), so the standardised rate is also the charitable one\.

### C\.2Per\-task results

End\-of\-stream accuracy per classification task, TRACE\-8 at 0\.5B \(canonical order, mean over55seeds; the per\-task view behind Table[3](https://arxiv.org/html/2607.15587#S6.T3)\)\.

Table 11:Per\-task end\-of\-stream accuracy on TRACE\-8 at 0\.5B\.Per\-task tables for the remaining settings follow the same pattern and are available with the released evaluation matrices\.

### C\.3Stream robustness

##### Task order\.

Across the greedy best\-source curriculum, its reversal, two deterministic reshufflings, and arbitrary reorderings, at both scales, the replay family varies by at most\.03\.03in ranking and magnitude; no\-memory baselines vary substantially, tracking the order’s*tail*— recency, not the matrix: at 0\.5B they swing by\+\.05\+\.05to\+\.09\+\.09under the curriculum, and an arbitrary reshuffle with a similar tail lifts SeqFT even more \(\+\.12\+\.12\); at 7B, where the base barely forgets, the same swings shrink below\.04\.04, and on NumGLUE all order effects are within seed noise \(§[6\.4](https://arxiv.org/html/2607.15587#S6.SS4)\)\. The one mild sensitivity is argmax selection when a target precedes its best sources — the transfer matrix is directional, and soft routing removes this by construction\.

##### Backbone\.

We further evaluate whether the framework generalises across backbone families\. Following §[4](https://arxiv.org/html/2607.15587#S4), we first audit headroom: on Llama\-3\.2\-1B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib28)\)and Gemma\-3\-1B\-it\(Gemma Team,[2025](https://arxiv.org/html/2607.15587#bib.bib29)\), the TRACE\-8 audit finds open headroom on every task \(target\-full minus target\-data,n=2n\{=\}2:\+\.045\+\.045to\+\.135\+\.135accuracy on Llama and\+\.045\+\.045to\+\.269\+\.269on Gemma across the five classification targets; the three generation targets improve by1\.81\.8–4\.64\.6in loss\), so transfer is measurable on both\. Table[12](https://arxiv.org/html/2607.15587#A3.T12)reports the stream under the exact configuration of Appendix[C\.1](https://arxiv.org/html/2607.15587#A3.SS1)\. The qualitative picture of Table[3](https://arxiv.org/html/2607.15587#S6.T3)is unchanged: replay\-based methods dominate; TSR attains the best overall accuracy, plasticity, and backward transfer on both backbones; and positive backward transfer remains confined to the replay\-based methods — every rehearsal\-free baseline, including the expansion\-based GainLoRA, forgets\.

Table 12:Backbone robustness of TSR on TRACE\-8under the standard sequential protocol \(Llama\-3\.2\-1B and Gemma\-3\-1B; mean±\\pmstd over 5 seeds\)\.

### C\.4Reproducibility

Base models: Qwen2\.5\-0\.5B\-Instruct and Qwen2\.5\-7B\-Instruct\(Qwen Team,[2024](https://arxiv.org/html/2607.15587#bib.bib27)\); the backbone\-robustness check additionally uses Llama\-3\.2\-1B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.15587#bib.bib28)\)and Gemma\-3\-1B\-it\(Gemma Team,[2025](https://arxiv.org/html/2607.15587#bib.bib29)\)\. Training configuration as in Appendix[C\.1](https://arxiv.org/html/2607.15587#A3.SS1)\(PP15 runs use max length256256\)\. Probe protocol, arm definitions, and per\-cell seed counts as stated inline; full\-seed statistics are reported throughout to avoid favourable\-subset selection bias\(Liet al\.,[2026](https://arxiv.org/html/2607.15587#bib.bib72)\)\. A minimal implementation reproducing the core TRACE\-8 experiments is released at[https://github\.com/ymeng3/transfer\-selective\-replay](https://github.com/ymeng3/transfer-selective-replay)\.