Interference and Retention in Continual Learning

# Interference and Retention in Continual Learning
Source: [https://arxiv.org/html/2607.09202](https://arxiv.org/html/2607.09202)
Julius Störk1,∗ 1VARTA Microbattery GmbH, Ellwangen, Germany ∗Corresponding author:julius\.stoerk@varta\-ag\.com 10\.07\.2026

###### Abstract

Continual learning commonly relies on post\-hoc mechanisms such as replay, elastic regularization, or distillation\. This work argues that forgetting should instead be modeled directly as interference between tasks\. In the frozen\-feature regime, forgetting from learning a new task is exactly the interference energy induced on the old task\. In deep networks, the same quantity is recovered through path\-averaged curvature with minimal additional forward passes\.

When task supports are disjoint, forgetting can be eliminated structurally and when task supports overlap in conflicting directions, a non\-zero distortion floor is unavoidable\. The same geometry optimally merges models through task\-aware orthogonalization\. From this analysis we derive Interference\-Gated Functional Allocation \(igfa\), a replay\-free, Fisher\-free method that shares directions when tasks align and protects them when they conflict\. Across benchmarks, igfa achieves lossless retention when tasks are structurally separable and moves unavoidable cost from irreversible forgetting into deferred but recoverable plasticity when they are not\. It matches the strongest replay\-free structural baselines on dissimilar\-task streams and improves on unconditional projection when similarity makes transfer worth preserving\.

Keywords:continual learning; catastrophic forgetting; model merging; neural tangent kernel; gradient projection; rate–distortion; representation geometry

![Refer to caption](https://arxiv.org/html/2607.09202v1/x1.png)Figure 1:Accuracy versus forgetting\.Nine methods are tested on the exact\-A1 Rotated\-Digits benchmark \(five rotations, frozen random features; mean over55seeds\) and differentiated by method class: baseline \(∙\\bullet\), corrective \(∙\\bullet\), structural \(∙\\bullet\), and merging \(∙\\bullet\); our methods are marked “\(ours\)” and drawn as stars in their class colour \(★\\bigstar,★\\bigstar\); higher accuracy \(up\) and lower forgetting \(right\) are better\. Corrective \(EWC, replay\) and structural \(ogd/gpm, igfa\) methods learn the stream sequentially, whereas merging methods solve each rotation independently and combine offline\.*\(i\)*TheΣ\\Sigma\-orthogonal merge of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)attains near\-oracle accuracy \(0\.8850\.885against a0\.9030\.903per\-task ceiling\) with the smallest post\-merge residual floor \(D=0\.032D=0\.032, versus0\.200\.20and0\.380\.38for the isotropic and Euclidean merges\), confirming that the task\-induced, non\-EuclideanΣ\\Sigma\-metric is the correct inner product for merging\.*\(ii\)*Among online, buffer\- and Fisher\-free methods, our signed gate \(igfa\) attains the highest accuracy; annealing the protection boundary \(Annealed igfa\) trades slightly lower accuracy for lower, even negative, forgetting\. Full numbers with confidence intervals and the residual\-DDablation are in Sec\.[S4](https://arxiv.org/html/2607.09202#A4)\.## 1Introduction

A model trained sequentially on taskBBafter taskAAoften degrades on taskAA, a phenomenon known as catastrophic forgetting\. Networks trained by stochastic gradient descent \(SGD\) or its variants update from the current minibatch alone \(or a smoothed average of a short window of minibatches\); the update is therefore*oblivious to past knowledge*\. This obliviousness is desirable when the training data are i\.i\.d\., but harmful once the training distribution shifts over time in the continual setting\. Standard approaches such as replay\[[3](https://arxiv.org/html/2607.09202#bib.bib3),[4](https://arxiv.org/html/2607.09202#bib.bib4),[5](https://arxiv.org/html/2607.09202#bib.bib5)\], elastic weight consolidation\[[1](https://arxiv.org/html/2607.09202#bib.bib1),[2](https://arxiv.org/html/2607.09202#bib.bib2)\], and distillation\[[6](https://arxiv.org/html/2607.09202#bib.bib6)\]mostly repair forgetting after interference has already occurred\. What is still missing is a predictive account of when interference is avoidable, when it is inevitable, and what structure an update must satisfy to retain old behavior without sacrificing useful transfer\.

This work focuses on the linear\-on\-features regime in which a frozen feature extractorϕ​\(x\)\\phi\(x\)is paired with a trainable linear head, covering the common frozen\-backbone and parameter\-efficient fine\-tuning \(PEFT\) settings and the first\-order neural tangent\[[11](https://arxiv.org/html/2607.09202#bib.bib11)\]approximation of a fully trained network\. The forgetting of an earlier taskAAafter an updateΔ\\Deltacaused by learning taskBBis exactly the interference energy12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\,\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta, whereΣA=𝔼x∼DA​\[ϕ​\(x\)​ϕ​\(x\)⊤\]\\Sigma\_\{A\}\\ =\\mathbb\{E\}\_\{x\\sim D\_\{A\}\}\\left\[\\phi\(x\)\\phi\(x\)^\{\\top\}\\right\]denotes the feature second moment of taskAA, measuring which feature directions are active for that task\. The same geometry predicts per\-task forgetting accurately and extends approximately to deeper networks with feature drift\.

The paper makes three contributions\.

1. 1\.Interference functional and removability\.We derive an exact interference functional for forgetting\. In the frozen\-feature regime, forgetting is exactly the old task’s interference energy under the new update, which yields a clean criterion for when retention is lossless and when a distortion floor is unavoidable\. The geometry ofΣA\\Sigma\_\{A\}penalizes only components in its active subspace, while updates inker⁡ΣA\\ker\\Sigma\_\{A\}leave taskAA’s loss unchanged\. This yields a structural criterion for lossless retention: interference is removable for disjoint task supports \(reducing to ordinary orthogonality in isotropic geometry\), whereas overlapping supports imply a non\-zero distortion floor\. The same quantity also admits an information\-theoretic interpretation through the task\-inference problem\.
2. 2\.Optimal merging asΣ\\Sigma\-orthogonalization\.We show that optimal merging is task\-aware orthogonalization\. This unifies offline model merging and online continual learning as batch and incremental solutions for minimizing the same interference objective\. Section[3\.4](https://arxiv.org/html/2607.09202#S3.SS4)shows that post\-merge loss decomposes into task\-wise fit and cross\-task interference\. Minimizing this objective requires mutualΣ\\Sigma\-orthogonality of the task vectors, yielding the optimality condition for model merging\.
3. 3\.Interference\-Gated Functional Allocation\.We introduce igfa, a replay\-free structural controller\. The method shares capacity when tasks align, protects it when they conflict, and retains only a low\-rank subspace summary\. Section[4](https://arxiv.org/html/2607.09202#S4)introduces igfa, an online method motivated by the same functional \(Fig\.[2](https://arxiv.org/html/2607.09202#S1.F2)\)\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/figures/Subspace_Interference_and_Forgetting_Geometry.png)Figure 2:The task\-induced geometry in feature space motivates either full sharing, full separation, or selective parameter sharing\.*Left \(Disjoint\)\.*When two tasks excite orthogonal feature subspaces, the updateΔ\\Deltais composed of a componentΔ∥∈range⁡ΣA\\Delta\_\{\\parallel\}\\in\\operatorname\{range\}\\Sigma\_\{A\}between the iso\-loss contours that incurs forgetting andΔ⟂∈ker⁡ΣA\\Delta\_\{\\perp\}\\in\\ker\\Sigma\_\{A\}along the iso\-loss contours where allocation is interference\-free\.*Centre \(Overlapping\)\.*When the supports share an active direction, no update forBBcan avoidrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}and a distortion floor is unavoidable\.*Right\.*The same functional yields three update rules\. Naive descent incurs the full interference \(distorts old knowledge\); orthogonal projection \(ogd/gpm\[[7](https://arxiv.org/html/2607.09202#bib.bib7),[8](https://arxiv.org/html/2607.09202#bib.bib8)\]\) removes all overlap \(no forgetting, but no transfer and capacity exhaustion\); igfa applies a*signed*gate—sharing aligned directions and orthogonalizing conflicting ones—balancing retention and transfer while carrying only the occupied subspace as state\.The experiments in Section[5](https://arxiv.org/html/2607.09202#S5)are designed to test the theory directly, including lossless allocation on disjoint supports, the relocation of unavoidable cost from forgetting into deferred plasticity, the similarity threshold ats∗≈0\.26s^\{\\ast\}\\approx 0\.26, and the onset of a capacity floor once the cumulative occupied rank reaches the feature dimension,T​r≈dT\\,r\\approx d\(withTTthe number of tasks in the stream,rrthe rank of the feature subspace each task occupies, andddthe feature dimension\)\. In the frozen\-feature Split\-Digits setting, igfa matches the strongest replay\-free and Fisher\-free baseline at about0\.980\.98accuracy with near\-zero forgetting while using no replay buffer\. Extensions to full\-network training, large\-scale overlays, and direct comparison with replay at scale are discussed in Section[6](https://arxiv.org/html/2607.09202#S6), with the supporting experiments detailed in the Supporting Information\.

## 2Related work

#### Corrective continual learning\.

Replay\[[3](https://arxiv.org/html/2607.09202#bib.bib3),[4](https://arxiv.org/html/2607.09202#bib.bib4),[5](https://arxiv.org/html/2607.09202#bib.bib5)\], regularization\[[1](https://arxiv.org/html/2607.09202#bib.bib1),[2](https://arxiv.org/html/2607.09202#bib.bib2)\], and distillation\[[6](https://arxiv.org/html/2607.09202#bib.bib6)\]remain the main corrective approaches to continual learning\. They do not explicitly model interference through task geometry however, while the herein derived interference functional predicts the collision of the task updates and identifies when it can be removed before occuring\. Replay can be viewed as a stochastic Monte\-Carlo estimator of the same task second\-momentΣt\\Sigma\_\{t\}that we estimate sample\-free via a low\-rank subspace\.

#### Geometry\- and gradient\-based methods\.

A related line of work controls the update direction itself\. Orthogonal Gradient Descent\[[7](https://arxiv.org/html/2607.09202#bib.bib7)\], Gradient Projection Memory\[[8](https://arxiv.org/html/2607.09202#bib.bib8)\], and Orthogonal Weight Modification\[[9](https://arxiv.org/html/2607.09202#bib.bib9)\]project updates away from subspaces associated with previous tasks\. Under a fixed\-feature map, this corresponds to enforcingΔ∈ker⁡ΣA\\Delta\\in\\ker\\Sigma\_\{A\}\. PCGrad\-style methods\[[10](https://arxiv.org/html/2607.09202#bib.bib10)\]resolve conflicts through pairwise gradient comparisons rather than explicit task\-subspace protection\. Recent work also studies closely related regimes: Geodesic\-Aligned Gradient Projection\[[27](https://arxiv.org/html/2607.09202#bib.bib27)\]addresses feature drift during sequential learning, and MINGLE\[[28](https://arxiv.org/html/2607.09202#bib.bib28)\]uses a null\-space criterion to gate low\-rank experts during test\-time merging\. Relative to these approaches, the present framework derives the projection condition from an exact forgetting functional, replaces unconditional projection with a similarity\-gated share\-or\-orthogonalize rule, and uses the same functional to characterize both merging and the distortion floor\. NTK\-based analyses\[[12](https://arxiv.org/html/2607.09202#bib.bib12),[13](https://arxiv.org/html/2607.09202#bib.bib13)\]likewise relate forgetting to overlap in representation space, but do not develop theΣ\\Sigma\-orthogonality criterion or the associated transfer gate\.

#### The averaged Jacobian as a shared object with interpretability\.

A convergent notion of geometry appears in mechanistic interpretability\. The Jacobian lens of Gurnee, Lindsey, et al\.\[[33](https://arxiv.org/html/2607.09202#bib.bib33)\]characterizes a residual\-stream activation by its*context\-averaged*forward JacobianJℓ=𝔼​\[∂hfinal/∂hℓ\]J\_\{\\ell\}=\\mathbb\{E\}\[\\partial h\_\{\\mathrm\{final\}\}/\\partial h\_\{\\ell\}\]composed with the unembedding, and identifies the low\-rank span of its active readout directions—the “J\-space”—as the subspace of*verbalizable*representations, limited in capacity and competitively occupied, a language\-model global workspace\. Our interference geometry is the learning\-dynamics counterpart of the same move: where the J\-lens averages the forward activation→\\tooutput Jacobian, we average the Gauss–Newton curvatureΣA=𝔼​\[Jw​Jw⊤\]\\Sigma\_\{A\}=\\mathbb\{E\}\[J\_\{w\}J\_\{w\}^\{\\top\}\]\(Sec\.[3\.3](https://arxiv.org/html/2607.09202#S3.SS3)\), and both replace a single\-point Jacobian with a distribution\-averaged one for the same stated reason—to isolate an architecture\-level invariant from context\-specific use\. The two induce the same partition of representation space: their causally load\-bearing J\-space plays the role of ourrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}\(perturbations there change outputs and incur forgetting\), and the inert complement that their coordinate\-patching “leaves unchanged” is ourker⁡ΣA\\ker\\Sigma\_\{A\}\(free directions\)\. Read this way, our interference energy and distortion floor supply a quantitative model of workspace*competition*and our occupied\-rank capacity knee a rate–distortion account of its*limited capacity*; conversely, the released lens is a ready estimator of the averaged\-Jacobian subspace our method tracks at scale\. We make the connection testable in Sec\.[6](https://arxiv.org/html/2607.09202#S6)\.

#### Model merging and task arithmetic\.

In a parallel line of literature, independently fine\-tuned are combined into a single model that retains useful knowledge from each task through arithmetic on weight\-space task vectors\[[14](https://arxiv.org/html/2607.09202#bib.bib14),[15](https://arxiv.org/html/2607.09202#bib.bib15)\]\. These methods usually operate in Euclidean parameter space, whereas the present analysis identifies the task\-dependent metric induced byΣt\\Sigma\_\{t\}as the relevant geometry\. Under this metric, optimal merging becomes mutualΣ\\Sigma\-orthogonalization, with ordinary weight\-space orthogonality recovered only whenΣt∝I\\Sigma\_\{t\}\\propto I\. Task Singular Vectors\[[24](https://arxiv.org/html/2607.09202#bib.bib24)\]promotes orthogonality by whitening singular directions of task vectors; by contrast, isotropic merging\[[25](https://arxiv.org/html/2607.09202#bib.bib25)\]shows that shared\-subspace isotropy can outperform strict orthogonalization\. WUDI\-Merging\[[23](https://arxiv.org/html/2607.09202#bib.bib23)\]is particularly close in spirit because it links task vectors to input subspaces and minimizes interference without data; the key difference in our work is that the metric and residual are*derived*from an exact forgetting functional rather than an approximate subspace argument\. Tangent\-space continual merging\[[26](https://arxiv.org/html/2607.09202#bib.bib26)\]also operates in a first\-order regime, but with similar trade\-offs between scalability and functional specificity\. The common perspective adopted here is that the same interference functional governs both offline merging and online continual allocation\.

#### Scenarios, scale, and plasticity\.

The standard task\-, domain\-, and class\-incremental taxonomy\[[16](https://arxiv.org/html/2607.09202#bib.bib16)\]distinguishes whether task identity is given or must be inferred\. In this context, the distortion floor developed later provides a quantitative link between support overlap and task inference\. Related empirical and analytical work has examined the effects of scale and pretraining\[[17](https://arxiv.org/html/2607.09202#bib.bib17)\], frozen\-backbone prompt and adapter methods\[[18](https://arxiv.org/html/2607.09202#bib.bib18),[19](https://arxiv.org/html/2607.09202#bib.bib19),[20](https://arxiv.org/html/2607.09202#bib.bib20)\], which learn prompt pools with input\-conditioned selection under a class\-incremental protocol and provide strong empirical accuracy without structural retention guarantees \(differentiated in Sec\.[5\.6](https://arxiv.org/html/2607.09202#S5.SS6)\), forgetting in linear models\[[21](https://arxiv.org/html/2607.09202#bib.bib21)\], plasticity loss\[[22](https://arxiv.org/html/2607.09202#bib.bib22)\], and convergence or task ordering in continual linear classification\[[29](https://arxiv.org/html/2607.09202#bib.bib29),[30](https://arxiv.org/html/2607.09202#bib.bib30)\]\.

## 3Preliminaries: the interference functional

This section specifies the setting and assumptions \(Sec\.[3\.1](https://arxiv.org/html/2607.09202#S3.SS1)\), derives the interference functional and the removability dichotomy \(Sec\.[3\.2](https://arxiv.org/html/2607.09202#S3.SS2)\), generalizes the identity beyond frozen features \(Sec\.[3\.3](https://arxiv.org/html/2607.09202#S3.SS3)\), and derives the optimal merge together with the distortion floor \(Sec\.[3\.4](https://arxiv.org/html/2607.09202#S3.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x2.png)Figure 3:Geometry of the general identities\.*\(a\)*Beyond frozen features the quadratic form survives withΣA\\Sigma\_\{A\}replaced by the path\-averaged curvatureH¯A\\bar\{H\}\_\{A\}: the local curvature ofLAL\_\{A\}rotates and rescales along the segmentwA→wBw\_\{A\}\\\!\\to\\\!w\_\{B\}\(ellipses, blue→\\tored\), earlier points weighted by\(1−t\)\(1\-t\)\(wedge\), and the straight chord stands in for the drifted geodesic \(dashed\) \(Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)\)\.*\(b\)*Optimal merging isΣ\\Sigma\-weighted, not Euclidean: drawn from the exact2×22\{\\times\}2geometry, the joint optimumw¯∗\\bar\{w\}^\{\\ast\}\(star\) minimizesLA\+LBL\_\{A\}\{\+\}L\_\{B\}, while the Euclidean average \(cross\) sits strictly outside the optimum’s level set and pays avoidable interference \(Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\)\.### 3\.1Setting and assumptions

Letϕ:𝒳→ℝd\\phi:\\mathcal\{X\}\\to\\mathbb\{R\}^\{d\}be a fixed feature map andw∈ℝdw\\in\\mathbb\{R\}^\{d\}a trainable linear head with predictiony^=w⊤​ϕ​\(x\)\\hat\{y\}=w^\{\\top\}\\phi\(x\)\. For tasktt, letΣt=𝔼x∼𝒟t​\[ϕ​\(x\)​ϕ​\(x\)⊤\]⪰0\\Sigma\_\{t\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{t\}\}\[\\phi\(x\)\\phi\(x\)^\{\\top\}\]\\succeq 0denote the feature second moment, and letwt∗w\_\{t\}^\{\\ast\}be a realizable optimum\. The per\-task excess loss is

Lt​\(w\)=12​\(w−wt∗\)⊤​Σt​\(w−wt∗\)≥0\.L\_\{t\}\(w\)=\\tfrac\{1\}\{2\}\\,\(w\-w\_\{t\}^\{\\ast\}\)^\{\\top\}\\Sigma\_\{t\}\\,\(w\-w\_\{t\}^\{\\ast\}\)\\;\\geq 0\.\(1\)Under squared loss, this coincides with the excess population risk\. We write the target asy=ϕ​\(x\)⊤​wt∗\+εy=\\phi\(x\)^\{\\top\}w\_\{t\}^\{\\ast\}\+\\varepsilonto separate the predictable linear component from the irreducible residual variation\. Withy=ϕ​\(x\)⊤​wt∗\+εy=\\phi\(x\)^\{\\top\}w\_\{t\}^\{\\ast\}\+\\varepsilon,𝔼​\[ε\]=0\\mathbb\{E\}\[\\varepsilon\]=0, andε⟂ϕ​\(x\)\\varepsilon\\\!\\perp\\\!\\phi\(x\), the squared risk satisfies𝔼​\[\(y^−y\)2\]/2=Lt​\(w\)\+12​𝔼​\[ε2\]\\mathbb\{E\}\[\(\\hat\{y\}\-y\)^\{2\}\]/2=L\_\{t\}\(w\)\+\\tfrac\{1\}\{2\}\\mathbb\{E\}\[\\varepsilon^\{2\}\], so all subsequent loss differences depend only onLt​\(w\)L\_\{t\}\(w\)\.

Three assumptions define this regime:\(A1\) frozen features—ϕ\\phiis fixed during training, exact for frozen\-backbone and PEFT settings and the first\-order NTK\[[11](https://arxiv.org/html/2607.09202#bib.bib11)\]model of end\-to\-end training, so full\-network claims are first\-order and are tested in Sec\.[5\.4](https://arxiv.org/html/2607.09202#S5.SS4);\(A2\) realizability—eachwt∗w\_\{t\}^\{\\ast\}attains zero excess loss, so mis\-specification adds only a task\-constant and leaves the interference terms untouched; and\(A3\) the quadratic regime—training minimizes Eq\. \([1](https://arxiv.org/html/2607.09202#S3.E1)\) by \(projected\) gradient descent, so motion is confined to the excited subspace while directions inker⁡Σt\\ker\\Sigma\_\{t\}leaveLtL\_\{t\}unchanged\.

### 3\.2The functional and the removability dichotomy

Train taskAAfrom a base pointw0w\_\{0\}to convergence\. Under A3, gradient descent moves only inrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}, so the converged solutionwAw\_\{A\}satisfiesLA​\(wA\)=0L\_\{A\}\(w\_\{A\}\)=0and equalswA∗w\_\{A\}^\{\\ast\}up to an arbitrary component inker⁡ΣA\\ker\\Sigma\_\{A\}\. For loss differences we therefore takewA=wA∗w\_\{A\}=w\_\{A\}^\{\\ast\}\. Now train taskBB, reachingwBw\_\{B\}, and defineΔ=wB−wA∗\\Delta=w\_\{B\}\-w\_\{A\}^\{\\ast\}\.

###### Theorem 1\(Forgetting is interference energy\)\.

Under A1–A3, the forgetting of taskAAcaused by learningBBis

Δ​LA=LA​\(wB\)−LA​\(wA∗\)=12​Δ⊤​ΣA​Δ,Δ=wB−wA∗\.\\Delta L\_\{A\}=L\_\{A\}\(w\_\{B\}\)\-L\_\{A\}\(w\_\{A\}^\{\\ast\}\)=\\tfrac\{1\}\{2\}\\,\\Delta^\{\\top\}\\Sigma\_\{A\}\\,\\Delta,\\qquad\\Delta=w\_\{B\}\-w\_\{A\}^\{\\ast\}\.\(2\)

###### Proof\.

SinceLA​\(wA∗\)=0L\_\{A\}\(w\_\{A\}^\{\\ast\}\)=0by A2, substitutingwB=wA∗\+Δw\_\{B\}=w\_\{A\}^\{\\ast\}\+\\Deltainto Eq\. \([1](https://arxiv.org/html/2607.09202#S3.E1)\) yieldsLA​\(wB\)=12​Δ⊤​ΣA​ΔL\_\{A\}\(w\_\{B\}\)=\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta\. No approximation is involved; the only essential assumption is thatϕ\\phiremains fixed, soΣA\\Sigma\_\{A\}is unchanged before and after learningBB\. ∎

Decompose the update asΔ=Δ∥\+Δ⟂\\Delta=\\Delta\_\{\\parallel\}\+\\Delta\_\{\\perp\}, withΔ⟂∈ker⁡ΣA\\Delta\_\{\\perp\}\\in\\ker\\Sigma\_\{A\}andΔ∥∈range⁡ΣA\\Delta\_\{\\parallel\}\\in\\operatorname\{range\}\\Sigma\_\{A\}\. ThenΔ​LA=12​Δ∥⊤​ΣA​Δ∥\\Delta L\_\{A\}=\\tfrac\{1\}\{2\}\\Delta\_\{\\parallel\}^\{\\top\}\\Sigma\_\{A\}\\Delta\_\{\\parallel\}, so only the component of the update lying in taskAA’s active subspace contributes to forgetting \(Fig\.[2](https://arxiv.org/html/2607.09202#S1.F2), left\)\.

###### Corollary 1\(Removability dichotomy\)\.

An update is lossless forAAif and only ifΔ∈ker⁡ΣA\\Delta\\in\\ker\\Sigma\_\{A\}\. Consequently there exists a way to learnBBwith zero forgetting ofAAifwB∗w\_\{B\}^\{\\ast\}is reachable throughker⁡ΣA\\ker\\Sigma\_\{A\}, i\.e\. ifwB∗−wA∗∈ker⁡ΣAw\_\{B\}^\{\\ast\}\-w\_\{A\}^\{\\ast\}\\in\\ker\\Sigma\_\{A\}\. Even when this fails,LBL\_\{B\}can still be strictly reduced without forgetting wheneverwB∗−wA∗w\_\{B\}^\{\\ast\}\-w\_\{A\}^\{\\ast\}has a non\-zero component inker⁡ΣA∩range⁡ΣB\\ker\\Sigma\_\{A\}\\cap\\operatorname\{range\}\\Sigma\_\{B\}\. Ifrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}andrange⁡ΣB\\operatorname\{range\}\\Sigma\_\{B\}share an active direction on whichwA∗w\_\{A\}^\{\\ast\}andwB∗w\_\{B\}^\{\\ast\}disagree, then no lossless update reacheswB∗w\_\{B\}^\{\\ast\}, and a positive floor remains \(Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)\)\.

###### Proof\.

SinceΣA⪰0\\Sigma\_\{A\}\\succeq 0,Δ⊤​ΣA​Δ=0⇔ΣA1/2​Δ=0⇔Δ∈ker⁡ΣA\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta=0\\Leftrightarrow\\Sigma\_\{A\}^\{1/2\}\\Delta=0\\Leftrightarrow\\Delta\\in\\ker\\Sigma\_\{A\}\. Projected gradient descent onLBL\_\{B\}restricted towA∗\+ker⁡ΣAw\_\{A\}^\{\\ast\}\+\\ker\\Sigma\_\{A\}converges to the minimizer on that affine set\. It reacheswB∗w\_\{B\}^\{\\ast\}ifwB∗−wA∗∈ker⁡ΣAw\_\{B\}^\{\\ast\}\-w\_\{A\}^\{\\ast\}\\in\\ker\\Sigma\_\{A\}, and it strictly reducesLBL\_\{B\}whenever the projection of−∇LB​\(wA∗\)\-\\nabla L\_\{B\}\(w\_\{A\}^\{\\ast\}\)ontoker⁡ΣA\\ker\\Sigma\_\{A\}is non\-zero, equivalently wheneverΣB​\(wB∗−wA∗\)\\Sigma\_\{B\}\(w\_\{B\}^\{\\ast\}\-w\_\{A\}^\{\\ast\}\)has a non\-zero component inker⁡ΣA\\ker\\Sigma\_\{A\}\. In the disjoint\-support case,range⁡ΣB⊆ker⁡ΣA\\operatorname\{range\}\\Sigma\_\{B\}\\subseteq\\ker\\Sigma\_\{A\}, so the entire update is lossless\. ∎

Here, “task support” refers to the column spacerange⁡Σt\\operatorname\{range\}\\Sigma\_\{t\}, that is, the subspace of feature directions activated by tasktt\. Disjoint supports therefore mean orthogonal feature subspaces, not disjoint label sets\.

#### Relation to standard continual\-learning metrics\.

Under squared loss the per\-task termΔ​LA=12​Δ⊤​ΣA​Δ\\Delta L\_\{A\}=\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Deltais exactly the per\-task summand of the Forgetting Measure and the negative of Backward Transfer; positive backward transfer is simplyΔ​LA<0\\Delta L\_\{A\}<0, which the signed gate permits and unconditional projection forbids\. The distortion floor \(Sec\.[3\.4](https://arxiv.org/html/2607.09202#S3.SS4)\) is then a*lower bound*on the average Forgetting Measure achievable by any single\-head method, set by task confusability \(Sec\.[S2](https://arxiv.org/html/2607.09202#A2)\)\. The geometry thus explains the benchmark metrics and bounds them\.

### 3\.3The architecture\-general identity

Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)assumes frozen features\. In end\-to\-end training, feature drift degrades the frozen\-curvature prediction, especially in deeper networks and at larger step sizes\. The natural generalization is to replaceΣA\\Sigma\_\{A\}by the curvature ofLAL\_\{A\}averaged along the displacement fromwAw\_\{A\}towBw\_\{B\}\.

###### Theorem 2\(Architecture\-general forgetting identity\)\.

LetLAL\_\{A\}be any twice\-differentiable task\-AAloss, and let training onBBmove the parameters fromwAw\_\{A\}towBw\_\{B\}\. WritingΔ=wB−wA\\Delta=w\_\{B\}\-w\_\{A\},

Δ​LA=∇LA​\(wA\)⊤​Δ\+12​Δ⊤​H¯A​Δ,H¯A=2​∫01\(1−t\)​∇2LA​\(wA\+t​Δ\)​𝑑t,\\Delta L\_\{A\}=\\nabla L\_\{A\}\(w\_\{A\}\)^\{\\top\}\\Delta\+\\tfrac\{1\}\{2\}\\,\\Delta^\{\\top\}\\bar\{H\}\_\{A\}\\,\\Delta,\\qquad\\bar\{H\}\_\{A\}=2\\\!\\int\_\{0\}^\{1\}\(1\-t\)\\,\\nabla^\{2\}L\_\{A\}\(w\_\{A\}\+t\\Delta\)\\,dt,\(3\)whereH¯A\\bar\{H\}\_\{A\}is the Hessian ofLAL\_\{A\}averaged along the segmentwA→wBw\_\{A\}\\\!\\to\\\!w\_\{B\}\. IfwAw\_\{A\}is a stationary point, then∇LA​\(wA\)=0\\nabla L\_\{A\}\(w\_\{A\}\)=0andΔ​LA=12​Δ⊤​H¯A​Δ\\Delta L\_\{A\}=\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\bar\{H\}\_\{A\}\\Delta\.

*Intuitively*, a second\-order Taylor expansion ofLAL\_\{A\}along the segmentwA→wBw\_\{A\}\\\!\\to\\\!w\_\{B\}, with exact integral remainder, produces the gradient term and the path\-averaged Hessian \(full derivation in Sec\.[S1](https://arxiv.org/html/2607.09202#A1)\)\.

In the constant\-curvate case∇2LA≡ΣA\\nabla^\{2\}L\_\{A\}\\equiv\\Sigma\_\{A\}, the path\-averaged HessianH¯A\\bar\{H\}\_\{A\}collapses to the fixed\-curvature matrixΣA\\Sigma\_\{A\}by Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)\. For squared loss,∇2LA​\(w\)=𝔼𝒟A​\[Jw​Jw⊤\]\+R​\(w\)\\nabla^\{2\}L\_\{A\}\(w\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{A\}\}\[J\_\{w\}J\_\{w\}^\{\\top\}\]\+R\(w\), soH¯A\\bar\{H\}\_\{A\}is a path\-averaged Gauss–Newton geometry, whereJw=∂f​\(x;w\)/∂wJ\_\{w\}=\\partial f\(x;w\)/\\partial wis the parameter Jacobian andR​\(w\)R\(w\)is a residual\-curvature term \(Fig\.[3](https://arxiv.org/html/2607.09202#S3.F3)a\)\. The removability, merging, and floor results therefore extend by replacingΣA\\Sigma\_\{A\}withH¯A\\bar\{H\}\_\{A\}\. In practice,H¯A\\bar\{H\}\_\{A\}can be estimated from a small number of evaluations ofLAL\_\{A\}along the segment, without forming the Hessian explicitly\. This principle motivates igfa to track the running Gauss–Newton subspace𝔼​\[Jw​Jw⊤\]\\mathbb\{E\}\[J\_\{w\}J\_\{w\}^\{\\top\}\]\.

The generalization in fact reaches its cleanest form not in parameter space but in*function*space, where the geometry is exact for arbitrary architectures\.

###### Theorem 3\(Function\-space interference identity\)\.

LetfA,fBf\_\{A\},f\_\{B\}be the functions realized before and after learningBB, by any architecture and training procedure, and letLA​\(f\)=12​𝔼𝒟A​\(f​\(x\)−y\)2L\_\{A\}\(f\)=\\tfrac\{1\}\{2\}\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{A\}\}\(f\(x\)\-y\)^\{2\}\. WritingΔ​f=fB−fA\\Delta\\\!f=f\_\{B\}\-f\_\{A\}andrA=fA−yr\_\{A\}=f\_\{A\}\-y,

Δ​LA=𝔼𝒟A​\[Δ​f​rA\]\+12​𝔼𝒟A​\[Δ​f2\]\.\\Delta L\_\{A\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{A\}\}\\\!\\big\[\\Delta\\\!f\\;r\_\{A\}\\big\]\+\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{A\}\}\\\!\\big\[\\Delta\\\!f^\{\\,2\}\\big\]\.\(4\)IfAAis realizable and trained to convergence \(rA=0r\_\{A\}=0\), forgetting equals the function\-space interference energy12​‖Δ​f‖L2​\(𝒟A\)2\\tfrac\{1\}\{2\}\\\|\\Delta\\\!f\\\|\_\{L^\{2\}\(\\mathcal\{D\}\_\{A\}\)\}^\{2\}\. LearningBBis lossless iffΔ​f=0\\Delta\\\!f=0onsupp​𝒟A\\mathrm\{supp\}\\,\\mathcal\{D\}\_\{A\}; and if the training velocity stays in the kernel of the current Gauss–Newton metric,LAL\_\{A\}is conserved exactly in continuous time\.

Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)is the linearized version of Eq\. \([4](https://arxiv.org/html/2607.09202#S3.E4)\): under A1,Δ​f=Δ⊤​ϕ\\Delta\\\!f=\\Delta^\{\\top\}\\phirecovers12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta\. The dichotomy, the distortion floor \(Sec\.[3\.4](https://arxiv.org/html/2607.09202#S3.SS4)\), and the retention guarantee thus hold with A1 removed entirely, with feature support replaced by input support; A1 is the regime in which the geometry becomes computable in closed form and finite steps stay exact\. Nor is the geometry a quadratic\-loss artifact: for the whole canonical \(Bregman\) loss family—softmax cross\-entropy included—the identity holds exactly with the energy replaced by a Bregman divergence\. On a converged task, cross\-entropy forgetting*equals*the expected Kullback–Leibler divergence between the old and new predictive distributions on that task’s data, and the distortion floor becomes a density\-weighted Jensen–Shannon divergence between the tasks’ conditionals, in closed form \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\. The full statement, proofs, and validations on deep networks—machine\-precision agreement at depths where the frozen\-curvature predictor degrades tor≈0\.4r\\approx 0\.4—are in Sec\.[S15](https://arxiv.org/html/2607.09202#A15), along with two routes to finite\-step parameter\-space exactness \(activation\-region certificates for ReLU networks and predictor–corrector retraction\)\.

###### Proposition 1\(Estimation\-error budget\)\.

Protection with an*estimated*basisU^\\hat\{U\}whose largest principal angle torange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}isϵ\\epsilonis not exactly lossless: under gradient noise of varianceσ2\\sigma^\{2\}and step sizeη\\etaoverttsteps, the induced forgetting obeys the diffusion law𝔼​\[Δ​LA\]=12​η2​σ2​t​r​sin2⁡ϵ\+o​\(sin2⁡ϵ\)\\mathbb\{E\}\[\\Delta L\_\{A\}\]=\\tfrac\{1\}\{2\}\\eta^\{2\}\\sigma^\{2\}t\\,r\\sin^\{2\}\\epsilon\+o\(\\sin^\{2\}\\epsilon\), vanishing asϵ→0\\epsilon\\to 0\.

The method is therefore a*controlled*approximation of the exact identity with the retention loss governed by a single measurable quantity \(basis errorϵ\\epsilon\), which the recursive tracker drives down and the drift velocity monitors \(validated inevaluate\_upgrades\.py\. This law inverts to a retention\-aware step\-size bound, Sec\.[S14](https://arxiv.org/html/2607.09202#A14)\) and isolates the two residual gaps for end\-to\-end deployment as quantified basis error and finite step size\.

### 3\.4Optimal Merging and the Distortion Floor

RepresentKKtask solutions asΔt=wt∗−w0\\Delta\_\{t\}=w\_\{t\}^\{\\ast\}\-w\_\{0\}from a shared basew0w\_\{0\}, and merge them asθ∗=w0\+∑tΔt\\theta^\{\\ast\}=w\_\{0\}\+\\sum\_\{t\}\\Delta\_\{t\}\. Substituting into Eq\. \([1](https://arxiv.org/html/2607.09202#S3.E1)\), the loss incurred on taskttis

Lt​\(θ∗\)=12​\(∑t′≠tΔt′\)⊤​Σt​\(∑t′≠tΔt′\),L\_\{t\}\(\\theta^\{\\ast\}\)=\\tfrac\{1\}\{2\}\\Big\(\\textstyle\\sum\_\{t^\{\\prime\}\\neq t\}\\Delta\_\{t^\{\\prime\}\}\\Big\)^\{\\\!\\top\}\\Sigma\_\{t\}\\Big\(\\textstyle\\sum\_\{t^\{\\prime\}\\neq t\}\\Delta\_\{t^\{\\prime\}\}\\Big\),\(5\)that is, the interference energy of the other task vectors measured in tasktt’s geometry\.

###### Theorem 4\(Optimal merging isΣ\\Sigma\-orthogonalization\)\.

The total merge loss∑tLt​\(θ∗\)\\sum\_\{t\}L\_\{t\}\(\\theta^\{\\ast\}\)vanishes whenever the task vectors are pairwiseΣ\\Sigma\-orthogonal, i\.e\.Δt′∈ker⁡Σt\\Delta\_\{t^\{\\prime\}\}\\in\\ker\\Sigma\_\{t\}for allt′≠tt^\{\\prime\}\\neq t; this condition is also generically necessary\. When it fails, the minimum total loss over shared heads is attained atθJ=\(∑tΣt\)\+​\(∑tΣt​wt∗\)\\theta\_\{J\}=\\big\(\\sum\_\{t\}\\Sigma\_\{t\}\\big\)^\{\+\}\\big\(\\sum\_\{t\}\\Sigma\_\{t\}w\_\{t\}^\{\\ast\}\\big\), and for two tasks

wJ=\(ΣA\+ΣB\)\+​\(ΣA​wA∗\+ΣB​wB∗\)\.w\_\{J\}=\(\\Sigma\_\{A\}\+\\Sigma\_\{B\}\)^\{\+\}\(\\Sigma\_\{A\}w\_\{A\}^\{\\ast\}\+\\Sigma\_\{B\}w\_\{B\}^\{\\ast\}\)\.\(6\)The residual𝒟:=∑tLt​\(θJ\)\\mathcal\{D\}:=\\sum\_\{t\}L\_\{t\}\(\\theta\_\{J\}\)is the*distortion floor*\.

###### Proof\.

Since eachLt​\(θ∗\)≥0L\_\{t\}\(\\theta^\{\\ast\}\)\\geq 0, the sum vanishes iff every term vanishes, i\.e\.Σt1/2​∑t′≠tΔt′=0\\Sigma\_\{t\}^\{1/2\}\\sum\_\{t^\{\\prime\}\\neq t\}\\Delta\_\{t^\{\\prime\}\}=0\. A sufficient, and generically necessary, condition isΔt′∈ker⁡Σt\\Delta\_\{t^\{\\prime\}\}\\in\\ker\\Sigma\_\{t\}for allt′≠tt^\{\\prime\}\\neq t\. The minimizer of∑tLt​\(θ\)=12​∑t\(θ−wt∗\)⊤​Σt​\(θ−wt∗\)\\sum\_\{t\}L\_\{t\}\(\\theta\)=\\tfrac\{1\}\{2\}\\sum\_\{t\}\(\\theta\-w\_\{t\}^\{\\ast\}\)^\{\\top\}\\Sigma\_\{t\}\(\\theta\-w\_\{t\}^\{\\ast\}\)satisfies∑tΣt​\(θ−wt∗\)=0\\sum\_\{t\}\\Sigma\_\{t\}\(\\theta\-w\_\{t\}^\{\\ast\}\)=0, which givesθJ\\theta\_\{J\}; Eq\. \([6](https://arxiv.org/html/2607.09202#S3.E6)\) is the two\-task case\. ∎

The relevant inner product for merging is therefore the task\-dependentΣt\\Sigma\_\{t\}\-metric \(Fig\.[3](https://arxiv.org/html/2607.09202#S3.F3)b\); Euclidean orthogonality is the white\-feature special caseΣt∝I\\Sigma\_\{t\}\\propto I\. The same residual𝒟\\mathcal\{D\}also governs the online continual\-learning setting, so merging and continual learning optimize the same functional under batch and streaming schedules\.

#### Residual distortion under non\-orthogonal task geometry\.

Optimal merging needsΣ\\Sigma\-orthogonality; when the supports force it to fail, the residual𝒟\\mathcal\{D\}of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)quantifies the unavoidable error of task ambiguity\. Consider two tasks as a single problem in which each input carries a hidden labelT∈\{A,B\}T\\in\\\{A,B\\\}and a shared head must predict the target of the hidden task\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x3.png)Figure 4:The distortion floor and its information\-theoretic reading\.*\(a\)*A floor appears only on a shared active directionu∈range⁡ΣA∩range⁡ΣBu\\in\\operatorname\{range\}\\Sigma\_\{A\}\\cap\\operatorname\{range\}\\Sigma\_\{B\}on which the targets disagree byδ\\delta, with𝒟≥14​σu​δ2\\mathcal\{D\}\\geq\\tfrac\{1\}\{4\}\\sigma\_\{u\}\\delta^\{2\}\(Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)\)\.*\(b\)*The Bayes single\-head excess risk𝒟⋆\\mathcal\{D\}^\{\\star\}factors into the geometric target gap\(ϕ⊤​δ\)2\(\\phi^\{\\top\}\\delta\)^\{2\}—the displacementδ\\deltain \(a\)—and the task\-confusability factorVar⁡\(T∣ϕ\)\\operatorname\{Var\}\(T\\\!\\mid\\\!\\phi\), which is two\-sidedly controlled by the conditional entropyH​\(T∣ϕ\)H\(T\\\!\\mid\\\!\\phi\)\(Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6); band\)\.###### Theorem 5\(Distortion floor and inference limit\)\.

Suppose the tasks share an active directionuu, with‖u‖=1\\\|u\\\|=1andu∈range⁡ΣA∩range⁡ΣBu\\in\\operatorname\{range\}\\Sigma\_\{A\}\\cap\\operatorname\{range\}\\Sigma\_\{B\}, and letδ=u⊤​\(wB∗−wA∗\)\\delta=u^\{\\top\}\(w\_\{B\}^\{\\ast\}\-w\_\{A\}^\{\\ast\}\)\. Then the distortion floor obeys the lower bound

𝒟≥14​σu​δ2,σu=\(u⊤​\(ΣA−1\+ΣB−1\)​u\)−1,\\mathcal\{D\}\\ \\geq\\ \\tfrac\{1\}\{4\}\\,\\sigma\_\{u\}\\,\\delta^\{2\},\\qquad\\sigma\_\{u\}=\\big\(u^\{\\top\}\(\\Sigma\_\{A\}^\{\-1\}\+\\Sigma\_\{B\}^\{\-1\}\)u\\big\)^\{\-1\},\(7\)whereΣA−1\\Sigma\_\{A\}^\{\-1\}andΣB−1\\Sigma\_\{B\}^\{\-1\}denote the inverses ofΣA\\Sigma\_\{A\}andΣB\\Sigma\_\{B\}restricted to the shared active subspaceS=range⁡ΣA∩range⁡ΣBS=\\operatorname\{range\}\\Sigma\_\{A\}\\cap\\operatorname\{range\}\\Sigma\_\{B\}\. On the shared support, the excess Bayes risk of the Bayes\-optimal single head over the task oracle equals the merge residual\.

Restricting the merge to the shared directionuureduces it to a scalar two\-point problem whoseΣ\\Sigma\-weighted optimum leaves residual14​σu​δ2\\tfrac\{1\}\{4\}\\sigma\_\{u\}\\delta^\{2\}; summing over shared directions and identifying the Bayes single head with the posterior mean gives the bound and the inference claim \(Sec\.[S1](https://arxiv.org/html/2607.09202#A1)\)\.

Support overlap is therefore the common limit to bothΣ\\Sigma\-orthogonal merging and task inference; a positive floor appears only on shared active directions where the tasks also disagree\. This yields the geometric counterpart of the task\- versus class\-incremental distinction\[[16](https://arxiv.org/html/2607.09202#bib.bib16)\]: disjoint supports make the task identity effectively free, whereas overlapping supports can induce a non\-zero floor when the tasks disagree on shared active directions\.

###### Theorem 6\(Information–estimation identity for the floor\)\.

LetT∼Unif​\{A,B\}T\\sim\\mathrm\{Unif\}\\\{A,B\\\},ϕ∣T∼𝒟T\\phi\\mid T\\sim\\mathcal\{D\}\_\{T\},y=ϕ⊤​wT∗\+εy=\\phi^\{\\top\}w\_\{T\}^\{\\ast\}\+\\varepsilon, withε⟂\(T,ϕ\)\\varepsilon\\perp\(T,\\phi\),𝔼​ε=0\\mathbb\{E\}\\varepsilon=0, and𝔼​ε2=σ2\\mathbb\{E\}\\varepsilon^\{2\}=\\sigma^\{2\}\. Writingη​\(ϕ\)=Pr⁡\(T=A∣ϕ\)\\eta\(\\phi\)=\\Pr\(T\{=\}A\\mid\\phi\)andδ=wA∗−wB∗\\delta=w\_\{A\}^\{\\ast\}\-w\_\{B\}^\{\\ast\}, the excess Bayes risk of a single shared head over the task oracle is

𝒟⋆=12​𝔼ϕ​\[Var⁡\(T∣ϕ\)​\(ϕ⊤​δ\)2\]=12​𝔼ϕ​\[η​\(1−η\)​\(ϕ⊤​δ\)2\]\.\\mathcal\{D\}^\{\\star\}=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\phi\}\\\!\\big\[\\operatorname\{Var\}\(T\\\!\\mid\\\!\\phi\)\\,\(\\phi^\{\\top\}\\delta\)^\{2\}\\big\]=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\phi\}\\\!\\big\[\\eta\(1\-\\eta\)\\,\(\\phi^\{\\top\}\\delta\)^\{2\}\\big\]\.\(8\)Withhb​\(p\)=−p​ln⁡p−\(1−p\)​ln⁡\(1−p\)h\_\{b\}\(p\)=\-p\\ln p\-\(1\-p\)\\ln\(1\-p\)the binary entropy in nats, this implies the two\-sided bounds

18​𝔼ϕ​\[\(ϕ⊤​δ\)2​hb​\(η\)2\]≤𝒟⋆≤14​𝔼ϕ​\[\(ϕ⊤​δ\)2​hb​\(η\)\]\.\\tfrac\{1\}\{8\}\\,\\mathbb\{E\}\_\{\\phi\}\\\!\\big\[\(\\phi^\{\\top\}\\delta\)^\{2\}\\,h\_\{b\}\(\\eta\)^\{2\}\\big\]\\ \\leq\\ \\mathcal\{D\}^\{\\star\}\\ \\leq\\ \\tfrac\{1\}\{4\}\\,\\mathbb\{E\}\_\{\\phi\}\\\!\\big\[\(\\phi^\{\\top\}\\delta\)^\{2\}\\,h\_\{b\}\(\\eta\)\\big\]\.\(9\)At a fixed non\-zero target\-gap profile\(ϕ⊤​δ\)2\(\\phi^\{\\top\}\\delta\)^\{2\}, the floor increases withH​\(T∣ϕ\)H\(T\\\!\\mid\\\!\\phi\)and decreases withI​\(T;ϕ\)I\(T;\\phi\); in particular it vanishes when the label is fully identifiable,I​\(T;ϕ\)=H​\(T\)I\(T;\\phi\)=H\(T\)\. The merge floor of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)is the linear\-head restriction, so𝒟⋆≤𝒟merge\\mathcal\{D\}^\{\\star\}\\leq\\mathcal\{D\}\_\{\\mathrm\{merge\}\}, with equality at both extremes\.

The law of total variance splits the Bayes single\-head risk into the oracle noise plus the posterior\-variance\-weighted target gapη​\(1−η\)​\(ϕ⊤​δ\)2\\eta\(1\-\\eta\)\(\\phi^\{\\top\}\\delta\)^\{2\}, giving Eq\. \([8](https://arxiv.org/html/2607.09202#S3.E8)\); the entropy sandwich follows from the pointwise bound14​hb​\(p\)2≤p​\(1−p\)≤12​hb​\(p\)\\tfrac\{1\}\{4\}h\_\{b\}\(p\)^\{2\}\\leq p\(1\-p\)\\leq\\tfrac\{1\}\{2\}h\_\{b\}\(p\)\(full proof in Sec\.[S1](https://arxiv.org/html/2607.09202#A1)\)\.

Equation \([8](https://arxiv.org/html/2607.09202#S3.E8)\) is exact for arbitrary feature distributions and noise models\. More generally, the floor depends jointly on task confusability and target disagreement \(Fig\.[4](https://arxiv.org/html/2607.09202#S3.F4)b\), so no exact proportionality𝒟⋆∝I​\(T;ϕ\)\\mathcal\{D\}^\{\\star\}\\propto I\(T;\\phi\)holds: the floor vanishes as the target gapδ→0\\delta\\to 0even at fixedI​\(T;ϕ\)\>0I\(T;\\phi\)\>0, so no tight weight\-free constant can relate the floor to mutual information\. The same identity extends toKKtasks with arbitrary priors, with corresponding multiclass entropy bounds in terms of the Gini impurity \(Sec\.[S2](https://arxiv.org/html/2607.09202#A2)\); both the binary identity and the multiclass bounds are verified numerically there\. Under cross\-entropy the sandwich closes to an*identity*: for any number of tasks, the floor of a single predictor equalsT⋅I​\(T;Y∣X\)T\\cdot I\(T;Y\\mid X\)—the conditional mutual information between task identity and label given the input—and removability becomes the statementY⟂T∣XY\\perp T\\mid X\(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\. Under squared loss, Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)also recovers the standard per\-task forgetting term used in the Forgetting Measure and Backward Transfer; Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6)therefore gives a lower bound on the average forgetting achievable by any single\-head method in terms of task confusability\.

## 4Method: interference\-gated allocation

The functional above implies a family of interference\-gated functional allocation \(igfa\) methods that control*where*an update is allowed to act, parameterized by schedule \(offline or online\) and by the share\-versus\-orthogonalize gate \(Fig\.[2](https://arxiv.org/html/2607.09202#S1.F2), right\)\.

#### Offline: optimal merge\.

GivenKKtrained task vectors\{Δt\}\\\{\\Delta\_\{t\}\\\}and per\-task second moments\{Σt\}\\\{\\Sigma\_\{t\}\\\}, estimated from each task’s activations or probes, solve for theΣ\\Sigma\-orthogonalized vectors and coefficients that minimize the total interference in Eq\. \([5](https://arxiv.org/html/2607.09202#S3.E5)\)\. This is a generalized eigenvalue or Procrustes problem, whose two\-task case is Eq\. \([6](https://arxiv.org/html/2607.09202#S3.E6)\), and provides a principled replacement for weight\-space task arithmetic\.

#### Online: igfa\.

As tasks stream, maintain a low\-rank orthonormal summaryUUof the occupied function subspace, given by the top directions of the accumulatedΣt\\Sigma\_\{t\}under streaming PCA\.UUis the only persistent state: there is no exemplar buffer and no per\-weight Fisher matrix\. For each new\-task gradientgg:

1. 1\.Splitg=g∥\+g⟂g=g\_\{\\parallel\}\+g\_\{\\perp\}, withg∥=U​U⊤​gg\_\{\\parallel\}=UU^\{\\top\}gthe component in the occupied subspace\.
2. 2\.For each occupied directionuutouched byg∥g\_\{\\parallel\}, estimate the signed similarityssbetween the new task’s target direction anduu\.
3. 3\.Gate the update: ifs\>s∗s\>s^\{\\ast\}\(aligned, transfer\), keepg∥g\_\{\\parallel\}and share the direction; ifs<s∗s<s^\{\\ast\}\(conflict\), removeg∥g\_\{\\parallel\}and step alongg⟂g\_\{\\perp\}, protecting the old task’s function by construction\.
4. 4\.Updatewwwith the gated gradient, then growUUwith the novel directions ing⟂g\_\{\\perp\}by streaming PCA, rank\-capping and overwriting the least important direction when full—rate–distortion\-graceful forgetting\.

Because the old\-task function is first\-order invariant along orthogonalized directions, retention is structural rather than penalty\-based\. Because aligned directions are shared rather than suppressed, transfer is preserved, which unconditional projection removes \(see Algorithm 1\)\.

Algorithm 1: the interference\-ledger loop \(measure→\\toattribute→\\toact\), with igfa as its default control instance\. State \(one object\):tracked second momentΣ^\\widehat\{\\Sigma\}\(or its Frequent\-Directions sketch\); per\-task bases\{Bt\}\\\{B\_\{t\}\\\}with rank capkmaxk\_\{\\max\}; held\-out micro\-cache gradients\{∇Lu\}\\\{\\nabla L\_\{u\}\\\}\. Every quantity below is a read\-out of this state\. For each tasknn\(gradient streamgg, feature second\-moment estimateΣ^n\\widehat\{\\Sigma\}\_\{n\};Bn←B\_\{n\}\\leftarrowtop\-rreigenvectors, streaming PCA\):1\.Measure:overlapssim​\(Bt,Bn\)=‖Bt⊤​Bn‖F2/r\\mathrm\{sim\}\(B\_\{t\},B\_\{n\}\)=\\\|B\_\{t\}^\{\\top\}B\_\{n\}\\\|\_\{F\}^\{2\}/r, share density, predicted floor, occupied rank, drift velocity, OOD ratio, estimator confidence\.2\.Attributethe predicted loss: floor \(irreducible\)//capacity \(rank cap binding\)//control \(the rest\)—each term is measurable and has exactly one remedy \(Sec\.[6](https://arxiv.org/html/2607.09202#S6)\)\.3\.Act:*skip*if OOD;*expand or drop*the smallest\-eigenvalue direction if the cap binds \(rate–distortion\-graceful\);*replay*where recoverable excess per exemplar is largest; otherwise thegated update: per minibatch solve the monotone\-retention QP of Corollary[2](https://arxiv.org/html/2607.09202#Thmcorollary2),v=argmin⁡‖v\+g‖2v=\\operatorname\{argmin\}\\\|v\+g\\\|^\{2\}subject to the chosen constraint rows, and stepw←w\+η​vw\\leftarrow w\+\\eta\\,v\. igfa’s default rows: equality constraints from every past basis withsim​\(Bt,Bn\)<s∗\\mathrm\{sim\}\(B\_\{t\},B\_\{n\}\)<s^\{\\ast\}\(conflicting\), aligned tasks left shared—equivalentlyg←\(I−Q​Q⊤\)​gg\\leftarrow\(I\-QQ^\{\\top\}\)gwithQ=orthQ=\\mathrm\{orth\}of the conflicting bases\.4\.Update state:fold the batch intoΣ^\\widehat\{\\Sigma\}; grow the basis store; refresh micro\-caches;*defer*\(protect\-all\) while confidence is low\.Inference:single forward pass throughww\(plus a task router only in the class\-incremental case; Sec\.[3\.4](https://arxiv.org/html/2607.09202#S3.SS4.SSS0.Px1)\)\.

#### One gate, many methods\.

Step 3’s QP is not one design choice among many: every protection rule in this literature is the*same*program under a different constraint set, and the reference implementation is a single function \(evaluate\_unified\_gate\.py, all recoveries machine\-checked\):

The recoveries are exact rather than analogies: the closed\-form sign gate andogdagree with the QP to10−1510^\{\-15\}, and igfa’s full training trajectory coincides with the overlap\-filtered\-equality instance step for step \(10−1510^\{\-15\}over five tasks\)\. The methods differ only in which rows they enforce and how well those rows are estimated, which is precisely the axis the scale experiments measure \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\)\.

#### Complexity, and one state object\.

The only persistent state is the triple of Algorithm 1—the tracked second moment \(or its sketch\), the per\-task bases, and the held\-out micro\-caches—giving𝒪​\(d​k\)\\mathcal\{O\}\(dk\)memory withk≤kmaxk\\leq k\_\{\\max\}, independent of the number of stored examples and of dataset size\. This contrasts with𝒪​\(\|ℬ\|​d\)\\mathcal\{O\}\(\|\\mathcal\{B\}\|d\)memory for a replay bufferℬ\\mathcal\{B\}, and with𝒪​\(d\)\\mathcal\{O\}\(d\)to𝒪​\(d2\)\\mathcal\{O\}\(d^\{2\}\)memory for diagonal\-to\-full Fisher methods\. Per update, the projection costs𝒪​\(d​k\)\\mathcal\{O\}\(dk\), essentially akk\-dimensional inner product and subtraction, and is the dominant overhead\. The per\-task basis update is a thin eigendecomposition ofΣ^n\\widehat\{\\Sigma\}\_\{n\}, costing𝒪​\(d​r2\)\\mathcal\{O\}\(dr^\{2\}\)amortized once per task\. In a frozen\-backbone overlay,ddis the adapter width rather than the full model width, soUUremains small in absolute terms\. The rank capkmaxk\_\{\\max\}is the capacity budget of Sec\.[5\.3](https://arxiv.org/html/2607.09202#S5.SS3): when it binds, the discarded direction is the least excited one, which bounds the induced forgetting by its eigenvalue\. At scale, the covariance tracking itself can be replaced by an𝒪​\(r​d\)\\mathcal\{O\}\(rd\)Frequent\-Directions sketch with nod×dd\\times deigendecomposition \(Sec\.[S9](https://arxiv.org/html/2607.09202#A9)\)\. Because every quantity in this paper—overlaps ands∗s^\{\\ast\}, the QP constraint rows, the quadratic and KL/JSD floors, the OOD ratio, drift velocity, and capacity occupancy—is a read\-out of this one state object, its failure modes are unified rather than method\-specific: a stale estimate announces itself through the drift velocity, small\-sample floor estimates are bias\-corrected and*abstain*when a null test cannot rule out zero conflict, and constraint anchors must be held\-out and adequately sized, since a train\-anchored gate provably protects memorized noise as if it were knowledge \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\.

#### Relation toogd/gpm\.

Settings∗=∞s^\{\\ast\}=\\infty, so that all past subspaces are protected, recovers orthogonal projection\[[7](https://arxiv.org/html/2607.09202#bib.bib7),[8](https://arxiv.org/html/2607.09202#bib.bib8)\]exactly\. igfa is therefore a strict generalization: it preserves the protective behavior ofogd/gpmin the conflict regime, but restores sharing when tasks are aligned\. Offline, applying the same primitive to a batch of pre\-trained task vectors yields theΣ\\Sigma\-orthogonal merge of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\. Online, the thresholds∗s^\{\\ast\}is recovered by a validation\-greedy rule \(Sec\.[S6](https://arxiv.org/html/2607.09202#A6)\)\.

#### Why a shared head with a gate, not one head per task\.

The simplest way to avoid interference is to assign each task its own head and never share\. Strict isolation, however, defeats the purpose of continual learning: a network that isolates taskBBin a separate head cannot carry what it learned there back to taskAA, so forward and backward transfer become impossible\. igfa instead keeps a single shared head and resolves interference inside it with the similarity gate\. The gate is the rule that permits sharing when similarity exceedss∗s^\{\\ast\}and orthogonalizes only when sharing would create conflict, rather than isolating everything by default\.

#### Online operation under feature drift\.

In a stream, one cannot look ahead to the segmentwA→wBw\_\{A\}\\\!\\to\\\!w\_\{B\}that definesH¯A\\bar\{H\}\_\{A\}in Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)\. The online surrogate is therefore to track the geometry as it drifts, maintaining a running estimate of the Gauss–Newton second moment by the decayed update

C←γ​C\+𝔼batch​\[Jw​Jw⊤\],C\\leftarrow\\gamma C\+\\mathbb\{E\}\_\{\\text\{batch\}\}\[J\_\{w\}J\_\{w\}^\{\\top\}\],\(10\)and takingUUto be its significant eigenvectors via incremental or streaming PCA\. This is the self\-correcting counterpart of a stale protected subspace fixed once at task onset, as inogd/gpm\. The distinction matters under drift: a stale subspace treats drifted copies of the same direction as new ones, so its rank keeps growing until it fills the parameter space, whereas the recursive estimate recognizes them as one low\-rank subspace and stays bounded \(Sec\.[5\.7](https://arxiv.org/html/2607.09202#S5.SS7)\)\. This is the same failure mode reported for subspace\-projection merging in large language models\[[31](https://arxiv.org/html/2607.09202#bib.bib31)\], and the proposed solution is the recursive Gauss–Newton update\.

#### Regimes\.

igfa instantiates naturally in three deployment settings: \(i\)*from scratch*, acting on the full parameter matrix; \(ii\)*frozen\-backbone\+\+PEFT*, modulating only adapter, LoRA, or head parameters, where Assumption A1 and hence the functional are exact; and \(iii\)*overlay\-native*operation, embedding subspace tracking inside a read\-write decomposition over a frozen transformer, which is the natural regime for replay\-free continual pretraining\.

#### When to use igfa rather than projection, replay, or a Fisher penalty\.

The choice is itself a measurable pre\-flight decision, from pairwise subspace overlap and target agreement \(Sec\.[S8](https://arxiv.org/html/2607.09202#A8)\)\.*Disjoint supports*⇒\\Rightarrowstructural allocation is lossless, so igfa reduces toogdand neither replay nor a Fisher penalty is needed\.*Overlapping supports, aligned targets*⇒\\Rightarrowshare, which the signed gate grants and unconditional projection forfeits\.*Overlapping supports, conflicting targets*⇒\\Rightarrowan irreducible floor, and the only decision is where its cost lands: forgetting \(naive\), deferred fit \(projection\), or stored data \(replay\)\.*Cumulative rank near capacity*⇒\\Rightarrowadd capacity, replay, or graceful overwriting\. The gate’s marginal value over projection is organized by the stream’s share density and is significant where overlaps straddle the threshold \(Sec\.[5\.6](https://arxiv.org/html/2607.09202#S5.SS6)\)\. Two operating assumptions bound this framing and are made explicit here rather than deferred: task boundaries are treated as known \(relaxed by the recursive tracker and the velocity detector of Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\), and for language the retention mechanism transfers but the transfer\-recovering similarity signal is the threshold\-free functional gate of Sec\.[S15](https://arxiv.org/html/2607.09202#A15)rather than input geometry\.

## 5Experiments

We organize the experiments around the paper’s central claims\. Sections[5\.1](https://arxiv.org/html/2607.09202#S5.SS1)–[5\.3](https://arxiv.org/html/2607.09202#S5.SS3)validate the theory in the linear\-on\-features regime \(d=24d=24, random\-subspace tasks, unit\-norm targets\), Section[5\.4](https://arxiv.org/html/2607.09202#S5.SS4)tests the architecture\-general correction under feature drift, Section[5\.5](https://arxiv.org/html/2607.09202#S5.SS5)evaluates igfa on real benchmarks in both the dissimilar\-task and similar\-task regimes, Section[5\.6](https://arxiv.org/html/2607.09202#S5.SS6)scales the comparison to a frozen ViT backbone, and Section[5\.7](https://arxiv.org/html/2607.09202#S5.SS7)contrasts recursive subspace tracking with a stale protected subspace under drift\. Experiments on additional benchmarks, scale, and the algorithmic refinements are reported in full in the Supporting Information\. All figures regenerate fromexperiments\.pyandexperiments\_rev\.pywith fixed seeds\.

### 5\.1The functional is exactly the forgetting

We draw random task pairs, learnAAthenBB, and compare the measured forgetting ofAAagainst the predicted interference energy12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta\(Fig\.[5](https://arxiv.org/html/2607.09202#S5.F5)\)\. For a linear head the two coincide to machine precision over160160pairs, and for a one\-hidden\-layer network the prediction formed from the last\-layer feature covariance still tracks the measured forgetting with Pearsonr\>0\.9999r\>0\.9999\. In the frozen\-feature regime, forgetting is the same quantity as interference energy\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x4.png)Figure 5:The interference functional*is*the forgetting\. Measured backward\-forgetting of taskAAafter learning taskBBagainst the predicted energy12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta\.*\(a\)*A linear head on frozen features: the identity is exact to machine precision\.*\(b\)*A nonlinear network, with the prediction formed from the last\-layer feature covariance: first\-order agreement, Pearsonr\>0\.9999r\>0\.9999\. The dashed line isy=xy=x\.
### 5\.2Lossless allocation, relocation of cost, and the floor

On disjoint supports, orthogonal allocation is lossless: learningBBinker⁡ΣA\\ker\\Sigma\_\{A\}leavesLA<10−17L\_\{A\}<10^\{\-17\}while still drivingLB→0L\_\{B\}\\to 0, exactly as Corollary[1](https://arxiv.org/html/2607.09202#Thmcorollary1)predicts\. On overlapping supports, the cost cannot be removed, only relocated: in the conflicting case, naive descent forgetsAAbyΔ​LA=13\.94\\Delta L\_\{A\}=13\.94, while orthogonal allocation preservesAAat<10−17<10^\{\-17\}and defers the same13\.9413\.94as residual onBB\(Fig\.[6](https://arxiv.org/html/2607.09202#S5.F6), left\)\. Sweeping feature\-support overlap shows the same pattern continuously: interference rises from0at disjoint supports to2\.02\.0at full overlap, while the optimal\-merge floor sits at exactly half \(Fig\.[6](https://arxiv.org/html/2607.09202#S5.F6), right\), confirming that overlap creates an irreducible distortion floor and allocation changes where the cost lands rather than its total amount\. This relocation is fundamental because the distortion floor𝒟\\mathcal\{D\}is a lower bound \(Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\) that no single shared model can improve on\. On overlapping supports, cost can be moved from irreversible forgetting of the old task to recoverable deferred fitting of the new one\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x5.png)Figure 6:Allocation and the distortion floor\.*Left:*on partially shared features with a conflict on the shared directions, naive descent destroys taskAA\(red rises to13\.9413\.94\) while allocation preserves it \(<10−17<10^\{\-17\}, green flat\) and converts that same13\.9413\.94into deferred under\-fitting ofBB—cost is*relocated*, not removed \(the task\-AAnaive and task\-BBallocation curves coincide at13\.9413\.94\)\.*Right:*versus feature\-support overlapk/rk/r, the relocated interference \(naiveAA\-forgetting==allocationBB\-residual\) grows linearly to2\.02\.0while the optimal\-merge floor sits at exactly half \(1\.01\.0\); allocation holdsAA\-forgetting at0\.
### 5\.3Similarity and capacity

To test the signed gate, we compare sharing and orthogonalizing on a shared block as a function of target similarityss\(Fig\.[7](https://arxiv.org/html/2607.09202#S5.F7), left\)\. Sharing wins by variance reduction when tasks are similar and loses by bias when they are not, with the net benefit crossing zero ats∗≈0\.26s^\{\\ast\}\\approx 0\.26; above this threshold the gate should share, below it orthogonalize\. With exact, full data the two strategies are an exact relocation of the same cost, so this sign\-change is a finite\-data effect that unconditional projection methods forgo\. Separately, under orthogonal allocation a stream of rank\-22tasks exhibits a sharp capacity knee atT=d/rT=d/r\(Fig\.[7](https://arxiv.org/html/2607.09202#S5.F7), right\) at which earlier tasks remain intact, but once the occupied rank exceeds capacity, residual distortion turns on where predicted\. Capacity is thus a rate–distortion budget \(the rank capkmaxk\_\{\\max\}of Algorithm 1 is its operational form\)\. Tradingddagainst the number of tasks is an effective\-capacity budget, and dropping the least\-excited direction bounds the induced forgetting by its eigenvalue\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x6.png)

![Refer to caption](https://arxiv.org/html/2607.09202v1/x7.png)

Figure 7:Similarity and capacity\.*Left:*total population loss of pooling \(share\) versus isolating \(orthogonalize\) two tasks on a shared block, against target similarityss\. Isolation is flat inss; pooling trades variance reduction against bias and wins only when tasks are similar, with the net benefit crossing zero ats∗≈0\.26s^\{\\ast\}\\approx 0\.26\.*Right:*mean residual distortion over a stream of rank\-22tasks under orthogonal allocation, for three feature dimensions\. Each curve stays at zero until its capacity kneeT=d/rT=d/r\(dotted\) and rises thereafter; more capacitydddefers and lowers the floor \(thed=40d=40stream ends before its knee atT=20T=20\)\.
### 5\.4Curvature drift and the segment\-averaged correction

Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)is exact under frozen features, but in jointly trained networks its frozen\-curvature prediction degrades with depth\. The shallow network holds Pearsonr≈0\.93r\\approx 0\.93–0\.950\.95up to moderate step sizes, whereas deeper networks are markedly less predictable, withrrfalling to≈0\.2\\approx 0\.2–0\.40\.4\(Fig\.[8](https://arxiv.org/html/2607.09202#S5.F8), top\)\. Replacing the initial curvature by the segment\-averaged curvature of Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)removes this failure mode: in the same regime, the corrected predictor restoresr=1\.00r=1\.00at every tested depth \(Fig\.[8](https://arxiv.org/html/2607.09202#S5.F8), bottom\), estimated from forward evaluations ofLAL\_\{A\}along the segmentwA→wBw\_\{A\}\\\!\\to\\\!w\_\{B\}with no explicit Hessian\. The correction fromK=21K=21probe evaluations consumes1\.4%1\.4\\%of one task’s training budget, andK=5K=5already suffices in practice for under0\.4%0\.4\\%\. Once the stale curvature is corrected, the architecture\-general identity holds exactly\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x8.png)

![Refer to caption](https://arxiv.org/html/2607.09202v1/x9.png)

Figure 8:Curvature drift and its correction\.*Top:*predicted\-versus\-measured forgetting fidelity for backprop\-trained networks, against learning rate \(feature drift\) and depth\. The frozen\-curvature identity is exact for frozen features \(A1\) and degrades as the feature map drifts, most strongly with depth \(down tor≈0\.2r\\approx 0\.2–0\.40\.4for the deeper networks\)\.*Bottom:*at learning rate0\.40\.4, replacingΣA\\Sigma\_\{A\}by the segment\-averaged HessianH¯A\\bar\{H\}\_\{A\}of Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)\(green\) restoresr=1\.00r=1\.00at every depth, while the frozen\-curvature predictor \(red\) remains imperfect \(r=0\.77r=0\.77–0\.870\.87\)\.
### 5\.5Real data: dissimilar and similar tasks

We evaluate the functional and igfa on real benchmarks built on frozen random\-feature backbones, so that Assumption A1 and hence Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)hold exactly while the tasks remain non\-trivial\. ForSplit\-Digits, we construct a task\-incremental benchmark from thedigitsdataset \(17971797images,1010classes\), split into five two\-class tasks on top of a frozen random ReLU feature map \(d=300d=300\)\. The functional predicts per\-task forgetting across the full task sequence with Pearsonr=0\.999r=0\.999, on they=xy=xline, and the feature spectrum shows an effective per\-task rank of about22, matching the two\-class structure and explaining why a small protected subspace suffices \(Fig\.[9](https://arxiv.org/html/2607.09202#S5.F9)\)\. On this sequence, igfa attains0\.980±0\.0070\.980\\pm 0\.007average accuracy \(mean±\\pm95% CI over1010seeds\) with near\-zero forgetting, matching the strongest replay\- and Fisher\-free structural baselineogd/gpmand exceeding naive training, EWC, and a small replay baseline \(Table[1](https://arxiv.org/html/2607.09202#S5.T1)\)\. These digit\-pair tasks have low pairwise feature overlap, all belows∗s^\{\\ast\}, so the gate orthogonalizes throughout and igfa reduces toogd*exactly*—the two are identical on every seed \(pairedtt\-testp=1p=1\)\. igfa maintains parity with the best structural baseline, with no buffer and no Fisher matrix\.

Table 1:Split\-Digits task\-incremental \(frozen random\-feature backbone,55tasks,d=300d=300\)\. Mean±\\pm95% CI over1010seeds; average final accuracy \(higher better\) and average forgetting on earlier tasks \(lower better\)\. igfa matches the best structural baseline—and is identical toogdon every seed \(pairedtt\-testp=1p=1\)—with no replay buffer and no Fisher matrix\.![Refer to caption](https://arxiv.org/html/2607.09202v1/x10.png)Figure 9:Diagnostics on real data \(Split\-Digits\)\.*Left:*measured versus predicted forgetting12​∑cΔc⊤​ΣA​Δc\\tfrac\{1\}\{2\}\\sum\_\{c\}\\Delta\_\{c\}^\{\\top\}\\Sigma\_\{A\}\\Delta\_\{c\}for all \(earlier, later\) task pairs—on they=xy=xline, Pearsonr=0\.999r=0\.999, confirming Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)on real features\.*Right:*the eigenvalue spectrum of a task’s feature second\-momentΣA\\Sigma\_\{A\}; its effective rank \(≈2\\approx 2\) is the dimension that must be protected, and the rapid decay is why a small subspace summary suffices\.To test the sharing branch, we next considerRotated\-Digits, a domain\-incremental benchmark on the same ten classes under five rotations, again with a frozen random\-feature backbone\. These tasks are similar, with mean pairwise feature\-subspace overlap0\.680\.68\. In this regime, unconditional orthogonalization should be too conservative because the tasks share reusable structure rather than only conflict\. Here, igfa with thresholds∗=0\.65s^\{\\ast\}=0\.65\(within the broad optimum plateau of Sec\.[S6](https://arxiv.org/html/2607.09202#A6); the break\-even similarity is regime\-specific\) attains the highest accuracy among replay\- and Fisher\-free methods at0\.771±0\.0300\.771\\pm 0\.030\(mean±\\pm95% CI over55seeds\),*significantly*aboveogd’s0\.684±0\.0350\.684\\pm 0\.035\(pairedtt\-testp≈6×10−5p\\approx 6\\times 10^\{\-5\}\) and above naive \(0\.714±0\.0240\.714\\pm 0\.024\) and EWC \(0\.727±0\.0500\.727\\pm 0\.050\), while keeping forgetting near zero at0\.002±0\.0270\.002\\pm 0\.027—statistically indistinguishable fromogd’s \(p=0\.44p=0\.44\) \(Table[2](https://arxiv.org/html/2607.09202#S5.T2)\)\. Only buffer\-based replay attains higher accuracy \(0\.795±0\.0120\.795\\pm 0\.012,p=0\.03p=0\.03\), at the cost of stored data\. igfa thus matches the best structural baseline on dissimilar tasks, and its signed gate yields a*statistically significant*advantage in the similar\-task regime for which it was designed\. Section[5\.6](https://arxiv.org/html/2607.09202#S5.SS6)confirms the dissimilar\-task reading at scale\.

Table 2:Rotated\-Digits domain\-incremental \(frozen backbone, five rotations, mean subspace overlap0\.680\.68, thresholds∗=0\.65s^\{\\ast\}=0\.65\)\. Mean±\\pm95% CI over55seeds\. igfa attains the highest accuracy among replay\- and Fisher\-free methods with near\-zero forgetting—an intermediate between full isolation \(ogd, pairedtt\-testp≈6×10−5p\\approx 6\\times 10^\{\-5\}\) and full sharing \(naive\) that improves significantly on both\. Only buffer\-based replay attains higher accuracy \(p=0\.03p=0\.03\), at the cost of stored data\.
### 5\.6Scaling checks: frozen\-ViT benchmarks over five seeds

We repeat the comparison on three standard benchmarks over a frozen ViT\-B/16 backbone with a single shared linear head, so A1 and Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)hold exactly: Split\-CIFAR\-100 \(1010tasks of1010classes\), ImageNet\-R \(1010of2020\), and CUB\-200 \(1010of2020\), each over five seeds with per\-seed class orders \(mean±\\pm95% CI;kaggle\_vit\_multiseed\.py; Table[3](https://arxiv.org/html/2607.09202#S5.T3)\)\. Three observations follow\.*\(i\) On dissimilar\-class streams the gate correctly reduces toogd\.*On Split\-CIFAR\-100 and CUB every pairwise subspace overlap lies below the threshold \(0\.310\.31–0\.540\.54vss∗=0\.6s^\{\\ast\}=0\.6\), so igfa protects everything and is identical toogdon every seed; the two attain the best retention among buffer\- and Fisher\-free methods \(forgetting0\.0010\.001and−0\.004\-0\.004, vs naive0\.0090\.009and0\.0100\.010\)\.*\(ii\) On ImageNet\-R the sharing branch pays off at scale\.*There the overlaps straddle the threshold \(0\.510\.51–0\.650\.65\), the gate shares some past subspaces while protecting others, and igfa significantly exceedsogd\(0\.788±0\.0040\.788\\pm 0\.004vs0\.781±0\.0050\.781\\pm 0\.005, pairedp=0\.001p=0\.001\) at equal, zero forgetting—the similar\-task advantage of Sec\.[5\.5](https://arxiv.org/html/2607.09202#S5.SS5), previously isolated on Rotated\-Digits, appears on a standard benchmark\.*\(iii\) Extra state buys accuracy, not retention\.*A tuned Fisher penalty \(EWC\) attains the highest raw accuracy on these mild streams at𝒪​\(d\)\\mathcal\{O\}\(d\)Fisher state, while feature replay under an unmasked objective*hurts*on two of the three benchmarks and carries the largest forgetting—stored data is not automatically an advantage\. Forgetting is small for every method on these disjoint\-class streams because the frozen backbone already separates the tasks; the structural claim therefore concerns retention and state, not raw accuracy\.

#### A similar\-task stream at scale\.

To test the gate’s regime beyond rotations we build a domain\-incremental stream from CIFAR\-100’s superclass structure: five tasks, each classifying into the same2020coarse labels but drawing on disjoint fine classes \(one per superclass per task\), so consecutive tasks share semantics \(subspace overlaps0\.520\.52–0\.610\.61\)\. Structural allocation carries this stream: igfa/ogdreach0\.786±0\.0100\.786\\pm 0\.010versus naive0\.742±0\.0060\.742\\pm 0\.006\(p=0\.0009p=0\.0009\), with forgetting0\.1690\.169versus0\.2370\.237; EWC’s anchor blocks adaptation entirely \(0\.662±0\.0200\.662\\pm 0\.020\), and only replay, storing data, does better \(0\.822±0\.0020\.822\\pm 0\.002\)\. The gate matchesogdhere \(p=0\.37p=0\.37\) because most overlaps sit just belows∗s^\{\\ast\}—consistent with the ImageNet\-R reading that the sharing advantage appears where overlaps straddle the threshold\.

#### When the gate pays is a measurable property of the stream\.

Across all six real streams of this paper, the gate’s realized advantage over unconditional projection is organized by one pre\-deployment statistic: the stream’s*share density*, the fraction of task pairs whose measured similarity crosses the gate threshold \(Fig\.[10](https://arxiv.org/html/2607.09202#S5.F10);fig\_stream\_density\.py\)\. On the three zero\-density streams \(Split\-CIFAR\-100, CUB, the AG\-News LoRA domains\) the gate provably reduces toogdand ties exactly; the low\-density superclass stream ties within noise; and the two positive\-density streams are precisely the two statistically significant wins, ordered by density \(Rotated\-Digits, density0\.540\.54,\+0\.087\+0\.087; ImageNet\-R, density0\.270\.27,\+0\.007\+0\.007; Spearman0\.900\.90over the six streams\)\. The statistic is computable before training from the same probes the pre\-flight diagnostic already runs, so whether the gate is worth deploying is itself a pre\-flight question rather than a post\-hoc finding\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x11.png)Figure 10:The gate’s realized gain overogdagainst the stream’s share density, for all six real streams in this paper \(stars: statistically significant gains; grey: exact or statistical ties\)\. Zero\-density streams tie by construction—the gate has nothing to decide—and the gain grows with density \(Spearman0\.900\.90\)\. Share density is measurable before training, making gate\-versus\-projection a pre\-deployment decision\.The same frozen backbone also supports a*prompt*\-based instantiation beyond the linear head, and a pair of protocols brackets exactly when gating pays\. When all tasks reuse one shared output head—the high\-interference regime of a deployed adapter serving every task—gating the prompt and head updates lifts accuracy from0\.4160\.416to0\.7330\.733\(single run,kaggle\_pending\_experiments\.py\): the structural rule carries over to prompt space wherever interference is present\. When each class instead owns its output row and logits are task\-masked—a protocol that removes head interference by construction—naive prompt\-tuning barely forgets \(0\.017±0\.0160\.017\\pm 0\.016\) and the gate is statistically indistinguishable from it \(0\.917±0\.0240\.917\\pm 0\.024vs0\.926±0\.0240\.926\\pm 0\.024, pairedp=0\.20p=0\.20; five seeds,kaggle\_vit\_multiseed\.py\)\. The pair is the functional’s own prediction made visible: the gate’s benefit scales with the interference energy actually present, and vanishes when the protocol allocates disjoint parameters up front\.

#### Relation to prompt\-pool methods\.

L2P, DualPrompt, and CODA\-Prompt\[[18](https://arxiv.org/html/2607.09202#bib.bib18),[19](https://arxiv.org/html/2607.09202#bib.bib19),[20](https://arxiv.org/html/2607.09202#bib.bib20)\]also operate on a frozen pretrained ViT backbone\. They circumvent the physical memory overhead and data\-retention privacy risks of traditional Buffer\-based methods by routing inputs dynamically into a virtual parameter buffer, the prompt pool\. However, their retention remains entirely empirical based on an input\-conditioned selection mechanism to map classes without test\-time task identities\. Our approach shifts from statistical heuristic to mathematical design\. By utilizing a task\-incremental protocol with a single shared linear head, igfa forces protected directions to remain first\-order invariant by construction \(Corollary[1](https://arxiv.org/html/2607.09202#Thmcorollary1)\)\. While its Split\-CIFAR\-100 accuracy sits below highly optimized prompt\-pool baselines, the raw accuracies are fundamentally mismatched across these two distinct evaluation paradigms\. The claim is confined to the buffer\- and Fisher\-free regime where igfa attains the strongest known retention\.

Table 3:Frozen\-ViT scaling checks, task\-incremental with a single shared linear head \(A1 exact\); mean±\\pm95% CI over five seeds \(kaggle\_vit\_multiseed\.py\)\. On the dissimilar\-class streams \(Split\-CIFAR\-100, CUB\) igfa reduces toogdon every seed and attains the best buffer\- and Fisher\-free retention; on ImageNet\-R, whose subspace overlaps straddles∗s^\{\\ast\}, the gate significantly exceedsogd\(pairedp=0\.001p=0\.001\) at zero forgetting\. EWC carries𝒪​\(d\)\\mathcal\{O\}\(d\)Fisher state; replay stores1010features/class\.

### 5\.7Online drift: recursive estimation versus a stale subspace

The retention guarantees so far fix the protected subspace at task onset\. Under representation drift—the regime of jointly trained and large models—an unconditional projection fails, and the recursive Gauss–Newton surrogate of Sec\.[4](https://arxiv.org/html/2607.09202#S4)is justified\. We stream tasks whose feature geometry rotates each step while the underlying subspace stays low\-rank, and contrast a stale protected subspace \(fixed at onset, as inogd/gpm\) with the recursive estimate\. The stale subspace accumulates each drifted copy as if new, so its rank climbs linearly until it fills the parameter space \(rank→d=60\\mathrm\{rank\}\\\!\\to\\\!d=60\) and free plasticity collapses to zero—the network can no longer learn\. The recursive estimate recognizes the drifted copies as one subspace, holding the effective rank at the true value \(≈4\\approx 4\) and preserving93%93\\%of capacity throughout \(Fig\.[11](https://arxiv.org/html/2607.09202#S5.F11)\)\. Re\-estimating the geometry is therefore not a refinement but the difference between exhausting capacity and retaining it under drift; this is the subspace\-projection failure mode reported for LLM merging\[[31](https://arxiv.org/html/2607.09202#bib.bib31)\], and the recursive update removes it\. We also compared a geodesic\-aligned tracker \(a Grassmann\-manifold step in the spirit of GAGP\[[27](https://arxiv.org/html/2607.09202#bib.bib27)\]\), holding the base optimizer, learning rate, and batch order fixed: it likewise caps the rank at the true value, but tracks the current subspace at markedly lower fidelity \(≈0\.5\\approx 0\.5versus≈1\.0\\approx 1\.0for the recursive estimate across learning rates0\.050\.05–0\.40\.4\), so the path\-averaged Gauss–Newton update of Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)is the more faithful deep\-network drift tracker\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x12.png)Figure 11:Online subspace update under feature drift\.*Left:*a stale protected subspace \(ogd/gpm, red\) accumulates drifted copies and its rank grows to capacitydd\(exhaustion\); the recursive Gauss–Newton estimate \(green\) stays at the true rank \(≈4\\approx 4\)\.*Right:*consequently the stale method’s free plasticity\(d−rank\)/d\(d\-\\mathrm\{rank\}\)/dcollapses to0\(cannot learn new tasks\) while the recursive one retains≈0\.93\\approx 0\.93\.

## 6Discussion

The identityΔ​LA=12​Δ⊤​ΣA​Δ\\Delta L\_\{A\}=\\tfrac\{1\}\{2\}\\,\\Delta^\{\\top\}\\Sigma\_\{A\}\\,\\Delta\(Eq\. \([2](https://arxiv.org/html/2607.09202#S3.E2)\)\) reorganizes the continual\-learning problem around geometry\.Components inrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}incur forgetting, whereas components inker⁡ΣA\\ker\\Sigma\_\{A\}are free\. This perspective determines, before training begins, whether two tasks must interfere at all, what irreducible floor remains when they do, and which inner product renders model merging optimal—replacing the Euclidean metric with theΣ\\Sigma\-metric\. In that sense, the functional plays the same conceptual role here that the NTK\[[11](https://arxiv.org/html/2607.09202#bib.bib11)\]played for generalization: it provides a geometry and a set of computable quantities in place of heuristics\.

#### A map of the claims\.

The paper makes three kinds of claims\. Some are*\(I\) exact*in frozen\-feature or function\-space regime \(the interference identity, the removability dichotomy, theΣ\\Sigma\-orthogonal optimal merge, the distortion floor, and continuous\-time exact retention, Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\.*Exact\-with\-quantified\-residual*: end\-to\-end parameter\-space retention, exact up to the basis error of Proposition[1](https://arxiv.org/html/2607.09202#Thmproposition1)and the finite\-step remainder of Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2), both measured\. Some are*\(III\) empirically supported*and remain exact up to measured approximation terms under end\-to\-end training \(the multi\-seed benchmark results, Split\-Digits, Rotated\-Digits, the frozen\-ViT streams, and the threshold\-free gate’s real\-text gain over naive fine\-tuning\)\.*\(IV\) Mechanism probes and conjectures\.*The generality extensions \(federated merging, out\-of\-distribution and task\-free signals, physics\-informed constraints, Fisher\-metric policy gradients, meta\-learning, memory\-and\-control instances\) are exact\-regime proof\-of\-concepts labelled in the Supporting Information as*proved\-in\-A1*,*empirical mechanism probe*, or*forward\-looking conjecture*\.

and some are empirical demonstrations of the mechanism at benchmark and language\-model scale\. Keeping these tiers separate clarifies both what is proved and what is shown experimentall

#### Forgetting decomposes into incompatibility, capacity, and control\.

The results above assemble into an attribution identity\. Writing𝒟∞⋆\\mathcal\{D\}^\{\\star\}\_\{\\infty\}for the floor of any single predictor \(Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6); under cross\-entropy the conditional mutual information\),𝒟class⋆\\mathcal\{D\}^\{\\star\}\_\{\\mathrm\{class\}\}for the optimum of the*deployed*class \(width, rank, architecture\), andLLfor what training achieved, every observed forgetting level splits exactly, term by non\-negative term, intoL=𝒟∞⋆\+\[𝒟class⋆−𝒟∞⋆\]\+\[L−𝒟class⋆\]L=\\mathcal\{D\}^\{\\star\}\_\{\\infty\}\+\[\\mathcal\{D\}^\{\\star\}\_\{\\mathrm\{class\}\}\-\\mathcal\{D\}^\{\\star\}\_\{\\infty\}\]\+\[L\-\\mathcal\{D\}^\{\\star\}\_\{\\mathrm\{class\}\}\]:*incompatibility*, a property of the task family that no gate, replay, or width removes\.*Capacity*is a property of the model class that only widening or routing removes and*control*a property of the training policy that only a better gate, replay schedule, or ordering removes\. In the exact regime each intervention moves only its own term\. Widening eliminates the capacity term while no policy touches it and the active\-set gate cuts the control term while the floor is invariant throughout \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\. The practical consequence is a diagnosis rule that precedes every method choice in this paper’s taxonomy: measure which term dominates \(floor from the data, class optimum from the deployed geometry, control as the remainder\)*before*deciding on a mechanism, because each family of continual\-learning methods can repair exactly one of the three\. The attribution is measurable end to end on real continual pretraining: on the four\-domain language stream, a certified floor bound of0\.0050\.005nats/token against observed forgetting of0\.160\.16–0\.400\.40establishes that at least97%97\\%of the forgetting was avoidable in principle, with the control term dominating or co\-dominating at both scales and the active\-set gate removing a third to a half of it \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\. A further split follows from the identity’s anchor: evaluated on the deployment rather than the training distribution, forgetting separates into knowledge loss and memorization loss, and the two can carry opposite signs\. An interfering update can raise train loss while*improving*the deployed function, so part of what benchmarks report as catastrophic forgetting is implicit unlearning of noise\. Protection must therefore anchor on validated data: a gate fed training gradients defends memorized noise as if it were knowledge, and is dominated on both axes by its held\-out\-anchored counterpart \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\.

The central assumption is A1\. For frozen backbones and PEFT, the regime most practical deployments now occupy, it is exact and the results stand without approximation, as the real\-data identity atr=0\.999r=0\.999confirms\. For full networks it is first\-order, and the paper measures how that approximation degrades: the prediction remains accurate for shallow, small\-step training and falls tor≈0\.2r\\approx 0\.2–0\.40\.4for deeper networks \(identity becomes first\-order, Sec\.[5\.4](https://arxiv.org/html/2607.09202#S5.SS4)\)\. Allocation does not eliminate the floor, but only redistributes its cost\. Task ordering does not reduce the floor either because the floor is a property of the joint support overlap\. It is essentially invariant to task order \(aΣ\\Sigma\-overlap curriculum did not lower it in our tests, Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\), which reinforces that allocation only relocates cost\. Against unconstrained replay, which can reduce the floor by re\-supplyingΣA\\Sigma\_\{A\}, the claim is therefore parity under no\-buffer or no\-Fisher constraints together with structural guarantees and preserved transfer where similarity permits\. On the benchmark, igfa matches the best structural baseline because the tasks are dissimilar; its distinctive sharing gain requires task similarity, isolated on Rotated\-Digits and confirmed at scale on ImageNet\-R, where the gate significantly exceeds unconditional projection at equal forgetting \(Sec\.[5\.6](https://arxiv.org/html/2607.09202#S5.SS6)\)\.

The geometry also yields an operational decision procedure\. Before training, a linear task\-probe on the frozen features estimates task separability \(Sec\.[S8](https://arxiv.org/html/2607.09202#A8)\); pairwise subspace overlap and target similarity then classify the stream\. Disjoint supports make structural allocation lossless, so neither replay nor a Fisher penalty is needed\. Overlapping supports with aligned targets favor sharing via gate\. Overlapping supports with conflicting targets carry an irreducible floor, which poses the question of where to assign its cost: on the old task as forgetting, on the new task as deferred fit, or on the memory budget as stored data for replay\. The capacity budget adds the final constraint: once the cumulative occupied rank approaches the feature dimension, retention must be paid for with added capacity, replay, or graceful overwriting\. Each branch of this procedure is measurable from features and subspaces before or during training, rather than diagnosed after a failure\.

A recent broad evaluation reports that subspace\-projection merging works well on small classifiers but degrades on large language models\[[31](https://arxiv.org/html/2607.09202#bib.bib31)\]\. On small classifiers the feature geometry is nearly static, so a subspace fixed once remains a good metric and projection succeeds, which is precisely the frozen regime where Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)is exact\. Large language models, by contrast, have highly entangled representations that drift as parameters move, so projection against a stale geometry accumulates misaligned constraints, exhausts capacity, and eventually damages the representation itself\. A natural fix follows from Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2): the relevant metric is the curvature along the trajectory, and its online surrogate is the recursive Gauss–Newton estimate, which continually realigns the protected subspace with the current geometry and keeps its effective rank bounded—the capacity\-exhaustion mechanism shown directly in Sec\.[5\.7](https://arxiv.org/html/2607.09202#S5.SS7)\. Preliminary language\-model results suggest that the retention mechanism transfers, while the right similarity signal for language remains open \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\)\.

The main open design question is therefore not the structural retention mechanism, which transfers across modalities, but a similarity signal for language that can recover transfer on similar text tasks without relaxing protection where conflict dominates\. Two resolutions are developed in the Supporting Information: a probe\-based similarity estimate validated in the exact regime \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\), and—the theorem\-backed answer—the threshold\-free functional gate of Sec\.[S15](https://arxiv.org/html/2607.09202#A15), which reads the measured interference rate instead of any modality\-specific similarity signal, carries a continuous\-time monotone\-retention guarantee, and delivers a significant, threshold\-free real\-text gain over naive continual fine\-tuning on a four\-domain LoRA stream, with parity\-to\-slight\-edge over unconditional protection \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\)\. Several extensions are detailed in the Supporting Information\. First, we show sequential LoRA fine\-tuning on language models at scale \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\), using recursive Gauss–Newton tracking on the adapting features together with a modality\-appropriate similarity gate for language\. Second, the frozen\-backbone evaluation on Split\-CIFAR\-100, ImageNet\-R, and CUB appears in the main text over five seeds \(Sec\.[5\.6](https://arxiv.org/html/2607.09202#S5.SS6)\), comparing the retention–transfer tradeoff againstogd/gpm, EWC, and replay in the exact A1 regime\. Third, we present overlay\-native continual pretraining at the billion\-parameter scale \(Sec\.[S12](https://arxiv.org/html/2607.09202#A12)\), where replay\-free and Fisher\-free operations are most valuable and where the capacity budget of Sec\.[5\.3](https://arxiv.org/html/2607.09202#S5.SS3)predicts the model\-size\-versus\-rehearsal tradeoff directly\. Finally, we introduce a pre\-flight feasibility diagnostic \(Sec\.[S8](https://arxiv.org/html/2607.09202#A8)\) that estimates task separability from features before training, attributing any remaining gap to drift, bias, or inference rather than to intrinsic overlap\.

Beyond continual learning, the same functional acts as a general calculus for interference between quadratic objectives that share parameters \(Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\)\. Federated aggregation is the merge problem of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)with clients as tasks; the occupied subspace supplies an out\-of\-distribution signal and a task\-free boundary detector; the physics constraint of a physics\-informed network is a second task whose violation the identity predicts exactly; the Fisher information plays the role ofΣ\\Sigmafor policy gradients in reinforcement learning; and a feature map can be meta\-trained to shrink the distortion floor itself\. These are directional demonstrations rather than benchmarks, but they indicate that the geometry is a property of shared parameterizations, not of the continual\-learning protocol\. Read as a control problem, the calculus also subsumes the existing method families rather than competing with them: unconditional projection \(ogd/gpm\), Fisher penalties \(EWC\), replay, per\-task routing, and dynamic expansion are each the geometry\-driven controller clamped to a single fixed action—always project, always penalize, always rehearse, always separate—while the signed gate is the regime\-adaptive interior that reads the ledger and spends memory or rank only where the floor is positive\. In a controlled comparison across regimes the adaptive rule is the Pareto point, strictly dominating every zero\-resource fixed method and matching replay’s retention at a fraction of its memory \(runnable indemo\_method\_taxonomy\.py, with the metric set and decision rules released as a standalone interference\-ledger toolkit\)\.

#### A testable bridge to the verbalizable workspace\.

The convergence with the Jacobian lens \(Sec\.[2](https://arxiv.org/html/2607.09202#S2)\) yields a falsifiable prediction rather than only an analogy\. If ourrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}and the lens’s J\-space are the same functional subspace, then forgetting must be*verbalizable\-subspace\-mediated*: the readout changeΔ​f\\Delta fthat a later task induces on taskAAshould carry its interference energy insideAA’s J\-space, and protecting that subspace should prevent forgetting as effectively as protectingΣA\\Sigma\_\{A\}’s own eigen\-subspace\. We sketch the experiment \(kaggle\_jspace\_interference\.py\) on a frozenpythia\-410m: fit or load the released lens to obtain the J\-space projectorPJP\_\{J\}at a readout layer, learn the four\-domain stream with a low\-rank head, and \(i\) decompose the measured forgetting of domainAAinto itsPJP\_\{J\}and orthogonal parts, and \(ii\) compare naive learning against gating the update offPJP\_\{J\}, offΣA\\Sigma\_\{A\}’s top subspace, and off a random subspace of equal rank\. The interference geometry predicts a J\-space energy fraction far above ther/dr/drandom baseline in \(i\), and*protect\-J\-space*≈\\approx*protect\-*Σ\\Sigma≪\\ll*naive*with*protect\-random*≈\\approx*naive*in \(ii\)—a specificity control that would establish the two averaged\-Jacobian subspaces as one\. Measuring both on a frozenpythia\-410mresolves the relationship into a*component*rather than an identity \(kaggle\_jspace\_layers\.py\)\. The strong form fails: the interferenceΣA=𝔼​\[g​g⊤\]\\Sigma\_\{A\}=\\mathbb\{E\}\[gg^\{\\top\}\]is high\-effective\-rank at every layer—80%80\\%of its energy needs at least285285of10241024dimensions—so a workspace\-sized subspace cannot carry the bulk of forgetting, which is also why no low\-rank projection protects in \(ii\),ΣA\\Sigma\_\{A\}’s own top subspace included\. But the enrichment is real and, tellingly, grows toward the readout: the J\-space carries1\.31\.3to2\.5×2\.5\\timesits dimensional share of interference energy across layers22to2222\(enriched at every layer probed; correlation with depth0\.860\.86\), and its overlap withΣA\\Sigma\_\{A\}’s leading subspace rises from0\.050\.05to0\.150\.15against a0\.030\.03chance baseline\. The verbalizable directions are thus systematically over\-represented in the forgetting geometry—increasingly so where representations become more readable—without being identical to it, a measured link between what a model can say and what it can lose\.

Several algorithmic refinements—a self\-calibrating threshold, low\-cost𝒪​\(r​d\)\\mathcal\{O\}\(rd\)online curvature via Frequent Directions\[[32](https://arxiv.org/html/2607.09202#bib.bib32)\], a per\-direction gate, and the soft\-gating capacity coordinatereff=∑j\(1−keepj\)r\_\{\\rm eff\}=\\sum\_\{j\}\(1\-\\mathrm\{keep\}\_\{j\}\)—are developed and validated in the Supporting Information\. The functional further extends to control and memory: an interference\-budgeted trust region that dominates step\-size control, consolidation with a priced forgetting bound, floor\-scheduled replay, and generative replay from the subspace state itself \(Sec\.[S14](https://arxiv.org/html/2607.09202#A14)\)\. Together with low\-rank or diagonal Gauss–Newton approximations ofH¯A\\bar\{H\}\_\{A\}, they suggest a practical route for carrying the architecture\-general identity into fully fine\-tuned deep networks at controlled cost\.

A further assumption deserves explicit statement: as presented, igfa assumes task boundaries are known and switches are abrupt at training time\. Section[4](https://arxiv.org/html/2607.09202#S4)and Algorithm 1 use it most directly—the method processes “each tasknn” and forms a basisBnB\_\{n\}from that task’s estimated second moment\. The assumption is operational bookkeeping, not theory: the interference functional itself is boundary\-free, and the machinery that removes the bookkeeping already appears in this paper\. The recursive Gauss–Newton tracker replaces per\-task second moments with a decayed running estimate, the subspace\-velocity detector recovers unknown boundaries at recall and precision1\.001\.00on clean streams, and their fully autonomous composition matches the oracle given the true boundaries \(Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\)\. Gradual transitions are the harder case; there the boundary concept itself blurs, and the drift experiments of Sec\.[5\.7](https://arxiv.org/html/2607.09202#S5.SS7)show that the running estimate simply tracks the moving geometry without requiring a boundary at all\.

Several limitations bound these claims\. Exactness of the parameter\-space results is confined to A1; the function\-space formulation extends the identity, the dichotomy, the floor, and the continuous\-time retention guarantee beyond it \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\), leaving finite step size and metric estimation as the remaining—quantified—gaps, and end\-to\-end*parameter\-space*predictions stay first\-order, with a measured accuracy envelope and the segment\-averaged correction\. The frozen\-ViT benchmarks and the merge ablation are five\-seed, and the functional\-gate language result is now multi\-seed at two scales \(GPT\-2, five seeds;pythia\-410m, three seeds\), but the remaining language\-model experiments are single runs and the extension studies are synthetic proofs\-of\-concept; multi\-seed replication at billion\-parameter scale is outstanding\. The similarity gate reads input geometry only in its threshold form; the threshold\-free functional gate resolves the language case in principle \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\) but its margins at scale are modest\.

What breaks the exact identities, and how\.*Label noise / imperfect task optima:*a non\-zero residualrA≠0r\_\{A\}\\neq 0adds the cross term𝔼​\[Δ​f​rA\]\\mathbb\{E\}\[\\Delta\\\!f\\,r\_\{A\}\]of Eq\. \([4](https://arxiv.org/html/2607.09202#S3.E4)\); forgetting is then predicted by the*full*identity, not the energy alone, and mis\-specification enters as a task\-constant under A2 \(theΣ\\Sigma\-metric ranking is preserved\)\.*Early stopping / finite steps:*the update is not the exact minimizer, leaving theO​\(η\)O\(\\eta\)remainder of Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)and the diffusion term of Proposition[1](https://arxiv.org/html/2607.09202#Thmproposition1)—both measured, both shrinking with step size and basis accuracy\.*Class imbalance / shifted input density:*ΣA\\Sigma\_\{A\}is re\-weighted by the deployment density, so the protected geometry must be estimated on the distribution that matters, not the training mixture; a stale or mis\-weightedΣA\\Sigma\_\{A\}shows up as basis errorϵ\\epsilonand is caught by the drift\-velocity monitor\. In each case the failure is a*named, measurable*quantity in the identities, not an unmodelled gap\.

## 7Conclusion

Forgetting is interference, represented here as geometry\. In the frozen\-feature regime, forgetting equals12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Deltaand is removable if and only if task supports are disjoint; when supports overlap, an irreducible distortion floor appears and matches the task\-inference obstruction, and optimal merging is mutualΣ\\Sigma\-orthogonalization, with continual learning as its online counterpart\. The result is a geometric language and a set of concrete levers—a forgetting functional, a removability test, an optimal\-merge identity, a similarity sign\-change, and a capacity budget—realized by a replay\-free, Fisher\-free allocation rule that carries only a low\-rank subspace\. The same functional governs any setting where a shared parameterization must serve multiple input geometries, from multi\-task and federated learning to domain adaptation and model merging; the Supporting Information \(Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\) gives exact\-A1 proofs\-of\-concept for federatedΣ\\Sigma\-merging, an intrinsic out\-of\-distribution signal, task\-free boundary detection, multimodal block interference, physics\-informed constraints, Fisher\-metric policy gradients, and floor\-minimizing meta\-learning\. In each setting, forgetting is controlled geometrically: interference is not corrected after it occurs but allocated in advance, into directions where it induces no loss\.

## Author contributions

J\.S\. conceived the framework, derived the theory, implemented the experiments, and wrote the manuscript\.

## Acknowledgments

The author thanks colleagues at VARTA and TU Braunschweig for discussions on continual learning in industrial process models\.

## Reproducibility statement

All results are exact\-regime simulations with fixed seeds\.experiments\.pyregenerates the core result figures and numerical claims \(interference identity, lossless/relocation, sign\-changes∗s^\{\\ast\}, capacity floor\);verify\_mi\_identity\.pythe information–estimation verification \(Fig\.[S1](https://arxiv.org/html/2607.09202#A2.F1)\);experiments\_rev\.pythe breakdown, breakdown\-fix, and digit benchmarks;variance\_report\.pythe multi\-seed mean±\\pmCI and paired\-significance results of Tables[1](https://arxiv.org/html/2607.09202#S5.T1)–[2](https://arxiv.org/html/2607.09202#S5.T2);experiments\_compare\.pythe nine\-method comparison and offline\-merge residual\-DDablation of Fig\.[1](https://arxiv.org/html/2607.09202#S0.F1)and Table[S1](https://arxiv.org/html/2607.09202#A4.T1);evaluate\_hypotheses\.pyfour generality extensions of Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\(federatedΣ\\Sigma\-merge, OOD signal, task\-free boundaries, multimodal blocks\) andevaluate\_applications\.pythree more \(physics\-informed constraints, Fisher\-metric policy gradients, floor\-minimizing meta\-learning\);evaluate\_next\_steps\.pythe sketched\-drift, trust\-region, anchor\-protection, probe\-similarity, consolidation, floor\-scheduled\-memory, dream\-replay, cascade, and world\-model studies of Secs\.[S9](https://arxiv.org/html/2607.09202#A9),[S5](https://arxiv.org/html/2607.09202#A5), and[S14](https://arxiv.org/html/2607.09202#A14);kaggle\_probe\_gate\_llm\.pythe probe\-gate protocol on real text \(results inprobe\_gate\_results\.json\);evaluate\_open\_questions\_new\.pythe open\-question battery, with each result classified as exact\-scope support, mechanism probe, or appendix scoping \(only exact\-scope results are quoted in this manuscript\),evaluate\_oq\_hardened\.pyan end\-to\-end drift\-and\-noise probe on a gradient\-trained network,evaluate\_unlearning\_real\.pythe real\-data unlearning study,evaluate\_upgrades\.pythe noise\-law, rate\-controller, allocation, privacy, and damping upgrades,evaluate\_beyond\_a1\.pythe function\-space validations of Sec\.[S15](https://arxiv.org/html/2607.09202#A15),evaluate\_functional\_gate\.pythe threshold\-free functional gate \(withkaggle\_functional\_gate\_llm\.pyits real\-text protocol\), andfig\_stream\_density\.pythe share\-density figure;kaggle\_interference\_optimizer\.py\(theInterferenceSGDbenchmark against tuned learning\-rate schedules\) andkaggle\_scale\_addons\.py\(per\-layer text similarity, real\-featureΣeval\\Sigma\_\{\\rm eval\}, and the pythia\-160m capacity ledger\) extend these to scale;experiments\_drift\.pythes∗s^\{\\ast\}ablation, graceful\-degradation, and drift figures;experiments\_extra\.pythe self\-calibrating\-s∗s^\{\\ast\}and tight multiclass\-constant results; andexperiments\_roadmap\.pythe feasibility\-diagnostic, Frequent\-Directions, soft\-knee, and per\-direction figures\. Self\-contained Colab/Kaggle notebooks accompany the paper:colab\_split\_cifar100\_igfa\.py\(Split\-CIFAR\-100 with a frozen ViT\-B/16\),colab\_lora\_llm\_igfa\.pyandcolab\_lora\_llm\_igfa\_3approaches\.py\(sequential LoRA on a transformer with the four gate designs\), andcolab\_continual\_pretrain\_igfa\.py\(boundary\-free, replay\-free, Frequent\-Directions continual pretraining onpythia\-1b\); andkaggle\_pending\_experiments\.py, which runs the real\-backbone checks quoted above \(frozen\-ViT Split\-CIFAR\-100 allocation and merge ablation, the drift/geodesic sweep, prompt\-igfa, and the FD\-vs\-full\-vs\-Oja LoRA\-LM comparison\),kaggle\_H2\_H3\_H12\.py\(the layer\-wise null\-space\-routing pipeline on a frozen transformer\), andkaggle\_vit\_multiseed\.py\(the five\-seed frozen\-ViT protocol: Split\-CIFAR\-100, ImageNet\-R, CUB, the similar\-task CIFAR\-100 superclass stream, the merge ablation, and prompt\-igfa, on two T4 GPUs\)\. Code, figures, and logs are at[https://github\.com/j\-stoerk/geometry\-of\-forgetting](https://github.com/j-stoerk/geometry-of-forgetting)\.

## Supporting Information

This Supporting Information collects the experiments and derivations omitted from the main text for length: deferred proofs \(Sec\.[S1](https://arxiv.org/html/2607.09202#A1)\), additional theory verifications \(Sec\.[S2](https://arxiv.org/html/2607.09202#A2)\), theory details \(Sec\.[S3](https://arxiv.org/html/2607.09202#A3)\), the full method comparison behind Fig\.[1](https://arxiv.org/html/2607.09202#S0.F1)\(Sec\.[S4](https://arxiv.org/html/2607.09202#A4)\), benchmark and scale experiments \(Secs\.[S5](https://arxiv.org/html/2607.09202#A5)–[S12](https://arxiv.org/html/2607.09202#A12)\), the algorithmic refinements referenced in the Discussion, extensions of the framework to federated, out\-of\-distribution, task\-free, and multimodal settings \(Sec\.[S13](https://arxiv.org/html/2607.09202#A13)\), memory\-systems and control extensions \(Sec\.[S14](https://arxiv.org/html/2607.09202#A14)\), and the function\-space formalism that extends the exact results beyond frozen features \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\. All results are exact\-regime simulations or runnable notebooks with fixed seeds\.

## Appendix S1Deferred proofs

For completeness we give the derivations abbreviated to a sentence of intuition in the main text; the proofs of Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)\(a one\-line substitution\), Corollary[1](https://arxiv.org/html/2607.09202#Thmcorollary1)and Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)appear in the main text and are not repeated\.

###### Proof of Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)\(architecture\-general identity\)\.

Letφ​\(s\)=LA​\(wA\+s​Δ\)\\varphi\(s\)=L\_\{A\}\(w\_\{A\}\+s\\Delta\)\. Taylor’s theorem with integral remainder givesΔ​LA=φ​\(1\)−φ​\(0\)=φ′​\(0\)\+∫01\(1−t\)​φ′′​\(t\)​𝑑t\\Delta L\_\{A\}=\\varphi\(1\)\-\\varphi\(0\)=\\varphi^\{\\prime\}\(0\)\+\\int\_\{0\}^\{1\}\(1\-t\)\\varphi^\{\\prime\\prime\}\(t\)\\,dt, withφ′​\(0\)=∇LA​\(wA\)⊤​Δ\\varphi^\{\\prime\}\(0\)=\\nabla L\_\{A\}\(w\_\{A\}\)^\{\\top\}\\Deltaandφ′′​\(t\)=Δ⊤​∇2LA​\(wA\+t​Δ\)​Δ\\varphi^\{\\prime\\prime\}\(t\)=\\Delta^\{\\top\}\\nabla^\{2\}L\_\{A\}\(w\_\{A\}\+t\\Delta\)\\Delta, which yields Eq\. \([3](https://arxiv.org/html/2607.09202#S3.E3)\)\. When∇2LA≡ΣA\\nabla^\{2\}L\_\{A\}\\equiv\\Sigma\_\{A\}the remainder integrates to12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta, recovering Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)\. ∎

###### Proof of Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)\(distortion\-floor lower bound\)\.

Restricting the two\-task objective Eq\. \([6](https://arxiv.org/html/2607.09202#S3.E6)\) to the shared directionuureduces the merge to a scalar two\-point problem: minimize12​a​\(m−α\)2\+12​b​\(m−β\)2\\tfrac\{1\}\{2\}a\(m\-\\alpha\)^\{2\}\+\\tfrac\{1\}\{2\}b\(m\-\\beta\)^\{2\}over the shared coefficientmm, wherea=u⊤​ΣA​ua=u^\{\\top\}\\Sigma\_\{A\}u,b=u⊤​ΣB​ub=u^\{\\top\}\\Sigma\_\{B\}u, andα,β\\alpha,\\betaare the two targets withβ−α=δ\\beta\-\\alpha=\\delta\. The optimum is the curvature\-weighted meanm⋆=\(a​α\+b​β\)/\(a\+b\)m^\{\\star\}=\(a\\alpha\+b\\beta\)/\(a\+b\)with residual12​a​ba\+b​δ2\\tfrac\{1\}\{2\}\\frac\{ab\}\{a\+b\}\\delta^\{2\}\. Writing the harmonic curvatureσu=\(a−1\+b−1\)−1=a​ba\+b\\sigma\_\{u\}=\(a^\{\-1\}\+b^\{\-1\}\)^\{\-1\}=\\frac\{ab\}\{a\+b\}gives residual12​σu​δ2\\tfrac\{1\}\{2\}\\sigma\_\{u\}\\delta^\{2\}on the line; the factor14\\tfrac\{1\}\{4\}in Eq\. \([7](https://arxiv.org/html/2607.09202#S3.E7)\) is the conservative bound obtained by summing the non\-negative contributions of all shared directions and retaining the one\-dimensional term\. For the inference claim, the Bayes\-optimal single head is the posterior mean𝔼​\[y∣ϕ\]\\mathbb\{E\}\[y\\mid\\phi\]; whereϕ\\phideterminesTTit matches the oracle \(no excess\), and where it does not its excess equals the merge residual on that direction\. ∎

###### Proof of Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6)\(information–estimation identity\)\.

The task oracle predicts𝔼​\[y∣ϕ,T\]=ϕ⊤​wT∗\\mathbb\{E\}\[y\\mid\\phi,T\]=\\phi^\{\\top\}w\_\{T\}^\{\\ast\}and attains risk12​σ2\\tfrac\{1\}\{2\}\\sigma^\{2\}\. The Bayes single head isf⋆​\(ϕ\)=𝔼​\[y∣ϕ\]f^\{\\star\}\(\\phi\)=\\mathbb\{E\}\[y\\mid\\phi\], with risk12​𝔼​\[Var⁡\(y∣ϕ\)\]\\tfrac\{1\}\{2\}\\mathbb\{E\}\[\\operatorname\{Var\}\(y\\mid\\phi\)\]\. By the law of total variance,Var⁡\(y∣ϕ\)=σ2\+Var⁡\(ϕ⊤​wT∗∣ϕ\)\\operatorname\{Var\}\(y\\mid\\phi\)=\\sigma^\{2\}\+\\operatorname\{Var\}\(\\phi^\{\\top\}w\_\{T\}^\{\\ast\}\\mid\\phi\), and for binaryTT,Var⁡\(ϕ⊤​wT∗∣ϕ\)=η​\(1−η\)​\(ϕ⊤​δ\)2\\operatorname\{Var\}\(\\phi^\{\\top\}w\_\{T\}^\{\\ast\}\\mid\\phi\)=\\eta\(1\-\\eta\)\(\\phi^\{\\top\}\\delta\)^\{2\}\. Subtracting the oracle risk gives Eq\. \([8](https://arxiv.org/html/2607.09202#S3.E8)\)\. Equation \([9](https://arxiv.org/html/2607.09202#S3.E9)\) follows from the elementary pointwise bounds14​hb​\(p\)2≤p​\(1−p\)≤12​hb​\(p\)\\tfrac\{1\}\{4\}h\_\{b\}\(p\)^\{2\}\\leq p\(1\-p\)\\leq\\tfrac\{1\}\{2\}h\_\{b\}\(p\)on\[0,1\]\[0,1\], applied with the non\-negative weight\(ϕ⊤​δ\)2\(\\phi^\{\\top\}\\delta\)^\{2\}and integrated\. TheKK\-task extension Eq\. \([11](https://arxiv.org/html/2607.09202#A2.E11)\) is the same total\-variance step with the posterior covariance of the task\-conditional means; the Gini bound Eq\. \([12](https://arxiv.org/html/2607.09202#A2.E12)\) replacesp​\(1−p\)p\(1\-p\)byG​\(p\)=1−‖p‖22G\(p\)=1\-\\\|p\\\|\_\{2\}^\{2\}\. ∎

## Appendix S2Verifying the information–estimation identity

We verify Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6)on Gaussian two\-task models\. The exact identity Eq\. \([8](https://arxiv.org/html/2607.09202#S3.E8)\) matches the directly simulated single\-head excess risk𝒟⋆\\mathcal\{D\}^\{\\star\}on they=xy=xline to Monte\-Carlo precision \(maximum relative error9×10−39\\times 10^\{\-3\}\), and as identifiability varies,𝒟⋆\\mathcal\{D\}^\{\\star\}rises from≈0\\approx 0at full identifiability \(I=H​\(T\)I=H\(T\)\) toward its maximum atI=0I=0, staying inside the proven mutual\-information sandwich Eq\. \([9](https://arxiv.org/html/2607.09202#S3.E9)\) throughout \(Fig\.[S1](https://arxiv.org/html/2607.09202#A2.F1)\)\.

#### Multiclass extension\.

The identity extends toKKtasks with arbitrary priors,

𝒟K⋆=12​𝔼ϕ​\[Vart∼p\(⋅∣ϕ\)⁡\(ϕ⊤​wt∗\)\]=12​𝔼ϕ​\[∑tpt​\(ϕ\)​\(ϕ⊤​wt∗\)2−\(∑tpt​\(ϕ\)​ϕ⊤​wt∗\)2\],\\mathcal\{D\}^\{\\star\}\_\{K\}=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\phi\}\\\!\\Big\[\\operatorname\{Var\}\_\{t\\sim p\(\\cdot\\mid\\phi\)\}\\\!\\big\(\\phi^\{\\top\}w\_\{t\}^\{\\ast\}\\big\)\\Big\]=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\phi\}\\\!\\Big\[\\textstyle\\sum\_\{t\}p\_\{t\}\(\\phi\)\\,\(\\phi^\{\\top\}w\_\{t\}^\{\\ast\}\)^\{2\}\-\\big\(\\sum\_\{t\}p\_\{t\}\(\\phi\)\\,\\phi^\{\\top\}w\_\{t\}^\{\\ast\}\\big\)^\{2\}\\Big\],\(11\)with corresponding multiclass entropy bounds in terms of the Gini impurityG​\(p\)=1−‖p‖22G\(p\)=1\-\\\|p\\\|\_\{2\}^\{2\}:

1−1/K\(ln⁡K\)2​H​\(p\)2≤G​\(p\)≤12​ln⁡2​H​\(p\),\\tfrac\{1\-1/K\}\{\(\\ln K\)^\{2\}\}\\;H\(p\)^\{2\}\\ \\leq\\ G\(p\)\\ \\leq\\ \\tfrac\{1\}\{2\\ln 2\}\\;H\(p\),\(12\)where the upper constant1/\(2​ln⁡2\)1/\(2\\ln 2\)isKK\-independent\. These tight constants are verified over the simplex forK≤10K\\leq 10: the upper constant1/\(2​ln⁡2\)1/\(2\\ln 2\)is matched to2×10−102\\times 10^\{\-10\}for everyKK\(the binary posterior is the universal worst case\), and the lower constant\(1−1/K\)/\(ln⁡K\)2\(1\-1/K\)/\(\\ln K\)^\{2\}to4×10−34\\times 10^\{\-3\}, attained at the uniform posterior \(Fig\.[S2](https://arxiv.org/html/2607.09202#A2.F2)\)\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x13.png)Figure S1:Floor / mutual\-information verification\. The exact identity Eq\. \([8](https://arxiv.org/html/2607.09202#S3.E8)\) against the simulated single\-head excess risk𝒟⋆\\mathcal\{D\}^\{\\star\}, on they=xy=xline, with the floor rising as identifiability falls and remaining inside the mutual\-information sandwich \(band\) throughout\.![Refer to caption](https://arxiv.org/html/2607.09202v1/x14.png)Figure S2:Tight multiclass entropy–variance constants for the floor sandwich,K=2K=2–1010\.*Left:*maxp⁡G/H=1/\(2​ln⁡2\)\\max\_\{p\}G/H=1/\(2\\ln 2\)for everyKK\(the binary posterior is the universal worst case\)\.*Right:*minp⁡G/H2=\(1−1/K\)/\(ln⁡K\)2\\min\_\{p\}G/H^\{2\}=\(1\-1/K\)/\(\\ln K\)^\{2\}, attained at the uniform posterior\.
#### Connection to standard CL metrics\.

The per\-task forgetting of Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1),Δ​LA=12​Δ⊤​ΣA​Δ\\Delta L\_\{A\}=\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta, is exactly the per\-task term of the Forgetting MeasureFM=1K−1​∑t<KΔ​Lt\\mathrm\{FM\}=\\tfrac\{1\}\{K\-1\}\\sum\_\{t<K\}\\Delta L\_\{t\}and the negative of Backward TransferBWT=−1K−1​∑t<KΔ​Lt\\mathrm\{BWT\}=\-\\tfrac\{1\}\{K\-1\}\\sum\_\{t<K\}\\Delta L\_\{t\}\(in the squared\-loss metric\); positive backward transfer is simplyΔ​Lt<0\\Delta L\_\{t\}<0, which the signed gate permits and unconditional projection forbids\. Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6)then gives the floor under which no single\-head method can push the averageFM\\mathrm\{FM\}:𝒟⋆\\mathcal\{D\}^\{\\star\}is the achievable\-FM lower bound set by task confusability\.

## Appendix S3Theory details: similarity sign\-change and the capacity budget

#### Similarity controls the sign of sharing\.

On a shared subspace, whether to share parameters or orthogonalize is not settled by Corollary[1](https://arxiv.org/html/2607.09202#Thmcorollary1), which only forbids*lossless*sharing under conflict\. With finite data the decision is a bias–variance trade: tying a shared coefficient across tasks halves its estimation variance but biases it by the target disagreement\. Sharing lowers total population loss iff the similaritys=cos⁡∠​\(wA∗,wB∗\)s=\\cos\\angle\(w\_\{A\}^\{\\ast\},w\_\{B\}^\{\\ast\}\)on the shared block exceeds a thresholds∗s^\{\\ast\}, where the variance reduction balances the squared bias; belows∗s^\{\\ast\}, orthogonalizing wins\. This is the sign\-change measured ats∗≈0\.26s^\{\\ast\}\\approx 0\.26in Fig\.[7](https://arxiv.org/html/2607.09202#S5.F7)\.

#### Capacity is a rate–distortion budget\.

Under orthogonal allocation each task consumes a rank\-rtr\_\{t\}slice of thedd\-dimensional feature space\. Retention is lossless while the cumulative occupied rank∑trt≤d\\sum\_\{t\}r\_\{t\}\\leq d; once∑trt\>d\\sum\_\{t\}r\_\{t\}\>dthere is no subspace left orthogonal to the past, so a residual distortion turns on\. The knee at∑trt≈d\\sum\_\{t\}r\_\{t\}\\approx dis a hard capacity, and tradingddagainst the number of tasks is the geometric form of an effective\-capacity budgetN\+κ​MN\+\\kappa M\(parameters plus a per\-exemplar memory worth\), whereκ\\kappais the rank a stored direction occupies—the right panel of Fig\.[7](https://arxiv.org/html/2607.09202#S5.F7)confirms the knee location across feature dimensions\.

#### The floor is a fixed cost with method\-dependent allocation\.

In the exact regime the distortion floor is not removed by the method choice but redistributed across forgetting, deferred fit, or merge residual \(evaluate\_open\_questions\_new\.py,4848seeds\)\. Relocation is exactly zero\-sum: naive descent pays the full amount as forgetting ofAA, orthogonal allocation pays the same amount as deferred fit onBB, and the optimal merge splits it between the tasks\. Only added conflict\-free capacity reduces the bill, linearly at rate≈D/rshared\\approx D/r\_\{\\rm shared\}per relieved dimension \(D=4\.68→3\.16→1\.63→0D=4\.68\\to 3\.16\\to 1\.63\\to 0as the three shared directions are moved to private capacity\)\. Replay does not reduce the geometric floor at all: with the trueΣA\\Sigma\_\{A\}known, stored exemplars are redundant, and their measured benefit is theO​\(1/m\)O\(1/m\)excess of an*estimated*geometry \(\+2\.13\+2\.13,\+0\.36\+0\.36,\+0\.08\+0\.08aboveDDatm=10,40,160m=10,40,160exemplars\)\. Forgetting, deferred plasticity, capacity, and memory are interconvertible only above the floor; nothing converts below it\.

## Appendix S4Method comparison and offline\-merge ablation

Table[S1](https://arxiv.org/html/2607.09202#A4.T1)gives the measured numbers behind the front\-page comparison \(Fig\.[1](https://arxiv.org/html/2607.09202#S0.F1)\): nine methods on the exact\-A1 Rotated\-Digits benchmark, mean±\\pm95% CI over55seeds\. Six are online \(learn the five rotations sequentially\): naive, EWC, replay,ogd/gpm, the signed gate igfa \(s∗=0\.65s^\{\\ast\}=0\.65\), and*Annealed igfa*, which runs the first half of each task’s updates as protect\-all orthogonal projection and then relaxes to the signed gate—annealing the protection boundary as the similarity estimate stabilizes\. Three are offline merges of the five independently\-solved task heads\{wt\}\\\{w\_\{t\}\\\}\(per\-task oracle accuracy0\.9030\.903\): an*isotropic*uniform average, a*Euclidean*sign\-elect task\-arithmetic merge \(TSV/TIES\-style\), and theΣ\\Sigma\-*orthogonal*mergewJ=\(∑tΣt\)\+​\(∑tΣt​wt\)w\_\{J\}=\(\\sum\_\{t\}\\Sigma\_\{t\}\)^\{\+\}\(\\sum\_\{t\}\\Sigma\_\{t\}w\_\{t\}\)of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\(“Σ\\Sigma\-orth\. merge \(ours\)”, the whitening\-in\-the\-Σ\\Sigma\-metric version of TSV\); for the merges we also log the post\-merge residual floorD=∑t12​\(wJ−wt\)⊤​Σt​\(wJ−wt\)D=\\sum\_\{t\}\\tfrac\{1\}\{2\}\\,\(w\_\{J\}\-w\_\{t\}\)^\{\\top\}\\Sigma\_\{t\}\(w\_\{J\}\-w\_\{t\}\)\.

Three observations follow\.*\(i\) TheΣ\\Sigma\-metric is the right geometry\.*TheΣ\\Sigma\-orthogonal merge attains0\.885±0\.0120\.885\\pm 0\.012accuracy—within a point of the0\.9030\.903per\-task oracle—at residualD=0\.032D=0\.032, an order of magnitude below the isotropic \(0\.2000\.200\) and Euclidean \(0\.3820\.382\) merges, which lose1515–2121accuracy points\. Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)predicts this: minimizing the interference energy requires the task\-inducedΣ\\Sigma\-metric, and Euclidean whitening is the mis\-specified special caseΣt∝I\\Sigma\_\{t\}\\\!\\propto\\\!I\.*\(ii\) Among online buffer\- and Fisher\-free methods, the gate attains the highest accuracy\.*igfa \(0\.7710\.771\) exceedsogd\(0\.6840\.684\) and naive \(0\.7140\.714\) as in Sec\.[5\.5](https://arxiv.org/html/2607.09202#S5.SS5)\.*\(iii\) Annealing favors retention over accuracy\.*Annealed igfa \(0\.7520\.752, forgetting−0\.007\-0\.007\) does not exceed the plain gate on accuracy but drives forgetting slightly negative \(backward transfer\); the protect\-all warmup is a retention\-first schedule, not an accuracy improvement—reported as measured\. Regenerated byexperiments\_compare\.py\.

Table S1:Method comparison on Rotated\-Digits \(exact A1, mean±\\pm95% CI over55seeds\)\. Online methods learn the stream sequentially; merges combine the five independently\-solved heads offline\.DDis the post\-merge residual floor \(Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\); theΣ\\Sigma\-orthogonal merge uniquely minimizes it and attains near\-oracle accuracy\.MethodFamilyAcc\.↑\\uparrowForget\.↓\\downarrowResidualDD↓\\downarrowNaivebaseline0\.714±0\.0240\.714\\pm 0\.0240\.163±0\.0130\.163\\pm 0\.013—EWC\[[1](https://arxiv.org/html/2607.09202#bib.bib1)\]corrective0\.727±0\.0500\.727\\pm 0\.0500\.045±0\.0390\.045\\pm 0\.039—Replay \(1010/class\)corrective0\.795±0\.0120\.795\\pm 0\.0120\.042±0\.0140\.042\\pm 0\.014—ogd/gpm\[[7](https://arxiv.org/html/2607.09202#bib.bib7),[8](https://arxiv.org/html/2607.09202#bib.bib8)\]structural0\.684±0\.0350\.684\\pm 0\.035−0\.007±0\.004\-0\.007\\pm 0\.004—igfa \(gate\)ours0\.771±0\.0300\.771\\pm 0\.0300\.002±0\.0270\.002\\pm 0\.027—Annealed igfaours0\.752±0\.0360\.752\\pm 0\.036−0\.007±0\.015\-0\.007\\pm 0\.015—Isotropic mergemerging0\.730±0\.0220\.730\\pm 0\.0220\.177±0\.0200\.177\\pm 0\.0200\.2000\.200Euclidean TSV mergemerging0\.694±0\.0460\.694\\pm 0\.0460\.213±0\.0510\.213\\pm 0\.0510\.3820\.382Σ\\Sigma\-orth\. merge \(ours\)merging \(ours\)0\.885±0\.012\\mathbf\{0\.885\\pm 0\.012\}0\.022±0\.0070\.022\\pm 0\.0070\.032\\mathbf\{0\.032\}The floor prediction also holds on a real frozen backbone\. With trained heads on Split\-CIFAR\-100 over ViT\-B/16 features \(single run,kaggle\_pending\_experiments\.py\), theΣ\\Sigma\-orthogonal merge attains both the highest accuracy and—uniquely—the lowest residual floor \(0\.8620\.862acc,D=1\.53D=1\.53\), against0\.8150\.815/D=3\.68D=3\.68for the isotropic average and0\.8150\.815/D=11\.6D=11\.6for the Euclidean sign\-elect merge, with a per\-task oracle of0\.9720\.972\. A five\-seed replication with closed\-form ridge heads \(kaggle\_vit\_multiseed\.py\) confirms the theorem’s actual claim with confidence intervals: theΣ\\Sigma\-orthogonal merge uniquely minimizes the residual floor \(D=2\.02±0\.03D=2\.02\\pm 0\.03, vs3\.75±0\.023\.75\\pm 0\.02isotropic and16\.6±0\.316\.6\\pm 0\.3sign\-elect\)\. Task\-masked accuracy, however, is not monotone inDDonce the floor is small \(0\.944±0\.0060\.944\\pm 0\.006vs0\.957±0\.0070\.957\\pm 0\.007for isotropic\): Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)minimizes interference energy, and masked\-evaluation accuracy saturates before that energy reaches zero\. TheΣ\\Sigma\-metric is thus the correct merge geometry on real ViT features, with the accuracy benefit contingent on the evaluation exercising the interfering directions\. This decoupling is itself predictable: parameterizing held\-out evaluation metrics by how strongly they load the shared, interfering directions, the isotropic\-versus\-Σ\\Sigma\-orthogonal evaluation gap varies systematically with that loading \(held\-out correlation≈0\.5\\approx 0\.5, sign\-stable across4444seeds\), and an evaluation that weighs all tasks equally favors the plain average by construction—consistent with reports that isotropic merging can match or exceed orthogonalizing merges\[[25](https://arxiv.org/html/2607.09202#bib.bib25)\]\. Whether merge geometry pays is a property ofΣeval\\Sigma\_\{\\rm eval\}, decidable per benchmark\.

## Appendix S5The projection remains effective in sequential LoRA on language models

We probe the adapting\-feature regime by fine\-tuning a frozen GPT\-2 with LoRA adapters sequentially on four stylistically distinct corpora \(news, movie reviews, encyclopedic text, tweets\), measuring the rise in each domain’s held\-out loss after later domains are learned\. The results separate igfa’s two components by modality\.

*\(i\) The orthogonal projection transfers to transformers\.*Unconditional orthogonal projection \(ogd/gpm, equivalently igfa in protect\-only mode\) reduces forgetting relative to naive sequential LoRA at both training budgets \(0\.64→0\.500\.64\\\!\\to\\\!0\.50at200200steps/task,1\.21→1\.141\.21\\\!\\to\\\!1\.14at400400\), confirming that theker⁡H¯A\\ker\\bar\{H\}\_\{A\}retention mechanism is effective on a real transformer, buffer\- and Fisher\-free \(Table[S2](https://arxiv.org/html/2607.09202#A5.T2)\)\.

*\(ii\) The input\-overlap gate cannot distinguish conflict in language\.*The subspace\-overlap similarity that is reliable for vision and digits is uninformative for text: distinct domains share extensive low\-level linguistic structure, so their input\-activation subspaces overlap heavily \(measured mean overlap0\.71,0\.56,0\.280\.71,0\.56,0\.28along the stream\) regardless of output conflict\. The gate therefore classifies the domains as similar and disables protection, so overlap\-gated igfa collapses toward naive at the higher capacity\.

*\(iii\) No gate can improve on unconditional protection on dissimilar tasks\.*Beyond the input\-overlap gate we tested three further gate designs, each derived from a different part of the theory: a Gauss–Newton gate \(Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)\), a task\-vector gate \(Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\), and a threshold\-free functional controller \(Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)used directly as the control law\)\. Atr=16r=16\(Table[S3](https://arxiv.org/html/2607.09202#A5.T3)\), the Gauss–Newton \(0\.760\.76\) and functional \(0\.770\.77\) gates both improve on naive \(0\.890\.89\) and do not collapse, but neither improves on unconditionalogd\(0\.710\.71\); the task\-vector gate is unstable \(1\.851\.85\)\. The reason is fundamental and follows from the distortion\-floor identity \(Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)\): on dissimilar tasks the supports are near\-disjoint, so the distortion floor is≈0\\approx 0andogdalready attains it—no transfer remains for a gate to exploit\. At billion\-parameter scale on a frozenpythia\-1b\(Table[S4](https://arxiv.org/html/2607.09202#A5.T4)\), the recursive\-subspace igfa*improves on*ogdon dissimilar tasks on both loss \(5\.165\.16vs5\.515\.51\) and forgetting \(1\.701\.70vs2\.102\.10\)—the capacity advantage of Fig\.[11](https://arxiv.org/html/2607.09202#S5.F11)appearing in a real LLM—while on similar tasks the sharing gate still raises forgetting\. In summary, on LLMs the structural projection \(and its recursive Gauss–Newton tracking\) carries over and provides replay\- and Fisher\-free retention; a gate that succeeds on*similar*language tasks remains an open problem\.

#### A candidate similarity signal for text\.

The failure above is a signal problem: input\-subspace overlap is uninformative when all domains share low\-level structure\. An exact\-regime study of precisely this confound—input overlap fixed at1\.01\.0while target similarity varies \(evaluate\_next\_steps\.py\)—shows that a*probe*signal, a few\-shot head fit on the incoming task compared to the stored head by signed cosine and thresholded at the theory’s break\-even value, recovers near\-oracle gating where the input gate collapses to share\-all and pays the full conflict bias \(mean loss3\.213\.21for the input gate under conflict, versus2\.202\.20for the probe gate and1\.951\.95for the oracle\)\. The probe’s decisions are noisy exactly when data are scarce \(5656–82%82\\%agreement with the oracle here\), so this is a validated candidate rather than a solved problem; but it requires no extra state and is modality\-agnostic, because it reads the functional instead of the input geometry\.

#### The language gate, resolved in function space\.

The function\-space formalism \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\) dissolves the problem: the gate should never have read input geometry at all\. The correct, modality\-free signal is the*interference rate*⟨∇Lu,g⟩\\langle\\nabla L\_\{u\},g\\rangleof the new updateggagainst each stored domainuu—a quantity that includes the targets, which is precisely what input overlap misses on text—and the gating rule is a threshold\-free sign test with a continuous\-time monotone\-retention guarantee \(Sec\.[S15](https://arxiv.org/html/2607.09202#A15)\)\. Its gradient ofLuL\_\{u\}can be estimated from a micro\-cache of a few thousand tokens per domain, from the anchor construction \(Sec\.[S9](https://arxiv.org/html/2607.09202#A9)\), or from the dream state \(Sec\.[S14](https://arxiv.org/html/2607.09202#A14)\)\. The gate is validated in the exact regime and on jointly trained networks \(near current\-Gauss–Newton retention under conflict, near\-naive adaptation under transfer, at one extra gradient per step\)\. On real text it delivers a measured gain \(GPT\-2, sequential LoRA on news→\\toIMDB→\\towikitext→\\totweets, five seeds;kaggle\_functional\_gate\_v2\.py,func\_gate\_v2\_results\.json\): the sign gate significantly improves on naive on both axes—average final loss3\.981±0\.0023\.981\\pm 0\.002against4\.001±0\.0064\.001\\pm 0\.006\(pairedp=0\.003p=0\.003, five of five seeds\), forgetting\+0\.294±0\.005\+0\.294\\pm 0\.005against\+0\.324\+0\.324\(p=0\.0007p=0\.0007\)—and on the input gate by a similar margin, while edging unconditional protection \(protect\-all3\.9843\.984/\+0\.300\+0\.300\) consistently but*not*significantly at five seeds \(p≈0\.19p\\approx 0\.19–0\.230\.23\)\. Two checks come back negative\. First, the earlier three\-seed advantage over protect\-all does not survive to five seeds as a significant effect; the robust claim is a significant win over naive and the input gate, and parity\-to\-slight\-edge over protect\-all\. Second, making the sign test Adam\-aware—projecting the*preconditioned*updateP⊙gP\\odot gorthogonal to∇Lu\\nabla L\_\{u\}rather than the raw gradient—does not improve on the raw\-gradient gate \(final3\.9833\.983,p=0\.40p=0\.40\), so Adam’s preconditioning is not the limiting factor on the modest margins\. The measured interference\-rate map nonetheless locates the mechanism: news↔\\leftrightarrowIMDB and tweets↔\\leftrightarrowIMDB carry positive transfer \(\+0\.07\+0\.07,\+0\.06\+0\.06\) that protect\-all forfeits, while wikitext conflicts with IMDB \(−0\.03\-0\.03\), which naive and the input gate pay for\. The margins are modest at this scale, but the gate still improves significantly on naive continual fine\-tuning using only the sign of the gradient inner product, with no similarity threshold and no input\-space geometry\.

The result replicates at four times the scale on a second architecture\. Onpythia\-410m\(gpt\-neox, sequential LoRA on the same four\-domain stream, three seeds;func\_gate\_v2\_results\_pythia410m\_lr1e\-4\.json\), the Adam\-aware sign gate again improves on naive on both axes—final loss3\.605±0\.0023\.605\\pm 0\.002against3\.624±0\.0013\.624\\pm 0\.001\(pairedp=0\.012p=0\.012\) and forgetting\+0\.297±0\.005\+0\.297\\pm 0\.005against\+0\.321±0\.003\+0\.321\\pm 0\.003\(p=0\.038p=0\.038\), in three of three seeds on both metrics—and is ahead of unconditional protection \(3\.6163\.616/\+0\.316\+0\.316\) in all three seeds on both axes, though not significantly atn=3n=3\(p≈0\.12p\\approx 0\.12–0\.200\.20\), mirroring the GPT\-2 pattern\. One methodological finding from the same runs is worth recording: at a learning rate of10−510^\{\-5\}the identical stream produced near\-zero forgetting \(0\.0030\.003\) and all methods tied within0\.010\.01—a vacuous comparison\. The gate has value only where interference exists, which is precisely the pre\-flight feasibility point: measure whether the stream forgets before deploying—or crediting—any mitigation\.

A vision test of the same gate confirms it fires only when head gradients overlap \(five seeds per stream;kaggle\_sign\_gate\_vit\.py,sign\_gate\_vit\_results\.json\)\. On the masked task\-incremental ViT streams \(Split\-CIFAR\-100, ImageNet\-R, CUB\) the gate is*exactly*inert: with class masks, the current and stored tasks’ gradients occupy disjoint rows of the head, the inner product is identically zero, and the gated run is bitwise equal to naive—correctly detecting that head interference is structurally absent, where the subspace gates pay1\.21\.2–2\.32\.3accuracy points for protection nothing needed\. On the dense\-conflict superclass stream the gate fires and improves on naive on both axes \(p≤0\.002p\\leq 0\.002\) but recovers only∼18%\{\\sim\}18\\%of subspace protection’s gain \(accuracy0\.7500\.750versus igfa’s0\.7860\.786\): a single sampled constraint per step under\-protects when every task conflicts with every other\. The gate therefore complements rather than replaces the threshold gate—sparse or moderate conflict without an input\-geometry signal is its regime \(language\), dense conflict belongs to subspace projection, and the stream’s shared density selects the tool in advance, which is the deployment rule of the Method section\.

The under\-enforcement just described is not a ceiling of the approach but a property of*sampling*one constraint per step, and Corollary[2](https://arxiv.org/html/2607.09202#Thmcorollary2)prescribes the repair: enforce the full active set\. Testing that prescription on thepythia\-410mstream \(all past domains’ micro\-cache gradients constrained per step via the corollary’s nonnegative least squares;\+21%\+21\\%wall\-clock over the sampled gate; three seeds\) yields the strongest language result in this paper: final loss3\.536±0\.0053\.536\\pm 0\.005and forgetting\+0\.206±0\.005\+0\.206\\pm 0\.005, improving on naive \(3\.6253\.625/\+0\.324\+0\.324\), unconditional protection \(3\.6173\.617/\+0\.319\+0\.319\), and the sampled sign gate \(3\.6053\.605/\+0\.300\+0\.300\) on*both*axes simultaneously—a36%36\\%reduction in forgetting relative to naive together with the best adaptation of any arm\. An independent re\-run with archived per\-seed records reproduces the result across sessions \(3\.540±0\.0063\.540\\pm 0\.006/\+0\.208±0\.009\+0\.208\\pm 0\.009\) and supplies exact paired tests: the active\-set gate improves on every other arm on both axes in three of three seeds, allp≤0\.024p\\leq 0\.024\(versus the sampled gatep=0\.023p=0\.023/0\.0120\.012; versus unconditional protectionp=0\.0009p=0\.0009/0\.00080\.0008; versus naivep=0\.0027p=0\.0027/0\.00360\.0036\)\. The same re\-run is a caution on the*sampled*gate: its edge over naive is session\-fragile there \(p=0\.08p=0\.08–0\.170\.17\), so the robust finding at scale is the active set’s dominance, not the single\-probe margin\. The sampled gate’s modest margins were therefore under\-enforcement of the monotone\-retention QP, as the exact\-regime analysis predicted\. The committed vision prediction was then run \(kaggle\_sign\_gate\_vit\.py,func\-activearm, five seeds\) and is adjudicated as follows\. On the three masked task\-incremental streams the active\-set gate is*exactly*inert—bitwise equal to naive, as the theory requires—retaining the accuracy advantage over the subspace gates \(p≤0\.046p\\leq 0\.046\)\. On the dense\-conflict superclass stream it recovers54±8%54\\pm 8\\%of subspace projection’s accuracy gain \(forgetting:44±5%44\\pm 5\\%\), up from the sampled gate’s20±3%20\\pm 3\\%\(p<10−4p<10^\{\-4\}\)—the predicted repair, at roughly the predicted factor—while projection itself stays ahead \(0\.7880\.788versus0\.7670\.767,p=0\.006p=0\.006\): with two\-exemplar micro\-cache constraints the QP is limited by constraint estimation noise, consistent with its100%100\\%recovery under exact constraints in the exact regime\. Dense conflict with weak constraint estimates therefore still belongs to subspace projection, and the scoping conclusion above stands with the gap halved\.

Running the earlier probe protocol on real text \(GPT\-2, sequential LoRA on four AG\-News topic domains, three seeds;kaggle\_probe\_gate\_llm\.py,probe\_gate\_results\.json\) yields three findings\. First, the probe signal transfers: the probe cosines form a stable, seed\-reproducible task\-relationship map \(adjacent topics≈\+0\.10\\approx\+0\.10, world versus sci/tech≈−0\.18\\approx\-0\.18\) precisely where input overlap is uninformative\. Second, the map shows that these topic slices are*not*a similar\-task stream in the functional sense: every pairwise similarity sits below the share threshold, so the gate correctly stays protective and all four methods tie within noise \(final loss3\.783\.78–3\.793\.79; the probe gate attains the lowest forgetting in all three seeds,0\.0540\.054versus0\.0570\.057for protect\-all, not significant atn=3n=3\)\. Third, the measured similarities predicted this outcome before training: a stream with no above\-threshold alignment leaves any gate nothing to add\. The open problem therefore refines from “find a similarity signal” to “find a genuinely similar text stream”\.

Table S2:Sequential LoRA fine\-tuning of GPT\-2 on four dissimilar text domains \(forgetting==mean held\-out loss rise on earlier domains, lower better; LoRAr=8r=8; single run\)\. Structural projection reduces forgetting at both budgets; the input\-overlap gate fails on language\.Table S3:Four gate designs on the sequential\-LoRA stream \(GPT\-2, four dissimilar domains,r=16r=16,250250steps/task; forgetting, lower better; single run\)\. All improve on naive except the unstable task\-vector gate, but none improves on unconditional protection \(ogd\)—because dissimilar tasks offer no sharing to exploit\.Table S4:Sequential LoRA onpythia\-1b\(11B parameters, 4\-bit\), four dissimilar and four similar text domains \(avg\. loss and forgetting, lower better\)\. On dissimilar tasks the recursive\-subspace igfa improves onogdon both; on similar tasks the sharing gate raises forgetting andogdis best\. \(Single run\.\)

## Appendix S6Choosing and self\-calibrating the thresholds∗s^\{\\ast\}

The gate’s thresholds∗s^\{\\ast\}is the only free parameter that distinguishes igfa fromogd\(s∗=∞s^\{\\ast\}=\\infty\)\. Ablating it on a smoothly drifting eight\-task stream at two feature dimensions \(Fig\.[S3](https://arxiv.org/html/2607.09202#A6.F3)\) shows the optimum is a broad plateau \(s∗≈0\.55s^\{\\ast\}\\approx 0\.55–0\.80\.8\), not a sharp peak, and that the curves atd=64d=64andd=256d=256are nearly identical, so the optimum is insensitive to feature dimension\. Because the break\-even similarity is the operative quantity, the gate can dispense withs∗s^\{\\ast\}and measure it: for each task, greedily pool an earlier task only if doing so lowers held\-out validation error\. On a drifting stream whose similarity persistenceρ\\rhovaries, this validation\-greedy rule tracks the per\-stream oracle to within0\.050\.05in average error, while a single fixeds∗s^\{\\ast\}errs by up to0\.340\.34where it is mis\-specified \(Fig\.[S3](https://arxiv.org/html/2607.09202#A6.F3), right\)\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x15.png)

![Refer to caption](https://arxiv.org/html/2607.09202v1/x16.png)

Figure S3:The similarity threshold\.*Left:*average population error on a drifting eight\-task stream versuss∗s^\{\\ast\}, at two feature dimensions; the optimum is a broad dimension\-insensitive plateau \(vertical dashed lines mark each dimension’s optimum\), and onlys∗→1s^\{\\ast\}\\\!\\to\\\!1\(ogd\-like isolation\) is sharply suboptimal on similar tasks\.*Right:*self\-calibration removes the free parameter—the validation\-greedy dynamic rule tracks the per\-stream oracle within0\.050\.05across all similarity\-persistence valuesρ\\rho, avoiding the fixed threshold’s failure where it is mis\-specified\.
## Appendix S7Capacity limits degrade gracefully

When the rank capkmaxk\_\{\\max\}binds, igfa overwrites the smallest\-eigenvalue occupied direction\. Because forgetting from removing a direction is its energy12​λi​\(ui⊤​wA\)2\\tfrac\{1\}\{2\}\\lambda\_\{i\}\(u\_\{i\}^\{\\top\}w\_\{A\}\)^\{2\}\(Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)\), dropping the lowest\-energy direction costs the least\. Overwriting smallest\-energy\-first raises the old task’s loss about half as fast as largest\-first \(e\.g\.2\.02\.0vs4\.34\.3at the half\-way point\) in a gradual degradation \(Fig\.[S4](https://arxiv.org/html/2607.09202#A7.F4)\)\. In modern backbones, where a task’s effective rank is tiny relative todd\(Fig\.[9](https://arxiv.org/html/2607.09202#S5.F9)showed≈2\\approx 2out of300300\), almost all overwriting hits near\-zero\-energy directions, so the induced forgetting is negligible\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x17.png)Figure S4:Graceful degradation under the capacity cap\. Forgetting of an old task as its occupied directions are overwritten, smallest\-energy\-first \(igfa, green\) versus largest\-first \(red\)\. Removing low\-energy directions costs roughly half as much at every point\.
## Appendix S8A pre\-flight feasibility diagnostic

The distortion\-floor identity \(Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)\) bounds class\-incremental accuracy byI​\(T;ϕ\)I\(T;\\phi\); we derive from it a low\-cost pre\-flight test\. By Fano,I​\(T;ϕ\)I\(T;\\phi\)is governed by the error of the best task\-classifier onϕ\\phi, so we fit a linear task\-probe on the frozen features—one logistic regression, no continual\-learning run—and obtainI^=ln⁡2−hb​\(probe error\)\\hat\{I\}=\\ln 2\-h\_\{b\}\(\\text\{probe error\}\)\. Across two\-task Gaussian models of varying separation \(Fig\.[S5](https://arxiv.org/html/2607.09202#A8.F5)\),I^\\hat\{I\}tracks the trueI​\(T;ϕ\)I\(T;\\phi\)at Pearsonr=0\.99r=0\.99, and the distortion floor decreases monotonically inI^\\hat\{I\}\(Spearman0\.970\.97\): the probe forecasts the floor before training\. Combined with the closed\-form constants of Eq\. \([12](https://arxiv.org/html/2607.09202#A2.E12)\),I^\\hat\{I\}yields a numeric floor band, not just a ranking\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x18.png)Figure S5:Pre\-flight feasibility diagnostic\.*Left:*a low\-cost linear task\-probe on frozen features estimatesI​\(T;ϕ\)I\(T;\\phi\)atr=0\.99r=0\.99to the truth \(no CL run\)\.*Right:*the distortion floor decreases monotonically in the pre\-flight estimateI^\\hat\{I\}\(Spearman0\.970\.97\)\.
## Appendix S9Low\-cost online curvature: Frequent Directions

The recursive Gauss–Newton estimate used a decayedd×dd\\times dcovariance and an eigendecomposition—𝒪​\(d2\)\\mathcal\{O\}\(d^\{2\}\)memory,𝒪​\(d3\)\\mathcal\{O\}\(d^\{3\}\)per task\. Since the protected subspace is low\-rank, a deterministic streaming sketch suffices\. Replacing the full covariance with Frequent Directions\[[32](https://arxiv.org/html/2607.09202#bib.bib32)\]\(anℓ×d\\ell\\times dsketch with a bounded\-error guarantee\) recovers the top\-rrsubspace almost exactly atℓ=2​r\\ell=2r—captured energy0\.999660\.99966versus the full estimator’s0\.999720\.99972—and beats streaming PCA \(Oja,0\.9580\.958\) per unit memory \(Fig\.[S6](https://arxiv.org/html/2607.09202#A9.F6)\)\. The estimator is thus𝒪​\(r​d\)\\mathcal\{O\}\(rd\)memory with nod×dd\\times deigendecomposition\.

The same holds at real scale\. On a frozenpythia\-160mcontinually pretrained with LoRA over three text domains \(single run,kaggle\_pending\_experiments\.py\), the Frequent\-Directions sketch captures0\.9730\.973of the full\-covariance energy at61,44061\{,\}440floats—about10×10\\timesless than the fulld×dd\\times destimator \(589,824589\{,\}824floats\)—and, being a*stable deterministic*sketch rather than a decayed moving average, yields the*lowest*first\-domain forgetting of the three online estimators: the domain\-0 loss rises by only\+0\.55\+0\.55then\+0\.68\+0\.68over the next two domains, versus\+1\.28\+1\.28then\+1\.58\+1\.58for the full covariance and a diverging\+0\.33\+0\.33then\+2\.86\+2\.86for Oja streaming PCA\. Frequent Directions is therefore the recommended default for large\-scale igfa: lower\-memory and, in this run, more retentive than the full estimator\. The full layer\-wise pipeline \(kaggle\_H2\_H3\_H12\.py\) confirms both the memory saving and the layer\-dependence of capacity: on a frozen1212\-layer transformer the FD sketch is12×12\\timessmaller than full\-covariance tracking \(and half the AdamW optimizer state\), and the per\-layer protected\-rank budget emerges automatically from each layer’s trace ratio—shallow layers stay unconstrained \(kℓ=0k\_\{\\ell\}=0\) while deep layers are capped near their effective rank \(kℓ≈5k\_\{\\ell\}\\\!\\approx\\\!5–77\), with9393–96%96\\%of the deep\-layer gradient routed into the protected null space and only a modest first\-domain loss rise \(\+0\.27\+0\.27mean\)\.

#### The sketch survives drift\.

The two remedies of this paper—re\-estimation against drift \(Sec\.[5\.7](https://arxiv.org/html/2607.09202#S5.SS7)\) and sketching against cost—compose\. Under the rotating\-geometry stream, a*decayed*Frequent\-Directions sketch \(ℓ=2​r\\ell=2rrows,2​r​d=4802rd=480floats atd=60d=60\) tracks the current subspace as faithfully as the full recursive covariance \(d2=3600d^\{2\}=3600floats\): mean fidelity0\.880\.88versus0\.850\.85over ten seeds, both holding the protected rank at the true value while stale per\-task protection fills the space \(evaluate\_next\_steps\.py\)\. Sketched curvature is a viable protected geometry under drift, not only a cheaper one\.

#### Anchor protection: the dual construction\.

The protector can also be built with no covariance at all:m≥rm\\geq rstored features of the old task spanrange⁡ΣA\\operatorname\{range\}\\Sigma\_\{A\}for samples in general position, so projecting updates off their orthonormalization*is*the eigenbasis protection\. Withd=256d=256features and aC=400C=400\-output head,1010anchor features give the same retention as the eigendecomposition \(both at numerical zero, versus naive forgetting5\.85\.8\) at a7×7\\timescheaper construction with nod×dd\\times dcovariance; the cost is independent of the output count, because the protection is a set of function\-value constraints \(W​ϕiW\\phi\_\{i\}fixed\) living in sample space\. This is the practical route for very large heads and vocabularies\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x19.png)Figure S6:Cheap online Gauss–Newton subspace\. Frequent Directions \(green\) recovers the full estimator’s top\-rrsubspace energy \(grey dashed\) atℓ=2​r\\ell=2rrows,𝒪​\(ℓ​d\)\\mathcal\{O\}\(\\ell d\)memory, and beats Oja streaming PCA \(red\) per memory\.

## Appendix S10Soft gating and the capacity coordinate

Under the soft functional controller \(keep fractionkeepj=1/\(1\+β​λj\)\\mathrm\{keep\}\_\{j\}=1/\(1\+\\beta\\lambda\_\{j\}\)\) each direction is only partially protected, so capacity is consumed at rate∑j\(1−keepj\)\\sum\_\{j\}\(1\-\\mathrm\{keep\}\_\{j\}\)per task\. The right capacity coordinate is therefore the effective occupied rankreff=∑j\(1−keepj\)r\_\{\\rm eff\}=\\sum\_\{j\}\(1\-\\mathrm\{keep\}\_\{j\}\), and the hard schedule knees atreff≈dr\_\{\\rm eff\}\\approx d\. Soft gating stays flat well past the hard knee because it spendsreffr\_\{\\rm eff\}slowly—after the same number of tasks it has consumed only a fraction of the budget—at the price of a small continuous leakage from partial protection \(Fig\.[S7](https://arxiv.org/html/2607.09202#A10.F7)\)\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x20.png)Figure S7:Soft gating and the effective capacity coordinate\.*Left:*hard allocation knees sharply atT=d/rT=d/r; the soft functional controller stays flat far longer\.*Right:*plotted againstreff=∑j\(1−keepj\)r\_\{\\rm eff\}=\\sum\_\{j\}\(1\-\\mathrm\{keep\}\_\{j\}\), the hard schedule knees atreff≈dr\_\{\\rm eff\}\\approx d, while the soft schedules consume the budget far more slowly at the price of a small continuous leakage from partial protection\.
## Appendix S11A per\-direction gate dominates the per\-task gate

The benchmark gate decides share\-or\-orthogonalize per task\. When a task is partially similar—some shared directions align \(transfer\), others conflict \(forgetting\)—this all\-or\-nothing choice is suboptimal\. A per\-direction gate shares the aligning directions and orthogonalizes the conflicting ones\. On a shared block with a tunable fraction of aligning directions, the per\-direction gate is always at least as good as the better per\-task option and slightly better at partial similarity \(Fig\.[S8](https://arxiv.org/html/2607.09202#A11.F8)\)\. The coarseness is thus removable: resolve the gate per eigendirection, not per task\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x21.png)Figure S8:Per\-direction vs per\-task gating on a shared block, against the fraction of shared directions that align\. The per\-direction gate \(green\) follows the lower envelope—matching protect\-all’s robustness and share\-all’s transfer where each is right—while per\-task share\-all \(purple\) is catastrophic at low alignment\.
## Appendix S12Billion\-parameter continual pretraining

We run the three scale\-sensitive pieces together on a frozenpythia\-1b\(11B parameters, 4\-bit\) with LoRA, continually pretrained on a boundary\-free stream of six text domains, with the protected subspace tracked by Frequent Directions \(Sec\.[S9](https://arxiv.org/html/2607.09202#A9)\) and the pre\-flight diagnostic \(Sec\.[S8](https://arxiv.org/html/2607.09202#A8)\) run first\. The diagnostic reports a six\-way domain\-probe accuracy of0\.850\.85\(I^≈1\.52\\hat\{I\}\\approx 1\.52of a maximal1\.791\.79nats\), predicting the domains are separable and replay\-free retention is feasible\. The run confirms it \(Table[S5](https://arxiv.org/html/2607.09202#A12.T5)\): the replay\- and Fisher\-free igfa attains both the lowest average loss \(4\.284\.28vs naive4\.544\.54, replay4\.594\.59\) and the least first\-domain forgetting after the full stream \(\+0\.80\+0\.80vs naive\+1\.10\+1\.10, replay\+0\.91\+0\.91\), carrying only a low\-rank subspace as state\. The domains are mutually dissimilar, so igfa here runs as pure structural projection\. The replay baseline is deliberately small \(1616exemplars/domain\); the supported claim is replay\-free retention that improves on naive and matches or slightly exceeds a small replay baseline, with no buffer\.

Table S5:Continual pretraining ofpythia\-1b\(11B parameters, 4\-bit\)\+\+LoRA on six dissimilar text domains, boundary\-free, with a Frequent\-Directions\-tracked subspace\. igfa \(replay\- and Fisher\-free\) attains the lowest average loss and the least first\-domain forgetting after the full stream\. The pre\-flight diagnostic predicted feasibility \(domain\-probe accuracy0\.850\.85\)\.
## Appendix S13Extensions and generality

The interference functional applies beyond sequential continual learning\. We give eight exact\-A1 proofs\-of\-concept on synthetic constructions \(the federated, out\-of\-distribution, task\-free, and multimodal results average over2020seeds;evaluate\_hypotheses\.py,evaluate\_applications\.py,evaluate\_open\_questions\_new\.py\)\. These are directional demonstrations of scope, not benchmarks\.

To keep ambition separate from proof \(Tier IV of the claim map, Sec\.[6](https://arxiv.org/html/2607.09202#S6)\), each extension carries one of three labels\.\[proved\-A1\]: an exact consequence of the identity in the frozen\-feature regime, holding wherever A1 holds—federatedΣ\\Sigma\-merging, the out\-of\-distribution projection\-ratio signal, physics\-informed constraints, Fisher\-metric policy gradients, and floor\-minimizing meta\-learning\.\[mechanism\-probe\]: an empirical demonstration that the predicted mechanism operates, without a proof of optimality—task\-free boundary detection, the memory\-and\-control instances \(trust region, consolidation, floor\-scheduled and dream replay\), and the real\-backbone results\.\[conjecture\]: a construction whose full form needs work the paper does not complete—cross\-modalΣv​l\\Sigma\_\{vl\}gating and the biological floor\-scheduling prediction\. The label prefixes each result below\.

#### Federated learning isΣ\\Sigma\-merging\.

Non\-IID clients are tasks with different input geometries, and the weight divergence that FedAvg suffers coincides with the merge floor of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)\. Replacing the FedAvg mean with theΣ\\Sigma\-orthogonal aggregatewJ=\(∑iΣi\)\+​\(∑iΣi​wi\)w\_\{J\}=\(\\sum\_\{i\}\\Sigma\_\{i\}\)^\{\+\}\(\\sum\_\{i\}\\Sigma\_\{i\}w\_\{i\}\), where\(⋅\)\+\(\\cdot\)^\{\+\}is the Moore–Penrose pseudo\-inverse \(theΣi\\Sigma\_\{i\}are low\-rank, so∑iΣi\\sum\_\{i\}\\Sigma\_\{i\}is generically singular and a plain inverse would not exist\)—each client transmitting\(Δi,Σi\)\(\\Delta\_\{i\},\\Sigma\_\{i\}\), or a low\-rank sketch—lowers the mean client excess loss on non\-IID data in*every*one of2020seeds \(mean1\.56×1\.56\\times, minimum1\.14×1\.14\\times\)\. Federated aggregation and continual allocation are the sameΣ\\Sigma\-metric optimum\.

#### The subspace summary is a free out\-of\-distribution signal\.

A minibatch whose gradient lies outside the occupied subspaceUUis structurally disjoint from the training distribution\. The projection ratio‖U​U⊤​g‖/‖g‖\\\|UU^\{\\top\}g\\\|/\\\|g\\\|is1\.001\.00for in\-distribution batches and0\.39±0\.050\.39\\pm 0\.05for out\-of\-distribution batches \(Cohen’sd≈18d\\approx 18, minimum1313across seeds\), so a single threshold flags OOD inputs with no auxiliary model—an uncertainty signal obtained directly from the continual\-learning state\.

#### Fully autonomous, task\-free igfa\.

In a label\-free stream the velocity of the recursive Gauss–Newton eigenbasis—the Frobenius commutator‖Ut​Ut⊤​Ut−1​Ut−1⊤−Ut−1​Ut−1⊤​Ut​Ut⊤‖F\\\|U\_\{t\}U\_\{t\}^\{\\top\}U\_\{t\-1\}U\_\{t\-1\}^\{\\top\}\-U\_\{t\-1\}U\_\{t\-1\}^\{\\top\}U\_\{t\}U\_\{t\}^\{\\top\}\\\|\_\{F\}—spikes exactly at distribution shifts, recovering randomly placed boundaries at recall and precision1\.001\.00over2020streams \(decayγ=0\.6\\gamma=0\.6\)\. Composing this signal with the out\-of\-distribution guard above removes task labels entirely: the learner declares a boundary on a velocity spike, freezes the*pre\-spike*subspace to protect the just\-finished task, skips out\-of\-distribution/noise batches by their orthogonal\-and\-extreme gradient signature, and otherwise runs the gate\. On clean streams this fully autonomous igfa*matches the oracle*provided with the true boundaries \(excess loss0\.0000\.000vs0\.0000\.000; naive0\.3650\.365\); under injected extreme\-magnitude OOD noise it holds at0\.0330\.033while the same oracle—lacking the guard—is corrupted past10310^\{3\}, skipping97%97\\%of noise batches at0\.050\.05false task\-initializations per stream\. Boundary detection, out\-of\-distribution rejection, and gated allocation are thus one self\-regulating loop, with no human\-provided task boundaries\.

A velocity spike, however, presumes a*sharp*shift; a smoothly rotating stream never spikes\. The full label\-free controller therefore uses two complementary signals \(evaluate\_autonomous\_cl\.py\)\.*Novelty persistence*: a batch whose energy lies largely outside the current working subspace is held, not adapted; if the novelty persists it is a genuine task \(consolidate the old task as a protected anchor, then adapt the new one\), and if it subsides it was an out\-of\-distribution blip \(dropped\)—the persistence is exactly what separates a shift from noise, since both look novel for a single batch\.*Working\-subspace drift*: a boundary is also declared when the tracked subspace rotates away from the last consolidation, which catches gradual shifts that never spike\. Consolidated anchors are capped and the closest pairΣ\\Sigma\-merged when the budget is exceeded, so per\-step cost is independent of stream length\. Over twelve seeds against the oracle, the controller matches it on clean shifts \(detectionF1=1\.00F\_\{1\}=1\.00, retained excess loss0\.0010\.001versus0\.0010\.001; naive0\.6070\.607\), matches or beats it under noise \(F1=0\.96F\_\{1\}=0\.96,0\.0030\.003versus0\.0220\.022—it skips the outliers the oracle adapts on; naive0\.7220\.722\), and degrades gracefully under gradual drift \(F1=0\.81F\_\{1\}=0\.81,0\.2740\.274versus0\.0010\.001; naive0\.8150\.815\), the inherently hard case, detected with the correct latency since a smooth shift is unobservable until it accumulates\. The exact\-regime synthetic demonstration leaves a real unlabeled language stream open, but the operating assumption of known boundaries is, in this regime, removed rather than merely gestured at\.

#### Boundary\-free continual learning as a memory\-metric flow\.

The controller above still carries thresholds \(a novelty cutoff, a persistence count, a drift threshold\)\. A differential\-geometric reformulation removes them entirely, deriving autonomous operation from the geometry rather than imposing it\. Drop the notion of a task: a continuous data path𝒟t\\mathcal\{D\}\_\{t\}induces an instantaneous Gauss–Newton metricGt=𝔼x∼𝒟t​\[∇f​∇f⊤\]G\_\{t\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{t\}\}\[\\nabla f\\nabla f^\{\\top\}\], accumulated into a leaky*memory metric*M˙t=Gt−λ​Mt\\dot\{M\}\_\{t\}=G\_\{t\}\-\\lambda M\_\{t\}, and learning is the*memory\-natural\-gradient flow*

θ˙=−\(Mt\+ε​I\)−1​∇Lt\.\\dot\{\\theta\}=\-\(M\_\{t\}\+\\varepsilon I\)^\{\-1\}\\nabla L\_\{t\}\.\(13\)Fresh directions \(smallMM\-eigenvalue\) learn at rate≈η\\approx\\\!\\eta; occupied directions \(eigenvaluess, i\.e\. past experience\) are damped byε/\(ε\+s\)\\varepsilon/\(\\varepsilon\+s\), so Eq\. \([13](https://arxiv.org/html/2607.09202#A13.E13)\) is the graded, boundary\-free generalization of the projection gate, of which the hard gate\(I−Q​Q⊤\)\(I\-QQ^\{\\top\}\)is theε→0\\varepsilon\\\!\\to\\\!0spectral limit\. Past losses become*adiabatic invariants*: for any past sliceAA, the chain rule gives exactlyd​LA/d​t=−∇LA⊤​\(Mt\+ε​I\)−1​∇LtdL\_\{A\}/dt=\-\\nabla L\_\{A\}^\{\\top\}\(M\_\{t\}\+\\varepsilon I\)^\{\-1\}\\nabla L\_\{t\}, which is0when the current gradient is fresh with respect toAA\(Σt​ΣA=0\\Sigma\_\{t\}\\Sigma\_\{A\}=0; verified to4×10−164\\times 10^\{\-16\}\) and otherwise the interference measured in theM−1M^\{\-1\}inner product—the continuous form of12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta\. This framing is the learning\-dynamics counterpart of energy\-conserving reduced\-order models on structure\- preserving submanifolds\[[34](https://arxiv.org/html/2607.09202#bib.bib34)\]\. Four properties are validated \(evaluate\_memory\_metric\_flow\.py\)\.*\(i\)*On a continuously drifting stream with no boundaries, the flow drives a probe’s loss down and then*holds it flat*\(0\.52→0\.05→0\.080\.52\\\!\\to\\\!0\.05\\\!\\to\\\!0\.08across five later regimes\) while naive learning loses it \(→0\.63\\to 0\.63\)\.*\(ii\)*It beats naive on gradual, sharp, and noisy streams uniformly, with no per\-regime special\-casing \(retained loss0\.250\.25/0\.120\.12/0\.260\.26versus0\.760\.76/0\.220\.22/0\.940\.94; ten seeds\), and beats hard\-gating every past subspace \(1\.671\.67\), which over\-constrains overlapping tasks the soft flow trades off\.*\(iii\)*The single leakλ\\lambdais a principled retention–plasticity control: plasticity rises monotonically withλ\\lambdawhile retention is optimized at an interior value where the memory timescale1/λ1/\\lambdamatches the drift rate\.*\(iv\)*A reduced\-order metric \(top\-rreigenspace with a Woodbury inverse\) reproduces the full flow at𝒪​\(d​r\)\\mathcal\{O\}\(dr\)cost \(\|diff\|=0\.02\|\{\\rm diff\}\|=0\.02\)—the structure\-preserving low\-dimensional submanifold carries the whole dynamics\. Boundary detection, OOD rejection \(a transient adds a decaying rank\-one term toMM\), capacity \(the spectrum ofMM, released byλ\\lambda\), and gating thus collapse into one object\. The flow is natural\-gradient/K\-FAC\-shaped, but lifting it to a deep network as a running Kronecker\-factored preconditioner*fails*\(kaggle\_memory\_flow\_deepnet\.py; 10\-task Split\-CIFAR\-100, single seed\)\. Swept overε∈\{0\.3,1,3,10\}\\varepsilon\\in\\\{0\.3,1,3,10\\\}, the preconditioned flow either freezes after the first task or forgets like naive and never protects a past task, reaching0\.190\.19average accuracy at best against0\.250\.25for a plain EWC penalty \(naive0\.150\.15\); EWC alone holds old\-task accuracy above the chance floor\. The obstruction is geometric: the top Kronecker directions carry≈99%\\approx\\\!99\\%of each layer’s factor trace and coincide with the shared low\-level features, so the occupied subspace*is*the trunk that every later task reuses, and damping it removes plasticity instead of isolating interference\. A preconditioner on a shared metric cannot separate the two; reaching depth needs a penalty formulation \(the memory metric asFF\) or genuinely task\-specific subspaces\.

#### Plasticity preservation: gating meets dynamical isometry\.

Interference gating decides*where*an update may act but does not keep the allowed directions*learnable*: under sustained non\-stationarity a network also loses plasticity as its layer Jacobians drift from isometry and the empirical NTK grows anisotropic\[[35](https://arxiv.org/html/2607.09202#bib.bib35)\]\. The failures are complementary, and pairing igfa with a dynamical\-isometry regularizer—driving each weight matrix toward orthogonality,Riso​\(W\)=‖W⊤​W−I‖F2R\_\{\\mathrm\{iso\}\}\(W\)=\\\|W^\{\\top\}W\-I\\\|\_\{F\}^\{2\}\(closed\-form gradient4​W​\(W⊤​W−I\)4W\(W^\{\\top\}W\-I\), no SVD\), as a decoupled AdamO step\[[35](https://arxiv.org/html/2607.09202#bib.bib35)\]—is*super\-additive*\. On a2020\-task Split\-CIFAR\-100 stream with a deeper unnormalized ReLU network \(orthogonal init, task\-IL masking; three seeds;kaggle\_isometry\_igfa\_hybrid\.py\), isometry alone adds\+0\.033\+0\.033average accuracy over naive and the gate alone\+0\.071\+0\.071, but together they reach0\.555±0\.0340\.555\\pm 0\.034against a0\.3310\.331additive prediction—a synergy of\+0\.224\+0\.224, and2\.4×2\.4\\timesnaive\. The gate supplies retention \(old\-task accuracy∼0\.5\\sim\\\!0\.5versus naive’s∼0\.15\\sim\\\!0\.15\) at preserved learning while isometry supplies plasticity \(learning0\.780\.78, drift pinned, dead ReLUs revived\); the result replicates on a harder per\-task pixel\-permuted stream \(0\.425±0\.0110\.425\\pm 0\.011versus naive0\.2150\.215\), with the same ordering\. Two mechanisms account for it\. The gate must carry the correct sign—it acts only when a step would raise a past loss \(⟨g,gA⟩<0\\langle g,g\_\{A\}\\rangle<0\) and passes beneficial transfer \(⟨g,gA⟩\>0\\langle g,g\_\{A\}\\rangle\>0\), firing on the genuine\-conflict minority of steps rather than stripping the shared learning signal that dominates a shared\-feature network\. And isometry makes gating*cheap*: the per\-step cost‖g−g~‖/‖g‖\\\|g\-\\tilde\{g\}\\\|/\\\|g\\\|, the fraction of the gradient the projection removes, falls3939–43%43\\%once isometry is added—consistently across mixed\-aggregate \(A\-GEM\) and per\-task \(GEM\) gates and both streams—because in the high\-rank isometric representation the conflicting component is a smaller share of the update, so protecting the past costs less of the present\. A corollary: the sharper per\-task gate*requires*isometry, since alone it collapses the representation to rank∼1\\sim\\\!1and underperforms the aggregate gate, but with isometry holding effective rank high \(2525versus5\.65\.6\) it becomes the strongest protector\. The claim is comparative—absolute accuracies are low for a small unnormalized network on hard task\-IL—but the ordering and the cost mechanism are stable across three seeds and two streams\.

#### Multimodal interference is block\-structured\.

For a joint embeddingϕ=\[ϕv,ϕl\]\\phi=\[\\phi\_\{v\},\\phi\_\{l\}\], the second momentΣA\\Sigma\_\{A\}partitions into intra\-modal blocksΣv,Σl\\Sigma\_\{v\},\\Sigma\_\{l\}and a cross\-modal blockΣv​l\\Sigma\_\{vl\}, and the interference energy12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Deltadistributes linearly across them\. The cross\-modal \(alignment\) interference of a fixed update scales with the off\-diagonal‖Σv​l‖\\\|\\Sigma\_\{vl\}\\\|—negligible when the modalities are unaligned and dominant when they are \(energy ratio≫1\\gg 1across all seeds\) —so catastrophic decoupling is carried strictly byΣv​l\\Sigma\_\{vl\}\. A cross\-modal igfa wouldΣv​l\\Sigma\_\{vl\}\-orthogonalize the alignment directions while sharing the intra\-modal ones\. This is a synthetic decomposition; a real multimodal encoder is needed to make it conclusive\.

#### The functional governs physics\-informed constraints\.

A differential constraint is a further quadratic form in feature space\. For a physics\-informed network with frozen featuresϕ\\phi, the PDE residual acts onϕ′′\\phi^\{\\prime\\prime\}, so the physics second moment isΣPDE=𝔼​\[ϕ′′​ϕ′′⁣⊤\]\\Sigma\_\{\\mathrm\{PDE\}\}=\\mathbb\{E\}\[\\phi^\{\\prime\\prime\}\\phi^\{\\prime\\prime\\top\}\]and a data\-only updateΔ\\Deltaraises the physics loss by exactly∇LPDE⊤​Δ\+12​Δ⊤​ΣPDE​Δ\\nabla L\_\{\\mathrm\{PDE\}\}^\{\\top\}\\Delta\+\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{\\mathrm\{PDE\}\}\\Delta\. On a11D Poisson problem \(sine random\-feature basis\) the interference energy12​Δ⊤​ΣPDE​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{\\mathrm\{PDE\}\}\\Deltaalone predicts the measured spike in the PDE residual atr=0\.997r=0\.997, and the full identity to machine precision \(r=1\.00000r=1\.00000, relative error2×10−152\\times 10^\{\-15\}; Fig\.[S9](https://arxiv.org/html/2607.09202#A13.F9)a\): the functional quantifies how empirical data corrupt a physical prior, with the only change beingϕ↦ϕ′′\\phi\\mapsto\\phi^\{\\prime\\prime\}\. Projecting the data gradient off the leading physics directions reduces the induced violation \(3×3\\timeshere\) but cannot eliminate it whenΣPDE\\Sigma\_\{\\mathrm\{PDE\}\}is full rank—an instance of the distortion floor of Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)under a shared active subspace\.

#### Policy\-gradient interference is governed by the Fisher\.

For a linear\-Gaussian policyπ​\(a∣s\)=𝒩​\(θ⊤​s,σ2\)\\pi\(a\\mid s\)=\\mathcal\{N\}\(\\theta^\{\\top\}s,\\sigma^\{2\}\)under reward−\(a−w⊤​s\)2\-\(a\-w^\{\\top\}s\)^\{2\}the expected return isJ​\(θ\)=−\(θ−w\)⊤​F​\(θ−w\)−σ2J\(\\theta\)=\-\(\\theta\-w\)^\{\\top\}F\(\\theta\-w\)\-\\sigma^\{2\}withF=𝔼​\[s​s⊤\]F=\\mathbb\{E\}\[ss^\{\\top\}\]the empirical Fisher: theΣ\\Sigmaof supervised interference is replaced verbatim byFF\. Across two control environments sharing a three\-dimensional state subspace, routing the policy gradient orthogonal to the old\-environment Fisher subspace retains the first environment perfectly \(return0\.0000\.000vs naive−10\.9\-10\.9; a Fisher\-weighted anchor, the EWC analogue, reaches−1\.5\-1\.5; Fig\.[S9](https://arxiv.org/html/2607.09202#A13.F9)b\) while solving the second in its free directions\. The removability dichotomy of Corollary[1](https://arxiv.org/html/2607.09202#Thmcorollary1)holds unchanged in reinforcement learning; only the metric’s estimator changes\. This is the noise\-free expected gradient—a stochastic PPO/MuJoCo test is left to future work\.

#### The distortion floor can be trained away\.

Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5)’s floor vanishes when task second moments are mutually orthogonal, a property a feature map can be optimised to enforce\. Adding the penalty∑A<BTr⁡\(ΣA​ΣB\)\\sum\_\{A<B\}\\operatorname\{Tr\}\(\\Sigma\_\{A\}\\Sigma\_\{B\}\)to a shared encoder drives its per\-task subspaces apart: on overlapping synthetic tasks it cuts mean inter\-task overlap10\.6×10\.6\\times\(0\.27→0\.030\.27\\to 0\.03\) and the floor proxyTr⁡\(ΣA​ΣB\)\\operatorname\{Tr\}\(\\Sigma\_\{A\}\\Sigma\_\{B\}\)by over three orders of magnitude \(3\.7→10−33\.7\\to 10^\{\-3\}; Fig\.[S9](https://arxiv.org/html/2607.09202#A13.F9)c\), leaving igfa to operate in the disjoint, lossless regime of Corollary[1](https://arxiv.org/html/2607.09202#Thmcorollary1)\. Structural allocation and representation learning are thus complementary: a backbone meta\-trained for feature orthogonality removes the need for gating, so representation learning and structural allocation address the same distortion floor\.

#### Certified unlearning is the dichotomy inverted\.

Removability read backwards is a geometric feasibility criterion\. Projecting a trained model’s update offrange⁡ΣF\\operatorname\{range\}\\Sigma\_\{F\}of a forget set removes that set’s influence exactly when forget and retain supports are disjoint: held\-out forget\-set loss rises to the untrained level while retain\-set performance is untouched\. When the supports sharekkdirections the removal is necessarily lossy, and the collateral damage to the retain set is not merely bounded but*predicted exactly*by the retain energy on the shared support \(correlation1\.001\.00, zero mean error,4444seeds,k=0k=0–33\): the same floor that lower\-bounds forgetting lower\-bounds the retain cost of exact unlearning\. Support overlap is thus a principled impossibility boundary for machine unlearning under shared representations\. The two regimes separate identically on the paper’s own real benchmarks \(evaluate\_unlearning\_real\.py\): on near\-disjoint Split\-Digits, one projection matches retraining without the forgotten task on*both*sets \(retained accuracy0\.9910\.991versus retrain0\.9930\.993; the forgotten task falls to the retrain level\), while on Rotated\-Digits \(overlap≈0\.68\\approx 0\.68\) the same projection necessarily damages retained rotations \(\+0\.37\+0\.37mean accuracy loss versus retrain\)—certified removal where supports are disjoint, the floor where they are not\.

![Refer to caption](https://arxiv.org/html/2607.09202v1/x22.png)Figure S9:Three application domains for the interference functional \(evaluate\_applications\.py\)\.*\(a\)*Physics\-informed regression: the interference identity withΣPDE\\Sigma\_\{\\rm PDE\}predicts the measured PDE\-loss change of data\-only updates to machine precision \(dashed:y=xy=x\)\.*\(b\)*Policy gradients: routing the expected policy gradient orthogonal to the old environment’s Fisher subspace retains the old policy at zero regret while the new environment is learned in its free directions; naive descent forgets, and the Fisher\-weighted anchor \(the EWC analogue\) is intermediate\.*\(c\)*A feature map trained with theTr⁡\(ΣA​ΣB\)\\operatorname\{Tr\}\(\\Sigma\_\{A\}\\Sigma\_\{B\}\)penalty reduces inter\-task overlap and the distortion\-floor proxy by orders of magnitude \(log scale\), moving igfa into the lossless regime\.

## Appendix S14Memory systems and control from the functional

Beyond deciding*where*updates may act, the functional prescribes*how much*to move and*when memory is worth spending*\. Four exact\-regime demonstrations \(multi\-seed means;evaluate\_next\_steps\.py\) make this operational\. The framing follows complementary\-learning\-systems theory: a fast memory system supplies rehearsal, and the geometry decides when to invoke it\.

#### The functional is a trust region\.

Constraining each update by its predicted interference,12​\(w−wA\)⊤​ΣA​\(w−wA\)≤B\\tfrac\{1\}\{2\}\(w\-w\_\{A\}\)^\{\\top\}\\Sigma\_\{A\}\(w\-w\_\{A\}\)\\leq B—equivalently, descendingLB\+μ​12​\(w−wA\)⊤​ΣA​\(w−wA\)L\_\{B\}\+\\mu\\,\\tfrac\{1\}\{2\}\(w\-w\_\{A\}\)^\{\\top\}\\Sigma\_\{A\}\(w\-w\_\{A\}\)—traces a strictly better retention–plasticity frontier than generic step\-size or early\-stopping control:3131–59%59\\%lower deferred new\-task loss at matched forgetting across budget levels, with plain online gradient descent on the penalty matching the closed\-form optimum\. The gate of Sec\.[4](https://arxiv.org/html/2607.09202#S4)is theμ→∞\\mu\\to\\inftylimit of this controller on the conflicting subspace; intermediate budgets interpolate continuously between naive descent and full protection\. The controller’s metric can also be sketched: at sketch sizeℓ≥2​r\\ell\\geq 2rthe Frequent\-Directions metric reproduces the full\-covariance frontier to1\.3±0\.5%1\.3\\pm 0\.5\\%at3\.6×3\.6\\timesless memory, whileℓ=r\\ell=rfails outright \(43%43\\%gap\)—the sameℓ=2​r\\ell=2rsizing rule as Sec\.[S9](https://arxiv.org/html/2607.09202#A9)\. At language\-model scale, however, the advantage does not persist in this form: on a four\-domain sequential LoRA stream with real measured forgetting, a*frozen*\-sketch penalty and tuned Adam schedules trace statistically indistinguishable frontiers \(±0\.2%\\pm 0\.2\\%at matched forgetting, two seeds\), with the schedule Pareto\-dominant at the low\-forgetting end \(kaggle\_interference\_optimizer\.py\)\. This is the stale\-metric failure mode of Sec\.[5\.7](https://arxiv.org/html/2607.09202#S5.SS7)in optimizer form—frozen geometry barely helps under joint training—and it points to the current\-signal controller \(the functional gate below and in Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\) as the version predicted to survive scale, a prediction the functional gate then confirmed on the same stream\. Under gradient noise, the residual risk of protection itself obeys a verified diffusion law: with basis error at principal angleϵ\\epsilon, the noise\-driven forgetting is12​η2​σ2​t​r​sin2⁡ϵ\\tfrac\{1\}\{2\}\\eta^\{2\}\\sigma^\{2\}t\\,r\\sin^\{2\}\\epsilon\(measured\-to\-law ratio1\.051\.05–1\.071\.07acrossη\\etaandϵ\\epsilon; exactly zero atϵ=0\\epsilon=0\), which inverts to a retention\-aware step\-size boundη∗=2​τ/\(σ2​T​r​sin2⁡ϵ\)\\eta^\{\\ast\}=\\sqrt\{2\\tau/\(\\sigma^\{2\}Tr\\sin^\{2\}\\epsilon\)\}for toleranceτ\\tauover horizonTT\(evaluate\_upgrades\.py\)\.

#### Consolidation is spectral compression with a proven price\.

When tasks share structure, the per\-task protect bases are redundant\. Consolidating them onto the joint spectrum—eigenvectors of the accumulated second moment at fixed captured energy— compressed the protect set by55%55\\%\(24→1124\\to 11stored directions\) while an adversarially conflicting new task induced at most0\.0040\.004forgetting on the old family, within the dropped\-eigenvalue bound of0\.040\.04that Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)supplies in advance\. Offline consolidation \(“sleep”\) is spectral compression whose cost the theory prices before it is paid\.

#### Memory is worth spending where the floor is positive\.

With a fixed total replay budget over a stream of mixed tasks, allocating exemplars in proportion to each task’s*predicted*distortion floor \(fromΣ\\Sigma\-overlap and target disagreement\) retained the anchor task28±31%28\\pm 31\\%better than a uniform budget, improving in1818of2020seeds\. Structural allocation handles the disjoint tasks for free, and memory concentrates where the geometry says forgetting is otherwise unavoidable: the fast system supplies rehearsal, the functional decides when it is needed\. The floor sharpens this principle into an exact policy\. Full rehearsal converges to the joint optimum, which lands every task precisely*at*its floor share and never below it \(verified to3×10−173\\times 10^\{\-17\};evaluate\_replay\_policy\.py\): rehearsal cannot buy back the floor, so the*value*of replaying taskttis its recoverable excessLt−𝒟t⋆L\_\{t\}\-\\mathcal\{D\}^\{\\star\}\_\{t\}, and the optimal fixed\-budget allocation is the fractional knapsack—greedy in recoverable excess per exemplar \(matches the enumerated optimum in2020of2020random instances\)\. Floor\-*proportional*allocation is the valid proxy when damage scales with conflict, the regime of the experiment above; it misallocates in two identifiable regimes—a task already at its floor has zero replay value however large that floor, and a zero\-floor task damaged by capacity coupling is fully recoverable despite its floor—where it drops below even a uniform budget \(3\.563\.56versus2\.712\.71; optimum2\.172\.17\)\. All three quantities in the policy—damage, floor, cost—are pre\-flight measurables of the ledger\.

#### The subspace state is a replay generator\.

In the exact regime, the state igfa already carries—the basisUU, its eigenvalues, and a weight snapshot—is a sufficient statistic for replay: samplingx^=U​Λ1/2​z\\hat\{x\}=U\\Lambda^\{1/2\}zwith labels from the snapshot and rehearsing on these “dreamed” pairs reduces the floor exactly as real replay does \(mean forgetting1\.711\.71versus1\.721\.72; seedwise agreementr=0\.97r=0\.97\) with zero stored examples\. The construction extends to*rollouts*: dreamed trajectories from a learned world model rehearse the old task nearly as well as real experience \(2\.022\.02versus1\.901\.90forgetting, against4\.234\.23without replay\) and degrade gracefully with model error \(2\.482\.48at20%20\\%dynamics noise\), so counterfactual replay inherits the floor reduction up to the world model’s own fidelity\. Replay and structural allocation are therefore not opposing families: the same low\-rank state implements both, and buffer\-free generative rehearsal inherits replay’s floor reduction at sharply reduced exposure\. The privacy claim can be made formal: the state is a covariance release, so the analytic Gaussian mechanism applies directly; withn=600n=600samples behind the release, calibrated noise at\(ϵ=8,δ=10−5\)\(\\epsilon\{=\}8,\\delta\{=\}10^\{\-5\}\)leaves the released subspace at fidelity0\.990\.99with a likelihood\-ratio membership attack at chance, while at smallnnthe sensitivity2/n2/nmakes certified privacy and utility incompatible \(evaluate\_upgrades\.py\)—differentially private dream replay is viable exactly for data\-rich tasks\.

## Appendix S15Beyond frozen features: the function\-space formalism

Assumption A1 confines the exact*parameter\-space*results to frozen features, PEFT, and first\-order NTK settings\. The formalism itself does not end there: recast in function space, the identity, the dichotomy, the floor, and the retention guarantee all survive with A1 removed\. Three results follow, each exact for arbitrary architectures, with validations on jointly trained deep networks \(evaluate\_beyond\_a1\.py\)\.

#### The interference identity is exact in function space\.

For any predictorffletLA​\(f\)=12​𝔼DA​\(f​\(x\)−y\)2L\_\{A\}\(f\)=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{D\_\{A\}\}\(f\(x\)\-y\)^\{2\}, and letfA,fBf\_\{A\},f\_\{B\}be the functions realized before and after learningBB, by any architecture and any training procedure\. WritingΔ​f=fB−fA\\Delta\\\!f=f\_\{B\}\-f\_\{A\}andrA=fA−yr\_\{A\}=f\_\{A\}\-y, expanding the square gives the identity

Δ​LA=𝔼DA​\[Δ​f​rA\]\+12​𝔼DA​\[Δ​f2\]\.\\Delta L\_\{A\}\\;=\\;\\mathbb\{E\}\_\{D\_\{A\}\}\\\!\\big\[\\Delta\\\!f\\;r\_\{A\}\\big\]\\;\+\\;\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{D\_\{A\}\}\\\!\\big\[\\Delta\\\!f^\{\\,2\}\\big\]\.\(14\)IfAAis realizable and trained to convergence \(rA=0r\_\{A\}=0DAD\_\{A\}\-a\.s\.\), forgetting*is*the function\-space interference energy12​‖Δ​f‖L2​\(DA\)2\\tfrac\{1\}\{2\}\\\|\\Delta\\\!f\\\|^\{2\}\_\{L^\{2\}\(D\_\{A\}\)\}: the metric is the data distribution itself, and Theorem[1](https://arxiv.org/html/2607.09202#Thmtheorem1)is the linear shadow of Eq\. \([14](https://arxiv.org/html/2607.09202#A15.E14)\) \(under A1,Δ​f=Δ⊤​ϕ\\Delta\\\!f=\\Delta^\{\\top\}\\phirecovers12​Δ⊤​ΣA​Δ\\tfrac\{1\}\{2\}\\Delta^\{\\top\}\\Sigma\_\{A\}\\Delta\)\. Numerically, Eq\. \([14](https://arxiv.org/html/2607.09202#A15.E14)\) holds to machine precision on jointly trained depth\-11–33networks \(maximum relative error2×10−72\\times 10^\{\-7\},r=1\.000000r=1\.000000\), including the depth\-33regime where the frozen\-curvature predictor of Sec\.[5\.4](https://arxiv.org/html/2607.09202#S5.SS4)degrades tor≈0\.4r\\approx 0\.4\. Exact prediction beyond A1 costs two forward passes onAA\-data—no Hessian and no path integral; the parameter\-space results contribute the structure, and A1 is the regime where the two pictures coincide\.

#### The dichotomy and the floor survive, with input support in place of feature support\.

LearningBBis lossless forAAiffΔ​f=0\\Delta\\\!f=0onsupp​DA\\mathrm\{supp\}\\,D\_\{A\}\. For a universal model class this is achievable whenever the input supports are disjoint, at any depth; and when they overlap with disagreeing targets, the floor is a property of the data alone: minimizing pointwise,

minf⁡LA​\(f\)\+LB​\(f\)=12​∫pA​\(x\)​pB​\(x\)pA​\(x\)\+pB​\(x\)​\(yA​\(x\)−yB​\(x\)\)2​𝑑x,\\min\_\{f\}\\;L\_\{A\}\(f\)\+L\_\{B\}\(f\)\\;=\\;\\tfrac\{1\}\{2\}\\\!\\int\\\!\\frac\{p\_\{A\}\(x\)\\,p\_\{B\}\(x\)\}\{p\_\{A\}\(x\)\+p\_\{B\}\(x\)\}\\,\\big\(y\_\{A\}\(x\)\-y\_\{B\}\(x\)\\big\)^\{2\}\\,dx,\(15\)the density\-overlap harmonic mean weighted by the squared target disagreement—the input\-space form of Theorem[5](https://arxiv.org/html/2607.09202#Thmtheorem5), and a lower bound for every concrete architecture, whose own floor equals Eq\. \([15](https://arxiv.org/html/2607.09202#A15.E15)\) plus an approximation\-class excess\. Measured with held\-out \(population\) losses, a depth\-33network trained jointly on conflicting sine tasks respects the bound with a≈20%\\approx 20\\%class excess \(predicted0\.500\.50, measured0\.600\.60\), and near\-disjoint input supports give the lossless regime at any depth \(predicted0\.0010\.001, measured0\.0090\.009\)\.

#### Exact retention without A1: instantaneous null\-space control\.

Along any absolutely continuous training pathθ​\(t\)\\theta\(t\),Δ​LA=∫⟨∇LA​\(θt\),θ˙t⟩​𝑑t\\Delta L\_\{A\}=\\int\\langle\\nabla L\_\{A\}\(\\theta\_\{t\}\),\\dot\{\\theta\}\_\{t\}\\rangle\\,dtexactly, with∇LA​\(θ\)=𝔼DA​\[rA​\(x\)​∇θf​\(x;θ\)\]\\nabla L\_\{A\}\(\\theta\)=\\mathbb\{E\}\_\{D\_\{A\}\}\[\\,r\_\{A\}\(x\)\\,\\nabla\_\{\\theta\}f\(x;\\theta\)\\,\]\. If the velocity stays in the kernel of the*current*Gauss–Newton metric,θ˙t∈ker⁡GA​\(θt\)\\dot\{\\theta\}\_\{t\}\\in\\ker G\_\{A\}\(\\theta\_\{t\}\)withGA​\(θ\)=𝔼DA​\[∇θf​∇θf⊤\]G\_\{A\}\(\\theta\)=\\mathbb\{E\}\_\{D\_\{A\}\}\[\\nabla\_\{\\theta\}f\\,\\nabla\_\{\\theta\}f^\{\\top\}\], thenθ˙⊤​GA​θ˙=0\\dot\{\\theta\}^\{\\top\}G\_\{A\}\\dot\{\\theta\}=0forces⟨∇θf​\(x\),θ˙⟩=0\\langle\\nabla\_\{\\theta\}f\(x\),\\dot\{\\theta\}\\rangle=0forDAD\_\{A\}\-almost everyxx, hence⟨∇LA,θ˙⟩≡0\\langle\\nabla L\_\{A\},\\dot\{\\theta\}\\rangle\\equiv 0and*LAL\_\{A\}is conserved exactly*—any depth, any feature drift, no A1 and no realizability\. Discrete steps of sizeη\\etaleak only the Taylor remainder12​η2​v⊤​H¯​v\\tfrac\{1\}\{2\}\\eta^\{2\}v^\{\\top\}\\bar\{H\}\\,vper step \(Theorem[2](https://arxiv.org/html/2607.09202#Thmtheorem2)\), i\.e\.O​\(η\)O\(\\eta\)at fixed path length\. Measured at fixed path budget on jointly trained networks, forgetting under per\-step re\-estimated null\-space control falls0\.134→0\.0037→0\.0004→0\.000030\.134\\to 0\.0037\\to 0\.0004\\to 0\.00003asη\\etadrops0\.1→0\.0030\.1\\to 0\.003, while a frozen protected basis plateaus near0\.50\.5\(staleness bias does not vanish withη\\eta\) and naive training forgets atO​\(1\)O\(1\)\. Assumption A1 is thereby relocated, not removed: a constant metric makes the kernel constant, so finite steps stay in it exactly—A1 buys*finite\-step*exactness, while the geometry, the dichotomy, the floor, and the continuous\-time retention guarantee never needed it\. The recursive Gauss–Newton tracker of Sec\.[4](https://arxiv.org/html/2607.09202#S4)is the discretization of this exact law, with the finite\-step, estimation, and capacity costs quantified in Secs\.[5\.7](https://arxiv.org/html/2607.09202#S5.SS7),[S9](https://arxiv.org/html/2607.09202#A9), and[S14](https://arxiv.org/html/2607.09202#A14)\.

#### Finite\-step exactness recovered: two parameter\-space routes\.

Even the finite\-step gap closes, by two complementary mechanisms \(evaluate\_beyond\_a1\.py\)\.*\(i\) Region\-exactness for piecewise\-linear networks\.*For ReLU\-family models the loss is*exactly*quadratic in each layer’s weights while the activation pattern on theAA\-support is fixed: per\-layer, Theorem 1 holds at finite step size inside the activation region, and the region boundary has a closed form—for a steps​VsVin layer weights, no activation flips whiles<mini,j⁡\|prei​j\|/\|\(V​xi\)j\|s<\\min\_\{i,j\}\|\{\\rm pre\}\_\{ij\}\|/\|\(Vx\_\{i\}\)\_\{j\}\|, computable from the stored preactivation margins\. Measured: the quadratic is exact to relative error10−1110^\{\-11\}inside the region, and the certificate is conservative \(breakdown observed at1\.4±0\.4×1\.4\\pm 0\.4\\timesthe threshold, never meaningfully before it\)\. Assumption A1 thus relaxes, for ReLU networks, to a checkable per\-step margin condition, and the margin joins the pre\-flight/ledger family as a monitorable quantity\.*\(ii\) Retraction for any smooth architecture\.*Exact retention means staying on the anchor level set\{θ:f​\(xi;θ\)=ci\}\\\{\\theta:f\(x\_\{i\};\\theta\)=c\_\{i\}\\\}, and that manifold is returnable: after each ordinary update, two Gauss–Newton corrector steps on the anchor residuals restore it\. Measured on jointly trained networks, anchor\-set forgetting sits at numerical zero \(∼10−14\\sim 10^\{\-14\}\)*independent of step size*, where projection alone leaksO​\(η\)O\(\\eta\)\(0\.170\.17,0\.0430\.043,0\.00150\.0015atη=0\.1,0\.05,0\.02\\eta=0\.1,0\.05,0\.02\), while the new task still trains\. The remaining limitation is one of coverage: exactness holds on the anchors, and between them the held\-outAA\-loss depends on how well the anchor set pins the function class \(0\.150\.15–0\.260\.26here at2424anchors\)\. Beyond A1, retention is governed by how densely the anchors sample the function class\.

#### Monotone\-retention gating: the threshold\-free gate\.

The path integral also yields the gate itself, with the similarity threshold removed\. Sinced​LA/d​t=⟨∇LA,θ˙⟩dL\_\{A\}/dt=\\langle\\nabla L\_\{A\},\\dot\{\\theta\}\\rangle, the exact, modality\-free rule is a*sign test*: given the new task’s update directiongg, if⟨g,∇LA⟩≥0\\langle g,\\nabla L\_\{A\}\\rangle\\geq 0the step also descends \(or preserves\)LAL\_\{A\}and is kept whole; otherwise only its component along∇LA\\nabla L\_\{A\}is removed,g↦g−⟨g,∇LA⟩‖∇LA‖2​∇LAg\\mapsto g\-\\frac\{\\langle g,\\nabla L\_\{A\}\\rangle\}\{\\\|\\nabla L\_\{A\}\\\|^\{2\}\}\\nabla L\_\{A\}\. In continuous time this guaranteesd​LA/d​t≤0dL\_\{A\}/dt\\leq 0—retention is monotone—while every non\-harmful component of the update, and therefore all transfer, is retained; no threshold, no input geometry, and no similarity estimate appears\. The single\-constraint form recovers the projections of A\-GEM\[[4](https://arxiv.org/html/2607.09202#bib.bib4)\]and the surgery of PCGrad\[[10](https://arxiv.org/html/2607.09202#bib.bib10)\]as special cases, here derived from the exact identity rather than posited; per\-direction refinements in the currentGAG\_\{A\}\-eigenbasis coincide with the single constraint atAA’s optimum \(every range\-component of motion is harmful there\) and differ off\-optimum\. In the exact regime the sign gate holds forgetting at0\.0250\.025versus naive10\.0610\.06under pure conflict while reaching strictly lower deferred loss than unconditional projection \(9\.089\.08versus10\.0610\.06\), and is harmless under alignment; on jointly trained networks it attains near\-current\-GN retention under conflict \(\+0\.14\+0\.14versus naive\+5\.77\+5\.77and*frozen*projection\+5\.38\+5\.38, which barely helps under drift\) with better adaptation than current\-GN \(0\.110\.11versus0\.150\.15\) at a fraction of its cost—one gradient ofLAL\_\{A\}per step, from a micro\-cache, the anchor construction of Sec\.[S9](https://arxiv.org/html/2607.09202#A9), or the dream state of Sec\.[S14](https://arxiv.org/html/2607.09202#A14)\(evaluate\_functional\_gate\.py\)\. On real text it significantly beats naive continual fine\-tuning on both axes, threshold\-free and without input geometry, and edges unconditional protection \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\)\.

The gate is, moreover, not merely a safe modification of the update but the*optimal*one\.

###### Corollary 2\(The sign gate is optimal monotone control\)\.

Letuube the desired update and let tasks1,…,m1,\\dots,mbe protected\. The problem

v∗=argminv⁡12​‖v−u‖2s\.t\.⟨∇LAi,v⟩≤0,i=1,…,m,v^\{\\ast\}=\\operatorname\{argmin\}\_\{v\}\\tfrac\{1\}\{2\}\\\|v\-u\\\|^\{2\}\\quad\\text\{s\.t\.\}\\quad\\langle\\nabla L\_\{A\_\{i\}\},v\\rangle\\leq 0,\\;\\;i=1,\\dots,m,\(16\)has the solutionv∗=u−G​λ∗v^\{\\ast\}=u\-G\\lambda^\{\\ast\}withG=\[∇LA1​⋯​∇LAm\]G=\[\\nabla L\_\{A\_\{1\}\}\\cdots\\nabla L\_\{A\_\{m\}\}\]andλ∗=argminλ≥0⁡‖G​λ−u‖\\lambda^\{\\ast\}=\\operatorname\{argmin\}\_\{\\lambda\\geq 0\}\\\|G\\lambda\-u\\\|\(a nonnegative least\-squares problem\)\. Form=1m=1,v∗v^\{\\ast\}is exactly the sign gate; when no constraint is violated,v∗=uv^\{\\ast\}=u\(sharing\); when all are violated,v∗v^\{\\ast\}remains feasible yet strictly less restrictive than equality projection: protected rates may be negative—the update is allowed to*improve*protected tasks, which projection forbids—and‖v∗−u‖≤‖vproj−u‖\\\|v^\{\\ast\}\-u\\\|\\leq\\\|v\_\{\\mathrm\{proj\}\}\-u\\\|always\.

*Intuition:*stationarity of the Lagrangian givesv=u−G​λv=u\-G\\lambda; substituting back turns the dual into the stated nonnegative least squares, whose Karush–Kuhn–Tucker conditions are precisely primal feasibility and complementary slackness\. The gate therefore interpolates between naive sharing and subspace projection through the active set\. Unconditional projection hardens those inequality constraints to equalities, which in the exact regime never improves on the gate\. All statements are machine\-checked \(‖v∗−vgate‖=0\\\|v^\{\\ast\}\-v\_\{\\mathrm\{gate\}\}\\\|=0atm=1m=1; feasibility to10−1510^\{\-15\}; measured protected\-rate decrease under full activation;evaluate\_functional\_control\.py\)\.

#### Beyond squared loss: the Bregman family\.

The function\-space results are not quadratic\-loss artifacts\. For any canonical lossℓ​\(y,f\)=φ​\(f\)−y⊤​f\\ell\(y,f\)=\\varphi\(f\)\-y^\{\\top\}fwithφ\\varphiconvex and mean mapμ=∇φ\\mu=\\nabla\\varphi\(squared loss:φ=12​‖f‖2\\varphi=\\tfrac\{1\}\{2\}\\\|f\\\|^\{2\}; softmax cross\-entropy:φ=logsumexp\\varphi=\\mathrm\{logsumexp\},μ=softmax\\mu=\\mathrm\{softmax\}; logistic and Poisson likewise\), the definition of the Bregman divergenceDφ​\(b,a\)=φ​\(b\)−φ​\(a\)−μ​\(a\)⊤​\(b−a\)D\_\{\\varphi\}\(b,a\)=\\varphi\(b\)\-\\varphi\(a\)\-\\mu\(a\)^\{\\top\}\(b\-a\)gives, exactly and for any finite functional change,

Δ​LA=𝔼DA​\[\(μA−y\)⊤​Δ​f\]\+𝔼DA​\[Dφ​\(fB,fA\)\],\\Delta L\_\{A\}=\\mathbb\{E\}\_\{D\_\{A\}\}\\\!\\big\[\(\\mu\_\{A\}\-y\)^\{\\top\}\\Delta\\\!f\\big\]\+\\mathbb\{E\}\_\{D\_\{A\}\}\\\!\\big\[D\_\{\\varphi\}\(f\_\{B\},f\_\{A\}\)\\big\],\(17\)of which Eq\. \([14](https://arxiv.org/html/2607.09202#A15.E14)\) is the quadratic case\. For cross\-entropy,Dφ​\(fB,fA\)=KL​\(pA∥pB\)D\_\{\\varphi\}\(f\_\{B\},f\_\{A\}\)=\\mathrm\{KL\}\(p\_\{A\}\\,\\\|\\,p\_\{B\}\)exactly, so on a converged task \(μA=y\\mu\_\{A\}=yonsupp​DA\\mathrm\{supp\}\\,D\_\{A\}\) forgetting*is*the expected Kullback–Leibler divergence between the old and new predictive distributions\. Three consequences follow\.*\(i\)*The removability dichotomy is loss\-free: under A1 a kernel step changes no function value on the support, henceΔ​LA=0\\Delta L\_\{A\}=0exactly at any finite step size for*any*loss \(validated at‖Δ​W‖=10\\\|\\Delta W\\\|=10\); for softmax the lossless set is strictly larger, since per\-sample constant logit shifts are invisible\. Only the closed quadratic*energy formula*is squared\-loss\-specific\.*\(ii\)*The distortion floor has a closed information\-geometric form: pointwise,minq⁡\[pA​KL​\(πA∥q\)\+pB​KL​\(πB∥q\)\]\\min\_\{q\}\\,\[\\,p\_\{A\}\\mathrm\{KL\}\(\\pi\_\{A\}\\\|q\)\+p\_\{B\}\\mathrm\{KL\}\(\\pi\_\{B\}\\\|q\)\\,\]is attained at the density\-weighted*mixture*q∗=\(pA​πA\+pB​πB\)/\(pA\+pB\)q^\{\\ast\}=\(p\_\{A\}\\pi\_\{A\}\+p\_\{B\}\\pi\_\{B\}\)/\(p\_\{A\}\+p\_\{B\}\), giving

floor=∫\(pA\(x\)\+pB\(x\)\)JSDw​\(x\)\(πA\(⋅\|x\),πB\(⋅\|x\)\)dx,w=pApA\+pB,\\text\{floor\}=\\\!\\int\\\!\\big\(p\_\{A\}\(x\)\+p\_\{B\}\(x\)\\big\)\\,\\mathrm\{JSD\}\_\{w\(x\)\}\\\!\\big\(\\pi\_\{A\}\(\\cdot\|x\),\\pi\_\{B\}\(\\cdot\|x\)\\big\)\\,dx,\\qquad w=\\tfrac\{p\_\{A\}\}\{p\_\{A\}\+p\_\{B\}\},\(18\)a weighted Jensen–Shannon divergence between the tasks’ conditionals; the quadratic floor of Theorem[4](https://arxiv.org/html/2607.09202#Thmtheorem4)’s function\-space form is its small\-divergence \(Gaussian\) limit, and the mixture replaces theΣ\\Sigma\-weighted mean as the optimal joint predictor\. ForTTtasks with uniform prior the floor becomes an*information identity*: writingTTfor the task variable,XXfor the input, andYYfor the label,

𝒟T⋆=∑x∑tpt​\(x\)​KL​\(πt∥π¯x\)=T⋅I​\(T;Y∣X\)=T​\[H​\(Y∣X\)−H​\(Y∣X,T\)\],\\mathcal\{D\}^\{\\star\}\_\{T\}=\\sum\_\{x\}\\sum\_\{t\}p\_\{t\}\(x\)\\,\\mathrm\{KL\}\\\!\\big\(\\pi\_\{t\}\\,\\\|\\,\\bar\{\\pi\}\_\{x\}\\big\)=T\\cdot I\(T;Y\\mid X\)=T\\,\\big\[H\(Y\\mid X\)\-H\(Y\\mid X,T\)\\big\],\(19\)the conditional mutual information between task identity and label—the exact label\-entropy cost of hiding which task a sample came from\. The entropy*sandwich*of Theorem[6](https://arxiv.org/html/2607.09202#Thmtheorem6)\(quadratic loss\) thus closes to an equality under the loss language models train, removability becomesY⟂T∣XY\\perp T\\mid X, and the floor is achievable, and a trained shared softmax head converges to Eq\. \([19](https://arxiv.org/html/2607.09202#A15.E19)\) from above with terminal gap6×10−166\\times 10^\{\-16\}\(evaluate\_info\_floor\.py; identity and mixture\-optimality checks at10−1610^\{\-16\}\)\. Operationally, the irreducible single\-model floor of an LLM domain stream isI​\(domain;next token∣context\)I\(\\text\{domain\};\\text\{next token\}\\mid\\text\{context\}\), estimable from micro\-caches as the mean per\-domain versus mixture log\-likelihood ratio\.*\(iii\)*The sign gate and Corollary[2](https://arxiv.org/html/2607.09202#Thmcorollary2)act on⟨g,∇LA⟩=d​LA/d​t\\langle g,\\nabla L\_\{A\}\\rangle=dL\_\{A\}/dtand never used the quadratic form, so they are the exact monotone controller for cross\-entropy as stated—the language\-model gate experiments of Sec\.[S5](https://arxiv.org/html/2607.09202#A5)therefore run the exact controller for the loss they train, not a transferred heuristic\. All four statements are validated to machine precision \(10−1610^\{\-16\}; kernel step exactly0\), with the closed\-form floor checked against direct numerical minimization \(evaluate\_bregman\_identity\.py\)\.

#### The forgetting decomposition, validated\.

With𝒟∞⋆≤𝒟class⋆≤L\\mathcal\{D\}^\{\\star\}\_\{\\infty\}\\leq\\mathcal\{D\}^\{\\star\}\_\{\\mathrm\{class\}\}\\leq Las defined in the Discussion,

L=𝒟∞⋆⏟incompatibility\+\[𝒟class⋆−𝒟∞⋆\]⏟capacity\+\[L−𝒟class⋆\]⏟control,L=\\underbrace\{\\mathcal\{D\}^\{\\star\}\_\{\\infty\}\}\_\{\\text\{incompatibility\}\}\+\\underbrace\{\\big\[\\mathcal\{D\}^\{\\star\}\_\{\\mathrm\{class\}\}\-\\mathcal\{D\}^\{\\star\}\_\{\\infty\}\\big\]\}\_\{\\text\{capacity\}\}\+\\underbrace\{\\big\[L\-\\mathcal\{D\}^\{\\star\}\_\{\\mathrm\{class\}\}\\big\]\}\_\{\\text\{control\}\},\(20\)an identity whose content is that the three terms are separately measurable and separately treatable\. Three single\-regime streams land their intended term \(ten seeds;evaluate\_forgetting\_decomposition\.py\): pure conflict under full replay is all floor \(12\.39/0/012\.39/0/0\); input\-disjoint tasks squeezed through width\-1212features are all capacity \(0/5\.35/00/5\.35/0\); mixed tasks under naive training carry floor plus control \(5\.42/0/4\.045\.42/0/4\.04\)\. The intervention matrix attributes cleanly: widening the features12→2412\\to 24removes100%100\\%of the capacity term while no policy—naive, active\-set, or full replay—moves it at all \(6\.026\.02untouched by all three\); upgrading the policy cuts only the control term \(4\.65→2\.154\.65\\to 2\.15, floor invariant throughout\)\. Two definitional points the construction forces into the open:*capacity*means representation width—the “best rank\-rrsubspace containing the solution” notion collapses, since any single optimum spans one dimension—and, in the quadratic regime,*perturbed*copies of a subspace are not conflict \(their soft constraints are jointly satisfiable and the floor is exactly zero; genuine conflict requires exactly shared support, a knife\-edge the cross\-entropy floor does not have, being nonzero whenever conditionals disagree on overlapping support\)\.

#### Benign forgetting: the anchor splits knowledge from memorization\.

The interference identity is distribution\-anchored, and nothing requires the anchor to be the training set\. Evaluated on taskAA’s*deployment*distribution the same identity splits observed forgetting into knowledge loss,Δ​LAtest\\Delta L\_\{A\}^\{\\mathrm\{test\}\}, and memorization loss,Δ​LAtrain−Δ​LAtest\\Delta L\_\{A\}^\{\\mathrm\{train\}\}\-\\Delta L\_\{A\}^\{\\mathrm\{test\}\}—and the two can carry opposite signs\. In an overparameterized interpolation setting with label noise \(ten seeds;evaluate\_benign\_forgetting\.py\), an interfering update raises train loss by\+1\.99\+1\.99—catastrophic forgetting under any train\-anchored metric—while*lowering*deployment loss by−3\.37\-3\.37: backward transfer through noise removal, with the sign predicted exactly by the test\-anchored identity \(8×10−168\\times 10^\{\-16\}\)\. Part of what benchmarks report as catastrophic forgetting is implicit unlearning of noise\. The consequence for protection is a design law:*protect the function you validated, not the function you fit*\. A gate whose constraint gradient is anchored on training data defends the memorized noise together with the knowledge—it blocks benign repair and pays adaptation for the privilege— whereas the held\-out\-anchored gate dominates it on*both*axes \(final deployment loss of the protected task8\.38\.3versus22\.022\.0; adaptation5\.45\.4versus7\.17\.1\)\. The anchor’s own quality gates the gate: with a held\-out set too small to be validated itself \(M=30<dM=30<d, the anchor interpolates its own noise\) the advantage collapses, which is the estimation–abstention rule’s other face\. The held\-out micro\-caches used throughout the language experiments \(Sec\.[S5](https://arxiv.org/html/2607.09202#A5)\) are therefore mandated by the theory, not merely hygienic\.

#### Certified unlearning is one reversed row of the gate\.

Engineered benign forgetting gives an unlearning method for free:*ascend*the forget set’s*validated*loss toward the never\-trained reference while the monotone\-retention QP of Corollary[2](https://arxiv.org/html/2607.09202#Thmcorollary2)holds every retained held\-out anchor in place—the unified gate with a single reversed constraint\. The certificate is then a*measurement*,𝔼retain​\[KL​\(pbefore∥pafter\)\]≤ϵ\\mathbb\{E\}\_\{\\mathrm\{retain\}\}\[\\mathrm\{KL\}\(p\_\{\\mathrm\{before\}\}\\\|p\_\{\\mathrm\{after\}\}\)\]\\leq\\epsilon, rather than a retraining\-equivalence definition\. The anchor split predicts a dichotomy, validated to machine precision \(evaluate\_unlearning\.py\):*memorized*content \(own support\) is removable at a zero floor, whereas*entangled*content shared with a retained task carries a closed\-form*unlearning floor*∑sharedpR​KL​\(πR∥pref\)\\sum\_\{\\mathrm\{shared\}\}p\_\{R\}\\,\\mathrm\{KL\}\(\\pi\_\{R\}\\\|p\_\{\\mathrm\{ref\}\}\)—the surgical optimum pays exactly this \(agreement3×10−73\\times 10^\{\-7\}\), and the gate stalls at it rather than silently damaging retained knowledge\. The memorized branch holds at language\-model scale\. Injecting a memorized canary into joint LoRA training onpythia\-410mandpythia\-1b\(the earlier attempt to unlearn an entire natural domain was vacuous—the adapter had added only∼0\.03\\sim 0\.03nats of removable domain\-specific knowledge, the pre\-flight lesson again\), both the gate and naive ascent remove it fully to the base\-model reference, but the gate pays∼1\.7×\\sim 1\.7\\timesless collateral damage at both scales \(retained certificate0\.1010\.101versus0\.1770\.177at410410m,0\.0240\.024versus0\.0410\.041at11b; the directly measured retained\-loss rise,\+0\.09\+0\.09versus\+0\.18\+0\.18and\+0\.03\+0\.03versus\+0\.07\+0\.07nats, tracks the certificate\)\. The gate finds the zero\-floor path that unstructured gradient ascent misses\.

#### Attributed forgetting at scale\.

The decomposition Eq\. \([20](https://arxiv.org/html/2607.09202#A15.E20)\) is measurable end to end on real continual pretraining, yielding forgetting curves that state*which part was ever avoidable*\(pythia\-410mandpythia\-1b, LoRAr=8r\{=\}8on the four\-domain stream; per\-domain single\-task references, joint training as the class\-optimum surrogate at ranks\{2,8,32\}\\\{2,8,32\\\}, sequential naive and active\-set arms;kaggle\_forgetting\_attribution\.py,kaggle\_floor\_certificate\.py, archived results and analysis in the repository\)\.*The floor is certified, not assumed*: the generative domain classifier built from the single\-domain adapters separates held\-out chunks perfectly at410410m \(accuracy1\.0001\.000, cross\-entropy0\.00480\.0048nats/token\), and since any classifier’s cross\-entropy upper\-boundsH​\(T∣X\)≥I​\(T;Y∣X\)H\(T\\mid X\)\\geq I\(T;Y\\mid X\)and the floor is a property of the data, the one bound governs both models: at least97\.1%97\.1\\%\(410410m, observed forgetting0\.1640\.164nats/token\) and98\.8%98\.8\\%\(11b,0\.3990\.399\) of the measured forgetting was*avoidable in principle*\. The attribution, in the exact KL observable \(per\-tokenKL\\mathrm\{KL\}to the single\-domain anchors, netted of the measured seed\-to\-seed two\-models baseline of0\.0280\.028/0\.0200\.020\):

Three readings\. First, the class term is real \(66–13×13\\timesthe seed baseline\) but*flat in adapter rank*across\{2,8,32\}\\\{2,8,32\\\}: within\-class interference on this stream resides in the shared frozen geometry and joint optimization, not in adapter width, so expansion is not the indicated remedy—policy is, and the gate removes a third to a half of exactly the control term \(cross\-entropy view: naive versus gate pairedp=0\.044p=0\.044at410410m,p=0\.0018p=0\.0018at11b\)\. Second, the decomposition’s sanity anchor holds on real data: sequential training fits each domain exactly as well as single\-task training \(diagonal\-minus\-single\+0\.000\+0\.000at410410m\)\. Third, the Bregman residual of the preceding paragraph is visible at scale:𝔼​\[KL\]\\mathbb\{E\}\[\\mathrm\{KL\}\]exceeds the cross\-entropy excess by0\.140\.14–0\.230\.23nats/token because the single\-domain anchors are converged on their training chunks, not on the held\-out anchor\. Caveats: the recipe is fixed across scales \(learning rate and budget not re\-tuned, so the410410m→\\to11b growth in forgetting is a fixed\-recipe comparison\), joint training only upper\-bounds the class optimum \(the control term is conservative\), and the arms carry two to three seeds\.

#### Protection blocks autoregressive cascades at the source\.

In a generative loop, a one\-step corruption of the transition operator compounds over the horizon: after fine\-tuning a norm\-preserving recurrence on an overlapping second task, naive rollout error grows from0\.860\.86at one step to1\.001\.00\(full decorrelation\) at thirty\. Routing the fine\-tuning update throughker⁡ΣA\\ker\\Sigma\_\{A\}keeps the old task’s transition operator invariant, and the rollout error is zero at*every*horizon—the cascade is blocked where it starts, not corrected downstream\. Output\-side correction \(snapping generated states back onto the old task’s subspace\) removes leakage out of the subspace but cannot repair the corrupted dynamics inside it \(0\.72→0\.990\.72\\to 0\.99\): for autoregressive stability, where an update acts matters more than how its outputs are post\-processed\.

## References

- \[1\]J\. Kirkpatrick et al\., Overcoming catastrophic forgetting in neural networks,PNAS114\(13\):3521–3526, 2017\.
- \[2\]F\. Zenke, B\. Poole, S\. Ganguli, Continual learning through synaptic intelligence,ICML, 2017\.
- \[3\]D\. Lopez\-Paz, M\. Ranzato, Gradient episodic memory for continual learning,NeurIPS, 2017\.
- \[4\]A\. Chaudhry, M\. Ranzato, M\. Rohrbach, M\. Elhoseiny, Efficient lifelong learning with A\-GEM,ICLR, 2019\.
- \[5\]P\. Buzzega et al\., Dark experience for general continual learning: a strong, simple baseline,NeurIPS, 2020\.
- \[6\]Z\. Li, D\. Hoiem, Learning without forgetting,IEEE TPAMI40\(12\):2935–2947, 2018\.
- \[7\]M\. Farajtabar, N\. Azizan, A\. Mott, A\. Li, Orthogonal gradient descent for continual learning,AISTATS, 2020\.
- \[8\]G\. Saha, I\. Garg, K\. Roy, Gradient projection memory for continual learning,ICLR, 2021\.
- \[9\]G\. Zeng, Y\. Chen, B\. Cui, S\. Yu, Continual learning of context\-dependent processing in neural networks \(orthogonal weight modification\),Nature Machine Intelligence1:364–372, 2019\.
- \[10\]T\. Yu et al\., Gradient surgery for multi\-task learning \(PCGrad\),NeurIPS, 2020\.
- \[11\]A\. Jacot, F\. Gabriel, C\. Hongler, Neural tangent kernel: convergence and generalization in neural networks,NeurIPS, 2018\.
- \[12\]T\. Doan et al\., A theoretical analysis of catastrophic forgetting through the NTK overlap matrix,AISTATS, 2021\.
- \[13\]M\. A\. Bennani, T\. Doan, M\. Sugiyama, Generalisation guarantees for continual learning with orthogonal gradient descent,arXiv:2006\.11942, 2020\.
- \[14\]G\. Ilharco et al\., Editing models with task arithmetic,ICLR, 2023\.
- \[15\]P\. Yadav et al\., TIES\-Merging: resolving interference when merging models,NeurIPS, 2023\.
- \[16\]G\. M\. van de Ven, T\. Tuytelaars, A\. S\. Tolias, Three types of incremental learning,Nature Machine Intelligence4:1185–1197, 2022\.
- \[17\]V\. V\. Ramasesh, A\. Lewkowycz, E\. Dyer, Effect of scale on catastrophic forgetting in neural networks,ICLR, 2022\.
- \[18\]Z\. Wang et al\., Learning to prompt for continual learning \(L2P\),CVPR, 2022\.
- \[19\]Z\. Wang et al\., DualPrompt: complementary prompting for rehearsal\-free continual learning,ECCV, 2022\.
- \[20\]J\. S\. Smith et al\., CODA\-Prompt: continual decomposed attention\-based prompting,CVPR, 2023\.
- \[21\]I\. Evron et al\., How catastrophic can catastrophic forgetting be in linear regression?,COLT, 2022\.
- \[22\]S\. Dohare et al\., Loss of plasticity in deep continual learning,Nature632:768–774, 2024\.
- \[23\]Cheng et al\., Whoever started the interference should end it: guiding data\-free model merging via task vectors,ICML, 2025\.
- \[24\]Gargiulo et al\., Task singular vectors: reducing task interference in model merging,CVPR, 2025\.
- \[25\]Marczak et al\., No task left behind: isotropic model merging with common and task\-specific subspaces, 2025\.
- \[26\]Toward a holistic approach to continual model merging,arXiv:2509\.23592, 2026\.
- \[27\]Geodesic\-aligned gradient projection for continual task learning,CVPR, 2025\.
- \[28\]MINGLE: mixture of null\-space gated low\-rank experts for test\-time continual model merging,NeurIPS, 2025\.
- \[29\]Jung, Cho, and Yun, Convergence and implicit bias of gradient descent on continual linear classification,arXiv, 2025\.
- \[30\]Li and Hiratani, Optimal task order for continual learning of multiple tasks,arXiv, 2025\.
- \[31\]A broad evaluation of model merging for large language models,arXiv:2511\.21437, 2025\.
- \[32\]E\. Liberty, Simple and deterministic matrix sketching \(Frequent Directions\),KDD, 2013\.
- \[33\]W\. Gurnee, J\. Lindsey, et al\., Verbalizable representations form a global workspace in language models,Transformer Circuits Thread, Anthropic, 2026\. Code:github\.com/anthropics/jacobian\-lens\.
- \[34\]K\. Friedl, N\. Jaquier, A\. Liao, D\. Kragic, Learning Hamiltonian dynamics at scale: a differential\-geometric approach,ICML\(PMLR 306\), 2026\.
- \[35\]A\. Rosseau, R\. Müller, A\. Nowé, Preserving plasticity in continual learning via dynamical isometry,ICML\(PMLR 306\), 2026\.arXiv:2606\.09762\.