Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting

# Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting
Source: [https://arxiv.org/html/2610.22439](https://arxiv.org/html/2610.22439)
Kamil Faber††thanks:Equal contributionsAffiliation:Faculty of Computer ScienceAffiliation:AGH University of KrakowAffiliation:Krakow, PolandEmail:[kfaber@agh\.edu\.pl](mailto:)Katarzyna Filus\*& Łukasz PawelaAffiliation:Institute of Theoretical and Applied Informatics,Affiliation:Polish Academy of SciencesAffiliation:Gliwice, PolandEmail:[\{kfilus,lpawela\}@iitis\.pl](mailto:)

###### Abstract

Catastrophic forgetting remains a challenge in supervised continual learning, as adapting to new tasks causes representation drift and loss of previously learned information\. This raises the question of which representational changes should be constrained while preserving plasticity\. We therefore introduce*Sparse Autoencoder Geometric Anchoring \(SAGA\)*, which converts a Sparse Autoencoder \(SAE\) trained at each task boundary into local, low\-rank constraints on representation drift\. Replay examples specify where the constraints apply, while leading right singular directions of the encoder Jacobian specify which changes are penalized\. The resulting geometry is input dependent and can be retained without keeping the SAE for subsequent tasks’ training\. We establish its local relation to latent code matching, define its spectral truncation error, and derive a conditional bound on frozen\-readout drift\. With replay and snapshot history, the method achieves the highest mean Class\-IL accuracy and lowest mean forgetting among the evaluated ER, DER\+\+, ER\-ACE, and SGD baselines on three pretrained ViT\-B/16 benchmarks, with accuracy gains of 12\.15, 4\.79, and 3\.22 percentage points over the strongest conventional baseline on CIFAR\-100, CUB\-200, and ImageNet\-R, respectively\. From scratch results are more mixed, while rank ablations and geometry controls show that SAE\-induced low\-rank geometry offers useful stability\-plasticity trade\-offs\.

## 1Introduction

Catastrophic forgetting, i\.e\., the degradation of previously acquired knowledge during sequential training on new tasks, remains one of the key challenges in supervised continual learning[Wang et al\. \(2024\)](https://arxiv.org/html/2610.22439#bib.bib16)\. Common mitigation strategies include rehearsal[Rolnick et al\. \(2019\)](https://arxiv.org/html/2610.22439#bib.bib7), regularization and distillation[Kirkpatrick et al\. \(2017\)](https://arxiv.org/html/2610.22439#bib.bib14);[Li and Hoiem \(2018\)](https://arxiv.org/html/2610.22439#bib.bib6), and hybrid approaches[Buzzega et al\. \(2020\)](https://arxiv.org/html/2610.22439#bib.bib9)\. More recently, forgetting has increasingly been studied at the representation level, where learning subsequent tasks can alter feature\-space geometry and move previously useful information into directions that are no longer accessible to the original readout[Gu et al\. \(2023\)](https://arxiv.org/html/2610.22439#bib.bib12);[Filus et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib3);[Masip et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib5)\. This motivates preserving task\-relevant representation subspaces, for example those derived from previous task heads[Yamashita et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib20)or representations[Saha et al\. \(2021\)](https://arxiv.org/html/2610.22439#bib.bib19)\. However, these approaches typically define global subspaces, leaving open whether local geometry could provide selective control of representation drift\.

Sparse autoencoders \(SAEs\) map dense representations to sparse feature codes[Huben et al\. \(2024\)](https://arxiv.org/html/2610.22439#bib.bib18);[Bereska and Gavves \(2024\)](https://arxiv.org/html/2610.22439#bib.bib11);[Filus and Pokuciński \(2026\)](https://arxiv.org/html/2610.22439#bib.bib21)\. In continual learning, they have been used to analyze changes in visual representations[Filus et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib3);[Masip et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib5)and to regularize sparse activations during language\-model adaptation[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib1);[Ning et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib2)\. We ask a different question:*can a sparse encoder specify which local changes in the original representation should be penalized, rather than only providing a latent code to match?*

To address this question, we introduce*SAGA*\(*Sparse Autoencoder Geometric Anchoring*\), which uses SAE encoder Jacobians to derive task\-specific local geometry for representation anchoring\. At each task boundary, a fresh SAE is fitted to the backbone representations of a replay buffer snapshot\. Its encoder Jacobian defines a local quadratic geometry at each stored example\. The snapshot specifies*where*to anchor the old representation, while the leading Jacobian singular directions and their weights specify*which changes*to penalize\. Unlike a single global subspace, these constraints can differ across examples because the SAE’s active features change with the input\. Once these elements are computed, the SAE can be discarded\. Full\-stack SAGA combines this Jacobian anchor with two components\.*History*keeps the anchors from all task boundaries, so older constraints are not overwritten by snapshots fitted to already\-drifted representations\.*Replay*adds cross\-entropy on snapshot examples, supplying the classifier supervision that anchoring alone lacks in Class\-IL\.

The empirical benefit is clearest with pretrained representations\. On CIFAR\-100, CUB\-200, and ImageNet\-R, full\-stack*SAGA*configuration achieves the highest mean accuracy and lowest mean forgetting among the evaluated conventional baselines\. Against the corresponding latent\-anchoring configuration, it has higher mean Class\-IL accuracy in four out of five dataset/backbone settings, with CIFAR\-100 being the exception\. Rank ablations show that a small retained subspace can be sufficient, and geometry controls identify where SAE\-derived constraints help relative to global alternatives\. We distinguish these empirical findings from a universal optimality claim: the mathematical analysis states the coverage of old\-readout sensitivity needed for a geometric anchor to control functional drift\. Our contributions can be summarized in the following way:

- •A task\-boundary anchoring method that turns sparse encoders into input\-dependent, low\-rank quadratic constraints, separating spatial memory in replay examples from directional memory in Jacobian factors\.
- •An analysis of the relation to latent matching, spectral truncation, and frozen\-readout drift, with an explicit distinction between geometric energy and the normalization of the implemented training losses\.
- •An empirical evaluation across pretrained and from scratch backbones, including replay/history ablations, rank compression, alternative geometries, and dictionary size, that characterizes both the benefits and the limits of SAE\-derived anchoring\.

## 2Related work

Traditional research on catastrophic forgetting in supervised continual learning has primarily focused on classifier\-level degradation and mitigation strategies such as replay, regularization, distillation, and architectural approaches[Kirkpatrick et al\. \(2017\)](https://arxiv.org/html/2610.22439#bib.bib14);[Li and Hoiem \(2018\)](https://arxiv.org/html/2610.22439#bib.bib6);[Parisi et al\. \(2019\)](https://arxiv.org/html/2610.22439#bib.bib17);[Wang et al\. \(2024\)](https://arxiv.org/html/2610.22439#bib.bib16);[Verwimp et al\. \(2024\)](https://arxiv.org/html/2610.22439#bib.bib15)\. More recently, attention has shifted toward representations, including analyses that distinguish information loss from loss of accessibility to an old readout[Davari et al\. \(2022\)](https://arxiv.org/html/2610.22439#bib.bib4), and methods that explicitly preserve representation structure\. LUCIR constrains feature directions[Hou et al\. \(2019\)](https://arxiv.org/html/2610.22439#bib.bib30), PODNet distills spatially pooled representations[Douillard et al\. \(2020\)](https://arxiv.org/html/2610.22439#bib.bib29), and Backward Feature Projection preserves linear separability of past tasks[Gu et al\. \(2023\)](https://arxiv.org/html/2610.22439#bib.bib12)\. Related works preserve task\-relevant subspaces defined from activations[Saha et al\. \(2021\)](https://arxiv.org/html/2610.22439#bib.bib19)or classifier weights[Yamashita et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib20)\. At the parameter level, OGD constrains updates using gradients of old outputs[Farajtabar et al\. \(2020\)](https://arxiv.org/html/2610.22439#bib.bib25), while compressed quadratic constraints have been considered through sketched structural and Jacobian regularization[Li et al\. \(2021\)](https://arxiv.org/html/2610.22439#bib.bib24);[Heckel \(2022\)](https://arxiv.org/html/2610.22439#bib.bib23)\. Thus, neither subspace nor Jacobian\-based regularization is new in isolation\. In contrast to these global or parameter\-space constraints, our method uses the Jacobian of a task\-anchored SAE encoder with respect to the backbone representation to induce an input\-dependent, low\-rank geometry that adapts to each replay anchor\.

The use of encoder Jacobians to capture local representation geometry has also been studied in earlier work on autoencoders\. In particular, the Manifold Tangent Classifier extracts local directions from top singular vectors of a contractive autoencoder encoder Jacobian and encourages classifier invariance along them[Rifai et al\. \(2011\)](https://arxiv.org/html/2610.22439#bib.bib22)\. Our objective is different: rather than interpreting these directions as manifold tangents and enforcing classifier invariance, we freeze SAE sensitivity factors at task boundaries and use them to constrain subsequent backbone representation drift in continual learning\. We make no assumption that the selected TopK\-Jacobian directions correspond to manifold tangents or individual semantic concepts\. Our sparse encoders build onkk\-sparse autoencoders[Makhzani and Frey \(2013\)](https://arxiv.org/html/2610.22439#bib.bib27)and subsequent TopK SAE formulations[Gao et al\. \(2025\)](https://arxiv.org/html/2610.22439#bib.bib8)\. Jacobian Sparse Autoencoders[Farnik et al\. \(2025\)](https://arxiv.org/html/2610.22439#bib.bib26)address a complementary problem of learning sparse computational connections between model representations, rather than deriving continual\-learning constraints from the local geometry of a frozen sparse encoder\.

Recently, Sparse Autoencoders \(SAEs\), which map dense representations into sparse feature vectors[Huben et al\. \(2024\)](https://arxiv.org/html/2610.22439#bib.bib18);[Bereska and Gavves \(2024\)](https://arxiv.org/html/2610.22439#bib.bib11);[Filus and Pokuciński \(2026\)](https://arxiv.org/html/2610.22439#bib.bib21), have been applied to continual learning both to analyze and mitigate forgetting\. Diagnostic studies[Filus et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib3);[Masip et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib5)show that forgetting can reflect changes in feature accessibility and geometry rather than complete information loss, with a substantial component of representational drift being approximately linear\. For mitigation in Low\-Rank Adaptation \(LoRA\)\-based continual learning for LLMs, SAE\-FD[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib1)uses directional feature matching and magnitude preservation, while SAE\-guided activation regularization[Ning et al\. \(2026\)](https://arxiv.org/html/2610.22439#bib.bib2)selectively protects SAE features\. In contrast, we study supervised continual learning in vision and train*task\-boundary SAEs*rather than relying on a shared pretrained SAE space\. Instead of directly preserving SAE activations, we export their local sensitivity geometry as low\-rank factors and constrain representation drift along its dominant directions, without retaining the SAE during subsequent training\. Our*Latent*baseline therefore serves as a direct TopK\-code matching ablation rather than a reproduction of these prior methods\.

While subspace regularization, Jacobian\-based geometry, and SAE\-based preservation have each been explored in different contexts, our approach introduces*task\-boundary, input\-dependent SAE geometry*for continual representation anchoring\. At each task boundary, the SAE is converted into local, low\-rank sensitivity directions that constrain subsequent representation drift and can be retained without keeping the SAE itself\. Comparisons with latent anchoring and random, PCA, and classifier SVD geometries isolate the role of the resulting SAE\-induced geometry\.

## 3Using SAE\-induced local geometry for forgetting mitigation

##### Spatial and directional anchors\.

Letssindex a completed task and letBs=\{xs,i\}i=1nsB\_\{s\}=\\\{x\_\{s,i\}\\\}\_\{i=1\}^\{n\_\{s\}\}be the reservoir\-buffer snapshot stored at that boundary\. It contains examples from the cumulative stream, not exclusively from taskss\. With backbone parametersθs\\theta\_\{s\}, storehs​\(x\)=fθs​\(x\)∈ℝdh\_\{s\}\(x\)=f\_\{\\theta\_\{s\}\}\(x\)\\in\\mathbb\{R\}^\{d\}and fit a fresh TopK SAE encoderEsE\_\{s\}[Makhzani and Frey \(2013\)](https://arxiv.org/html/2610.22439#bib.bib27);[Gao et al\. \(2025\)](https://arxiv.org/html/2610.22439#bib.bib8)to the snapshot representations\. At each stored example, compute

Js​\(x\)=JEs​\(hs​\(x\)\)=Us​\(x\)​Σs​\(x\)​Vs​\(x\)⊤,δs​\(x,θ\)=fθ​\(x\)−hs​\(x\)\.J\_\{s\}\(x\)=J\_\{E\_\{s\}\}\(h\_\{s\}\(x\)\)=U\_\{s\}\(x\)\\Sigma\_\{s\}\(x\)V\_\{s\}\(x\)^\{\\top\},\\qquad\\delta\_\{s\}\(x;\\theta\)=f\_\{\\theta\}\(x\)\-h\_\{s\}\(x\)\.\(1\)Define the full pullback semimetric and its spectral truncation by

Qs​\(h\)\\displaystyle Q\_\{s\}\(h\)=JEs​\(h\)⊤​JEs​\(h\),\\displaystyle=J\_\{E\_\{s\}\}\(h\)^\{\\top\}J\_\{E\_\{s\}\}\(h\),\(2\)Qs,r​\(x\)\\displaystyle Q\_\{s,r\}\(x\)=Vs,r​\(x\)​Σs,r​\(x\)2​Vs,r​\(x\)⊤⪰0,\\displaystyle=V\_\{s,r\}\(x\)\\Sigma\_\{s,r\}\(x\)^\{2\}V\_\{s,r\}\(x\)^\{\\top\}\\succeq 0,\(3\)whereVs,rV\_\{s,r\}contains therrretained right singular vectors\. The per\-example*geometric energy*and its snapshot average are

Gs,r​\(x,θ\)\\displaystyle G\_\{s,r\}\(x;\\theta\)=‖Σs,r​\(x\)​Vs,r​\(x\)⊤​δs​\(x,θ\)‖22=δs​\(x,θ\)⊤​Qs,r​\(x\)​δs​\(x,θ\),\\displaystyle=\\\|\\Sigma\_\{s,r\}\(x\)V\_\{s,r\}\(x\)^\{\\top\}\\delta\_\{s\}\(x;\\theta\)\\\|\_\{2\}^\{2\}=\\delta\_\{s\}\(x;\\theta\)^\{\\top\}Q\_\{s,r\}\(x\)\\delta\_\{s\}\(x;\\theta\),\(4\)R^s,r​\(θ\)\\displaystyle\\widehat\{R\}\_\{s,r\}\(\\theta\)=1\|Bs\|​∑x∈BsGs,r​\(x,θ\)\.\\displaystyle=\\frac\{1\}\{\|B\_\{s\}\|\}\\sum\_\{x\\in B\_\{s\}\}G\_\{s,r\}\(x;\\theta\)\.\(5\)The snapshot specifies*where*the old representation is anchored andQs,r​\(x\)Q\_\{s,r\}\(x\)specifies*which local changes*are penalized\. In Jacobian mode, the encoder is discarded after these factors have been computed; the cached representations and factors define the subsequent constraint\. Appendix[D\.4](https://arxiv.org/html/2610.22439#A4.SS4)specifies the architecture and construction\.

##### Training losses and snapshot history\.

For an anchor minibatchℳs⊂Bs\\mathcal\{M\}\_\{s\}\\subset B\_\{s\}, the raw Jacobian loss averages over examples*and*retained directions, whereas the latent baseline sums latent\-coordinate errors and averages over examples:

ℒJac,simpl\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{Jac\},s\}^\{\\mathrm\{impl\}\}=1\|ℳs\|​r​∑x∈ℳsGs,r​\(x,θ\),\\displaystyle=\\frac\{1\}\{\|\\mathcal\{M\}\_\{s\}\|r\}\\sum\_\{x\\in\\mathcal\{M\}\_\{s\}\}G\_\{s,r\}\(x;\\theta\),\(6\)ℒlatent,simpl\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{latent\},s\}^\{\\mathrm\{impl\}\}=1\|ℳs\|​∑x∈ℳs‖Es​\(fθ​\(x\)\)−Es​\(hs​\(x\)\)‖22\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{M\}\_\{s\}\|\}\\sum\_\{x\\in\\mathcal\{M\}\_\{s\}\}\\\|E\_\{s\}\(f\_\{\\theta\}\(x\)\)\-E\_\{s\}\(h\_\{s\}\(x\)\)\\\|\_\{2\}^\{2\}\.\(7\)Thus the raw training metric isQs,r/rQ\_\{s,r\}/r, notQs,rQ\_\{s,r\}\.Vanillaretains the latest snapshot;Historyretains all completed\-task snapshots\. Letℋt\\mathcal\{H\}\_\{t\}be this retained index set when learning tasktt\. An independent minibatch is drawn from each snapshot and the default mean reduction is

ℒanchorimpl=1\|ℋt\|​∑s∈ℋtℒsimpl\.\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}^\{\\mathrm\{impl\}\}=\\frac\{1\}\{\|\\mathcal\{H\}\_\{t\}\|\}\\sum\_\{s\\in\\mathcal\{H\}\_\{t\}\}\\mathcal\{L\}\_\{s\}^\{\\mathrm\{impl\}\}\.\(8\)Replayadds cross\-entropy on the minibatch from the*latest frozen snapshot*, rather than drawing from the live reservoir\. Withλ\\lambdathe anchor coefficient andρ\\rhothe replay coefficient, the objective is

ℒ=ℒstream​CE\+λ​ℒanchorimpl\+ρ​ℒsnapshot​CE\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{stream\\ CE\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}^\{\\mathrm\{impl\}\}\+\\rho\\mathcal\{L\}\_\{\\mathrm\{snapshot\\ CE\}\}\.\(9\)WithoutReplay,ρ=0\\rho=0\. Before the first snapshot is created, only the stream loss is used\. Stored images have deterministic preprocessing for anchor comparisons so that the old and current features refer to the same input view\.

##### Relation to latent matching and rank truncation\.

ForE⁡\(h\)=TopK⁡\(ReLU⁡\(W​h\+b\)\)E\(h\)=\\operatorname\{TopK\}\(\\operatorname\{ReLU\}\(Wh\+b\)\), TopK retains theKKlargest post\-ReLU values in their original coordinates\. If the positive active mask is unchanged alongh0\+τ​δh\_\{0\}\+\\tau\\delta,0≤τ≤10\\leq\\tau\\leq 1, then

E⁡\(h0\+δ\)−E⁡\(h0\)=JE​\(h0\)​δ\.E\(h\_\{0\}\+\\delta\)\-E\(h\_\{0\}\)=J\_\{E\}\(h\_\{0\}\)\\delta\.\(10\)At full retained rank, the unnormalized Jacobian energy therefore equals latent displacement energy\. The*implemented*raw losses satisfyℒlatentimpl=r​ℒJacimpl\\mathcal\{L\}\_\{\\mathrm\{latent\}\}^\{\\mathrm\{impl\}\}=r\\mathcal\{L\}\_\{\\mathrm\{Jac\}\}^\{\\mathrm\{impl\}\}within a fixed cell, so equal coefficients do not give equal local regularization strength\. Our latent comparison concerns these implemented objectives, not a matched\-strength isolation of support geometry\.

ForQ=J⊤​JQ=J^\{\\top\}JandQr=∑i=1rσi2​vi​vi⊤Q\_\{r\}=\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}^\{2\}v\_\{i\}v\_\{i\}^\{\\top\}, spectral truncation is optimal in operator and Frobenius norms and gives

0≤δ⊤​\(Q−Qr\)​δ≤σr\+12​‖δ‖22,dimker⁡Qr≥d−r\.0\\leq\\delta^\{\\top\}\(Q\-Q\_\{r\}\)\\delta\\leq\\sigma\_\{r\+1\}^\{2\}\\\|\\delta\\\|\_\{2\}^\{2\},\\qquad\\dim\\ker Q\_\{r\}\\geq d\-r\.\(11\)Missing singular values are set to zero\. The nullspace is local to an anchor: multiple examples or snapshots need not leave a shared subspace of that dimension\. Support changes and truncation distinguish the geometric energies; the1/r1/rreduction additionally changes the scale of the training loss\.

##### Functional relevance\.

Letgs:ℝd→ℝcsg\_\{s\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{c\_\{s\}\}be a frozen old readout,As=Jgs​\(h0\)A\_\{s\}=J\_\{g\_\{s\}\}\(h\_\{0\}\), and letVr,ΣrV\_\{r\},\\Sigma\_\{r\}retain positive singular values only\. WritingPr=Vr​Vr⊤P\_\{r\}=V\_\{r\}V\_\{r\}^\{\\top\}, defineαs,r=‖As​Vr​Σr−1‖op\\alpha\_\{s,r\}=\\\|A\_\{s\}V\_\{r\}\\Sigma\_\{r\}^\{\-1\}\\\|\_\{\\mathrm\{op\}\}andβs,r=‖As​\(I−Pr\)‖op\\beta\_\{s,r\}=\\\|A\_\{s\}\(I\-P\_\{r\}\)\\\|\_\{\\mathrm\{op\}\}\. IfJgsJ\_\{g\_\{s\}\}isLgL\_\{g\}\-Lipschitz on a convex neighborhood containing the displacement segment, then

‖gs​\(h0\+δ\)−gs​\(h0\)‖2≤αs,r​δ⊤​Qr​δ\+βs,r​‖δ‖2\+Lg2​‖δ‖22\.\\\|g\_\{s\}\(h\_\{0\}\+\\delta\)\-g\_\{s\}\(h\_\{0\}\)\\\|\_\{2\}\\leq\\alpha\_\{s,r\}\\sqrt\{\\delta^\{\\top\}Q\_\{r\}\\delta\}\+\\beta\_\{s,r\}\\\|\\delta\\\|\_\{2\}\+\\frac\{L\_\{g\}\}\{2\}\\\|\\delta\\\|\_\{2\}^\{2\}\.\(12\)This bound separates controlled geometric energy from old\-readout sensitivity outside the protected subspace\. It is a conditional frozen\-readout statement, not a guarantee for the changing Class\-IL classifier\. We do not estimateβs,r\\beta\_\{s,r\}in the reported experiments\. Appendix[A](https://arxiv.org/html/2610.22439#A1)provides the proofs, support\-boundary example, and further consequences\.

## 4Experimental setup

Datasets\.CIFAR\-100 \(100 classes, split into 10 tasks of 10 classes each; used both at native32×3232\\times 32resolution with a from scratch backbone, and upsampled to224×224224\\times 224with a pretrained backbone\)\. CUB\-200 \(200 fine\-grained bird species, 10 tasks of 20 classes each,224×224224\\times 224\)\. ImageNet\-R \(200 classes – renditions/art styles of ImageNet classes, 10 tasks of 20 classes each,224×224224\\times 224\)\. TinyImageNet \(200 classes, split into 10 tasks of 20 classes each\)\. For the from scratch regime we use CIFAR\-100 and TinyImageNet, which provide 500 training images per class, rather than CUB\-200 and ImageNet\-R \(30 and 120 images per class\), as training on such limited data is not possible\. more details are described in Appendix[D](https://arxiv.org/html/2610.22439#A4)\.

Backbones\.ViT\-B/16, pretrained on ImageNet\-21k\. ResNet\-18, trained entirely from scratch\. We include more details on the hyperparameters in Appendix[D\.2](https://arxiv.org/html/2610.22439#A4.SS2)

Scenario and metrics\.All experiments use the class\-incremental learning \(Class\-IL\) scenario, where the task identity is not available at test time\. Task\-IL scenario \(task identity given at test time\) is reported alongside as a secondary, less demanding diagnostic in Appendix[B](https://arxiv.org/html/2610.22439#A2)\. We implement our proposed methods within the Mammoth continual learning framework111[https://github\.com/aimagelab/mammoth](https://github.com/aimagelab/mammoth)[Buzzega et al\. \(2020\)](https://arxiv.org/html/2610.22439#bib.bib9);[Boschini et al\. \(2022\)](https://arxiv.org/html/2610.22439#bib.bib13)and use it for all experiments\. The primary metrics are final mean accuracy \(accuracy on all classes seen so far, averaged over tasks, after training on the last task\) and forgetting \(mean drop from each task’s peak accuracy to its accuracy at the end of training\), computed using Mammoth’s standard evaluation protocol\.

Protocol\.A replay buffer of size500500\(51205120for TinyImagenet\) is used throughout for every rehearsal\-based method \(ours and all four baselines\)\. Every number reported as “final” anywhere in this paper is the mean±\\pmstandard deviation over three seeds \(\{0,1,2\}\\\{0,1,2\\\}\) on the true test set; hyperparameter selection \(Section[D\.2](https://arxiv.org/html/2610.22439#A4.SS2)\) is always performed separately on a held\-out validation split and never touches the test set used for the numbers reported here\.

Anchor snapshots and memory\.The live reservoir contains500500examples, butHistoryretains an additional frozen task\-boundary snapshot after each completed task\. Each snapshot stores its then\-current reservoir images together with old representations and the corresponding anchor factors\. Thus History has auxiliary memory and anchor\-computation costs that grow with the number of completed tasks, even though the live replay reservoir remains fixed at500500examples; we therefore do not claim a strictly memory\-matched comparison to ER, DER\+\+, or ER\-ACE\. The default mean History reduction in Eq\.[8](https://arxiv.org/html/2610.22439#S3.E8)keeps the nominal anchor\-loss scale approximately independent of the number of retained snapshots\.

Evaluation plan\.Our main experimental evaluation can be organized into four parts:\(1\)Ablation regarding the design choices for the task\-boundary SAE\-based anchoring and a comparison between SAE\-Jacobian anchoring and direct SAE\-latent anchoring\. In the ablation we include combinations ofVanilla Latent/Vanilla Jacobian,HistoryandReplayconfigurations on a full experimental stack \(Section[5\.1](https://arxiv.org/html/2610.22439#S5.SS1)\);\(2\)Comparison of the Jacobian\-based anchoring \(SAGA\) with established continual learning baselines, including SGD \(no forgetting mitigation\), Experience Replay \(ER\)[Chaudhry et al\. \(2019\)](https://arxiv.org/html/2610.22439#bib.bib28);[Rolnick et al\. \(2019\)](https://arxiv.org/html/2610.22439#bib.bib7), Dark Experience Replay with Knowledge Distillation \(DER\+\+\)[Buzzega et al\. \(2020\)](https://arxiv.org/html/2610.22439#bib.bib9), and Experience Replay with Asymmetric Cross\-Entropy \(ER\-ACE\)[Caccia et al\. \(2022\)](https://arxiv.org/html/2610.22439#bib.bib10)on a full experimental stack \(for further experimental details see Appendix[D\.3](https://arxiv.org/html/2610.22439#A4.SS3), and for the results see Section[5\.2](https://arxiv.org/html/2610.22439#S5.SS2)\);\(3\)Analysis of the effect of the retained Jacobian rank \(r\) to assess performance in low\-rank anchoring regimes on example configurations \(Section[5\.3](https://arxiv.org/html/2610.22439#S5.SS3)\);\(4\)Comparison of SAE\-induced geometry with alternative reference subspaces used for anchoring derived from random directions, Principal Component Analysis \(PCA\), and classifier SVD on example experimental configuration \(Section[5\.4](https://arxiv.org/html/2610.22439#S5.SS4)\)\. We also perform the analysis of the effect of SAE dictionary size in Appendix[C](https://arxiv.org/html/2610.22439#A3), ranging from undercomplete to strongly overcomplete configurations, to determine whether overcompleteness is necessary for effective anchoring\. For computational efficiency, we use undercomplete SAEs in the main experiments, while our dictionary\-size ablation in Appendix[C](https://arxiv.org/html/2610.22439#A3)shows that overcomplete SAEs achieve comparable anchoring performance and can therefore be used when interpretability is also desired\.

## 5Experimental results

### 5\.1Comparison of Jacobian\-based anchoring with Latent\-based anchoring with SAEs

In this section, we compareJacobianand directLatentanchoring separately forpretrainedbackbones \(model: ViT\-B/16, datasets: CIFAR\-100, CUB\-200, ImageNet\-R\) and backbones trainedfrom scratch\(model: ResNet\-18, datasets: CIFAR\-100, TinyImageNet\), along with the ablation study of the design choices for both these methods\. Tables[1](https://arxiv.org/html/2610.22439#S5.T1)and[2](https://arxiv.org/html/2610.22439#S5.T2)report the Class\-IL accuracy and forgetting results for the pretrained and from scratch cases respectively\. Task\-IL results are provided in Appendix[B](https://arxiv.org/html/2610.22439#A2)\.

Table 1:SAE anchoring ablations for Class\-IL scenario withpretrainedViT\-B/16\. Results are mean±\\pmstd over three seeds\. Higher accuracy \(↑\\uparrow\) and lower forgetting \(↓\\downarrow\) are better\.The clearest ablation result is the benefit of combining replay with historical anchors\. For both anchor types and all five settings, the fullReplay \+ Historyconfiguration improves both mean Class\-IL accuracy and mean forgetting over the correspondingReplay\-only configuration\. History alone helps substantially on pretrained representations, but is less reliable from scratch\. Thus the useful setting is geometric anchoring together with replay and retained snapshots, rather than either standalone anchor\.

The comparison between the full latent and Jacobian configurations is more specific\. Jacobian anchoring improves mean accuracy in four of five settings\. On from scratch CIFAR\-100, accuracy rises from21\.44%21\.44\\%to28\.53%28\.53\\%and forgetting falls from48\.9048\.90to38\.4638\.46; both metrics also improve on TinyImageNet and pretrained ImageNet\-R\. CUB\-200 trades slightly higher accuracy for higher forgetting, while latent anchoring is better on CIFAR\-100\. These outcomes characterize the implemented methods; the differing loss reductions in Eqs\.[6](https://arxiv.org/html/2610.22439#S3.E6)and equation[7](https://arxiv.org/html/2610.22439#S3.E7)prevent attributing every difference solely to the choice of local geometry\.

Table 2:SAE anchoring ablations for Class\-IL scenario with ResNet\-18 trainedfrom scratch\. Results are mean±\\pmstd over three seeds\. Higher accuracy \(↑\\uparrow\) and lower forgetting \(↓\\downarrow\) are better\.
### 5\.2Comparison between Jacobian\-based SAE anchoring and common forgetting mitigation methods

In this section, we compareJacobiananchoring with common continual learning strategies – SGD \(no forgetting mitigation\), ER, ER\-ACE and DER\+\+ – separately forpretrainedbackbones \(model: ViT\-B/16, datasets: CIFAR\-100, CUB\-200, ImageNet\-R\) and backbones trainedfrom scratch\(model: ResNet\-18, datasets: CIFAR\-100, TinyImageNet\)\. Tables[3](https://arxiv.org/html/2610.22439#S5.T3)and[4](https://arxiv.org/html/2610.22439#S5.T4)report Class\-IL accuracy and forgetting for the pretrained and from scratch cases respectively\. Task\-IL results are provided in Appendix[B](https://arxiv.org/html/2610.22439#A2)\.

Table 3:Class\-IL accuracy and forgetting against continual learning baselines on threepretrainedViT\-B/16 settings\. Results are mean±\\pmstd over three seeds\. Higher accuracy \(↑\\uparrow\) and lower forgetting \(↓\\downarrow\) are better\.The strongest result is consistent performance on the three pretrained benchmarks\. In Table[3](https://arxiv.org/html/2610.22439#S5.T3), our full method has the highest mean accuracy and lowest mean forgetting among the conventional baselines in every setting\. Its Class\-IL accuracy exceeds the strongest such baseline by12\.1512\.15percentage points on CIFAR\-100,4\.794\.79on CUB\-200, and3\.223\.22on ImageNet\-R\. These comparisons include the additional historical snapshot storage described above; they are not equal\-total\-memory comparisons\.

The from scratch results define the limits of this advantage \(Table[4](https://arxiv.org/html/2610.22439#S5.T4)\)\. Our method improves on ER in accuracy on both datasets, but DER\+\+ or ER\-ACE attains higher final accuracy\. On TinyImageNet it has the lowest mean forgetting, while on CIFAR\-100 its forgetting is similar to DER\+\+ and ER\-ACE\. The contrast is consistent with anchoring being particularly useful when adapting pretrained representations, but architecture, data, and initialization differ between the regimes, therefore these experiments do not isolate pretraining as the causal factor\.

Table 4:Class\-IL accuracy and forgetting against continual learning baselines with a ResNet\-18 trainedfrom scratch\. Results are mean±\\pmstd over three seeds\. Higher accuracy \(↑\\uparrow\) and lower forgetting \(↓\\downarrow\) are better\.
### 5\.3Effect of anchor rankrr\.

Table[5](https://arxiv.org/html/2610.22439#S5.T5)studies the effect of the retained Jacobian rankr∈4,8,16,32,64r\\in\{4,8,16,32,64\}for SAGA \(Jacobian \+ Replay \+ History\), using a TopK SAE withK=64K=64and a pretrained ViT\-B/16 backbone and three datasets\. With raw weights, changingrralso changes the effective penalty strength because the loss averages over retained directions with decaying singular values\. We therefore normalize each anchor’s weights to unit mean square, preserving their relative weighting while keeping the nominal penalty strength constant across ranks\.

Class\-IL performance generally improves, however, substantially lower ranks remain competitive: evenr≪Kr\\ll Kpreserves strong performance, while Task\-IL accuracy is comparatively insensitive to the retained rank\. This indicates that the SAE\-induced geometry admits substantial low\-rank compression, although retaining a larger fraction of the Jacobian\-sensitive subspace can be beneficial for Class\-IL performance\.

Table 5:Effect of the retained Jacobian rankrrfor a TopK SAE withK=64K=64on accuracy, with normalized anchor weights \(constant nominal penalty strength acrossrr\) and pretrained ViT\-B/16\. Results are mean±\\pmstd over three seeds\. Higher results are better \(↑\\uparrow\)\.
### 5\.4Comparison of Low\-Rank Reference Geometries

To disentangle the contribution of the SAE\-induced geometry from low\-rank anchoring more broadly, we compare SAE\-Jacobian anchoring with random, PCA, and classifier SVD reference*geometries*of matched nominal rank for pretrained Class\-IL settings \(Table[6](https://arxiv.org/html/2610.22439#S5.T6)\) and a from scratch setting \(Table[7](https://arxiv.org/html/2610.22439#S5.T7)\)\. All rows use normalized anchor weighting, which fixes the total trace of the quadratic metric while retaining each geometry’s protected directions and relative within\-subspace spectral weights \(Appendix[D\.3](https://arxiv.org/html/2610.22439#A4.SS3)\)\. Thus PCA and classifier SVD comparisons vary more than the subspace alone, whereas the SAE geometry is additionally input dependent\.

The normalized controls show that the choice of geometry matters, but not in a uniform way\. SAE\-Jacobian has the highest mean accuracy and lowest mean forgetting on CUB\-200\. On ImageNet\-R, it is close to PCA and random anchoring: the0\.070\.07\-point accuracy difference from PCA is small relative to the reported standard deviations\. On the other hand, classifier SVD performs best on CIFAR\-100\. Therefore, these results support local SAE geometry as a useful option, not a universal replacement for global subspace constructions\.

Table 6:Class\-IL accuracy and forgetting for different reference geometries onpretrainedViT\-B/16\. All variants use normalized anchor weighting\. Results are mean±\\pmstd over three seeds\. Higher accuracy \(↑\\uparrow\) and lower forgetting \(↓\\downarrow\) are better\.For the from scratch setting \(Tab\.[7](https://arxiv.org/html/2610.22439#S5.T7)\), we additionally vary the nominal retained rank across all reference geometries\. SAE\-Jacobian is particularly effective atr=4r=4andr=8r=8in Class\-IL accuracy, while PCA has lower mean forgetting at every displayed rank\. Atr=16r=16, PCA and SAE\-Jacobian have nearly identical mean accuracy, and atr=64r=64SAE\-Jacobian again has higher mean accuracy\. These results show that alternative geometries occupy different stability–plasticity operating points rather than supporting a universal ordering\. The normalized comparison controls the total metric trace, but PCA, classifier SVD, and SAE\-Jacobian retain their own relative spectral weighting; the experiment is therefore a comparison of low\-rank geometries rather than of directions alone\. The functional relevance of the SAE geometry is characterized by the leakage termβs,r\\beta\_\{s,r\}in Eq\.[12](https://arxiv.org/html/2610.22439#S3.E12), which describes old\-readout sensitivity outside the protected subspace\.

Table 7:Class\-IL accuracy and forgetting for different reference geometries and ranks on CIFAR\-100 with ResNet\-18 trainedfrom scratch\(r=Kr=K\)\. All variants use normalized anchor weighting\. Results are mean±\\pmstd over three seeds\. Higher accuracy \(↑\\uparrow\) and lower forgetting \(↓\\downarrow\) are better\.

## 6Conclusions

We introduced Sparse Autoencoder Geometric Anchoring \(SAGA\), a representation\-level continual learning regularizer in which task\-specific sparse autoencoders induce local positive\-semidefinite geometries over stored representations\. Replay snapshots specify where anchoring is applied, while truncated Jacobian singular directions determine which local changes are constrained\. Our results show that SAEs can serve not only as latent feature spaces, but also as sources of sample\-specific geometric constraints on backbone drift\. These constraints can be compressed to a chosen rank and combined with replay supervision\. The strongest empirical result is the full\-stack SAGA’s accuracy and forgetting relative to conventional baselines across all three pretrained benchmarks\. From scratch and geometry control results show that the benefit is regime dependent, while the rank ablation shows that the SAE\-induced geometry can be substantially compressed with limited performance loss\. The contribution, therefore, is a concrete use of SAEs as local geometric memory, together with a theoretical analysis of what that memory controls and an empirical characterization of where it helps\.

##### Limitations\.

The Jacobian is a local linear description of the SAE and, for hard TopK encoders, its exact relation to latent anchoring holds only within a fixed activation/support cell; support transitions are a genuine source of discrepancy\. An unsupervised SAE also does not guarantee that its sensitive directions coincide with old\-task functional sensitivity, which is why our theoretical bound contains an explicit leakage term and why the PCA/classifier/random controls are important\. Results use three seeds, so small differences in means should not be over\-interpreted\. Historical anchors also retain complete task\-boundary reservoir snapshots and their stored representations/geometric factors, so their auxiliary memory and compute grow with the number of completed tasks even though the live replay reservoir remains fixed at500500examples\. Future work should therefore investigate anchor compression, more efficient SAE/Jacobian construction, and the resulting memory–accuracy–computation trade\-offs\.

### AI use statement

Generative AI tools were used to assist with research ideation, mathematical checking, drafting, and document organization\. The authors reviewed the AI\-assisted material and independently verified the mathematical statements, experimental results, citations, implementation decisions, and final claims included in the submission\. The authors take responsibility for the final content of the work\. Generative AI also was used to rewrite handwritten formulas and whiteboard photos into LaTeX equations\.

### Ethics statement

This work does not involve any sensitive aspects, such as human subjects, personal data, or user studies\. Continual learning is an important enabler for developing adaptive machine learning systems that can incorporate new knowledge while preserving previously acquired information\. Our work aims to advance this area by addressing catastrophic forgetting – its key challenge – through an anchoring method\. In conducting and reporting the study, we aimed to uphold high standards of scientific excellence through transparent methodology, reproducible experimental procedures, sharing our code, making it compatible with a commonly used framework \(Mammoth\) and careful reporting of both strengths and limitations\. We are not aware of direct harmful applications specific to the proposed method\. Nevertheless, as with other machine learning methods, its use in deployed systems may inherit risks associated with the underlying application domains, and such downstream effects should be considered\.

### Reproducibility Statement

To support reproducibility, we evaluated our method on publicly available continual learning datasets and implement it within the widely used Mammoth continual learning framework\. Our implementation follows the experimental settings and evaluation protocols described in the paper, and we provide the source code required to reproduce the experiments\. The code will be made publicly available on GitHub upon acceptance\.

#### Acknowledgments

This work was supported by the National Science Centre, Poland, under the OPUS grant 2025/59/B/ST6/00193\.

## References

- L\. Bereska and E\. GavvesMechanistic Interpretability for AI Safety – A Review\.arXiv preprint arXiv:2404\.14082\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.
- Boschiniet al\.\(2022\)M\. Boschini, L\. Bonicelli, P\. Buzzega, A\. Porrello, and S\. CalderaraClass\-Incremental Continual Learning into the eXtended DER\-verse\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§4](https://arxiv.org/html/2610.22439#S4.p3.1)\.
- Buzzegaet al\.\(2020\)P\. Buzzega, M\. Boschini, A\. Porrello, D\. Abati, and S\. CalderaraDark Experience for General Continual Learning: a Strong, Simple Baseline\.Advances in Neural Information Processing Systems33,pp\. 15920–15930\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§4](https://arxiv.org/html/2610.22439#S4.p3.1),[§4](https://arxiv.org/html/2610.22439#S4.p6.1)\.
- Cacciaet al\.\(2022\)L\. Caccia, R\. Aljundi, N\. Asadi, T\. Tuytelaars, J\. Pineau, and E\. BelilovskyNew Insights on Reducing Abrupt Representation Change in Online Continual Learning\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2104.05025)Cited by:[§4](https://arxiv.org/html/2610.22439#S4.p6.1)\.
- Chaudhryet al\.\(2019\)A\. Chaudhry, M\. Rohrbach, M\. Elhoseiny, T\. Ajanthan, P\. K\. Dokania, P\. H\. S\. Torr, and M\. RanzatoOn Tiny Episodic Memories in Continual Learning\.arXiv preprint arXiv:1902\.10486\.External Links:[Link](https://arxiv.org/abs/1902.10486)Cited by:[§4](https://arxiv.org/html/2610.22439#S4.p6.1)\.
- Davariet al\.\(2022\)M\. Davari, N\. Asadi, S\. Mudur, R\. Aljundi, and E\. BelilovskyProbing Representation Forgetting in Supervised and Unsupervised Continual Learning\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16712–16721\.Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Douillardet al\.\(2020\)A\. Douillard, M\. Cord, C\. Ollion, T\. Robert, and E\. VallePODNet: Pooled Outputs Distillation for Small\-Tasks Incremental Learning\.InEuropean Conference on Computer Vision,External Links:[Link](https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/3384_ECCV_2020_paper.php)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Farajtabaret al\.\(2020\)M\. Farajtabar, N\. Azizan, A\. Mott, and A\. LiOrthogonal Gradient Descent for Continual Learning\.InProceedings of the 23rd International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.108,pp\. 3762–3773\.External Links:[Link](https://proceedings.mlr.press/v108/farajtabar20a.html)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Farniket al\.\(2025\)L\. Farnik, T\. Lawson, C\. Houghton, and L\. AitchisonJacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 16178–16236\.External Links:[Link](https://proceedings.mlr.press/v267/farnik25a.html)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p2.1)\.
- Filuset al\.\(2026\)K\. Filus, K\. Faber, R\. Corizzo, and C\. KananLost or hidden? a concept\-level forgetting in supervised continual learning\.arXiv preprint arXiv:2605\.16374\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.
- Filus and Pokuciński \(2026\)K\. Filus and S\. PokucińskiMeasuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence\.arXiv preprint arXiv:2607\.17770\.External Links:[Link](https://arxiv.org/abs/2607.17770)Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.
- Gaoet al\.\(2025\)L\. Gao, T\. Dupre la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. WuScaling and evaluating sparse autoencoders\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 26721–26754\.Cited by:[§D\.4](https://arxiv.org/html/2610.22439#A4.SS4.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p2.1),[§3](https://arxiv.org/html/2610.22439#S3.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2023\)Q\. Gu, D\. Shim, and F\. ShkurtiPreserving linear separability in continual learning by backward feature projection\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 24286–24295\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Heckel \(2022\)R\. HeckelProvable Continual Learning via Sketched Jacobian Approximations\.InProceedings of the 25th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.151,pp\. 10448–10470\.External Links:[Link](https://proceedings.mlr.press/v151/heckel22a.html)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Houet al\.\(2019\)S\. Hou, X\. Pan, C\. C\. Loy, Z\. Wang, and D\. LinLearning a Unified Classifier Incrementally via Rebalancing\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 831–839\.External Links:[Link](https://openaccess.thecvf.com/content_CVPR_2019/html/Hou_Learning_a_Unified_Classifier_Incrementally_via_Rebalancing_CVPR_2019_paper.html)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Hubenet al\.\(2024\)R\. Huben, H\. Cunningham, L\. R\. Smith, A\. Ewart, and L\. SharkeySparse Autoencoders Find Highly Interpretable Features in Language Models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan,et al\.Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Liet al\.\(2021\)H\. Li, A\. Krishnan, J\. Wu, S\. Kolouri, P\. K\. Pilly, and V\. BravermanLifelong Learning with Sketched Structural Regularization\.InProceedings of the 13th Asian Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.157,pp\. 985–1000\.External Links:[Link](https://proceedings.mlr.press/v157/li21b.html)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Li and Hoiem \(2018\)Z\. Li and D\. HoiemLearning without Forgetting\.IEEE Transactions on Pattern Analysis and Machine Intelligence40\(12\),pp\. 2935–2947\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2017.2773081)Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Makhzani and Frey \(2013\)A\. Makhzani and B\. Freyk\-Sparse Autoencoders\.arXiv preprint arXiv:1312\.5663\.External Links:[Link](https://arxiv.org/abs/1312.5663)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p2.1),[§3](https://arxiv.org/html/2610.22439#S3.SS0.SSS0.Px1.p1.1)\.
- Masipet al\.\(2026\)S\. Masip, G\. M\. van de Ven, J\. Ferrando, and T\. TuytelaarsPutting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability\.arXiv preprint arXiv:2601\.22012\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.
- Ninget al\.\(2026\)E\. Ning, W\. Xue, D\. Lou, and Y\. GuoFrom weights to features: sae\-guided activation regularization for llm continual learning\.arXiv preprint arXiv:2606\.26629\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.
- Parisiet al\.\(2019\)G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. WermterContinual Lifelong Learning with Neural Networks: A Review\.Neural Networks113,pp\. 54–71\.Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Rifaiet al\.\(2011\)S\. Rifai, Y\. N\. Dauphin, P\. Vincent, Y\. Bengio, and X\. MullerThe Manifold Tangent Classifier\.InAdvances in Neural Information Processing Systems,Vol\.24\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2011/hash/d1f44e2f09dc172978a4d3151d11d63e-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p2.1)\.
- Rolnicket al\.\(2019\)D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. WayneExperience Replay for Continual Learning\.Advances in Neural Information Processing Systems32\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§4](https://arxiv.org/html/2610.22439#S4.p6.1)\.
- Sahaet al\.\(2021\)G\. Saha, I\. Garg, and K\. RoyGradient projection memory for continual learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Verwimpet al\.\(2024\)E\. Verwimp, R\. Aljundi, S\. Ben\-David, M\. Bethge, A\. Cossu, A\. Gepperth,et al\.Continual Learning: Applications and the Road Forward\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, X\. Zhang, H\. Su, and J\. ZhuA Comprehensive Survey of Continual Learning: Theory, Method and Application\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(8\),pp\. 5362–5383\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Yamashitaet al\.\(2026\)H\. Yamashita, T\. Kurita, and M\. OnishiContinual learning by regularization in row space of weight matrix of previous task for deep neural network\.InInternational Conference on Agents and Artificial Intelligence,Vol\.2,pp\. 1138–1148\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p1.1),[§2](https://arxiv.org/html/2610.22439#S2.p1.1)\.
- Zhanget al\.\(2026\)M\. Zhang, Y\. Li, L\. Li, D\. Shen, H\. Xiong, and Y\. SunSAE\-fd: sparse autoencoder feature distillation for continual learning of large language models\.arXiv preprint arXiv:2605\.25525\.Cited by:[§1](https://arxiv.org/html/2610.22439#S1.p2.1),[§2](https://arxiv.org/html/2610.22439#S2.p3.1)\.

## Appendix AMathematical details of task\-anchored SAE geometry

### A\.1Local pullback approximation

For a frozen task\-boundary SAE encoderEsE\_\{s\}, define the nonlinear displacement energy at an anchorh0h\_\{0\}by

ds​\(h0,h0\+δ\)2=‖Es​\(h0\+δ\)−Es​\(h0\)‖22\.d\_\{s\}\(h\_\{0\},h\_\{0\}\+\\delta\)^\{2\}=\\\|E\_\{s\}\(h\_\{0\}\+\\delta\)\-E\_\{s\}\(h\_\{0\}\)\\\|\_\{2\}^\{2\}\.\(13\)Assume thatEsE\_\{s\}is differentiable nearh0h\_\{0\}and that its Jacobian isLsL\_\{s\}\-Lipschitz in operator norm\. WritingJ=JEs​\(h0\)J=J\_\{E\_\{s\}\}\(h\_\{0\}\), Taylor’s theorem gives

Es​\(h0\+δ\)−Es​\(h0\)=J​δ\+r⁡\(δ\),‖r⁡\(δ\)‖2≤Ls2​‖δ‖22\.E\_\{s\}\(h\_\{0\}\+\\delta\)\-E\_\{s\}\(h\_\{0\}\)=J\\delta\+r\(\\delta\),\\qquad\\\|r\(\\delta\)\\\|\_\{2\}\\leq\\frac\{L\_\{s\}\}\{2\}\\\|\\delta\\\|\_\{2\}^\{2\}\.\(14\)Therefore

\|ds​\(h0,h0\+δ\)2−δ⊤​J⊤​J​δ\|\\displaystyle\\left\|d\_\{s\}\(h\_\{0\},h\_\{0\}\+\\delta\)^\{2\}\-\\delta^\{\\top\}J^\{\\top\}J\\delta\\right\|≤Ls​‖J‖op​‖δ‖23\+Ls24​‖δ‖24\.\\displaystyle\\leq L\_\{s\}\\\|J\\\|\_\{\\mathrm\{op\}\}\\\|\\delta\\\|\_\{2\}^\{3\}\+\\frac\{L\_\{s\}^\{2\}\}\{4\}\\\|\\delta\\\|\_\{2\}^\{4\}\.\(15\)ThusJ⊤​JJ^\{\\top\}Jis the local pullback semimetric induced by Euclidean distance in SAE latent space\.

### A\.2Exact affine ReLU–TopK geometry

Assume

E⁡\(h\)=TopK⁡\(ReLU⁡\(W​h\+b\)\)E\(h\)=\\mathrm\{TopK\}\(\\mathrm\{ReLU\}\(Wh\+b\)\)\(16\)whereTopK\\mathrm\{TopK\}is the hard coordinate\-masking operator used in the implementation: it keeps theKKlargest post\-ReLU values in their original latent coordinates and sets all remaining coordinates to zero\. Leth0h\_\{0\}be a regular point whose selected TopK coordinates are strictly positive and separated from the remaining coordinates\. LetS⁡\(h0\)S\(h\_\{0\}\)be the selected support and letMS⁡\(h0\)M\_\{S\(h\_\{0\}\)\}denote the corresponding diagonal mask\. Then

JE​\(h0\)=MS⁡\(h0\)​W\.J\_\{E\}\(h\_\{0\}\)=M\_\{S\(h\_\{0\}\)\}W\.\(17\)Consequently, for every displacementδ\\delta,

δ⊤​JE​\(h0\)⊤​JE​\(h0\)​δ=‖MS⁡\(h0\)​W​δ‖22,\\delta^\{\\top\}J\_\{E\}\(h\_\{0\}\)^\{\\top\}J\_\{E\}\(h\_\{0\}\)\\delta=\\\|M\_\{S\(h\_\{0\}\)\}W\\delta\\\|\_\{2\}^\{2\},\(18\)so full frozen\-Jacobian anchoring is exactly matching the old active preactivation changes under the frozen anchor mask\.

If the ReLU/TopK mask remains unchanged on the segment\{h0\+τ​δ:0≤τ≤1\}\\\{h\_\{0\}\+\\tau\\delta:0\\leq\\tau\\leq 1\\\}, the encoder is affine on that segment and hence

E⁡\(h0\+δ\)−E⁡\(h0\)=JE​\(h0\)​δ\.E\(h\_\{0\}\+\\delta\)\-E\(h\_\{0\}\)=J\_\{E\}\(h\_\{0\}\)\\delta\.\(19\)Therefore the*unnormalized geometric energy*of the full Jacobian equals the SAE\-latent displacement exactly inside one cell\. Becauserank⁡\(JE​\(h0\)\)≤K\\mathrm\{rank\}\(J\_\{E\}\(h\_\{0\}\)\)\\leq K, retaining every nonzero singular direction also gives equality of these per\-example energies\. The training reductions are not identical: the implementation divides the Jacobian energy by the retained rankrrthrough its mean over directions, whereas the latent loss sums latent\-coordinate errors\. Hence fixed\-cell equality should not be read as equality of the two implemented losses at the same numerical anchor weight\.

A simple sufficient support\-stability certificate can be written in terms of the TopK gap\. Let

z⁡\(h\)=ReLU⁡\(W​h\+b\)z\(h\)=\\mathrm\{ReLU\}\(Wh\+b\)\(20\)and letz\(1\)​\(h0\)≥⋯≥z\(m\)​\(h0\)z\_\{\(1\)\}\(h\_\{0\}\)\\geq\\cdots\\geq z\_\{\(m\)\}\(h\_\{0\}\)be the ordered coordinates\. ForK<mK<m, define

γK​\(h0\)=z\(K\)​\(h0\)−z\(K\+1\)​\(h0\)\.\\gamma\_\{K\}\(h\_\{0\}\)=z\_\{\(K\)\}\(h\_\{0\}\)\-z\_\{\(K\+1\)\}\(h\_\{0\}\)\.\(21\)IfγK​\(h0\)\>0\\gamma\_\{K\}\(h\_\{0\}\)\>0and

‖W​δ‖∞<γK​\(h0\)2,\\\|W\\delta\\\|\_\{\\infty\}<\\frac\{\\gamma\_\{K\}\(h\_\{0\}\)\}\{2\},\(22\)then the TopK support is unchanged\. Since coordinatewise ReLU is 1\-Lipschitz, every post\-ReLU coordinate moves by less thanγK/2\\gamma\_\{K\}/2, so the ordering across theK/\(K\+1\)K/\(K\+1\)gap cannot reverse\. A sufficient Euclidean condition is

∥δ∥2<γK​\(h0\)2​‖W‖2→∞,∥W∥2→∞=maxj∥Wj,:∥2\.\\\|\\delta\\\|\_\{2\}<\\frac\{\\gamma\_\{K\}\(h\_\{0\}\)\}\{2\\\|W\\\|\_\{2\\to\\infty\}\},\\qquad\\\|W\\\|\_\{2\\to\\infty\}=\\max\_\{j\}\\\|W\_\{j,:\}\\\|\_\{2\}\.\(23\)
Support changes can nevertheless separate latent and frozen\-Jacobian anchoring sharply\. ForK=1K=1,W=I2W=I\_\{2\},b=0b=0,a\>0a\>0,

h0=\(a\+ε,a\),h1=\(a,a\+ε\),h\_\{0\}=\(a\+\\varepsilon,a\),\\qquad h\_\{1\}=\(a,a\+\\varepsilon\),\(24\)we have‖h1−h0‖2=2​ε→0\\\|h\_\{1\}\-h\_\{0\}\\\|\_\{2\}=\\sqrt\{2\}\\,\\varepsilon\\to 0, while

‖E⁡\(h1\)−E⁡\(h0\)‖22=2​\(a\+ε\)2→2​a2,\\\|E\(h\_\{1\}\)\-E\(h\_\{0\}\)\\\|\_\{2\}^\{2\}=2\(a\+\\varepsilon\)^\{2\}\\to 2a^\{2\},\(25\)but the frozen full\-Jacobian penalty ath0h\_\{0\}is onlyε2→0\\varepsilon^\{2\}\\to 0\. Thus hard TopK support transitions are one mechanism by which latent matching and frozen local geometry genuinely differ even without rank truncation\.

### A\.3Optimal rank truncation

Let

Q=J⊤​J=∑j=1qσj2​vj​vj⊤,σ1≥⋯≥σq\>0,Q=J^\{\\top\}J=\\sum\_\{j=1\}^\{q\}\\sigma\_\{j\}^\{2\}v\_\{j\}v\_\{j\}^\{\\top\},\\qquad\\sigma\_\{1\}\\geq\\cdots\\geq\\sigma\_\{q\}\>0,\(26\)For1≤r≤q1\\leq r\\leq q, setσq\+1=0\\sigma\_\{q\+1\}=0and define

Qr=∑j=1rσj2​vj​vj⊤\.Q\_\{r\}=\\sum\_\{j=1\}^\{r\}\\sigma\_\{j\}^\{2\}v\_\{j\}v\_\{j\}^\{\\top\}\.\(27\)By the Eckart–Young–Mirsky theorem,QrQ\_\{r\}is a best rank\-rrapproximation toQQin both operator and Frobenius norms, with

‖Q−Qr‖op=σr\+12,‖Q−Qr‖F2=∑j\>rσj4\.\\\|Q\-Q\_\{r\}\\\|\_\{\\mathrm\{op\}\}=\\sigma\_\{r\+1\}^\{2\},\\qquad\\\|Q\-Q\_\{r\}\\\|\_\{F\}^\{2\}=\\sum\_\{j\>r\}\\sigma\_\{j\}^\{4\}\.\(28\)For everyδ\\delta,

0≤δ⊤​\(Q−Qr\)​δ=∑j\>rσj2​\(vj⊤​δ\)2≤σr\+12​‖δ‖22\.0\\leq\\delta^\{\\top\}\(Q\-Q\_\{r\}\)\\delta=\\sum\_\{j\>r\}\\sigma\_\{j\}^\{2\}\(v\_\{j\}^\{\\top\}\\delta\)^\{2\}\\leq\\sigma\_\{r\+1\}^\{2\}\\\|\\delta\\\|\_\{2\}^\{2\}\.\(29\)Moreover,rank⁡\(Qr\)≤r\\mathrm\{rank\}\(Q\_\{r\}\)\\leq r, sodimker⁡Qr≥d−r\\dim\\ker Q\_\{r\}\\geq d\-r\.

### A\.4Spatial sampling and directional compression

Letμssnap\\mu\_\{s\}^\{\\mathrm\{snap\}\}denote the distribution of input examples in the cumulative stream at task boundaryssthat the reservoir snapshotBsB\_\{s\}is intended to approximate\. Define the ideal nonlinear snapshot\-anchor energy

RsSAE​\(θ\)=𝔼x∼μssnap​\[‖Es​\(fθ​\(x\)\)−Es​\(hs​\(x\)\)‖22\]\.R\_\{s\}^\{\\mathrm\{SAE\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\mu\_\{s\}^\{\\mathrm\{snap\}\}\}\\left\[\\\|E\_\{s\}\(f\_\{\\theta\}\(x\)\)\-E\_\{s\}\(h\_\{s\}\(x\)\)\\\|\_\{2\}^\{2\}\\right\]\.\(30\)Withδs​\(x,θ\)=fθ​\(x\)−hs​\(x\)\\delta\_\{s\}\(x;\\theta\)=f\_\{\\theta\}\(x\)\-h\_\{s\}\(x\), set

qs​\(x,θ\)=δs⊤​Qs​\(hs​\(x\)\)​δs,qs,r​\(x,θ\)=δs⊤​Qs,r​\(x\)​δs,q\_\{s\}\(x;\\theta\)=\\delta\_\{s\}^\{\\top\}Q\_\{s\}\(h\_\{s\}\(x\)\)\\delta\_\{s\},\\qquad q\_\{s,r\}\(x;\\theta\)=\\delta\_\{s\}^\{\\top\}Q\_\{s,r\}\(x\)\\delta\_\{s\},\(31\)whereQsQ\_\{s\}is the full pullback metric from Eq\.[2](https://arxiv.org/html/2610.22439#S3.E2)\. Then

\|RsSAE​\(θ\)−R^s,r​\(θ\)\|≤\\displaystyle\\left\|R\_\{s\}^\{\\mathrm\{SAE\}\}\(\\theta\)\-\\widehat\{R\}\_\{s,r\}\(\\theta\)\\right\|\\leq\{\}𝔼x∼μssnap​\|‖Es​\(hs\+δs\)−Es​\(hs\)‖22−qs​\(x,θ\)\|⏟ℰlocal\\displaystyle\\underbrace\{\\mathbb\{E\}\_\{x\\sim\\mu\_\{s\}^\{\\mathrm\{snap\}\}\}\\left\|\\\|E\_\{s\}\(h\_\{s\}\+\\delta\_\{s\}\)\-E\_\{s\}\(h\_\{s\}\)\\\|\_\{2\}^\{2\}\-q\_\{s\}\(x;\\theta\)\\right\|\}\_\{\\mathcal\{E\}\_\{\\mathrm\{local\}\}\}\(32\)\+𝔼x∼μssnap​\[qs​\(x,θ\)−qs,r​\(x,θ\)\]⏟ℰrank\\displaystyle\+\\underbrace\{\\mathbb\{E\}\_\{x\\sim\\mu\_\{s\}^\{\\mathrm\{snap\}\}\}\\left\[q\_\{s\}\(x;\\theta\)\-q\_\{s,r\}\(x;\\theta\)\\right\]\}\_\{\\mathcal\{E\}\_\{\\mathrm\{rank\}\}\}\(33\)\+\|𝔼x∼μssnap​qs,r​\(x,θ\)−1\|Bs\|​∑xi∈Bsqs,r​\(xi,θ\)\|⏟ℰsample\.\\displaystyle\+\\underbrace\{\\left\|\\mathbb\{E\}\_\{x\\sim\\mu\_\{s\}^\{\\mathrm\{snap\}\}\}q\_\{s,r\}\(x;\\theta\)\-\\frac\{1\}\{\|B\_\{s\}\|\}\\sum\_\{x\_\{i\}\\in B\_\{s\}\}q\_\{s,r\}\(x\_\{i\};\\theta\)\\right\|\}\_\{\\mathcal\{E\}\_\{\\mathrm\{sample\}\}\}\.\(34\)The three terms quantify local/support error, directional compression, and finite spatial sampling\. The rank term obeys

ℰrank≤𝔼x∼μssnap​\[σs,r\+1​\(x\)2​‖δs​\(x,θ\)‖22\],\\mathcal\{E\}\_\{\\mathrm\{rank\}\}\\leq\\mathbb\{E\}\_\{x\\sim\\mu\_\{s\}^\{\\mathrm\{snap\}\}\}\\left\[\\sigma\_\{s,r\+1\}\(x\)^\{2\}\\\|\\delta\_\{s\}\(x;\\theta\)\\\|\_\{2\}^\{2\}\\right\],\(35\)andℰlocal=0\\mathcal\{E\}\_\{\\mathrm\{local\}\}=0inside a fixed affine ReLU–TopK cell\. This decomposition is written for the unnormalized geometric energyR^s,r\\widehat\{R\}\_\{s,r\}; the raw Jacobian training loss carries the additional factor1/r1/rfrom Eq\.[6](https://arxiv.org/html/2610.22439#S3.E6)\.

For intuition about finite spatial resolution, fix the entire loss function, includinghsh\_\{s\},EsE\_\{s\},rr, andθ\\theta, independently of a hypothetical i\.i\.d\. evaluation sampleBsB\_\{s\}fromμssnap\\mu\_\{s\}^\{\\mathrm\{snap\}\}\. If0≤qs,r​\(x,θ\)≤Ms0\\leq q\_\{s,r\}\(x;\\theta\)\\leq M\_\{s\}almost surely, Hoeffding’s inequality gives, with probability at least1−η1\-\\eta,

ℰsample≤Ms​log⁡\(2/η\)2​\|Bs\|\.\\mathcal\{E\}\_\{\\mathrm\{sample\}\}\\leq M\_\{s\}\\sqrt\{\\frac\{\\log\(2/\\eta\)\}\{2\|B\_\{s\}\|\}\}\.\(36\)This is an illustrative fixed\-model sampling statement, not a generalization guarantee for the trained continual learner: the implementation uses reservoir snapshots to fit the SAE and later optimize the backbone, so the independence assumption does not hold for those same stored examples\.

### A\.5Functional leakage outside the protected subspace

Letgs:ℝd→ℝcsg\_\{s\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{c\_\{s\}\}be a frozen old\-task logit map and letAs=Jgs​\(h0\)A\_\{s\}=J\_\{g\_\{s\}\}\(h\_\{0\}\)\. LetQr=Vr​Σr2​Vr⊤Q\_\{r\}=V\_\{r\}\\Sigma\_\{r\}^\{2\}V\_\{r\}^\{\\top\}be formed fromrrretained*nonzero*singular values, withσr\>0\\sigma\_\{r\}\>0, and setPr=Vr​Vr⊤P\_\{r\}=V\_\{r\}V\_\{r\}^\{\\top\}\. Define

αs,r=‖As​Vr​Σr−1‖op,βs,r=‖As​\(I−Pr\)‖op\.\\alpha\_\{s,r\}=\\\|A\_\{s\}V\_\{r\}\\Sigma\_\{r\}^\{\-1\}\\\|\_\{\\mathrm\{op\}\},\\qquad\\beta\_\{s,r\}=\\\|A\_\{s\}\(I\-P\_\{r\}\)\\\|\_\{\\mathrm\{op\}\}\.\(37\)Ifgsg\_\{s\}has anLgL\_\{g\}\-Lipschitz Jacobian on a convex neighborhood containing the displacement segment, Taylor expansion gives

gs​\(h0\+δ\)−gs​\(h0\)=As​δ\+ρ⁡\(δ\),‖ρ⁡\(δ\)‖2≤Lg2​‖δ‖22\.g\_\{s\}\(h\_\{0\}\+\\delta\)\-g\_\{s\}\(h\_\{0\}\)=A\_\{s\}\\delta\+\\rho\(\\delta\),\\qquad\\\|\\rho\(\\delta\)\\\|\_\{2\}\\leq\\frac\{L\_\{g\}\}\{2\}\\\|\\delta\\\|\_\{2\}^\{2\}\.\(38\)DecomposingAs​δ=As​Pr​δ\+As​\(I−Pr\)​δA\_\{s\}\\delta=A\_\{s\}P\_\{r\}\\delta\+A\_\{s\}\(I\-P\_\{r\}\)\\deltaand using

As​Pr​δ=As​Vr​Σr−1​\(Σr​Vr⊤​δ\)A\_\{s\}P\_\{r\}\\delta=A\_\{s\}V\_\{r\}\\Sigma\_\{r\}^\{\-1\}\(\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\\delta\)\(39\)yields

‖gs​\(h0\+δ\)−gs​\(h0\)‖2≤αs,r​δ⊤​Qr​δ\+βs,r​‖δ‖2\+Lg2​‖δ‖22\.\\\|g\_\{s\}\(h\_\{0\}\+\\delta\)\-g\_\{s\}\(h\_\{0\}\)\\\|\_\{2\}\\leq\\alpha\_\{s,r\}\\sqrt\{\\delta^\{\\top\}Q\_\{r\}\\delta\}\+\\beta\_\{s,r\}\\\|\\delta\\\|\_\{2\}\+\\frac\{L\_\{g\}\}\{2\}\\\|\\delta\\\|\_\{2\}^\{2\}\.\(40\)Thus a low\-rank SAE anchor can control old\-task function only to the extent that old\-task sensitivity outside the protected subspace, described byβs,r\\beta\_\{s,r\}, is small\.

This statement concerns compatibility with the frozen old\-task readout\. In the Class\-IL experiments the live classifier continues to change, so classifier\-parameter drift and competition from newly introduced classes can alter logits even when the representation remains compatible with the old frozen readout\.

For a linear old readoutgs​\(h\)=As​h\+bsg\_\{s\}\(h\)=A\_\{s\}h\+b\_\{s\}and anyQ⪰0Q\\succeq 0, there exists a finiteccsuch that

‖As​δ‖22≤c​δ⊤​Q​δfor all​δ\\\|A\_\{s\}\\delta\\\|\_\{2\}^\{2\}\\leq c\\,\\delta^\{\\top\}Q\\delta\\qquad\\text\{for all \}\\delta\(41\)if and only if

ker⁡Q⊆ker⁡As\.\\ker Q\\subseteq\\ker A\_\{s\}\.\(42\)When this holds, the smallest constant is

c⋆=∥AsQ†/2∥op2\.c\_\{\\star\}=\\\|A\_\{s\}Q^\{\\dagger/2\}\\\|\_\{\\mathrm\{op\}\}^\{2\}\.\(43\)This gives an exact algebraic version of the same requirement: no old\-task\-sensitive direction may lie completely in the anchor nullspace\.

### A\.6Same\-point historical quadratic aggregation

The following identity applies when several historical quadratic anchors constrain the*same*current representationhh\. It is useful for interpreting how multiple local PSD constraints combine, but it is not a literal reduction of the full implementation, where different task\-boundary snapshots generally contain different replay examples and are evaluated on independently sampled minibatches\. Let

Lhist​\(h\)=∑s=1Tωs​\(h−hs\)⊤​Qs​\(h−hs\),ωs\>0,L\_\{\\mathrm\{hist\}\}\(h\)=\\sum\_\{s=1\}^\{T\}\\omega\_\{s\}\(h\-h\_\{s\}\)^\{\\top\}Q\_\{s\}\(h\-h\_\{s\}\),\\qquad\\omega\_\{s\}\>0,\(44\)withQs⪰0Q\_\{s\}\\succeq 0\. Define

QH=∑sωs​Qs,bH=∑sωs​Qs​hs,cH=∑sωs​hs⊤​Qs​hs\.Q\_\{H\}=\\sum\_\{s\}\\omega\_\{s\}Q\_\{s\},\\qquad b\_\{H\}=\\sum\_\{s\}\\omega\_\{s\}Q\_\{s\}h\_\{s\},\\qquad c\_\{H\}=\\sum\_\{s\}\\omega\_\{s\}h\_\{s\}^\{\\top\}Q\_\{s\}h\_\{s\}\.\(45\)Withh⋆=QH†​bHh\_\{\\star\}=Q\_\{H\}^\{\\dagger\}b\_\{H\},

Lhist​\(h\)=\(h−h⋆\)⊤​QH​\(h−h⋆\)\+κ,L\_\{\\mathrm\{hist\}\}\(h\)=\(h\-h\_\{\\star\}\)^\{\\top\}Q\_\{H\}\(h\-h\_\{\\star\}\)\+\\kappa,\(46\)whereκ=cH−bH⊤​QH†​bH≥0\\kappa=c\_\{H\}\-b\_\{H\}^\{\\top\}Q\_\{H\}^\{\\dagger\}b\_\{H\}\\geq 0\. Since allQsQ\_\{s\}are positive semidefinite,

ker⁡QH=⋂s=1Tker⁡Qs\.\\ker Q\_\{H\}=\\bigcap\_\{s=1\}^\{T\}\\ker Q\_\{s\}\.\(47\)Thus, at a common point, adding PSD historical anchors can only shrink the common unconstrained subspace\. In the actual training procedure, History instead averages or sums stochastic losses from separately stored task\-boundary snapshots; the default is the mean reduction, so total nominal anchor\-loss scale does not grow linearly with the number of snapshots, although storage and compute do\.

## Appendix BTask\-IL results

For completeness, we report Task\-IL accuracy and forgetting corresponding to the main Class\-IL experiments\. Task\-IL assumes that the task identity is available at test time and serves as a secondary diagnostic\.

Table 8:Task\-IL accuracy and forgetting for SAE anchoring ablations with a pretrained ViT\-B/16\. Results are mean±\\pmstd over three seeds\.Table 9:Task\-IL accuracy and forgetting for SAE anchoring ablations with a ResNet\-18 trained from scratch\.Overall, the Task\-IL results are consistent with the Class\-IL findings included in the main paper\. In particular, combining replay with persistent task history remains the strongest SAE\-anchoring configuration, while SAE\-Jacobian anchoring shows particularly strong performance in the pretrained setting, as well compared to the SAE latent anchoring in the from scratch scenario\. Our method also consistently reduces forgetting relative to standard baselines\.

Table 10:Task\-IL accuracy and forgetting against continual learning baselines on pretrained ViT\-B/16\.Table 11:Task\-IL accuracy and forgetting against continual learning baselines with a ResNet\-18 trained from scratch\.
## Appendix CEffect of SAE dictionary size \(overcompleteness\)

We perform all of our main experiments with SAE dictionary of size256256, i\.e\. an*undercomplete*dictionary relative to the ResNet\-18 and ViT feature dimensions \(D=512D=512, an expansion factor of0\.5×0\.5\\times\), whereas SAEs are more commonly trained overcomplete when used for representational disentanglement and interpretability\. Table[12](https://arxiv.org/html/2610.22439#A3.T12)tests different dictionary sizes on CIFAR\-100 with ResNet\-18, ranging from0\.5×0\.5\\times\(the default for dictionary size256256in the ResNet\-18 case\) up to8×8\\times, at fixedsae\_topk=jacobian\_k=16\\texttt\{sae\\\_topk\}=\\texttt\{jacobian\\\_k\}=16\. Class\-IL accuracy is rather flat across dictionary sizes and within one standard deviation of the default256256\-unit dictionary at every size tested\. The only configuration that is both worse and markedly less stable across seeds is dictionary size10241024\(expansion factor22\)\. Our obtained results suggest that overcomplete dictionaries are not essential for the proposed anchoring mechanism, however, they can still be used when interpretability of continual learning models representations is an additional objective, while the same SAE can be used to derive the Jacobian\-based anchors for continual learning\.

Table 12:Effect of SAE dictionary size on accuracy for CIFAR\-100 with a ResNet\-18 trained from scratch, at fixedK=r=16K=r=16\. Expansion factor is relative to the backbone’s512512\-dimensional feature space\. Results are mean±\\pmstd over three seeds\. Higher results are better \(↑\\uparrow\)\.
## Appendix DDetailed experimental setup

### D\.1Datasets

Table[13](https://arxiv.org/html/2610.22439#A4.T13)summarizes the datasets\. Classes are assigned to tasks in the class order of the dataset \(no class permutation\), so the task sequence is identical for every seed; each task is trained for a fixed number of epochs \(Table[14](https://arxiv.org/html/2610.22439#A4.T14)\)\. CIFAR\-100 is used at native32×3232\\times 32resolution with a from scratch ResNet\-18 and upsampled to224×224224\\times 224with the pretrained ViT\-B/16\. TinyImageNet has500500training images per class and its official validation split serves as the test set\. CUB\-200 \(fine\-grained bird species\) has about3030training images per class and ImageNet\-R \(renditions such as art and cartoons of200200ImageNet classes\)120120per class, with the fixed train/test split of the respective data release\. In all settings the seed changes the initialization of the new layers and of the SAE, the data order and the buffer contents, but not the task sequence\.

Table 13:Datasets\. “Tasks” is the number of tasks×\\timesclasses per task\. Input: image size seen by the network\. Train\-time augmentation is listed in the text; test images are only resized \(and, where stated, center\-cropped\) and normalized\.Datasets in training from scratchA network trained from scratch needs many images per class\. CIFAR\-100 and TinyImageNet provide 500 training images per class, whereas ImageNet\-R provides 120 \(24,000 images for 200 classes, moreover renditions of varying style\) and CUB\-200 only 30 \(5,994 images\)\. In preliminary runs of a ResNet\-18 trained from scratch on these two datasets \(10 epochs per task, three seeds\), all methods, including plain SGD, stayed below 10% Class\-IL accuracy, so they cannot be meaningfully compared\. In contrast, a ResNet\-18 trained from scratch on TinyImageNet reaches 25–32% with the rehearsal\-based methods \(and for SGD: 6\.8%\)\. We therefore evaluate the from\-scratch regime on CIFAR\-100 and TinyImageNet and the pretrained regime on the 224\-pixel datasets \(upscaled CIFAR\-100, ImageNet\-R and CUB\-200\)\.

### D\.2Hyperparameters

Table[14](https://arxiv.org/html/2610.22439#A4.T14)lists the tuned hyperparameters of our method \(learning rate and anchor weight\) together with the fixed training settings, for the main results\. The SAE\-specific hyperparameters are fixed in all of them \(sae\_epochs=20\\texttt\{sae\\\_epochs\}=20;sae\_lr=10−3\\texttt\{sae\\\_lr\}=10^\{\-3\}; dictionary size256256\)\.

Table 14:Tuned hyperparameters and fixed training settings for our method SAGA \(Jacobian \+ Replay \+ History\), by setting\.For every setting, our method’s learning rate and anchor weight were selected by a grid search \(learning rate×\\timesanchor weight\) evaluated on a held\-out validation split \(10%10\\%of the training set held out per class\), using a single screening seed\. All numbers reported elsewhere in the paper are then produced by a separate run at the winning hyperparameters, over three seeds on the separate test set\.

### D\.3Comparisons with reference geometries

To test whether SAE\-Jacobian anchoring benefits from SAE\-induced geometry specifically or from low\-rank anchoring more generally, we compare it with random, PCA, and classifier SVD geometries under the same Replay\+History protocol\. Each geometry supplies an orthonormal basisVrV\_\{r\}and nonnegative direction weightsSrS\_\{r\}, defining

Qr\(m\)=Vr\(m\)​Sr\(m\)​2​Vr\(m\)⊤\.Q\_\{r\}^\{\(m\)\}=V\_\{r\}^\{\(m\)\}S\_\{r\}^\{\(m\)2\}V\_\{r\}^\{\(m\)\\top\}\.\(48\)For these comparisons we useanchor\_weight\_norm=normalized, which rescales the weights according to

S~r=Sr1r​∑i=1rSi2\.\\widetilde\{S\}\_\{r\}=\\frac\{S\_\{r\}\}\{\\sqrt\{\\frac\{1\}\{r\}\\sum\_\{i=1\}^\{r\}S\_\{i\}^\{2\}\}\}\.\(49\)Because the implementation also averages squared projected drift over therrretained directions, the normalized per\-example loss is

1r​‖S~r​Vr⊤​δ‖22=δ⊤​Qr\(m\)​δtr⁡Qr\(m\)\.\\frac\{1\}\{r\}\\\|\\widetilde\{S\}\_\{r\}V\_\{r\}^\{\\top\}\\delta\\\|\_\{2\}^\{2\}=\\frac\{\\delta^\{\\top\}Q\_\{r\}^\{\(m\)\}\\delta\}\{\\operatorname\{tr\}Q\_\{r\}^\{\(m\)\}\}\.\(50\)Thus the comparison normalizes the total metric trace while preserving each geometry’s directions and relative within\-subspace spectral weighting\.

The concrete reference geometries are:

- •*Random*: drawrrGaussian directions and orthonormalize them by QR\. The directions use unit weights, soQr=Vr​Vr⊤Q\_\{r\}=V\_\{r\}V\_\{r\}^\{\\top\}and the implemented loss is‖Vr⊤​δ‖22/r\\\|V\_\{r\}^\{\\top\}\\delta\\\|\_\{2\}^\{2\}/r\. The seed of the basis is derived from the run seed and the task index\.
- •*PCA*: center the same representation dataset that would otherwise be used to fit the SAE, computeHc=U​S​V⊤H\_\{c\}=USV^\{\\top\}, and use the leading right\-singular directions together with their singular values\. Before normalization,QrPCA=Vr​Sr2​Vr⊤Q\_\{r\}^\{\\mathrm\{PCA\}\}=V\_\{r\}S\_\{r\}^\{2\}V\_\{r\}^\{\\top\}\.
- •*Classifier SVD*: decompose the frozen classifier weight matrixW=U​S​V⊤W=USV^\{\\top\}and use its leading right\-singular directions and singular values\. Before normalization, the full\-rank energy satisfies‖S​V⊤​δ‖22=‖W​δ‖22\\\|SV^\{\\top\}\\delta\\\|\_\{2\}^\{2\}=\\\|W\\delta\\\|\_\{2\}^\{2\}, i\.e\. squared drift of a frozen linear readout\. Under Eq\.[50](https://arxiv.org/html/2610.22439#A4.E50), the implemented full\-rank quantity is proportional to logit drift,‖W​δ‖22/‖W‖F2\\\|W\\delta\\\|\_\{2\}^\{2\}/\\\|W\\\|\_\{F\}^\{2\}\. The requested rank is capped by the numerical rank ofWW\.
- •*SAE Jacobian*: use the per\-example right\-singular directions and singular values of the local SAE encoder Jacobian\. Unlike the other three controls, this basis varies across anchor examples\.

PCA and classifier SVD therefore vary both the protected directions and their within\-subspace spectral weights\. The experiment should be interpreted as a comparison of alternative low\-rank geometries rather than of directions alone\.

### D\.4SAE setup

At every task boundary a fresh TopK SAE[Gao et al\. \(2025\)](https://arxiv.org/html/2610.22439#bib.bib8)is trained on the frozen backbone’s features of the reservoir snapshot; its encoder Jacobian is evaluated at the snapshot’s samples and, in the Jacobian mode used for all main results, the SAE is then discarded, keeping only the low\-rank directions and weights\.

Architecture\.SAEs act on the pooled pre\-logit featureh∈ℝDh\\in\\mathbb\{R\}^\{D\}\(class token after the final layer normalization for ViT\-B/16,D=768D=768; globally average\-pooled last\-stage output for ResNet\-18,D=512D=512\), used without centering or rescaling\. It has one hidden layer with a hard TopK activation,

E⁡\(h\)=TopKK⁡\(ReLU⁡\(We​h\+be\)\),h^=Wd​E​\(h\)\+bd,E\(h\)=\\operatorname\{TopK\}\_\{K\}\\\!\\big\(\\operatorname\{ReLU\}\(W\_\{e\}h\+b\_\{e\}\)\\big\),\\qquad\\hat\{h\}=W\_\{d\}E\(h\)\+b\_\{d\},\(51\)withWe∈ℝm×DW\_\{e\}\\in\\mathbb\{R\}^\{m\\times D\}andWd∈ℝD×mW\_\{d\}\\in\\mathbb\{R\}^\{D\\times m\};TopKK\\operatorname\{TopK\}\_\{K\}keeps theKKlargest post\-ReLU activations of each sample in place and zeroes the rest\. The weights are untied, the decoder columns are unconstrained, initialization is the PyTorch default, and there are2​D​m\+m\+D2Dm\+m\+Dparameters\. The values used for the main results are listed in Table[15](https://arxiv.org/html/2610.22439#A4.T15):m=256m=256andK=16K=16\(undercomplete:0\.33×0\.33\\timesthe feature dimension for ViT\-B/16,0\.5×0\.5\\timesfor ResNet\-18\)\. The ablations change them as follows:K=64K=64for all ranksr∈\{4,8,16,32,64\}r\\in\\\{4,8,16,32,64\\\}in the rank ablation, so that the SAE is identical across ranks;K=rK=rin the reference\-geometry comparison, exceptK=16K=16atr=4r=4; andm∈\{256,…,4096\}m\\in\\\{256,\\ldots,4096\\\}atK=r=16K=r=16in the dictionary\-size ablation\.

Training\.The SAE minimizes only the mean squared reconstruction error over minibatchesℬ\\mathcal\{B\},

ℒSAE=1\|ℬ\|​D​∑h∈ℬ‖h^−h‖22,\\mathcal\{L\}\_\{\\mathrm\{SAE\}\}=\\frac\{1\}\{\|\\mathcal\{B\}\|\\,D\}\\sum\_\{h\\in\\mathcal\{B\}\}\\big\\\|\\hat\{h\}\-h\\big\\\|\_\{2\}^\{2\},\(52\)with no sparsity penalty \(TopK provides the sparsity\), auxiliary loss or weight decay\. It uses Adam \(learning rate10−310^\{\-3\}, default momenta\) for2020epochs with batch size6464, reshuffled every epoch, independently of the continual learner’s optimizer\. It is initialized from scratch at every boundary and then frozen in evaluation mode:88steps per epoch \(160160in total\) for500500samples,16001600steps for the51205120samples of TinyImageNet\. These values are identical in all settings and were not tuned\.

Training data\.The SAE is trained without labels on the features of allN=min⁡\(images seen,buffer capacity\)N=\\min\(\\text\{images seen\},\\text\{buffer capacity\}\)images in the reservoir buffer at the boundary \(N≤500N\\leq 500;N≤5120N\\leq 5120for TinyImageNet\)\. As in the baselines, the reservoir is updated at every training step, so it approximates a uniform sample of the stream seen so far, contains all classes seen up to that task, and, when a task is trained for several epochs, may hold the same image more than once\. Images are stored without augmentation and passed through the deterministic part of the dataset’s test\-time preprocessing \(resizing, normalization\), as when the anchor loss compareshnewh\_\{\\mathrm\{new\}\}withholdh\_\{\\mathrm\{old\}\}; features come from the backbone at the boundary in evaluation mode\.

From the SAE to the geometry\.For a buffer imagexxwith featureholdh\_\{\\mathrm\{old\}\}, letπ1​\(x\),…,πK​\(x\)\\pi\_\{1\}\(x\),\\ldots,\\pi\_\{K\}\(x\)index itsKKlargest post\-ReLU activationsa=ReLU⁡\(We​hold\+be\)a=\\operatorname\{ReLU\}\(W\_\{e\}h\_\{\\mathrm\{old\}\}\+b\_\{e\}\)\. Away from ReLU and TopK boundaries the encoder is locally linear and its Jacobian is read from the weights in closed form:

Jx=\[1\[aπi​\(x\)\>0\]\(We\)πi\(x\),:\]i=1K∈ℝK×D\.J\_\{x\}=\\big\[\\,\\mathbb\{1\}\[a\_\{\\pi\_\{i\}\(x\)\}\>0\]\\;\(W\_\{e\}\)\_\{\\pi\_\{i\}\(x\),:\}\\,\\big\]\_\{i=1\}^\{K\}\\in\\mathbb\{R\}^\{K\\times D\}\.\(53\)We take the thin SVDJx=U​Σ​V⊤J\_\{x\}=U\\Sigma V^\{\\top\}\(in chunks of6464samples\), keep ther≤Kr\\leq Kleadingσi\\sigma\_\{i\}andviv\_\{i\}, and store them withholdh\_\{\\mathrm\{old\}\}, the image and its label in the snapshot; the normalized weighting rescalesσ\\sigmatoσ~\\tilde\{\\sigma\}\(Eq\.[49](https://arxiv.org/html/2610.22439#A4.E49)\)\. In Latent mode the frozen SAE is kept andE⁡\(hold\)∈ℝmE\(h\_\{\\mathrm\{old\}\}\)\\in\\mathbb\{R\}^\{m\}is stored instead\. Anchors therefore need only the buffer images, the backbone at the boundary and the SAE \(transiently, while the directions are computed\); no data of the finished task, old\-task classifier or gradients are used\. The reference geometries of Section[5\.4](https://arxiv.org/html/2610.22439#S5.SS4)train no SAE:*random*needs onlyDD,rrand a seed \(Gaussian matrix orthonormalized by QR; seed derived from the run seed and task index\),*PCA*the centered buffer features \(leading right singular vectors and values\),*classifier SVD*the classifier weights at the boundary\. These bases are global, oneD×rD\\times rmatrix shared by all anchors, whereas SAE\-Jacobian directions are per anchor\.

Table 15:SAE hyperparameters for the main results\.DD: feature dimension;mm: dictionary size;KK: TopK sparsity;rr: retained Jacobian rank;NN: buffer samples per snapshot \(SAE training set and anchors\); SAE parameters:2​D​m\+m\+D2Dm\+m\+D\. The training hyperparameters \(Adam, learning rate10−310^\{\-3\},2020epochs, batch size6464\) are identical in all settings\.

## Appendix EComputational Overhead

Table[16](https://arxiv.org/html/2610.22439#A5.T16)reports the mean wall\-clock training time per task for our method and SGD, included as the least computationally expensive baseline, across five different settings for which per\-task timing was recorded\. Independently measured GPU\-device time using CUDA events differs from wall\-clock time by less than1%1\\%in all cases, indicating that the runs are predominantly GPU\-bound with negligible CPU and data\-loading overhead\. We therefore omit GPU\-device time as a separate column\.

Table 16:Mean wall\-clock time per task \(seconds\), our method vs\. SGD, across thefivesettings with timing instrumentation\. Ratio = Ours / SGD\.The proposed method introduces a consistent computational overhead across all evaluated settings, with training time remaining within a narrow6\.4×6\.4\\times–7\.1×7\.1\\timesrange relative to SGD\. This indicates that the additional cost is predictable and primarily comes from SAE training, Jacobian/SVD computation at task boundaries, and the additional anchoring passes during optimization\.