Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learning
Summary
This paper introduces a diagnostic framework using Sparse Autoencoders to analyze concept-level forgetting in continual learning, finding that much forgetting is due to representational inaccessibility rather than erasure.
arXiv:2605.16374v1 Announce Type: new
Abstract: Continual learning studies how models can adapt to new tasks while retaining previously acquired knowledge. Although a broad spectrum of methods has been proposed to mitigate catastrophic forgetting, the field remains predominantly performance-driven, with limited insight into what forgetting actually corresponds to within the vision model's representation space. Prior work has primarily analyzed forgetting through task-level performance or coarse measures of representational drift, without disentangling output-level accessibility from changes in finer-grained internal structure. To this end, we propose a diagnostic framework that leverages Sparse Autoencoders (SAEs) to define a task-anchored latent feature space, enabling analysis of how task-specific information evolves at a finer granularity, where individual SAE latents are treated as concept proxies for recurring and relatively disentangled visual patterns in the model's internal computations. Within this framework, we decompose forgetting into apparent concept deletion, recoverability, and decodability. We show that a large portion of seemingly lost concept-level information can often be recovered under linearity assumption, with concept decodability degrading as more tasks are introduced. Overall, our findings suggest that a significant part of concept-level forgetting can be attributed to changes in the representational accessibility rather than complete information erasure.
View Cached Full Text
Cached at: 05/19/26, 06:42 AM
# Lost or Hidden? A Concept-Level Forgetting in Supervised Continual Learning
Source: [https://arxiv.org/html/2605.16374](https://arxiv.org/html/2605.16374)
Katarzyna Filus Institute of Theoretical and Applied Informatics Polish Academy of Sciences, Gliwice, Poland kfilus@iitis\.pl &Kamil Faber AGH University of Krakow Krakow, Poland kfaber@agh\.edu\.pl Roberto Corizzo American University Washington DC, USA rcorizzo@american\.edu &Christopher Kanan University of Rochester Rochester, New York, USA ckanan@cs\.rochester\.edu
###### Abstract
Continual learning studies how models can adapt to new tasks while retaining previously acquired knowledge\. Although a broad spectrum of methods has been proposed to mitigate catastrophic forgetting, the field remains predominantly performance\-driven, with limited insight into what forgetting actually corresponds to within the vision model’s representation space\. Prior work has primarily analyzed forgetting through task\-level performance or coarse measures of representational drift, without disentangling output\-level accessibility from changes in finer\-grained internal structure\. To this end, we propose a diagnostic framework that leverages Sparse Autoencoders \(SAEs\) to define a task\-anchored latent feature space, enabling analysis of how task\-specific information evolves at a finer granularity, where individual SAE latents are treated as concept proxies for recurring and relatively disentangled visual patterns in the model’s internal computations\. Within this framework, we decompose forgetting into apparent concept deletion, recoverability, and decodability\. We show that a large portion of seemingly lost concept\-level information can often be recovered under linearity assumption, with concept decodability degrading as more tasks are introduced\. Overall, our findings suggest that a significant part of concept\-level forgetting can be attributed to changes in the representational accessibility rather than complete information erasure\.
## 1Introduction
The degradation of previously learned knowledge during sequential training, known as catastrophic forgetting, remains a central challenge in continual learningWanget al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib20)\)\. Various forgetting mitigation strategies have been proposed, including rehearsal\-based methodsRolnicket al\.\([2019](https://arxiv.org/html/2605.16374#bib.bib11)\), regularization and distillationLi and Hoiem \([2017](https://arxiv.org/html/2605.16374#bib.bib9)\), as well as hybrid methodsBuzzegaet al\.\([2020](https://arxiv.org/html/2605.16374#bib.bib12)\)\. Despite this progress, evaluation remains largely performance\-driven, equating forgetting to decrease in task accuracy after learning new tasks, while leaving unresolved the fundamental questions of how forgetting manifests within modelsMasipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\)\. Recent works notice this gap and attempt to shift from the output\-level evaluation to representationsHuet al\.\([2025](https://arxiv.org/html/2605.16374#bib.bib10)\); Masipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\); Davariet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib1)\); Guet al\.\([2023](https://arxiv.org/html/2605.16374#bib.bib15)\)\. Previous studies measure representational drift using mostly similarity measuresKornblithet al\.\([2019](https://arxiv.org/html/2605.16374#bib.bib3)\); Huet al\.\([2025](https://arxiv.org/html/2605.16374#bib.bib10)\); Masipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\)\. Other worksDavariet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib1)\); Guet al\.\([2023](https://arxiv.org/html/2605.16374#bib.bib15)\)show that much apparent performance loss can be recovered via retrained classifiers or representation constraints\. However, these analyses remain global and tied to task\-level performance, and thus do not capture how fine\-grained information is organized, transformed, or lost\.
Figure 1:The taxonomy of concept transitions for taskttdata after taskt\+1t\+1training\.Similarly to how humans recognize objects by combining recurring concepts, recent progress in mechanistic interpretability enables the study of neural representations in terms of more disentangled and interpretable features used to perform learned tasksBereska and Gavves \([2024](https://arxiv.org/html/2605.16374#bib.bib13)\)\. It is particularly important because deep networks are considered to rely on superposition, where multiple concepts – understood as recurring patterns within the model computation – are encoded in overlapping neural directions, making feature\-level analysis difficultBereska and Gavves \([2024](https://arxiv.org/html/2605.16374#bib.bib13)\)\. Methods such as Sparse Autoencoders \(SAEs\)Hubenet al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib23)\)and transcodersDunefskyet al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib24)\)map representations into higher\-dimensional sparse latent spaces that can isolate more atomic and interpretable features\. Although these latent dimensions do not correspond to the ground\-truth semantic concepts, they often capture recurring and relatively disentangled patterns in model computations, serving as useful*concept proxies*that often align with abstract motifs such as furry animals or red objectsBereska and Gavves \([2024](https://arxiv.org/html/2605.16374#bib.bib13)\)\. In the following, we refer to them as*concepts*and leverage them as a structured basis for analyzing how fine\-grained information evolves during continual learning\.
Studying forgetting at a fine\-grained level should be as important as in terms of accuracy degradation, indicating whether the model retains knowledge and supports stable interpretations rather than shifts representations\. However, despite the potential of mechanistic interpretability, it has been scarcely applied to this problem\. The only workMasipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\)uses transcoders to analyze forgetting as geometric transformations of features in a shared latent space\. However, it focuses on*how*features transform, rather than analyzing task\-specific information accessibility\. In particular, it does not quantify concept\-level activation dynamics, the decodability of individual features, or the extent to which seemingly lost information can be recovered under standard readout assumptions\. As a result, the relationship between representational change and functional forgetting remains insufficiently understood\.
In this work, we address this gap by proposing a concept\-level framework for analyzing forgetting through the accessibility of task\-specific information\. Using Sparse Autoencoders \(SAEs\), we define a fixed, task\-anchored latent space in which individual features act as concept proxies, enabling fine\-grained tracking of how information evolves during continual training\. Within this framework, we decompose forgetting into concept deletion, recoverability, and decodability, distinguishing between information that is removed, misaligned, or still present but not directly accessible\. Our results show that much of the seemingly lost concept\-level information remains linearly recoverable, indicating that forgetting often reflects reduced accessibility to the underlying computational patterns rather than true erasure\. At the same time, some concepts become less linearly decodable over time, making loss of accessibility functionally equivalent to forgetting under standard linear readouts\. Moreover, continual strategies differ in how well they preserve the fine\-grained information\. Our findings provide a unified view of forgetting that links representational drift, readout limitations, and concept\-level information dynamics\. Our contributions can be summarized as follows:
- •We introduce a*concept\-level*framework for analyzing forgetting in continual learning, leveraging SAEs to define a task\-anchored latent space whose features act as concept proxies\. It enables studying task\-specific knowledge preservation and linear recoverability through concept activation dynamics and task\- and concept\-level decodability\.
- •We define a taxonomy of concept transitions after learning subsequent tasks in continual learning: retained, seemingly deleted, recovered, decodable, and lost concepts \(see Fig\.[1](https://arxiv.org/html/2605.16374#S1.F1)\)\.
- •We show that a substantial portion of task\-specific information remains linearly recoverable at the level of individual concepts, and that continual learning strategies differ in how well they preserve the fine\-grained knowledge and its recoverability\.
## 2Related work
Most work in continual learning quantifies forgetting through degradation in the model’s classifier performanceVerwimpet al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib18)\); Soutif\-Cormeraiset al\.\([2023](https://arxiv.org/html/2605.16374#bib.bib19)\); Kemkeret al\.\([2018](https://arxiv.org/html/2605.16374#bib.bib14)\); Kirkpatricket al\.\([2017](https://arxiv.org/html/2605.16374#bib.bib17)\)\. Most works focused on developing mitigation strategies, including replay\-based methods, regularization and distillation, architectural, and hybrid variantsWanget al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib20)\); Parisiet al\.\([2019](https://arxiv.org/html/2605.16374#bib.bib21)\); Faberet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib22)\), with forgetting typically evaluated through accuracy\-based metrics\. Although this line of work has established effective methods for reducing catastrophic forgetting, it offers limited insight into how forgetting manifests within internal representations\. More recent studies relate forgetting to representational drift and changes in feature geometryDavariet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib1)\); Guet al\.\([2023](https://arxiv.org/html/2605.16374#bib.bib15)\); Masipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\); Huet al\.\([2025](https://arxiv.org/html/2605.16374#bib.bib10)\)\. The authors ofDavariet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib1)\)shows that classifier\-level forgetting can overestimate degradation of the underlying representations\. In contrast, we move beyond task\-level probing on raw features by using an SAE\-defined, task\-anchored concept space to analyze whether more fine\-grained task\-relevant information remains preserved and linearly decodable at the concept level\. The authors ofGuet al\.\([2023](https://arxiv.org/html/2605.16374#bib.bib15)\)enforces linear separability across tasks through backward feature projection\. In contrast, we use linear mappings not as a training constraint, but as a diagnostic tool to measure how much past information remains linearly recoverable under standard continual learning strategies, and where this recoverability breaks down\. In terms of understanding forgetting, mechanistic interpretability offers a complementary perspective by decomposing neural representations into more atomic and interpretable components, e\.g\., by using Sparse Autoencoders \(SAEs\) and transcodersHubenet al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib23)\); Dunefskyet al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib24)\); Bereska and Gavves \([2024](https://arxiv.org/html/2605.16374#bib.bib13)\)\. Such methods construct overcomplete, sparse latent spaces that expose recurring disentangled patterns, offering a stronger basis for feature\-level analysis\. Yet, mechanistic interpretability remains rarely used to study forgetting in continual learning\. The closest work to ours isMasipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\), in which transcoders are used to map representations from different tasks into a shared space and describes forgetting in terms of geometric transformations such as rotations and scaling\. In contrast, our goal is not to characterize how features transform but to study and quantify whether task\-specific information remains accessible at the concept level\. To this end, we use task\-anchored SAE latent spaces and analyze concept activation dynamics, linear recoverability, and concept\-level decodability\. This allows us to distinguish between information that is preserved but misaligned, implicitly encoded, or no longer accessible under linear readouts\.
## 3Forgetting as Loss of Accessibility: A Concept\-Level Framework
In this section, we formulate our concept\-level analysis of forgetting as a loss of accessibility to fine\-grained task\-specific information rather than complete information loss\. The central question is whether information that no longer supports the original task readout after continual training has been erased, or it remains present in the representation but is misaligned, less active, or harder to decode\. Our framework is diagnostic, analyzing how past task information evolves within the model\.
#### Problem setup\.
Let𝒟t=\{\(xit,yit\)\}i=1Nt\\mathcal\{D\}^\{t\}=\\\{\(x\_\{i\}^\{t\},y\_\{i\}^\{t\}\)\\\}\_\{i=1\}^\{N\_\{t\}\}denote the dataset of tasktt\. Letfθtf\_\{\\theta\_\{t\}\}denote the continual model’s feature extractor after training on tasktt, andfθt\+sf\_\{\\theta\_\{t\+s\}\}the same extractor after further training on taskst\+1,…,t\+st\+1,\\dots,t\+s\. We study representations of taskttdata under these two models:
ht\(x\)=fθt\(x\),ht\+s\(x\)=fθt\+s\(x\),x∈𝒟t\.h\_\{t\}\(x\)=f\_\{\\theta\_\{t\}\}\(x\),\\quad h\_\{t\+s\}\(x\)=f\_\{\\theta\_\{t\+s\}\}\(x\),\\quad x\\in\\mathcal\{D\}^\{t\}\.\(1\)The goal is to analyze how the representation of taskttevolves after subsequent training and to what extent task\-relevant information remains accessible\.
#### Linear translation of representations\.
Continual training may introduce geometric shifts in the representation space without necessarily removing the information\. To study this behavior, we introduce a linear translation mapping:
Tt\+s→t\(h\)=Wt\+sh\+bt\+s,T\_\{t\+s\\rightarrow t\}\(h\)=W\_\{t\+s\}h\+b\_\{t\+s\},\(2\)trained to alignht\+s\(x\)h\_\{t\+s\}\(x\)withht\(x\)h\_\{t\}\(x\)using tasktttraining data:
minW,b𝔼x∼𝒟traint‖Wht\+s\(x\)\+b−ht\(x\)‖22\.\\min\_\{W,b\}\\ \\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}^\{t\}\_\{\\text\{train\}\}\}\\left\\\|Wh\_\{t\+s\}\(x\)\+b\-h\_\{t\}\(x\)\\right\\\|\_\{2\}^\{2\}\.\(3\)The learned mappingTt\+s→tT\_\{t\+s\\rightarrow t\}is then applied totttest data\. We use linear mappings to match standard readouts in continual learning, enabling recovery analysis under typical decision heads\. This choice is supported by prior work showing that forgetting can be mitigated by retraining linear headsDavariet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib1)\), that task representations may remain linearly relatedGuet al\.\([2023](https://arxiv.org/html/2605.16374#bib.bib15)\), and aligns with the linear representation hypothesisElhageet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib6)\); Bereska and Gavves \([2024](https://arxiv.org/html/2605.16374#bib.bib13)\); Masipet al\.\([2026](https://arxiv.org/html/2605.16374#bib.bib5)\)\. If task\-ttinformation can be recovered via a linear mapping, forgetting may reflect representational change rather than complete erasure\.
#### Concept proxies via sparse autoencoders\.
To analyze forgetting with finer granularity than task accuracy or global representation similarity, for each tasktt, we train an SAE on the train split ofht\(x\)h\_\{t\}\(x\), and use the trained SAE’s encoder to obtain different test representations:
zt\(x\)=SAEt\(ht\(x\)\),zt\+s\(x\)=SAEt\(ht\+s\(x\)\),zT\(x\)=SAEt\(Tt\+s→t\(ht\+s\(x\)\)\),z\_\{t\}\(x\)=\\text\{SAE\}\_\{t\}\(h\_\{t\}\(x\)\),\\quad z\_\{t\+s\}\(x\)=\\text\{SAE\}\_\{t\}\(h\_\{t\+s\}\(x\)\),\\quad z\_\{T\}\(x\)=\\text\{SAE\}\_\{t\}\(T\_\{t\+s\\rightarrow t\}\(h\_\{t\+s\}\(x\)\)\),\(4\)wherez\(x\)∈ℝKz\(x\)\\in\\mathbb\{R\}^\{K\}denotes latent activations\. The SAE trained on taskttdefines a fixed coordinate system – an anchor for concept\-level analysis to track how specific representational components evolve across training\. Individual task\-anchored latent features serve asconcept proxies\. We refer to them asconcepts: not ground\-truth semantic entities, but more disentangled, fine\-grained, recurring features in model computation \(see Appendix[A](https://arxiv.org/html/2605.16374#A1)for a detailed definition\), in contrast to the highly entangled features of standard continual models that hinder feature\-level analysis\.
#### Identifying active concepts through binarization of activations\.
Following recent work showing that binarization of SAE features can provide an effective alternative representationGallifantet al\.\([2025](https://arxiv.org/html/2605.16374#bib.bib2)\); Aswal and Hudelot \([2025](https://arxiv.org/html/2605.16374#bib.bib4)\), and to focus on salient and reusable features, we introduce a frequency\-based binarization rule to assess whether the concept isactive\. For each latent dimensionkk, we compute its mean activation over tasktttraining dataμk\\mu\_\{k\}and use it to define a mean\-based binarization ruleAswal and Hudelot \([2025](https://arxiv.org/html/2605.16374#bib.bib4)\):
ak\(x\)=𝕀\[zk\(x\)\>μk\]\.a\_\{k\}\(x\)=\\mathbb\{I\}\[z^\{k\}\(x\)\>\\mu\_\{k\}\]\.\(5\)Then, a conceptkkis consideredactivefor taskttand a frequency thresholdτ\\tauif
1\|𝒟traint\|∑xak\(x\)≥τ,\\frac\{1\}\{\|\\mathcal\{D\}^\{t\}\_\{\\text\{train\}\}\|\}\\sum\_\{x\}a\_\{k\}\(x\)\\geq\\tau,\(6\)as consistently and strongly activated, corresponding to a likely informative and reusable feature\. Therefore, we consider: raw featuresh\(x\)h\(x\), continuousz\(x\)z\(x\)and binarizeda\(x\)a\(x\)latent activations\.
#### Analysis modules\.
\(1\) Binary concept activation analysis\.We analyze the structure of the binary activation space by measuring: i\) the number of*active concepts*, ii\) the*deletion ratio*\(fraction of concepts active atttbut inactive att\+st\+s; by subtracting this value from 1, we obtain the ratio of retained concepts\), iii\)*regained concept count*after applyingTt\+s→tT\_\{t\+s\\rightarrow t\}and*regained activation mass*\(the non\-negative recovery of latent activation magnitude in deleted features provided by translationTTrelative tozt\+s\(x\)z\_\{t\+s\}\(x\), normalized by the total activation loss incurred betweenzt\(x\)z\_\{t\}\(x\)andzt\+s\(x\)z\_\{t\+s\}\(x\)\)\. These metrics quantify how concept\-level information is preserved, potentially lost, or recoverable\. \(2\) Concept prediction from raw features\.We assess concept\-level decodability by predicting binary concept activations from raw features\. For each conceptkk, we define a binary label:
yk\(x\)=𝕀\[ztk\(x\)\>0\],y\_\{k\}\(x\)=\\mathbb\{I\}\[z\_\{t\}^\{k\}\(x\)\>0\],\(7\)and train a classifier to predictyk\(x\)y\_\{k\}\(x\)fromht\+s\(x\)h\_\{t\+s\}\(x\)\. We use a less strict threshold than in the activity analysis, since the selected latent features have already been identified as salient concept proxies at the dataset level and to reduce sparse space class imbalance\. We evaluate performance using balanced accuracy and F1 score on thet\+st\+stest splits\. High values indicate that concept\-level information remains decodable from the representation\. Mean scores reflect overall concept preservation, while their distribution reflects that concepts can have different decodability levels\. \(3\) Task prediction via linear probes\.We train linear classifiers on taskttdata after tasktttraining to predict task labels using: \(i\) raw featuresht\(x\)h\_\{t\}\(x\), \(ii\) concept activationszt\(x\)z\_\{t\}\(x\)\. We evaluate these trained probes on testttdata attt,t\+st\+s, andTt\+s→tT\_\{t\+s\\rightarrow t\}using accuracy\. This measures how well task\-level information remains linearly decodable when using raw features and concept\-level features, and whether translation can fix some of the linear readout issues\.
#### The concept taxonomy\.
We distinguish five possible outcomes for a task\-ttconcept proxy after training on taskt\+st\+s:*Retained*, when the concept is active atttand remains active in the rawt\+st\+srepresentation;*Seemingly deleted*, when it was active atttbut is absent in rawt\+st\+sunder the binarization rule;*Recovered*, when it is absent in rawt\+st\+sbut reappears after translation;*Decodable*, when it does not reappear as an active concept, but its presence can be predicted from the representation \(F1\>0\\mathrm\{F1\}\>0\); and*Lost*, when it is neither recovered by translation nor decodable \(F1=0\\mathrm\{F1\}=0\)\.*Decodable*concepts do not reactivate but remain predictable \(non\-zero performance\), whereas*Lost*concepts are neither recoverable nor decodable\. Among decodable concepts, balanced accuracy and F1 quantify recoverability, with higher values indicating greater accessibility \(see Fig\.[1](https://arxiv.org/html/2605.16374#S1.F1)\)\.
## 4Experimental setup
Our experimental design aims to evaluate the accessibility\-based formulation of forgetting introduced in Section[3](https://arxiv.org/html/2605.16374#S3)\. Ourcontinual learning settingincludes evaluation of our framework on 3 task\-incremental benchmarks: i\)2seq\-CIFAR10, splitting CIFAR\-10Krizhevskyet al\.\([2009](https://arxiv.org/html/2605.16374#bib.bib7)\)into 2 tasks, 5 classes each; ii\)2seq\-tiny\-ImageNet, using the first 40 classes of tiny\-ImageNetDenget al\.\([2009](https://arxiv.org/html/2605.16374#bib.bib8)\), divided into 2 tasks with 20 classes each, enabling a coarse\-grained comparison with 2seq\-CIFAR10; iii\)10seq\-tiny\-ImageNet, using a split of tiny\-ImageNet into 10 tasks with 20 classes each\. We leverage four representative continual learning strategies: i\)SGD, corresponding to naive fine\-tuning without knowledge retention, serves as a baseline for natural continual\-learning behavior; ii\) Learning without Forgetting \(LWF\)Li and Hoiem \([2017](https://arxiv.org/html/2605.16374#bib.bib9)\), a distillation\-based strategy; iii\) Elastic Weight Consolidation \(EWC\), a regularization\-based strategyKirkpatricket al\.\([2017](https://arxiv.org/html/2605.16374#bib.bib17)\), and iv\) Dark Experience Replay \+\+ \(DER\+\+\)Buzzegaet al\.\([2020](https://arxiv.org/html/2605.16374#bib.bib12)\), a hybrid strategy combining experience replay with knowledge distillation and regularization\. As the backbone, we use Resnet18Heet al\.\([2016](https://arxiv.org/html/2605.16374#bib.bib25)\), one of the most popular models in continual image classificationBianet al\.\([2024](https://arxiv.org/html/2605.16374#bib.bib26)\); Masanaet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib27)\)\. To ensure reproducibility, we use implementations, data splits and hyperparameters fromMammoth111[https://github\.com/aimagelab/mammoth](https://github.com/aimagelab/mammoth)Boschiniet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib16)\)\. Appendix[B](https://arxiv.org/html/2605.16374#A2)reports Task\-Incremental accuracies: all strategies achieve relatively strong performance on new tasks, while exhibiting different forgetting levels, making them suitable for our analysis\.
In ourconcept\-level analysis setup, for each task, we train an SAE withBatchTopKSAEregime viaovercomplete222[https://github\.com/KempnerInstitute/overcomplete](https://github.com/KempnerInstitute/overcomplete)on the same Mammoth’s train/test splits\. We use Mean Squared Error \(MSE\) with anovercomplete’s dead\-neuron reactivation loss weighted by10−210^\{\-2\}\. Across experiments, manual inspection showed reconstructionR2\>0\.6R^\{2\}\>0\.6and close to 0% dead\-neuron rate, indicating meaningful and non\-degenerate SAEs\. We also examined top 9 activating images for selected SAE latents and found consistent shared visual motifs, indicating coherent patterns rather than arbitrary feature mixtures \(App\.[J](https://arxiv.org/html/2605.16374#A10)\)\. App\.[L](https://arxiv.org/html/2605.16374#A12)shows that active SAE latents can achieve substantially higher monosemanticity scores \(MS\)Pachet al\.\([2025](https://arxiv.org/html/2605.16374#bib.bib28)\)than randomized baselines, quantitatively supporting their quality and coherence as concept proxies\. We use a single linear layer for translation, and Logistic Regression for task\-level and concept\-level probes\. Though nonlinear mappings are less constrained and relevant under standard readouts, we additionally evaluate a simple nonlinear translator in App\.[I](https://arxiv.org/html/2605.16374#A9), which yields no significant gains\. We verified the analysis robustness across 10 runs \(App\.[D](https://arxiv.org/html/2605.16374#A4)\), 6 frequency\-based binarization thresholds \(App\.[E](https://arxiv.org/html/2605.16374#A5)\), 5 SAE batch sizes \(App\.[H](https://arxiv.org/html/2605.16374#A8)\), and 4 SAEKKvalues \(App\.[G](https://arxiv.org/html/2605.16374#A7)\)\. The results show consistent behavior of key metrics and support our conclusions’ robustness\. We validate that the selected active neurons retain most task\-relevant information through SAE\-based deletion experiments on non\-active neurons under different parameters and binarization thresholds \(App\.[E](https://arxiv.org/html/2605.16374#A5),[F](https://arxiv.org/html/2605.16374#A6),[G](https://arxiv.org/html/2605.16374#A7),[H](https://arxiv.org/html/2605.16374#A8)\)\. We used NVIDIA GH200 GPUs for continual training and Titan RTX GPU for SAE training and evaluation\. Average execution time for an example task and 10 runs was204\.199±42\.473204\.199\\pm 42\.473s on the Titan GPU\. We report hyperparameter values in App\.[K](https://arxiv.org/html/2605.16374#A11)\. We provide our code as Supplementary Material and will release it in a public GitHub repository upon publication\.
## 5Experimental analysis
Our experiments follow the*analysis modules*defined in Section[3](https://arxiv.org/html/2605.16374#S3): \(1\) Binary concept activation analysis, \(2\) Concept prediction from raw features and \(3\) Task prediction via linear probes\.
### 5\.1Concept retention, deletion and recoverability: concept activation analysis
Fig\.[2a](https://arxiv.org/html/2605.16374#S5.F2.sf1)shows the number of active neurons for 2seq\-CIFAR10 and 2seq\-tiny\-ImageNet\. Under the same binarization rule, substantially more latent neurons are active for 2seq\-tiny\-ImageNet, likely reflecting its higher complexity\. In all cases, only a small fraction of the SAE latent space is active, corresponding to salient, reusable features\. The number of active neurons decreases when task\-ttdata are encoded after training on taskt\+1t\+1for most setups, but a large fraction is recovered after linear translation\. An exception is 2seq\-CIFAR10 with DER\+\+, where the number of active concepts increases att\+1t\+1, possibly due to replay\-driven expansion of the active feature set\. Fig\.[2b](https://arxiv.org/html/2605.16374#S5.F2.sf2)reveals clear differences between continual strategies: deletion ratios are the highest for SGD and EWC, lower for DER\+\+ and LwF\. Linear translation noticeably improves deletion ratios for 2seq\-tiny\-ImageNet, especially for LwF, where deletion becomes almost zero\. Fig\.[2c](https://arxiv.org/html/2605.16374#S5.F2.sf3)shows that while the regained mass is similar across settings, the regained count ratio is the highest for LwF, followed by EWC\. It suggests that these strategies may preserve a more linearly recoverable fine\-grained space\. Overall, these results indicate that many apparently forgotten concepts are not irreversibly lost, but become less readable in the original task\-ttcoordinate system due to approximately linear representational drift\. As a qualitative complement, Appendix[J](https://arxiv.org/html/2605.16374#A10)compares the most activating images for exampleretainedconcepts atttandt\+1t\+1\(Figs\.[J\.1](https://arxiv.org/html/2605.16374#A10.F1),[J\.2](https://arxiv.org/html/2605.16374#A10.F2),[J\.3](https://arxiv.org/html/2605.16374#A10.F3)\)\. Shared visual motifs indicate overlapping information across the two representations\. We also showrecoveredconcepts \(Figs\.[J\.4](https://arxiv.org/html/2605.16374#A10.F4),[J\.5](https://arxiv.org/html/2605.16374#A10.F5),[J\.6](https://arxiv.org/html/2605.16374#A10.F6)\), whose common motifs further support that concept\-level information can be linearly restored\.
\(a\)Active neurons
\(b\)Deletion ratio
\(c\)Regained concept statistics
Figure 2:Concept activation analysis:Active concept count, \(seeming\)deletion ratioandregained concept statisticsfor 2seq\-CIFAR10 and 2seq\-tiny\-ImageNet\.Figure[3](https://arxiv.org/html/2605.16374#S5.F3)shows deletion ratios for each taskttafter all subsequent taskst\+st\+son 10seq\-tiny\-ImageNet\. For raw representations, deletion ratios increase withssunder all strategies, with consistently higher values for SGD and EWC\. Under LwF and DER\+\+, later tasks start from lower deletion levels \(att\+1t\+1\), suggesting improved resistance to degradation\. For DER\+\+, deletion increases progressively withss, although with a decelerating trend over time \(log\-like\)\. Across all tasks, linear alignment substantially reduces deletion ratios, indicating that much of the drift is partially reversible\. The strength of this reduction is weaker for DER\+\+ than for LwF\. The regained count ratios in Fig\.[4](https://arxiv.org/html/2605.16374#S5.F4)follow similar patterns as in the bar plots: they are the highest for LwF and EWC, comparable for SGD and DER\+\+\. Increases in deletion rates at higherssand slight decreases in regained counts \(SGD, DER\+\+, EWC\), suggest gradually reduced linear recoverability of concept activations\.
\(a\)SGD \- Raw features
\(b\)SGD \- Linearly translated features
\(c\)LwF \- Raw features
\(d\)LwF \- Linearly translated features
\(e\)EWC \- Raw features
\(f\)EWC \- Linearly translated features
\(g\)DER\+\+ \- Raw features
\(h\)DER\+\+ \- Linearly translated features
\(i\)Common legend
Figure 3:Deletion ratiofor all tasks of 10seq\-tiny\-ImageNet throughout the continual training\.\(a\)SGD
\(b\)LwF
\(c\)EWC
\(d\)DER\+\+
Figure 4:Regained concept count ratiofor all tasks of 10seq\-tiny\-ImageNet throughout the continual training after the linear translation\. We use the same legend as in Fig\.[3](https://arxiv.org/html/2605.16374#S5.F3)\.
### 5\.2Concept decodability: concept prediction from raw features
Fig\.[5a](https://arxiv.org/html/2605.16374#S5.F5.sf1)reports the mean balanced accuracy and F1 of linear classifiers trained to predict concepts that become inactive in raw task\-ttrepresentations after learning taskt\+1t\+1, with the corresponding score distributions shown in Figs\.[5c](https://arxiv.org/html/2605.16374#S5.F5.sf3)\(F1\) and[5b](https://arxiv.org/html/2605.16374#S5.F5.sf2)\(balanced accuracy\)\. The results show that LwF and EWC preserve substantially higher concept decodability att\+1t\+1than SGD and DER\+\+, suggesting that, beyond coarse\-grained class informationDavariet al\.\([2022](https://arxiv.org/html/2605.16374#bib.bib1)\), finer\-grained concept\-level information can also remain accessible, especially under forgetting mitigation with strategies differing in decodability preservation \(LwF, EWC being superior to DER\+\+\)\. At the same time, the spread of balanced accuracy and F1 across concepts indicates that the recoverability is non\-uniform: some concepts remain easily decodable, whereas others degrade strongly\. For LwF and EWC, the distributions are shifted toward higher values and are more compact, further indicating their broader and more consistent preservation of information at the level of concept proxies\. Overall, these results suggest that fine\-grained forgetting is often selective and partial rather than uniform or complete\.
\(a\)Mean balanced accuracy and F1 score
\(b\)Balanced accuracy distribution
\(c\)F1 score distribution
Figure 5:Meanbalanced accuracyandF1 scorealong with their distributions for the concept prediction probe on 2seq\-CIFAR10 and 2seq\-tiny\-ImageNet\.Fig\.[6](https://arxiv.org/html/2605.16374#S5.F6)shows the evolution of mean balanced accuracy and F1 of concept predictions throughout all future continual tasks for all tasks\. These plots also show significant differences: for SGD and DER\+\+ both balanced accuracy and F1 drop with subsequent training tasks\. For LwF it is almost constant \(only some small drops appear\) showing that this forgetting\-aware strategy preserves task information also at the more fine\-grained level of concepts\. As the ground truth labels are constant \(task t data at t determines them\), this is true decodability degradation caused by possible information loss or increasing readout issues at the level of individual concepts\. Our analysis did not identify fullylostneurons at the strictest possible criterion, i\.e\.,F1=0\\mathrm\{F1\}=0, suggesting that concept\-level information was at most partially lost rather than entirely erased in our setup\. As a qualitative complement, in Appendix[J](https://arxiv.org/html/2605.16374#A10)we compare the top activating images of selecteddecodableconcepts \(see Figs\.[J\.7](https://arxiv.org/html/2605.16374#A10.F7),[J\.8](https://arxiv.org/html/2605.16374#A10.F8)\)\. Concepts with varying decodability \(F1, balanced accuracy\) still share motifs acrossttandt\+1t\+1, supporting that at least some part of concept\-level information can remain decodable\.
\(a\)SGD \- Mean balanced accuracy
\(b\)SGD \- Mean F1 score
\(c\)LwF \- Mean balanced accuracy
\(d\)LwF \- Mean F1 score
\(e\)EWC \- Mean balanced accuracy
\(f\)EWC \- Mean F1 score
\(g\)DER\+\+ \- Mean balanced accuracy
\(h\)DER\+\+ \- Mean F1 score
Figure 6:Mean balanced accuracyandmean F1 scorefor theconcept prediction probeon 10seq\-tiny\-ImageNet for all tasks throughout continual training\. We use the same legend as in Fig\.[3](https://arxiv.org/html/2605.16374#S5.F3)\.
### 5\.3Task prediction via linear probes
Fig\.[7](https://arxiv.org/html/2605.16374#S5.F7)reports task 0 accuracy for 2seq\-CIFAR10 and 2seq\-tiny\-ImageNet, using probes trained on continual model features and SAE latents\. As expected, the largest drops att\+1t\+1are observed for SGD and EWC\. Linear translation restores most performance\. However, incomplete recovery for all strategies except LwF indicates that degradation cannot be explained solely by linear misalignment, pointing to residual readability limitations and partial loss of task information\. Importantly, the recovery of accuracy for latent\-based probes suggests that task\-relevant information remains largely compatible with the original task\-anchored concept basis after alignment, and thus provides functional evidence that the translation restores access to fine\-grained information at the level of the analyzed concept proxies\. This further indicates that a large portion of fine\-grained information is not fully deleted due to forgetting, but becomes misaligned and less directly accessible\. The remaining performance gap after alignment implies, however, partial degradation of concept\-level structure, consistent with the observed variability in F1 scores for concept prediction\.
To further understand the temporal dynamics of probe performance, we provide an extended analysis across all tasksttaftert\+st\+sof 10seq\-tiny\-ImageNet in Appendix[C](https://arxiv.org/html/2605.16374#A3)\(Fig\.[C\.1](https://arxiv.org/html/2605.16374#A3.F1)– continual model features\-based probes, Fig\.[C\.2](https://arxiv.org/html/2605.16374#A3.F2)– concept\-based probes\)\. Accuracy att\+1t\+1without translation declines asssincreases for all strategies, with the strongest effect on SGD and EWC\. Operating on SAE latents seems to stabilize the probe accuracies on non\-translated data for LwF and DER\+\+\. Translation recovers much of the performance at allss, thus information recoverability persists over time\. However, it slightly decreases asssgrows for some strategies \(SGD, DER\+\+\), further suggesting that stronger forgetting may involve more advanced drifts or partial information erasure\.
\(a\)Representation\-level
\(b\)Concept\-level
Figure 7:Accuracy of probespredicting 2seq\-CIFAR10 and 2seq\-tiny\-ImageNet task 0 classes from the continual model features \(representation\-\) and SAE latent activations \(concept\-level\)\.
## 6Conclusion
In this paper, we proposed an SAE\-based framework that defines a fixed, more disentangled concept space to analyze forgetting in continual learning through task\-specific knowledge preservation and recoverability under a linear readout assumption\. Our results show that forgetting mitigation strategies affect not only the magnitude but also the nature of forgetting\. Methods such as LwF keep representational drift largely linear, making class\- and concept\-level information more decodable and reversible\. In contrast, other strategies induce less recoverable distortions\. Overall, in the studied settings, much apparent forgetting reflects approximately linear representational changes, with seemingly deleted information often recoverable through linear remapping\. Importantly, our results suggest that knowledge preservation extends beyond coarse\-grained class information to a more fine\-grained level\. At the same time, our results indicate that some part of the concept\-level information becomes no longer fully accessible in a linear setting\. Under the common assumption of a linear readout as a classifier, this leads to a practical loss of decodability, which can be functionally equivalent to forgetting\. Thanks to the observation of the distributions of concept\-prediction scores we find that concept\-level forgetting is not uniform\. At the same time, our results suggest that continual learning strategies preserve past knowledge at different levels: some retain their functionality more at the level of outputs \(DER\+\+\), whereas others better preserve it at the level of the internal organization of concept proxies \(LwF, EWC\)\. Overall, our findings suggest that, in supervised continual learning, past fine\-grained structure of model knowledge and computations are often not lost but hidden: they may remain linearly recoverable, although their preservation depends on the continual learning strategy\. This emphasizes the importance of preserving such structures for interpretable continual learning\. As improving transparency and trustworthiness of continual models is a positive societal impact, future work will extend our framework to broader settings and problems, as well as design strategies that better preserve fine\-grained task\-relevant information\.
#### Limitations\.
One may view the use of SAE latent features in a concept\-level analysis as a limitation, since they are concept proxies rather than ground\-truth semantic entities\. However, they provide an automatic, label\-free, and computationally feasible approximation of more disentangled, fine\-grained structure in the continual model’s representation space, supported by strong task\-level concept\-based probe performance, quantitative quality assessment in App\.[L](https://arxiv.org/html/2605.16374#A12), and qualitative examples in App\.[J](https://arxiv.org/html/2605.16374#A10)showing clear shared motifs\. A limitation of our taxonomy is that the boundary for lost concepts is not absolute: very lowF1\\mathrm\{F1\}may indicate only marginal recoverability\. For clarity, we useF1=0\\mathrm\{F1\}=0to define*Lost*, which represents a clear non\-parametric case of zero decodability\. However, we emphasize in the paper that residual concept degradation can still be functionally equivalent to partial concept loss under a linear readout\. Finally, a potential negative societal impact is that concept\-level analyses may increase confidence in deploying continual models in high\-stakes settings, even though the identified concepts are proxies and may miss sensitive or safety\-critical information\.
## References
- ConceptGuard: Neuro\-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Concepts\.External Links:2508\.16325Cited by:[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px4.p1.3)\.
- L\. Bereska and E\. Gavves \(2024\)Mechanistic Interpretability for AI Safety – A Review\.arXiv preprint arXiv:2404\.14082\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p2.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1),[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px2.p1.6)\.
- A\. Bian, W\. Li, H\. Yuan, C\. Yu, M\. Wang, Z\. Zhao, A\. Lu, P\. Ji, and T\. Feng \(2024\)Make Continual Learning Stronger via C\-Flat\.Advances in Neural Information Processing Systems37,pp\. 7608–7630\.Cited by:[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- M\. Boschini, L\. Bonicelli, P\. Buzzega, A\. Porrello, and S\. Calderara \(2022\)Class\-Incremental Continual Learning into the eXtended DER\-verse\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- P\. Buzzega, M\. Boschini, A\. Porrello, D\. Abati, and S\. Calderara \(2020\)Dark Experience for General Continual Learning: a Strong, Simple Baseline\.Advances in Neural Information Processing Systems33,pp\. 15920–15930\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- M\. Davari, N\. Asadi, S\. Mudur, R\. Aljundi, and E\. Belilovsky \(2022\)Probing Representation Forgetting in Supervised and Unsupervised Continual Learning\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16712–16721\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1),[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px2.p1.6),[§5\.2](https://arxiv.org/html/2605.16374#S5.SS2.p1.3)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei \(2009\)ImageNet: A Large\-Scale Hierarchical Image Database\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 248–255\.Cited by:[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- J\. Dunefsky, P\. Chlenski, and N\. Nanda \(2024\)Transcoders Find Interpretable LLM Feature Circuits\.Advances in Neural Information Processing Systems37,pp\. 24375–24410\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p2.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen,et al\.\(2022\)Toy models of superposition\.arXiv preprint arXiv:2209\.10652\.Cited by:[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px2.p1.6)\.
- K\. Faber, C\. Kanan, V\. Lomonaco, and R\. Corizzo \(2026\)Continual anomaly detection: a comprehensive survey and research roadmap\.Preprints\.External Links:[Document](https://dx.doi.org/10.20944/preprints202601.1931.v1),[Link](https://doi.org/10.20944/preprints202601.1931.v1)Cited by:[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- J\. Gallifant, S\. Chen, K\. Sasse, H\. Aerts, T\. Hartvigsen, and D\. Bitterman \(2025\)Sparse Autoencoder Features for Classifications and Transferability\.InConference on Empirical Methods in Natural Language Processing,pp\. 29927–29951\.Cited by:[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px4.p1.3)\.
- Q\. Gu, D\. Shim, and F\. Shkurti \(2023\)Preserving linear separability in continual learning by backward feature projection\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 24286–24295\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1),[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px2.p1.6)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep Residual Learning for Image Recognition\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- J\. Hu, Z\. Lian, Z\. Wen, C\. Li, G\. Chen, X\. Wen, B\. Xiao, and M\. Tan \(2025\)Continual Knowledge Adaptation for Reinforcement Learning\.arXiv preprint arXiv:2510\.19314\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- R\. Huben, H\. Cunningham, L\. R\. Smith, A\. Ewart, and L\. Sharkey \(2024\)Sparse Autoencoders Find Highly Interpretable Features in Language Models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p2.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- R\. Kemker, M\. McClure, A\. Abitino, T\. Hayes, and C\. Kanan \(2018\)Measuring catastrophic forgetting in neural networks\.InAAAI Conference on Artificial Intelligence,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§2](https://arxiv.org/html/2605.16374#S2.p1.1),[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton \(2019\)Similarity of Neural Network Representations Revisited\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1)\.
- A\. Krizhevsky, G\. Hinton,et al\.\(2009\)Learning Multiple Layers of Features from Tiny Images\.Cited by:[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- Z\. Li and D\. Hoiem \(2017\)Learning without Forgetting\.IEEE Transactions on Pattern Analysis and Machine Intelligence40\(12\),pp\. 2935–2947\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- M\. Masana, X\. Liu, B\. Twardowski, M\. Menta, A\. D\. Bagdanov, and J\. Van De Weijer \(2022\)Class\-incremental learning: survey and performance evaluation on image classification\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(5\),pp\. 5513–5533\.Cited by:[§4](https://arxiv.org/html/2605.16374#S4.p1.1)\.
- S\. Masip, G\. M\. van de Ven, J\. Ferrando, and T\. Tuytelaars \(2026\)Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability\.arXiv preprint arXiv:2601\.22012\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§1](https://arxiv.org/html/2605.16374#S1.p3.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1),[§3](https://arxiv.org/html/2605.16374#S3.SS0.SSS0.Px2.p1.6)\.
- M\. Pach, S\. Karthik, Q\. Bouniot, S\. Belongie, and Z\. Akata \(2025\)Sparse Autoencoders Learn Monosemantic Features in Vision\-Language Models\.Advances in Neural Information Processing Systems\.Cited by:[Appendix L](https://arxiv.org/html/2605.16374#A12.p1.1),[Appendix L](https://arxiv.org/html/2605.16374#A12.p2.1),[§4](https://arxiv.org/html/2605.16374#S4.p2.4)\.
- G\. I\. Parisi, R\. Kemker, J\. L\. Part, C\. Kanan, and S\. Wermter \(2019\)Continual Lifelong Learning with Neural Networks: A Review\.Neural Networks113,pp\. 54–71\.Cited by:[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. Wayne \(2019\)Experience Replay for Continual Learning\.Advances in Neural Information Processing Systems32\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1)\.
- A\. Soutif\-Cormerais, A\. Carta, A\. Cossu, J\. Hurtado, V\. Lomonaco, J\. Van de Weijer, and H\. Hemati \(2023\)A Comprehensive Empirical Evaluation on Online Continual Learning\.InIEEE/CVF International Conference on Computer Vision,pp\. 3518–3528\.Cited by:[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- E\. Verwimp, R\. Aljundi, S\. Ben\-David, M\. Bethge, A\. Cossu, A\. Gepperth,et al\.\(2024\)Continual Learning: Applications and the Road Forward\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024\)A Comprehensive Survey of Continual Learning: Theory, Method and Application\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(8\),pp\. 5362–5383\.Cited by:[§1](https://arxiv.org/html/2605.16374#S1.p1.1),[§2](https://arxiv.org/html/2605.16374#S2.p1.1)\.
## Appendix AConcept definition
In this work, we understood aconceptas a computational representation of a shared, recurring pattern that helps a model structure information\. Methods from mechanistic interpretability such as Sparse Autoencoders \(SAEs\) can extract such representations via self\-supervised learning, often providing more coherent patterns than standard continual models\. Analogous to human perception, in which recognizing an object \(e\.g\. a dog\) relies on features like ears, a nose, or paws, model concepts capture recurring patterns in data\. While they do not fully align with human semantic concepts and remain approximate \(see Fig\.[A\.1](https://arxiv.org/html/2605.16374#A1.F1)\), potentially less coherent representations, they provide a practical and well\-established way to model abstract, fine\-grained information, serving as usefulconcept proxies, to which we refer to asconcepts\. Studying whether these concepts persist in continual learning is equally as important as accuracy, as it reveals whether the model truly retains knowledge and preserves a consistent way of organizing information and computations, rather than merely sustaining performance through shifting internal representations\. These all can be also connected to whether the model maintains stable and trustworthy interpretations, which is crucial for explainability and trustworthiness in artificial intelligence\.
Figure A\.1:SAE latents provide cleaner, more concept\-aligned representations than raw model features, and are therefore used here as proxies for concepts\. i\) Feature from a CL model corresponds to a single neuron in the representation layer in CL model; ii\) SAE allows us to build a concept proxy, disentangling features; iii\) ground truth semantic concept in real\-world
## Appendix BAccuracy of the original continual models’ classification heads
Table[B\.1](https://arxiv.org/html/2605.16374#A2.T1)presents the accuracy results for both tasks for the 2seq\-CIFAR10 case\. After learning the second task, performance on the first task drops significantly for SGD, and notably less for DER\+\+ and LwF, while accuracy on the new task remains similarly high for SGD, LwF and DER\+\+ \(96\.4 for SGD, 96\.5 for DER\+\+, 95\.3 for LwF\), and slightly lower for EWC \(89\)\. This suggests that on CIFAR10, all methods learn the new task well, but forgetting mitigation mainly affects retention of the old one, with LwF preserving past\-task accuracy best, DER\+\+ offering an intermediate preservation, and SGD, EWC forgetting the most\.
Table B\.1:Task\-IL accuracy \(%\) on 2seq\-CIFAR10\. For task 0, we report the accuracy after learning task 0\. For task 1, we report the accuracies on task 0 and task 1 after learning task 1\.We present the accuracy matrices for all tasks under the Task\-IL setting for 10seq\-tiny\-ImageNet in Fig\.[B\.1](https://arxiv.org/html/2605.16374#A2.F1)\. The four examined strategies resulted in expected behaviors\. SGD achieves reasonable accuracy on the currently learned task, but older\-task performance drops sharply as training proceeds – as an indication of strong forgetting\. DER\+\+ provides the best overall Task\-IL accuracy, with noticeably better retention of past tasks than SGD while still maintaining strong performance on newer tasks\. In contrast, LwF is more uniform across tasks, but at a lower accuracy level, suggesting better stability at the cost of plasticity and overall per\-task performance\.
\(a\)SGD
\(b\)DER\+\+
\(c\)LwF
\(d\)EWC
Figure B\.1:Accuracy of the original continual models’ classification heads in the task incremental learning setup on 10seq\-tiny\-ImageNet\.
## Appendix CEvolution of task\-level linear probes performance when trained on raw features and latent features
Figures[C\.1](https://arxiv.org/html/2605.16374#A3.F1)and[C\.2](https://arxiv.org/html/2605.16374#A3.F2)show the temporal dynamics of feature\- and concept\-based probe accuracy for tasktt, measured on task\-ttdata after subsequent taskst\+st\+s\. In all cases, accuracy on raw task\-ttrepresentations declines asssincreases, but with different dynamics: under SGD and EWC it drops quickly and stabilizes only slightly above 0, whereas under DER\+\+ and LwF it remains much higher\. DER\+\+ preserves task performance in non\-translated data better than LwF, likely due to combining replay and distillation\. For LwF, accuracy after linear translation stays nearly constant over time, indicating that the apparent forgetting is largely reversible by a simple linear remapping\. For SGD, EWC and DER\+\+, translation recovers much of the lost performance, but less so asssgrows, further suggesting that stronger forgetting may involve more advanced drifts or partial information erasure\. Similar trends both at the concept level and the level of continual model entangled features, show that these effects persists even in sparse, more disentangled representations\. Additionally, operating at the concept level appears to slow down the decline for DER\+\+ and LwF relative to the raw\-feature probe\. Importantly, the recovery of accuracy for probes operating on SAE latents suggests that task\-relevant information remains largely compatible with the original task\-anchored concept basis after alignment\. This indicates that a significant portion of fine\-grained, concept\-level information is not fully deleted during continual learning, but instead becomes misaligned and less directly accessible\. Moreover, this recovery provides functional evidence that the linear translation of the continual model features restores access to fine\-grained concept\-based information\.
\(a\)SGD \- Raw features
\(b\)SGD \- Linear translation
\(c\)LwF \- Raw features
\(d\)LwF \- Linear translation
\(e\)EWC \- Raw features
\(f\)EWC \- Linear translation
\(g\)DER\+\+ \- Raw features
\(h\)DER\+\+ \- Linear translation
Figure C\.1:Accuracy of probesfor 10seq\-tiny\-ImageNet classes prediction throughout continual training based on theraw representationsfrom the continual model \(representation\-level\)\. Accuracy is higher in all cases after applying the linear translation to the raw features\. We use the same legend as in Fig\.[3](https://arxiv.org/html/2605.16374#S5.F3)\.\(a\)SGD \- Raw features
\(b\)SGD \- Linear translation
\(c\)LwF \- Raw features
\(d\)LwF \- Linear translation
\(e\)EWC \- Raw features
\(f\)EWC \- Linear translation
\(g\)DER\+\+ \- Raw features
\(h\)DER\+\+ \- Linear translation
Figure C\.2:Accuracy of probesfor 10seq\-tiny\-ImageNet classes prediction throughout continual training based on theSAE latent neuron activations\(concept\-level\)\. Accuracy is higher in all cases after applying the linear translation to the raw features\. We use the same legend as in Fig\.[3](https://arxiv.org/html/2605.16374#S5.F3)\.
## Appendix DStability of concept\-based forgetting analysis across runs
Box plots of key metric values across 10 runs for an example configuration \(2seq\-tiny\-ImageNet, LwF\) presented in Fig\.[D\.1](https://arxiv.org/html/2605.16374#A4.F1)show that the results are largely stable across runs\. The probe\-based measures, especially the representation\- and concept\-level probe accuracies and the mean balanced accuracy and F1 for concept prediction, show the most compact distributions, indicating high consistency of the main conclusions\. Larger variability is observed for the regained count ratio, although its main mass remains concentrated in a relatively narrow range\. This suggests that the exact number of regained concepts may vary between runs, but the overall conclusion that linear alignment restores a substantial fraction of the concept information remains robust\.
\(a\)Active neurons
\(b\)Deletion ratio
\(c\)Regained count and mass ratio
\(d\)Representation\-based probe accuracy
\(e\)Concept\-based probe accuracy
\(f\)Mean balanced accuracy and F1
Figure D\.1:Stability analysis: box plots of the distribution of values of our key measures for the 2seq\-tiny\-ImageNet on the example LwF continual learning strategy case for 10 individual runs\. For all measures bounded to the\[0,1\]\[0,1\]range, the y\-axes were rescaled to dataset\-specific intervals of width0\.50\.5in order to facilitate visual comparison of the spread across plots
## Appendix EImpact of frequency\-based binarization threshold on concept\-based forgetting analysis
With the frequency\-based binarization criterion, a latent is considered active if it exceeds its task\-tttrain\-set mean on at least a specified fraction of task\-ttsamples\. As shown in Fig\.[E\.1](https://arxiv.org/html/2605.16374#A5.F1)for the example LwF 2seq\-tiny\-ImageNet case, increasing this threshold reduces the number of detected active neurons, as could be predicted, but the overall forgetting picture remains largely unchanged\. Across all thresholds, rawt\+1t\+1representations ofttexhibit fewer active concepts and non\-zero deletion, while linear alignment restores most of the lost activity and keeps both feature\-level and concept\-level probe accuracy close to the task\-ttreference\. This indicates that our concept\-forgetting observations are not an artifact of a particular binarization threshold, but are qualitatively robust across a range of frequency\-based thresholds\.
\(a\)Active neurons
\(b\)Deletion ratio
\(c\)Regained count and mass ratio
\(d\)Representation\-based probe accuracy
\(e\)Concept\-based probe accuracy
\(f\)Mean balanced accuracy and F1
Figure E\.1:Theimpact of frequency\-based binarization thresholdon metrics for 2seq\-tiny\-ImageNet task 0 under LwF\.To verify that the set of binary active neurons is functionally relevant, we additionally measure task\-0 classification accuracy after zeroing out, through the SAE decoder, all latent neurons that are not marked as active under the given frequency\-based rule\. Figure[E\.2b](https://arxiv.org/html/2605.16374#A5.F2.sf2)shows that this filtering leads only to a limited accuracy drop across the examined thresholds, confirming that the selected active neurons retain most of the task\-level information\. For 2seq\-tiny\-ImageNet under LwF, the performance remains relatively stable across thresholds, although slightly larger drops appear for thresholds below 0\.05\. This suggests that very permissive thresholds include less functionally relevant activations, while stricter thresholds can entail some loss of the task\-relevant information\. Therefore, thresholds around 0\.05 provide a reasonable trade\-off between retaining task information and enforcing a meaningful notion of concept activity\. We therefore use 0\.05 in all setups, including CIFAR10, to keep the analysis unified across datasets\.
\(a\)Balanced accuracy
\(b\)F1 score
Figure E\.2:Theimpact of frequency\-based binarization thresholdon accuracy of probes built for prediction of 2seq\-tiny\-ImageNet task 0 classes under LwF based on the raw representations from the continual model \(representation\-level\) and based on the SAE latent neuron activations \(concept\-level\)\. All accuracy values are computed for the task 0 data after training on task 0, but for the "Raw before filtered" case, we use our SAE decoder to zero out the neurons that are not active binary under our binarization rule\. We examine frequency\-based threshold in\[0\.00625,0\.0125,0\.025,0\.05,0\.1,0\.2\]\[0\.00625,0\.0125,0\.025,0\.05,0\.1,0\.2\]\.
## Appendix FImpact of0\.050\.05frequency\-based binarization on probe\-based task accuracy
As an additional check, to examine whether the active set of neurons under the frequency\-based binarization rule used in our main experiments retains the functionally relevant task information for both datasets, we perform an experiment, in which we zero\-out the neurons that are overall not active under the binarization rule \(i\.e\. due to values lower than mean train activations or due to the frequency\)\. We use the SAE’s decoder for this intervention by first zeroing out the activations of the selected neurons, decoding the modified latent vector, and comparing it with the decoder output obtained from the raw latent activations\. We then use this difference to update the original input representation, rather than replacing it directly with the decoded vector, in order to avoid propagating the SAE reconstruction error\.
Fig\.[F\.1](https://arxiv.org/html/2605.16374#A6.F1)shows that using the frequency\-based threshold of 0\.05 preserves most of the probe accuracy across all configurations, both at the representation and concept level\. This indicates that the set of neurons selected as active retains the majority of task\-relevant information despite being only a subset of the full latent space\. Overall, these results support the reliability of the adopted binarization rule as the basis for our concept\-level analysis across all examined settings\.
\(a\)Representation\-level
\(b\)Concept\-level
Figure F\.1:Accuracy of probes built for prediction of 2seq\-CIFAR10 and 2seq\-tiny\-ImageNet task 0 classes based on the raw representations from the continual model \(representation\-level\) and based on the SAE latent neuron activations \(concept\-level\)\. Both accuracy values are computed for the task 0 data after training on task 0, but for theF\(t\+1\)F\(t\+1\)case, we use our SAE decoder to zero out the neurons that are not active binary under our binarization rule \(e\.g\. the frequency of their values that are higher than the activation threshold is lower than the frequency threshold\)\.
## Appendix GImpact of SAE’sKKon concept\-based forgetting analysis
Fig\.[G\.1](https://arxiv.org/html/2605.16374#A7.F1)shows the impact of the sparsity\-control parameterKKused for the BatchTopK SAE on the key metrics for the example 2seq\-tiny\-ImageNet \+ LwF setup\. Overall, the key results are qualitatively stable across the testedKKvalues, indicating that our conclusions are not sensitive to this hyperparameter\. The main difference lies in the number of active neurons, which is expected since parameterKKdirectly controls the sparsity of the activations\. Although some other small quantitative differences in other metrics can be observed, the overall patterns remain unchanged: raw task\-t\+1t\+1representations show concept deletion, linear alignment recovers a large fraction of the lost information, and both task\- and concept\-level probe performance remain close to the reference values\. Importantly, most values fall within the spread previously observed across repeated runs of the same method, suggesting that the effect of batch size is comparable to the run\-to\-run variability\.
\(a\)Active neurons
\(b\)Deletion ratio
\(c\)Regained count and mass ratio
\(d\)Representation\-based probe accuracy
\(e\)Concept\-based probe accuracy
\(f\)Mean balanced accuracy and F1
Figure G\.1:Theimpact of SAE’s Kon metrics for 2seq\-tiny\-ImageNet task 0 under LwF\.To verify that the set of binary active neurons is functionally relevant under differentKKvalues, we additionally measure task\-0 classification accuracy after zeroing out, through the SAE decoder, all latent neurons that are not marked as active under the given frequency\-based rule\. Figure[G\.2](https://arxiv.org/html/2605.16374#A7.F2)shows that the approach is robust to the chosenKKfor BatchTopK SAE regime\.
\(a\)Balanced accuracy
\(b\)F1 score
Figure G\.2:Theimpact of SAE’s Kon accuracy of probes built for prediction of 2seq\-tiny\-ImageNet task 0 classes under LwF based on the raw representations from the continual model \(representation\-level\) and based on the SAE latent neuron activations \(concept\-level\)\. All accuracy values are computed for the task 0 data after training on task 0, but for the "Raw before filtered" case, we use our SAE decoder to zero out the neurons that are not active binary under our binarization rule\.
## Appendix HImpact of SAE batch size on concept\-based forgetting analysis
Fig\.[H\.1](https://arxiv.org/html/2605.16374#A8.F1)shows the impact of the SAE training batch size on the key metrics for the example 2seq\-tiny\-ImageNet \+ LwF setup\. Overall, the results are qualitatively stable across the tested batch sizes, indicating that our conclusions are not sensitive to this hyperparameter\. Although small quantitative differences appear, the overall patterns remain unchanged: raw task\-t\+1t\+1representations show concept deletion, linear alignment recovers a large fraction of the lost information, and both task\- and concept\-level probe performance stay close to the reference values\. Similarly to the results forKK, most values remain within the spread observed across repeated runs, suggesting that the effect of batch size is comparable to the run\-to\-run variability\.
\(a\)Active neurons
\(b\)Deletion ratio
\(c\)Regained count and mass ratio
\(d\)Representation\-based probe accuracy
\(e\)Concept\-based probe accuracy
\(f\)Mean balanced accuracy and F1
Figure H\.1:Theimpact of SAE batch sizeon metrics for 2seq\-tiny\-ImageNet task 0 under LwF\.To verify that the set of binary active neurons is functionally relevant under different SAE batch sizes, we additionally measure task\-0 classification accuracy after zeroing out, through the SAE decoder, all latent neurons that are not marked as active under the given frequency\-based rule\. Figure[H\.2](https://arxiv.org/html/2605.16374#A8.F2)shows that the approach is robust to the chosen batch size for SAE training\.
\(a\)Balanced accuracy
\(b\)F1 score
Figure H\.2:Theimpact of SAE batch sizeon accuracy of probes built for prediction of 2seq\-tiny\-ImageNet task 0 classes under LwF based on the raw representations from the continual model \(representation\-level\) and based on the SAE latent neuron activations \(concept\-level\)\. All accuracy values are computed for the task 0 data after training on task 0, but for the "Raw before filtered" case, we use our SAE decoder to zero out the neurons that are not active binary under our binarization rule\.
## Appendix IImpact of using a non\-linear translation model on concept decodability and accessibility
#### Nonlinear recoverability setup\.
To test whether the information could be better restored by a low\-capacity nonlinear mapping instead of a linear mapping, we additionally considered a nonlinear translation model\. It was implemented as atorchmodel with three fully connected layers and GELU nonlinearities after the first two layers\. The hidden dimensionality was set equal to the input dimensionality\. The model was trained, analogously to the linear translator, to predict the task\-ttrepresentations from the task\-\(t\+s\)\(t\+s\)representations by minimizing mean squared error\. We trained it for100100epochs using AdamW, with learning rate10−310^\{\-3\}, weight decay10−410^\{\-4\}, batch size128128, and a validation split of0\.20\.2\.
Fig\.[I\.1](https://arxiv.org/html/2605.16374#A9.F1)shows that the nonlinear translation model does not provide substantial improvements over the linear translation model\. In most cases, the nonlinear model yields slightly worse performance across all considered metrics\. Only two exceptions can be observed for deletion/regained count metrics, namely for 2seq\-CIFAR10 with DER\+\+ and 2seq\-tiny\-ImageNet with SGD, where the nonlinear model leads to minor improvements\. However, these gains are marginal and correspond to restoring only a very small number of additional concepts \(1\-3 concepts\), without meaningful impact on the overall metrics\. Overall, both models achieve highly comparable performance, with the linear translation often performing slightly better\. The overall results, therefore, indicate that the recoverable component of the representational drift is largely linear, both at the task level and at the level of individual concepts, and that simple nonlinear mappings do not provide additional explanatory power in this setting\. While it is possible that more expressive nonlinear models could recover additional information, identifying such models and their configurations would require exploring a largely unconstrained architectural and hyperparameter space, which is beyond the scope of this work\.
\(a\)Active neurons
\(b\)Deletion ratio
\(c\)Regained count and mass ratio
\(d\)Representation\-based probe accuracy
\(e\)Concept\-based probe accuracy
Figure I\.1:Theimpact of using a non\-linear translation modelon metrics for 2seq\-tiny\-ImageNet compared to using linear translation\. It is visible that the utilized model makes the recoverability and decodability worse in most cases, while in two improved cases, the improvement is marginal\.
## Appendix JQualitative analysis of concepts
In this section, we provide a qualitative examination of SAE neurons across datasets and continual learning strategies\. The goal of this analysis is to visually assess whether SAE latents after tasksttandt\+1t\+1correspond to reasonably coherent and interpretable patterns\. For data from tasktt, the visual coherence of the top\-activating examples suggests that SAE latent neurons capture recurring patterns in the representation space and suggests that it may therefore serve as a useful proxy for concept detectors\. Comparing taskttexamples after training on taskt\+1t\+1with those obtained immediately after taskttprovides a qualitative indication of concept preservation\. Visual similarity suggests that the concept remains at least partially encoded\. Substantial differences, visible even in the most prototypical and central part of the neuron activation space, point either to information loss or to representational changes that prevent faithful recovery of the original concept\. These results further support analysis of cases for which concept predictionF1\>0F1\>0and can be used as a visual sanity check\. In particular, we study the visual quality of neurons identified for taskttafterttandt\+1t\+1training, by inspecting their top\-99activating images, ordered by activation rank from11to99\. To make this comparison meaningful in the presence of representation drift, the maximally activating examples att\+1t\+1are determined using the linearly translated feature vectors\. This allows us to examine how the center of the concept changes after subsequent training, and whether the same visual motif can still be recovered despite the apparent shift in representation space\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.1:The exampleretainedconcept, which stayed active in raw vector aftert\+1t\+1\. Top\-ranked examples for latent neuron 274 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-tiny\-ImageNet with DER\+\+\. The pictures show aclear motif: brown, furry objects, and the motif is shared between representations atttandt\+1t\+1\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.2:The exampleretainedconcept, which stayed active in raw vector aftert\+1t\+1\. Top\-ranked examples for latent neuron 370 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-tiny\-ImageNet with DER\+\+\. The pictures show aclear motif: penguins, and the motif is shared between representations atttandt\+1t\+1\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.3:The exampleretainedconcept, which stayed active in raw vector aftert\+1t\+1\. Top\-ranked examples for latent neuron 425 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-tiny\-ImageNet with DER\+\+\. The pictures show aclear motif: transport vehicles, and the motif is shared between representations atttandt\+1t\+1\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.4:The example of alower quality concept\. The concept can be described asseemingly deleted, as it was absent in rawt\+1t\+1andrecovered, because it reappeared in translatedt\+1t\+1\. Top\-ranked examples for latent neuron 687 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-tiny\-ImageNet with LwF\. The concept obtained a relativelyhigh decodability level: mean bacc was0\.970\.97and F1 was0\.80\.8for concept decodability\. It had7676images with\>0\>0activations atttand7777at translatedt\+1t\+1\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.5:The example of alower quality concept\. The concept can be described asseemingly deleted, as it was absent in rawt\+1t\+1andrecovered, because it reappeared in translatedt\+1t\+1\. Top\-ranked examples for latent neuron 358 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-tiny\-ImageNet with DER\+\+\. The concept obtained amedium decodability level: F1 \(0\.60\.6\) at mean bacc of0\.970\.97for concept decodability\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.6:The example of alower quality concept\. The concept can be described asseemingly deleted, as it was absent in rawt\+1t\+1andrecovered, because it reappeared in translatedt\+1t\+1\. Top\-ranked examples for latent neuron 146 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-tiny\-ImageNet with LwF\. The concept obtained a relativelyhigh decodability level: F1 of0\.710\.71and mean bacc of0\.920\.92for concept decodability\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.7:The example of agood quality concept\. The concept can be described asseemingly deleted, as it was absent in rawt\+1t\+1anddecodable, because it didn’t reappear in translatedt\+1t\+1but obtained concept decodabilityF1\>0F1\>0\. Top\-ranked examples for latent neuron 14 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-CIFAR10 with SGD\. The concept obtained a relativelylow concept decodability level: F1 of0\.440\.44and mean bacc of0\.880\.88\.In the qualitative analysis, we present representative examples of concepts drawn from the proposed taxonomy, illustrating different post\-task outcomes such asretained\(Figs\.[J\.1](https://arxiv.org/html/2605.16374#A10.F1),[J\.2](https://arxiv.org/html/2605.16374#A10.F2),[J\.3](https://arxiv.org/html/2605.16374#A10.F3)\),deleted\(Figs\.[J\.4](https://arxiv.org/html/2605.16374#A10.F4),[J\.6](https://arxiv.org/html/2605.16374#A10.F6),[J\.7](https://arxiv.org/html/2605.16374#A10.F7),[J\.8](https://arxiv.org/html/2605.16374#A10.F8)\),recovered\(Figs\.[J\.4](https://arxiv.org/html/2605.16374#A10.F4),[J\.6](https://arxiv.org/html/2605.16374#A10.F6),[J\.5](https://arxiv.org/html/2605.16374#A10.F5)\), anddecodable\(Figs\.[J\.7](https://arxiv.org/html/2605.16374#A10.F7),[J\.8](https://arxiv.org/html/2605.16374#A10.F8)\) concepts\. These examples complement the quantitative results by showing how the central visual structure of individual concepts expressed with top\-9 activating images changes after subsequent task learning\. Notably, across the cases examined in our analysis, we did not observe any concepts that would fall into theLostcategory at the strictest possible F1 threshold, i\.e\., concepts that were neither recoverable through translation nor decodable from the representation in our setup\.
\(a\)Aftertttraining
\(b\)Aftert\+1t\+1training
Figure J\.8:The example of alower quality concept\. The concept can be described asseemingly deleted, as it was absent in rawt\+1t\+1anddecodable, because it didn’t reappear in translatedt\+1t\+1but obtained concept decodabilityF1\>0F1\>0\. Top\-ranked examples for latent neuron 677 are presented attt\(top\) and att\+1t\+1\(bottom\)\. The setup was 2seq\-CIFAR10 with SGD\. The concept obtained a relativelylow decodability level: F1 of0\.280\.28and mean bacc of0\.750\.75\.In many of presented cases images representing the concepts atttshow a clear common motif, e\.g\. brown, furry objects \(Fig\.[J\.1](https://arxiv.org/html/2605.16374#A10.F1)\), penguins \(Fig\.[J\.2](https://arxiv.org/html/2605.16374#A10.F2)\), bisons \(Fig\.[J\.4](https://arxiv.org/html/2605.16374#A10.F4)\) or birds \(Fig\.[J\.7](https://arxiv.org/html/2605.16374#A10.F7)\), regardless of thet\+1t\+1outcome of the concept transition, which wasretainedfor the first two examples,recoveredfor the third example anddecodablefor the fourth example\. Some lower\-quality neurons can be showed as well, e\.g\. see Figs\.[J\.6](https://arxiv.org/html/2605.16374#A10.F6)and[J\.8](https://arxiv.org/html/2605.16374#A10.F8)\. Nevertheless, even in these cases, some recurring objects or motifs can still be identified\. For example, in Fig\.[J\.6](https://arxiv.org/html/2605.16374#A10.F6), several highly activating examples contain backpacks or towers, while in Fig\.[J\.8](https://arxiv.org/html/2605.16374#A10.F8), recurring elements such as cats and cars can be observed\. This behavior is consistent with standard SAEs, which typically extract features of varying quality and degrees of disentanglement\. Overall, the presented examples still suggest that the SAE and our mean\-based and frequency\-based thresholding criteria capture at least partially disentangled features associated with recurring visual patterns, supporting our use of these latent units as proxies for concepts\.
Across all presented examples, the top\-99activating images at taskttand after learning taskt\+1t\+1remain highly aligned, indicating that the centers of the corresponding concepts’ \(latent features\) activation spaces are largely preserved\. This is reflected in shared visual motifs and, in many cases, even repeated images among the strongest activations\. Importantly, this observation is valid even for the seemingly deleted, recovered and decodable categories\. Fig\.[J\.4](https://arxiv.org/html/2605.16374#A10.F4)shows that some contamination of the taskttconcept center can occur att\+1t\+1\. While at taskttthe dominant motifs include green plants, water, and bisons on grass, at taskt\+1t\+1an out\-of\-pattern image of a girl in a sombrero appears among the top activations\. These qualitative observations provide graphical support for the quantitative results in the paper, which indicate that a large portion of concept\-level knowledge is preserved during updates on subsequent tasks, although some information may become harder to read out or partially lost\.
## Appendix KHyperparameters
In ourconcept\-level analysis setup, for each task, we train an SAE withBatchTopKSAEregime via theovercomplete333[https://github\.com/KempnerInstitute/overcomplete](https://github.com/KempnerInstitute/overcomplete)on the same Mammoth’s train/test splits\. We use the unified setup: batch sizeB=16B=16, top\-K=10K=10,1010epochs, learning rate5⋅10−35\\cdot 10^\{\-3\}, expansion rate22, Mean Squared Error \(MSE\) loss augmented with anovercomplete’s dead\-neuron reactivation loss weighted by10−210^\{\-2\}\. Across experiments, manual inspection showed reconstructionR2\>0\.6R^\{2\}\>0\.6and close to 0% dead\-neuron rate, indicating meaningful and non\-degenerate SAEs\. As a qualitative complement, we examine top 9 activating images for selected SAE latents and find consistent shared visual motifs, indicating that they capture coherent patterns rather than arbitrary feature mixtures \(Appendix[J](https://arxiv.org/html/2605.16374#A10)\)\. We use frequency\-based binarization thresholdτ=0\.05\\tau=0\.05\. We use a single linear layer as for translation, trained for100100epochs withAdamWusing MSE, learning rate10−310^\{\-3\}, weight decay10−410^\{\-4\}, batch size128128, and a validation split0\.20\.2\. For task\-level probes, we use Logistic Regression withlbfgssolver and 1500 maximum iterations, while for concept\-level prediction we use Logistic Regression withliblinearsolver, 1000 iterations, andbalancedclass weight\. We verified the analysis robustness across 10 runs \(App\.[D](https://arxiv.org/html/2605.16374#A4)\), frequency\-based binarization thresholds \(App\.[E](https://arxiv.org/html/2605.16374#A5), \{0\.00625, 0\.0125, 0\.025, 0\.05, 0\.1, 0\.2\}\), SAE batch size \(App\.[H](https://arxiv.org/html/2605.16374#A8),B∈\{16,32,64,128,256\}B\\in\\\{16,32,64,128,256\\\}\) and SAEKK\(App\.[G](https://arxiv.org/html/2605.16374#A7),K∈\{10,16,32,64\}K\\in\\\{10,16,32,64\\\}\)\. The results show consistent behavior of key metrics, with variability largely within run\-to\-run fluctuations, and support robustness of our conclusions\. We further validate that the selected active neurons retain most task\-relevant information with a SAE\-based experiments on deletion of non\-active neurons under different values of parameters and under our binarization rule for all methods and datasets, further supporting the use of threshold0\.050\.05\. We used multiple NVIDIA GH200 GPUs for continual learning experiments and 1 NVIDIA Titan RTX GPU for SAE training and evaluation\. Average execution time for an example task, including saving all the required and additional data, was204\.199±42\.473204\.199\\pm 42\.473s on the Titan RTX GPU\.
## Appendix LMonosemanticity of SAE concept proxies
As an additional sanity check of the quality of the SAE latent features used as concept proxies in our framework, we evaluate their monosemanticity using the Monosemanticity Score \(MS\) introduced in work\[[23](https://arxiv.org/html/2605.16374#bib.bib28)\]\. The aim of this experiment is not to claim that SAE latents correspond to ground\-truth semantic concepts, but to quantitatively assess whether the selected latent features capture more coherent visual patterns than random associations\.
MS measures how coherent are the images that strongly activate a given SAE latent neuron\. Pairwise cosine similarities are computed between embeddings from a pretrained encoder\. For a neuronkk, representing a given concept proxy, latent activations are used to weight similarities between samples\. The MS is then defined as the activation\-weighted average of pairwise embedding similarities, assigning higher scores to neurons whose most ctivating examples correspond to coherent visual patterns, and lower scores to neurons activating for diverse unrelated images\[[23](https://arxiv.org/html/2605.16374#bib.bib28)\]\.
We perform the analysis for an example configuration: 2seq\-tiny\-ImageNet under the LwF continual learning strategy\. For each active SAE latent neuron identified under our frequency\-based binarization criterion, we compute the MS using embeddings extracted from the continual model backbone used as an encoder\. As a baseline, we additionally compute the same score after randomly permuting the embeddings from the encoder, which destroys the sample similarity structure while preserving the structure of the encoder embedding space\. This lets us assess whether latent neurons capture more monosemantic patterns than expected by chance, disentangling embedding\-space geometry from neuron\-induced meaningful similarity\.
Figure L\.1:Monosemanticity Score \(MS\) analysis for active SAE latent neurons in the example 2seq\-tiny\-ImageNet \+ LwF configuration\. The box plots compare MS values of active SAE latent neurons with those obtained by permuting embeddings\. Active SAE latent neurons achieve substantially higher MS values, indicating that they capture more coherent visual patterns than expected from random associations within the embedding space\.Fig\.[L\.1](https://arxiv.org/html/2605.16374#A12.F1)shows that the SAE latents active under our binarization rule achieve substantially higher MS values than the permutation baseline\. Although the scores vary across neurons, indicating different degrees of concept quality and disentanglement, which is expected from SAE latents, the majority of features are significantly more coherent than the baseline\. In contrast, the baseline produces consistently lower and stable scores, as expected for randomly associated examples\. The presented quantitative results support that the analyzed SAE latent features capture recurring and coherent visual patterns rather than arbitrary mixtures of activations, and thus can serve as concept proxies\.Similar Articles
Erased, but Not Gone: Output Forgetting Is Not True Forgetting
This paper argues that standard output-level evaluations of machine unlearning overestimate success, showing that methods can appear successful at the output layer while retaining structured representation-level discrepancies relative to retrained models. The authors propose retraining-consistent representation forgetting as a stronger evaluative lens.
The Art of Not Forgetting A Local Learning Architecture for Continual Learning
This paper introduces CMP (Cognitive Memory Primitive), a continual-learning architecture that uses sparse relational codes and local learning to reduce catastrophic forgetting, demonstrating better backward transfer than a Transformer with EWC on a byte-level language modeling protocol.
A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders
This paper proposes a unified geometric framework for understanding concept learning and neuron interpretation in sparse autoencoders, formalizing concepts as sets and defining detection, separation, and approximation. It provides error bounds, capacity constraints, and links to formal concept analysis, with experiments on synthetic data.
Forgetting is Not Erasure: Recovering Latent Knowledge via Transport Keys
This paper argues that catastrophic forgetting in neural networks is not erasure but an interface alignment problem. It introduces 'transport keys' to recover latent task-specific features from sequentially trained models, demonstrating significant performance recovery on split CIFAR-100.
Rethinking Backdoor Adversarial Unlearning through the Lens of Catastrophic Forgetting in Continual Learning
This paper rethinks backdoor unlearning from a continual learning perspective, defining complete backdoor unlearning and proposing Blind Inversion-Backdoor Adversarial Unlearning (BI-BAU) that integrates adversarial training into an EM algorithm to effectively eliminate backdoor effects across various attack types and modalities.