Metaplasticity as adaptive gradient preconditioning for incremental learning

# Metaplasticity as adaptive gradient preconditioning for incremental learning
Source: [https://arxiv.org/html/2608.14634](https://arxiv.org/html/2608.14634)
###### Abstract

Biological intelligence naturally prevents catastrophic forgetting through Complementary Learning Systems \(CLS\) theory, a macroscopic consolidation process driven at the local level by synaptic metaplasticity: the continuous, history\-dependent neuromodulation of individual synapses\. While artificial neural networks struggle with the stability\-plasticity dilemma in non\-stationary environments, existing solutions often require task labels or incur massive memory overhead, diverging from biological reality\. Re\-framing this localized neuromodulation as an optimization\-driven process, we introduceSynGAP:SynapticGeometricAdaptivePreconditioning\. SynGAP is a task\-free continual learning framework based on adaptive gradient preconditioning\. Rather than relying on explicit episodic triggers, SynGAP simulates real\-time metaplasticity by maintaining an exponential moving average of the Fisher Information Matrix over a continuous data stream\. During the optimization step, these dynamic metaplastic states are translated into a bounded multiplicative mask that preconditions raw gradients, selectively attenuating updates to critical historical parameters\. Empirical evaluations demonstrate SynGAP’s superior ability to mitigate catastrophic forgetting compared to established baselines\. On the Split CIFAR\-100 benchmark, SynGAP delivers a4×4\\timesincrease in accuracy compared to EWC\+\+ and outperforms Experience Replay \(ER\) by almost10%10\\%, while reducing the forgetting measure by over10%10\\%against both methods\. Furthermore, on the CORe50 benchmark, SynGAP achieves about68%68\\%, a10%10\\%improvement over optimizer baselines\. By mathematically formalizing continuous biological metaplasticity as stable gradient\-based regularization, SynGAP offers a highly robust and memory\-efficient solution for adaptive intelligence at the edge\.

## Introduction

Biological intelligence is grounded in its ability to learn continuously, adapting to a non\-stationary stream of environmental stimuli without catastrophically forgetting previously acquired knowledge\. In artificial neural networks \(ANNs\), this stability\-plasticity dilemma remains a fundamental challenge\(Kudithipudiet al\.[2022](https://arxiv.org/html/2608.14634#bib.bib1)\)\(Fig\.[1](https://arxiv.org/html/2608.14634#Sx1.F1)a\)\. Standard gradient descent algorithms optimize parameters myopically for immediate performance, overwriting latent representations vital to historical tasks, resulting in a phenomenon known ascatastrophic forgetting\(McCloskey and Cohen[1989](https://arxiv.org/html/2608.14634#bib.bib2); French[1999](https://arxiv.org/html/2608.14634#bib.bib48)\)\. Complementary Learning Systems \(CLS\) theory proposes that the brain resolves the stability\-plasticity dilemma through systems consolidation, in which the hippocampus acts as a rapid buffer for new information that is subsequently integrated into the neocortex\(O’Reillyet al\.[2014](https://arxiv.org/html/2608.14634#bib.bib3)\)\. This gradual integration is likely governed at the cellular level by metaplasticity, or the "plasticity of synaptic plasticity"\(Fusiet al\.[2005](https://arxiv.org/html/2608.14634#bib.bib4); Jedlickaet al\.[2022](https://arxiv.org/html/2608.14634#bib.bib15)\)\(Fig\.[1](https://arxiv.org/html/2608.14634#Sx1.F1)b, c\)\.

Deep learning architectures have employed architectural, replay\-based, or regularization strategies to facilitate continual learning\(Wanget al\.[2024](https://arxiv.org/html/2608.14634#bib.bib55)\)\. However, these paradigms frequently diverge from biological constraints\(Hesset al\.[2023](https://arxiv.org/html/2608.14634#bib.bib56)\):

- •Architectural and parameter\-isolation methodsoften suffer from capacity limits or require explicit task boundaries to allocate new subnets\.
- •Replay\-based frameworks, while empirically robust, impose substantial memory overhead via episodic buffers and suffer from "stability gaps" at task transition boundaries due to sudden distribution shifts\.
- •Regularization techniquessuch as Elastic Weight Consolidation \(EWC\)\(Kirkpatricket al\.[2017](https://arxiv.org/html/2608.14634#bib.bib8)\)capture the geometry of the historical loss landscape via the Fisher Information Matrix \(FIM\)\. However, these methods typically require explicit task IDs to compute and freeze importance metrics at the end of distinct training phases, relying on additive loss penalties that distort the global optimization landscape\.

In this work, we argue that synaptic metaplasticity can be mathematically re\-framed as a normative, geometry\-aware optimization strategy\. Rather than retroactively evaluating the importance of parameters, a continual learning agent must treat optimization as navigation through a partially observable Riemannian manifold\(Vastolaet al\.[2025](https://arxiv.org/html/2608.14634#bib.bib6)\)\. In this space, the local curvature corresponds to the critical structural knowledge of past data\. Standard stochastic gradient descent \(SGD\) operates myopically in Euclidean space\. By blindly following the steepest descent of the current stimuli, it violently displaces parameters along directions of varying loss curvatures\(Kingma and Ba[2015](https://arxiv.org/html/2608.14634#bib.bib35); Chnget al\.[2025](https://arxiv.org/html/2608.14634#bib.bib52)\)\. We demonstrate that it can also be used to accelerate convergence by adjusting for these varying loss curvatures\(Kingma and Ba[2015](https://arxiv.org/html/2608.14634#bib.bib35); Chnget al\.[2025](https://arxiv.org/html/2608.14634#bib.bib52)\)\. We demonstrate that it can serve a crucial inverse objective\. By mapping the parameter\-space geometry of joint tasks in real time, preconditioning can serve as a structural stabilizer for continual learning\.

![Refer to caption](https://arxiv.org/html/2608.14634v1/x1.png)Figure 1:Balancing the stability\-plasticity dilemma requires metaplastic control\.\(a\)Conventional learning leads to catastrophic forgetting of an old task \(Task A\), where the introduction of a new task \(Task B\) causes an unstable balance on the plasticity spectrum\.\(b\)To counteract this, a metaplasticity mechanism shifts the "fulcrum" of learning on a seesaw that holds both tasks, balancing the consolidation of old knowledge with the intake of new information\.\(c\)At the network level, this is implemented via localized gradient flow modulation at the synapse\. As an adaptive valve, metaplasticity regulates gradient updates to determine the most balanced position along the plasticity spectrum\.Building upon this geometric intuition, we introduceSynapticGeometricAdaptivePreconditioning \(SynGAP\), a gradient\-based, task\-free continual learning framework\. Operating without explicit episodic triggers, SynGAP simulates continuous neuromodulation by maintaining an online exponential moving average of the FIM over a continuous data stream\. During each optimization step, this dynamic approximation of the historical loss curvature is mapped to a bounded multiplicative mask\. Rather than altering the global loss function with additive penalties, this mask directly preconditions raw stochastic gradients, geometrically attenuating update vectors in directions of high historical curvature while preserving plasticity along flat, non\-interfering parameter trajectories\.

SynGAP balances the stability\-plasticity trade\-off by enforcing rigorous mathematical bounds on its preconditioning mask, thereby systematically preventing the loss of plasticity or parameter intransigence that commonly affects rigid regularization methods\. By unifying biological metaplasticity with Riemannian parameter space geometry, SynGAP avoids the severe memory footprint of episodic data replay and relies only on a small episodic buffer\. Our framework delivers a highly robust, computationally lightweight optimizer for continuous, adaptive intelligence at the edge\.

Our core contributions are structured as follows:

- •We mathematically formalize biological metaplasticity as a gradient preconditioning problem\.
- •We propose the SynGAP optimizer, which continuously tracks parameter importance across non\-stationary streams without requiring task boundaries or incurring extensive data storage overhead\.
- •We introduce a bounded multiplicative masking mechanism that explicitly maintains parameter plasticity, providing safeguards against network freezing\.

![Refer to caption](https://arxiv.org/html/2608.14634v1/x2.png)Figure 2:SynGAP\.\(a\)To achieve continual learning success, the model must optimize the joint loss of all tasks\.\(b\)Naive optimization struggles to find a joint minimum for multi\-task continual learning, as updates are executed globally on the newest task data\.\(c\)SynGAPintroduces local metaplastic updates that are governed by a functiong​\(F\)g\(\\textbf\{F\}\)that modulates the optimization step based on Fisher Information of the joint loss\. This figure is produced with the aid of Gemini 3\.
## Related Works

### Continual learningviaparameter regularization

A common strategy for mitigating catastrophic forgetting is parameter regularization, which aims to facilitate continual learning by restricting the plasticity of individual weights according to their estimated importance to previously learned tasks\. These approaches typically append a penalty term to the global loss function, gently pulling the updated weights back toward their consolidated values\. The primary distinction between these methods lies in how they estimate parameter importance\. EWC\(Kirkpatricket al\.[2017](https://arxiv.org/html/2608.14634#bib.bib8)\)leverages the diagonal of the empirical FIM to measure local parameter sensitivity, but requires strict task IDs and access to true labels\. Synaptic Intelligence \(SI\)\(Zenkeet al\.[2017](https://arxiv.org/html/2608.14634#bib.bib9)\)continuously tracks a path integral of gradients during training, and Memory\-Aware Synapses \(MAS\)\(Aljundiet al\.[2018](https://arxiv.org/html/2608.14634#bib.bib10)\)use the gradient of the network’s output function, thereby adding an unsupervised estimate of parameter importance\. More recently, Test Time Adaptation\-based methods such as Efficient Anti\-forgetting Test\-time Adaptation \(EATA\)\(Niuet al\.[2022](https://arxiv.org/html/2608.14634#bib.bib72)\)have extended this paradigm to label\-free, streaming environments by employing pseudo\-labels to estimate Fisher information and actively filtering high\-entropy samples to reduce the computational overhead of backward passes\.

Despite their widespread adoption and recent adaptations for efficient inference, regularization methods typically rely on post hoc additive penalties on the loss function, which can cause scaling issues during optimization as the number of tasks increases\. By treating consolidation as a static constraint rather than a dynamic property of the active learning process, these methods often suffer from a trade\-off between retaining past knowledge and adapting to new data distributions\(Dohareet al\.[2024](https://arxiv.org/html/2608.14634#bib.bib27)\)\. This limitation underscores the need for approaches that intervene directly in the optimization step, moving away from global loss penalties in favor of embedding memory protection seamlessly into the network’s update dynamics\.

### Gradient modification and preconditioning

Moving beyond static loss penalties, gradient preconditioning methods intervene directly in optimization dynamics by scaling weight updates based on historical gradients or loss geometry\(Kingma and Ba[2015](https://arxiv.org/html/2608.14634#bib.bib35); Amari[1998](https://arxiv.org/html/2608.14634#bib.bib63)\)\. Recent advancements include gradient projection techniques, which update models along orthogonal directions to the old tasks’ gradients to minimize interference\(Farajtabaret al\.[2020](https://arxiv.org/html/2608.14634#bib.bib73)\)\. To mitigate the rigidity of strict orthogonality, Scaled Gradient Projection \(SGP\) combines orthogonal projections with scaled updates along past core gradient spaces\(Saha and Roy[2023](https://arxiv.org/html/2608.14634#bib.bib74)\)\. SGP computes this basis importance via Singular Value Decomposition \(SVD\) on input representations, bypassing the need for data replay\(Sahaet al\.[2021](https://arxiv.org/html/2608.14634#bib.bib75)\)\.

However, these methods face severe computational bottlenecks, particularly limiting their viability for low\-power edge applications\. Standard optimizers use transient scaling factors that rapidly overwrite historical importance\. Conversely, rigorous projection methods such as SGP require the continuous computation and manipulation of large basis matrices via SVD\(Sahaet al\.[2021](https://arxiv.org/html/2608.14634#bib.bib75)\), incurring prohibitive global overhead\. Consequently, the field lacks a localized, permanently stateful mechanism that can embed enduring memory protection directly into gradient updates without expensive mathematical operations\. Looking to neuroscience, this gap naturally points toward the complex, history\-dependent metaplasticity observed in biological synapses\.

### Biological metaplasticity

To overcome the limitations of transient optimizers and computationally prohibitive curvature matrices, recent continual learning architectures draw inspiration from the brain’s Complementary Learning Systems \(CLS\)\(O’Reillyet al\.[2014](https://arxiv.org/html/2608.14634#bib.bib3); Phamet al\.[2021](https://arxiv.org/html/2608.14634#bib.bib76)\)and the micro\-level dynamics of "complex" synapses\. In biological networks, hidden internal states dictate the metaplasticity of a synapse \("the plasticity of its plasticity"\), ensuring that prior consolidation structurally preconditions future susceptibility to change\(Benna and Fusi[2016](https://arxiv.org/html/2608.14634#bib.bib14); Fusiet al\.[2005](https://arxiv.org/html/2608.14634#bib.bib4)\)\. Although some continual learning approaches have translated this concept into artificial neural networks\(Aguilaret al\.[2025](https://arxiv.org/html/2608.14634#bib.bib16); Laborieuxet al\.[2021](https://arxiv.org/html/2608.14634#bib.bib51)\), they are not architecture\-agnostic and suffer from a capacity bottleneck that prevents them from capturing complex, high\-dimensional distributions or large\-scale tasks\. Consequently, there is a lack of a continuous, model\-agnostic, and highly scalable metaplastic framework\.

## Preliminaries

#### Continual learning setting\.

In standard task\-based continual learning \(CL\), a model is trained over a sequence of tasks𝒯=\{T1,T2,…,TT\}\\mathcal\{T\}=\\\{T\_\{1\},T\_\{2\},\\dots,T\_\{T\}\\\}\(Mirzadehet al\.[2020](https://arxiv.org/html/2608.14634#bib.bib40)\)\. Each taskTtT\_\{t\}is associated with a specific dataset𝒟t=\{\(xit,yit\)\}i=1Nt\\mathcal\{D\}\_\{t\}=\\\{\(x\_\{i\}^\{t\},y\_\{i\}^\{t\}\)\\\}\_\{i=1\}^\{N\_\{t\}\}, wherexit∈𝒳x\_\{i\}^\{t\}\\in\\mathcal\{X\}represents an input sample andyit∈𝒴y\_\{i\}^\{t\}\\in\\mathcal\{Y\}is the corresponding label, drawn from a task\-specific data distributionPt​\(X,Y\)P\_\{t\}\(X,Y\)\.

During the training phase for taskTtT\_\{t\}, the model only has access to the current dataset𝒟t\\mathcal\{D\}\_\{t\}\. Access to data from previous tasks is typically restricted to a small memory buffer\. The overarching objective after training ontttasks is to find the optimal parametersθ\\thetathat minimize the expected risk across all tasks observed so far:

minθ​∑k=1t𝔼\(x,y\)∼Pk​\[ℒ​\(fθ​\(x\),y\)\]\\min\_\{\\theta\}\\sum\_\{k=1\}^\{t\}\\mathbb\{E\}\_\{\(x,y\)\\sim P\_\{k\}\}\[\\mathcal\{L\}\(f\_\{\\theta\}\(x\),y\)\]\(1\)wherefθf\_\{\\theta\}is the neural network parameterized byθ\\theta, andℒ\\mathcal\{L\}is a standard loss function \(here, we use cross\-entropy\)\. The fundamental challenge in this setting is to successfully learn the distributionPtP\_\{t\}without severely degrading the performance of previous distributionsP1:t−1P\_\{1:t\-1\}\. Minimizing this expected risk is typically done by optimizing the joint loss of all datasets,⋃k=1t𝒟k\\bigcup\_\{k=1\}^\{t\}\\mathcal\{D\}\_\{k\}, which CL methods must approximate without full access to past data\.

#### The Fisher information matrix\.

To measure the importance of parameters for past data, we rely on the Fisher Information Matrix \(FIM\)\(Rattrayet al\.[1998](https://arxiv.org/html/2608.14634#bib.bib24); Amari[1998](https://arxiv.org/html/2608.14634#bib.bib63)\)\. The theoretical FIM is defined as the covariance of the score function:

ℱ​\(θ\)=𝔼x∼PX,y∼Pθ​\(Y\|X\)​\[∇θlog⁡pθ​\(y\|x\)​∇θlog⁡pθ​\(y\|x\)⊤\]\\mathcal\{F\}\(\\theta\)=\\mathbb\{E\}\_\{x\\sim P\_\{X\},y\\sim P\_\{\\theta\}\(Y\|X\)\}\\left\[\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(y\|x\)\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(y\|x\)^\{\\top\}\\right\]\(2\)Note that the targetsyyare sampled from the model’s predictive distributionPθ​\(Y\|X\)P\_\{\\theta\}\(Y\|X\), rather than the ground\-truth labels \(which constitute the Empirical Fisher\)\(van de Ven[2025](https://arxiv.org/html/2608.14634#bib.bib28)\)\. To maintain computational feasibility, we restrict our formulation to the diagonal approximation of the FIM, denoted asF∈ℝ\|θ\|\\textbf\{F\}\\in\\mathbb\{R\}^\{\|\\theta\|\}\(Soen and Sun[2024](https://arxiv.org/html/2608.14634#bib.bib50)\)\.

#### Gradient preconditioning\.

Standard gradient descent updates parameters viaθt\+1=θt−η​∇θℒt\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{t\}\. Preconditioned gradient methods modify the update direction by applying a transformation matrixP−1P^\{\-1\}:

θt\+1=θt−η​P−1​∇θℒt\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta P^\{\-1\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{t\}\(3\)While adaptive optimizers like Adam\(Kingma and Ba[2015](https://arxiv.org/html/2608.14634#bib.bib35)\)constructP−1P^\{\-1\}to accelerate optimization in high\-curvature directions, our framework leverages preconditioning in the opposite direction to attenuate updates in directions critical to preserving historical knowledge\.

## Methodology

In this section, we introduce SynGAP\. Letθ∈ℝD\\theta\\in\\mathbb\{R\}^\{D\}denote the flattened parameter vector of the neural network andℒ​\(θ\)\\mathcal\{L\}\(\\theta\)denote the empirical loss function over the continuous data stream\.

#### Online Fisher information tracking\.

Unlike methods that compute importance metrics at the task boundary, SynGAP maintains a continuous estimate of parameter importance\. We simulate the accumulation of the metaplastic state by computing an online Exponential Moving Average \(EMA\) of the diagonal FIM\. The global state vectorFis updated continuously:

Ft=α​Ft−1\+\(1−α\)​diag​\(ℱt​\(θ\)\)F\_\{t\}=\\alpha F\_\{t\-1\}\+\(1\-\\alpha\)\\text\{diag\}\(\\mathcal\{F\}\_\{t\}\(\\theta\)\)\(4\)whereα∈\[0,1\)\\alpha\\in\[0,1\)is the metaplastic retention rate, andℱt​\(θ\)\\mathcal\{F\}\_\{t\}\(\\theta\)is the instantaneous Fisher information evaluated on the current batch\.

#### Bounded metaplastic preconditioning\.

Raw FIM values can vary by orders of magnitude, contributing to instability in optimization\. To translate the unbounded metaplastic state vectorFFinto a stable learning mechanism, we introduce a globally normalized plasticity function, denotedg​\(F\)g\(\\textbf\{F\}\)\.

Letμ=1D​∑i=1DFi\\mu=\\frac\{1\}\{D\}\\sum\_\{i=1\}^\{D\}F\_\{i\}be the global mean of the Fisher EMA across all network parameters\. We define a dynamic scaling factorccto normalize the landscape:

c=arctanh​\(τ\)μ\+λc=\\frac\{\\text\{arctanh\}\(\\tau\)\}\{\\mu\+\\lambda\}\(5\)whereτ∈\(0,1\)\\tau\\in\(0,1\)is the target consolidation hyperparameter\.

We define the plasticity function as:

g​\(F\)=1−tanh⁡\(c​\(F\+λ\)\)\+ϵg\(\\textbf\{F\}\)=1\-\\tanh\(c\(\\textbf\{F\}\+\\lambda\)\)\+\\epsilon\(6\)
whereλ\\lambdaacts as a Tikhonov damping term for numerical stability, andϵ\\epsilonis a small scalar representing ambient plasticity \(to prevent complete intransigence\)\. The output of this function yields the consolidation mask,Ω=g​\(F\)\\Omega=g\(\\textbf\{F\}\), which is strictly bounded within\[ϵ,1\+ϵ\]\[\\epsilon,1\+\\epsilon\]\. All operations in Eq\.[6](https://arxiv.org/html/2608.14634#Sx4.E6)are applied element\-wise\.

As illustrated in Fig\.[2](https://arxiv.org/html/2608.14634#Sx1.F2), the SynGAP architecture operates via two parallel streams during the optimization step\. While the standard backward pass computes the raw task gradients∇θℒ​\(θ\)\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\), the metaplastic tracking module updates simultaneously the historical Fisher EMA stateFF\.

The plasticity functiong​\(F\)g\(F\)acts as a gating mechanism, transforming the unbounded historical stateFFinto the bounded multiplicative maskΩ\\Omega\. The raw gradients are then preconditioned by this mask before altering the network\. The final SynGAP update rule at optimization stepttis formulated as:

θt\+1=θt−η​\(g​\(F\)⊙∇θℒ​\(θ\)\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\(g\(\\textbf\{F\}\)\\odot\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\)\)\(7\)whereη\\etais the global learning rate and⊙\\odotdenotes the element\-wise Hadamard product\. By mathematically decoupling the gradient computation from the metaplastic gating, we demonstrate that SynGAP requires no additive loss penalty, cleanly avoiding the catastrophic gradient scaling and loss\-landscape distortion issues common in prior continual learning methods\(Liet al\.[2023](https://arxiv.org/html/2608.14634#bib.bib49)\)\.

## Experiments

To validate the efficacy of SynGAP, we evaluate our framework on challenging benchmarks for class\-incremental and continuous domain\-incremental learning\. We aim to answer three key questions: \(1\) How does SynGAP compare to state\-of\-the\-art continual learning methods, \(2\) How does metaplastic preconditioning compare to standard adaptive optimizers, and \(3\) What is the impact of the bounded plasticity mask?

Table 1:Performance metrics on Split CIFAR\-100 and CORe50\. Average Accuracy \(ACC\), Average Forgetting \(FM\), and Intransigence \(INT\) are reported in%\\%\.Joint \(oracle\)serves as an upper bound for accuracy\. All methods are averaged over five full runs\. Best results are in bold\.### Experimental setup

#### Benchmarks\.

We evaluate on two standard datasets:

- •Split CIFAR\-100:A class\-incremental benchmark of100100classes, split into1010tasks \(with1010classes per task\)\. Although the dataset contains discrete shifts, SynGAP processes the stream continuously without access to task identifiers or boundary signals\.
- •CORe50:A continuous object recognition benchmark designed for highly non\-stationary environments, comprising5050classes\. We utilize the New Classes \(NC\) scenario, which mimics a smooth, continuous data stream\(Lomonaco and Maltoni[2017](https://arxiv.org/html/2608.14634#bib.bib31)\)\.

#### Baselines\.

We compare SynGAP with two categories of baselines:

- •Continual Learning Baselines:Experience Replay \(ER\)\(Rolnicket al\.[2019](https://arxiv.org/html/2608.14634#bib.bib12)\)and the online version of Elastic Weight Consolidation, EWC\+\+\(Kirkpatricket al\.[2017](https://arxiv.org/html/2608.14634#bib.bib8); Chaudhryet al\.[2018](https://arxiv.org/html/2608.14634#bib.bib5)\)\. All strategies operate without requiring task IDs\. All sampling buffers used are equal to20​n20nfornnclasses\.
- •Optimizer Baselines:SGD, SGD with Momentum \(SGDM\), Adam, and Ballistic\(Vastolaet al\.[2025](https://arxiv.org/html/2608.14634#bib.bib6)\)\.

#### Evaluation Metrics\.

To assess performance, we report the Average Accuracy \(ACC\)\(Lopez\-Paz and Ranzato[2017](https://arxiv.org/html/2608.14634#bib.bib58)\), where the mean performance across the data stream is measured after the model is completely trained\. LetA​C​C∈\[0,100\]ACC\\in\[0,100\]as a percentage value\.

ACC=1T​∑i=1TRT,i\\text\{ACC\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}R\_\{T,i\}\(8\)
In Equ\.[8](https://arxiv.org/html/2608.14634#Sx5.E8),Ri,jR\_\{i,j\}is the test classification accuracy of the model on taskTjT\_\{j\}after observing the last sample from taskTiT\_\{i\}\. For additional continual learning metrics, we also report Average Forgetting \(FM\) and Intransigence \(INT\), as first introduced in\(Chaudhryet al\.[2018](https://arxiv.org/html/2608.14634#bib.bib5)\)\. LetF​M∈\[−100,100\]FM\\in\[\-100,100\]andI​N​T∈\[−100,100\]INT\\in\[\-100,100\]be in percent\. Specifically,FMevaluates catastrophic forgetting by measuring the average drop in accuracy for each task from its peak performance to its final performance at the end of training\.

Conversely,INTmeasures the model’s resistance to acquiring new information, calculated as the performance gap between a specific model and a reference upper\-bound model \(here, we use one trained jointly on all data\)\.

Together, these metrics provide a holistic assessment of the balance between the stability and plasticity of the model capacity\(Jedlickaet al\.[2022](https://arxiv.org/html/2608.14634#bib.bib15)\)\. All evaluation metrics are presented as percentage values\.

### Main results

Table[1](https://arxiv.org/html/2608.14634#Sx5.T1)summarizes the performance of our proposed method, SynGAP, against several baselines on the Split CIFAR\-100 and CORe50 benchmarks\. We evaluate the models based on Average Accuracy \(ACC\), Average Forgetting \(FM\), and Intransigence \(INT\)\. TheJoint \(oracle\)training performance is provided as an upper bound\.

#### Superior Accuracy and Forgetting Mitigation

SynGAP significantly outperforms all baseline methods in both overall accuracy and memory retention across both datasets\.

On the challenging Split CIFAR\-100 benchmark, standard optimization strategies \(SGD, SGDM, Adam, and Ballistic\), along with the regularization\-based EWC\+\+ method, suffer from severe catastrophic forgetting, yielding final accuracies below9%9\\%and forgetting measures exceeding51%51\\%\. While ER provides a strong baseline by boosting accuracy to17\.60%17\.60\\%, SynGAP establishes superior accuracy of27\.28%27\.28\\%and reduces average forgetting to37\.87%37\.87\\%\.

Similar trends are observed with the CORe50 dataset\. SynGAP achieves the highest accuracy at68\.65%68\.65\\%, outperforming the ER baseline \(67\.22%67\.22\\%\)\. EWC\+\+ notably struggles on this continuous benchmark, achieving a much lower accuracy of 40\.51% and a high forgetting measure of 60\.53%\. Furthermore, SynGAP demonstrates the highest resistance to catastrophic forgetting, with an FM of30\.85%30\.85\\%, compared to ER’s33\.72%33\.72\\%and standard SGD’s41\.56%41\.56\\%\.

#### The intransigence trade\-off

While SynGAP achieves the best ACC and FM, the Intransigence \(INT\) metric reveals an expected trade\-off inherent to continual learning systems\. Methods utilizing momentum or adaptive learning rates \(SGDM and Adam\) achieve negative intransigence scores on Split CIFAR\-100 \(−5\.90%\-5\.90\\%and−5\.20%\-5\.20\\%, respectively\) and near\-zero scores on CORe50\. This indicates high plasticity and the ability to learn new tasks rapidly\. However, this comes at the direct cost of catastrophic forgetting, as evidenced by their high FM scores and extremely low average accuracies on CIFAR\-100\.

Regularization approaches like EWC\+\+ can struggle with this balance, sometimes becoming overly rigid—as seen by its high INT score of 5\.67% on CORe50\. In contrast, SynGAP maintains a stable INT score \(6\.91%6\.91\\%on CIFAR\-100 and4\.09%4\.09\\%on CORe50\), which is only a few percentage points higher than those of the other methods\. This demonstrates that SynGAP successfully restricts excessive parameter updates to protect historical knowledge \(i\.e\., lowering FM\) without becoming overly rigid or severely impeding the model’s ability to learn incoming tasks\.

### Optimization trajectory

![Refer to caption](https://arxiv.org/html/2608.14634v1/x3.png)Figure 3:SynGAP smooths the optimization trajectory compared to standard optimizers\.On the initial task loss landscape, the optimization trajectory from initialization \(white circle\) to task 1 training \(cyan star\) to task 2 training \(yellow star\) is shown using\(a\)SGD, and\(b\)our method, SynGAP\. The proposed method navigates towards a wider, more stable basin between task 1 and task 2\.\(c\)The dynamic preconditioning function, as described in Equ\.[6](https://arxiv.org/html/2608.14634#Sx4.E6)is applied in SynGAP to modulate the gradient steps shown in\(b\)\. Plasticity is dictated by the metaplastic state as defined by Fisher\.Table 2:Ablation study of core components and hyperparameter sensitivity on Split CIFAR\-100\. Performance is evaluated across Average Accuracy \(ACC\), Forgetting Measure \(FM\), and Intransigence \(INT\) in%\\%\. Each configuration is averaged over five full experimental runs\.To better understand how our approach prevents catastrophic forgetting, we visualize the model’s optimization trajectory in Fig\.[3](https://arxiv.org/html/2608.14634#Sx5.F3)\. The figure projects the loss landscape onto a two\-dimensional plane defined by the direction vectors𝐮\\mathbf\{u\}and𝐯\\mathbf\{v\}\. Following the visualization techniques in\(Liet al\.[2018](https://arxiv.org/html/2608.14634#bib.bib37)\), we use filter\-normalized random directions to plot the loss contours of Task 1 of the Split CIFAR\-100 setting\. As illustrated in Fig\.[3](https://arxiv.org/html/2608.14634#Sx5.F3), the dotted lines trace the optimization path of the model parameters \(θ\\theta\) as it learns Task 2, starting from model initialization\.

As shown in Fig\.[3](https://arxiv.org/html/2608.14634#Sx5.F3)\(a\), the Naive SGD trajectory exhibits clear signs of catastrophic forgetting\. Although the model successfully locates a low\-loss minimum at the end of Task 1, the subsequent training phase for Task 2 drives the parameters into a region of significantly higher loss, demonstrating the destruction of previously acquired representations\.

However, our method visualized in Fig\.[3](https://arxiv.org/html/2608.14634#Sx5.F3)\(b\) shows how the trajectory during Task 2 is visibly restricted, ensuring that the final parameters remain anchored within a broader low\-loss basin associated with Task 1\.

This constrained trajectory is directly driven by the function detailed in Fig\.[3](https://arxiv.org/html/2608.14634#Sx5.F3)\(c\), which depicts the dynamic preconditioning function\. The plasticity termg​\(𝐅\)g\(\\mathbf\{F\}\)parameter\-wise gates on the optimization steps, decaying as the metaplastic state𝐅\\mathbf\{F\}increases\. The decay rate is dynamically adjusted based on the global mean of the Fisher information\. Through this plasticity function, SynGAP heavily penalizes movement along dimensions critical to historical tasks while maintaining high plasticity in directions where𝐅\\mathbf\{F\}remains low, naturally guiding the optimization trajectory along the contours of the previous task’s loss landscape\.

### Ablation studies

To isolate the contributions of the SynGAP method, we perform the following ablations:

- •The Impact of the Bounded Mask:We replace thetanh\\tanhbounded plasticity functiong​\(𝐅\)g\(\\bf\{F\}\)with an unbounded linear penalty \(similar to standard regularization\)\.
- •Ablation of Preconditioning:To isolate the effect of metaplastic preconditioning, we evaluate the full model against an ablated variant where preconditioning is removed, effectively reducing the system to standard ER\.
- •Metaplastic Retention Rate \(α\\alpha\):We analyze the sensitivity of the system by decreasing the EMA decay parameterα\\alpha, inducing extreme plasticity\.

Table[2](https://arxiv.org/html/2608.14634#Sx5.T2)presents the empirical results of these configurations on the Split CIFAR\-100 benchmark\.

First, substituting the bounded plasticity mask with an unbounded linear penalty results in severe performance degradation\. The average accuracy \(ACC\) drops significantly from 27\.28% \(Full Model\) to 14\.72%, while the forgetting measure \(FM\) increases from 37\.87% to 49\.46%\. Intransigence \(INT\) also rises slightly to 8\.18%\. This confirms that the globally normalizedtanh\\tanhmapping is essential for preventing gradient\-scaling instability and effectively shielding consolidated parameters from destructive updates\.

Second, removing the metaplastic preconditioning entirely, which reduces the architecture to a standard ER baseline, results in a substantial increase in forgetting\. Without preconditioning, the FM spikes to 54\.23% and overall accuracy falls to 17\.60%\. Although this ablation yields a lower INT of 5\.23%, indicating greater plasticity, it comes at a greater cost to memory retention\. This validates the use of preconditioning and its role in retaining historical knowledge\.

Finally, lowering the metaplastic retention rate \(α\\alpha\) induces plasticity, causing the model to prematurely overwrite the historical importance of its parameters\. This high\-plasticity regime destabilizes the learning process, reducing the final accuracy to 19\.27% and worsening the forgetting measure to 46\.32%\. This shows that a slower decay rate \(i\.e\., deep consolidation\) is necessary to ensure the model retains long\-term structural knowledge while sequentially adapting to new distributions\.

## Discussion

The empirical success of SynGAP highlights a fundamental flaw in traditional parameter\-regularization techniques for CL\. Methods like EWC rely on additive loss penalties, which inherently alter the global optimization landscape and often lead to gradients that scale disproportionately with the number of tasks\. By shifting to a strictly multiplicative preconditioning paradigm bounded bytanh\\tanh, SynGAP guarantees that the loss landscape of the current stream remains undistorted, while structurally critical weights are protected via gradient attenuation rather than loss manipulation\. Furthermore, replacing task\-based Fisher calculations with an online EMA allows SynGAP to organically track non\-stationary distributions, bridging the gap between CL regularization and standard adaptive optimization\.

Beyond correcting the optimization landscape, SynGAP exhibits a powerful synergy with experience replay\. Standard Experience Replay \(ER\) often struggles with gradient interference, where updates from the memory buffer conflict with the high\-magnitude gradients of the novel data stream\. By preconditioning the gradients, SynGAP acts as a geometric gatekeeper, ensuring that updates driven by new data are projected away from the critical dimensions of previously learned tasks\. This alignment reduces catastrophic interference at the synaptic level, explaining the significant performance gap between SynGAP and standard ER observed in our experiments\.

Additionally, the task\-free nature of SynGAP offers a distinct operational advantage\. Traditional consolidation methods require discrete task boundaries to compute and anchor the importance matrices, rendering them unsuitable for continuous, real\-world data streams\. By integrating the Fisher estimation directly into the online training step, SynGAP achieves continuous consolidation with minimal computational overhead\. It eliminates the need for expensive offline Fisher re\-computations and complex regularization\-scaling heuristics\.

#### Limitations\.

Despite its robustness, the SynGAP framework operates under several assumptions that present avenues for future work\. First, to maintain computational efficiency, SynGAP relies on the diagonal approximation of the Fisher Information Matrix \(FIM\)\. While sufficient, this ignores off\-diagonal dependencies\. In highly complex loss landscapes, a block\-diagonal or Kronecker\-factored \(K\-FAC\) approximation may be required to capture true geometric constraints\(Martens and Grosse[2015](https://arxiv.org/html/2608.14634#bib.bib25)\)\. Second, while metaplastic preconditioning is memory\-efficient, SynGAP still relies on a sampling buffer to provide the active\-learning signal\. Finally, the hyperparameterα\\alpha\(the metaplastic retention rate\) assumes a uniform timescale of forgetting across all layers, which may not accurately reflect the abstraction of hierarchical features in deep networks\(Ly and Gong[2025](https://arxiv.org/html/2608.14634#bib.bib47); Erhanet al\.[2009](https://arxiv.org/html/2608.14634#bib.bib22)\)\.

#### Future directions\.

Future work will explore hierarchical timescales to more accurately simulate localized biological metaplasticity\(Behrouzet al\.[2025](https://arxiv.org/html/2608.14634#bib.bib70)\)\. Additionally, extending the scalar plasticity maskΩ\\Omegato incorporate second\-order off\-diagonal geometric constraints could eliminate the need for the ambient plasticity termϵ\\epsilon, allowing for true parameter lockdown without risking dead networks\.

## Conclusion

In this work, we introduced SynGAP, a task\-free Continual Learning framework that re\-conceptualizes biological metaplasticity as online geometric gradient preconditioning\. By maintaining a continuous exponential moving average of the Fisher Information Matrix and projecting it through a globally normalized bounding function, SynGAP selectively attenuates gradients to protect historical knowledge without requiring explicit task boundaries or additive loss penalties\. Our empirical evaluations demonstrate that SynGAP successfully prevents the catastrophic forgetting exhibited by standard adaptive optimizers \(such as Adam\) on non\-stationary streams, while substantially outperforming traditional regularization and replay\-based baselines\. By mathematically formalizing the biological concept of synaptic consolidation into a stable, multiplicative optimization rule, SynGAP offers a highly robust solution for continuous, adaptive intelligence\.

### Acknowledgements

The authors acknowledge the support from the Australian Research Council under Project DP230100019\. I\.A\. acknowledges support from the Australian Government’s Research Training Program \(RTP\)\.

### Code Availability

The code generated during this study are available from the corresponding author upon reasonable request\.

### Data Availability

We employed publicly accessible datasets that can be accessed through the following links:

- —CIFAR\-100:https://www\.cs\.toronto\.edu/˜kriz/cifar\.html, and
- —CORe50:https://vlomonaco\.github\.io/core50\.

## References

- I\. Aguilar, T\. Bersani–Veroni, L\. F\. Herbozo Contreras, T\. Love, A\. Nikpour, D\. Querlioz, and O\. Kavehei \(2025\)Continuous metaplastic training on brain signals\.npj Unconventional Computing2\(1\),pp\. 9\.Cited by:[Biological metaplasticity](https://arxiv.org/html/2608.14634#Sx2.SSx3.p1.1)\.
- R\. Aljundi, F\. Babiloni, M\. Elhoseiny, M\. Rohrbach, and T\. Tuytelaars \(2018\)Memory aware synapses: Learning what \(not\) to forget\.InProceedings of the European Conference on Computer Vision \(ECCV\),pp\. 139–154\.Cited by:[Continual learningviaparameter regularization](https://arxiv.org/html/2608.14634#Sx2.SSx1.p1.1)\.
- S\. Amari \(1998\)Natural gradient works efficiently in learning\.Neural Computation10\(2\),pp\. 251–276\.External Links:[Document](https://dx.doi.org/10.1162/089976698300017746)Cited by:[Gradient modification and preconditioning](https://arxiv.org/html/2608.14634#Sx2.SSx2.p1.1),[The Fisher information matrix\.](https://arxiv.org/html/2608.14634#Sx3.SSx3.SSS0.Px2.p1.4)\.
- A\. Behrouz, M\. Razaviyayn, P\. Zhong, and V\. Mirrokni \(2025\)Nested learning: The illusion of deep learning architectures\.InNeural Information Processing Systems,Cited by:[Future directions\.](https://arxiv.org/html/2608.14634#Sx6.SSx4.SSS0.Px2.p1.2)\.
- M\. K\. Benna and S\. Fusi \(2016\)Computational principles of synaptic memory consolidation\.Nature Neuroscience19\(12\),pp\. 1697–1706\.Cited by:[Biological metaplasticity](https://arxiv.org/html/2608.14634#Sx2.SSx3.p1.1)\.
- A\. Chaudhry, P\. K\. Dokania, T\. Ajanthan, and P\. H\. Torr \(2018\)Riemannian walk for incremental learning: Understanding forgetting and intransigence\.InProceedings of the European Conference on Computer Vision \(ECCV\),pp\. 532–547\.Cited by:[1st item](https://arxiv.org/html/2608.14634#Sx5.I4.i1.p1.2),[Evaluation Metrics\.](https://arxiv.org/html/2608.14634#Sx5.SSx1.SSS0.Px3.p3.5)\.
- S\. Chng, H\. Saratchandran, and S\. Lucey \(2025\)Preconditioners for the stochastic training of implicit neural representations\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p3.1)\.
- S\. Dohare, J\. F\. Hernandez\-Garcia, Q\. Lan, P\. Rahman, A\. R\. Mahmood, and R\. S\. Sutton \(2024\)Loss of plasticity in deep continual learning\.Nature632\(8026\),pp\. 768–774\.Cited by:[Continual learningviaparameter regularization](https://arxiv.org/html/2608.14634#Sx2.SSx1.p2.1)\.
- D\. Erhan, P\. Manzagol, Y\. Bengio, S\. Bengio, and P\. Vincent \(2009\)The difficulty of training deep architectures and the effect of unsupervised pre\-training\.InArtificial Intelligence and Statistics,pp\. 153–160\.Cited by:[Limitations\.](https://arxiv.org/html/2608.14634#Sx6.SSx4.SSS0.Px1.p1.1)\.
- M\. Farajtabar, N\. Azizan, A\. Mott, and A\. Li \(2020\)Orthogonal gradient descent for continual learning\.InInternational conference on artificial intelligence and statistics,pp\. 3762–3773\.Cited by:[Gradient modification and preconditioning](https://arxiv.org/html/2608.14634#Sx2.SSx2.p1.1)\.
- R\. M\. French \(1999\)Catastrophic forgetting in connectionist networks\.Trends in Cognitive Sciences3\(4\),pp\. 128–135\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p1.1)\.
- S\. Fusi, P\. J\. Drew, and L\. F\. Abbott \(2005\)Cascade models of synaptically stored memories\.Neuron45\(4\),pp\. 599–611\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p1.1),[Biological metaplasticity](https://arxiv.org/html/2608.14634#Sx2.SSx3.p1.1)\.
- T\. Hess, T\. Tuytelaars, and G\. M\. van de Ven \(2023\)Two complementary perspectives to continual learning: ask not only what to optimize, but also how\.arXiv preprint arXiv:2311\.04898\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p2.1)\.
- P\. Jedlicka, M\. Tomko, A\. Robins, and W\. C\. Abraham \(2022\)Contributions by metaplasticity to solving the catastrophic forgetting problem\.Trends in Neurosciences45\(9\),pp\. 656–666\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p1.1),[Evaluation Metrics\.](https://arxiv.org/html/2608.14634#Sx5.SSx1.SSS0.Px3.p5.1)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: A method for stochastic optimization\.InInternational Conference for Learning Representations \(ICLR\),External Links:[Link](http://arxiv.org/abs/1412.6980)Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p3.1),[Gradient modification and preconditioning](https://arxiv.org/html/2608.14634#Sx2.SSx2.p1.1),[Gradient preconditioning\.](https://arxiv.org/html/2608.14634#Sx3.SSx3.SSS0.Px3.p1.3)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[3rd item](https://arxiv.org/html/2608.14634#Sx1.I1.i3.p1.1),[Continual learningviaparameter regularization](https://arxiv.org/html/2608.14634#Sx2.SSx1.p1.1),[1st item](https://arxiv.org/html/2608.14634#Sx5.I4.i1.p1.2)\.
- D\. Kudithipudi, M\. Aguilar\-Simon, J\. Babb, M\. Bazhenov, D\. Blackiston, J\. Bongard, A\. P\. Brna, S\. Chakravarthi Raja, N\. Cheney, J\. Clune,et al\.\(2022\)Biological underpinnings for lifelong learning machines\.Nature Machine Intelligence4\(3\),pp\. 196–210\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p1.1)\.
- A\. Laborieux, M\. Ernoult, T\. Hirtzlin, and D\. Querlioz \(2021\)Synaptic metaplasticity in binarized neural networks\.Nature communications12\(1\),pp\. 2549\.Cited by:[Biological metaplasticity](https://arxiv.org/html/2608.14634#Sx2.SSx3.p1.1)\.
- H\. Li, Z\. Xu, G\. Taylor, C\. Studer, and T\. Goldstein \(2018\)Visualizing the loss landscape of neural nets\.Advances in Neural Information Processing Systems31\.Cited by:[Optimization trajectory](https://arxiv.org/html/2608.14634#Sx5.SSx3.p1.3)\.
- H\. Li, J\. Wu, and V\. Braverman \(2023\)Fixed design analysis of regularization\-based continual learning\.InConference on Lifelong Learning Agents,pp\. 513–533\.Cited by:[Bounded metaplastic preconditioning\.](https://arxiv.org/html/2608.14634#Sx4.SSx3.SSS0.Px2.p6.6)\.
- V\. Lomonaco and D\. Maltoni \(2017\)CORe50: A new dataset and benchmark for continuous object recognition\.InProceedings of the 1st Annual Conference on Robot Learning,Vol\.78,pp\. 17–26\.Cited by:[2nd item](https://arxiv.org/html/2608.14634#Sx5.I3.i2.p1.1)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.Advances in neural information processing systems30\.Cited by:[Evaluation Metrics\.](https://arxiv.org/html/2608.14634#Sx5.SSx1.SSS0.Px3.p1.1)\.
- A\. Ly and P\. Gong \(2025\)Optimization on multifractal loss landscapes explains a diverse range of geometrical and dynamical properties of deep learning\.Nature Communications16\(1\),pp\. 3252\.Cited by:[Limitations\.](https://arxiv.org/html/2608.14634#Sx6.SSx4.SSS0.Px1.p1.1)\.
- J\. Martens and R\. Grosse \(2015\)Optimizing neural networks with kronecker\-factored approximate curvature\.InInternational conference on machine learning,pp\. 2408–2417\.Cited by:[Limitations\.](https://arxiv.org/html/2608.14634#Sx6.SSx4.SSS0.Px1.p1.1)\.
- M\. McCloskey and N\. J\. Cohen \(1989\)Catastrophic interference in connectionist networks: The sequential learning problem\.InPsychology of Learning and Motivation,Vol\.24,pp\. 109–165\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p1.1)\.
- S\. I\. Mirzadeh, M\. Farajtabar, R\. Pascanu, and H\. Ghasemzadeh \(2020\)Understanding the role of training regimes in continual learning\.Advances in Neural Information Processing Systems33\.Cited by:[Continual learning setting\.](https://arxiv.org/html/2608.14634#Sx3.SSx3.SSS0.Px1.p1.6)\.
- S\. Niu, J\. Wu, Y\. Zhang, Y\. Chen, S\. Zheng, P\. Zhao, and M\. Tan \(2022\)Efficient test\-time model adaptation without forgetting\.InInternational conference on machine learning,pp\. 16888–16905\.Cited by:[Continual learningviaparameter regularization](https://arxiv.org/html/2608.14634#Sx2.SSx1.p1.1)\.
- R\. C\. O’Reilly, R\. Bhattacharyya, M\. D\. Howard, and N\. Ketz \(2014\)Complementary learning systems\.Cognitive science38\(6\),pp\. 1229–1248\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p1.1),[Biological metaplasticity](https://arxiv.org/html/2608.14634#Sx2.SSx3.p1.1)\.
- Q\. Pham, C\. Liu, and S\. Hoi \(2021\)Dualnet: continual learning, fast and slow\.Advances in neural information processing systems34,pp\. 16131–16144\.Cited by:[Biological metaplasticity](https://arxiv.org/html/2608.14634#Sx2.SSx3.p1.1)\.
- M\. Rattray, D\. Saad, and S\. Amari \(1998\)Natural gradient descent for on\-line learning\.Physical Review Letters81\(24\),pp\. 5461\.Cited by:[The Fisher information matrix\.](https://arxiv.org/html/2608.14634#Sx3.SSx3.SSS0.Px2.p1.4)\.
- D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. Wayne \(2019\)Experience replay for continual learning\.Advances in Neural Information Processing Systems32\.Cited by:[1st item](https://arxiv.org/html/2608.14634#Sx5.I4.i1.p1.2)\.
- G\. Saha, I\. Garg, and K\. Roy \(2021\)Gradient projection memory for continual learning\.arXiv preprint arXiv:2103\.09762\.Cited by:[Gradient modification and preconditioning](https://arxiv.org/html/2608.14634#Sx2.SSx2.p1.1),[Gradient modification and preconditioning](https://arxiv.org/html/2608.14634#Sx2.SSx2.p2.1)\.
- G\. Saha and K\. Roy \(2023\)Continual learning with scaled gradient projection\.InProceedings of the AAAI conference on artificial intelligence,Vol\.37,pp\. 9677–9685\.Cited by:[Gradient modification and preconditioning](https://arxiv.org/html/2608.14634#Sx2.SSx2.p1.1)\.
- A\. Soen and K\. Sun \(2024\)Trade\-offs of Diagonal Fisher Information Matrix estimators\.Advances in Neural Information Processing Systems37\.Cited by:[The Fisher information matrix\.](https://arxiv.org/html/2608.14634#Sx3.SSx3.SSS0.Px2.p1.3)\.
- G\. M\. van de Ven \(2025\)On the computation of the fisher information in continual learning\.InInternational Conference for Learning Representations \(ICLR\),External Links:[Link](https://iclr.cc/media/iclr-2025/Slides/31332.pdf)Cited by:[The Fisher information matrix\.](https://arxiv.org/html/2608.14634#Sx3.SSx3.SSS0.Px2.p1.3)\.
- J\. J\. Vastola, S\. J\. Gershman, and K\. Rajan \(2025\)Gradient descent as loss landscape navigation: A normative framework for deriving learning rules\.InConference on Neural Information Processing Systems,Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p3.1),[2nd item](https://arxiv.org/html/2608.14634#Sx5.I4.i2.p1.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024\)A comprehensive survey of continual learning: Theory, method and application\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(8\),pp\. 5362–5383\.Cited by:[Introduction](https://arxiv.org/html/2608.14634#Sx1.p2.1)\.
- F\. Zenke, B\. Poole, and S\. Ganguli \(2017\)Continual learning through synaptic intelligence\.InInternational Conference on Machine Learning,pp\. 3987–3995\.Cited by:[Continual learningviaparameter regularization](https://arxiv.org/html/2608.14634#Sx2.SSx1.p1.1)\.