Fast and Slow Variational Continual Learning

# Fast and Slow Variational Continual Learning
Source: [https://arxiv.org/html/2606.24007](https://arxiv.org/html/2606.24007)
Subarnaduti Paul1, Yohan Jung2, Mohammad Emtiyaz Khan3,4,5,Siddharth Swaroop6, Thomas Möllenhoff4,Martin Mundt1 1Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany, 2Department of Computer Science & AI, Jeonbuk National University, Republic of Korea, 3Department of Computer Science, TU Darmstadt, Darmstadt, Germany, 4RIKEN Center for Advanced Intelligence Project, Tokyo, Japan, 5Hessian Center for AI \(hessian\.AI\), Darmstadt, Germany, 6University College London, United Kingdom\. \{spaul, mundtm\}@uni\-bremen\.de

###### Abstract

Continual learning remains a major challenge for modern deep networks, partly because commonly used optimizers lack inherent mechanisms for continual adaptation\. One such natural mechanism is ‘fast and slow adaptation’ to balance stability and plasticity\. This mechanism has deep roots in neuroscience and biology, but there is no consensus on how to best incorporate it in commonly used optimizers\. Here, we show that this can be easily done via the VCL framework, where past posteriors are used as priors in the future\. Our key idea is to incorporate slow adaptation via merging of past posteriors to slow down the drift in the knowledge as learning progresses\. The merged posterior is then used as the prior in the VCL update to implement the fast\-weight updates\. These steps can be seamlessly implemented in the IVON optimizer, whose form and costs are nearly identical to that of Adam\. We call this new optimizer the Continual IVON \(CoVON\) optimizer and show that it not only consistently improves over existing VCL optimizers, but also performs better than other weight\-regularization strategies across domain\-incremental learning, continual pre\-training, and fine\-tuning of large language models\.

## 1Introduction

Modern deep learning systems, especially large language models \(LLMs\), are increasingly deployed in non\-stationary environments, with data and tasks evolving over time\(Silver and Sutton,[2025](https://arxiv.org/html/2606.24007#bib.bib1030)\)\. This is the central problem in continual and lifelong learning\(Thrun,[1998](https://arxiv.org/html/2606.24007#bib.bib895); Hadsellet al\.,[2020](https://arxiv.org/html/2606.24007#bib.bib828)\), where a fundamental challenge is to sequentially update these models without catastrophically interfering with the previously learned knowledge\. Forgetting occurs partly because standard optimizers, such as SGD and Adam, are not designed to handle non\-stationary settings and therefore do not have built\-in mechanisms to preserve old knowledge\. As a result, continual learning is treated as an add\-on problem, typically addressed through external regularization\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib814); Zenkeet al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib873); Leeet al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib886)\), experience or data replay\(Rebuffiet al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib952); Rolnicket al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib1026); Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2606.24007#bib.bib948)\), or isolation of architectural components\(Rusuet al\.,[2016](https://arxiv.org/html/2606.24007#bib.bib887); Yoonet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib879); Mitchellet al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib880)\), see surveys for a full overview\(Mundtet al\.,[2023](https://arxiv.org/html/2606.24007#bib.bib897); Wanget al\.,[2024a](https://arxiv.org/html/2606.24007#bib.bib898)\)\. This is a rather complicated way to deal with forgetting\. Ideally, we want knowledge retention to be inherently rooted in the optimization process itself, such that an optimizer should naturally be able to balance effective adaptation to the current task as well as careful preservation of previous knowledge\.

One way to naturally balance such stability\-plasticity is to use fast and slow adaptation\. Essentially, we want to operate learning on two distinct timescales: a fast component that adapts rapidly to new data, and a slow component that consolidates and preserves previously acquired knowledge\. This is rooted in Complementary Learning Systems \(CLS\) theory from neuroscience\(McClellandet al\.,[1995](https://arxiv.org/html/2606.24007#bib.bib882); O’Reilly and Norman,[2002](https://arxiv.org/html/2606.24007#bib.bib883)\), where the hippocampus is associated with fast adaptation and the neocortex with slow consolidation\. Motivated by this view, many works for continual learning have drawn loose inspiration from CLS theory\. However, instead of incorporating the fast and slow dynamics explicitly in the optimization of a single model, previous works use it implicitly to inform the design of structure or memory\(Dorovataset al\.,[2026](https://arxiv.org/html/2606.24007#bib.bib945)\)\. For example, they focus on the interplay between generative and discriminative models \(neocortex and hippocampus\) with a replay mechanism\(Shinet al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib947); Van de Venet al\.,[2020](https://arxiv.org/html/2606.24007#bib.bib899); Leeet al\.,[2016](https://arxiv.org/html/2606.24007#bib.bib313)\)and the separation into disjoint sets of parameters\(Phamet al\.,[2021](https://arxiv.org/html/2606.24007#bib.bib885); Araniet al\.,[2022](https://arxiv.org/html/2606.24007#bib.bib889); Leeet al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib888)\)\. Little work has been done on directly incorporating such updates into the optimizers for continual learning, and so far, there is no consensus on how to add this feature to commonly used optimizers\.

In this work, we take a different perspective and show that fast and slow adaptation can be naturally implemented within the variational continual learning \(VCL\) framework\(Nguyenet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib806)\)\. This framework uses past posteriors as prior distributions to update the new posterior\. We show that slow\-weight updates can be implemented via “posterior merging” of old posteriors, essentially slowing down the drift in the knowledge as new tasks are observed in an uncertainty\-weighted manner\. With this slow\-moving prior, the fast\-weight updates are implemented using the VCL objective\. For this, we use a recently\-proposed variational training algorithm, called Improved Variational Online Newton optimizer\(Shenet al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib17)\), which uses diagonal Gaussian posteriors and whose form and costs are nearly identical to that of Adam\. We call this new algorithm Continual IVON \(CoVON\)\. Here, we show the benefits of CoVON for continual learning over existing VCL and other weight\-regularization strategies through various experiments across classic domain\-incremental learning, continual pre\-training, and fine\-tuning of large language models\.

![Refer to caption](https://arxiv.org/html/2606.24007v1/x1.png)Figure 1:CoVON balances stability and plasticity in Variational Continual Learning through fast and slow adaptation steps in a single model, where the VCL block denotes the variational update in[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)\. The slow\-moving posterior \(red\) slowly consolidates information from the previous posterior and the new fast posterior \(blue\), merging the models together\. The fast posterior is trained on the current data using the previous slow posterior as a new prior in a Bayesian continual update\.
## 2Background and Related Work

Continual learning algorithms for neural networks must balance complementary properties:*plasticity*, the ability to adapt rapidly to new tasks without losing the capacity to learn; and*stability*, the ability to preserve knowledge acquired in previous tasks without catastrophic forgetting\. A natural mechanism to balance this is ‘fast and slow adaptation’, where plasticity is maintained through fast\-moving elements, and stability is maintained in the slow\-moving parts\.

However, most popular continual learning methods attempt to balance these properties using external mechanisms added on top of the standard training pipeline\. These include regularization\-based, replay/memory\-based, and architecture\-based approaches, see surveys for a full overview\(Mundtet al\.,[2023](https://arxiv.org/html/2606.24007#bib.bib897); Wanget al\.,[2024a](https://arxiv.org/html/2606.24007#bib.bib898)\)\. These approaches attempt to correct for a broader limitation of current optimization algorithms, such as SGD and AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2606.24007#bib.bib58)\): while effective to learn the task at hand, they do not explicitly preserve knowledge over time, offering limited balance to the stability\-plasticity tradeoff, and eventually succumbing to catastrophic forgetting\(Mirzadehet al\.,[2020](https://arxiv.org/html/2606.24007#bib.bib1010)\)\. Other recent works introduce simple methods that do not have external mechanisms and are similar to existing deep\-learning optimizers:Liet al\.\([2025](https://arxiv.org/html/2606.24007#bib.bib1017)\)proposes reusing AdamW’s scale vector for continual learning, but this can be inaccurate for larger batch sizes\. Utility\-Perturbed Gradient Descent \(UPGD\)\(Elsayed and Mahmood,[2024](https://arxiv.org/html/2606.24007#bib.bib1029)\)introduces a continual optimizer, but it relies on second\-derivative computations, which are difficult to scale to modern, larger\-scale neural networks\. In this work, we introduce a simple standard training pipeline that computes second\-order estimates in an online fashion within the optimization steps without significant overhead, enabling continuous adaptation at scale\.

Instead of finding external mechanisms that may or may not balance fast adaptation and slow consolidation well, we can instead directly use learning dynamics that operate at the two different paces\. This has been studied in neuroscience and is used to motivate machine learning algorithms\. The neuroscience theory of complementary learning systems \(CLS\) byMcClellandet al\.\([1995](https://arxiv.org/html/2606.24007#bib.bib882)\)argues that the human brain relies on fast and slow learning through the hippocampus and neocortex\. Continual learning works inspired by this CLS view, however, use heuristic methods\. They either explicitly use two sets of weights or dual\-system mechanisms; these include CLS\-ER\(Araniet al\.,[2022](https://arxiv.org/html/2606.24007#bib.bib889)\), Dual\-Nets\(Phamet al\.,[2021](https://arxiv.org/html/2606.24007#bib.bib885)\), RL\-based CLS\(Leeet al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib888)\), and MERLIN\(Joseph and Balasubramanian,[2020](https://arxiv.org/html/2606.24007#bib.bib884)\)\. In contrast, we naturally incorporate fast and slow updates within the optimization process of a single model\.

Another line of work is inspired by biological views on fast and slow updates of synapses\(Hinton and Plaut,[1987](https://arxiv.org/html/2606.24007#bib.bib825)\), using fast weights to deblur compressed memories stored in slow\-moving weights\. This perspective was later revisited to speed up optimization in deep learning\(Zhanget al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib698)\), for sequence modeling\(Baet al\.,[2016](https://arxiv.org/html/2606.24007#bib.bib890)\), for meta\-learning\(Finnet al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib1034); Nicholet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib1033)\), and recently, for continual learning\(Abbeset al\.,[2025](https://arxiv.org/html/2606.24007#bib.bib1035)\)\. Our work presents a variational Bayesian formulation of such views, along with a scalable implementation for Gaussian posteriors, which gives new curvature\-weighted variants of these methods\.

Here, our goal is to show how we can naturally implement fast and slow adaptation in the VCL framework\(Nguyenet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib806)\)\. Unlike standard training methods that aim to find a parameter𝜽\\boldsymbol\{\\theta\}by minimizing a loss functionℓt​\(𝜽\)=∑i∈𝒟tℓi​\(𝜽\)/\|𝒟t\|\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)=\\sum\_\{i\\in\\mbox\{$\{\\cal D\}$\}\_\{t\}\}\\ell\_\{i\}\(\\boldsymbol\{\\theta\}\)/\|\\mbox\{$\{\\cal D\}$\}\_\{t\}\|, VCL aims to find a distributionq​\(𝜽\)q\(\\boldsymbol\{\\theta\}\)by optimizing the Kullback\-Leibler divergence objective given a past posteriorqt−1q\_\{t\-1\}as prior at tasktt\. The optimization proceeds as follows,

qt​\(𝜽\)←arg⁡minq∈𝒬⁡λt​𝔼q​\(𝜽\)​\[ℓt​\(𝜽\)\]\+𝔻KL​\(q​\(𝜽\)∥qt−1​\(𝜽\)\)\.q\_\{t\}\(\\boldsymbol\{\\theta\}\)\\leftarrow\\arg\\min\_\{q\\in\\mathcal\{Q\}\}~\\lambda\_\{t\}\\mathbb\{E\}\_\{q\(\\text\{$\\boldsymbol\{\\theta\}$\}\)\}\[\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\]\+\\mathbb\{D\}\_\{\\text\{KL\}\}\(q\(\\boldsymbol\{\\theta\}\)\\,\\\|\\,q\_\{t\-1\}\(\\boldsymbol\{\\theta\}\)\)\.\(1\)It can be shown that by making certain approximations, solutions of standard optimizers can be recovered as special cases of this objective\. Contrary to the original VCL work, we introduce a scaling factorλt\>0\\lambda\_\{t\}\>0, which recovers VCL when set to the number of data examples\|𝒟t\|\|\\mbox\{$\{\\cal D\}$\}\_\{t\}\|\. Although VCL can perform decently in continual learning due to the Bayesian update structure, it does not have a sense of fast and slow updating: only one set of weights are updated using the same optimization procedure\. This means that there is no explicit tradeoff between fast adaptation and slow consolidation\.Nguyenet al\.\([2018](https://arxiv.org/html/2606.24007#bib.bib806)\)argued that VCL satisfies this automatically due to the Bayesian updates; however, in practice, VCL’s performance can deteriorate rapidly due to approximations required for neural networks \(such as mean\-field Gaussian approximations\)\. In our method, we will introduce a separate slow update to naturally include fast and slow dynamics\. We next describe how we build on VCL\.

## 3Fast and Slow VCL via Continual Variational Online Newton \(CoVON\)

We now show how we naturally implement fast and slow mechanisms within the VCL framework, where the fast adaptation controls the plasticity on the new task and the slow adaptation consolidates the prior knowledge acquired from the past tasks\. We first describe the general framework in[Sec\.˜3\.1](https://arxiv.org/html/2606.24007#S3.SS1)and then present two scalable implementations using diagonal Gaussian posteriors: a naive approximation that uses AdamW, and then our full CoVON algorithm \(that uses IVON\)\.

### 3\.1Fast and Slow Variational Continual Learning

We propose to generalize VCL into a two\-stage fast and slow update \(see also[Fig\.˜1](https://arxiv.org/html/2606.24007#S1.F1)\), where \(1\) a fast adapted per\-task posterior is learned using[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1), and then \(2\) it is consolidated into a slow\-moving outer posterior obtained through merging of past information with new information\. For fast adaptation, we use[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1), optimizing forqtfastq\_\{t\}^\{\\textrm\{fast\}\}while setting the prior to beqt−1slowq\_\{t\-1\}^\{\\textrm\{slow\}\}\.

Slow Adaptation:After learning the fast posteriorqtfastq\_\{t\}^\{\\textrm\{fast\}\}on the current task, we consolidate it into a slow\-moving outer posterior by merging it with the previous slow\-moving posteriorqt−1slowq\_\{t\-1\}^\{\\textrm\{slow\}\}with the newly learned posteriorqtfastq\_\{t\}^\{\\textrm\{fast\}\}:

qtslow​\(𝜽\)∝qt−1slow​\(𝜽\)​\(qtfast​\(𝜽\)qt−1slow​\(𝜽\)\)γ\.q\_\{t\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\propto q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\left\(\\frac\{q\_\{t\}^\{\\textrm\{fast\}\}\(\\boldsymbol\{\\theta\}\)\}\{q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\}\\right\)^\{\\gamma\}\.\(2\)This choice of update enables us to strike a better balance between the past and the future, in a similar fashion as other online learning methods\. This can be shown by noting a property of the solutions obtained by solving[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)\. Essentially, as shown byKhan \([2025](https://arxiv.org/html/2606.24007#bib.bib819), Eq\. 4\)\(and many other works before\), a solution of[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)has the following form in terms of the*site*functionℓ^t​\(𝜽\)\\hat\{\\ell\}\_\{t\}\(\\boldsymbol\{\\theta\}\),

qtfast​\(𝜽\)∝qt−1slow​\(𝜽\)​exp⁡\[−λt​ℓ^t​\(𝜽\)\],q\_\{t\}^\{\\textrm\{fast\}\}\(\\boldsymbol\{\\theta\}\)\\propto q^\{\\textrm\{slow\}\}\_\{t\-1\}\(\\boldsymbol\{\\theta\}\)\\exp\\left\[\-\\lambda\_\{t\}\\hat\{\\ell\}\_\{t\}\(\\boldsymbol\{\\theta\}\)\\right\],\(3\)where the site functionℓ^t​\(𝜽\)\\hat\{\\ell\}\_\{t\}\(\\boldsymbol\{\\theta\}\)summarizes the effect of the task lossℓt\\ell\_\{t\}on our learned distributionqtfastq\_\{t\}^\{\\textrm\{fast\}\}\(see[Sec\.˜A\.1](https://arxiv.org/html/2606.24007#A1.SS1)for derivation\)\. Substituting this in[Eq\.˜2](https://arxiv.org/html/2606.24007#S3.E2), we can write the slow update posterior as

qtslow​\(𝜽\)∝qt−1slow​\(𝜽\)​exp⁡\[−γ​λt​ℓ^t​\(𝜽\)\]∝q0slow​\(𝜽\)​∏i=1texp⁡\[−γ​λi​ℓ^i​\(𝜽\)\]\.q\_\{t\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\propto q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\exp\\left\[\-\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}\\gamma\}\\lambda\_\{t\}\\hat\{\\ell\}\_\{t\}\(\\boldsymbol\{\\theta\}\)\\right\]\\,\\,\\,\\propto q\_\{0\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\prod\_\{i=1\}^\{t\}\\exp\\left\[\-\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}\\gamma\}\\lambda\_\{i\}\\hat\{\\ell\}\_\{i\}\(\\boldsymbol\{\\theta\}\)\\right\]\.\(4\)The second expression is obtained by recursively applying the form\. The expression shows that the slow update is down\-weighting all the past site functions \(byγ\\gammahighlighted in red\)\. This has a net effect to slow down the learning, essentially reversing the effect of a largerλt\\lambda\_\{t\}used in the fast update\. Such exponentially weighted aggregation\(Vovk,[1990](https://arxiv.org/html/2606.24007#bib.bib1037)\)is a common strategy in online learning and can recover optimal guarantees by changing the posterior form\(van der Hoevenet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib812)\)\.

Our updates also have a similar property, where changing the form of the posterior can recover previous proposals for fast and slow adaptation\. For instance, for simple isotropic Gaussian distributions with fixed\-variance,[Eq\.˜2](https://arxiv.org/html/2606.24007#S3.E2)recovers shortest descent\(Hinton and Plaut,[1987](https://arxiv.org/html/2606.24007#bib.bib825)\), the lookahead optimizer\(Zhanget al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib698)\)and first\-order meta\-learning methods such as Reptile\(Nicholet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib1033)\)\. We would expect that using more expressive posteriors improves the quality of this knowledge consolidation step\. In the next subsection, we will see that using Gaussians with diagonal variance, one obtains curvature or uncertainty\-weighted variants of these methods\.

#### Fast adaptation\.

Given the slow posteriorqt−1slowq\_\{t\-1\}^\{\\textrm\{slow\}\}as a stabilizing prior, we optimize for the fast posteriorqtfastq\_\{t\}^\{\\textrm\{fast\}\}by solving[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)withqt−1slowq\_\{t\-1\}^\{\\textrm\{slow\}\}in place ofqt−1q\_\{t\-1\}:

qtfast​\(𝜽\)←arg⁡minq∈𝒬⁡λt​𝔼q​\(𝜽\)​\[ℓt​\(𝜽\)\]\+𝔻KL​\(q​\(𝜽\)∥qt−1slow​\(𝜽\)\)\.q\_\{t\}^\{\\textrm\{fast\}\}\(\\boldsymbol\{\\theta\}\)\\leftarrow\\arg\\min\_\{q\\in\\mathcal\{Q\}\}~\\lambda\_\{t\}\\mathbb\{E\}\_\{q\(\\boldsymbol\{\\theta\}\)\}\[\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\]\+\\mathbb\{D\}\_\{\\text\{KL\}\}\(q\(\\boldsymbol\{\\theta\}\)\\,\\\|\\,q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\)\.\(5\)Hereλt\>0\\lambda\_\{t\}\>0acts as a tunable learning\-rate\-like parameter controlling the balance between fitting the new task and retaining the slow posterior\. While a natural default isλt=\|𝒟t\|\\lambda\_\{t\}=\|\\mathcal\{D\}\_\{t\}\|, this is not a natural choice in online settings\(van der Hoevenet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib812); Chérief\-Abdellatifet al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib1038)\), and we therefore treat it as a hyperparameter\.

### 3\.2Diagonal\-Gaussian Implementation and the Proposed CoVON Method

We now derive two practical variants of the VCL algorithm\. We start with a simple variant of VCL that reuses curvature estimates obtained from training with AdamW, and then present our more faithful CoVON algorithm, which also uses the fast and slow model merging updates\.

To derive the Adam variant, we first note that any Gaussian solution of formqt​\(𝜽\)=𝒩​\(𝜽\|𝐦t,𝚺t\)q\_\{t\}\(\\boldsymbol\{\\theta\}\)=\\mbox\{$\{\\cal N\}$\}\(\\boldsymbol\{\\theta\}\\,\|\\,\\mathbf\{m\}\_\{t\},\\mathbf\{\\Sigma\}\_\{t\}\)to[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)satisfies the following optimality conditions,

𝐒t−1​\(𝐦t−𝐦t−1\)=−λt​𝔼qt​\(𝜽\)​\[∇ℓt​\(𝜽\)\],\\displaystyle\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\(\\mathbf\{m\}\_\{t\}\-\\mathbf\{m\}\_\{t\-1\}\)=\-\\lambda\_\{t\}\\mathbb\{E\}\_\{q\_\{t\}\(\\text\{$\\boldsymbol\{\\theta\}$\}\)\}\[\\nabla\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\],𝐒t=𝐒t−1\+λt​𝔼qt​\(𝜽\)​\[∇2ℓt​\(𝜽\)\],\\displaystyle\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\}=\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\mathbb\{E\}\_\{q\_\{t\}\(\\text\{$\\boldsymbol\{\\theta\}$\}\)\}\[\\nabla^\{2\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\],\(6\)where𝐒t=𝚺t−1\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\}=\\mathbf\{\\Sigma\}\_\{t\}^\{\-1\}is the precision matrix \(inverse covariance\)\. This is obtained by setting the gradient with respect to mean and precision in[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)to zero \(we have dropped superscripts ‘fast’ and ‘slow’\), see\(Khan and Rue,[2023](https://arxiv.org/html/2606.24007#bib.bib743), Sec\. 3\.1\)\. We present a self\-contained derivation of this in[Sec\.˜A\.2](https://arxiv.org/html/2606.24007#A1.SS2)\.

Performing an approximation to the expectation using a single sample at the distribution’s mean in[Eq\.˜6](https://arxiv.org/html/2606.24007#S3.E6)leads to the following approximate VCL update as shown below:

𝜽t←arg⁡min𝜽⁡ℓt​\(𝜽\)\+12​λt​‖𝜽−𝜽t−1‖𝐒t−12\\boldsymbol\{\\theta\}\_\{t\}\\leftarrow\\arg\\min\_\{\\text\{$\\boldsymbol\{\\theta\}$\}\}\\,\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\+\\frac\{1\}\{2\\lambda\_\{t\}\}\\\|\\boldsymbol\{\\theta\}\-\\boldsymbol\{\\theta\}\_\{t\-1\}\\\|\_\{\\text\{$\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}$\}\}^\{2\}\(7\)
𝐒t←𝐒t−1\+λt​∇2ℓt​\(𝜽t\)\.\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\}\\leftarrow\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\nabla^\{2\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\_\{t\}\)\.\(8\)
To derive this, we have rewritten the first optimality condition as an optimization problem and switched the notation from𝐦\\mathbf\{m\}to𝜽\\boldsymbol\{\\theta\}to highlight the approximation\. This yields the precision for the next task𝐒t=𝐒0\+∑k=1tλk​∇2ℓk​\(𝜽k\)\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\}=\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{0\}\+\\sum\_\{k=1\}^\{t\}\\lambda\_\{k\}\\nabla^\{2\}\\ell\_\{k\}\(\\boldsymbol\{\\theta\}\_\{k\}\)as an accumulation of past Hessians, where𝐒0\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{0\}is a prior chosen for the first task, typically𝐒0=λ1​δ​𝐈\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{0\}=\\lambda\_\{1\}\\delta\\mbox\{$\\mbox\{$\\mathbf\{I\}$\}$\}corresponds to weight\-decay with strengthδ\>0\\delta\>0\.

In practice, one further approximates the Hessian using the squared gradients:

∇2ℓt​\(𝜽\)≈diag​\(𝐡\),\\displaystyle\\nabla^\{2\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\\approx\\text\{diag\}\(\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\),𝐡=𝔼\(𝐱,𝐲\)∼𝒟t​\[\(∇𝜽ℒ​\(f​\(𝜽,𝐱\),𝐲\)\)2\]\.\\displaystyle\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}=\\mathbb\{E\}\_\{\(\\text\{\\mbox\{$\\mbox\{$\\mathbf\{x\}$\}$\}\},\\text\{\\mbox\{$\\mbox\{$\\mathbf\{y\}$\}$\}\}\)\\sim\\mathcal\{D\}\_\{t\}\}\\big\[\(\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\(f\(\\boldsymbol\{\\theta\},\\mbox\{$\\mbox\{$\\mathbf\{x\}$\}$\}\),\\mbox\{$\\mbox\{$\\mathbf\{y\}$\}$\}\)\)^\{2\}\\big\]\.\(9\)The resulting update affects[Eq\.˜7](https://arxiv.org/html/2606.24007#S3.E7)by changing the regularizer term to become12​∑t=1T\(𝜽−𝜽t\)⊤​diag​\(𝐡t\)​\(𝜽−𝜽t\)\\smash\{\\mbox\{$\\frac\{1\}\{2\}$\}\\sum\_\{t=1\}^\{T\}\(\\boldsymbol\{\\theta\}\-\\boldsymbol\{\\theta\}\_\{t\}\)^\{\\top\}\\mathrm\{diag\}\(\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\_\{t\}\)\(\\boldsymbol\{\\theta\}\-\\boldsymbol\{\\theta\}\_\{t\}\)\}for each tasktt\. This is the regularizer used in classical works such as Elastic Weight Consolidation \(EWC\)\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib814)\)and EWC\*\(Huszár,[2018](https://arxiv.org/html/2606.24007#bib.bib813)\)\. Both methods approximate posterior precision through Hessians at a single point, but do so only after training each task, requiring an extremely expensive additional full pass over the dataset to compute[Eq\.˜8](https://arxiv.org/html/2606.24007#S3.E8)together with[Eq\.˜9](https://arxiv.org/html/2606.24007#S3.E9)in a post\-hoc fashion\. As a cheap approximation, we propose to recycle the squared\-gradient accumulator in AdamW, and we refer to this method as Ada\-Reg\. The pseudo\-code of our EWC\-like and Ada\-Reg methods are provided in[Sec\.˜A\.3](https://arxiv.org/html/2606.24007#A1.SS3)\.

Algorithm 1CoVON: Fast and Slow Variational Continual Learning1:Init:

𝐦←\(NN\-weight init\),𝐦0←0,𝐬0←prior\-precision\\mathbf\{m\}\\leftarrow\(\\text\{NN\-weight init\}\),\\;\\mathbf\{m\}\_\{0\}\\leftarrow 0,\\;\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{0\}\\leftarrow\\text\{prior\-precision\}
2:Init:

β1,β2∈\(0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\(0,1\),

λt\\lambda\_\{t\}for each task

tt, Hessian init

𝐡0=h0⋅𝟏\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\_\{0\}=h\_\{0\}\\cdot\\mathbf\{1\},learning rates

α\\alpha,

γ\\gamma
3:for

t=1​…​Tt=1\\ldots Tdo

4:

𝐠←0\\mathbf\{g\}\\leftarrow 0,

𝐡←h0\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow h\_\{0\},

5:for

i=1,2,…i=1,2,\\ldotsdo

6:Sample

𝜽∼𝒩​\(𝜽\|𝐦,𝝈2\)\\boldsymbol\{\\theta\}\\sim\\mbox\{$\{\\cal N\}$\}\(\\boldsymbol\{\\theta\}\\,\|\\,\\mathbf\{m\},\\mbox\{$\\mbox\{$\\boldsymbol\{\\sigma\}$\}$\}^\{2\}\)with

𝝈2=1/\(λt​𝐡\+𝐬t−1\)\\mbox\{$\\mbox\{$\\boldsymbol\{\\sigma\}$\}$\}^\{2\}=1/\(\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\)
7:

𝐠^←∇^​ℓt​\(𝜽\)\\widehat\{\\mathbf\{g\}\}\\leftarrow\\widehat\{\\nabla\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)
8:

^​𝐡←𝐠^​\(𝜽−𝐦\)/𝝈2\\widehat\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow\\widehat\{\\mathbf\{g\}\}\(\\boldsymbol\{\\theta\}\-\\mathbf\{m\}\)/\\mbox\{$\\mbox\{$\\boldsymbol\{\\sigma\}$\}$\}^\{2\}\}​Fast VCL Updateusing IVON &Slow\-Updated Prior\\left\.\\begin\{array\}\[\]\{@\{\}c@\{\}\}\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\end\{array\}\\color\[rgb\]\{0\.6171875,0\.1328125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6171875,0\.1328125,0\.2734375\}\\right\\\}\\color\[rgb\]\{0\.6171875,0\.1328125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6171875,0\.1328125,0\.2734375\}\\begin\{tabular\}\[\]\{l\}Fast VCL Update\\\\ using IVON \\&\\\\ Slow\-Updated Prior\\end\{tabular\}

9:

𝐠←β1​𝐠\+\(1−β1\)​𝐠^\\mathbf\{g\}\\leftarrow\\beta\_\{1\}\\mathbf\{g\}\+\(1\-\\beta\_\{1\}\)\\widehat\{\\mathbf\{g\}\},

𝐠¯←𝐠/\(1−β1i\)\\bar\{\\mathbf\{g\}\}\\leftarrow\\mathbf\{g\}/\(1\-\\beta\_\{1\}^\{i\}\)
10:

𝐡←β2​𝐡\+\(1−β2\)​^​𝐡\+12​\(1−β2\)2​\(𝐡−^​𝐡\)2/\(𝐡\+λt−1​𝐬t−1\)\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow\\beta\_\{2\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\(1\-\\beta\_\{2\}\)\\widehat\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\\mbox\{$\\frac\{1\}\{2\}$\}\(1\-\\beta\_\{2\}\)^\{2\}\(\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\-\\widehat\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\)^\{2\}/\(\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\\lambda\_\{t\}^\{\-1\}\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\)
11:

𝐦←𝐦−α​\(𝐠¯\+λt−1​𝐬t−1​\(𝐦−𝐦t−1\)\)/\(𝐡\+λt−1​𝐬t−1\)\\mathbf\{m\}\\leftarrow\\mathbf\{m\}\-\\alpha\(\\bar\{\\mathbf\{g\}\}\+\{\\color\[rgb\]\{0\.6171875,0\.1328125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6171875,0\.1328125,0\.2734375\}\\displaystyle\\lambda\_\{t\}^\{\-1\}\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\)\}\)/\(\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\\lambda\_\{t\}^\{\-1\}\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\)
12:endfor

13:

𝐬t←\(1−γ\)​𝐬t−1\+γ​\(𝐬t−1\+λt​𝐡\)\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\}\\leftarrow\(1\-\\gamma\)\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\gamma\(\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\)
14:

𝐦t←\[\(1−γ\)​𝐬t−1​𝐦t−1\+γ​\(𝐬t−1\+λt​𝐡\)​𝐦\]/𝐬t\\mathbf\{m\}\_\{t\}\\leftarrow\\left\[\(1\-\\gamma\)\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\\mathbf\{m\}\_\{t\-1\}\+\\gamma\(\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\)\\mathbf\{m\}\\right\]/\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\}\}​Slow Update\\left\.\\begin\{array\}\[\]\{@\{\}c@\{\}\}\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\\\ \{\}\\hfil\\end\{array\}\\color\[rgb\]\{0\.6171875,0\.1328125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6171875,0\.1328125,0\.2734375\}\\right\\\}\\color\[rgb\]\{0\.6171875,0\.1328125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6171875,0\.1328125,0\.2734375\}\\begin\{tabular\}\[\]\{l\}Slow Update\\end\{tabular\}

15:

𝐦←𝐦t\\mathbf\{m\}\\leftarrow\\mathbf\{m\}\_\{t\}
16:endfor

Instead of approximating the expectation with a single sample at the mean, our method CoVON uses the Improved Variational Online Newton method \(IVON\), seeShenet al\.\([2024](https://arxiv.org/html/2606.24007#bib.bib17), Alg\. 1\)to directly solve[Eq\.˜6](https://arxiv.org/html/2606.24007#S3.E6)\. CoVON also uses the additional proposed slow update\. Algorithm[1](https://arxiv.org/html/2606.24007#alg1)summarizes the final method, and in the following, we will describe the individual components\.

#### CoVON’s Inner Loop \(Fast Adaptation\)\.

CoVON closely follows the Ada\-Reg pseudo\-code \(see[Sec\.˜A\.3](https://arxiv.org/html/2606.24007#A1.SS3)\), but solves the variational problem[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)\. This is achieved by sampling parameters from the current posteriorqtq\_\{t\}inline 4of Algorithm[1](https://arxiv.org/html/2606.24007#alg1), and evaluating gradients at the sampled parameters inline 5\. This stochastic parameter sampling allows CoVON to propagate posterior uncertainty through the optimization trajectory and estimate curvature information rather than collapsing prematurely to a single solution\. This noise is also crucial for controlling the model’s plasticity, with noisy weights corresponding to plastic ones and static weights to stable ones\. The noise is also used to estimate the Hessian𝐡\\mathbf\{h\}in[Eq\.˜9](https://arxiv.org/html/2606.24007#S3.E9)via the reparametrization trick in line 6, giving curvature estimates for free\. To retain the posterior estimate from the previous tasks, which is essential for continual learning, we augment each parameter update step with a quadratic regularizer centered at the previous posterior mean𝐦t−1\\mathbf\{m\}\_\{t\-1\}in line 9\. The regularizer is also scaled by task\-specific precision𝐬t−1\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}, which captures curvature information updated online across tasks\. Together, this implements the KL\-term in[Eq\.˜1](https://arxiv.org/html/2606.24007#S2.E1)\.

#### CoVON’s Outer Loop \(Slow Consolidation\)\.

Instead of simply accumulating the Hessians in the outer loop as suggested by[Eq\.˜6](https://arxiv.org/html/2606.24007#S3.E6)and done in Ada\-Reg \([Sec\.˜A\.3](https://arxiv.org/html/2606.24007#A1.SS3)\), CoVON now uses the merging from[Eq\.˜2](https://arxiv.org/html/2606.24007#S3.E2), which in our Gaussian case takes the following form:

𝐬t←\(1−γ\)​𝐬t−1\+γ​\(𝐬t−1\+λt​𝐡\),𝐦t←\[\(1−γ\)​𝐬t−1​𝐦t−1\+γ​\(𝐬t−1\+λt​𝐡\)​𝐦\]/𝐬t,\\displaystyle\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\}\\leftarrow\(1\-\\gamma\)\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\gamma\(\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\),\\quad\\mathbf\{m\}\_\{t\}\\leftarrow\\left\[\(1\-\\gamma\)\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\\mathbf\{m\}\_\{t\-1\}\+\\gamma\(\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\)\\mathbf\{m\}\\right\]/\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\},\(10\)
where𝐡\\mathbf\{h\}is the online Hessian estimated during the inner loop\. This avoids any expensive post\-hoc Fisher or Hessian evaluations as in EWC\. This can also be well achieved without any task boundaries, where one may just need to set a threshold in the number of optimization steps or track the loss curve to determine the point of convergence as an accumulation point\.

Forγ=1\\gamma=1, the above simplifies to[Eq\.˜6](https://arxiv.org/html/2606.24007#S3.E6)which we use in Ada\-Reg and EWC\. A choice ofγ<1\\gamma<1favors retaining old information, and consolidates the knowledge from the inner fast adaptation into the slow\-moving outer posterior\. Such ideas have existed for a long time, seeHinton and Plaut \([1987](https://arxiv.org/html/2606.24007#bib.bib825)\)\. The idea is that the fast moving set of weights in the inner loop attends to the new tasks and*deblurs*old memories stored in the slow moving weights \(outer loop\)\. Knowledge in the fast moving weights then gets merged or consolidated into a set of slow moving weights in the outer loop that stores the long term memories\. The updates[Eq\.˜10](https://arxiv.org/html/2606.24007#S3.E10)yield a curvature\-weighted uncertainty\-aware generalization of such fast\-and\-slow learning algorithms\(Hinton and Plaut,[1987](https://arxiv.org/html/2606.24007#bib.bib825); Nicholet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib1033); Zhanget al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib698)\)\. Finally, we note that a choice ofγ\>1\\gamma\>1is also possible and corresponds to an overrelaxation which is frequently used in optimization\(Eckstein and Bertsekas,[1992](https://arxiv.org/html/2606.24007#bib.bib736)\), but appears less useful for continual learning\. The updates in[Eq\.˜10](https://arxiv.org/html/2606.24007#S3.E10)are also an instance of uncertainty\-guided model merging\(Daheimet al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib773)\), which augments regular model\-merging with a weighting\. The intuition is that weights with high uncertainty \(noisy, plastic weights which can take on any value\) should move more in the model merging than weights which have little uncertainty and are static\.

## 4Experiments: Fast & Slow Variational Continual Learning with CoVON

We now empirically corroborate our fast and slow VCL framework and its instantiation through CoVON\. We begin by revisiting VCL through the lens of fast and slow adaptation in Section[4\.1](https://arxiv.org/html/2606.24007#S4.SS1), demonstrating that CoVON improves over existing VCL optimizers and ablating key design choices in the process\. We then run prevalent domain incremental benchmarks in Section[4\.2](https://arxiv.org/html/2606.24007#S4.SS2), demonstrating that CoVON performs favorably against other \(non\-VCL\-based\) SOTA CL algorithms\. Finally, we validate CoVON’s scalability and robustness in more challenging and realistic scenarios\. Here, we highlight CoVON’s aptitude for continual pre\-training across multilingual data \([Sec\.˜4\.3](https://arxiv.org/html/2606.24007#S4.SS3)\) in LLMs and knowledge preservation in fine\-tuning on common reasoning benchmark \([Sec\.˜4\.4](https://arxiv.org/html/2606.24007#S4.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2606.24007v1/x2.png)\\phantomcaption

UpdatePriorOuter loop𝜸\\boldsymbol\{\\gamma\}Avg AccFast onlyFastw/o−\-=1=140\.23±2\.9840\.23\\pm 2\.98Fastw/−\-=1=177\.03±1\.0377\.03\\pm 1\.03Fast & slowFast & sloww/EMA=1=189\.26±0\.2489\.26\\pm 0\.24Fast & sloww/EMA<1<189\.56±0\.2589\.56\\pm 0\.25Fast & sloww/Full VCL=1=190\.32±0\.2790\.32\\pm 0\.27Fast & sloww/Full VCL<1<192\.12±0\.3592\.12\\pm 0\.35

\\phantomcaption

Figure 2:\(a\) Comparison of CoVON against recent VCL approaches on a Sequential Permuted MNIST\-Hard benchmark\. Here, CoVON\-NoM denotes the variant without posterior merging \(this is then the same as VCL using natural\-gradient updates\), whereas CoVON includes this slow update, as shown in[Eq\.˜10](https://arxiv.org/html/2606.24007#S3.E10)\. CoVON\-NoM already outperforms the strongest baseline TD\-VCL, and the merging\-based update further improves the model performance, demonstrating the importance of the interplay between fast and slow components\. \(b\) Ablation study of the fast and slow components of CoVON\. Considering only fast IVON\-like updates unsurprisingly leads to catastrophic forgetting, which is partially alleviated by adding a prior\. On the contrary, using both fast and slow mechanisms offers a significant improvement, where the full fast and slow VCL mechanism of CoVON outperforms a curvature\-unaware EMA\-based merging of past posteriors\.Table 1:Comparison of CoVON against non\-VCL SOTA baselines on domain\-incremental benchmarks\. We report the average accuracyATA\_\{T\}and the average forgettingFTF\_\{T\}on DomainNet, CDDB\-Hard, and CORe50 under two backbone settings: a standard ViT\-B/16 and CLIP with a pretrained ViT\-B/16 vision encoder\. Best results per section are highlighted inbold\. CoVON consistently improves both retention and final accuracy over prior methods and further outperforms the no\-merging variant \(CoVON\-NoM\), demonstrating the benefit of slow\-merged posterior consolidation\.Continual learning MethodsDomainNetCDDB\-HardCORe50ATA\_\{T\}\(↑\\uparrow\)FTF\_\{T\}\(↓\\downarrow\)ATA\_\{T\}\(↑\\uparrow\)FTF\_\{T\}\(↓\\downarrow\)ATA\_\{T\}\(↑\\uparrow\)Model: Vision Transformer \(ViT\-B/16\)DyToxDouillardet al\.\([2022](https://arxiv.org/html/2606.24007#bib.bib867)\)62\.94–86\.21−1\.55\-1\.5579\.21EWC\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib814)\)47\.62±0\.947\.62\{\\pm 0\.9\}−1\.93\-1\.9350\.59±1\.150\.59\{\\pm 1\.1\}−27\.62\-27\.6274\.82±1\.274\.82\{\\pm 1\.2\}EWC\*\(Huszár,[2018](https://arxiv.org/html/2606.24007#bib.bib813)\)51\.01±0\.851\.01\{\\pm 0\.8\}−1\.79\-1\.7970\.94±0\.970\.94\{\\pm 0\.9\}−1\.54\-1\.5485\.35±1\.285\.35\{\\pm 1\.2\}LwF\(Li and Hoiem,[2017](https://arxiv.org/html/2606.24007#bib.bib28)\)45\.01±0\.445\.01\{\\pm 0\.4\}−1\.99\-1\.9960\.94±0\.360\.94\{\\pm 0\.3\}−16\.24\-16\.2475\.35±0\.375\.35\{\\pm 0\.3\}L2P\(Wanget al\.,[2022c](https://arxiv.org/html/2606.24007#bib.bib30)\)40\.15±0\.040\.15\{\\pm 0\.0\}−2\.25\-2\.2561\.08±0\.061\.08\{\\pm 0\.0\}−9\.23\-9\.2378\.45±0\.078\.45\{\\pm 0\.0\}DualPrompt\(Wanget al\.,[2022b](https://arxiv.org/html/2606.24007#bib.bib29)\)43\.79±0\.0343\.79\{\\pm 0\.03\}−2\.03\-2\.0364\.80±0\.0164\.80\{\\pm 0\.01\}−8\.74\-8\.7480\.68±0\.0280\.68\{\\pm 0\.02\}CODA\-P\(Smithet al\.,[2023](https://arxiv.org/html/2606.24007#bib.bib31)\)54\.29±0\.2054\.29\{\\pm 0\.20\}−1\.34\-1\.3473\.15±0\.1073\.15\{\\pm 0\.10\}−0\.99\-0\.9985\.31±0\.1085\.31\{\\pm 0\.10\}S\-iPrompts\(Wanget al\.,[2022a](https://arxiv.org/html/2606.24007#bib.bib33)\)50\.62±0\.0550\.62\{\\pm 0\.05\}−2\.85\-2\.8574\.51±0\.0274\.51\{\\pm 0\.02\}−1\.30\-1\.3083\.50±0\.0183\.50\{\\pm 0\.01\}PINA\(Wanget al\.,[2024b](https://arxiv.org/html/2606.24007#bib.bib32)\)54\.86±0\.0154\.86\{\\pm 0\.01\}−2\.24\-2\.2477\.35±0\.0177\.35\{\\pm 0\.01\}−0\.98\-0\.9887\.26±0\.0287\.26\{\\pm 0\.02\}CoVON\-NoM58\.96±0\.32\\mathbf\{58\.96\}\{\\pm 0\.32\}−1\.86\\mathbf\{\-1\.86\}80\.11±0\.45\\mathbf\{80\.11\}\{\\pm 0\.45\}−0\.67\\mathbf\{\-0\.67\}93\.84±0\.34\\mathbf\{93\.84\}\{\\pm 0\.34\}CoVON59\.89±0\.08\\mathbf\{59\.89\}\{\\pm 0\.08\}−1\.61\\mathbf\{\-1\.61\}82\.24±0\.12\\mathbf\{82\.24\}\{\\pm 0\.12\}−0\.47\\mathbf\{\-0\.47\}95\.14±0\.09\\mathbf\{95\.14\}\{\\pm 0\.09\}Model: CLIP with pretrained ViT\-B/16 as vision encoderPINA\(Wanget al\.,[2024b](https://arxiv.org/html/2606.24007#bib.bib32)\)66\.00±0\.266\.00\{\\pm 0\.2\}−1\.59\-1\.5985\.71±0\.085\.71\{\\pm 0\.0\}−1\.21\-1\.2187\.38±0\.187\.38\{\\pm 0\.1\}MoP\-CLIP\(Nicolaset al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib859)\)69\.70±0\.269\.70\{\\pm 0\.2\}−0\.76\-0\.7688\.65±0\.288\.65\{\\pm 0\.2\}−0\.69\-0\.6992\.29±0\.292\.29\{\\pm 0\.2\}S\-liPrompts\(Wanget al\.,[2022a](https://arxiv.org/html/2606.24007#bib.bib33)\)68\.55±0\.168\.55\{\\pm 0\.1\}−1\.64\-1\.6486\.08±0\.186\.08\{\\pm 0\.1\}−1\.12\-1\.1290\.10±0\.190\.10\{\\pm 0\.1\}CoVON70\.59±0\.2\\mathbf\{70\.59\}\{\\pm 0\.2\}−0\.53\\mathbf\{\-0\.53\}89\.10±0\.1\\mathbf\{89\.10\}\{\\pm 0\.1\}−0\.56\\mathbf\{\-0\.56\}92\.64±0\.3\\mathbf\{92\.64\}\{\\pm 0\.3\}

### 4\.1Revisiting Variational Continual Learning With Fast & Slow Updates

We first compare CoVON with recent VCL approaches\(Meloet al\.,[2025](https://arxiv.org/html/2606.24007#bib.bib1012); Batra and Clark,[2024](https://arxiv.org/html/2606.24007#bib.bib1011)\)on the Permuted MNIST benchmark across 10 tasks in[Fig\.˜2](https://arxiv.org/html/2606.24007#S4.F2)\. Although initially simplistic, we consider this similar experimental setup to the baselines\(Meloet al\.,[2025](https://arxiv.org/html/2606.24007#bib.bib1012)\)in order to ensure comparability and avoid directly running into known scalability issues for select methods\(Nguyenet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib806)\)\.

From the figure, we can observe that CoVON already outperforms the strongest VCL variant \(TD\-VCL\)\(Meloet al\.,[2025](https://arxiv.org/html/2606.24007#bib.bib1012)\)without a slow\-update merging mechanism \(γ<1\\gamma<1, hence referred to as CoVON\-NoM \(CoVON No Merge\)\)\. Although CoVON is easy to implement in an Adam\-style optimizer, it is further evident that both Adam and IVON have not been designed for such an adaptive scenario and both forget catastrophically\. CoVON’s approximately 2% observed gain in final accuracy over the best VCL methods is further substantiated by the inclusion of the slow consolidation mechanism \(CoVON\), which merges past posteriors, boosting the final accuracy to 92\.12%\.

To shed more light on these observed improvements, we ablate the role of the interplay between the fast\-adapted new posterior and the slow\-consolidated old posterior in[Fig\.˜2](https://arxiv.org/html/2606.24007#S4.F2)\. Considering only fast updates, i\.e\., a standard IVON optimizer, unsurprisingly yields catastrophic forgetting \(38%\)\. On the contrary, including the posterior as the prior in standard VCL alleviates some forgetting \(77%\)\. The clearest improvement appears when fast and slow updates are coupled through posterior merging\. We ablate two variants of this slow consolidation\. The first, which is the CoVON update in[Eq\.˜10](https://arxiv.org/html/2606.24007#S3.E10), follows the full VCL formulation and incorporates precision information throughStS\_\{t\}\. The second is a standard EMA\-style merge, which performs a simpler weighted averaging update without the precision information \(𝐦t←\(1−γ\)​𝐦t−1\+γ​𝐦\\mathbf\{m\}\_\{t\}\\leftarrow\(1\-\\gamma\)\\mathbf\{m\}\_\{t\-1\}\+\\gamma\\mathbf\{m\}\)\. The ablation results show that the slow merging step benefits from accumulated curvature information, with precision\-aware consolidation consistently outperforming the simple EMA\-style merge\.

### 4\.2Contextualizing CoVON In Domain Incremental Settings

We now validate CoVON on a set of more challenging domain\-incremental learning \(DIL\) benchmarks without access to the task identifiers or replay buffers\. Here, we further compare CoVON to non\-VCL SOTA CL methods, in order to further contextualize CoVON in the broader literature landscape\. To this end, we consider three standard benchmarks: CDDB\(Liet al\.,[2023](https://arxiv.org/html/2606.24007#bib.bib35)\), CORe50\(Lomonaco and Maltoni,[2017](https://arxiv.org/html/2606.24007#bib.bib34)\), and DomainNet\(Penget al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib36)\)\. CDDB is a continual deepfake detection benchmark, where we adopt the most demanding “hard” track, consisting of∼\\sim27k images across five generative models\. CORe50\(Lomonaco and Maltoni,[2017](https://arxiv.org/html/2606.24007#bib.bib34)\)evaluates continual object recognition over 50 categories and 11 domains, where train sequentially on 8 domains \(120k images\) and test on 3 unseen domains\. DomainNet\(Penget al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib36)\)is a large\-scale benchmark with 345 classes and∼\\sim600k images spanning six diverse domains\. Regarding the methods, we compare our approach against EWC\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.24007#bib.bib814)\), EWC\-Corrected\(Huszár,[2018](https://arxiv.org/html/2606.24007#bib.bib813)\), L2P\(Wanget al\.,[2022c](https://arxiv.org/html/2606.24007#bib.bib30)\), and LwF\(Li and Hoiem,[2017](https://arxiv.org/html/2606.24007#bib.bib28)\), as well as several exemplar\-free SOTA DIL approaches, including S\-Prompt\(Wanget al\.,[2022a](https://arxiv.org/html/2606.24007#bib.bib33)\), PINA\(Wanget al\.,[2024b](https://arxiv.org/html/2606.24007#bib.bib32)\), CODA\-P\(Smithet al\.,[2023](https://arxiv.org/html/2606.24007#bib.bib31)\), and our own degenerate case of Ada\-Reg\. We follow the same experimental protocol as our baselines, adopting a ViT\-B/16 backbone pretrained on ImageNet\-1k\. Further experimental details are in[Appendix˜C](https://arxiv.org/html/2606.24007#A3)\.

Table[1](https://arxiv.org/html/2606.24007#S4.T1)reports the average accuracy \(ATA\_\{T\}\) and forgetting \(FTF\_\{T\}\) across tasks at the end of training\. We can observe that CoVON consistently strikes a strong balance between stability and plasticity across all exemplar\-free domain\-incremental settings\. On DomainNet, it achieves an average accuracy of 59\.89%, outperforming the strongest baseline \(PINA, 54\.86%\) by 4\.1%, while maintaining a low forgetting value \(−1\.86\-1\.86\)\. Similarly strong performance is visible for CDDB\-Hard and Core50, where the latter features the largest observed gains\. Changing the backbone to CLIP improves overall performance, but does not change these observed trends for CoVON\. We attribute this to CoVON’s fast and slow adaptation with online curvature estimates, which regularize the fast\-adapted posterior mean against excessive drift under changing task distributions\. As in the VCL comparison, the slow posterior\-merging update can again be empirically observed to provide additional benefit\.

Finally, we note that CoVON further reduces the cost of the computation over traditional regularization methods such as EWC and EWC∗\. On Core50, EWC’s post\-hoc estimation increases computational cost by more than 200% at task transitions, whereas CoVON incurs no substantial overhead beyond standard AdamW training\. We provide a detailed cost comparison in[Table˜3](https://arxiv.org/html/2606.24007#A3.T3)\. Overall, both CoVON’s efficacy in balancing stability\-plasticity and their computational efficiency make it an ideal contender not only for domain\-incremental learning, but also for modern fine\-tuning of larger models\.

![Refer to caption](https://arxiv.org/html/2606.24007v1/x3.png)

![Refer to caption](https://arxiv.org/html/2606.24007v1/x4.png)

Figure 3:We show that a fast and slow VCL realization through CoVON can effectively scale to LLMs, demonstrating a strong stability\-plasticity trade\-off across diverse continual language modeling settings\.\(a\)On continual multilingual pretraining from English to German to French \(transitions after every 5k iterations\), CoVON maintains low perplexity on previously seen languages while adapting effectively to each new language, exhibiting less forgetting than AdamW and IVON\.\(b\)On continual finetuning across sequential reasoning tasks for GSM8K mathematics, Alpaca\-Code, and the ARC\-Challenge, CoVON retains approximately 95% of mathematical reasoning accuracy after finetuning on coding and commonsense distributions \(visualized as shaded bars with the color of the current fine\-tuning task\), while achieving plasticity on each new task comparable to AdamW\.
### 4\.3Precision\-Weighted Continual Pretraining on Multilingual Data

We continually pretrain a GPT\-based language model \(125M parameters\) across three languages in sequence: English→\\rightarrowGerman→\\rightarrowFrench, transitioning to a new language every 5k iterations after observing convergence\. To assess model stability under shifting language distributions, we report the average validation perplexity across all languages seen throughout the training \([Fig\.˜3](https://arxiv.org/html/2606.24007#S4.F3)\), measuring the model’s ability to acquire new linguistic knowledge, while retaining capabilities in learned languages\.

CoVON maintains a consistently stable trajectory throughout, achieving the lowest averaged perplexity of 31\.89 by the end of training\. A similar trend is observed for the variant without the slow\-update posterior merging, which prevents the fast\-adapted weights from overwriting old linguistic ability as each new language is introduced\. In contrast, AdamW\-FT and IVON\-FT exhibit very high forgetting \(perplexity of approx\. 85\), especially when French is introduced, reflecting the fact that these optimizers were designed for static learning purposes\. EWC and the degenerate Ada\-Reg implementation fall in between these results, alleviating some forgetting, but falling behind either CoVON variant\. This highlights that both the variational Bayesian perspective and the addition of the curvature\-weighted posterior merging in fast and slow VCL are relevant for continual adaptation at a larger scale\. We hypothesize that the remaining gap to the cumulative training baseline, i\.e\., the baseline where all data is stored over time, is a reflection of the lack of an explicit buffer for highly memorable experiences, which present an auxiliary avenue in complement to optimizer improvements\.

### 4\.4Uncertainty\-Aware Continual Finetuning on Diverse Reasoning tasks

Finally, we continually fine\-tune a pre\-trained Qwen3\-1\.7B model\(Qwen Team,[2025](https://arxiv.org/html/2606.24007#bib.bib1015)\)on three sequential reasoning tasks: mathematical problem solving \(GSM8K\)\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.24007#bib.bib1031)\), code generation \(Alpaca\-Code\)\(Chaudhary,[2023](https://arxiv.org/html/2606.24007#bib.bib904)\), and commonsense reasoning \(ARC\-Challenge\)\(Clarket al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib925)\), in this order\. In addition to the performance of the new tasks \(gauging plasticity\), we evaluate the stability of CoVON across these tasks by reporting the ability of the model to solve math problems \(the initial fine\-tuning set\) after continuing training in coding skills and common\-sense reasoning \(visualized in the form of shading of the respective task’s color in[Fig\.˜3](https://arxiv.org/html/2606.24007#S4.F3)\)\. As the previous experimental subsections have already corroborated the positive impact of the slow\-update posterior merging, we directly evaluate the full CoVON variant only\.

[Fig\.˜3](https://arxiv.org/html/2606.24007#S4.F3)illustrates that CoVON achieves comparable plasticity on each new reasoning task to standard finetuning with AdamW, matching∼\\sim65% on math problems,∼\\sim52% on code generation, and∼\\sim80% on the ARC\-Challenge\. When we then re\-evaluate the model’s initial math\-solving ability after learning these new skills, CoVON exhibits only minimal forgetting: performance drops by just 2% after code generation and 4% after common\-sense reasoning\. Similarly, CoVON\-NoM also observes high stability across tasks, reporting only 8% forgetting at the end of the training\. In contrast, AdamW exhibits a∼\\sim70% degradation in mathematical reasoning after commonsense finetuning, and EWC∗incurs additional computational overhead without matching CoVON’s retention \(∼\\sim18% drop\)\. These results further highlight the importance of the posterior merging as part of the slow consolidation step\. Embedded together, they show that the interplay between fast adaptation and slow consolidation makes CoVON a robust continual optimization method that scales to challenging tasks\.

## 5Conclusion

In this work, we have realized a natural fast and slow adaptation mechanism within a Variational Continual Learning framework and have shown that it can be instantiated in a practical optimizer for continual adaptation of a single model\. In our proposed CoVON approach, the two timescales were implemented through task\-wise posterior updates, which entail rapid adaptation to new data, and posterior merges, which entail slower, uncertainty\-aware knowledge preservation across tasks\. Across domain\-incremental classification, continual pre\-training, and fine\-tuning of large language models, these mechanisms were shown to consistently improve over existing VCL optimizers and weight\-regularization methods\. For future work, CoVON’s ease of use and Adam\-like implementation position it favorably for combination with auxiliary structural continual learning perspectives, such as memory replay, dynamic architecture modifications, or in\-context learning mechanisms\(Dorovataset al\.,[2026](https://arxiv.org/html/2606.24007#bib.bib945)\)\. One limitation of CoVON is that it relies on Gaussian posteriors with diagonal variance, extending it to more expressive posteriors is therefore a natural next step\. Another promising avenue lies in addressing related knowledge interference challenges, e\.g\., in federated learning or unlearning, which we posit could be generalized by drawing inspiration from the outlined perspective of knowledge adaptation as posterior correction inKhan \([2025](https://arxiv.org/html/2606.24007#bib.bib819)\)\.

## Acknowledgments

This work was supported by the Bayes duality project, JST CREST Grant Number JPMJCR211\. TM acknowledges the support of JSPS KAKENHI Grant Number 26H02541\. This research has also benefited from Germany’s Excellence Strategy EXC\-3057 “RAI: Reasonable Artificial Intelligence”, funded by the Deutsche Forschungsgemeinschaft \(DFG, German Research Foundation\), as well as the high\-profile area “Minds, Media, Machines” \(MMM\) at the University of Bremen\.

## References

- I\. Abbes, G\. Subbaraj, M\. Riemer, N\. Islah, B\. Therien, T\. Tabaru, H\. Kingetsu, S\. Chandar, and I\. Rish \(2025\)Revisiting replay and gradient alignment for continual pre\-training of large language models\.InConference on Lifelong Learning Agents \(CoLLAs\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p4.1)\.
- Learning fast, learning slow: a general continual learning method based on complementary learning system\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1),[§2](https://arxiv.org/html/2606.24007#S2.p3.1)\.
- J\. Ba, G\. E\. Hinton, V\. Mnih, J\. Z\. Leibo, and C\. Ionescu \(2016\)Using fast weights to attend to the recent past\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p4.1)\.
- H\. Batra and R\. Clark \(2024\)EVCL: elastic variational continual learning with weight consolidation\.InICML Workshop on Structured Probabilistic Inference & Generative Modeling,Cited by:[§4\.1](https://arxiv.org/html/2606.24007#S4.SS1.p1.1)\.
- S\. Chaudhary \(2023\)Code Alpaca: an instruction\-following LLaMA model for code generation\.GitHub\.Note:[https://github\.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca)Cited by:[§4\.4](https://arxiv.org/html/2606.24007#S4.SS4.p1.1)\.
- B\. Chérief\-Abdellatif, P\. Alquier, and M\. E\. Khan \(2019\)A generalization bound for online variational inference\.InAsian Conference on Machine Learning \(ACML\),Cited by:[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.SSS0.Px1.p1.6)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? Try ARC, the AI2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[§4\.4](https://arxiv.org/html/2606.24007#S4.SS4.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv:2110\.14168\.Cited by:[§4\.4](https://arxiv.org/html/2606.24007#S4.SS4.p1.1)\.
- N\. Daheim, T\. Möllenhoff, E\. M\. Ponti, I\. Gurevych, and M\. E\. Khan \(2024\)Model merging by uncertainty\-based gradient matching\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.SSS0.Px2.p3.3)\.
- V\. Dorovatas, M\. Schwerin, A\. D\. Bagdanov, L\. Caccia, A\. Carta, L\. Charlin, B\. Hammer, T\. L\. Hayes, T\. Hess, C\. Kanan, D\. Kudithipudi, X\. Liu, V\. Lomonaco, J\. Mendez\-Mendez, D\. Patil, A\. Prabhu, E\. Ricci, T\. Tuytelaars, G\. M\. van de Ven, L\. Wang, J\. van de Weijer, J\. Choi, M\. Mundt, and R\. Aljundi \(2026\)Modular memory is the key to continual learning agents\.arXiv:2603\.01761\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1),[§5](https://arxiv.org/html/2606.24007#S5.p1.1)\.
- A\. Douillard, A\. Ramé, G\. Couairon, and M\. Cord \(2022\)DyTox: transformers for continual learning with dynamic token expansion\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Table 1](https://arxiv.org/html/2606.24007#S4.T1.15.11.11.2)\.
- J\. Eckstein and D\. P\. Bertsekas \(1992\)On the Douglas\-Rachford splitting method and the proximal point algorithm for maximal monotone operators\.Math\. Program\.55\(1\),pp\. 293–318\.Cited by:[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.SSS0.Px2.p3.3)\.
- M\. Elsayed and A\. R\. Mahmood \(2024\)Addressing loss of plasticity and catastrophic forgetting in continual learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p2.1)\.
- C\. Finn, P\. Abbeel, and S\. Levine \(2017\)Model\-agnostic meta\-learning for fast adaptation of deep networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p4.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Cited by:[§C\.4](https://arxiv.org/html/2606.24007#A3.SS4.p1.1)\.
- R\. Hadsell, D\. Rao, A\. A\. Rusu, and R\. Pascanu \(2020\)Embracing change: continual learning in deep neural networks\.Trends in Cognitive Sciences24\(12\),pp\. 1028–1040\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- G\. E\. Hinton and D\. C\. Plaut \(1987\)Using fast weights to deblur old memories\.InProceedings of the Conference of the Cognitive Science Society,Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p4.1),[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.p4.1),[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.SSS0.Px2.p3.3)\.
- F\. Huszár \(2018\)Note on the quadratic penalties in elastic weight consolidation\.Proceedings of the National Academy of Sciences115\(11\),pp\. E2496–E2497\.Cited by:[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.p7.2),[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.25.21.21.6)\.
- K\. J\. Joseph and V\. N\. Balasubramanian \(2020\)Meta\-consolidation for continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p3.1)\.
- M\. E\. Khan and H\. Rue \(2023\)The Bayesian learning rule\.J\. Mach\. Learn\. Res\. \(JMLR\)24\(281\),pp\. 1–46\.Cited by:[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.p2.2)\.
- M\. E\. Khan \(2025\)Knowledge adaptation as posterior correction\.arXiv:2506\.14262\.Cited by:[§A\.1](https://arxiv.org/html/2606.24007#A1.SS1.p2.7),[§A\.1](https://arxiv.org/html/2606.24007#A1.SS1.p2.8),[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.p3.1),[§5](https://arxiv.org/html/2606.24007#S5.p1.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1),[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.p7.2),[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.20.16.16.6)\.
- H\. Lee, H\. Cho, H\. Kim, D\. Kim, D\. Min, J\. Choo, and C\. Lyle \(2024\)Slow and steady wins the race: maintaining plasticity with hare and tortoise networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1),[§2](https://arxiv.org/html/2606.24007#S2.p3.1)\.
- S\. W\. Lee, J\. H\. Kim, J\. Jun, J\. W\. Ha, and B\. T\. Zhang \(2017\)Overcoming catastrophic forgetting by incremental moment matching\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- S\. W\. Lee, C\. Y\. Lee, D\. H\. Kwak, J\. Kim, J\. Kim, and B\. T\. Zhang \(2016\)Dual\-memory deep learning architectures for lifelong learning of everyday human behaviors\.InInternational Joint Conference on Artificial Intelligence \(IJCAI\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1)\.
- C\. Li, Z\. Huang, D\. P\. Paudel, Y\. Wang, M\. Shahbazi, X\. Hong, and L\. Van Gool \(2023\)A continual deepfake detection benchmark: dataset, methods, and essentials\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),Cited by:[§C\.2](https://arxiv.org/html/2606.24007#A3.SS2.p1.5),[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2)\.
- Y\. Li, F\. Dangel, D\. Tam, and C\. Raffel \(2025\)Fishers for free? Approximating the Fisher information matrix by recycling the squared gradient accumulator\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p2.1)\.
- Z\. Li and D\. Hoiem \(2017\)Learning without forgetting\.IEEE Trans\. Pattern Anal\. Mach\. Intell\. \(PAMI\)40\(12\),pp\. 2935–2947\.Cited by:[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.30.26.26.6)\.
- V\. Lomonaco and D\. Maltoni \(2017\)Core50: a new dataset and benchmark for continuous object recognition\.InConference on Robot Learning \(CoRL\),Cited by:[§C\.2](https://arxiv.org/html/2606.24007#A3.SS2.p1.5),[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p2.1)\.
- J\. McClelland, B\. McNaughton, and R\. O’Reilly \(1995\)Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory\.Psychological Review102,pp\. 419–457\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1),[§2](https://arxiv.org/html/2606.24007#S2.p3.1)\.
- L\. C\. Melo, A\. Abate, and Y\. Gal \(2025\)Temporal\-difference variational continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§C\.1](https://arxiv.org/html/2606.24007#A3.SS1.p1.3),[§4\.1](https://arxiv.org/html/2606.24007#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.24007#S4.SS1.p2.1)\.
- S\. I\. Mirzadeh, M\. Farajtabar, R\. Pascanu, and H\. Ghasemzadeh \(2020\)Understanding the role of training regimes in continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p2.1)\.
- R\. Mitchell, R\. Menzenbach, K\. Kersting, and M\. Mundt \(2024\)Self\-expanding neural networks\.arXiv:2307\.04526\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- M\. Mundt, Y\. Hong, I\. Pliushch, and V\. Ramesh \(2023\)A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning\.Neural Networks160,pp\. 306–336\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1),[§2](https://arxiv.org/html/2606.24007#S2.p2.1)\.
- C\. V\. Nguyen, Y\. Li, T\. D\. Bui, and R\. E\. Turner \(2018\)Variational continual learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p3.1),[§2](https://arxiv.org/html/2606.24007#S2.p5.5),[§2](https://arxiv.org/html/2606.24007#S2.p5.7),[§4\.1](https://arxiv.org/html/2606.24007#S4.SS1.p1.1),[Algorithm 4](https://arxiv.org/html/2606.24007#alg4)\.
- A\. Nichol, J\. Achiam, and J\. Schulman \(2018\)On first\-order meta\-learning algorithms\.arXiv:1803\.02999\.Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p4.1),[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.p4.1),[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.SSS0.Px2.p3.3)\.
- J\. Nicolas, F\. Chiaroni, I\. Ziko, O\. Ahmad, C\. Desrosiers, and J\. Dolz \(2024\)MoP\-CLIP: a mixture of prompt\-tuned CLIP models for domain incremental learning\.InIEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),Cited by:[Table 1](https://arxiv.org/html/2606.24007#S4.T1.75.71.71.6)\.
- R\. C\. O’Reilly and K\. A\. Norman \(2002\)Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework\.Trends in Cognitive Sciences6\(12\),pp\. 505–510\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1)\.
- X\. Peng, Q\. Bai, X\. Xia, Z\. Huang, K\. Saenko, and B\. Wang \(2019\)Moment matching for multi\-source domain adaptation\.InInternational Conference on Computer Vision \(ICCV\),Cited by:[§C\.2](https://arxiv.org/html/2606.24007#A3.SS2.p1.5),[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2)\.
- Q\. Pham, C\. Liu, and S\. Hoi \(2021\)Dualnet: Continual learning, fast and slow\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1),[§2](https://arxiv.org/html/2606.24007#S2.p3.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§C\.4](https://arxiv.org/html/2606.24007#A3.SS4.p1.1),[§4\.4](https://arxiv.org/html/2606.24007#S4.SS4.p1.1)\.
- S\. Rebuffi, A\. Kolesnikov, G\. Sperl, and C\. H\. Lampert \(2017\)iCaRL: incremental classifier and representation learning\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. Wayne \(2019\)Experience replay for continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- A\. A\. Rusu, N\. C\. Rabinowitz, G\. Desjardins, H\. Soyer, J\. Kirkpatrick, K\. Kavukcuoglu, R\. Pascanu, and R\. Hadsell \(2016\)Progressive Neural Networks\.arXiv:1606\.04671\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- Y\. Shen, N\. Daheim, B\. Cong, P\. Nickl, G\. M\. Marconi, C\. Bazan, R\. Yokota, I\. Gurevych, D\. Cremers, M\. E\. Khan, and T\. Möllenhoff \(2024\)Variational learning is effective for large deep networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p3.1),[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.p8.1)\.
- H\. Shin, J\. K\. Lee, J\. Kim, and J\. Kim \(2017\)Continual learning with deep generative replay\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1)\.
- D\. Silver and R\. S\. Sutton \(2025\)Welcome to the era of experience\.Google AI\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- J\. S\. Smith, L\. Karlinsky, V\. Gutta, P\. Cascante\-Bonilla, D\. Kim, A\. Arbelle, R\. Panda, R\. Feris, and Z\. Kira \(2023\)Coda\-prompt: continual decomposed attention\-based prompting for rehearsal\-free continual learning\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.45.41.41.6)\.
- S\. Thrun \(1998\)Lifelong learning algorithms\.Learning to Learn8,pp\. 181–209\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- G\. M\. Van de Ven, H\. T\. Siegelmann, and A\. S\. Tolias \(2020\)Brain\-inspired replay for continual learning with artificial neural networks\.Nature communications11\(1\),pp\. 4069\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p2.1)\.
- D\. van der Hoeven, T\. van Erven, and W\. Kotłowski \(2018\)The many faces of exponential weights in online learning\.InConference on Learning Theory \(COLT\),pp\. 2067–2092\.Cited by:[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.SSS0.Px1.p1.6),[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.p3.6)\.
- V\. G\. Vovk \(1990\)Aggregating strategies\.InConference on Learning Theory \(COLT\),Cited by:[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.p3.6)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024a\)A comprehensive survey of continual learning: theory, method and application\.IEEE Trans\. Pattern Anal\. Mach\. Intell\. \(PAMI\)46\(8\),pp\. 5362–5383\.Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1),[§2](https://arxiv.org/html/2606.24007#S2.p2.1)\.
- Q\. Wang, Y\. He, S\. Dong, X\. Gao, S\. Wang, and Y\. Gong \(2024b\)Non\-exemplar domain incremental learning via cross\-domain concept integration\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§C\.2](https://arxiv.org/html/2606.24007#A3.SS2.p1.5),[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.55.51.51.6),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.70.66.66.6)\.
- Y\. Wang, Z\. Huang, and X\. Hong \(2022a\)S\-prompts learning with pre\-trained transformers: an Occam’s razor for domain incremental learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.50.46.46.6),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.80.76.76.6)\.
- Z\. Wang, Z\. Zhang, S\. Ebrahimi, R\. Sun, H\. Zhang, C\. Lee, X\. Ren, G\. Su, V\. Perot, J\. Dy, and T\. Pfister \(2022b\)DualPrompt: complementary prompting for rehearsal\-free continual learning\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[Table 1](https://arxiv.org/html/2606.24007#S4.T1.40.36.36.6)\.
- Z\. Wang, Z\. Zhang, C\. Lee, H\. Zhang, R\. Sun, X\. Ren, G\. Su, V\. Perot, J\. Dy, and T\. Pfister \(2022c\)Learning to prompt for continual learning\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§4\.2](https://arxiv.org/html/2606.24007#S4.SS2.p1.2),[Table 1](https://arxiv.org/html/2606.24007#S4.T1.35.31.31.6)\.
- J\. Yoon, E\. Yang, J\. Lee, and S\. J\. Hwang \(2018\)Lifelong learning with dynamically expandable networks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- F\. Zenke, B\. Poole, and S\. Ganguli \(2017\)Continual learning through synaptic intelligence\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2606.24007#S1.p1.1)\.
- M\. R\. Zhang, J\. Lucas, G\. Hinton, and J\. Ba \(2019\)Lookahead optimizer: k steps forward, 1 step back\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§2](https://arxiv.org/html/2606.24007#S2.p4.1),[§3\.1](https://arxiv.org/html/2606.24007#S3.SS1.p4.1),[§3\.2](https://arxiv.org/html/2606.24007#S3.SS2.SSS0.Px2.p3.3)\.

## Appendix ADerivation and Algorithms

### A\.1Optimality Condition of the Variational Learning Problem

We start from the variational continual learning objective,

qtfast←arg⁡minq∈𝒬⁡λt​𝔼q​\[ℓt\]\+𝔻KL​\(q∥qt−1slow\)\.q\_\{t\}^\{\\textrm\{fast\}\}\\leftarrow\\arg\\min\_\{q\\in\\mathcal\{Q\}\}\\,\\lambda\_\{t\}\\mathbb\{E\}\_\{q\}\[\\ell\_\{t\}\]\+\\mathbb\{D\}\_\{\\text\{KL\}\}\(q\\,\\\|\\,q\_\{t\-1\}^\{\\textrm\{slow\}\}\)\.
We write the optimality condition of this objective, followingKhan \([2025](https://arxiv.org/html/2606.24007#bib.bib819)\)\(see their Eq\. 3\),

qtfast\(𝜽\)∝qt−1slow\(𝜽\)exp\(−λt𝐓\(𝜽\)⊤∇𝔼qtfast\[−ℓt\]\)=qt−1slow\(𝜽\)exp\(−ℓ^t\)λt,q\_\{t\}^\{\\textrm\{fast\}\}\(\\boldsymbol\{\\theta\}\)\\propto q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\exp\\left\(\-\\lambda\_\{t\}\\mathbf\{T\}\(\\boldsymbol\{\\theta\}\)^\{\\top\}\{\\nabla\}\\mathbb\{E\}\_\{q\_\{t\}^\{\\textrm\{fast\}\}\}\[\-\\ell\_\{t\}\]\\right\)=q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)\\exp\(\-\\hat\{\\ell\}\_\{t\}\)^\{\\lambda\_\{t\}\},where∇\{\\nabla\}is the natural gradient,𝐓​\(𝜽\)\\mathbf\{T\}\(\\boldsymbol\{\\theta\}\)is the sufficient statistics of the exponential family distribution parameterizingqq, and we defineℓ^t​\(𝜽\)=𝐓​\(𝜽\)⊤​∇𝔼qtfast​\[−ℓt\]\\hat\{\\ell\}\_\{t\}\(\\boldsymbol\{\\theta\}\)=\\mathbf\{T\}\(\\boldsymbol\{\\theta\}\)^\{\\top\}\{\\nabla\}\\mathbb\{E\}\_\{q\_\{t\}^\{\\textrm\{fast\}\}\}\[\-\\ell\_\{t\}\]to ease notation\. Further details for this derivation are inKhan \([2025](https://arxiv.org/html/2606.24007#bib.bib819)\)\. We can see that this is equal to multiplying the priorqt−1slow​\(𝜽\)q\_\{t\-1\}^\{\\textrm\{slow\}\}\(\\boldsymbol\{\\theta\}\)by the effect of the new dataexp⁡\(−ℓ^t\)\\exp\(\-\\hat\{\\ell\}\_\{t\}\)raised to the powerλt\\lambda\_\{t\}\.

### A\.2Optimality Condition of the Variational Learning Problem for Gaussians

In this section, we perform the same derivation as in[Sec\.˜A\.1](https://arxiv.org/html/2606.24007#A1.SS1), but using a different approach, and specializing from the beginning to Gaussian approximate distributions\.

We start from the variational continual learning objective,

q←arg⁡minq∈𝒬⁡λt​𝔼q​\[ℓt\]\+𝔻KL​\(q∥qt−1\)\.q\\leftarrow\\arg\\min\_\{q\\in\\mathcal\{Q\}\}\\,\\lambda\_\{t\}\\mathbb\{E\}\_\{q\}\[\\ell\_\{t\}\]\+\\mathbb\{D\}\_\{\\text\{KL\}\}\(q\\,\\\|\\,q\_\{t\-1\}\)\.We restrict the variational family to be Gaussian,

q=𝒩​\(𝐦,𝚺\)\.q=\\mathcal\{N\}\(\\mathbf\{m\},\\mathbf\{\\Sigma\}\)\.𝐦\\mathbf\{m\}denotes the parameter mean and the corresponding precision is defined as:

𝐒=𝚺−1\.\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}=\\mathbf\{\\Sigma\}^\{\-1\}\.
We write the objective overqqin terms of the parametrization\(𝐦,𝚺\)\(\\mathbf\{m\},\\mathbf\{\\Sigma\}\)as follows:

𝒥​\(𝐦,𝚺\)=λt​𝔼q​\[ℓt\]\+𝔻KL​\(𝒩​\(𝐦,𝚺\)∥𝒩​\(𝐦t−1,𝚺t−1\)\)\.\\mathcal\{J\(\\mathbf\{m\},\\mathbf\{\\Sigma\}\)\}=\\lambda\_\{t\}\\mathbb\{E\}\_\{q\}\[\\ell\_\{t\}\]\+\\mathbb\{D\}\_\{\\text\{KL\}\}\(\\mathcal\{N\}\(\\mathbf\{m\},\\mathbf\{\\Sigma\}\)\\,\\\|\\,\\mathcal\{N\}\(\\mathbf\{m\}\_\{t\-1\},\\mathbf\{\\Sigma\}\_\{t\-1\}\)\)\.The KL divergence between two Gaussians is given as,

𝔻KL​\(\(𝐦,𝚺\)∥\(𝐦t−1,𝚺t−1\)\)=\\displaystyle\\mathbb\{D\}\_\{\\text\{KL\}\}\(\(\\mathbf\{m\},\\mathbf\{\\Sigma\}\)\\,\\\|\\,\(\\mathbf\{m\}\_\{t\-1\},\\mathbf\{\\Sigma\}\_\{t\-1\}\)\)=12​\[tr⁡\(𝐒t−1​𝚺\)\+\(𝐦−𝐦t−1\)⊤​𝐒t−1​\(𝐦−𝐦t−1\)−d\+log​det𝚺t−1−log​det𝚺\]\.\\displaystyle\\quad\\frac\{1\}\{2\}\\Big\[\\operatorname\{tr\}\(\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\\mathbf\{\\Sigma\}\)\+\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\)^\{\\top\}\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\)\-d\+\\log\\det\\mathbf\{\\Sigma\}\_\{t\-1\}\-\\log\\det\\mathbf\{\\Sigma\}\\Big\]\.
Discarding constants independent of\(𝐦,𝚺\)\(\\mathbf\{m\},\\mathbf\{\\Sigma\}\), the remaining terms relevant for the optimization are:

12​tr⁡\(𝐒t−1​𝚺\)\+12​\(𝐦−𝐦t−1\)⊤​𝐒t−1​\(𝐦−𝐦t−1\)−12​log​det𝚺\.\\frac\{1\}\{2\}\\operatorname\{tr\}\(\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\\mathbf\{\\Sigma\}\)\+\\frac\{1\}\{2\}\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\)^\{\\top\}\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\)\-\\frac\{1\}\{2\}\\log\\det\\mathbf\{\\Sigma\}\.
The derivative of the objective with respect to𝐦\\mathbf\{m\}is

∇𝐦𝒥=λt​𝔼q​\[∇ℓt​\(𝜽\)\]\+𝐒t−1​\(𝐦−𝐦t−1\)\.\\nabla\_\{\\mathbf\{m\}\}\\mathcal\{J\}=\\lambda\_\{t\}\\,\\mathbb\{E\}\_\{q\}\[\\nabla\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\]\+\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\)\.Setting this gradient to zero gives

𝐒t−1​\(𝐦−𝐦t−1\)=−λt​𝔼q​\[∇ℓt\]\.\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\\big\(\\mathbf\{m\}\-\\mathbf\{m\}\_\{t\-1\}\\big\)=\-\\lambda\_\{t\}\\,\\mathbb\{E\}\_\{q\}\[\\nabla\\ell\_\{t\}\]\.This is the first optimality condition shown in the main paper\.

Next, differentiating with respect to the covariance, we get:

∇𝚺𝔼q​\[ℓ​\(𝜽\)\]=12​𝔼q​\[∇2ℓ​\(𝜽\)\]\.\\nabla\_\{\\mathbf\{\\Sigma\}\}\\mathbb\{E\}\_\{q\}\[\\ell\(\\boldsymbol\{\\theta\}\)\]=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{q\}\[\\nabla^\{2\}\\ell\(\\boldsymbol\{\\theta\}\)\]\.Together with

∇𝚺12​tr⁡\(𝐒t−1​𝚺\)=12​𝐒t−1,∇𝚺\(−12​log​det𝚺\)=−12​𝚺−1,\\nabla\_\{\\mathbf\{\\Sigma\}\}\\frac\{1\}\{2\}\\operatorname\{tr\}\(\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\\mathbf\{\\Sigma\}\)=\\frac\{1\}\{2\}\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\},\\qquad\\nabla\_\{\\mathbf\{\\Sigma\}\}\\\!\\left\(\-\\frac\{1\}\{2\}\\log\\det\\mathbf\{\\Sigma\}\\right\)=\-\\frac\{1\}\{2\}\\mathbf\{\\Sigma\}^\{\-1\},we obtain

∇𝚺𝒥=λt2​𝔼q​\[∇2ℓt​\(𝜽\)\]\+12​𝐒t−1−12​𝚺−1\.\\nabla\_\{\\mathbf\{\\Sigma\}\}\\mathcal\{J\}=\\frac\{\\lambda\_\{t\}\}\{2\}\\mathbb\{E\}\_\{q\}\[\\nabla^\{2\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)\]\+\\frac\{1\}\{2\}\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\-\\frac\{1\}\{2\}\\mathbf\{\\Sigma\}^\{\-1\}\.
Setting this gradient to zero yields

𝐒=𝐒t−1\+λt​𝔼q​\[∇2ℓt\]\.\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}=\\mbox\{$\\mbox\{$\\mathbf\{S\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\,\\mathbb\{E\}\_\{q\}\[\\nabla^\{2\}\\ell\_\{t\}\]\.This is the second optimality condition shown in the main paper\.

### A\.3Baseline Implementations of VCL

We show that VCL can be approximately implemented with standard first\-order optimizers like AdamW using its online Hessian estimates, as shown in[Alg\.˜2](https://arxiv.org/html/2606.24007#alg2)\. More specifically, CoVON can be instantiated as a standard first\-order continual optimization algorithm, which we call AdamW\-Regularized \(Ada\-Reg\)\. For completeness, we also provide pseudocode for EWC and EWC∗[Alg\.˜3](https://arxiv.org/html/2606.24007#alg3)as Laplace approximations to VCL, as well as variational continual learning implemented with Bayes\-by\-Backprop[Alg\.˜4](https://arxiv.org/html/2606.24007#alg4)\.

We emphasize that EWC and Ada\-Reg arise as a degenerate special case of CoVON under the following assumptions:

1. 1\.We approximate posterior expectations at the mean \(no sampling\)\.
2. 2\.For EWC and EWC∗, curvature is estimated once per task using squared gradients and then frozen\. For Ada\-Reg, Hessian curvature is approximated using the squared\-gradient accumulator of Adam\.
3. 3\.Optimization proceeds deterministically using only the posterior mean\.

CoVON relaxes all three assumptions, yielding a strictly more general optimizer that retains uncertainty awareness while remaining cheap for large\-scale training\.

Algorithm 2AdamW\-Regularized \(Ada\-Reg\)1:Init:

𝜽←\(NN\-weight init\),𝜽0←0,𝐬0←prior\-precision\\boldsymbol\{\\theta\}\\leftarrow\(\\text\{NN\-weight init\}\),\\;\\boldsymbol\{\\theta\}\_\{0\}\\leftarrow 0,\\;\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{0\}\\leftarrow\\text\{prior\-precision\}
2:Init:

β1,β2∈\(0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\(0,1\),

λt\\lambda\_\{t\}for each task

tt, Hessian init

𝐡0=0\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\_\{0\}=0,learning rate

α\\alpha
3:for

t=1​…​Tt=1\\ldots Tdo

4:

𝐠←0\\mathbf\{g\}\\leftarrow 0,

𝐡←h0\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow h\_\{0\},

5:for

i=1,2,…i=1,2,\\ldotsdo

6:

𝐠^←∇^​ℓt​\(𝜽\)\\widehat\{\\mathbf\{g\}\}\\leftarrow\\widehat\{\\nabla\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)
7:

^​𝐡←𝐠2\\widehat\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow\\mathbf\{g\}^\{2\}
8:

𝐠←β1​𝐠\+\(1−β1\)​𝐠^\\mathbf\{g\}\\leftarrow\\beta\_\{1\}\\mathbf\{g\}\+\(1\-\\beta\_\{1\}\)\\widehat\{\\mathbf\{g\}\},

𝐠¯←𝐠/\(1−β1i\)\\bar\{\\mathbf\{g\}\}\\leftarrow\\mathbf\{g\}/\(1\-\\beta\_\{1\}^\{i\}\)
9:

𝐡←β2​𝐡\+\(1−β2\)​^​𝐡\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow\\beta\_\{2\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\(1\-\\beta\_\{2\}\)\\widehat\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\},

¯​𝐡←𝐡/\(1−β2i\)\\bar\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}/\(1\-\\beta\_\{2\}^\{i\}\)
10:

𝜽←𝜽−α​\(𝐠¯/\(¯​𝐡\+ϵ\)\+λt−1​𝐬t−1​\(𝜽−𝜽t−1\)\)\\boldsymbol\{\\theta\}\\leftarrow\\boldsymbol\{\\theta\}\-\\alpha\(\\bar\{\\mathbf\{g\}\}/\(\\sqrt\{\\bar\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\}\+\\epsilon\)\+\\lambda\_\{t\}^\{\-1\}\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\(\\boldsymbol\{\\theta\}\-\\boldsymbol\{\\theta\}\_\{t\-1\}\)\)
11:endfor

12:

𝐬t←𝐬t−1\+λt​¯​𝐡\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\}\\leftarrow\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+\\lambda\_\{t\}\\bar\{\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}
13:

𝜽t←𝜽\\boldsymbol\{\\theta\}\_\{t\}\\leftarrow\\boldsymbol\{\\theta\}
14:endfor

Algorithm 3EWC and EWC∗using post\-hoc uncertainty with squared gradients1:Init:

𝜽←\(NN\-weight init\)\\boldsymbol\{\\theta\}\\leftarrow\(\\text\{NN\-weight init\}\),

𝜽0←0\\boldsymbol\{\\theta\}\_\{0\}\\leftarrow 0,

𝐬0←prior\-precision\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{0\}\\leftarrow\\text\{prior\-precision\}
2:Init:Weightings

λt\\lambda\_\{t\}for each task

tt
3:for

t=1​…​Tt=1\\ldots Tdo

4:for

i=1,2,…i=1,2,\\ldotsdo

5:

𝐠^←∇^​ℓt​\(𝜽\)\\widehat\{\\mathbf\{g\}\}\\leftarrow\\widehat\{\\nabla\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)
6:Update

𝜽\\boldsymbol\{\\theta\}by optimizer step on

𝐠^\\widehat\{\\mathbf\{g\}\}and

∑k=0t−1λk−1​𝐬k​\(𝜽−𝜽k\)\\sum\_\{k=0\}^\{t\-1\}\\lambda\_\{k\}^\{\-1\}\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{k\}\(\\boldsymbol\{\\theta\}\-\\boldsymbol\{\\theta\}\_\{k\}\)\(decoupled in AdamW\)

7:endfor

8:

𝐡←0\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow 0,

k←\|𝒟t\|/Bk\\leftarrow\|\\mbox\{$\{\\cal D\}$\}\_\{t\}\|/B
9:foreach minibatch of size

BBin

𝒟t\\mathcal\{D\}\_\{t\}do

10:

𝐡←𝐡\+∇^​ℓt​\(𝜽\)2/k\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\\leftarrow\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}\+\\widehat\{\\nabla\}\\ell\_\{t\}\(\\boldsymbol\{\\theta\}\)^\{2\}/k
11:endfor

12:

𝐬t←B​λt​𝐡\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\}\\leftarrow B\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}for EWC,

𝐬t←𝐬t−1\+B​λt​𝐡\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\}\\leftarrow\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{t\-1\}\+B\\lambda\_\{t\}\\mbox\{$\\mbox\{$\\mathbf\{h\}$\}$\}for EWC∗

13:

𝜽t←𝜽\\boldsymbol\{\\theta\}\_\{t\}\\leftarrow\\boldsymbol\{\\theta\},

14:endfor

Algorithm 4Variational Continual Learning \(VCL\) with Bayes by Backprop\(Nguyenet al\.,[2018](https://arxiv.org/html/2606.24007#bib.bib806)\)1:Init:Initial posterior

q​\(𝜽\)←p0​\(𝜽\)q\(\\boldsymbol\{\\theta\}\)\\leftarrow p\_\{0\}\(\\boldsymbol\{\\theta\}\), prior

p0​\(𝜽\)←𝒩​\(𝜽\|0,𝐬0−1\)p\_\{0\}\(\\boldsymbol\{\\theta\}\)\\leftarrow\\mbox\{$\{\\cal N\}$\}\(\\boldsymbol\{\\theta\}\\,\|\\,0,\\mbox\{$\\mbox\{$\\mathbf\{s\}$\}$\}\_\{0\}^\{\-1\}\)
2:Init:

λt\\lambda\_\{t\}for each task

tt
3:for

t=1​…​Tt=1\\ldots Tdo

4:for

i=1,2,…i=1,2,\\ldotsdo

5:Update

qqvia Bayes\-by\-Backprop on

λt​𝔼q​\[ℓt\]\+𝔻KL​\(q∥qt−1\)\\lambda\_\{t\}\\mathbb\{E\}\_\{q\}\[\\ell\_\{t\}\]\+\\mathbb\{D\}\_\{\\text\{KL\}\}\(q\\,\\\|\\,q\_\{t\-1\}\)and SGD

6:endfor

7:

qt←qq\_\{t\}\\leftarrow q
8:endfor

## Appendix BUnderstanding the Hyperparameters of CoVON

In this section, we provide practical guidance for choosing the hyperparameters that govern the training dynamics of CoVON\. In particular, the behavior of the fast inner loop and slow outer loop depends on several key choices, including Hessian initialization, effective sample size, Hessian Momentum,γ\\gammafor the posterior\-merging mechanism, etc\. We now analyze the sensitivity of CoVON to these hyperparameters and identify the regimes in which it achieves a favorable stability–plasticity trade\-off\.

![Refer to caption](https://arxiv.org/html/2606.24007v1/x5.png)

![Refer to caption](https://arxiv.org/html/2606.24007v1/x6.png)

![Refer to caption](https://arxiv.org/html/2606.24007v1/x7.png)

![Refer to caption](https://arxiv.org/html/2606.24007v1/x8.png)

Figure 4:We analyze the importance of the different hyperparameters that influence the training dynamics of CoVON\.Hessian Initializationh0h\_\{0\}: The initialization of the curvatureh0h\_\{0\}defines the precision of the Gaussian prior before any data is observed\. In a continual setting, this choice is critical: an overly largeh0h\_\{0\}prevents meaningful posterior formation on the initial task, whereas an overly smallh0h\_\{0\}causes the initial task to dominate, leading to excessive curvature growth and premature freezing in later tasks, especially in image classification tasks\. For vision tasks across different domains, values ofh0h\_\{0\}in the range of1​e−11e^\{\-1\}to1​e−21e^\{\-2\}tend to work well, whereas for language models, an initialization around1​e−31e^\{\-3\}provides the best results\. It is also worth noting that one needs to re\-initialize the Hessian at the start of each new task\.

Effective sample size: Another crucial hyperparameter in CoVON is the effective sample sizeλt\\lambda\_\{t\}for each task\. This quantity controls both the amount of noise injected into the parameters and the influence of posterior precision in the update\. In particular, the variance of the injected noise is inversely proportional to the effective sample size, \(noise∝1ess\\text\{noise\}\\propto\\frac\{1\}\{\\text\{ess\}\}\)\. Larger values therefore reduce the noise and the strength of the quadratic regularization term, causing the optimizer to rely more strongly on the new data\. Smaller values have the opposite effect, regularizing stronger towards the previous task\. Across both language modeling and the image classification task, we found the suitable range of ess to be around10810^\{8\}to101010^\{10\}\. Ideally, setting ess to the size of the dataset could be a good starting point\.

Learning rateα\\alpha:For ViT\-based architectures trained on Core50, CDDB, initial learning has been set around1​e−51e^\{\-5\}, except for DomainNet, it requires a higher learning rate of1​e−21e^\{\-2\}\. From the second task onwards, we decayed the lr in the range of5​e−55e^\{\-5\}to1​e−61e^\{\-6\}In the case of continually pretraining GPT\-2 with 125M parameters, we have set the initial learning rate to be 0\.97, whereas the finetuning of the Qwen3\-1\.7B architecture, we have set the learning rate to be3​e−33e^\{\-3\}\.

Hessian momentumβ2\\beta\_\{2\}:The Hessian momentum needs to be set rather close to one; for instance, values in the range of 0\.9999 to 0\.99999 have worked well for our experiments\.

Gradient momentumβ1\\beta\_\{1\}:We have set the gradient momentumβ1\\beta\_\{1\}to 0\.9 for all of our experiments\.

Merging coefficientγ\\gamma:Theγ\\gammaacts as a merging coefficient for the slow component in CoVON, controlling how strongly the current posterior is consolidated with the past posteriors\. Across different domain\-incremental classification benchmarks, a value of 0\.1\-0\.5 worked well, whereas for the language modeling, a value in the range of 1e\-2 worked well\.

## Appendix CAdditional Training and Experimental Details

### C\.1Comparing against the SOTA VCL benchmark

We follow the same experimental protocol asMeloet al\.\([2025](https://arxiv.org/html/2606.24007#bib.bib1012)\)and evaluate on the Permuted MNIST Hard benchmark\. The benchmark consists of 10 sequential tasks, each with 50k training samples and 10k test samples\. All methods are trained with a simple MLP architecture with two hidden layers \[100,100\] and a single head with 10 units, and the baseline methods use the same hyperparameters reported in\(Meloet al\.,[2025](https://arxiv.org/html/2606.24007#bib.bib1012)\)\. For CoVON, we use a learning rate of 0\.02 on the first task and reduce it to 0\.003 for all subsequent tasks, training for 30 epochs on each task\. The Hessian is initialized to 0\.01, whileλ\\lambdaandγ\\gammaare chosen in the ranges of1​e7−1​e81e^\{7\}\-1e^\{8\}and 0\.8 \- 0\.9, respectively, across the tasks\.

### C\.2Contextualizing CoVON In Domain Incremental Settings

We consider three standard domain\-incremental benchmarks: DomainNet\(Penget al\.,[2019](https://arxiv.org/html/2606.24007#bib.bib36)\), CDDB\(Liet al\.,[2023](https://arxiv.org/html/2606.24007#bib.bib35)\), and CORe50\(Lomonaco and Maltoni,[2017](https://arxiv.org/html/2606.24007#bib.bib34)\)\. For the fairness of comparison, we follow the same experimental protocol as prior baselines\(Wanget al\.,[2024b](https://arxiv.org/html/2606.24007#bib.bib32)\), using a ViT\-B/16 backbone pretrained on ImageNet\-1K\. For each dataset, we use the official train/test splits with the hyperparameters used in the prior works\. The corresponding CoVON hyperparameters are as follows: an initial Hessian value of5​e−35e^\{\-3\}, with the Hessian for subsequent tasks set to 1e\-3,λ\\lambda: 1e8,γ\\gamma: 0\.3, and a learning rate of1​e−41e^\{\-4\}and decayed to1​e−61e^\{\-6\}from the second task onwards\.

### C\.3Continual Pretraining on multilingual data

For continual pretraining of the language model, we use a GPT\-style architecture withnlayer=12n\_\{\\mathrm\{layer\}\}=12,nhead=12n\_\{\\mathrm\{head\}\}=12, and embedding dimensionnembd=768n\_\{\\mathrm\{embd\}\}=768, with dropout set to0\.00\.0and no bias terms\. Training is performed with micro\-batch size2020, gradient accumulation steps2424, and sequence length10241024, yielding a larger effective batch through accumulation\. We use1500015000optimization steps in total, corresponding to50005000updates per language for the sequence\{en,de,fr\}\\\{\\mathrm\{en\},\\mathrm\{de\},\\mathrm\{fr\}\\\}\.

For the CoVON\-specific hyperparameters, we use learning rate9\.72×10−19\.72\\times 10^\{\-1\}, weight decay1×10−61\\times 10^\{\-6\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99995\\beta\_\{2\}=0\.99995, gradient clipping1\.01\.0, and Hessian update valuehess,2=1\.92×10−2h\_\{\\mathrm\{ess\},2\}=1\.92\\times 10^\{\-2\}\. Learning\-rate decay is enabled with500500warmup iterations, decay over the full1500015000iterations, and minimum learning rate0\. We set the initial Hessian to1×10−31\\times 10^\{\-3\}, the effective sample size to1×10101\\times 10^\{10\}for English,1\.44×10131\.44\\times 10^\{13\}for German, and1×10151\\times 10^\{15\}for French, and the clipping radius to3×10−43\\times 10^\{\-4\}\. The slow\-update coefficients areγ=10−2\\gamma=10^\{\-2\}for English,γ=10−3\\gamma=10^\{\-3\}for German, andγ=10−2\\gamma=10^\{\-2\}for French\.

For the baselines trained with AdamW, we use a learning rate6×10−46\\times 10^\{\-4\}, a weight decay1×10−11\\times 10^\{\-1\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, and gradient clipping1\.01\.0\. Training runs for1500015000iterations, with learning\-rate decay enabled,500500warmup iterations, decay over the full50005000iterations per language, and a minimum learning rate of6×10−56\\times 10^\{\-5\}\.

### C\.4Continual finetuning of Qwen models on a set of sequential reasoning tasks

We provide additional details on the training and evaluation pipeline used in our continual fine\-tuning experiments\. In this setting, we fine\-tune a pretrained Qwen3\-1\.7B model\(Qwen Team,[2025](https://arxiv.org/html/2606.24007#bib.bib1015)\)sequentially on three reasoning tasks: mathematical reasoning, code generation, and common\-sense reasoning\. The datasets are GSM8K, without additional augmentation, with 7\.8k training examples and 1\.3k test examples; Alpaca Code, with roughly 20k training samples; and ARC\-Challenge, which we use as the common\-sense reasoning task, with 7\.8k question–answer pairs\. For evaluation, we use lm\-eval\-harness\(Gaoet al\.,[2024](https://arxiv.org/html/2606.24007#bib.bib1016)\)as a unified benchmark pipeline\. We evaluate its test sets and set the generation length to 1024 tokens\.

The math dataset is trained for33epochs with learning rate3×10−33\\times 10^\{\-3\}, batch size44, gradient accumulation88, and maximum sequence length512512\. The code stage is trained for22epochs with learning rate3×10−13\\times 10^\{\-1\}, batch size22, gradient accumulation1616, and maximum sequence length10241024\. The ARC stage is trained for33epochs with learning rate3×10−13\\times 10^\{\-1\}, batch size44, gradient accumulation88, and maximum sequence length256256\. We set the initial Hessian to1×10−11\\times 10^\{\-1\}, the effective sample size to1×1091\\times 10^\{9\}, the clipping radius to1×10−21\\times 10^\{\-2\}, and the momentum parameters toβ1=0\.9\\beta\_\{1\}=0\.9andβ2=0\.99995\\beta\_\{2\}=0\.99995on the first task\. At the transition to the code task, we set hess to5×10−15\\times 10^\{\-1\}andλ=1×107\\lambda=1\\times 10^\{7\}\. At the transition to the ARC task, we again set hess to5×10−15\\times 10^\{\-1\}, withλ=1×106\\lambda=1\\times 10^\{6\}\. Across all stages, we use evaluation batch size44and warmup ratio0\.030\.03

We compared CoVON with standard baselines such as AdamW and IVON finetuning \(AdamW\-FT / IVON\-FT\), Ada\-Reg, a deterministic variant of CoVON, and finally regularization techniques such as EWC​\\\*\. In each column of[Table˜2](https://arxiv.org/html/2606.24007#A3.T2), we report the performance of the model on each new task and its ability to retain the ability to solve math tasks after learning a new skill\. CoVON shows the best retention ability \(61%\) after it has been finetuned on common\-sense reasoning skills\. Note, we don’t use any replay for any of our methods and only opted for pure Supervised Fine\-Tuning\. It establishes CoVON as a robust and scalable optimization algorithm for LLM models\.

Compute Resources:All experiments were conducted on four NVIDIA A600 GPUs, each with 24,GB of memory\. A single run on DomainNet, CDDB, and CORe50 took approximately 300, 120, and 120 minutes, respectively\. For language\-model experiments, continual pretraining required about 900 minutes per run, while continual fine\-tuning took approximately 100 minutes\.

Table 2:Continual Fine\-tuning of Qwen3\-1\.7B\-Base on Math \(GSM8K\)→\\rightarrowCode \(Code\-Alpaca 20K\)→\\rightarrowCommon\-sense reasoning \(Ai2\_\\\_arc\)\. We specifically evaluate CoVON’s ability to retain mathematical problem\-solving capabilities after the model is sequentially fine\-tuned on code generation and common\-sense question\-answering tasks\. Each column reports performance on the new task and on the math task\. We observe that AdamW exhibits significantly higher forgetting than CoVON\. After fine\-tuning on the code, AdamW suffers a nearly 30% drop in math performance, compared to only 3% for CoVON\. A similar trend is observed after finetuning on the common\-sense dataset thereafter\.Table 3:Comparison of accuracy, runtime overhead, and GPU memory usage in the domain\-incremental setting\. We report the final averaged accuracy together with wall\-clock overhead measured per batch, per epoch, and per task, as well as peak GPU memory consumption in GB\. While EWC and EWC∗require substantial task\-level overhead due to additional post\-hoc computations, CoVON remains nearly as efficient as standard fine\-tuning and Ada\-Reg, with only marginal increases in per\-batch and per\-epoch time and no meaningful extra memory cost\. Despite this comparable computational footprint, CoVON achieves the highest average accuracy, demonstrating a substantially better efficiency–performance trade\-off than existing regularization\-based baselines\.
### C\.5Computational Overhead and Memory Cost

In[Table˜3](https://arxiv.org/html/2606.24007#A3.T3), we compare the runtime and memory cost of CoVON against standard continual learning baselines in the domain\-incremental setting\. CoVON achieves the highest average accuracy of 56\.75%, improving over Ada\-Reg \(49\.75%\) and outperforming finetuning \(28\.48%\) and EWC\-based methods\. At the same time, its computational overhead remains very small: the per\-batch time is 0\.93s and the per\-epoch time is 36\.5s, which are nearly identical to AdamW\-Reg \(0\.92s and 36\.0s\) and only slightly above plain finetuning \(0\.89s and 31\.89s\)\. Its total per\-task cost is 537s, again very close to AdamW\-Reg \(534s\), while EWC and EWC\-Corrected require much larger task\-level overheads of 1956s and 1789s, respectively, due to additional post\-hoc computations\. In terms of memory, CoVON uses 16\.57 GB of GPU memory, matching AdamW\-Reg and only slightly exceeding finetuning and EWC \(16\.13 GB\)\. Overall, the table shows that CoVON improves accuracy without introducing meaningful runtime or memory overhead, providing a substantially better efficiency–performance trade\-off than existing regularization\-based baselines\.