ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

# ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models
Source: [https://arxiv.org/html/2606.28719](https://arxiv.org/html/2606.28719)
Zihan Zhai1 &Hongwei Yan1 &Hang Su5 &Jun Zhu5,∗ &Yi Zhong1,∗ & 1School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China 2Institute of Software Chinese Academy of Sciences, Beijing, China 3Beijing Academy of Artificial Intelligence, Beijing, China 4Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China 5Dept\. of Comp\. Sci\. and Tech\., Institute for AI, Tsinghua\-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, China 6These authors contributed equally ∗Correspondence:dcszj@tsinghua\.edu\.cn, zhongyithu@tsinghua\.edu\.cn

###### Abstract

Test\-time adaptation \(TTA\) of vision\-language models \(VLMs\) is essential for their robust deployment in dynamic, real\-world environments\. However, existing TTA methods often adapt locally without accumulating knowledge over time, or operating within a single modality without exploiting VLMs’ inherently multi\-modal nature\. Inspired by theComplementaryMemory systems of the biological brain, we proposeComMem, an innovative approach that mimics the distinct but cooperative roles of the hippocampus and neocortex to enable effective TTA for VLMs\. ComMem consists of two key components: a fast\-adapting detailed memory, akin to the hippocampus, that forms a dynamic visual cache from high\-confidence test samples; and a slow\-integrating abstract memory, akin to the neocortex, that continually refines global textual prototypes\. For each test instance, ComMem jointly optimizes both memory systems to ensure cross\-modal consistency\. Extensive experiments on 15 benchmark datasets show that ComMem significantly outperforms state\-of\-the\-art methods under both natural distribution shifts and cross\-dataset generalization, offering a promising direction for enhancing VLMs’ practical adaptability\.

## 1Introduction

The advent of pioneering vision\-language models \(VLMs\), such as CLIP\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\), has transformed computer vision by enabling strong zero\-shot generalization through learning from vast image\-text corpora\. Despite their impressive capabilities, these models often suffer significant performance degradation when deployed in real\-world environments where the test\-time data distribution diverges from their pre\-training data distribution\(Agarwal et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib2); Menon et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib29)\)\. To address this challenge, test\-time adaptation \(TTA\) has emerged as a powerful paradigm, allowing pre\-trained models to adapt to different target domains using only a stream of unlabeled test samples\(Liang et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib23); Dong et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib8)\)\.

Recent TTA efforts for VLMs have evolved from memoryless schemes, in which models are optimized independently for each sample or mini\-batch, to more advanced memory\-based approaches111Due to the page limit, we present a more comprehensive summary of related work in Appendix[A](https://arxiv.org/html/2606.28719#A1)\.\. Early methods, exemplified by TPT\(Shu et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib38)\)and its variants such as DiffTPT\(Feng et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib11)\)and SwapPrompt\(Ma et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib25)\), operate in an “amnesic” manner by adapting to each test instance in isolation without retaining experience over time \(Fig\.[1](https://arxiv.org/html/2606.28719#S1.F1)A\)\. Recognizing this limitation, subsequent studies introduced memory mechanisms, which can be broadly categorized by their adaptive component: some accumulate knowledge at the textual level through prompt tuning, such as HisTPT\(Zhang et al\.,[2024b](https://arxiv.org/html/2606.28719#bib.bib48)\)and DynaPrompt\(Xiao et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib45)\)\(Fig\.[1](https://arxiv.org/html/2606.28719#S1.F1)B\), whereas others maintain dynamic caches at the visual representation level, such as TDA\(Karmanov et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib20)\)and DMN\-ZS\(Zhang et al\.,[2024c](https://arxiv.org/html/2606.28719#bib.bib50)\)\(Fig\.[1](https://arxiv.org/html/2606.28719#S1.F1)C\)\. However, restricting adaptation to single modality limits the model’s ability to align visual and textual representations effectively\. Without coordinated cross\-modal updates, these methods fail to balance the complementary fast \(plastic\) and slow \(stable\) learning dynamics, thereby limiting the ability to rapidly adapt to new domain\-specific cues while preserving the robustness of pre\-trained knowledge under distribution shifts \(Fig\.[1](https://arxiv.org/html/2606.28719#S1.F1)D\-E\)\.

![Refer to caption](https://arxiv.org/html/2606.28719v1/x1.png)Figure 1:Comparison of ComMem with recent TTA methods\. Norm\., Normalization; Con\., Consolidation; and Recon\., Reconsolidation\.![Refer to caption](https://arxiv.org/html/2606.28719v1/x2.png)Figure 2:Overview of the complementary memory systems theory\(McClelland et al\.,[1995](https://arxiv.org/html/2606.28719#bib.bib28); Tonegawa et al\.,[2018](https://arxiv.org/html/2606.28719#bib.bib41); Lei et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib22)\)in neuroscience\.In neuroscience,*complementary memory systems*theory\(McClelland et al\.,[1995](https://arxiv.org/html/2606.28719#bib.bib28); Tonegawa et al\.,[2018](https://arxiv.org/html/2606.28719#bib.bib41); Lei et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib22)\), also known as*complementary learning systems*, has been widely acknowledged to achieve an integration of new experience into prior knowledge for optimal behavioral output\. This theory posits a functional synergy between two distinct memory systems \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)A\): the hippocampus \(HPC\) rapidly encodes episodic and detailed memories; the neocortex \(NC\) slowly forms semantic and generalizable knowledge\. Over time, memory traces are gradually transferred from the HPC to the NC through*systems consolidation*\(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)B\)\(Golbabaei & Frankland,[2025](https://arxiv.org/html/2606.28719#bib.bib12)\)process\. More importantly, a recent work revealed that recalled of an NC\-dependent memory can induce*memory reconsolidation*phase in HPC \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)C\), allowing the HPC to incorporate new information and reconstruct an memory trace\(Lei et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib22)\)\.

Motivated by this well\-established biological mechanism, we introduceComMem, an innovative TTA approach that instantiates the dynamics of complementary memory systems for robust VLMs adaptation \(Fig\.[3](https://arxiv.org/html/2606.28719#S2.F3), see pseudo\-code in Appendix Alg\.[1](https://arxiv.org/html/2606.28719#alg1)\)\. ComMem models the fast, plastic learning of the HPC as a detailed memory cache that rapidly stores and updates visual features from high\-confidence test samples\. In parallel, it models the NC’s slow, stable learning as an abstract memory composed of global textual prototypes, which evolve gradually through a process analogous to memory consolidation\. For each incoming test sample, ComMem simulates memory reconsolidation by retrieving from both memory systems and jointly optimizing them via learnable residuals to enforce cross\-modal consistency\. This enables the model not only to adapt to individual samples but also to progressively accumulate and refine domain knowledge across the test data stream\.

We evaluate ComMem with extensive experiments across 15 challenging benchmark datasets\. Our results demonstrate that ComMem consistently and significantly outperforms current state\-of\-the\-art methods in both robustness to natural distribution shifts \(e\.g\., ImageNet\-A/R/S\) and broad cross\-dataset generalization scenarios\. ComMem achieves a new state\-of\-the\-art average accuracy of 48\.84% on ImageNet\-OOD variants with ResNet\-50\(He et al\.,[2016](https://arxiv.org/html/2606.28719#bib.bib14)\)and 65\.36% with ViT\-B/16\(Dosovitskiy et al\.,[2020](https://arxiv.org/html/2606.28719#bib.bib9)\)\. Furthermore, our approach ensures both strong performance and efficiency, incurring substantially less computational overhead than prominent prompt\-tuning methods like TPT\(Shu et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib38)\)\. Detailed ablation studies confirm that the synergistic combination of the fast\-adapting detailed memory and the slow\-integrating abstract memory, alongside our reconsolidation process, is critical to its success\.

Our main contributions are as follows:

- •We introduce ComMem, a brain\-inspired approach to TTA of VLMs that, for the first time, operationalizes the distinct fast and slow learning dynamics underlying the brain’s complementary memory systems\.
- •We design a synergistic mechanism involving a fast\-updating detailed cache and a slow\-consolidating abstract memory, which are jointly optimized through a reconsolidation process for cross\-modal consistency\.
- •We conduct extensive experiments on 15 benchmark datasets, demonstrating that ComMem significantly outperforms state\-of\-the\-art TTA methods under distribution shifts and cross\-dataset generalization\.

## 2Preliminaries

In this section, we describe the problem formulation of zero\-shot VLMs classification, test\-time adaptation in VLMs, and our brain\-inspired motivations\.

### 2\.1Formulation

#### Zero\-Shot VLMs Classification

Representative VLMs such as CLIP\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)consist of a visual encoderℰv​\(⋅\)\\mathcal\{E\}\_\{v\}\(\\cdot\)and a textual encoderℰt​\(⋅\)\\mathcal\{E\}\_\{t\}\(\\cdot\)that project images and text into a shared, high\-dimensional embedding spaceℝD\\mathbb\{R\}^\{D\}\. For aCC\-way classification task, the class names\{yc\}c=1C\\\{y\_\{c\}\\\}\_\{c=1\}^\{C\}are embedded into textual features using prompt ensembling, e\.g\., “a photo of a \{CLASS\}”, yielding a text prototype matrix𝐏t∈ℝD×C\\mathbf\{P\}^\{t\}\\in\\mathbb\{R\}^\{D\\times C\}\.

For an input image𝐱\\mathbf\{x\}, its visual feature𝐟v=ℰv​\(𝐱\)\\mathbf\{f\}\_\{v\}=\\mathcal\{E\}\_\{v\}\(\\mathbf\{x\}\)is compared with the text features using cosine similarity, producing the zero\-shot class posterior:

p​\(y=c\|𝐱\)=exp⁡\(sim​\(𝐟v,𝐩ct\)/τ\)∑j=1Cexp⁡\(sim​\(𝐟v,𝐩jt\)/τ\),p\(y=c\|\\mathbf\{x\}\)=\\frac\{\\exp\(\\text\{sim\}\(\\mathbf\{f\}\_\{v\},\\mathbf\{p\}^\{t\}\_\{c\}\)/\\tau\)\}\{\\sum\_\{j=1\}^\{C\}\\exp\(\\text\{sim\}\(\\mathbf\{f\}\_\{v\},\\mathbf\{p\}^\{t\}\_\{j\}\)/\\tau\)\},\(1\)where𝐩ct\\mathbf\{p\}^\{t\}\_\{c\}is thecc\-th text prototype,sim​\(⋅,⋅\)\\text\{sim\}\(\\cdot,\\cdot\)denotes cosine similarity, andτ\\tauis the temperature hyperparameter\.

#### Test\-Time Adaptation in VLMs

We consider anonline streamingtest\-time adaptation \(TTA\) scenario, where a pre\-trained VLM\(ℰv,ℰt\)\(\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\}\)is deployed in a target domain with unlabeled test samples\{𝐱s\}s=1S\\\{\\mathbf\{x\}\_\{s\}\\\}\_\{s=1\}^\{S\}arriving sequentially\. No ground\-truth labels are available, and the target distribution𝒫tgt​\(x\)\\mathcal\{P\}\_\{\\text\{tgt\}\}\(x\)differs from the source distribution𝒫src​\(x\)\\mathcal\{P\}\_\{\\text\{src\}\}\(x\)\. The objective is to adapt the modelon\-the\-flyto minimize the expected target error:

minθ⁡𝔼𝐱∼𝒫tgt​\[ℓ​\(y^​\(𝐱;θ\),y\)\],\\min\_\{\\theta\}\\ \\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\mathcal\{P\}\_\{\\text\{tgt\}\}\}\\big\[\\ell\(\\hat\{y\}\(\\mathbf\{x\};\\theta\),y\)\\big\],\(2\)whereθ\\thetadenotes the parameters of the model and its auxiliary adaptation modules,ℓ​\(⋅\)\\ell\(\\cdot\)is the classification loss, andy^\\hat\{y\}represents the model’s prediction\.

Unlike regular domain adaptation, TTA operates without access to labeled target data\. In theonline streamingsetting, test samples arrive sequentially in a continuous stream\. Accordingly, the adaptation algorithm must satisfy the following dynamics:

𝐱s\\displaystyle\\mathbf\{x\}\_\{s\}∼𝒫tgt,\\displaystyle\\sim\\mathcal\{P\}\_\{\\text\{tgt\}\},\(3\)θs\\displaystyle\\theta\_\{s\}=Update​\(θs−1,𝐱s\),\\displaystyle=\\text\{Update\}\(\\theta\_\{s\-1\},\\mathbf\{x\}\_\{s\}\),\(4\)y^s\\displaystyle\\hat\{y\}\_\{s\}=f​\(𝐱s;θs\),\\displaystyle=f\(\\mathbf\{x\}\_\{s\};\\theta\_\{s\}\),\(5\)wheressindexes the time step,Update​\(⋅\)\\text\{Update\}\(\\cdot\)denotes an unsupervised adaptation rule based on model confidence, entropy, or consistency\.

A central challenge lies inrapidly adaptingto domain\-specific variations to mitigate error accumulation from self\-training noise, while simultaneouslymaintaining stabilityto prevent catastrophic forgetting\. To address this trade\-off, we draw inspirations from the robust biological brain, especially thecomplementary memory systemstheory\.

### 2\.2Brain\-Inspired Motivation

![Refer to caption](https://arxiv.org/html/2606.28719v1/x3.png)Figure 3:The proposed ComMem framework for test\-time adaptation of vision\-language models\.The remarkable adaptability of the biological brain arises from the interaction between the following two memory systems \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)A\)\(McClelland et al\.,[1995](https://arxiv.org/html/2606.28719#bib.bib28); Tonegawa et al\.,[2018](https://arxiv.org/html/2606.28719#bib.bib41); Lei et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib22)\):

- •Neocortex \(NC\):The NC encodes semantic, abstract knowledge \(e\.g\., the concept of a “dog”\) throughslow and integrative learning, ensuring knowledge stability and generalization across experiences\.
- •Hippocampus \(HPC\):The HPC rapidly captures specific, episodic experiences \(e\.g\., “the particular dog I saw in the park”\), supportingfast, instance\-based learningwith high plasticity\.

The key mechanisms underlying interactions of the two memory systems aresystems consolidationandsystems reconsolidation\. Duringsystems consolidation\(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)B\), episodic traces encoded in the HPC are gradually integrated into cortical representations within the NC, enabling the abstraction of long\-term generalizable knowledge\. Conversely, duringsystems reconsolidation\(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)C\), when novel stimuli are encountered, the NC retrieves existing abstract representation memory to interpret new episodic inputs from the HPC\. Both memory traces are then jointly updated, allowing the brain to maintain coherent and adaptive representations over time\.

Motivated by this effective synergy between fast and slow learning systems, our proposedComMemframework \(Fig\.[3](https://arxiv.org/html/2606.28719#S2.F3)\) computationally instantiates these dynamics for TTA in VLMs\. Specifically, we model adetailed memory cache, analogous to the HPC, that rapidly stores and updates high\-confidence visual features from the test data stream to enable quick adaptation; and anabstract memory systems, analogous to the NC, represented by slowly evolving textual prototypes that accumulate stable, generalizable knowledge\. For each incoming test sample, ComMem performs a computational analogue of memory consolidation and memory reconsolidation, where information from both memories is retrieved and jointly optimized to ensure cross\-modal alignment, followed by gradual consolidation of reliable updates into the abstract memory\.

## 3Method

The ComMem framework is illustrated in Fig\.[3](https://arxiv.org/html/2606.28719#S2.F3)\(see pseudo\-code in Appendix Alg\.[1](https://arxiv.org/html/2606.28719#alg1)\)\. We first introduce the two complementary memory components and their distinct update mechanisms, then detail the joint optimization process that computationally simulates complementary memory systems\.

### 3\.1Visual Representation Encoding

To facilitate rapid, instance\-based adaptation, ComMem maintains an episodic memory cacheℳ\\mathcal\{M\}that stores class\-conditional visual instances as feature–entropy pairs\{\(𝐟i,ℋi\)\}i=1\|ℳc\|\\\{\(\\mathbf\{f\}\_\{i\},\\mathcal\{H\}\_\{i\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{M\}\_\{c\}\|\}, where each classccretains up toKKentries ranked by confidence \(i\.e\., inverse entropy\)\. This cache serves as the HPC\-like memory, corresponding to precise, high\-certainty visual representations from the target domain\.

#### Entropy\-Guided Encoding\.

For each test image𝐱t\\mathbf\{x\}\_\{t\}, we generate a set ofNNrandomly augmented views\{𝐱t\(i\)\}i=1N\\\{\\mathbf\{x\}\_\{t\}^\{\(i\)\}\\\}\_\{i=1\}^\{N\}and extract their visual embeddings𝐟v\(i\)=ℰv​\(𝐱t\(i\)\)\\mathbf\{f\}\_\{v\}^\{\(i\)\}=\\mathcal\{E\}\_\{v\}\(\\mathbf\{x\}\_\{t\}^\{\(i\)\}\)using the frozen visual encoderℰv\\mathcal\{E\}\_\{v\}\. Each embedding is passed through the classifier \(Eq\.[1](https://arxiv.org/html/2606.28719#S2.E1)\) to obtain the predicted probability distributionp\(i\)=p​\(y\|𝐟v\(i\)\)p^\{\(i\)\}=p\(y\|\\mathbf\{f\}\_\{v\}^\{\(i\)\}\)\. The prediction entropyℋ\(i\)\\mathcal\{H\}^\{\(i\)\}of each view is then computed as:

ℋ\(i\)=−∑c=1Cpc\(i\)​log⁡pc\(i\),\\mathcal\{H\}^\{\(i\)\}=\-\\sum\_\{c=1\}^\{C\}p^\{\(i\)\}\_\{c\}\\log p^\{\(i\)\}\_\{c\},\(6\)whereCCis the number of classes and lower entropy indicates higher prediction confidence\.

To ensure that only reliable representations contribute to adaptation, we perform a confidence\-weighted aggregation of these augmented views:

w\(i\)=exp⁡\(−ℋ\(i\)/τe\)∑j=1Nexp⁡\(−ℋ\(j\)/τe\),𝐟v∗=∑i=1Nw\(i\)​𝐟v\(i\),w^\{\(i\)\}=\\frac\{\\exp\(\-\\mathcal\{H\}^\{\(i\)\}/\\tau\_\{e\}\)\}\{\\sum\_\{j=1\}^\{N\}\\exp\(\-\\mathcal\{H\}^\{\(j\)\}/\\tau\_\{e\}\)\},\\quad\\mathbf\{f\}\_\{v\}^\{\*\}=\\sum\_\{i=1\}^\{N\}w^\{\(i\)\}\\mathbf\{f\}\_\{v\}^\{\(i\)\},\(7\)whereτe\\tau\_\{e\}is an entropy temperature controlling the sharpness of the weighting distribution\. The resulting𝐟v∗\\mathbf\{f\}\_\{v\}^\{\*\}represents an entropy\-refined, confidence\-weighted embedding of the target instance, while its pseudo\-labely^=argmaxp\(⋅\|𝐟v∗\)\\hat\{y\}=\\arg\\max p\(\\cdot\|\\mathbf\{f\}\_\{v\}^\{\*\}\)is subsequently used to update the detailed memory cache\.

### 3\.2HPC Detailed Memory Fast Adaptation

For an incoming feature𝐟v∗\\mathbf\{f\}\_\{v\}^\{\*\}with pseudo\-labely^\\hat\{y\}and entropyℋ\\mathcal\{H\}, the episodic cacheℳ\\mathcal\{M\}is updated as follows:

- •If the cache is not full \(\|ℳy^\|<K\|\\mathcal\{M\}\_\{\\hat\{y\}\}\|<K\):The new feature–entropy pair\(𝐟v∗,ℋ\)\(\\mathbf\{f\}\_\{v\}^\{\*\},\\mathcal\{H\}\)is directly added to the cache of classy^\\hat\{y\}\.
- •If the cache is full \(\|ℳy^\|=K\|\\mathcal\{M\}\_\{\\hat\{y\}\}\|=K\):The cache entries for classy^\\hat\{y\}are sorted by their entropy\. Let\(𝐟max,ℋmax\)\(\\mathbf\{f\}\_\{\\max\},\\mathcal\{H\}\_\{\\max\}\)denote the entry with the highest entropy \(i\.e\., lowest confidence\)\. If the new feature is less confident \(ℋ\>ℋmax\\mathcal\{H\}\>\\mathcal\{H\}\_\{\\max\}\), the cache remains unchanged\. Otherwise, it replaces this entry through aninstance\-level evolutionstep\.

#### HPC Memory Update Rule\.

Before replacing the least confident entry\(𝐟max,ℋmax\)\(\\mathbf\{f\}\_\{\\max\},\\mathcal\{H\}\_\{\\max\}\), the incoming high\-confidence feature𝐟v∗\\mathbf\{f\}\_\{v\}^\{\*\}is evolved to form a more robust representation\. In this step, only the pair withℋmax\\mathcal\{H\}\_\{\\max\}is updated, and the prototype of its pseudo\-label class is computed as𝐩y^v=mean​\(\{𝐟i∈ℳy^\}\)\\mathbf\{p\}^\{v\}\_\{\\hat\{y\}\}=\\text\{mean\}\(\\\{\\mathbf\{f\}\_\{i\}\\in\\mathcal\{M\}\_\{\\hat\{y\}\}\\\}\)\. The final stored representation𝐟v′\\mathbf\{f\}^\{\\prime\}\_\{v\}is obtained via a weighted fusion of the class prototype and the new instance:

𝐟v′=\(1−λhpc\)​𝐩y^v\+λhpc​𝐟v∗,\\mathbf\{f\}^\{\\prime\}\_\{v\}=\(1\-\\lambda\_\{\\text\{hpc\}\}\)\\mathbf\{p\}^\{v\}\_\{\\hat\{y\}\}\+\\lambda\_\{\\text\{hpc\}\}\\mathbf\{f\}\_\{v\}^\{\*\},\(8\)whereλhpc\\lambda\_\{\\text\{hpc\}\}controls the update plasticity\. A largerλhpc\\lambda\_\{\\text\{hpc\}\}corresponds to afast, instance\-dominant update, while a smaller value leads to aslow, prototype\-dominant integration\. The updated pair\(𝐟v′,ℋ\)\(\\mathbf\{f\}^\{\\prime\}\_\{v\},\\mathcal\{H\}\)then replaces\(𝐟max,ℋmax\)\(\\mathbf\{f\}\_\{\\max\},\\mathcal\{H\}\_\{\\max\}\)inℳy^\\mathcal\{M\}\_\{\\hat\{y\}\}\.

#### Detailed Memory Classifier Generation\.

After each update, the complete set of visual prototypes𝐏v=\[𝐩1v,…,𝐩Cactivev\]∈ℝD×Cactive\\mathbf\{P\}^\{v\}=\[\\mathbf\{p\}^\{v\}\_\{1\},\\dots,\\mathbf\{p\}^\{v\}\_\{C\_\{\\text\{active\}\}\}\]\\in\\mathbb\{R\}^\{D\\times C\_\{\\text\{active\}\}\}is re\-computed by averaging all features currently stored in each active class\. This matrix𝐏v\\mathbf\{P\}^\{v\}serves as the base classifier for the detailed \(HPC\-like\) memory and is subsequently refined by a learnable residual termδv\\delta\_\{v\}during the test\-time optimization stage \(see Sec\.[3\.4](https://arxiv.org/html/2606.28719#S3.SS4)\)\.

### 3\.3NC Abstract Memory Slow Integration

To model the stable, semantic knowledge of the neocortex, theAbstract Memoryis represented by a set of global textual prototypes𝐏t∈ℝD×C\\mathbf\{P\}^\{t\}\\in\\mathbb\{R\}^\{D\\times C\}\. This component reflects the model’s pre\-trained, generalized knowledge and undergoes aslow integrationprocess during test time\.

#### NC Memory Update Rule\.

This gradual update mechanism integrates newly acquired task\-specific knowledge back into the stable abstract memory, analogous to systems consolidation in neuroscience\. The update is triggeredafterthe test\-time optimization step \(Sec\.[3\.4](https://arxiv.org/html/2606.28719#S3.SS4)\) for each test instance𝐱t\\mathbf\{x\}\_\{t\}\. If the final prediction is sufficiently confident \(i\.e\., prediction entropy is below a thresholdτconf\\tau\_\{\\text\{conf\}\}\), the locally optimized prototype𝐏^localt\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\mathrm\{local\}\}\(defined in Eq\.[11](https://arxiv.org/html/2606.28719#S3.E11)\) is used to update the global abstract memory𝐏t\\mathbf\{P\}^\{t\}\. The update follows a weighted fusion controlled by the rateλnc\\lambda\_\{\\text\{nc\}\}:

𝐏t←\(1−λnc\)​𝐏t\+λnc​𝐏^localt,\\mathbf\{P\}^\{t\}\\leftarrow\(1\-\\lambda\_\{\\text\{nc\}\}\)\\mathbf\{P\}^\{t\}\+\\lambda\_\{\\text\{nc\}\}\\hat\{\\mathbf\{P\}\}\_\{\\mathrm\{local\}\}^\{t\},\(9\)whereλnc\\lambda\_\{\\text\{nc\}\}is the update rate\. A value approaching 0 results in aslow, stable integrationthat preserves the stability of pre\-trained knowledge \(an NC trait\)\. Conversely, a value approaching 1 leads to afast, unstable update, risking catastrophic forgetting\. This ensures𝐏t\\mathbf\{P\}^\{t\}evolves gradually while slowly absorbing new, high\-confidence information from the target domain\.

### 3\.4Test\-Time Optimization

For each test sample, ComMem retrieves and integrates both memory systems, performing a single\-step optimization over a small set of learnable parameters to achieve adaptive yet stable prediction\. These parameters correspond to distinct brain\-inspired modules with fast and slow learning rates:

- •Visual Residuals \(δv∈ℝD×Cactive\\delta\_\{v\}\\in\\mathbb\{R\}^\{D\\times C\_\{\\text\{active\}\}\}\):A learnable residual term added to the visual prototypes𝐏v\\mathbf\{P\}^\{v\}\(retrieved from the hippocampal\-like cache\) to construct a sample\-specific visual classifier: 𝐏^v=Normalize​\(𝐏v\+δv\)\.\\hat\{\\mathbf\{P\}\}^\{v\}=\\text\{Normalize\}\(\\mathbf\{P\}^\{v\}\+\\delta\_\{v\}\)\.\(10\)This component belongs to the fast\-learning HPC system, responsible for adapting episodic visual details, and is updated with a learning ratel​rvlr\_\{v\}\.
- •Textual Residuals \(δt∈ℝD×C\\delta\_\{t\}\\in\\mathbb\{R\}^\{D\\times C\}\):A learnable residual added to the global textual prototypes𝐏t\\mathbf\{P\}^\{t\}to yield a context\-adaptive local classifier: 𝐏^localt=Normalize​\(𝐏t\+δt\)\.\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\text\{local\}\}=\\text\{Normalize\}\(\\mathbf\{P\}^\{t\}\+\\delta\_\{t\}\)\.\(11\)This component is associated with the slow\-learning NC system, which encodes abstract, semantic knowledge, and is updated with a learning ratel​rtlr\_\{t\}\.
- •Normalization Layers:The parameters of layer normalization \(LN\) or batch normalization \(BN\) layers, denotedθn\\theta\_\{n\}, within the visual encoder are fine\-tuned to enhance low\-level feature adaptation: 𝐟^v=LNθn​\(𝐟v\)​or​BNθn​\(𝐟v\)\.\\hat\{\\mathbf\{f\}\}\_\{v\}=\\text\{LN\}\_\{\\theta\_\{n\}\}\(\\mathbf\{f\}\_\{v\}\)\\ \\text\{or\}\\ \\text\{BN\}\_\{\\theta\_\{n\}\}\(\\mathbf\{f\}\_\{v\}\)\.\(12\)These parameters belong to the NC system, capturing slower cortical adaptation of perceptual representations, and are updated with a learning ratel​rnlr\_\{n\}\.

#### Prediction Integration\.

Given a visual feature𝐟v\\mathbf\{f\}\_\{v\}, ComMem integrates predictions from both the abstract \(NC\) and detailed \(HPC\) systems:

𝐳final=𝐟v⊤​𝐏^localt\+𝒜​\(𝐟v⊤​𝐏^v\),\\mathbf\{z\}\_\{\\text\{final\}\}=\\mathbf\{f\}\_\{v\}^\{\\top\}\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\text\{local\}\}\+\\mathcal\{A\}\(\\mathbf\{f\}\_\{v\}^\{\\top\}\\hat\{\\mathbf\{P\}\}^\{v\}\),\(13\)where𝒜​\(s\)=α​exp⁡\(−β​\(1−s\)\)\\mathcal\{A\}\(s\)=\\alpha\\exp\(\-\\beta\(1\-s\)\)is an affinity function that adaptively weights the contribution of the detailed memory, following\(Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\)\. This integration mimics biological memory reconsolidation, where episodic and semantic traces jointly determine the final perception or decision\.

#### Joint Optimization Objective\.

The learnable parameters\(δt,δv,θn\)\(\\delta\_\{t\},\\delta\_\{v\},\\theta\_\{n\}\)are optimized in a single gradient step by minimizing a composite loss function:

ℒtotal=ℒent\+λalign​ℒalign\+λsparse​ℒsparse\.\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{ent\}\}\+\\lambda\_\{\\text\{align\}\}\\mathcal\{L\}\_\{\\text\{align\}\}\+\\lambda\_\{\\text\{sparse\}\}\\mathcal\{L\}\_\{\\text\{sparse\}\}\.\(14\)Each term is framed with neuro\-computational analogues:

- •Entropy Minimization \(ℒent\\mathcal\{L\}\_\{\\text\{ent\}\}\):Reduces the prediction uncertainty by minimizing the entropy of the final logits, promoting confident, stable decisions: ℒent=ℋ​\(Softmax​\(𝐳final/τ\)\)\.\\mathcal\{L\}\_\{\\text\{ent\}\}=\\mathcal\{H\}\\\!\\left\(\\text\{Softmax\}\(\\mathbf\{z\}\_\{\\text\{final\}\}/\\tau\)\\right\)\.\(15\)This mirrors the neural process of uncertainty resolution during memory updating\(Mason et al\.,[2017](https://arxiv.org/html/2606.28719#bib.bib27); Radvansky et al\.,[1995](https://arxiv.org/html/2606.28719#bib.bib36)\)\.
- •Inter\-Memory Consistency \(ℒalign\\mathcal\{L\}\_\{\\text\{align\}\}\):Encourages coherence between the visual and textual representations by aligning their respective prototypes via a symmetric InfoNCE loss\(Oord et al\.,[2018](https://arxiv.org/html/2606.28719#bib.bib31)\): ℒalign=−1Cactive∑c=1Cactive\(log⁡exp⁡\(𝐩^t,c⊤​𝐩^v,c/τc\)∑j=1Cactiveexp⁡\(𝐩^t,c⊤​𝐩^v,j/τc\)\+logexp⁡\(𝐩^v,c⊤​𝐩^t,c/τc\)∑j=1Cactiveexp⁡\(𝐩^v,c⊤​𝐩^t,j/τc\)\)\.\\begin\{split\}\\mathcal\{L\}\_\{\\text\{align\}\}=\-\\frac\{1\}\{C\_\{\\text\{active\}\}\}\\sum\_\{c=1\}^\{C\_\{\\text\{active\}\}\}\\Biggl\(&\\log\\frac\{\\exp\(\\hat\{\\mathbf\{p\}\}\_\{t,c\}^\{\\top\}\\hat\{\\mathbf\{p\}\}\_\{v,c\}/\\tau\_\{c\}\)\}\{\\sum\_\{j=1\}^\{C\_\{\\text\{active\}\}\}\\exp\(\\hat\{\\mathbf\{p\}\}\_\{t,c\}^\{\\top\}\\hat\{\\mathbf\{p\}\}\_\{v,j\}/\\tau\_\{c\}\)\}\\\\ &\+\\log\\frac\{\\exp\(\\hat\{\\mathbf\{p\}\}\_\{v,c\}^\{\\top\}\\hat\{\\mathbf\{p\}\}\_\{t,c\}/\\tau\_\{c\}\)\}\{\\sum\_\{j=1\}^\{C\_\{\\text\{active\}\}\}\\exp\(\\hat\{\\mathbf\{p\}\}\_\{v,c\}^\{\\top\}\\hat\{\\mathbf\{p\}\}\_\{t,j\}/\\tau\_\{c\}\)\}\\Biggr\)\.\\end\{split\}\(16\)This term maintains consistency between abstract \(NC\) and detailed \(HPC\) representations, ensuring coherent reconsolidation\(Colyer & Kahana,[2025](https://arxiv.org/html/2606.28719#bib.bib5)\)\.
- •Intra\-Memory Sparse Coding \(ℒsparse\\mathcal\{L\}\_\{\\text\{sparse\}\}\):Inspired by sparse neural coding\(Wixted et al\.,[2014](https://arxiv.org/html/2606.28719#bib.bib43); Palm,[2013](https://arxiv.org/html/2606.28719#bib.bib32)\), which minimizes representational overlap across memory traces, this term penalizes prototype redundancy within each memory system\. For a prototype matrix𝐏∈\{𝐏^localt,𝐏^v\}\\mathbf\{P\}\\in\\\{\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\text\{local\}\},\\hat\{\\mathbf\{P\}\}^\{v\}\\\}: ℒsparse​\(𝐏\)=1C​\(C−1\)​∑i≠j\(𝐩i⊤​𝐩j\)2\.\\mathcal\{L\}\_\{\\text\{sparse\}\}\(\\mathbf\{P\}\)=\\frac\{1\}\{C\(C\-1\)\}\\sum\_\{i\\neq j\}\(\\mathbf\{p\}\_\{i\}^\{\\top\}\\mathbf\{p\}\_\{j\}\)^\{2\}\.\(17\)

The optimization is executed with a single\-step update using the AdamW optimizer\(Loshchilov & Hutter,[2017](https://arxiv.org/html/2606.28719#bib.bib24)\)\. After adaptation, the refined residuals yield the final prediction for the current sample\. If the prediction confidence surpasses a predefined threshold, the global semantic prototypes𝐏t\\mathbf\{P\}^\{t\}are further updated via Eq\.[9](https://arxiv.org/html/2606.28719#S3.E9), preparing the system for subsequent samples in the test data stream\.

Table 1:Performance comparison under natural distribution shifts\.Top\-1 accuracy \(%\) of all evaluated methods using ResNet\-50 and ViT\-B/16 as CLIP vision backbones\. The best and second\-best results are marked inboldandunderlined, respectively\.

## 4Experiments

In this section, we describe the experimental setups \(further detailed in Appendix Sec\.[B\.1](https://arxiv.org/html/2606.28719#A2.SS1)\), including datasets, implementation details and baselines, and then present the experimental results with an in\-depth analysis\.

### 4\.1Performance Evaluation

#### Robustness to Natural Distribution Shifts\.

In Table[1](https://arxiv.org/html/2606.28719#S3.T1), we present the performance comparison on ImageNet and its variants\. Our proposed ComMem consistently achieves state\-of\-the\-art performance across both CLIP\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)backbones\. With the ResNet\-50\(He et al\.,[2016](https://arxiv.org/html/2606.28719#bib.bib14)\)backbone, ComMem attains an average Out\-of\-Distribution \(OOD\) accuracy of 48\.84%, surpassing the next\-best method, DPE\(Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\), which scores 47\.66%\. A similar trend is observed with the ViT\-B/16\(Dosovitskiy et al\.,[2020](https://arxiv.org/html/2606.28719#bib.bib9)\)backbone, where ComMem achieves an average OOD accuracy of 65\.36%, outperforming all other competing methods\. These results underscore ComMem’s superior ability to generalize to unseen domains throughout training\.

Table 2:Performance comparisons on cross\-dataset generalization\.Top\-1 accuracy \(%\) of all evaluated methods using two CLIP vision backbones\. The best and second\-best results are marked inboldandunderlined, respectively\. Results for DynaPrompt\(Xiao et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib45)\)on Cars\(Krause et al\.,[2013](https://arxiv.org/html/2606.28719#bib.bib21)\)and SUN397\(Xiao et al\.,[2010](https://arxiv.org/html/2606.28719#bib.bib44)\)are marked as “–” due to out\-of\-memory errors during reproduction\.
#### Cross\-Dataset Generalization\.

We further assess the generalization of ComMem on 10 diverse datasets, with results summarized in Table[2](https://arxiv.org/html/2606.28719#S4.T2)\. The significant distributional differences across these datasets pose a substantial challenge for adaptation methods\. Nevertheless, ComMem demonstrates remarkable robustness\. With the ResNet\-50\(He et al\.,[2016](https://arxiv.org/html/2606.28719#bib.bib14)\)backbone, ComMem again leads with an average accuracy of 64\.60%, outperforming the strong baselines DMN\-ZS\(Zhang et al\.,[2024c](https://arxiv.org/html/2606.28719#bib.bib50)\)of 63\.32% and DPE\(Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\)of 61\.93%\. Similarly, with the ViT\-B/16\(Dosovitskiy et al\.,[2020](https://arxiv.org/html/2606.28719#bib.bib9)\)backbone, ComMem achieves the highest average accuracy of 70\.74%\. The consistent performance gains highlight the adaptability and effectiveness of our brain\-inspired framework\.

### 4\.2Ablation Studies

#### Effects of Learnable Modules\.

Fig\.[4](https://arxiv.org/html/2606.28719#S4.F4)A presents a progressive ablation of the learnable components in our dual\-memory architecture\. Starting from the base CLIP\-ResNet\-50\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)model, we incrementally enable three modules: inter\-memory consistency, intra\-memory sparsity, and normalization adaptation\. Each additional component leads to a consistent improvement across all datasets, demonstrating their complementary contributions\.

![Refer to caption](https://arxiv.org/html/2606.28719v1/x4.png)Figure 4:Ablation and hyperparameter analysis of ComMem\.\(A\) Effects of learnable modules\. \(B\) Comparison of learning rates for residuals layers \(l​rtlr\_\{t\}andl​rvlr\_\{v\}\) and normalization layers \(l​rnlr\_\{n\}\)\. \(C\) Influence of HPC\-like memory cache capacityKK\. RN50 and ViT16 denote CLIP\-ResNet\-50\(He et al\.,[2016](https://arxiv.org/html/2606.28719#bib.bib14)\)and CLIP\-ViT\-B/16\(Dosovitskiy et al\.,[2020](https://arxiv.org/html/2606.28719#bib.bib9)\)backbones, respectively\.
#### Learning Rate of Complementary Memory Systems\.

As shown in Fig\.[4](https://arxiv.org/html/2606.28719#S4.F4)B, we examine the interaction between the learning rates of the residual layers \(l​rvlr\_\{v\}forδv\\delta\_\{v\}andl​rtlr\_\{t\}forδt\\delta\_\{t\}, withl​rv=l​rtlr\_\{v\}=lr\_\{t\}\) and the normalization layers \(l​rnlr\_\{n\}forθn\\theta\_\{n\}\) using CLIP\-ResNet\-50\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)backbone\. The results reveal distinct optimal learning\-rate regimes for these two components\. The normalization layers achieve peak performance at relatively small learning rates \(e\.g\.,10−6∼10−510^\{\-6\}\\\!\\sim\\\!10^\{\-5\}\), aligning with their role in slow and stable integration of visual representation knowledge\. In contrast, the residual layers perform best with larger learning rates \(on the order of10−410^\{\-4\}\), supporting rapid adaptation to episodic inputs\. This clear difference in optimal magnitude reflects the complementary dynamics between the neocortical and hippocampal memory systems\.

#### HPC\-like Memory Capacity\.

Fig\.[4](https://arxiv.org/html/2606.28719#S4.F4)C analyzes the effect of HPC\-like memory capacityKKacross different datasets and backbones\. We observe that performance consistently peaks whenKKis between 3 and 5\. Smaller capacities limit the diversity of stored instances, while larger ones introduce redundancy and noise\. This moderate optimal capacity suggests that a compact but selective detailed memory is most effective for balancing flexibility and stability during test\-time adaptation\.

### 4\.3Extended Analysis

#### Efficiency Comparison\.

Table 3:Efficiency comparison on ImageNet\(Deng et al\.,[2009](https://arxiv.org/html/2606.28719#bib.bib7)\)\. We report the testing time, the achieved accuracy, and the performance gains compared to zero\-shot CLIP\.In Table[3](https://arxiv.org/html/2606.28719#S4.T3), we analyze the computational efficiency of ComMem by measuring the total time required to process the 50,000 test samples of ImageNet\(Deng et al\.,[2009](https://arxiv.org/html/2606.28719#bib.bib7)\)\. ComMem achieves its state\-of\-the\-art accuracy in around 1\.5 hours\. This is substantially more efficient than prompt\-tuning methods like TPT\(Shu et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib38)\)and DiffTPT\(Feng et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib11)\)\(\>\>15 hours\), which require computationally intensive backpropagation through the text encoder for every sample\. While training\-free methods like TDA\(Karmanov et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib20)\)\(53 minutes\) are faster, ComMem offers a far greater performance gain \(a \+4\.24% accuracy gain for ComMem vs\. \+1\.54% for TDA\(Karmanov et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib20)\)\)\. With only 4\.08M learnable parameters, ComMem strikes an excellent balance between high accuracy and practical test\-time efficiency\.

Table 4:Comparison of different memory update rules for the HPC\-like and NC\-like memory systems on the StanfordCars\(Krause et al\.,[2013](https://arxiv.org/html/2606.28719#bib.bib21)\)\.Each rule determines how a new feature𝐟v∗\\mathbf\{f\}\_\{v\}^\{\*\}\(for HPC, Sec[3\.2](https://arxiv.org/html/2606.28719#S3.SS2.SSS0.Px1)\) or a locally optimized prototype𝐏^localt\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\mathrm\{local\}\}\(for NC, Sec[3\.3](https://arxiv.org/html/2606.28719#S3.SS3.SSS0.Px1)\) is integrated with the existing memory representation \(𝐩y^v\\mathbf\{p\}^\{v\}\_\{\\hat\{y\}\}or𝐏t\\mathbf\{P\}^\{t\}\)\. Results are reported using CLIP\-ResNet\-50\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)with a memory capacity of 5 entries per class\.
#### Different Memory Update Rules\.

As described in Sec\.[3\.2](https://arxiv.org/html/2606.28719#S3.SS2.SSS0.Px1)and[3\.3](https://arxiv.org/html/2606.28719#S3.SS3.SSS0.Px1), we evaluate different memory update rules between two memory systems\. As shown in Table[4](https://arxiv.org/html/2606.28719#S4.T4), the HPC\-like detailed memory𝐏v\\mathbf\{P\}^\{v\}achieves optimal performance underfast update rules, such as full updates or fast exponential averaging, confirming its role as a rapidly adapting, high\-plasticity system\. In contrast, the NC\-like abstract memory𝐏t\\mathbf\{P\}^\{t\}performs best withslow update rules\(e\.g\., cumulative or exponential averaging with smallλnc\\lambda\_\{\\text\{nc\}\}\), which promote gradual and stable integration\. These complementary findings validate our dual\-memory design: the HPC benefits from fast learning for flexible adaptation, whereas the NC evolves slowly to maintain semantic stability\.

#### Visualization of cache features using t\-SNE\.

Fig\.[5](https://arxiv.org/html/2606.28719#S4.F5)and Appendix Fig\.[6](https://arxiv.org/html/2606.28719#A2.F6)provides t\-SNE visualizations of the HPC\-like detailed memoryPvP^\{v\}as it evolves\. Compared to DPE\(Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\), the feature clusters of ComMem become significantly more compact and more clearly separated by iteration 7000\. This indicates that our ComMem successfully learns more discriminative class representations over time, validating our complementary memory mechanism\.

![Refer to caption](https://arxiv.org/html/2606.28719v1/x5.png)Figure 5:t\-SNE visualizations of the HPC\-like detailed memory \(with cache size K = 20\) over time using CLIP\-ResNet\-50\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)on EuroSAT\(Helber et al\.,[2019](https://arxiv.org/html/2606.28719#bib.bib15)\)\.

## 5Conclusion

In this paper, we introduced ComMem, a versatile and effective test\-time adaptation framework for VLMs inspired by the brain’s complementary memory systems\. By modeling a fast\-adapting HPC\-like detailed memory and a slow\-consolidating NC\-like abstract memory, our approach successfully accumulates and refines multi\-modal knowledge from an unlabeled test data stream\. The core of our method lies in mimicking systems consolidation and systems reconsolidation, where both memory systems are jointly optimized for each sample to enforce consistency and produce robust predictions\. Extensive evaluations on 15 diverse benchmarks demonstrate that ComMem sets a new state\-of\-the\-art, consistently outperforming existing methods in handling natural distribution shifts and generalizing across datasets\.

Despite its strong performance, our work has limitations\. The use of dynamic memory cache and per\-sample optimization slightly increases the computational and memory overhead compared to zero\-shot inference, a trade\-off for improved accuracy\. Future work could explore more efficient implementations of the memory systems, perhaps through sparse or quantized representations\. Furthermore, delving deeper into more nuanced brain\-inspired mechanisms, such as the process of memory forgetting to prune irrelevant information, could pave the way for even more robust and intelligent adaptation systems\.

#### Acknowledgment

This work is supported by the Beijing Major Science and Technology Project \(No\. Z251100008425003\), the STI2030\-Major Projects \(No\. 2022ZD0204900\), the NSFC Projects \(Nos\. 62406160, 92370124, U25B6003, 62350080, 62595773\), the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China \(No\. JYB2025XDXM101\), Beijing Natural Science Foundation \(No\. L247011\), the Shandong Provincial Natural Science Foundation \(No\. ZR2022ZD01\), and the High Performance Computing Center, Tsinghua University\.

## References

- Abdul Samadh et al\. \(2023\)Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan\.Align your prompts: Test\-time prompting with distribution alignment for zero\-shot generalization\.*Advances in Neural Information Processing Systems*, 36:80396–80413, 2023\.
- Agarwal et al\. \(2021\)Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage\.Evaluating clip: towards characterization of broader capabilities and downstream implications\.*arXiv preprint arXiv:2108\.02818*, 2021\.
- Bossard et al\. \(2014\)Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool\.Food\-101–mining discriminative components with random forests\.In*European Conference on Computer Vision*, pp\. 446–461\. Springer, 2014\.
- Cimpoi et al\. \(2014\)Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi\.Describing textures in the wild\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 3606–3613, 2014\.
- Colyer & Kahana \(2025\)Ryan A Colyer and Michael J Kahana\.Phase consistency dynamics of memory encoding\.*Journal of Neuroscience*, 45\(35\), 2025\.
- Cui et al\. \(2025\)Shuang Cui, Jinglin Xu, Yi Li, Xiongxin Tang, Jiangmeng Li, Jiahuan Zhou, Fanjiang Xu, Fuchun Sun, and Hui Xiong\.Bayestta: Continual\-temporal test\-time adaptation for vision\-language models via gaussian discriminant analysis\.*arXiv preprint arXiv:2507\.08607*, 2025\.
- Deng et al\. \(2009\)Jia Deng, Wei Dong, Richard Socher, Li\-Jia Li, Kai Li, and Li Fei\-Fei\.Imagenet: A large\-scale hierarchical image database\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 248–255, 2009\.
- Dong et al\. \(2025\)Hao Dong, Lijun Sheng, Jian Liang, Ran He, Eleni Chatzi, and Olga Fink\.Adapting vision\-language models without labels: A comprehensive survey\.*arXiv preprint arXiv:2508\.05547*, 2025\.
- Dosovitskiy et al\. \(2020\)Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al\.An image is worth 16x16 words: Transformers for image recognition at scale\.*arXiv preprint arXiv:2010\.11929*, 2020\.
- Fei\-Fei et al\. \(2007\)Li Fei\-Fei, Rob Fergus, and Pietro Perona\.Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories\.*Computer Vision and Image Understanding*, 106\(1\):59–70, 2007\.
- Feng et al\. \(2023\)Chun\-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo\.Diverse data augmentation with diffusions for effective test\-time prompt tuning\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp\. 2704–2714, 2023\.
- Golbabaei & Frankland \(2025\)Ali Golbabaei and Paul W Frankland\.The post\-“standard model” age: Updating theories of systems consolidation\.*Neuron*, 113\(3\):339–341, 2025\.
- Grandvalet & Bengio \(2004\)Yves Grandvalet and Yoshua Bengio\.Semi\-supervised learning by entropy minimization\.*Advances in neural information processing systems*, 17, 2004\.
- He et al\. \(2016\)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Deep residual learning for image recognition\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pp\. 770–778, 2016\.
- Helber et al\. \(2019\)Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth\.Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification\.*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12\(7\):2217–2226, 2019\.
- Hendrycks et al\. \(2021a\)Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al\.The many faces of robustness: A critical analysis of out\-of\-distribution generalization\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp\. 8340–8349, 2021a\.
- Hendrycks et al\. \(2021b\)Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song\.Natural adversarial examples\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 15262–15271, 2021b\.
- Ioffe & Szegedy \(2015\)Sergey Ioffe and Christian Szegedy\.Batch normalization: Accelerating deep network training by reducing internal covariate shift\.In*International conference on machine learning*, pp\. 448–456\. pmlr, 2015\.
- Jia et al\. \(2021\)Chao Jia, Yinfei Yang, Ye Xia, Yi\-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun\-Hsuan Sung, Zhen Li, and Tom Duerig\.Scaling up visual and vision\-language representation learning with noisy text supervision\.In*International conference on machine learning*, pp\. 4904–4916\. PMLR, 2021\.
- Karmanov et al\. \(2024\)Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing\.Efficient test\-time adaptation of vision\-language models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024\.
- Krause et al\. \(2013\)Jonathan Krause, Michael Stark, Jia Deng, and Li Fei\-Fei\.3d object representations for fine\-grained categorization\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops*, pp\. 554–561, 2013\.
- Lei et al\. \(2025\)Bo Lei, Bilin Kang, Yuejun Hao, Haoyu Yang, Zihan Zhong, Zihan Zhai, and Yi Zhong\.Reconstructing a new hippocampal engram for systems reconsolidation and remote memory updating\.*Neuron*, 113\(3\):471–485, 2025\.
- Liang et al\. \(2025\)Jian Liang, Ran He, and Tieniu Tan\.A comprehensive survey on test\-time adaptation under distribution shifts\.*International Journal of Computer Vision*, 133\(1\):31–64, 2025\.
- Loshchilov & Hutter \(2017\)Ilya Loshchilov and Frank Hutter\.Decoupled weight decay regularization\.*arXiv preprint arXiv:1711\.05101*, 2017\.
- Ma et al\. \(2023\)Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu\.Swapprompt: Test\-time prompt adaptation for vision\-language models\.*Advances in Neural Information Processing Systems*, 36:65252–65264, 2023\.
- Maji et al\. \(2013\)Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi\.Fine\-grained visual classification of aircraft\.*arXiv preprint arXiv:1306\.5151*, 2013\.
- Mason et al\. \(2017\)Alice Mason, Simon Farrell, Paul Howard\-Jones, and Casimir JH Ludwig\.The role of reward and reward uncertainty in episodic memory\.*Journal of memory and language*, 96:62–77, 2017\.
- McClelland et al\. \(1995\)James L McClelland, Bruce L McNaughton, and Randall C O’Reilly\.Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory\.*Psychological review*, 102\(3\):419, 1995\.
- Menon et al\. \(2024\)Sachit Menon, Ishaan Preetam Chandratreya, and Carl Vondrick\.Task bias in contrastive vision\-language models\.*International Journal of Computer Vision*, 132\(6\):2026–2040, 2024\.
- Nilsback & Zisserman \(2008\)Maria\-Elena Nilsback and Andrew Zisserman\.Automated flower classification over a large number of classes\.In*Indian Conference on Computer Vision, Graphics and Image Processing*, pp\. 722–729\. IEEE, 2008\.
- Oord et al\. \(2018\)Aaron van den Oord, Yazhe Li, and Oriol Vinyals\.Representation learning with contrastive predictive coding\.*arXiv preprint arXiv:1807\.03748*, 2018\.
- Palm \(2013\)Günther Palm\.Neural associative memories and sparse coding\.*Neural Networks*, 37:165–171, 2013\.
- Parkhi et al\. \(2012\)Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar\.Cats and dogs\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 3498–3505, 2012\.
- Pratt et al\. \(2023\)Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi\.What does a platypus look like? generating customized prompts for zero\-shot image classification\.In*Proceedings of the IEEE/CVF international conference on computer vision*, pp\. 15691–15701, 2023\.
- Radford et al\. \(2021\)Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al\.Learning transferable visual models from natural language supervision\.In*International Conference on Machine Learning*, pp\. 8748–8763\. PMLR, 2021\.
- Radvansky et al\. \(1995\)Gabriel A Radvansky, Laura A Carlson\-Radvansky, and David E Irwin\.Uncertainty in estimating distances from memory\.*Memory & Cognition*, 23\(5\):596–606, 1995\.
- Recht et al\. \(2019\)Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar\.Do imagenet classifiers generalize to imagenet?In*International Conference on Machine Learning*, pp\. 5389–5400\. PMLR, 2019\.
- Shu et al\. \(2022\)Manli Shu, Weili Nie, De\-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao\.Test\-time prompt tuning for zero\-shot generalization in vision\-language models\.*Advances in Neural Information Processing Systems*, 35:14274–14289, 2022\.
- Soomro et al\. \(2012\)Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah\.Ucf101: A dataset of 101 human actions classes from videos in the wild\.*arXiv preprint arXiv:1212\.0402*, 2012\.
- Sui et al\. \(2025\)Elaine Sui, Xiaohan Wang, and Serena Yeung\-Levy\.Just shift it: Test\-time prototype shifting for zero\-shot generalization with vision\-language models\.In*2025 IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*, pp\. 825–835\. IEEE, 2025\.
- Tonegawa et al\. \(2018\)Susumu Tonegawa, Mark D Morrissey, and Takashi Kitamura\.The role of engram cells in the systems consolidation of memory\.*Nature Reviews Neuroscience*, 19\(8\):485–498, 2018\.
- Wang et al\. \(2019\)Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing\.Learning robust global representations by penalizing local predictive power\.In*Advances in Neural Information Processing Systems*, volume 32, pp\. 10506–10518, 2019\.
- Wixted et al\. \(2014\)John T Wixted, Larry R Squire, Yoonhee Jang, Megan H Papesh, Stephen D Goldinger, Joel R Kuhn, Kris A Smith, David M Treiman, and Peter N Steinmetz\.Sparse and distributed coding of episodic memory in neurons of the human hippocampus\.*Proceedings of the National Academy of Sciences*, 111\(26\):9621–9626, 2014\.
- Xiao et al\. \(2010\)Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba\.Sun database: Large\-scale scene recognition from abbey to zoo\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 3485–3492, 2010\.
- Xiao et al\. \(2025\)Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang, and Cees GM Snoek\.Dynaprompt: Dynamic test\-time prompt tuning\.*arXiv preprint arXiv:2501\.16404*, 2025\.
- Yuan et al\. \(2023\)Longhui Yuan, Binhui Xie, and Shuang Li\.Robust test\-time adaptation in dynamic scenarios\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 15922–15932, 2023\.
- Zhang et al\. \(2024a\)Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie\.Dual prototype evolving for test\-time generalization of vision\-language models, 2024a\.URL[https://arxiv\.org/abs/2410\.12790](https://arxiv.org/abs/2410.12790)\.
- Zhang et al\. \(2024b\)Jingyi Zhang, Jiaxing Huang, Xiaoqin Zhang, Ling Shao, and Shijian Lu\.Historical test\-time prompt tuning for vision foundation models\.*Advances in Neural Information Processing Systems*, 37:12872–12896, 2024b\.
- Zhang et al\. \(2022\)Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li\.Tip\-adapter: Training\-free adaption of clip for few\-shot classification\.In*European conference on computer vision*, pp\. 493–510\. Springer, 2022\.
- Zhang et al\. \(2024c\)Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang\.Dual memory networks: A versatile adaptation approach for vision\-language models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp\. 28718–28728, 2024c\.
- Zhou et al\. \(2022\)Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu\.Learning to prompt for vision\-language models\.*International Journal of Computer Vision*, 130\(9\):2337–2348, 2022\.
- Zhou et al\. \(2025\)Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen Lei\.Bayesian test\-time adaptation for vision\-language models, 2025\.URL[https://arxiv\.org/abs/2503\.09248](https://arxiv.org/abs/2503.09248)\.

## Appendix ARelated work

#### Vision\-Language Models \(VLMs\)\.

Recent years have witnessed the emergence of powerful VLMs, such as CLIP\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)and ALIGN\(Jia et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib19)\), trained on large\-scale image\-text pairs\. These models learn a joint embedding space that aligns visual and textual representations, enabling strong zero\-shot generalization across a wide range of downstream tasks\(Liang et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib23); Dong et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib8)\)\. To further enhance VLMs’ performance, various adaptation strategies have been proposed, most notably in few\-shot settings\. For example, prompt tuning methods like CoOp\(Zhou et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib51)\)learn continuous prompt vectors to steer model predictions, while adapter\-based methods like Tip\-Adapter\(Zhang et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib49)\)introduce lightweight modules to adapt features with minimal supervision\. While effective, these methods require labeled data, a constraint that our work aims to overcome by operating in a purely test\-time setting\.

#### Test Time Adaptation \(TTA\)\.

TTA aims to adapt a model pretrained on a source domain to an unlabeled target domain during inference\(Liang et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib23); Dong et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib8)\)\. Early methods primarily focused on updating batch normalization statistics or minimizing prediction entropy to encourage confident outputs\(Ioffe & Szegedy,[2015](https://arxiv.org/html/2606.28719#bib.bib18); Grandvalet & Bengio,[2004](https://arxiv.org/html/2606.28719#bib.bib13); Yuan et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib46)\)\. Subsequently, Bayesian inference\-based methods such as BCA\(Zhou et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib52)\)and BayesTTA\(Cui et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib6)\)leverage information from previous test batches for model refinement\. Prompt\-based methods such as TPT \(Fig\.[1](https://arxiv.org/html/2606.28719#S1.F1)A\)\(Shu et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib38)\)and its variants\(Feng et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib11); Abdul Samadh et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib1); Ma et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib25)\)learn a tailored prompt for each test sample\. However, these methods are inherently amnesic, adapting in isolation without accumulating knowledge over time\. More recent efforts, such as HisTPT\(Zhang et al\.,[2024b](https://arxiv.org/html/2606.28719#bib.bib48)\)and DynaPrompt\(Xiao et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib45)\)\(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)B\), attempt to address this by dynamically selecting prompts across the test stream, but still operate at the prompt\-tuning level, limiting long\-term adaptation\.

To overcome these limitations, memory\-based methods have been introduced\. TDA\(Karmanov et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib20)\)maintains a dynamic visual cache to enhance adaptation, while DMN\-ZS\(Zhang et al\.,[2024c](https://arxiv.org/html/2606.28719#bib.bib50)\)and DPE\(Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\)extend this idea to dual memory systems\. DPE evolves visual and textual prototypes jointly, and DMN\-ZS incorporates both dynamic test\-time and optional static few\-shot memories \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)C\)\. However, these methods generally treat memory updates as symmetric and fast across modalities, lacking a principled separation of roles\. This design leads to overfitting recent examples or under\-adapting to novel domains, failing to strike a balance between plasticity \(rapid adaptation\) and stability \(retention of generalizable knowledge\)\. In contrast, our proposed ComMem framework introduces a brain\-inspired architecture that explicitly models distinct fast and slow learning pathways DPE evolves visual and textual prototypes jointly \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)D\-E and Fig\.[3](https://arxiv.org/html/2606.28719#S2.F3)\)\. By doing so, it enables both per\-sample responsiveness and long\-term knowledge accumulation through coordinated cross\-modal memory fusion\.

#### Complementary Memory Systems\.

ComMem is grounded in the complementary learning systems \(CLS\) theory, also referred to as the complementary memory systems, from neuroscience\(McClelland et al\.,[1995](https://arxiv.org/html/2606.28719#bib.bib28); Tonegawa et al\.,[2018](https://arxiv.org/html/2606.28719#bib.bib41)\), which explains how the brain learns across multiple timescales\. CLS posits a functional specialization between the hippocampus \(HPC\) that rapidly encodes detailed, episodic experiences and the neocortex \(NC\) that slowly incorporates structured, generalizable knowledge \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)A\)\(Tonegawa et al\.,[2018](https://arxiv.org/html/2606.28719#bib.bib41)\)\. The gradual reorganization and transfer of memory traces from HPC to NC is referred to as system consolidation \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)B\), allowing temporary experiences to become part of stable long\-term knowledge\(Lei et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib22); Golbabaei & Frankland,[2025](https://arxiv.org/html/2606.28719#bib.bib12)\)\. Importantly, consolidated memories can become labile upon retrieval through a process known as memory reconsolidation \(Fig\.[2](https://arxiv.org/html/2606.28719#S1.F2)C\), which re\-engages the HPC to incorporate new information and update existing memories\(Lei et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib22)\)\. To our knowledge, ComMem is the first to explicitly operationalize both the fast \(HPC\-like\) and slow \(NC\-like\) learning dynamics, along with mechanisms analogous to consolidation and reconsolidation, for VLMs’ TTA\. This biologically grounded design enables ComMem to achieve more robust and generalizable adaptation\.

## Appendix BAdditional Implementation Details

### B\.1Experimental Setups

Datasets\.Our evaluation follows the established benchmarks used in prior work\(Shu et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib38); Feng et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib11); Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\)\. To assessrobustness to natural distribution shifts, we use the ImageNet dataset\(Deng et al\.,[2009](https://arxiv.org/html/2606.28719#bib.bib7)\)and its four out\-of\-distribution variants: ImageNet\-A\(Hendrycks et al\.,[2021b](https://arxiv.org/html/2606.28719#bib.bib17)\), ImageNet\-V2\(Recht et al\.,[2019](https://arxiv.org/html/2606.28719#bib.bib37)\), ImageNet\-R\(Hendrycks et al\.,[2021a](https://arxiv.org/html/2606.28719#bib.bib16)\), and ImageNet\-Sketch\(Wang et al\.,[2019](https://arxiv.org/html/2606.28719#bib.bib42)\)\. Forcross\-dataset generalization, we evaluate on a diverse suite of 10 recognition datasets: FGVCAircraft\(Maji et al\.,[2013](https://arxiv.org/html/2606.28719#bib.bib26)\), Caltech101\(Fei\-Fei et al\.,[2007](https://arxiv.org/html/2606.28719#bib.bib10)\), StandfordCars\(Krause et al\.,[2013](https://arxiv.org/html/2606.28719#bib.bib21)\), DTD\(Cimpoi et al\.,[2014](https://arxiv.org/html/2606.28719#bib.bib4)\), EuroSAT\(Helber et al\.,[2019](https://arxiv.org/html/2606.28719#bib.bib15)\), Flowers102\(Nilsback & Zisserman,[2008](https://arxiv.org/html/2606.28719#bib.bib30)\), Food101\(Bossard et al\.,[2014](https://arxiv.org/html/2606.28719#bib.bib3)\), OxfordPets\(Parkhi et al\.,[2012](https://arxiv.org/html/2606.28719#bib.bib33)\), SUN397\(Xiao et al\.,[2010](https://arxiv.org/html/2606.28719#bib.bib44)\), and UCF101\(Soomro et al\.,[2012](https://arxiv.org/html/2606.28719#bib.bib39)\)\. These datasets offer a comprehensive benchmark for evaluating the robustness of various methods across different distributional variations\.

Implementation Details\.We employ CLIP\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)models with both ResNet\-50\(He et al\.,[2016](https://arxiv.org/html/2606.28719#bib.bib14)\)and ViT\-B/16\(Dosovitskiy et al\.,[2020](https://arxiv.org/html/2606.28719#bib.bib9)\)visual backbones for all experiments\. Following standard practice, we utilize a broad set of hand\-crafted text prompts and also incorporate prompts from CuPL to enhance the initial textual prototypes \(Supplementary Table[6](https://arxiv.org/html/2606.28719#A2.T6)\)\. Followed by previous studies\(Karmanov et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib20); Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\), we generate multiple augmented views for each sample to facilitate robust optimization\. The learnable residuals for our complementary memory systems are optimized for a single step per sample using the AdamW optimizer\(Loshchilov & Hutter,[2017](https://arxiv.org/html/2606.28719#bib.bib24)\)\. The adaptation of batch normalization or layer normalization parameters\(Ioffe & Szegedy,[2015](https://arxiv.org/html/2606.28719#bib.bib18)\)is performed with a distinct, slower learning rate\. Our primary experiments are conducted on a single NVIDIA RTX 3090 GPU with 24GB of memory\.

Baselines\.We compare ComMem against a comprehensive set of state\-of\-the\-art methods for VLM adaptation\. These include: zero\-shotCLIP\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)and its promptEnsembleversion; few\-shot supervised methods likeCoOp\(Zhou et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib51)\); and leading test\-time adaptation methods such asTPT\(Shu et al\.,[2022](https://arxiv.org/html/2606.28719#bib.bib38)\), which learns sample\-specific prompts;DiffTPT\(Feng et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib11)\), which enhances TPT with diffusion\-based augmentations;DynaPrompt\(Xiao et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib45)\), which performs continual adaptation by adaptively selecting and optimizing relevant prompts for each test sample;TDA\(Karmanov et al\.,[2024](https://arxiv.org/html/2606.28719#bib.bib20)\), a training\-free approach that builds a visual cache;TPS\(Sui et al\.,[2025](https://arxiv.org/html/2606.28719#bib.bib40)\), which shifts prototypes at test\-time;DPE\(Zhang et al\.,[2024a](https://arxiv.org/html/2606.28719#bib.bib47)\), which evolves dual prototypes, andDMN\-ZS\(Zhang et al\.,[2024c](https://arxiv.org/html/2606.28719#bib.bib50)\), which uses a dynamic memory for historical test data and a static memory for few\-shot data\.

Algorithm 1ComMem: Complementary Memory Systems for Test\-Time Adaptation of VLMs1:Input: Pretrained VLM

\(ℰv,ℰt\)\(\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\}\), target stream

\{𝐱t\}t=1T\\\{\\mathbf\{x\}\_\{t\}\\\}\_\{t=1\}^\{T\},

2:NC\-like abstract memory

𝐏t\\mathbf\{P\}^\{t\}, HPC\-like detailed memory cache

ℳ\\mathcal\{M\}, hyperparameters

\(τe,τconf,λhpc,λnc\)\(\\tau\_\{e\},\\tau\_\{\\text\{conf\}\},\\lambda\_\{\\text\{hpc\}\},\\lambda\_\{\\text\{nc\}\}\)
3:

4:foreach incoming test sample

𝐱t\\mathbf\{x\}\_\{t\}do

5:Step 1: Entropy\-Guided Visual Encoding

6:Generate

NNaugmented views

\{𝐱t\(i\)\}i=1N\\\{\\mathbf\{x\}\_\{t\}^\{\(i\)\}\\\}\_\{i=1\}^\{N\}
7:foreach view

i=1​…​Ni=1\\dots Ndo

8:

𝐟v\(i\)←ℰv​\(𝐱t\(i\)\)\\mathbf\{f\}\_\{v\}^\{\(i\)\}\\leftarrow\\mathcal\{E\}\_\{v\}\(\\mathbf\{x\}\_\{t\}^\{\(i\)\}\)
9:

p\(i\)←p​\(y\|𝐟v\(i\)\)p^\{\(i\)\}\\leftarrow p\(y\|\\mathbf\{f\}\_\{v\}^\{\(i\)\}\)via Eq\. \([1](https://arxiv.org/html/2606.28719#S2.E1)\)

10:

ℋ\(i\)←−∑c=1Cpc\(i\)​log⁡pc\(i\)\\mathcal\{H\}^\{\(i\)\}\\leftarrow\-\\sum\_\{c=1\}^\{C\}p\_\{c\}^\{\(i\)\}\\log p\_\{c\}^\{\(i\)\}⊳\\trianglerightEq\. \([6](https://arxiv.org/html/2606.28719#S3.E6)\)

11:endfor

12:Compute weights

w\(i\)∝exp⁡\(−ℋ\(i\)/τe\)w^\{\(i\)\}\\propto\\exp\(\-\\mathcal\{H\}^\{\(i\)\}/\\tau\_\{e\}\)
13:Obtain entropy\-refined feature

𝐟v∗=∑iw\(i\)​𝐟v\(i\)\\mathbf\{f\}\_\{v\}^\{\*\}=\\sum\_\{i\}w^\{\(i\)\}\\mathbf\{f\}\_\{v\}^\{\(i\)\}⊳\\trianglerightEq\. \([7](https://arxiv.org/html/2606.28719#S3.E7)\)

14:Assign pseudo\-label

y^=argmaxp\(⋅\|𝐟v∗\)\\hat\{y\}=\\arg\\max p\(\\cdot\|\\mathbf\{f\}\_\{v\}^\{\*\}\)
15:

16:Step 2: HPC Detailed Memory Update

17:if

\|ℳy^\|<K\|\\mathcal\{M\}\_\{\\hat\{y\}\}\|<Kthen

18:Add

\(𝐟v∗,ℋ\)\(\\mathbf\{f\}\_\{v\}^\{\*\},\\mathcal\{H\}\)to

ℳy^\\mathcal\{M\}\_\{\\hat\{y\}\}
19:else

20:Identify least confident entry

\(𝐟max,ℋmax\)\(\\mathbf\{f\}\_\{\\max\},\\mathcal\{H\}\_\{\\max\}\)
21:if

ℋ<ℋmax\\mathcal\{H\}<\\mathcal\{H\}\_\{\\max\}then

22:Compute class prototype

𝐩y^v=mean​\(ℳy^\)\\mathbf\{p\}^\{v\}\_\{\\hat\{y\}\}=\\text\{mean\}\(\\mathcal\{M\}\_\{\\hat\{y\}\}\)
23:Update

𝐟v′=Norm​\(\(1−λhpc\)​𝐩y^v\+λhpc​𝐟v∗\)\\mathbf\{f\}^\{\\prime\}\_\{v\}=\\text\{Norm\}\(\(1\-\\lambda\_\{\\text\{hpc\}\}\)\\mathbf\{p\}^\{v\}\_\{\\hat\{y\}\}\+\\lambda\_\{\\text\{hpc\}\}\\mathbf\{f\}\_\{v\}^\{\*\}\)⊳\\trianglerightEq\. \([8](https://arxiv.org/html/2606.28719#S3.E8)\)

24:Replace

\(𝐟max,ℋmax\)←\(𝐟v′,ℋ\)\(\\mathbf\{f\}\_\{\\max\},\\mathcal\{H\}\_\{\\max\}\)\\leftarrow\(\\mathbf\{f\}^\{\\prime\}\_\{v\},\\mathcal\{H\}\)
25:endif

26:endif

27:Recompute HPC prototypes

𝐏v=\[𝐩1v,…,𝐩Cactivev\]\\mathbf\{P\}^\{v\}=\[\\mathbf\{p\}^\{v\}\_\{1\},\\dots,\\mathbf\{p\}^\{v\}\_\{C\_\{\\text\{active\}\}\}\]
28:

29:Step 3: Test\-Time Optimization

30:Initialize learnable parameters: residuals

\(δt,δv\)\(\\delta\_\{t\},\\delta\_\{v\}\), normalization

θn\\theta\_\{n\}
31:Compute updated prototypes:

32:

𝐏^localt=Norm​\(𝐏t\+δt\)\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\text\{local\}\}=\\text\{Norm\}\(\\mathbf\{P\}^\{t\}\+\\delta\_\{t\}\)⊳\\trianglerightEq\. \([11](https://arxiv.org/html/2606.28719#S3.E11)\)

33:

𝐏^v=Norm​\(𝐏v\+δv\)\\hat\{\\mathbf\{P\}\}^\{v\}=\\text\{Norm\}\(\\mathbf\{P\}^\{v\}\+\\delta\_\{v\}\)
34:Compute final logits:

35:

𝐳final=𝐟v⊤​𝐏^localt\+𝒜​\(𝐟v⊤​𝐏^v\)\\mathbf\{z\}\_\{\\text\{final\}\}=\\mathbf\{f\}\_\{v\}^\{\\top\}\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\text\{local\}\}\+\\mathcal\{A\}\(\\mathbf\{f\}\_\{v\}^\{\\top\}\\hat\{\\mathbf\{P\}\}^\{v\}\)
36:Compute composite loss:

37:

ℒtotal=ℒent\+ℒalign\+ℒsparse\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{ent\}\}\+\\mathcal\{L\}\_\{\\text\{align\}\}\+\\mathcal\{L\}\_\{\\text\{sparse\}\}
38:Update

\(δt,δv,θn\)\(\\delta\_\{t\},\\delta\_\{v\},\\theta\_\{n\}\)with one\-step AdamW

39:Obtain final prediction

y^t=arg⁡max⁡Softmax​\(𝐳final\)\\hat\{y\}\_\{t\}=\\arg\\max\\text\{Softmax\}\(\\mathbf\{z\}\_\{\\text\{final\}\}\)
40:

41:Step 4: NC Abstract Memory Update \(Consolidation\)

42:if

ℋ​\(𝐳final\)<τconf\\mathcal\{H\}\(\\mathbf\{z\}\_\{\\text\{final\}\}\)<\\tau\_\{\\text\{conf\}\}then

43:

𝐏t←Norm​\(\(1−λnc\)​𝐏t\+λnc​𝐏^localt\)\\mathbf\{P\}^\{t\}\\leftarrow\\text\{Norm\}\\big\(\(1\-\\lambda\_\{\\text\{nc\}\}\)\\mathbf\{P\}^\{t\}\+\\lambda\_\{\\text\{nc\}\}\\hat\{\\mathbf\{P\}\}^\{t\}\_\{\\text\{local\}\}\\big\)⊳\\trianglerightEq\. \([9](https://arxiv.org/html/2606.28719#S3.E9)\)

44:endif

45:endfor

46:

47:returnAdapted VLM

\(ℰv,ℰt,𝐏t,ℳ\)\(\\mathcal\{E\}\_\{v\},\\mathcal\{E\}\_\{t\},\\mathbf\{P\}^\{t\},\\mathcal\{M\}\)

### B\.2Dataset Details

In Table[5](https://arxiv.org/html/2606.28719#A2.T5), we present the detailed statistics of each dataset we used in our experiments, including the number of classes, the sizes of training, validation and testing sets, and their original tasks\.

Table 5:Detailed statistics of datasets used in experiments\. Note that the last 4 ImageNet variant datasets are designed for evaluation and only contain the test sets\.
### B\.3Textual Prompts Used in Experiments

In Table[6](https://arxiv.org/html/2606.28719#A2.T6), we detail the specific hand\-crafted prompts utilized for each dataset\.

Table 6:Textual prompts used in experiments\. In addition to these prompts, we also employ CuPL\(Pratt et al\.,[2023](https://arxiv.org/html/2606.28719#bib.bib34)\)prompts to further enhance performance\.
### B\.4Textual Prompts Used in Experiments

In Table[6](https://arxiv.org/html/2606.28719#A2.T6), we detail the specific hand\-crafted prompts utilized for each dataset\.

![Refer to caption](https://arxiv.org/html/2606.28719v1/x6.png)Figure 6:t\-SNE visualizations of the HPC\-like detailed memory \(with cache sizeKK= 30\) over time using CLIP\-ResNet\-50\(Radford et al\.,[2021](https://arxiv.org/html/2606.28719#bib.bib35)\)on Food101\(Bossard et al\.,[2014](https://arxiv.org/html/2606.28719#bib.bib3)\)\. Features from 10% of randomly selected classes are shown in different colors, while all others are indicated in gray\.