MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

# MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models
Source: [https://arxiv.org/html/2607.22556](https://arxiv.org/html/2607.22556)
Yanchi LiuNEC Laboratories America \{yanchi, xuzhao, weicheng, zchen, haifeng\}@nec\-labs\.comXujiang ZhaoNEC Laboratories America \{yanchi, xuzhao, weicheng, zchen, haifeng\}@nec\-labs\.comWei ChengNEC Laboratories America \{yanchi, xuzhao, weicheng, zchen, haifeng\}@nec\-labs\.comZhengzhang ChenNEC Laboratories America \{yanchi, xuzhao, weicheng, zchen, haifeng\}@nec\-labs\.comXintao WuUniversity of Arkansas xintaowu@uark\.eduZhong ChenSouthern Illinois University zhong\.chen@cs\.siu\.eduChen ZhaoBaylor University \{dong\_li1, chen\_zhao\}@baylor\.eduHaifeng ChenNEC Laboratories America \{yanchi, xuzhao, weicheng, zchen, haifeng\}@nec\-labs\.com

###### Abstract

Continual learning \(CL\) is essential for small language models \(SLMs\) to adapt to evolving real\-world needs in resource\-constrained deployments\. However, directly updating their limited parameter space causes catastrophic forgetting\. While memory\-based methods naturally address this by decoupling knowledge retention from parameters, existing approaches designed for large language models \(LLMs\) rely on abundant storage and strong in\-context reasoning that SLMs lack\. To address these challenges, we propose MIITA, a Memory\-Induced Inference\-Time Adaptation framework for supervised CL under constrained storage\. MIITA stores supervised experiences as compact correction\-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty\-based cues\. The retrieved directions are applied through gated temporary hidden\-state adaptation, enabling non\-destructive reuse of past supervision without backbone updates, prompt extensions, or test\-time backpropagation\. A local theoretical analysis links this design to first\-order loss reduction, uncertainty\-guided retrieval, and directional coverage for retaining old\-stage knowledge\. Extensive experiments across diverse supervised CL settings show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets\.

## 1Introduction

Small language models \(SLMs\) are increasingly being deployed in localized, low\-latency, privacy\-sensitive, and resource\-constrained settings, where they need to continually adapt to evolving real\-world needs\[[14](https://arxiv.org/html/2607.22556#bib.bib22),[24](https://arxiv.org/html/2607.22556#bib.bib23)\]\. However, continual learning \(CL\) for SLMs is fundamentally difficult to achieve through the parameter space alone\. Their limited capacity and high representational redundancy mean that repeatedly writing new experiences into the backbone induces severe catastrophic forgetting and unstable updates\[[5](https://arxiv.org/html/2607.22556#bib.bib1),[2](https://arxiv.org/html/2607.22556#bib.bib2),[21](https://arxiv.org/html/2607.22556#bib.bib3)\]\. Therefore, the key challenge is to continuously utilize new supervised experiences without overwriting the scarce parameter space\.

Memory\-based CL naturally addresses this by decoupling experience retention from the model’s parameter space, allowing the backbone to remain stable while auxiliary storage accumulates new knowledge\[[20](https://arxiv.org/html/2607.22556#bib.bib29),[16](https://arxiv.org/html/2607.22556#bib.bib24),[7](https://arxiv.org/html/2607.22556#bib.bib25)\]\. However, existing LLM\-centric memory methods translate poorly to SLMs because they typically preserve past experience as textual examples, summaries, or demonstrations\[[18](https://arxiv.org/html/2607.22556#bib.bib6),[3](https://arxiv.org/html/2607.22556#bib.bib26)\]\. Such memories are storage\-inefficient under a fixed budget: they retain surface content in addition to the reusable correction signal, and functionally redundant experiences are difficult to consolidate in text form\[[15](https://arxiv.org/html/2607.22556#bib.bib28)\]\. Moreover, they rely on strong in\-context reasoning to infer how retrieved memories should affect predictions, which is constrained in smaller models\[[26](https://arxiv.org/html/2607.22556#bib.bib27)\]\. As shown in Figure[1](https://arxiv.org/html/2607.22556#S1.F1), the performance of representative memory\-based LLM CL methods degrades significantly as the base model size decreases, especially under fixed storage budgets\. This trend highlights a critical dual bottleneck for SLMs: the inability to maintain massive textual memories under tight storage limits, and the limited capacity to effectively exploit them during inference\.

![Refer to caption](https://arxiv.org/html/2607.22556v1/x1.png)Figure 1:Model\-size scaling of CIS\[[18](https://arxiv.org/html/2607.22556#bib.bib6)\]and FOREVER\[[6](https://arxiv.org/html/2607.22556#bib.bib5)\], two LLM\-oriented memory\-based CL baselines, onSuperNI\[[27](https://arxiv.org/html/2607.22556#bib.bib19)\]benchmark\. The “\-B” suffix indicates variants restricted to a fixed memory budget, contrasting with their unconstrained counterparts\. We evaluate Overall Performance \(OP\) and Backward Transfer \(BWT\); higher values indicate better accuracy and less forgetting, respectively\.Overcoming this dual bottleneck requires rethinking what a memory item should preserve\. Inspired by von Oswaldet al\.\[[25](https://arxiv.org/html/2607.22556#bib.bib18)\], who characterize in\-context examples as inducing gradient\-descent\-like local corrections during the forward pass, the value of a memory item lies not in its raw semantic content alone, but in the functional correction it imposes on the model’s hidden representation\. This perspective reveals adivergence between semantic and functional redundancy: semantically disparate examples may induce identical corrections and can thus be merged, whereas semantically similar examples might require distinct corrections\. Consequently, rather than preserving historical text and expecting the SLM to infer how to use it, an alternative paradigm is to organize memory directly around reusable adaptation directions\. Storing these functional signals in a compact form addresses both tight storage limits and the reliance on in\-context reasoning\.

Motivated by this paradigm, we proposeMIITA, aMemory\-Induced Inference\-Time Adaptationframework for supervised CL with SLMs under constrained persistent storage\. MIITA converts each supervised experience into a compact correction direction within the model’s internal hidden space, consolidating these functional signals into budget\-aware memory prototypes that utilize semantic anchors purely as lightweight retrieval keys\. At inference time, the framework retrieves relevant historical directions through both semantic relevance and an uncertainty\-based directional proxy\. These directions are aggregated into a query\-specific correction and applied via a gated, temporary hidden\-state update to directly guide the prediction\. This adaptation is disposable, requiring no permanent parameter updates, test\-time backpropagation, or lengthy prompt extensions\. Our main contributions are summarized as follows:

- •We propose a direction\-oriented memory representation that stores supervised experiences as correction\-direction prototypes with lightweight semantic anchors, overcoming the high storage and reasoning costs of traditional textual memory in SLMs\.
- •We introduce a memory\-induced inference\-time adaptation mechanism that applies retrieved historical directions via gated temporary hidden\-state updates, safely reusing past supervision without test\-time backpropagation or destructive backbone updates\.
- •We provide a local theoretical analysis showing that correction directions are first\-order optimal for decreasing the supervised loss, that uncertainty directions identify the steepest direction for decreasing local predictive entropy and serve as label\-free retrieval cues, and that old\-stage retention is controlled by the directional coverage of stored memory\.
- •We provide comprehensive empirical evaluation across diverse CL settings, showing that MIITA improves final performance and reduces forgetting under fixed memory budgets, with further analyses demonstrating its robustness across model scales and memory budgets, as well as the effectiveness of functional correction memory and uncertainty\-guided retrieval\.\.

## 2Related Work

CL for SLMs\.CL for SLMs is crucial for resource\-constrained deployment but remains underexplored\. Existing methods address catastrophic forgetting through various mechanisms: DKVB\[[5](https://arxiv.org/html/2607.22556#bib.bib1)\]uses localized updates via a discrete key\-value bottleneck; DA\-GRPO\[[2](https://arxiv.org/html/2607.22556#bib.bib2)\]optimizes continual post\-training while learning to invoke a cloud LLM under a cloud\-assistance budget; and Srinath Ket al\.\[[21](https://arxiv.org/html/2607.22556#bib.bib3)\]use POS\-guided code\-switching to mitigate cross\-lingual forgetting through replay adapters\. However, these approaches still rely on training\-time parameter updates, cloud assistance, or sample replay\. In contrast, MIITA addresses these limitations by bypassing persistent backbone modifications and training\-time replay, thereby overcoming the inherent unreliability of SLMs in continual fine\-tuning and text\-based prompting\.

Memory\-Based CL for LLMs\.Existing memory\-based CL methods for LLMs generally fall into two categories based on when memory is used\. Training\-time replay methods reuse stored examples during optimization; for example, Continual\-T0 \(CT0\)\[[19](https://arxiv.org/html/2607.22556#bib.bib4)\]employs a small rehearsal buffer to retain previous abilities, while FOREVER\[[6](https://arxiv.org/html/2607.22556#bib.bib5)\]schedules memory reviews based on update dynamics\. Alternatively, inference\-time methods like CIS\[[18](https://arxiv.org/html/2607.22556#bib.bib6)\]and InCA\[[17](https://arxiv.org/html/2607.22556#bib.bib7)\]utilize textual summaries or in\-context examples to avoid forgetting without updating the LLM\. Other inference\-time methods \(e\.g\., MAC\[[22](https://arxiv.org/html/2607.22556#bib.bib8)\], CMT\[[13](https://arxiv.org/html/2607.22556#bib.bib9)\], HippoRAG 2\[[8](https://arxiv.org/html/2607.22556#bib.bib10)\], MBC\[[10](https://arxiv.org/html/2607.22556#bib.bib11)\]\) retrieve external documents for continual knowledge updates\. However, these methods primarily focus on knowledge augmentation rather than directly addressing task\-level forgetting in supervised CL, which differs from our target problem\. Furthermore, the remaining LLM\-centric approaches are ill\-suited for SLMs, where limited capacity and weaker in\-context abilities make replay and textual memories unreliable\. In contrast, MIITA stores supervised experiences as compact functional correction directions, invoking them via lightweight, optimization\-free hidden\-state adaptation at inference time\.

## 3Methodology

### 3\.1Problem Setting

We studysupervised continual learning \(CL\) for small language models \(SLMs\) under a constrained persistent auxiliary storage budget\. An SLM receives a sequence of data stages𝒟stream=\{𝒟1,…,𝒟T\}\\mathcal\{D\}\_\{\\mathrm\{stream\}\}=\\\{\\mathcal\{D\}\_\{1\},\\ldots,\\mathcal\{D\}\_\{T\}\\\}, where each stage𝒟t=\{\(xit,yit\)\}i=1Nt\\mathcal\{D\}\_\{t\}=\\\{\(x\_\{i\}^\{t\},y\_\{i\}^\{t\}\)\\\}\_\{i=1\}^\{N\_\{t\}\}contains supervised input\-output examples\. After each stage, a CL algorithm updates its system state, which may include model parameters and/or auxiliary persistent information such as replay samples, summaries, prompts, adapters, statistics, or memory items\. To reflect the resource constraints of SLM deployment, the total size of this auxiliary persistent information is bounded by a fixed storage budgetCmemC\_\{\\mathrm\{mem\}\}\. The goal is to learn from each new stage while preserving performance on all previously seen stages, i\.e\., maximizing final average performance and minimizing forgetting\.

### 3\.2MIITA: Memory\-Induced Inference\-Time Adaptation

To address budget\-constrained supervised CL for SLMs, we introduceMIITA, a memory\-induced inference\-time adaptation framework that decouples long\-term experience retention from the frozen model parameter space by preserving historical supervision as compact correction directions rather than raw examples or textual summaries\. Tailored to SLMs’ limited capacity and weaker in\-context abilities, MIITA models continual memory as reusable adaptation signals: during CL, supervised experiences are converted into pre\-LM\-head correction directions, which the memory manager then organizes into budget\-aware prototypes with semantic keys for retrieval\. At inference, MIITA retrieves relevant prototypes using semantic and uncertainty cues, applying them via a gated temporary hidden update\. We provide local theoretical analyses for correction\-direction extraction, uncertainty\-based retrieval, temporary hidden adaptation, and direction\-memory coverage; formal statements and proofs are given in Appendix[A](https://arxiv.org/html/2607.22556#A1)\. The overall framework of MIITA is shown in Algorithm[1](https://arxiv.org/html/2607.22556#alg1)in Appendix[B\.2](https://arxiv.org/html/2607.22556#A2.SS2)

#### 3\.2\.1Direction\-Oriented Memory Representation

MIITA maintains a memory bankℬ=\{u1,…,uJ\}\\mathcal\{B\}=\\\{u\_\{1\},\\ldots,u\_\{J\}\\\}, where each unit is a direction prototype:

uj=\(dj,𝒜j,nj\)\.u\_\{j\}=\(d\_\{j\},\\mathcal\{A\}\_\{j\},n\_\{j\}\)\.This design reflects the need to separate functional retention from semantic access in SLMs, which cannot reliably depend on large textual memories due to tighter context budgets and weaker in\-context abilities\. The prototype directiondj∈ℝHd\_\{j\}\\in\\mathbb\{R\}^\{H\}is a normalized correction direction in the pre\-LM\-head hidden space and serves as the primary stored signal, preserving the functional adaptation effect of historical supervised experiences\. The anchor set𝒜j\\mathcal\{A\}\_\{j\}contains lightweight semantic keys that are used only to retrieve this functional signal from different query contexts, andnjn\_\{j\}records how many experiences have been merged into the prototype\. The memory bankℬ\\mathcal\{B\}is maintained under the global storage budgetCmemC\_\{\\mathrm\{mem\}\}, which is enforced during memory writing through memory repair when new directions or anchors exceed the budget, as detailed in Section[3\.2\.3](https://arxiv.org/html/2607.22556#S3.SS2.SSS3)\.

#### 3\.2\.2Extracting Direction Signal from an Experience

Given a supervised experience\(xm,ym\)\(x\_\{m\},y\_\{m\}\), MIITA extracts two signals: a semantic keykmk\_\{m\}for memory access and a correction directiondmd\_\{m\}for memory writing\.

Semantic key\.Let𝒳m\\mathcal\{X\}\_\{m\}be the input token positions andhr\(ℓk\)h\_\{r\}^\{\(\\ell\_\{k\}\)\}the hidden state at layerℓk\\ell\_\{k\}\. We define

km=Norm​\(1\|𝒳m\|​∑r∈𝒳mhr\(ℓk\)\)\.k\_\{m\}=\\mathrm\{Norm\}\\left\(\\frac\{1\}\{\|\\mathcal\{X\}\_\{m\}\|\}\\sum\_\{r\\in\\mathcal\{X\}\_\{m\}\}h\_\{r\}^\{\(\\ell\_\{k\}\)\}\\right\)\.\(1\)whereNorm​\(v\)=v/\(‖v‖2\+ϵ\)\\mathrm\{Norm\}\(v\)=v/\(\\\|v\\\|\_\{2\}\+\\epsilon\)\. The keykmk\_\{m\}is used only for later retrieval\.

Correction vector\.We define the correction signal in the pre\-LM\-head hidden space, i\.e\., the space of final hidden states before they are projected to vocabulary logits\. For an experience\(xm,ym\)\(x\_\{m\},y\_\{m\}\), its unnormalized correction vectorRmR\_\{m\}is

Rm=1\|𝒴m\|​∑t∈𝒴m\(−∂ℒt∂htout\)=1\|𝒴m\|​∑t∈𝒴mWLM⊤​\(eyt−pt\)\.R\_\{m\}=\\frac\{1\}\{\|\\mathcal\{Y\}\_\{m\}\|\}\\sum\_\{t\\in\\mathcal\{Y\}\_\{m\}\}\\left\(\-\\frac\{\\partial\\mathcal\{L\}\_\{t\}\}\{\\partial h\_\{t\}^\{\\mathrm\{out\}\}\}\\right\)=\\frac\{1\}\{\|\\mathcal\{Y\}\_\{m\}\|\}\\sum\_\{t\\in\\mathcal\{Y\}\_\{m\}\}W\_\{\\mathrm\{LM\}\}^\{\\top\}\(e\_\{y\_\{t\}\}\-p\_\{t\}\)\.\(2\)Here,𝒴m\\mathcal\{Y\}\_\{m\}denotes the target\-output token positions,htout∈ℝHh\_\{t\}^\{\\mathrm\{out\}\}\\in\\mathbb\{R\}^\{H\}is the final hidden state before the LM head at positiontt,WLMW\_\{\\mathrm\{LM\}\}is the LM\-head projection matrix,pt=softmax​\(WLM​htout\)p\_\{t\}=\\mathrm\{softmax\}\(W\_\{\\mathrm\{LM\}\}h\_\{t\}^\{\\mathrm\{out\}\}\)is the predicted token distribution,yty\_\{t\}is the ground\-truth token,eyte\_\{y\_\{t\}\}is its one\-hot vector, andℒt=−log⁡pt​\[yt\]\\mathcal\{L\}\_\{t\}=\-\\log p\_\{t\}\[y\_\{t\}\]is the token\-level cross\-entropy loss\. The vectorRmR\_\{m\}represents the supervised correction signal exposed by this experience: it specifies the local hidden\-space direction that would increase the likelihood of the target output under the current model\. Unlike a text summary or a semantic embedding, this signal captures the functional training value of the experience\. We therefore store its normalized formdm=Norm​\(Rm\)d\_\{m\}=\\mathrm\{Norm\}\(R\_\{m\}\)as a reusable adaptation direction in memory\. This direction is first\-order optimal for locally decreasing the supervised loss of the experience in the pre\-LM\-head hidden space \(Appendix[A](https://arxiv.org/html/2607.22556#A1), Proposition[A\.1](https://arxiv.org/html/2607.22556#A1.Thmtheorem1)\)\.

#### 3\.2\.3Budget\-Aware Memory Management

Given the semantic keykmk\_\{m\}and correction directiondmd\_\{m\}extracted from a new experience, MIITA writes it into the memory bank by first updating the direction prototypes and then repairing the memory if the storage budget is exceeded\.

Prototype assignment and update\.If\|ℬ\|=0\|\\mathcal\{B\}\|=0, MIITA directly creates a new prototypeunew=\(dm,\{km\},1\)u\_\{\\mathrm\{new\}\}=\(d\_\{m\},\\\{k\_\{m\}\\\},1\)\. Otherwise, it finds the closest existing prototype in the correction\-direction space:

j⋆=arg⁡max1≤j≤J⁡cos⁡\(dm,dj\)\.j^\{\\star\}=\\arg\\max\_\{1\\leq j\\leq J\}\\cos\(d\_\{m\},d\_\{j\}\)\.Letτd\\tau\_\{d\}be the direction\-merge threshold\. Ifcos⁡\(dm,dj⋆\)<τd\\cos\(d\_\{m\},d\_\{j^\{\\star\}\}\)<\\tau\_\{d\}, the experience is not sufficiently aligned with any existing prototype, so it is treated as a new direction and stored asunew=\(dm,\{km\},1\)u\_\{\\mathrm\{new\}\}=\(d\_\{m\},\\\{k\_\{m\}\\\},1\)\. Otherwise, it is merged intouj⋆u\_\{j^\{\\star\}\}by updating the prototype direction with a running average:

dj⋆←Norm​\(nj⋆​dj⋆\+dm\),nj⋆←nj⋆\+1\.d\_\{j^\{\\star\}\}\\leftarrow\\mathrm\{Norm\}\\left\(n\_\{j^\{\\star\}\}d\_\{j^\{\\star\}\}\+d\_\{m\}\\right\),\\qquad n\_\{j^\{\\star\}\}\\leftarrow n\_\{j^\{\\star\}\}\+1\.\(3\)
After merging intouj⋆u\_\{j^\{\\star\}\}, MIITA useskmk\_\{m\}only to expand the semantic access points of the assigned prototype\. With semantic\-anchor novelty thresholdτk\\tau\_\{k\}, we update

𝒜j⋆←𝒜j⋆∪\{km\}if​maxa∈𝒜j⋆⁡cos⁡\(km,a\)<τk\.\\mathcal\{A\}\_\{j^\{\\star\}\}\\leftarrow\\mathcal\{A\}\_\{j^\{\\star\}\}\\cup\\\{k\_\{m\}\\\}\\quad\\text\{if \}\\max\_\{a\\in\\mathcal\{A\}\_\{j^\{\\star\}\}\}\\cos\(k\_\{m\},a\)<\\tau\_\{k\}\.\(4\)Otherwise,kmk\_\{m\}is discarded as a redundant anchor\. This enables one correction direction to be retrieved from diverse contexts without storing redundant anchors or raw examples\.

Memory repair\.When the memory footprint exceedsCmemC\_\{\\mathrm\{mem\}\}, MIITA repairs memory in a direction\-preserving order: it first prunes redundant semantic anchors, then merges the most similar prototypes with minimal distortion, and finally evicts the prototype with the lowest value:

value​\(uj\)=log⁡\(1\+nj\)​\(1−maxr≠j⁡cos⁡\(dj,dr\)\)Size​\(uj\)\.\\mathrm\{value\}\(u\_\{j\}\)=\\frac\{\\log\(1\+n\_\{j\}\)\\left\(1\-\\max\_\{r\\neq j\}\\cos\(d\_\{j\},d\_\{r\}\)\\right\)\}\{\\mathrm\{Size\}\(u\_\{j\}\)\}\.\(5\)This value favors prototypes that are well supported, directionally distinctive, and storage\-efficient\. As formalized by the directional coverage view at the end of Section[3\.2\.4](https://arxiv.org/html/2607.22556#S3.SS2.SSS4), the benefit of memory depends on how well stored directions cover historical corrections; thus, the limited budget is allocated to diverse reusable correction directions rather than raw examples\.

#### 3\.2\.4Inference\-Time Retrieval and Adaptation

After continual training, given a queryqqat inference time, MIITA first constructs query\-side retrieval signals, then retrieves relevant historical correction directions, and finally applies the retrieved direction through temporary hidden\-state adaptation to locally improve the current prediction without permanently modifying the model\.

Query\-side retrieval signals\.MIITA computes the query semantic keykqk\_\{q\}by applying the same pooling function in Eq\. \([1](https://arxiv.org/html/2607.22556#S3.E1)\) to the query token positions\. This key provides the semantic signal for accessing memory\. Since the ground\-truth output is unavailable at inference time, MIITA also constructs an unsupervised directional proxy from local prediction uncertainty\. Lethqh\_\{q\}be the pre\-LM\-head hidden state at the answer\-start or first probe decoding position, and letpq=softmax​\(WLM​hq\)p\_\{q\}=\\mathrm\{softmax\}\(W\_\{\\mathrm\{LM\}\}h\_\{q\}\)\. We keep the top\-KvK\_\{v\}token set𝒱q\\mathcal\{V\}\_\{q\}underpqp\_\{q\}and renormalize their probabilities asp^q​\(v\)=pq​\(v\)/∑v′∈𝒱qpq​\(v′\)\\hat\{p\}\_\{q\}\(v\)=p\_\{q\}\(v\)/\\sum\_\{v^\{\\prime\}\\in\\mathcal\{V\}\_\{q\}\}p\_\{q\}\(v^\{\\prime\}\)forv∈𝒱qv\\in\\mathcal\{V\}\_\{q\}\. The restricted entropy is thenH^q=−∑v∈𝒱qp^q​\(v\)​log⁡p^q​\(v\)\\hat\{H\}\_\{q\}=\-\\sum\_\{v\\in\\mathcal\{V\}\_\{q\}\}\\hat\{p\}\_\{q\}\(v\)\\log\\hat\{p\}\_\{q\}\(v\)\. The gradient ofH^q\\hat\{H\}\_\{q\}with respect to the restricted logits is

gq=−p^q⊙\(log⁡p^q\+H^q​𝟏\)\.g\_\{q\}=\-\\hat\{p\}\_\{q\}\\odot\\left\(\\log\\hat\{p\}\_\{q\}\+\\hat\{H\}\_\{q\}\\mathbf\{1\}\\right\)\.\(6\)Projecting the restricted entropy gradient back to the hidden space gives

d~q=Norm​\(−rq\),where​rq=W𝒱q⊤​gq\.\\tilde\{d\}\_\{q\}=\\mathrm\{Norm\}\(\-r\_\{q\}\),\\qquad\\text\{where \}r\_\{q\}=W\_\{\\mathcal\{V\}\_\{q\}\}^\{\\top\}g\_\{q\}\.\(7\)If the norm ofrqr\_\{q\}exceeds a stability thresholdϵr\\epsilon\_\{r\}, MIITA usesd~q\\tilde\{d\}\_\{q\}as the entropy\-decreasing query direction\. This direction is the steepest first\-order direction for decreasing the restricted predictive entropyH^q\\hat\{H\}\_\{q\}at the current query state \(Appendix[A](https://arxiv.org/html/2607.22556#A1), Proposition[A\.2](https://arxiv.org/html/2607.22556#A1.Thmtheorem2)\), and therefore serves as a label\-free geometric cue for retrieving historical directions that may affect the current prediction\.

Memory retrieval and direction aggregation\.Givenkqk\_\{q\}and, when available,d~q\\tilde\{d\}\_\{q\}, MIITA retrieves memory prototypes from both semantic and directional channels\. For each prototypeuju\_\{j\}, its semantic relevance is measured by the best matching anchor,Sk​\(q,j\)=maxa∈𝒜j⁡cos⁡\(kq,a\)S\_\{k\}\(q,j\)=\\max\_\{a\\in\\mathcal\{A\}\_\{j\}\}\\cos\(k\_\{q\},a\)\. The candidate set is defined as

𝒞q=TopRk⁡\(Sk​\(q,j\)\)∪\{TopRd⁡\(cos⁡\(d~q,dj\)\),if​d~q​is available,∅,otherwise\.\\mathcal\{C\}\_\{q\}=\\operatorname\{Top\}\_\{R\_\{k\}\}\\\!\\left\(S\_\{k\}\(q,j\)\\right\)\\cup\\begin\{cases\}\\operatorname\{Top\}\_\{R\_\{d\}\}\\\!\\left\(\\cos\(\\tilde\{d\}\_\{q\},d\_\{j\}\)\\right\),&\\text\{if \}\\tilde\{d\}\_\{q\}\\text\{ is available\},\\\\ \\varnothing,&\\text\{otherwise\}\.\\end\{cases\}\(8\)Here,TopRk\\operatorname\{Top\}\_\{R\_\{k\}\}andTopRd\\operatorname\{Top\}\_\{R\_\{d\}\}return the prototype indices with the highest semantic and directional scores, respectively\. For each candidateuju\_\{j\}withj∈𝒞qj\\in\\mathcal\{C\}\_\{q\}, MIITA computes

score​\(q,j\)=λk​Sk​\(q,j\)\+λu​U​\(q,j\),where​U​\(q,j\)=max⁡\(0,−gq⊤​ϵ​W𝒱q​dj\)\.\\mathrm\{score\}\(q,j\)=\\lambda\_\{k\}S\_\{k\}\(q,j\)\+\\lambda\_\{u\}U\(q,j\),\\qquad\\text\{where \}U\(q,j\)=\\max\\left\(0,\-g\_\{q\}^\{\\top\}\\epsilon W\_\{\\mathcal\{V\}\_\{q\}\}d\_\{j\}\\right\)\.\(9\)Here,U​\(q,j\)U\(q,j\)estimates whether moving alongdjd\_\{j\}would locally reduce the query uncertainty,ϵ\\epsilonis a small probe scale, andλk,λu≥0\\lambda\_\{k\},\\lambda\_\{u\}\\geq 0balance semantic relevance and directional utility\. If the directional proxy is unavailable, we setU​\(q,j\)=0U\(q,j\)=0\. The top\-MMcandidates underscore​\(q,j\)\\mathrm\{score\}\(q,j\)are selected asSqS\_\{q\}and aggregated into a query\-specific correction directiond¯q\\bar\{d\}\_\{q\}:

d¯q=Norm​\(∑j∈Sqwj​dj\),where​wj=exp⁡\(score​\(q,j\)/Tagg\)∑r∈Sqexp⁡\(score​\(q,r\)/Tagg\)\.\\bar\{d\}\_\{q\}=\\mathrm\{Norm\}\\left\(\\sum\_\{j\\in S\_\{q\}\}w\_\{j\}d\_\{j\}\\right\),\\qquad\\text\{where \}w\_\{j\}=\\frac\{\\exp\(\\mathrm\{score\}\(q,j\)/T\_\{\\mathrm\{agg\}\}\)\}\{\\sum\_\{r\\in S\_\{q\}\}\\exp\(\\mathrm\{score\}\(q,r\)/T\_\{\\mathrm\{agg\}\}\)\}\.\(10\)Here,TaggT\_\{\\mathrm\{agg\}\}is the softmax temperature\. The resultingd¯q\\bar\{d\}\_\{q\}summarizes the historical correction directions retrieved for the current query\.

Temporary hidden adaptation\.The aggregated directiond¯q\\bar\{d\}\_\{q\}is applied during generation through a query\-specific hidden update\. Rather than applying the same correction uniformly to all decoding states, MIITA uses an activation gate to make the correction stronger when the current generation state is close to the query state that triggered retrieval:

pt′=softmax​\(WLM​\(htout\+η​αt​d¯q\)\),where​αt=max⁡\(0,cos⁡\(Norm​\(hq\),Norm​\(htout\)\)\)\.p\_\{t\}^\{\\prime\}=\\mathrm\{softmax\}\(W\_\{\\mathrm\{LM\}\}\(h\_\{t\}^\{\\mathrm\{out\}\}\+\\eta\\alpha\_\{t\}\\bar\{d\}\_\{q\}\)\),\\qquad\\text\{where \}\\alpha\_\{t\}=\\max\(0,\\cos\(\\mathrm\{Norm\}\(h\_\{q\}\),\\mathrm\{Norm\}\(h\_\{t\}^\{\\mathrm\{out\}\}\)\)\)\.\(11\)Here,η\\etais the adaptation scale\. Both the direction aggregation in Eq\. \([10](https://arxiv.org/html/2607.22556#S3.E10)\) and the activation gate in Eq\. \([11](https://arxiv.org/html/2607.22556#S3.E11)\) admit a local least\-squares interpretation \(Appendix[A](https://arxiv.org/html/2607.22556#A1), Proposition[A\.3](https://arxiv.org/html/2607.22556#A1.Thmtheorem3)\)\. An ablation in Appendix[D\.1](https://arxiv.org/html/2607.22556#A4.SS1)further confirms that this activation gate is necessary, as uniformly applying the retrieved direction to all decoding states leads to worse performance and stronger forgetting\. Thus, the inference\-time update can be viewed as a low\-cost rank\-one correction in hidden space, modulated by local state similarity\.

###### Theorem \(Directional coverage bound, informal\)\.

Suppose the supervised loss isβ\\beta\-smooth in the pre\-LM\-head hidden space\. For an old\-stage examplezz, applying inference\-time hidden adaptation with retrieved directiond¯z\\bar\{d\}\_\{z\}reduces the old\-stage risk to first order by a term proportional to

η​‖Gzα‖2​⟨bz,d¯z⟩−β​η22,\\eta\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}\\langle b\_\{z\},\\bar\{d\}\_\{z\}\\rangle\-\\frac\{\\beta\\eta^\{2\}\}\{2\},whereGzαG\_\{z\}^\{\\alpha\}is the gated old\-task correction vector andbz=Gzα/‖Gzα‖2b\_\{z\}=G\_\{z\}^\{\\alpha\}/\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}\. Thus, the benefit of memory is controlled by its*directional coverage*of historical corrections, rather than by raw example coverage\. The full statement and proof are given in Appendix[A](https://arxiv.org/html/2607.22556#A1), Theorem[A\.4](https://arxiv.org/html/2607.22556#A1.Thmtheorem4)\.

This bound clarifies the role of memory in MIITA: stored prototypes should cover diverse historical correction directions that can reduce old\-stage risk after retrieval and adaptation\. This provides a unified theoretical view of the design choices, including direction prototypes in Section[3\.2\.1](https://arxiv.org/html/2607.22556#S3.SS2.SSS1), direction\-preserving memory repair in Section[3\.2\.3](https://arxiv.org/html/2607.22556#S3.SS2.SSS3), and temporary hidden adaptation\. The update is discarded after generation, enabling local adaptation without modifying the backbone, updating the LM head, extending the prompt with retrieved examples, or running test\-time backpropagation\. This is especially suitable for SLM CL, where repeatedly writing new experiences into a capacity\-limited backbone can overwrite prior capabilities\. An inference\-time complexity analysis is in Appendix[B\.1](https://arxiv.org/html/2607.22556#A2.SS1)\.

## 4Experiments

### 4\.1Experimental Setup

Benchmarks\.We evaluate our method on four supervised CL benchmarks covering classification, question answering \(QA\), and generation tasks: \(i\)class\-incremental learning \(CIL\)onBanking77\[[1](https://arxiv.org/html/2607.22556#bib.bib12)\], an intent classification dataset with 77 classes, split into 7 sequential tasks with 11 new intents per task; \(ii\)domain\-incremental learning \(DIL\)on two benchmarks:Document Sentiment Classification\(DSC\)\[[11](https://arxiv.org/html/2607.22556#bib.bib13)\], which contains 10 product\-review domains with a fixed positive/negative label space, andPAXQA\[[12](https://arxiv.org/html/2607.22556#bib.bib14)\], a multilingual QA dataset where 4 languages are treated as sequential domains; and \(iii\)task\-incremental learning \(TIL\)on theSuperNI Benchmark\[[27](https://arxiv.org/html/2607.22556#bib.bib19)\], a 15\-task instruction\-following benchmark covering dialogue generation, information extraction, QA, summarization, and sentiment classification\.

Baselines\.We compare MIITA with three groups of baselines: \(i\) general language model CL baselines, including LoRA\[[9](https://arxiv.org/html/2607.22556#bib.bib17)\]and MBPA\+\+\[[4](https://arxiv.org/html/2607.22556#bib.bib16)\]; \(ii\) memory\-based LLM CL baselines, including CIS\[[18](https://arxiv.org/html/2607.22556#bib.bib6)\], InCA\[[17](https://arxiv.org/html/2607.22556#bib.bib7)\], CT0\[[19](https://arxiv.org/html/2607.22556#bib.bib4)\], and FOREVER\[[6](https://arxiv.org/html/2607.22556#bib.bib5)\]; and \(iii\) SLM\-oriented CL baselines, including DKVB\[[5](https://arxiv.org/html/2607.22556#bib.bib1)\]and Srinath Ket al\.\[[21](https://arxiv.org/html/2607.22556#bib.bib3)\]\. Implementation details and the adaptation of each baseline to our supervised SLM CL setting are provided in Appendix[C\.1](https://arxiv.org/html/2607.22556#A3.SS1)\.

Evaluation Metrics\.We evaluate CL performance using a performance matrix𝐀∈ℝT×T\\mathbf\{A\}\\in\\mathbb\{R\}^\{T\\times T\}, whereAt,τA\_\{t,\\tau\}denotes the task\-specific score on taskτ\\tauafter the model has learned tasktt\. For classification tasks,At,τA\_\{t,\\tau\}is accuracy; for QA tasks,At,τA\_\{t,\\tau\}is exact match; and for generation tasks inSuperNI,At,τA\_\{t,\\tau\}is Rouge\-L\. All scores are scaled to\[0,100\]\[0,100\]\. We reportOverall Performance \(OP\)asOP=1T​∑τ=1TAT,τ\\mathrm\{OP\}=\\frac\{1\}\{T\}\\sum\_\{\\tau=1\}^\{T\}A\_\{T,\\tau\}, which measures the final average task score over all tasks, andBackward Transfer \(BWT\)asBWT=1T−1​∑τ=1T−1\(AT,τ−Aτ,τ\)\\mathrm\{BWT\}=\\frac\{1\}\{T\-1\}\\sum\_\{\\tau=1\}^\{T\-1\}\(A\_\{T,\\tau\}\-A\_\{\\tau,\\tau\}\), which measures how learning later tasks affects earlier tasks\. Higher OP and larger BWT are better\.

Implementation Details\.We evaluate all methods on instruction\-tuned base models, including Qwen3 \(0\.6B, 1,7B, 4B\)\[[28](https://arxiv.org/html/2607.22556#bib.bib20)\]and LLaMA \(3\.2\-3B, 3\.1\-8B\)\[[23](https://arxiv.org/html/2607.22556#bib.bib21)\]\. Unless otherwise specified, we use a fixed 1% memory budget, measured as the byte size of 1% of the full training stream\. For MIITA, we set the direction\-merge threshold toτd=0\.85\\tau\_\{d\}=0\.85and the semantic\-anchor novelty threshold toτk=0\.80\\tau\_\{k\}=0\.80\. A sensitivity analysis in Appendix[D\.2](https://arxiv.org/html/2607.22556#A4.SS2)shows that MIITA remains stable across both thresholds\. During inference, we retrieveRk=Rd=8R\_\{k\}=R\_\{d\}=8candidates and aggregate the topM=4M=4prototypes after reranking with weightsλk=1\.0\\lambda\_\{k\}=1\.0andλu=0\.5\\lambda\_\{u\}=0\.5, usingT=0\.2T=0\.2andη=0\.5\\eta=0\.5\. Results average three independent runs\. Algorithm[1](https://arxiv.org/html/2607.22556#alg1)in Appendix[B\.2](https://arxiv.org/html/2607.22556#A2.SS2)details the implementation\.

### 4\.2Overall Performance

Table 1:Overall continual learning performance on four datasets\.Boldindicates the best result,underlineindicates the second\-best result, and↑\\uparrowindicates that higher values are better\.BaseModelMethodMemory\-BasedDesignedForBanking77DSCPAXQASuperNIOP↑\\uparrowBWT↑\\uparrowOP↑\\uparrowBWT↑\\uparrowOP↑\\uparrowBWT↑\\uparrowOP↑\\uparrowBWT↑\\uparrowQwen3\-0\.6BLoRA\[[9](https://arxiv.org/html/2607.22556#bib.bib17)\]✗–65\.31\-11\.4466\.04\-11\.5340\.75\-5\.5731\.24\-8\.91MBPA\+\+\[[4](https://arxiv.org/html/2607.22556#bib.bib16)\]✓LM67\.82\-10\.2267\.22\-9\.1240\.48\-6\.1231\.58\-10\.21CIS\[[18](https://arxiv.org/html/2607.22556#bib.bib6)\]✓LLM66\.78\-10\.6065\.47\-10\.6638\.96\-6\.7426\.51\-11\.05InCA\[[17](https://arxiv.org/html/2607.22556#bib.bib7)\]✓LLM71\.20\-8\.7266\.79\-10\.2539\.47\-6\.1728\.58\-11\.17CT0\[[19](https://arxiv.org/html/2607.22556#bib.bib4)\]✓LLM68\.79\-9\.7867\.71\-9\.7541\.46\-5\.7530\.43\-10\.82FOREVER\[[6](https://arxiv.org/html/2607.22556#bib.bib5)\]✓LLM70\.58\-9\.1468\.12\-9\.6342\.54\-5\.2432\.82\-9\.95DKVB\[[5](https://arxiv.org/html/2607.22556#bib.bib1)\]✗SLM73\.12\-7\.4375\.47\-7\.2353\.45\-4\.6437\.89\-6\.72Srinath Ket al\.\[[21](https://arxiv.org/html/2607.22556#bib.bib3)\]✓SLM70\.44\-7\.9269\.42\-9\.2455\.64\-4\.4335\.54\-7\.44MIITA \(ours\)✓SLM77\.67\-6\.1778\.29\-6\.1460\.01\-3\.1240\.58\-4\.89Qwen3\-4BLoRA\[[9](https://arxiv.org/html/2607.22556#bib.bib17)\]✗–68\.42\-9\.8569\.38\-9\.7444\.92\-4\.8234\.17\-7\.96MBPA\+\+\[[4](https://arxiv.org/html/2607.22556#bib.bib16)\]✓LM71\.36\-8\.7170\.54\-7\.8244\.15\-5\.0536\.35\-8\.41CIS\[[18](https://arxiv.org/html/2607.22556#bib.bib6)\]✓LLM71\.84\-8\.8869\.75\-8\.9243\.05\-5\.7338\.40\-8\.92InCA\[[17](https://arxiv.org/html/2607.22556#bib.bib7)\]✓LLM76\.48\-6\.9570\.95\-8\.7643\.74\-5\.2840\.12\-8\.64CT0\[[19](https://arxiv.org/html/2607.22556#bib.bib4)\]✓LLM73\.92\-7\.7871\.86\-8\.1546\.18\-4\.5242\.68\-8\.37FOREVER\[[6](https://arxiv.org/html/2607.22556#bib.bib5)\]✓LLM75\.10\-7\.2172\.64\-7\.6547\.32\-4\.1244\.00\-8\.05DKVB\[[5](https://arxiv.org/html/2607.22556#bib.bib1)\]✗SLM76\.22\-6\.3178\.10\-6\.0557\.82\-3\.9644\.76\-6\.08Srinath Ket al\.\[[21](https://arxiv.org/html/2607.22556#bib.bib3)\]✓SLM73\.18\-6\.9472\.68\-7\.8661\.47\-3\.6241\.85\-6\.91MIITA \(ours\)✓SLM81\.05\-5\.0281\.66\-4\.9565\.21\-2\.3848\.21\-4\.27Table[1](https://arxiv.org/html/2607.22556#S4.T1)reports the main results for Qwen3\-0\.6B and Qwen3\-4B under a 1% persistent memory budget\. MIITA consistently achieves the best OP and BWT across all benchmarks\. This simultaneous improvement confirms that MIITA genuinely enhances the stability\-plasticity trade\-off, successfully incorporating new experiences without simply overfitting later stages at the expense of earlier ones\. Crucially, MIITA overcomes the fundamental limitations of both parameter\-update and textual\-memory paradigms\. Parameter\-updating baselines \(e\.g\., LoRA\) suffer from severe forgetting, while SLM\-oriented methods show only task\-dependent gains, confirming that repeatedly modifying limited SLM capacity is unreliable\. Similarly, memory\-based LLM methods struggle under strict storage budgets, especially on the smaller 0\.6B backbone and open\-ended tasks\. This validates our intuition that textual memory is inefficient for SLMs, as it consumes surface\-form storage and demands strong in\-context reasoning\. Overall, MIITA’s consistent superiority validates its core design\. Rather than destructively overwriting parameters or relying on surface textual prompts, MIITA preserves compact functional correction directions\. Applying these directions via disposable inference\-time hidden adaptation provides a more direct, reusable, and non\-destructive mechanism for continual learning\.

### 4\.3Model\-Size Scaling under a Fixed Memory Budget

![Refer to caption](https://arxiv.org/html/2607.22556v1/x2.png)Figure 2:Model\-size scaling onSuperNIunder 1% memory budget\.To investigate whether existing methods remain effective as the base model becomes smaller under the same memory budget, we conduct model scaling experiments, as shown in Figure[2](https://arxiv.org/html/2607.22556#S4.F2)\. The results reveal a severe degradation in memory methods designed for LLMs\. Specifically, replay\-based methods \(CT0, FOREVER\) drop substantially, highlighting their reliance on large parameter capacity to absorb and retain knowledge\. Similarly, prompting\-based methods \(CIS, InCA\) falter because SLMs lack the strong in\-context reasoning required to utilize textual memories\. While SLM\-oriented baselines \(DKVB, Srinath Ket al\.\) exhibit better relative robustness, their performance still declines, indicating that purely architectural or replay\-adapter solutions are insufficient\. In contrast, MIITA consistently achieves the best OP and BWT across all scales\. By preserving compact functional correction directions rather than raw text or examples, MIITA ensures that historical memory remains highly effective despite limited model capacity and tight storage budgets\.

### 4\.4Memory Budget Sensitivity

![Refer to caption](https://arxiv.org/html/2607.22556v1/x3.png)Figure 3:Budget sensitivity onSuperNIwith Qwen3\-0\.6B for memory\-based methods\. Each column denotes a fixed auxiliary memory budget;∞\\inftydenotes the unlimited\-memory setting\.Figure[3](https://arxiv.org/html/2607.22556#S4.F3)evaluates memory\-based CL methods under increasing storage budgets, focusing on those that scale with external memory and omitting fixed\-storage baselines such as LoRA and DKVB\. The results show that existing memory\-based LLM CL methods are highly sensitive to available storage: replay\-based methods like CT0 and FOREVER improve substantially as the budget increases, with the latter becoming strongest in the practically unlimited\-memory setting by leveraging large buffers to retain extensive raw supervision from previous tasks\. However, their performance drops sharply under tight budgets, indicating that replaying old examples is inefficient when memory is minimal\. In contrast, MIITA remains consistently strong in the low\-budget regime \(0\.1% to 2%\), where the gap between our approach and memory methods designed for LLMs is most pronounced\. This supports our central motivation: for SLM continual learning, the key challenge is not merely whether memory is used, but whether the stored representation is compact and functionally useful\. While methods storing raw examples, summaries, or prompt contexts require significantly larger budgets to be competitive, MIITA stores supervised correction directions that directly encode reusable update signals\.

### 4\.5Effect of Functional Correction Directions

A key question in MIITA is whether supervised experiences should be stored as textual or semantic content, or as functional correction directions that encode their effect on the model prediction\.

![Refer to caption](https://arxiv.org/html/2607.22556v1/x4.png)

\(a\)

![Refer to caption](https://arxiv.org/html/2607.22556v1/x5.png)

\(b\)

Figure 4:Analysis of why MIITA stores functional correction directions onSuperNI\.\(a\)Comparison of memory variants with Qwen3\-0\.6B under a fixed 1% memory budget\.\(b\)Pairwise comparison between semantic and functional similarity, where each point represents a pair of training examples\.Memory variant comparison\.To first isolate the effect of what is stored in memory, we compare four variants differing exclusively in their memory content:ExampleMemstores raw supervised examples\(x,y\)\(x,y\),SummaryMemstores textual summaries of historical examples,SemanticMemstores semantic embedding prototypes, andDirectionMemstores correction\-direction prototypes induced by supervised examples\. Implementation details are provided in Appendix[C\.2](https://arxiv.org/html/2607.22556#A3.SS2)\. As shown in Figure[4](https://arxiv.org/html/2607.22556#S4.F4)\(a\), ExampleMem and SummaryMem are limited by the weak in\-context utilization of small models, since they require the SLM to reuse retrieved textual contexts\. SemanticMem avoids long textual memories and is more compact, but its semantic\-redundancy\-based compression does not reliably preserve the functional update signals needed under a tight storage budget\. In contrast, DirectionMem directly stores correction\-direction prototypes, preserving how past supervised examples should change the model’s prediction\. Its superior OP and forgetting results show that MIITA benefits from capturing functional redundancy, rather than merely storing human\-readable text or semantic similarity\.

Semantic vs\. functional redundancy\.To further validate the divergence between semantic and functional redundancy, we compare pairwise cosine similarity among training examples, using semantic keys for semantic similarity and correction directions for functional similarity\. The two measures are only weakly correlated, with Pearson correlation 0\.18 and Spearman correlation 0\.21, as visualized in Figure[4](https://arxiv.org/html/2607.22556#S4.F4)\(b\)\. The Pearson correlation corresponds to only aboutR2≈0\.03R^\{2\}\\approx 0\.03, meaning that semantic similarity explains less than 5% of the variance in functional similarity\. Consistent with this, the points are widely scattered rather than concentrated along an increasing diagonal trend: semantically similar examples can still require very different correction directions, while semantically distant examples may share useful functional updates\. This suggests that purely semantic compression can miss many functionally important directions, whereas direction\-level prototypes preserve supervised update signals according to their functional effects\.

### 4\.6Validating Uncertainty\-Guided Retrieval

Table 2:Validation of uncertainty\-guided retrieval onSuperNIwith Qwen3\-0\.6B under 1% memory budget\. OracleRet is the label\-informed upper bound\.Retrieval VariantOP↑\\uparrowBWT↑\\uparrowOA↑\\uparrowRandomRet28\.85\-12\.630\.03SemanticRet36\.21\-6\.980\.35UncertaintyRet34\.74\-8\.920\.42SemUncRet \(Ours\)40\.58\-4\.890\.52OracleRet47\.29\-2\.42–To evaluate MIITA’s retrieval design by ablating inference\-time query signals, we compare variants differing exclusively in how prototypes are retrieved:RandomRetuniformly samples prototypes;SemanticRetrelies solely on semantic\-key similaritySk​\(q,j\)S\_\{k\}\(q,j\);UncertaintyRetscores stored directionsdjd\_\{j\}using only the uncertainty\-induced direction;SemUncRetemploys the full MIITA strategy; andOracleRetprovides an upper bound using a label\-derived oracle correction direction\. To quantify consistency with supervised corrections, we reportOracle Alignment \(OA\), the average cosine alignment between the retrieved and oracle directions\. Implementation details of the retrieval variants and OA are provided in Appendix[C\.3](https://arxiv.org/html/2607.22556#A3.SS3)\. Table[2](https://arxiv.org/html/2607.22556#S4.T2)confirms uncertainty is an effective query\-side proxy\. UncertaintyRet outperforms RandomRet and achieves higher OA than SemanticRet, proving it provides a meaningful label\-free geometric cue\. However, UncertaintyRet yields lower OP and BWT than SemanticRet because uncertainty captures local geometry but lacks semantic relevance\. The optimal variant, SemUncRet, resolves this: semantic recall ensures relevance, while uncertainty prioritizes correction utility\. This validates MIITA’s design: uncertainty is a powerful label\-free proxy, maximizing effectiveness when combined with semantic recall for reranking\.

## 5Conclusion

We introduced MIITA, a memory\-induced inference\-time adaptation framework for supervised continual learning \(CL\) in small language models \(SLMs\) under constrained persistent storage\. Unlike replay\-based or prompting\-based memory methods, MIITA preserves historical experiences as compact functional correction directions rather than raw examples, textual summaries, or long contexts\. These directions are organized into budget\-aware prototypes with lightweight semantic anchors and are invoked at inference time through semantic and uncertainty\-guided retrieval followed by gated temporary hidden\-state adaptation\. This enables non\-destructive reuse of past supervision without backbone updates, prompt extension, or test\-time backpropagation\. Experiments across diverse CL settings show that MIITA improves final performance and mitigates forgetting under fixed memory budgets, while further analyses demonstrate its robustness across model scales and memory budgets and validate the roles of functional correction memory and uncertainty\-guided retrieval\. Overall, this work suggests that SLM CL can benefit from shifting memory design away from storing textual content toward storing reusable functional adaptation signals\.

## References

- \[1\]I\. Casanueva, T\. Temčinas, D\. Gerz, M\. Henderson, and I\. Vulić\(2020\-07\)Efficient intent detection with dual sentence encoders\.InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI,T\. Wen, A\. Celikyilmaz, Z\. Yu, A\. Papangelis, M\. Eric, A\. Kumar, I\. Casanueva, and R\. Shah \(Eds\.\),Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.nlp4convai-1.5/),[Document](https://dx.doi.org/10.18653/v1/2020.nlp4convai-1.5)Cited by:[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p1.1)\.
- \[2\]E\. Chen, W\. Fang, S\. Wang, and C\. Brinton\(2026\)Joint continual learning of local language models and cloud offloading decisions with budget constraints\.arXiv preprint arXiv:2602\.00166\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p1.1),[§2](https://arxiv.org/html/2607.22556#S2.p1.1)\.
- \[3\]P\. Das, S\. Chaudhury, E\. Nelson, I\. Melnyk, S\. Swaminathan, S\. Dai, A\. Lozano, G\. Kollias, V\. Chenthamarakshan, S\. Dan,et al\.\(2024\)Larimar: large language models with episodic memory control\.arXiv preprint arXiv:2403\.11901\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p2.1)\.
- \[4\]C\. de Masson D’Autume, S\. Ruder, L\. Kong, and D\. Yogatama\(2019\)Episodic memory in lifelong language learning\.Advances in Neural Information Processing Systems32\.Cited by:[2nd item](https://arxiv.org/html/2607.22556#A3.I1.i2.p1.2),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.11.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.20.1)\.
- \[5\]A\. Diera, L\. Galke, F\. Karl, and A\. Scherp\(2025\)Efficient continual learning for small language models with a discrete key\-value bottleneck\.InProceedings of the 8th International Conference on Natural Language and Speech Processing \(ICNLSP\-2025\),pp\. 162–176\.Cited by:[7th item](https://arxiv.org/html/2607.22556#A3.I1.i7.p1.1),[§1](https://arxiv.org/html/2607.22556#S1.p1.1),[§2](https://arxiv.org/html/2607.22556#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.16.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.25.1)\.
- \[6\]Y\. Feng, H\. Wang, J\. Li, X\. Chu, Z\. Kang, Y\. Liu, Y\. Wang, P\. S\. Yu, and X\. Wu\(2026\)FOREVER: forgetting curve\-inspired memory replay for language model continual learning\.arXiv preprint arXiv:2601\.03938\.Cited by:[4th item](https://arxiv.org/html/2607.22556#A3.I1.i4.p1.1),[Figure 1](https://arxiv.org/html/2607.22556#S1.F1),[Figure 1](https://arxiv.org/html/2607.22556#S1.F1.4.2),[§2](https://arxiv.org/html/2607.22556#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.15.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.24.1)\.
- \[7\]B\. J\. Gutiérrez, Y\. Shu, Y\. Gu, M\. Yasunaga, and Y\. Su\(2024\)Hipporag: neurobiologically inspired long\-term memory for large language models\.Advances in neural information processing systems37,pp\. 59532–59569\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p2.1)\.
- \[8\]B\. J\. Gutiérrez, Y\. Shu, W\. Qi, S\. Zhou, and Y\. Su\(2025\)From rag to memory: non\-parametric continual learning for large language models\.arXiv preprint arXiv:2502\.14802\.Cited by:[§2](https://arxiv.org/html/2607.22556#S2.p2.1)\.
- \[9\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[1st item](https://arxiv.org/html/2607.22556#A3.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.10.2),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.19.2)\.
- \[10\]T\. Katraouras and D\. Rafailidis\(2026\)Memory bank compression for continual adaptation of large language models\.arXiv preprint arXiv:2601\.00756\.Cited by:[§2](https://arxiv.org/html/2607.22556#S2.p2.1)\.
- \[11\]Z\. Ke, B\. Liu, N\. Ma, H\. Xu, and L\. Shu\(2021\)Achieving forgetting prevention and knowledge transfer in continual learning\.Advances in Neural Information Processing Systems34,pp\. 22443–22456\.Cited by:[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p1.1)\.
- \[12\]B\. Li and C\. Callison\-Burch\(2023\)Paxqa: generating cross\-lingual question answering examples at training scale\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 439–454\.Cited by:[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p1.1)\.
- \[13\]D\. Li, Z\. Sun, X\. Hu, B\. Hu, and M\. Zhang\(2025\)Cmt: a memory compression method for continual knowledge learning of large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 24413–24421\.Cited by:[§2](https://arxiv.org/html/2607.22556#S2.p2.1)\.
- \[14\]Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, X\. Zhang, N\. D\. Lane, and M\. Xu\(2024\)Small language models: survey, measurements, and insights\.arXiv preprint arXiv:2409\.15790\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p1.1)\.
- \[15\]Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang\(2025\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.IEEE Transactions on Audio, Speech and Language Processing\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p2.1)\.
- \[16\]J\. Mok, J\. Do, S\. Lee, T\. Taghavi, S\. Yu, and S\. Yoon\(2023\)Large\-scale lifelong learning of in\-context instructions and how to tackle it\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12573–12589\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p2.1)\.
- \[17\]S\. Momeni, S\. Mazumder, Z\. Ke, and B\. Liu\(2025\)In\-context continual learning assisted by an external continual learner\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 7292–7306\.Cited by:[6th item](https://arxiv.org/html/2607.22556#A3.I1.i6.p1.1),[§2](https://arxiv.org/html/2607.22556#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.13.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.22.1)\.
- \[18\]J\. Qiu, Z\. Ke, and B\. Liu\(2025\)Continual learning using only large language model prompting\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 6014–6023\.Cited by:[5th item](https://arxiv.org/html/2607.22556#A3.I1.i5.p1.1),[Figure 1](https://arxiv.org/html/2607.22556#S1.F1),[Figure 1](https://arxiv.org/html/2607.22556#S1.F1.4.2),[§1](https://arxiv.org/html/2607.22556#S1.p2.1),[§2](https://arxiv.org/html/2607.22556#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.12.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.21.1)\.
- \[19\]T\. Scialom, T\. Chakrabarty, and S\. Muresan\(2022\-12\)Fine\-tuned language models are continual learners\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 6107–6122\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.410/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.410)Cited by:[3rd item](https://arxiv.org/html/2607.22556#A3.I1.i3.p1.1),[§2](https://arxiv.org/html/2607.22556#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.14.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.23.1)\.
- \[20\]H\. Shi, Z\. Xu, H\. Wang, W\. Qin, W\. Wang, Y\. Wang, Z\. Wang, S\. Ebrahimi, and H\. Wang\(2025\)Continual learning of large language models: a comprehensive survey\.ACM Computing Surveys58\(5\),pp\. 1–42\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p2.1)\.
- \[21\]M\. Somani, V\. R\. Padala, P\. Upadhyay, A\. Das,et al\.\(2026\)Continual\-learning for modelling low\-resource languages from large language models\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6258–6275\.Cited by:[8th item](https://arxiv.org/html/2607.22556#A3.I1.i8.p1.1),[§1](https://arxiv.org/html/2607.22556#S1.p1.1),[§2](https://arxiv.org/html/2607.22556#S2.p1.1),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.17.1),[Table 1](https://arxiv.org/html/2607.22556#S4.T1.10.26.1)\.
- \[22\]J\. Tack, J\. Kim, E\. Mitchell, J\. Shin, Y\. W\. Teh, and J\. R\. Schwarz\(2024\)Online adaptation of language models with a memory of amortized contexts\.Advances in Neural Information Processing Systems37,pp\. 130109–130135\.Cited by:[§2](https://arxiv.org/html/2607.22556#S2.p2.1)\.
- \[23\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p4.8)\.
- \[24\]C\. Van Nguyen, X\. Shen, R\. Aponte, Y\. Xia, S\. Basu, Z\. Hu, J\. Chen, M\. Parmar, S\. Kunapuli, J\. Barrow,et al\.\(2025\)A survey on small language models\.InProceedings of the 15th International Conference on Recent Advances in Natural Language Processing\-Natural Language Processing in the Generative AI Era,pp\. 807–821\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p1.1)\.
- \[25\]J\. Von Oswald, E\. Niklasson, E\. Randazzo, J\. Sacramento, A\. Mordvintsev, A\. Zhmoginov, and M\. Vladymyrov\(2023\)Transformers learn in\-context by gradient descent\.InInternational Conference on Machine Learning,pp\. 35151–35174\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p3.1)\.
- \[26\]F\. Wang, Z\. Zhang, X\. Zhang, Z\. Wu, T\. Mo, Q\. Lu, W\. Wang, R\. Li, J\. Xu, X\. Tang,et al\.\(2025\)A comprehensive survey of small language models in the era of large language models: techniques, enhancements, applications, collaboration with llms, and trustworthiness\.ACM Transactions on Intelligent Systems and Technology16\(6\),pp\. 1–87\.Cited by:[§1](https://arxiv.org/html/2607.22556#S1.p2.1)\.
- \[27\]Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ nlp tasks\.InProceedings of the 2022 conference on empirical methods in natural language processing,pp\. 5085–5109\.Cited by:[Figure 1](https://arxiv.org/html/2607.22556#S1.F1),[Figure 1](https://arxiv.org/html/2607.22556#S1.F1.4.2),[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p1.1)\.
- \[28\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2607.22556#S4.SS1.p4.8)\.

## Appendix ATheoretical Analysis

We provide a local theoretical analysis of MIITA to justify the main components used in Section[3\.2](https://arxiv.org/html/2607.22556#S3.SS2): correction\-direction extraction, uncertainty\-guided retrieval, temporary hidden adaptation, and direction\-memory coverage\. The analysis does not assume global convexity or task stationarity; it only uses differentiability of the LM\-head loss and first\-order local approximations\.

##### Notation\.

For a target tokenyywith pre\-LM\-head hidden stateh∈ℝHh\\in\\mathbb\{R\}^\{H\}, define

ℓy​\(h\)=−log⁡softmax​\(WLM​h\)​\[y\]\.\\ell\_\{y\}\(h\)=\-\\log\\mathrm\{softmax\}\(W\_\{\\mathrm\{LM\}\}h\)\[y\]\.Letp=softmax​\(WLM​h\)p=\\mathrm\{softmax\}\(W\_\{\\mathrm\{LM\}\}h\)\. Then

∇hℓy​\(h\)=WLM⊤​\(p−ey\),−∇hℓy​\(h\)=WLM⊤​\(ey−p\)\.\\nabla\_\{h\}\\ell\_\{y\}\(h\)=W\_\{\\mathrm\{LM\}\}^\{\\top\}\(p\-e\_\{y\}\),\\qquad\-\\nabla\_\{h\}\\ell\_\{y\}\(h\)=W\_\{\\mathrm\{LM\}\}^\{\\top\}\(e\_\{y\}\-p\)\.For an experiencemm, define its hidden\-perturbed loss

ℒm​\(δ\)=1\|𝒴m\|​∑t∈𝒴mℓyt​\(htout\+δ\)\.\\mathcal\{L\}\_\{m\}\(\\delta\)=\\frac\{1\}\{\|\\mathcal\{Y\}\_\{m\}\|\}\\sum\_\{t\\in\\mathcal\{Y\}\_\{m\}\}\\ell\_\{y\_\{t\}\}\(h\_\{t\}^\{\\mathrm\{out\}\}\+\\delta\)\.Then the correction vector used by MIITA is exactly

Rm=−∇δℒm​\(0\)\.R\_\{m\}=\-\\nabla\_\{\\delta\}\\mathcal\{L\}\_\{m\}\(0\)\.
###### Proposition A\.1\(First\-order optimality of correction directions\)\.

AssumeRm≠0R\_\{m\}\\neq 0and letdm=Rm/‖Rm‖2d\_\{m\}=R\_\{m\}/\\\|R\_\{m\}\\\|\_\{2\}\. For any unit directionddand sufficiently smallη\>0\\eta\>0,

ℒm​\(η​d\)=ℒm​\(0\)−η​⟨Rm,d⟩\+o​\(η\)\.\\mathcal\{L\}\_\{m\}\(\\eta d\)=\\mathcal\{L\}\_\{m\}\(0\)\-\\eta\\langle R\_\{m\},d\\rangle\+o\(\\eta\)\.Therefore, among all unit\-norm hidden\-space directions,dmd\_\{m\}maximizes the first\-order decrease ofℒm\\mathcal\{L\}\_\{m\}:

dm=arg⁡max‖d‖2=1⁡⟨Rm,d⟩\.d\_\{m\}=\\arg\\max\_\{\\\|d\\\|\_\{2\}=1\}\\langle R\_\{m\},d\\rangle\.

###### Proof\.

By the first\-order Taylor expansion ofℒm\\mathcal\{L\}\_\{m\}aroundδ=0\\delta=0,

ℒm​\(η​d\)=ℒm​\(0\)\+η​⟨∇δℒm​\(0\),d⟩\+o​\(η\)\.\\mathcal\{L\}\_\{m\}\(\\eta d\)=\\mathcal\{L\}\_\{m\}\(0\)\+\\eta\\langle\\nabla\_\{\\delta\}\\mathcal\{L\}\_\{m\}\(0\),d\\rangle\+o\(\\eta\)\.SinceRm=−∇δℒm​\(0\)R\_\{m\}=\-\\nabla\_\{\\delta\}\\mathcal\{L\}\_\{m\}\(0\), we obtain

ℒm​\(η​d\)=ℒm​\(0\)−η​⟨Rm,d⟩\+o​\(η\)\.\\mathcal\{L\}\_\{m\}\(\\eta d\)=\\mathcal\{L\}\_\{m\}\(0\)\-\\eta\\langle R\_\{m\},d\\rangle\+o\(\\eta\)\.The first\-order loss decrease is therefore proportional to⟨Rm,d⟩\\langle R\_\{m\},d\\rangle\. By Cauchy–Schwarz,

⟨Rm,d⟩≤‖Rm‖2​‖d‖2=‖Rm‖2,\\langle R\_\{m\},d\\rangle\\leq\\\|R\_\{m\}\\\|\_\{2\}\\\|d\\\|\_\{2\}=\\\|R\_\{m\}\\\|\_\{2\},with equality if and only ifd=Rm/‖Rm‖2=dmd=R\_\{m\}/\\\|R\_\{m\}\\\|\_\{2\}=d\_\{m\}\. Hencedmd\_\{m\}is the steepest first\-order correction direction in the pre\-LM\-head hidden space\. ∎

###### Proposition A\.2\(Uncertainty direction as a local geometric proxy\)\.

Fix the restricted vocabulary set𝒱q\\mathcal\{V\}\_\{q\}at query statehqh\_\{q\}\. Let

p^q​\(v\)=pq​\(v\)∑v′∈𝒱qpq​\(v′\),v∈𝒱q,\\hat\{p\}\_\{q\}\(v\)=\\frac\{p\_\{q\}\(v\)\}\{\\sum\_\{v^\{\\prime\}\\in\\mathcal\{V\}\_\{q\}\}p\_\{q\}\(v^\{\\prime\}\)\},\\qquad v\\in\\mathcal\{V\}\_\{q\},and

H^q=−∑v∈𝒱qp^q​\(v\)​log⁡p^q​\(v\)\.\\hat\{H\}\_\{q\}=\-\\sum\_\{v\\in\\mathcal\{V\}\_\{q\}\}\\hat\{p\}\_\{q\}\(v\)\\log\\hat\{p\}\_\{q\}\(v\)\.Letz^q=W𝒱q​hq\\hat\{z\}\_\{q\}=W\_\{\\mathcal\{V\}\_\{q\}\}h\_\{q\}be the restricted logits\. Then

gq=∂H^q∂z^q=−p^q⊙\(log⁡p^q\+H^q​𝟏\),g\_\{q\}=\\frac\{\\partial\\hat\{H\}\_\{q\}\}\{\\partial\\hat\{z\}\_\{q\}\}=\-\\hat\{p\}\_\{q\}\\odot\\left\(\\log\\hat\{p\}\_\{q\}\+\\hat\{H\}\_\{q\}\\mathbf\{1\}\\right\),and

rq=W𝒱q⊤​gq=∇hqH^q\.r\_\{q\}=W\_\{\\mathcal\{V\}\_\{q\}\}^\{\\top\}g\_\{q\}=\\nabla\_\{h\_\{q\}\}\\hat\{H\}\_\{q\}\.Therefore, ifrq≠0r\_\{q\}\\neq 0, the direction

d~q=−rq‖rq‖2\\tilde\{d\}\_\{q\}=\-\\frac\{r\_\{q\}\}\{\\\|r\_\{q\}\\\|\_\{2\}\}is the steepest first\-order direction for decreasing the restricted entropy\.

###### Proof\.

For restricted softmax probabilitiesp^\\hat\{p\}, the entropy is

H^=−∑vp^v​log⁡p^v\.\\hat\{H\}=\-\\sum\_\{v\}\\hat\{p\}\_\{v\}\\log\\hat\{p\}\_\{v\}\.Using the softmax Jacobian

∂p^i∂z^j=p^i​\(𝟏​\[i=j\]−p^j\),\\frac\{\\partial\\hat\{p\}\_\{i\}\}\{\\partial\\hat\{z\}\_\{j\}\}=\\hat\{p\}\_\{i\}\(\\mathbf\{1\}\[i=j\]\-\\hat\{p\}\_\{j\}\),we have

∂H^∂z^j=∑i−\(log⁡p^i\+1\)​p^i​\(𝟏​\[i=j\]−p^j\)\.\\frac\{\\partial\\hat\{H\}\}\{\\partial\\hat\{z\}\_\{j\}\}=\\sum\_\{i\}\-\(\\log\\hat\{p\}\_\{i\}\+1\)\\hat\{p\}\_\{i\}\(\\mathbf\{1\}\[i=j\]\-\\hat\{p\}\_\{j\}\)\.Rearranging gives

∂H^∂z^j=−p^j​\(log⁡p^j\+H^\)\.\\frac\{\\partial\\hat\{H\}\}\{\\partial\\hat\{z\}\_\{j\}\}=\-\\hat\{p\}\_\{j\}\(\\log\\hat\{p\}\_\{j\}\+\\hat\{H\}\)\.Thus

gq=−p^q⊙\(log⁡p^q\+H^q​𝟏\)\.g\_\{q\}=\-\\hat\{p\}\_\{q\}\\odot\(\\log\\hat\{p\}\_\{q\}\+\\hat\{H\}\_\{q\}\\mathbf\{1\}\)\.Sincez^q=W𝒱q​hq\\hat\{z\}\_\{q\}=W\_\{\\mathcal\{V\}\_\{q\}\}h\_\{q\}, the chain rule gives

∇hqH^q=W𝒱q⊤​gq=rq\.\\nabla\_\{h\_\{q\}\}\\hat\{H\}\_\{q\}=W\_\{\\mathcal\{V\}\_\{q\}\}^\{\\top\}g\_\{q\}=r\_\{q\}\.For any unit directiondd,

H^q​\(hq\+η​d\)=H^q​\(hq\)\+η​⟨rq,d⟩\+o​\(η\)\.\\hat\{H\}\_\{q\}\(h\_\{q\}\+\\eta d\)=\\hat\{H\}\_\{q\}\(h\_\{q\}\)\+\\eta\\langle r\_\{q\},d\\rangle\+o\(\\eta\)\.The first\-order entropy decrease is maximized by minimizing⟨rq,d⟩\\langle r\_\{q\},d\\rangle, whose unit\-norm solution isd=−rq/‖rq‖2d=\-r\_\{q\}/\\\|r\_\{q\}\\\|\_\{2\}\. ∎

Moreover,

U​\(q,j\)=max⁡\(0,−gq⊤​ϵ​W𝒱q​dj\)=ϵ​max⁡\(0,−⟨rq,dj⟩\)\.U\(q,j\)=\\max\\left\(0,\-g\_\{q\}^\{\\top\}\\epsilon W\_\{\\mathcal\{V\}\_\{q\}\}d\_\{j\}\\right\)=\\epsilon\\max\\left\(0,\-\\langle r\_\{q\},d\_\{j\}\\rangle\\right\)\.ThereforeU​\(q,j\)U\(q,j\)is exactly the positive part of the first\-order restricted\-entropy decrease induced by moving along historical directiondjd\_\{j\}\. This proxy does not assume that the entropy\-decreasing direction equals the ground\-truth correction direction\. Rather, it exposes the local geometry of the current predictive distribution and is used as a label\-free auxiliary retrieval cue for comparing the query state with stored historical directions\.

###### Proposition A\.3\(Local least\-squares derivation of the hidden adapter\)\.

Given selected memory directions\{dj\}j∈Sq\\\{d\_\{j\}\\\}\_\{j\\in S\_\{q\}\}and weightswj≥0w\_\{j\}\\geq 0,∑jwj=1\\sum\_\{j\}w\_\{j\}=1, consider the problem of representing them by a single unit correction direction:

min‖d‖2=1​∑j∈Sqwj​‖d−dj‖22\.\\min\_\{\\\|d\\\|\_\{2\}=1\}\\sum\_\{j\\in S\_\{q\}\}w\_\{j\}\\\|d\-d\_\{j\}\\\|\_\{2\}^\{2\}\.If∑jwj​dj≠0\\sum\_\{j\}w\_\{j\}d\_\{j\}\\neq 0, the solution is

d¯q=∑j∈Sqwj​dj‖∑j∈Sqwj​dj‖2\.\\bar\{d\}\_\{q\}=\\frac\{\\sum\_\{j\\in S\_\{q\}\}w\_\{j\}d\_\{j\}\}\{\\left\\\|\\sum\_\{j\\in S\_\{q\}\}w\_\{j\}d\_\{j\}\\right\\\|\_\{2\}\}\.Furthermore, with

h^q=Norm​\(hq\),h^t=Norm​\(htout\),\\hat\{h\}\_\{q\}=\\mathrm\{Norm\}\(h\_\{q\}\),\\qquad\\hat\{h\}\_\{t\}=\\mathrm\{Norm\}\(h\_\{t\}^\{\\mathrm\{out\}\}\),the activation gate

αt=max⁡\(0,cos⁡\(h^t,h^q\)\)\\alpha\_\{t\}=\\max\(0,\\cos\(\\hat\{h\}\_\{t\},\\hat\{h\}\_\{q\}\)\)is the nonnegative least\-squares coefficient for projecting the current decoding state onto the query state\.

###### Proof\.

Expanding the direction\-aggregation objective,

∑jwj​‖d−dj‖22=∑jwj​\(‖d‖22\+‖dj‖22−2​⟨d,dj⟩\)\.\\sum\_\{j\}w\_\{j\}\\\|d\-d\_\{j\}\\\|\_\{2\}^\{2\}=\\sum\_\{j\}w\_\{j\}\(\\\|d\\\|\_\{2\}^\{2\}\+\\\|d\_\{j\}\\\|\_\{2\}^\{2\}\-2\\langle d,d\_\{j\}\\rangle\)\.Since‖d‖2=1\\\|d\\\|\_\{2\}=1,‖dj‖2=1\\\|d\_\{j\}\\\|\_\{2\}=1, and∑jwj=1\\sum\_\{j\}w\_\{j\}=1, minimizing the objective is equivalent to maximizing

⟨d,∑jwj​dj⟩\.\\left\\langle d,\\sum\_\{j\}w\_\{j\}d\_\{j\}\\right\\rangle\.By Cauchy–Schwarz, the maximizer is the normalized weighted mean direction\.

For the gate, consider the nonnegative least\-squares projection

αt=arg⁡minα∈\[0,1\]⁡‖h^t−α​h^q‖22\.\\alpha\_\{t\}=\\arg\\min\_\{\\alpha\\in\[0,1\]\}\\\|\\hat\{h\}\_\{t\}\-\\alpha\\hat\{h\}\_\{q\}\\\|\_\{2\}^\{2\}\.Since‖h^q‖2=‖h^t‖2=1\\\|\\hat\{h\}\_\{q\}\\\|\_\{2\}=\\\|\\hat\{h\}\_\{t\}\\\|\_\{2\}=1, the unconstrained minimizer is

αt⋆=⟨h^t,h^q⟩\.\\alpha\_\{t\}^\{\\star\}=\\langle\\hat\{h\}\_\{t\},\\hat\{h\}\_\{q\}\\rangle\.Projecting it to\[0,1\]\[0,1\]gives

αt=max⁡\(0,⟨h^t,h^q⟩\)=max⁡\(0,cos⁡\(h^t,h^q\)\)\.\\alpha\_\{t\}=\\max\(0,\\langle\\hat\{h\}\_\{t\},\\hat\{h\}\_\{q\}\\rangle\)=\\max\(0,\\cos\(\\hat\{h\}\_\{t\},\\hat\{h\}\_\{q\}\)\)\.Thus, the gate is the least\-squares coefficient measuring how much the current decoding state lies in the local query direction\. ∎

Combining the least\-squares prototype estimatord¯q\\bar\{d\}\_\{q\}with the least\-squares local activation coefficientαt\\alpha\_\{t\}, the lowest\-cost rank\-one hidden adapter takes the form

ht′=htout\+η​αt​d¯q\.h\_\{t\}^\{\\prime\}=h\_\{t\}^\{\\mathrm\{out\}\}\+\\eta\\alpha\_\{t\}\\bar\{d\}\_\{q\}\.This update realizes the retrieved historical correction as a single vector addition in hidden space, modulated by local state similarity, and is discarded after generation\. Therefore, the temporary hidden adapter can be interpreted as the local least\-squares projection of multiple retrieved historical correction signals onto a single low\-cost inference\-time correction field\.

###### Theorem A\.4\(Direction coverage bound for old\-stage risk\)\.

Consider an old\-stage examplezzevaluated by the final frozen backbone\. Let

ℒzα​\(η,d\)=1\|𝒴z\|​∑t∈𝒴zℓyt​\(htout\+η​αt​d\),\\mathcal\{L\}\_\{z\}^\{\\alpha\}\(\\eta,d\)=\\frac\{1\}\{\|\\mathcal\{Y\}\_\{z\}\|\}\\sum\_\{t\\in\\mathcal\{Y\}\_\{z\}\}\\ell\_\{y\_\{t\}\}\\left\(h\_\{t\}^\{\\mathrm\{out\}\}\+\\eta\\alpha\_\{t\}d\\right\),whereαt∈\[0,1\]\\alpha\_\{t\}\\in\[0,1\]is the inference\-time activation gate\. Define the gated old\-task correction vector

Gzα=1\|𝒴z\|​∑t∈𝒴zαt​WLM⊤​\(eyt−pt\)\.G\_\{z\}^\{\\alpha\}=\\frac\{1\}\{\|\\mathcal\{Y\}\_\{z\}\|\}\\sum\_\{t\\in\\mathcal\{Y\}\_\{z\}\}\\alpha\_\{t\}W\_\{\\mathrm\{LM\}\}^\{\\top\}\(e\_\{y\_\{t\}\}\-p\_\{t\}\)\.Letbz=Gzα/‖Gzα‖2b\_\{z\}=G\_\{z\}^\{\\alpha\}/\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}whenGzα≠0G\_\{z\}^\{\\alpha\}\\neq 0\. Suppose the memory retrieval and aggregation procedure returns a unit directiond¯z\\bar\{d\}\_\{z\}\. Define the directional coverage error

εℬ​\(z\)=1−⟨bz,d¯z⟩\.\\varepsilon\_\{\\mathcal\{B\}\}\(z\)=1\-\\langle b\_\{z\},\\bar\{d\}\_\{z\}\\rangle\.If the token loss isβ\\beta\-smooth in hidden space, then

ℒzα​\(η,d¯z\)≤ℒzα​\(0,d¯z\)−η​‖Gzα‖2​\(1−εℬ​\(z\)\)\+β​η22\.\\mathcal\{L\}\_\{z\}^\{\\alpha\}\(\\eta,\\bar\{d\}\_\{z\}\)\\leq\\mathcal\{L\}\_\{z\}^\{\\alpha\}\(0,\\bar\{d\}\_\{z\}\)\-\\eta\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}\\bigl\(1\-\\varepsilon\_\{\\mathcal\{B\}\}\(z\)\\bigr\)\+\\frac\{\\beta\\eta^\{2\}\}\{2\}\.

###### Proof\.

For fixedzzandd¯z\\bar\{d\}\_\{z\}, define

ϕz​\(η\)=ℒzα​\(η,d¯z\)\.\\phi\_\{z\}\(\\eta\)=\\mathcal\{L\}\_\{z\}^\{\\alpha\}\(\\eta,\\bar\{d\}\_\{z\}\)\.Its derivative atη=0\\eta=0is

ϕz′​\(0\)=1\|𝒴z\|​∑t∈𝒴zαt​⟨∇hℓyt​\(htout\),d¯z⟩\.\\phi\_\{z\}^\{\\prime\}\(0\)=\\frac\{1\}\{\|\\mathcal\{Y\}\_\{z\}\|\}\\sum\_\{t\\in\\mathcal\{Y\}\_\{z\}\}\\alpha\_\{t\}\\left\\langle\\nabla\_\{h\}\\ell\_\{y\_\{t\}\}\(h\_\{t\}^\{\\mathrm\{out\}\}\),\\bar\{d\}\_\{z\}\\right\\rangle\.Using

∇hℓyt​\(htout\)=WLM⊤​\(pt−eyt\),\\nabla\_\{h\}\\ell\_\{y\_\{t\}\}\(h\_\{t\}^\{\\mathrm\{out\}\}\)=W\_\{\\mathrm\{LM\}\}^\{\\top\}\(p\_\{t\}\-e\_\{y\_\{t\}\}\),we get

ϕz′​\(0\)=−⟨Gzα,d¯z⟩=−‖Gzα‖2​⟨bz,d¯z⟩\.\\phi\_\{z\}^\{\\prime\}\(0\)=\-\\left\\langle G\_\{z\}^\{\\alpha\},\\bar\{d\}\_\{z\}\\right\\rangle=\-\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}\\langle b\_\{z\},\\bar\{d\}\_\{z\}\\rangle\.Since⟨bz,d¯z⟩=1−εℬ​\(z\)\\langle b\_\{z\},\\bar\{d\}\_\{z\}\\rangle=1\-\\varepsilon\_\{\\mathcal\{B\}\}\(z\), and byβ\\beta\-smoothness,

ϕz​\(η\)≤ϕz​\(0\)\+η​ϕz′​\(0\)\+β​η22,\\phi\_\{z\}\(\\eta\)\\leq\\phi\_\{z\}\(0\)\+\\eta\\phi\_\{z\}^\{\\prime\}\(0\)\+\\frac\{\\beta\\eta^\{2\}\}\{2\},which gives the desired result\. ∎

Let𝒫old\\mathcal\{P\}\_\{\\mathrm\{old\}\}be the distribution over previously seen examples\. Denote the frozen final old\-task risk by

ℛold0=𝔼z∼𝒫old​\[ℒzα​\(0,d¯z\)\],\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{0\}=\\mathbb\{E\}\_\{z\\sim\\mathcal\{P\}\_\{\\mathrm\{old\}\}\}\[\\mathcal\{L\}\_\{z\}^\{\\alpha\}\(0,\\bar\{d\}\_\{z\}\)\],and the adapted risk by

ℛoldℬ​\(η\)=𝔼z∼𝒫old​\[ℒzα​\(η,d¯z\)\]\.\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{\\mathcal\{B\}\}\(\\eta\)=\\mathbb\{E\}\_\{z\\sim\\mathcal\{P\}\_\{\\mathrm\{old\}\}\}\[\\mathcal\{L\}\_\{z\}^\{\\alpha\}\(\\eta,\\bar\{d\}\_\{z\}\)\]\.Then

ℛoldℬ​\(η\)≤ℛold0−η​𝔼z​\[‖Gzα‖2​⟨bz,d¯z⟩\]\+β​η22\.\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{\\mathcal\{B\}\}\(\\eta\)\\leq\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{0\}\-\\eta\\mathbb\{E\}\_\{z\}\\left\[\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}\\langle b\_\{z\},\\bar\{d\}\_\{z\}\\rangle\\right\]\+\\frac\{\\beta\\eta^\{2\}\}\{2\}\.Thus, the benefit of memory is controlled by the directional coverage term

Cov​\(ℬ\)=𝔼z​\[‖Gzα‖2​⟨bz,d¯z⟩\]\.\\mathrm\{Cov\}\(\\mathcal\{B\}\)=\\mathbb\{E\}\_\{z\}\\left\[\\\|G\_\{z\}^\{\\alpha\}\\\|\_\{2\}\\langle b\_\{z\},\\bar\{d\}\_\{z\}\\rangle\\right\]\.If memory preserves old\-stage directions with positive alignment, then inference\-time adaptation reduces old\-stage risk to first order\. Relative to any reference old\-stage riskℛoldref\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{\\mathrm\{ref\}\}, the forgetting gap satisfies

ℛoldℬ​\(η\)−ℛoldref≤ℛold0−ℛoldref−η​Cov​\(ℬ\)\+β​η22\.\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{\\mathcal\{B\}\}\(\\eta\)\-\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{\\mathrm\{ref\}\}\\leq\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{0\}\-\\mathcal\{R\}\_\{\\mathrm\{old\}\}^\{\\mathrm\{ref\}\}\-\\eta\\mathrm\{Cov\}\(\\mathcal\{B\}\)\+\\frac\{\\beta\\eta^\{2\}\}\{2\}\.

## Appendix BExtended Methodology

### B\.1Inference\-Time Efficiency

At inference time, MIITA adds only retrieval, reranking, and temporary hidden\-state adaptation on top of standard generation\. Exact semantic and directional recall costO​\(A​H\)O\(AH\)andO​\(J​H\)O\(JH\), respectively, whereA=∑j\|𝒜j\|A=\\sum\_\{j\}\|\\mathcal\{A\}\_\{j\}\|is the number of anchors andJJis the number of prototypes\. Both are bounded by the fixed memory budget and can be further accelerated with approximate indexing\. The uncertainty proxy and reranking costO​\(Kv​H\)O\(K\_\{v\}H\)andO​\(\|𝒞q\|​Kv​H\)O\(\|\\mathcal\{C\}\_\{q\}\|K\_\{v\}H\), respectively, whereKvK\_\{v\}is the restricted vocabulary size and𝒞q\\mathcal\{C\}\_\{q\}is the candidate set\. With fixed retrieval hyperparameters and bounded memory, the additional query\-level overhead is linear in the hidden dimensionHH\. During generation, the adapter introduces only a cosine gate and a vector addition per token, which also costO​\(H\)O\(H\)\. Therefore, MIITA avoids prompt extension, replay, parameter updates, and test\-time backpropagation, making it compatible with the low\-storage and low\-latency constraints of SLM deployment\.

### B\.2Detailed Implementation of MIITA

Algorithm[1](https://arxiv.org/html/2607.22556#alg1)summarizes the full implementation of MIITA, including memory construction, budget\-aware memory repair, inference\-time retrieval, and temporary hidden\-state adaptation\.

Algorithm 1MIITA: Memory\-Induced Inference\-Time Adaptation Framework for CL1:Stream

\{𝒟t\}t=1T\\\{\\mathcal\{D\}\_\{t\}\\\}\_\{t=1\}^\{T\}, base SLM

ff, budget

CmemC\_\{\\mathrm\{mem\}\}, hyperparameters

\{τd,τk,Rk,Rd,M,λk,λu,Tagg,η\}\\\{\\tau\_\{d\},\\tau\_\{k\},R\_\{k\},R\_\{d\},M,\\lambda\_\{k\},\\lambda\_\{u\},T\_\{\\mathrm\{agg\}\},\\eta\\\}
2:Memory bank

ℬ\\mathcal\{B\}, prediction

y^\\hat\{y\}
3:

ℬ←∅\\mathcal\{B\}\\leftarrow\\varnothing
4:

5:// Phase 1: Build budgeted direction memory

6:for

t=1t=1to

TTdo

7:foreach

\(xm,ym\)∈𝒟t\(x\_\{m\},y\_\{m\}\)\\in\\mathcal\{D\}\_\{t\}do

8:

km←k\_\{m\}\\leftarrowEq\. \([1](https://arxiv.org/html/2607.22556#S3.E1)\),

dm←d\_\{m\}\\leftarrowEq\. \([2](https://arxiv.org/html/2607.22556#S3.E2)\)

9:if

ℬ=∅\\mathcal\{B\}=\\varnothingthen

10:Add

u=\(dm,\{km\},1\)u=\(d\_\{m\},\\\{k\_\{m\}\\\},1\)to

ℬ\\mathcal\{B\}
11:else

12:

j⋆←arg⁡maxj⁡cos⁡\(dm,dj\)j^\{\\star\}\\leftarrow\\arg\\max\_\{j\}\\cos\(d\_\{m\},d\_\{j\}\)
13:if

cos⁡\(dm,dj⋆\)<τd\\cos\(d\_\{m\},d\_\{j^\{\\star\}\}\)<\\tau\_\{d\}then

14:Add

u=\(dm,\{km\},1\)u=\(d\_\{m\},\\\{k\_\{m\}\\\},1\)to

ℬ\\mathcal\{B\}
15:else

16:Update

uj⋆u\_\{j^\{\\star\}\}by Eqs\. \([3](https://arxiv.org/html/2607.22556#S3.E3)\) and \([4](https://arxiv.org/html/2607.22556#S3.E4)\)

17:endif

18:endif

19:while

Size​\(ℬ\)\>Cmem\\mathrm\{Size\}\(\\mathcal\{B\}\)\>C\_\{\\mathrm\{mem\}\}do

20:Repair

ℬ\\mathcal\{B\}by anchor pruning, prototype merging, or eviction via Eq\. \([5](https://arxiv.org/html/2607.22556#S3.E5)\)

21:endwhile

22:endfor

23:endfor

24:

25:// Phase 2: Retrieve and aggregate memory directions

26:Given query

qq, compute

kqk\_\{q\}by Eq\. \([1](https://arxiv.org/html/2607.22556#S3.E1)\) and

d~q\\tilde\{d\}\_\{q\}by Eq\. \([7](https://arxiv.org/html/2607.22556#S3.E7)\)

27:Construct

𝒞q\\mathcal\{C\}\_\{q\}by Eq\. \([8](https://arxiv.org/html/2607.22556#S3.E8)\)

28:foreach

j∈𝒞qj\\in\\mathcal\{C\}\_\{q\}do

29:Compute

score​\(q,j\)\\mathrm\{score\}\(q,j\)by Eq\. \([9](https://arxiv.org/html/2607.22556#S3.E9)\)

30:endfor

31:

Sq←TopM⁡\{score​\(q,j\):j∈𝒞q\}S\_\{q\}\\leftarrow\\operatorname\{Top\}\_\{M\}\\\{\\mathrm\{score\}\(q,j\):j\\in\\mathcal\{C\}\_\{q\}\\\}
32:Aggregate

d¯q\\bar\{d\}\_\{q\}by Eq\. \([10](https://arxiv.org/html/2607.22556#S3.E10)\)

33:

34:// Phase 3: Apply temporary hidden adaptation

35:foreach decoding step

ssdo

36:Apply gated update

hs′h\_\{s\}^\{\\prime\}by Eq\. \([11](https://arxiv.org/html/2607.22556#S3.E11)\)

37:Generate next token from

hs′h\_\{s\}^\{\\prime\}
38:endfor

39:Discard the temporary update

40:return

y^\\hat\{y\}

## Appendix CExperimental Details

### C\.1Baseline Details

General protocol\.All baselines are evaluated under the same budget\-constrained supervised SLM CL setting\. Unless otherwise specified, each benchmark is converted into the unified instruction\-style format\(x,y\)\(x,y\), and all methods use the same SLM backbone\. For methods that store auxiliary information, including replay samples, summaries, statistics, episodic memories, adapters, or bottleneck parameters, we constrain the total persistent storage to the same budgetCbytesC\_\{\\mathrm\{bytes\}\}\. Methods designed only for classification are evaluated on the classification benchmarks and are marked as not applicable to open\-ended QA\.

Baseline descriptions and adaptations\.

- •LoRA\[[9](https://arxiv.org/html/2607.22556#bib.bib17)\]\. LoRA is a parameter\-efficient sequential tuning baseline that updates low\-rank adapters while keeping the SLM backbone frozen\. We train one LoRA module sequentially on each incoming stage using only the current data𝒟t\\mathcal\{D\}\_\{t\}, without replay or external memory\. The LoRA parameters are carried across stages as the model’s continual update state\.
- •MBPA\+\+\[[4](https://arxiv.org/html/2607.22556#bib.bib16)\]\. MBPA\+\+ is an episodic\-memory language\-model CL method that stores past examples and uses sparse replay together with inference\-time local adaptation\. We replace its original language model with the same SLM backbone used by our method, store serialized\(x,y\)\(x,y\)examples in episodic memory underCbytesC\_\{\\mathrm\{bytes\}\}, retrieve nearest neighbors using frozen SLM representations, and perform temporary local LoRA adaptation for each query before discarding the update\.
- •CT0\[[19](https://arxiv.org/html/2607.22556#bib.bib4)\]\. CT0 is a training\-time rehearsal baseline that continually learns new instruction\-following tasks by replaying a small buffer of previous examples\. We adapt it by maintaining a budgeted replay buffer of old supervised examples and mixing sampled replay data with the current stage data during LoRA\-based continual tuning\.
- •FOREVER\[[6](https://arxiv.org/html/2607.22556#bib.bib5)\]\. FOREVER is a replay\-based LLM CL method that schedules memory replay according to model\-centric update dynamics\. We use the same budgeted replay buffer as CT0, but trigger replay according to FOREVER’s model\-time schedule computed from accumulated LoRA update norms, and apply its replay\-time regularization during continual tuning\.
- •CIS\[[18](https://arxiv.org/html/2607.22556#bib.bib6)\]\. CIS is the concrete method under the CLOB paradigm, which performs black\-box CL by storing class knowledge as incrementally updated textual summaries\. For classification benchmarks, we follow its original design by generating and updating class\-level summaries underCbytesC\_\{\\mathrm\{bytes\}\}and using them in inference\-time prompts\. For QA benchmarks, where the output space is open\-ended rather than class\-based, we adapt CIS by maintaining stage\-level or domain\-level summaries of previously seen QA examples\. At inference time, the query is paired with the relevant stored summaries in the prompt, and the SLM generates the answer directly\.
- •InCA\[[17](https://arxiv.org/html/2607.22556#bib.bib7)\]\. InCA is an inference\-time ICL method that uses an external continual learner to select candidate classes before prompting the LLM\. For classification benchmarks, we keep this mechanism and incrementally store class\-level tag statistics and summaries underCbytesC\_\{\\mathrm\{bytes\}\}\. For QA benchmarks, we adapt the external learner from class selection to memory selection: the learner ranks stored stage/domain summaries or representative QA memories according to the semantic tags of the query, and the top\-ranked memories are used to construct the inference prompt\. The final prediction is generated by the SLM rather than selected from a finite class set\.
- •DKVB\[[5](https://arxiv.org/html/2607.22556#bib.bib1)\]\. DKVB is an SLM\-oriented CL method that uses a discrete key\-value bottleneck to enable localized continual updates\. Since the original method is designed for encoder\-only classification, we adapt it by inserting the bottleneck at the SLM hidden representation before the prediction head\. For instruction\-style tasks, labels are verbalized as output tokens, and the bottleneck capacity is selected to fit the same storage budget\.
- •Srinath Ket al\.\[[21](https://arxiv.org/html/2607.22556#bib.bib3)\]\. Srinath Ket al\.propose an SLM\-oriented multilingual CL method for mitigating cross\-lingual forgetting with replay adapters\. For multilingual QA, we treat each language as a domain and use its replay\-adapter strategy in the original language\-incremental form\. For non\-multilingual benchmarks, we adapt the method by retaining the shared replay\-adapter mechanism but replacing POS\-guided code\-switching with task\- or domain\-preserving replay examples from the same benchmark\. Thus, the baseline becomes a replay\-adapter CL method that uses the same instruction\-style input\-output format while removing the language\-specific code\-switching component when it is not applicable\.

### C\.2Details of Memory Item Variants

We evaluate four memory item formats under the same retrieval backbone, retrieval budget, and total storage constraint\.

- •ExampleMem\.This variant stores serialized supervised examples\(x,y\)\(x,y\)as memory items\. At inference time, retrieved examples are inserted into the prompt as demonstrations or contextual evidence\. The storage cost is measured by the byte size of the stored text\.
- •SummaryMem\.This variant compresses historical examples into natural\-language summaries\. Retrieved summaries are inserted into the prompt before generation\. The storage cost includes the byte size of the stored summaries\.
- •SemanticMem\.This variant stores semantic embedding prototypes obtained from the same retrieval backbone\. At inference time, retrieved semantic prototypes are used through the same retrieval interface, but they do not contain supervised correction information\. The storage cost includes the stored embedding vectors and associated prototype metadata\.
- •DirectionMem\.This is the memory format used by MIITA\. Each supervised example induces a correction direction in hidden space, and similar directions are merged into prototypes under the memory budget\. At inference time, retrieved direction prototypes are used for temporary hidden\-state adaptation without gradient\-based test\-time training\.

### C\.3Details of Retrieval Variants

We evaluate retrieval variants under the same DirectionMem memory bank, retrieval backbone, retrieval budget, and total storage constraint\. All variants use the same supervised stream and Qwen3\-0\.6B backbone\. They differ only in how memory prototypes are selected or reranked at inference time\.

- •RandomRet\.This variant uniformly samples memory prototypes from the DirectionMem bank\. It ignores both semantic relevance and uncertainty information and serves as a lower\-bound retrieval baseline\.
- •SemanticRet\.This variant retrieves memory prototypes using only semantic\-key similaritySk​\(q,j\)S\_\{k\}\(q,j\)between the query and stored memory anchors\. It tests whether semantic relevance alone is sufficient for selecting useful historical correction directions\.
- •UncertaintyRet\.This variant retrieves memory prototypes using only the uncertainty\-induced query directiond~q\\tilde\{d\}\_\{q\}\. Sinced~q\\tilde\{d\}\_\{q\}and the stored correction directionsdjd\_\{j\}are both defined in the LM\-head input hidden space, they can be compared by directional alignment\. This variant tests whether uncertainty provides a label\-free query\-side signal for selecting historical directions that may locally affect the current prediction distribution\.
- •SemUncRet\.This is the retrieval strategy used by MIITA\. It first applies semantic recall to construct a query\-relevant candidate memory set and then applies uncertainty\-guided soft reranking within this set\. Semantic recall keeps retrieved memories related to the query, while uncertainty prioritizes directions that are more likely to influence the current predictive state\.
- •OracleRet\.This variant uses the ground\-truth target to compute an oracle correction direction and retrieves memory prototypes according to their alignment with this direction\. For a single target token, the oracle direction is computed asdq⋆=WLM⊤​\(ey−p\)d\_\{q\}^\{\\star\}=W\_\{\\mathrm\{LM\}\}^\{\\top\}\(e\_\{y\}\-p\), whereppis the predicted distribution andeye\_\{y\}is the one\-hot target distribution\. For generation tasks, token\-level oracle directions are averaged over the target sequence\. OracleRet is used only as an upper bound and is not available at inference time\.

##### Oracle Alignment\.

For retrieval analysis, we computeOracle Alignment \(OA\)on a held\-out evaluation set where labels are available\. For each queryqq, we first compute the oracle correction directiondq⋆d\_\{q\}^\{\\star\}from the ground\-truth target\. Each retrieval variant then produces an aggregated retrieved directiond¯q\\bar\{d\}\_\{q\}using the same direction aggregation rule as MIITA\. OA is defined as the average cosine similarity betweend¯q\\bar\{d\}\_\{q\}anddq⋆d\_\{q\}^\{\\star\}:

OA=1\|𝒬\|​∑q∈𝒬cos⁡\(d¯q,dq⋆\)\.\\mathrm\{OA\}=\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\}\\cos\(\\bar\{d\}\_\{q\},d\_\{q\}^\{\\star\}\)\.A higher OA means that the retrieved direction has stronger alignment with the label\-derived oracle correction direction\. OA should not be interpreted as accuracy or as the fraction of queries whose predictions become correct\. It measures directional agreement in hidden space\. In high\-dimensional spaces, random directions have cosine similarity close to zero, so an OA around0\.50\.5already indicates a strong positive projection onto the oracle correction direction\. A retrieved direction does not need to be perfectly collinear with the oracle direction to be useful: as long as it has a positive projection on the oracle direction, a small hidden\-state update can still increase the probability of target tokens or reduce the supervised loss\. Conversely, OA close to11would mean near\-perfect recovery of the label\-derived correction direction, which is unrealistic for a label\-free retrieval signal in generation tasks\. Even OracleRet does not obtain OA equal to11because it still retrieves and aggregates finite historical memory prototypes rather than directly applying the oracle direction itself\. Thus, OA is used only as a diagnostic measure of retrieval\-direction quality, while OP and BWT measure the final continual learning performance\.

## Appendix DSupplementary Experiments

### D\.1Effect of the Activation Gate

We additionally evaluate the activation gateαt\\alpha\_\{t\}used in the temporary hidden adaptation step\. The experiment is conducted onSuperNIwith Qwen3\-0\.6B under the fixed 1% memory budget\. The full MIITA uses the gated updateht′=htout\+η​αt​d¯qh\_\{t\}^\{\\prime\}=h\_\{t\}^\{\\mathrm\{out\}\}\+\\eta\\alpha\_\{t\}\\bar\{d\}\_\{q\}, while the ablated variant removes the gate by settingαt=1\\alpha\_\{t\}=1for all decoding steps, so that the same correction direction is uniformly applied throughout generation\.

Removing the gate reduces performance from40\.5840\.58OP /−4\.89\-4\.89BWT to38\.7238\.72OP /−6\.18\-6\.18BWT\. This confirms that the gate is not a redundant component: uniformly injecting the retrieved direction can over\-correct decoding states that are no longer close to the query state that triggered retrieval\. In contrast, the full gated design applies stronger correction only when the current decoding state remains locally aligned with the query state\. This is consistent with Proposition[A\.3](https://arxiv.org/html/2607.22556#A1.Thmtheorem3), which interpretsαt\\alpha\_\{t\}as a nonnegative least\-squares projection coefficient measuring how much the current decoding state lies in the local query direction\.

### D\.2Hyperparameter Sensitivity

We further study the sensitivity of MIITA to the two memory\-construction thresholds,τd\\tau\_\{d\}andτk\\tau\_\{k\}, onSuperNIwith Qwen3\-0\.6B under the fixed 1% memory budget\. When sweeping one hyperparameter, all others are fixed to the default configuration\. The default setting used in the main experiments isτd=0\.85\\tau\_\{d\}=0\.85andτk=0\.80\\tau\_\{k\}=0\.80\.

Table 3:Hyperparameter sensitivity onSuperNIwith Qwen3\-0\.6B under a fixed 1% memory budget\. When sweeping one threshold, all other hyperparameters are fixed to the default configuration\.Boldindicates the default setting used in the main experiments\.HyperparameterValueOP↑\\uparrowBWT↑\\uparrowτd\\tau\_\{d\}0\.7539\.82\-5\.310\.8040\.21\-5\.080\.8540\.58\-4\.890\.9040\.34\-5\.020\.9539\.76\-5\.36τk\\tau\_\{k\}0\.7040\.05\-5\.170\.7540\.31\-5\.020\.8040\.58\-4\.890\.8540\.39\-4\.970\.9040\.12\-5\.15Table[3](https://arxiv.org/html/2607.22556#A4.T3)shows that MIITA is robust across a reasonable range of threshold choices\. Forτd\\tau\_\{d\}, overly small values merge directions too aggressively and may blur distinct functional corrections, while overly large values create more fragmented prototypes and reduce the benefit of direction sharing\. The default valueτd=0\.85\\tau\_\{d\}=0\.85provides the best balance between compression and functional diversity\. Forτk\\tau\_\{k\}, lower values keep too few semantic anchors and may weaken retrieval access, while higher values preserve more anchors but consume budget with redundant semantic entry points\. The default valueτk=0\.80\\tau\_\{k\}=0\.80achieves the best overall trade\-off\. Across all tested values, performance remains close to the default setting, indicating that the gains of MIITA are not due to a fragile hyperparameter choice\.

## Appendix ELimitations

MIITA is designed for supervised continual learning with small language models under a fixed persistent\-memory budget, and its conclusions should be interpreted within this setting\. The method assumes that each incoming stage provides input\-output supervision from which correction directions can be extracted, and it does not directly cover unsupervised, reinforcement\-learning, multimodal, or tool\-use continual adaptation\. Because the backbone remains frozen and adaptation is applied only as a temporary hidden\-state perturbation, MIITA cannot recover capabilities that are absent from the base model or fully replace parameter updates when new tasks require substantial new reasoning skills or knowledge that is poorly represented in the existing hidden space\.

The effectiveness of MIITA also depends on the quality and coverage of the stored direction prototypes\. If historical examples induce highly diverse or conflicting correction directions under a very small memory budget, memory repair may merge or evict directions that are useful for rare tasks\. Similarly, retrieval relies on semantic anchors and an uncertainty\-based label\-free proxy, so out\-of\-domain queries, ambiguous prompts, poorly calibrated predictive uncertainty, or weak semantic representations may retrieve suboptimal directions and reduce the benefit of inference\-time adaptation\. The theoretical analysis is local: it relies on differentiability, first\-order approximations around the current hidden state, fixed LM\-head geometry, and smoothness assumptions, and therefore does not guarantee global performance improvements for large adaptation scales or across arbitrary distribution shifts\.

Although MIITA avoids prompt extension, test\-time backpropagation, and persistent backbone updates, it still introduces additional inference\-time computation for memory retrieval, reranking, and per\-token hidden\-state gating\. Exact retrieval scales with the number of stored anchors and prototypes, so very long streams or larger memory budgets may require approximate indexing or further compression to maintain low latency\. Our experiments cover multiple supervised CL benchmarks, task types, model families, model scales, and memory budgets, but they do not exhaustively evaluate very long\-horizon streams, safety\-critical deployments, non\-English domains beyond the tested multilingual QA setting, or all possible SLM architectures\. Finally, storing compact correction directions reduces the amount of raw text retained but should not be interpreted as a formal privacy guarantee; if the training stream contains sensitive or biased supervision, additional auditing, privacy protection, and fairness evaluation would be needed before deployment\.