Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory

# Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory
Source: [https://arxiv.org/html/2303.01421](https://arxiv.org/html/2303.01421)
Guangyue Peng♡\\heartsuit, Tao Ge♠, Wen Luo♡\\heartsuit, Wei Li♡\\heartsuit, Houfeng Wang♡\\heartsuit† ♡\\heartsuitState Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University ♠Microsoft \{agy,wanghf\}@pku\.edu\.cn,taoge@microsoft\.com llvvvv22222@gmail\.com,weili22@stu\.pku\.edu\.cn

###### Abstract

Semiparametric language models \(LMs\) have shown promise in various Natural Language Processing \(NLP\) tasks\. However, they utilize non\-parametric memory as static storage, which lacks learning capability and remains disconnected from the internal information flow of the parametric models, limiting scalability and efficiency\. Based on recent interpretability theories of LMs, we reconceptualize the non\-parametric memory represented bykkNN\-LM as a learnable Mixture\-of\-Neighbors Induction Memory \(MoNIM\), which synergizes the induction capabilities of attention heads with the memorization strength of feed\-forward networks \(FFN\)\. By integrating into the model’s information flow, MoNIM functions as an FFN\-like bypass layer within the Transformer architecture, enabling effective learning of new knowledge\. Extensive experiments demonstrate that MoNIM is a retentive and scalable continual learner in both data\- and model\-wise, enhancing the scalability and continual learning performance of semiparametric LMs\.111Code is publicly available at[https://github\.com/viniferagy/MoNIM](https://github.com/viniferagy/MoNIM)\.

Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture\-of\-Neighbors Induction Memory

Guangyue Peng♡\\heartsuit††thanks:This work was done during the author’s internship at Microsoft Research Asia\., Tao Ge♠††thanks:Corresponding author, Wen Luo♡\\heartsuit, Wei Li♡\\heartsuit, Houfeng Wang♡\\heartsuit†♡\\heartsuitState Key Laboratory for Multimedia Information Processing,School of Computer Science, Peking University♠Microsoft\{agy,wanghf\}@pku\.edu\.cn,taoge@microsoft\.comllvvvv22222@gmail\.com,weili22@stu\.pku\.edu\.cn

## 1Introduction

Semiparametric language models \(LMs\) have drawn increasing attention\(Guuet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib38); Yogatamaet al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib108); Ramet al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib3)\)for their photographic memorization capabilities and in\-domain accuracy\. Combining a parameterized neural model with an extensible non\-parametric memory, they are skillful at aiding prediction with memorization\.

However, current semiparametric LMs are unsuitable for our fast\-changing world because of inefficient memory management strategies\(Heet al\.,[2021a](https://arxiv.org/html/2303.01421#bib.bib41)\)\. They typically record all training data in the static memory, and the search for useful information then relies on additional modules or tunable hyperparameters\. A lack of learning ability hinders memory compression, a process crucial for efficient learning in LMs\(Delétanget al\.,[2024](https://arxiv.org/html/2303.01421#bib.bib25)\), and separates the memory component from the information flow of the model\. As a result, these models experience linear growth of memory usage and search time as data or model dimensions increase\. This inefficiency becomes especially impractical for large language models \(LLMs\), which deal with huge volumes of training data\.

In this paper, we deal with two questions about semiparametric LMs:Why they are powerful but inefficientandcan we build an efficient memory strategy? Enlightened by research interpreting the learning abilities of LLMs\(Elhageet al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib27); Gevaet al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib31); Olssonet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib73)\), we propose that the non\-parametric memory, specificallykkNN\-LM\(Khandelwalet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib58)\)inherently possesses abilities akin to the induction heads in Multi\-Headed Self\-Attention \(MHSA\) layers\(Olssonet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib73)\)\. Perfect memorization brings perfect local next\-token induction and good performance, but deficiencies in global visions prevent the memory from efficient reasoning\.

![Refer to caption](https://arxiv.org/html/2303.01421v2/x1.png)

![Refer to caption](https://arxiv.org/html/2303.01421v2/x2.png)

Figure 1:The learning mechanism of MoNIM\.\(a\)The analogy between induction heads \(up\) and MoNIM \(down\)\. The shades of lines indicate allocated attention scores\. While induction heads assimilate related information in context, MoNIM gathers similar samples memorized in learning history\.\(b\)MoNIM as a learnable module\. MoNIM shares the same input, working flow, and output with the final FFN layer\. While FFN utilizes parametric keys𝐧L\\boldsymbol\{\\mathrm\{n\}\}^\{L\}to match the query𝐱~L\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}for the promotion of learned concepts encoded in parametric values𝐨L\\boldsymbol\{\\mathrm\{o\}\}^\{L\}\(green\), MoNIM promotes concept mixtures𝐜\\boldsymbol\{\\mathrm\{c\}\}\(green with stripes\) that are embedded in neighboring memorized values\. The learning process of MoNIM is controlled by a compressorM^Θ​\(𝐱\)\\hat\{M\}\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\)optimized for the model’s loss function\.To promote local advantages while avoiding global weaknesses, we build a learnable Mixture\-of\-Neighbor Induction Memory \(MoNIM\) based on the components of concept promotion in Feed\-Forward Networks \(FFN\)\(Gevaet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib32)\)\. As an FFN\-like bypass layer, MoNIM can select and absorb knowledge with a gradually smaller memory footprint, demonstrating its consistency with gradient descent in parametric models, where the impact of new information is lessened with more data or larger model sizes\(Kaplanet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib2)\)\. Consequently, MoNIM’s memory grows sub\-linearly with the enhancement of the model’s capabilities, resulting in a scalable continual learner free of training\. In continual learning scenarios, MoNIM performs comparably to vanilla models while consuming only half the memory space\.

Our contributions can be summarized as follows:

- •We introduce Mixture\-of\-Neighbor Induction Memory \(MoNIM\), a learnable memory in semiparametric continual learning settings that functions as an FFN\-like bypass layer\.
- •MoNIM achieves its prowess by integrating the inductive capabilities of Multi\-Head Self\-Attention \(MHSA\) with the memorization functions of Feed\-Forward Networks \(FFN\)\.
- •Extensive experiments in language modeling and downstream tasks show that MoNIM effectively compresses seen information and is both data\- and model\-wise scalable, thus suitable for continual learning over streaming data with semiparametric LMs\.

## 2MoNIM: Mixture\-of\-Neighbor Induction Memory

### 2\.1Preliminaries:kkNN\-LM

Formally, we useΘ=\(θ,ℳ\)\\Theta=\(\\theta,\\mathcal\{M\}\)to denote a semiparametric LM, whereθ\\thetastands for the parametric LM andℳ\\mathcal\{M\}for the non\-parametric memory\.

As a representative,kkNN\-LM\(Khandelwalet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib58)\)enhances the prediction ofθ\\thetaby leveraging the information ofkk\-nearest neighbors inℳ\\mathcal\{M\}\. Given a leftward context𝐱=\(x1,…,xt\)\\boldsymbol\{\\mathrm\{x\}\}=\(x\_\{1\},\\dots,x\_\{t\}\),kkNN\-LM uses the hidden states in the final position before an FFN layer𝓁∈\{1,\.\.,L\}\\mathscr\{l\}\\in\\\{1,\.\.,L\\\}as the contextualized representation𝐱~𝓁∈ℝd\\widetilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\\in\\mathbb\{R\}^\{d\}, and computes its next wordyy’s probability as follows:

P​\(y\|𝐱;Θ\)=f​\(P​\(y\|𝐱;θ\)⏟Model,P​\(y\|𝐱~𝓁;ℳ\)⏟Memory,λ\)P\(y\|\\boldsymbol\{\\mathrm\{x\}\};\\Theta\)=f\(\\underbrace\{P\(y\|\\boldsymbol\{\\mathrm\{x\}\};\\theta\)\}\_\{\\@add@raggedright\\text\{\\raggedright\{Model\}\}\},\\underbrace\{P\(y\|\\widetilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\};\\mathcal\{M\}\)\}\_\{\\@add@raggedright\\text\{\\raggedright\{Memory\}\}\},\\lambda\)\(1\)whereffrepresents the interpolation function to weigh the predictions of the model and memory byλ\\lambda\.kkNN\-LMs constructℳ\\mathcal\{M\}by the training set𝒟\\mathcal\{D\}as a key\-value lookup, with an entry for each token in the training setxtx\_\{t\}\(as value\) and the representation of its context𝐱<𝐭~𝓁\\widetilde\{\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\}\}^\{\\mathscr\{l\}\}\(as key\):

ℳ=\{\(𝐱<𝐭~𝓁→xt\)\|x∈𝒟\}\\mathcal\{M\}=\\\{\(\\widetilde\{\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\}\}^\{\\mathscr\{l\}\}\\to x\_\{t\}\)\|\\mathrm\{x\}\\in\\mathcal\{D\}\\\}
During inference, we first use𝐱~𝓁\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}as a query to retrievekknearest neighbors from the memoryℳ\\mathcal\{M\}:

𝒩​\(𝐱~𝓁\)=\{\(𝐤~→y~\)i\|i=1,2,…,k\}⊆ℳ\\mathcal\{N\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\)=\\\{\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\\to\\tilde\{y\}\)\_\{i\}\|i=1,2,\\dots,k\\\}\\subseteq\\mathcal\{M\}Then, we obtain the prediction fromℳ\\mathcal\{M\}by computing the weighted sum of retrieved targets:

P​\(y\|𝐱~𝓁;ℳ\)∝∑𝒩​\(𝐱~𝓁\)𝟙y~=y​exp⁡\(−d​\(𝐤~,𝐱~𝓁\)\)P\(y\|\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\};\\mathcal\{M\}\)\\propto\\sum\_\{\\mathcal\{N\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\)\}\\mathbbm\{1\}\_\{\\tilde\{y\}=y\}\\exp\(\-d\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\},\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\)\)\\vskip\-5\.69046pthered\(\.,\.\)d\(\.,\.\)denotes the semantic similarity\.Khandelwalet al\.\([2021](https://arxiv.org/html/2303.01421#bib.bib59)\)proved that𝓁=L\\mathscr\{l\}=Lproduces the best retrieval quality\.

### 2\.2Induction Memory

Recent interpretability studies\(Olssonet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib73); Wanget al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib99)\)have shown that induction heads, namely attention heads that implement a simple algorithm to complete sequences in the form of \[A\]\[B\] … \[A\] \-\> \[B\], might constitute the fundamental abilities for in\-context learning in LLMs\. Attention heads exhibit two typical properties: prefix matching, to attend to the tokens with similar context; and copying, to increase the logit of the output corresponding to the attended\-to tokens\(Bansalet al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib12)\)\.

[Figure˜1](https://arxiv.org/html/2303.01421#S1.F1)\(a\) demonstrates the reasoning mechanisms between induction heads and thekkNN memory\. It is clear to observe their closeness: as induction heads assimilate related information scattered in the contexts,kkNN memory gathers similar information from𝒩​\(𝐱~\)\\mathcal\{N\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}\)memorized in history\. The memory encodes the prefix into the key vector, uses it to match the query, and copies the memorized value to the position to be predicted as the function of induction heads\. From this perspective,kkNN memory is an induction buffer, considering all training data as neighbors, thus the name Mixture\-of\-Neighbor Induction Memory \(MoNIM\)\.

While induction heads serve as basic components in the early layers, transferring context information in the model to implement more complex global reasoning pathways, MoNIM, as a highly localized source of information, only provides the memorized labels\. The capability to induce the most similar tokens also suggests its inability to perform efficient prediction\. To make the best use of MoNIM’s local precision, we move on to develop a new conceptual architecture of MoNIM\.

### 2\.3MoNIM for Local Concept Mixture Promotion

According to previous work\(Sukhbaataret al\.,[2015](https://arxiv.org/html/2303.01421#bib.bib6); Gevaet al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib31)\), FFN layers function as key\-value memory, and the value vectors can be projected to vocabulary space to represent comprehensible concepts such as food or movie characters\(Gevaet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib32)\)\. An FFN updateℱ​\(𝐱~\)\\mathcal\{F\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}\)thus can be interpreted as successive concept promotions towards the connotation of the target token:

ℱ𝓁​\(𝐱~𝓁\)=∑i=1dℱf​\(𝐱~𝓁⋅𝐧i𝓁\)​𝐨i𝓁=∑i=1dℱmi𝓁​𝐨i𝓁\\mathcal\{F\}^\{\\mathscr\{l\}\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\)=\\sum\_\{i=1\}^\{d\_\{\\mathcal\{F\}\}\}f\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\\cdot\\boldsymbol\{\\mathrm\{n\}\}^\{\\mathscr\{l\}\}\_\{i\}\)\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\_\{i\}=\\sum\_\{i=1\}^\{d\_\{\\mathcal\{F\}\}\}m^\{\\mathscr\{l\}\}\_\{i\}\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\_\{i\}
𝐧i𝓁,𝐨i𝓁\\boldsymbol\{\\mathrm\{n\}\}^\{\\mathscr\{l\}\}\_\{i\},\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\_\{i\}is theii\-th column ofWi​nT,Wo​u​t∈ℝdℱ×dW\_\{in\}^\{\\mathrm\{T\}\},W\_\{out\}\\in\\mathbb\{R\}^\{d\_\{\\mathcal\{F\}\}\\times d\}inℱ\\mathcal\{F\},mi𝓁m^\{\\mathscr\{l\}\}\_\{i\}represents the weight of𝐨i𝓁\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\_\{i\}, where the model stores its understanding of concepts\. Similarly, for the hidden state before an MHSA layer𝐱^\\hat\{\\boldsymbol\{\\mathrm\{x\}\}\}, given the attention pattern𝐚𝓁∈ℝT\\boldsymbol\{\\mathrm\{a\}\}^\{\\mathscr\{l\}\}\\in\\mathbb\{R\}^\{T\}for a context of lengthTTand corresponding𝐯j𝓁\\boldsymbol\{\\mathrm\{v\}\}^\{\\mathscr\{l\}\}\_\{j\}, the j\-th column ofWV∈ℝT×d𝒜W\_\{\\mathrm\{V\}\}\\in\\mathbb\{R\}^\{T\\times d\_\{\\mathcal\{A\}\}\}, an MHSA update𝒜​\(𝐱^\)\\mathcal\{A\}\(\\hat\{\\boldsymbol\{\\mathrm\{x\}\}\}\)is the linear combination of vectors of the output matrixWO∈ℝd𝒜×dW\_\{\\mathrm\{O\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathcal\{A\}\}\\times d\}\.

𝒜𝓁​\(𝐱^𝓁\)=∑j=1d𝒜\(𝐚𝓁⋅𝐯j𝓁\)​𝐨j𝓁=∑i=jd𝒜mj𝓁​𝐨j𝓁\\mathcal\{A\}^\{\\mathscr\{l\}\}\(\\hat\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}\)=\\sum\_\{j=1\}^\{d\_\{\\mathcal\{A\}\}\}\(\\boldsymbol\{\\mathrm\{a\}\}^\{\\mathscr\{l\}\}\\cdot\\boldsymbol\{\\mathrm\{v\}\}^\{\\mathscr\{l\}\}\_\{j\}\)\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\_\{j\}=\\sum\_\{i=j\}^\{d\_\{\\mathcal\{A\}\}\}m^\{\\mathscr\{l\}\}\_\{j\}\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\_\{j\}
SinceGevaet al\.\([2022](https://arxiv.org/html/2303.01421#bib.bib32)\)has proved that the layer normalization \(LN\) is almost linear and does not affect the linear properties of MHSA and FFN outputs, we assert that the final prediction before the unembedding matrixE∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}can be decomposed to the reweighted sum of information gained in MHSA and FFN layers, that

𝐱~𝓁=∑i=1𝓁⋅\(dℱ\+d𝒜\)m~i<𝓁​𝐨i<𝓁\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{\\mathscr\{l\}\}=\\sum\_\{i=1\}^\{\\mathscr\{l\}\\cdot\(d\_\{\\mathcal\{F\}\}\+d\_\{\\mathcal\{A\}\}\)\}\\tilde\{m\}^\{<\\mathscr\{l\}\}\_\{i\}\\boldsymbol\{\\mathrm\{o\}\}^\{<\\mathscr\{l\}\}\_\{i\}𝒚~=softmax​\(E​𝐱~L\)\\tilde\{\\boldsymbol\{y\}\}=\\mathrm\{softmax\}\(E\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)
The prediction of𝒚~\\tilde\{\\boldsymbol\{y\}\}is determined by the mixture of concepts𝒐<L\\boldsymbol\{o\}^\{<L\}in FFN\. We define the bestlocal mixture of concepts𝐜\\boldsymbol\{\\mathrm\{c\}\}which outputs the golden prediction𝒚∗\\boldsymbol\{y^\{\*\}\}:

𝐜=∑i=1L⋅\(dℱ\+d𝒜\)m~i∗⁣<L​𝐨i<L\\boldsymbol\{\\mathrm\{c\}\}=\\sum\_\{i=1\}^\{L\\cdot\(d\_\{\\mathcal\{F\}\}\+d\_\{\\mathcal\{A\}\}\)\}\\tilde\{m\}^\{\*<L\}\_\{i\}\\boldsymbol\{\\mathrm\{o\}\}^\{<L\}\_\{i\}𝒚∗=softmax​\(E​𝐜\)\\boldsymbol\{y^\{\*\}\}=\\mathrm\{softmax\}\(E\\boldsymbol\{\\mathrm\{c\}\}\)
We can infer that in MoNIMℳ\\mathcal\{M\}, the functioning form of memory entries is\(𝐤~→𝐜\)\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\\to\\boldsymbol\{\\mathrm\{c\}\}\), while the actual memory entries\(𝐤~→y~\)\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\\to\\tilde\{y\}\)can be explained as economical and practicable appearance\. MoNIM update can be viewed as a collection of sub\-updates, each corresponding to a local mixture of concepts in the MoNIM output:

ℳ​\(𝐱~L\)=∑𝒩​\(𝐱~L\)fℳ​\(𝐱~L⋅𝐤~\)⋅𝐜=∑𝒩​\(𝐱~L\)m~​𝐜\\displaystyle\\mathcal\{M\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)=\\sum\_\{\\mathcal\{N\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)\}f\_\{\\mathcal\{M\}\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\\cdot\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\)\\cdot\\boldsymbol\{\\mathrm\{c\}\}=\\sum\_\{\\mathcal\{N\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)\}\\tilde\{m\}\\boldsymbol\{\\mathrm\{c\}\}
\(𝐤~→y~\)\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\\to\\tilde\{y\}\)integrates into the information stream of Transformer for sake ofEEto transform to\(𝐤~→𝐜\)\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\\to\\boldsymbol\{\\mathrm\{c\}\}\)to operate\. ThroughEE, MoNIM transforms into the general form as inEq\\mathrm\{Eq\}[1](https://arxiv.org/html/2303.01421#S2.E1)\.

log​P​\(y\|𝐱;Θ\)\\displaystyle\\mathrm\{log\}P\(y\|\\boldsymbol\{\\mathrm\{x\}\};\\Theta\)=log​exp​\(\(1−λ\)​𝒆y​𝐨L\+λ​𝒆y​ℳ​\(𝐱~L\)\)Z​\(E​\(\(1−λ\)​𝐨L\+λ​ℳ​\(𝐱~L\)\)\)\\displaystyle=\\mathrm\{log\}\\frac\{\\mathrm\{exp\}\(\(1\-\\lambda\)\\boldsymbol\{e\}\_\{y\}\\boldsymbol\{\\mathrm\{o\}\}^\{L\}\+\\lambda\\boldsymbol\{e\}\_\{y\}\\mathcal\{M\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)\)\}\{Z\(E\(\(1\-\\lambda\)\\boldsymbol\{\\mathrm\{o\}\}^\{L\}\+\\lambda\\mathcal\{M\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)\)\)\}∝\\displaystyle\\proptolog​exp​\(\(1−λ\)​𝒆y​𝐨L\)Z​\(E​𝐨L\)​exp​\(λ​𝒆y​ℳ​\(𝐱~L\)\)Z​\(E​ℳ​\(𝐱~L\)\)\\displaystyle\\mathrm\{log\}\\frac\{\\mathrm\{exp\}\(\(1\-\\lambda\)\\boldsymbol\{e\}\_\{y\}\\boldsymbol\{\\mathrm\{o\}\}^\{L\}\)\}\{Z\(E\\boldsymbol\{\\mathrm\{o\}\}^\{L\}\)\}\\frac\{\\mathrm\{exp\}\(\\lambda\\boldsymbol\{e\}\_\{y\}\\mathcal\{M\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)\)\}\{Z\(E\\mathcal\{M\}\(\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}^\{L\}\)\)\}=\\displaystyle=\(1−λ\)​log​P​\(y\|𝐱;θ\)\+λ​log​P​\(y\|𝐱~;ℳ\)\\displaystyle\(1\-\\lambda\)\\mathrm\{log\}P\(y\|\\boldsymbol\{\\mathrm\{x\}\};\\theta\)\+\\lambda\\mathrm\{log\}P\(y\|\\widetilde\{\\boldsymbol\{\\mathrm\{x\}\}\};\\mathcal\{M\}\)where𝒆y\\boldsymbol\{e\}\_\{y\}is the embedding ofyy, andZ​\(⋅\)Z\(\\cdot\)is the constant softmax normalization factor\.[Figure˜1](https://arxiv.org/html/2303.01421#S1.F1)\(b\) demonstrates the equivalent working flows between\(𝐤~→𝐜\)\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\}\\to\\boldsymbol\{\\mathrm\{c\}\}\)in MoNIM and\(𝐧𝓁,𝐨𝓁\)\(\\boldsymbol\{\\mathrm\{n\}\}^\{\\mathscr\{l\}\},\\boldsymbol\{\\mathrm\{o\}\}^\{\\mathscr\{l\}\}\)inℱL\\mathcal\{F\}^\{L\}, formalizing MoNIM as an FFN\-like bypass layer\. From this perspective, MoNIM focuses on promoting local mixtures of concepts induced by the memorized neighbors, augmenting the induction abilities of LMs in the final layers\. The complicated reasoning tasks are left to the parametric model which handles them better\.

## 3MoNIM is a Scalable Continual Learner

MoNIM’s blend of memorization and induction suggests its potential to adapt to new knowledge, namely it canlearnas induction heads and compress worthless data for its induction task\(Jianget al\.,[2024](https://arxiv.org/html/2303.01421#bib.bib49); Delétanget al\.,[2024](https://arxiv.org/html/2303.01421#bib.bib25)\)\. When the model confidently relies on global reasoning to tackle problems, MoNIM should step back to avoid impacting the model’s performance\. However, when the model lacks information for a decision, MoNIM should step in, promoting memorized local concept mixtures to help the model generate a more probable prediction\.

### 3\.1Learning Strategies of MoNIM

We propose learning strategies for MoNIM that adopt cross\-entropy, the optimization objectives of gradient descent\.222Other possible strategies are discussed in[SectionC\.1](https://arxiv.org/html/2303.01421#A3.SS1)\.

In gradient descent, the greater the cross entropy, the greater the gradient and the impact of data on parameter updates\. For MoNIM learning, we indicate the same effect of data on memory capacity\. The memory effect of a sampleMΘ​\(𝐱\)M\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\)can be expressed by its loss on modelΘ\\Theta:

MΘ​\(𝐱\)∝log⁡P​\(xt\|𝐱<𝐭;Θ\)M\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\)\\propto\\log P\(x\_\{t\}\|\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\};\\Theta\)
To compress the data, rather than assign weights to indicate the importance of data points, we transform the weighted update into a "full\-or\-none" compressorM^Θ​\(𝐱\)\\hat\{M\}\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\), namely only updates that weigh above a thresholdδ\\deltawill be saved into memory\. Through this approximate method, we compress the unimportant part of data to take up no space and prove that the compressed part of data has very little effect on results\.

M^Θ​\(𝐱\)=\{1if​log⁡P​\(xt\|𝐱<𝐭;Θ\)<δ0else\\hat\{M\}\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\)=\\begin\{cases\}1&\\text\{if\}~\\log P\(x\_\{t\}\|\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\};\\Theta\)<\\delta\\\\ 0&\\text\{else\}\\end\{cases\}\(2\)ℳ←ℳ∪\{𝟙M^Θ​\(𝐱\)​\(𝐱<𝐭~→xt\)\}\\mathcal\{M\}\\leftarrow\\mathcal\{M\}~\\cup~\\\{\\mathbbm\{1\}\_\{\\hat\{M\}\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\)\}\(\\boldsymbol\{\\widetilde\{\\mathrm\{x\_\{<t\}\}\}\}\\to x\_\{t\}\)\\\}\(3\)

### 3\.2Adaptive MoNIM weight

Instead of using a fixed threshold \(δ\\deltainEq\\mathrm\{Eq\}[2](https://arxiv.org/html/2303.01421#S3.E2)\), we propose to use an adaptive memorization threshold \(AMT\) to enhance the effect of MoNIM:333The ablation of AMT is placed in[SectionC\.4](https://arxiv.org/html/2303.01421#A3.SS4)\.

δada=δmaxy⁡log⁡P​\(y\|𝐱<𝐭;Θ\)P​\(xt\|𝐱<𝐭;Θ\)\+0\.5\\delta\_\{\\text\{ada\}\}=\\frac\{\\delta\}\{\\max\_\{y\}\\log\\frac\{P\\left\(y\|\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\};\\Theta\\right\)\}\{P\\left\(x\_\{t\}\|\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\};\\Theta\\right\)\}\+0\.5\}
The best form of threshold is not the focus of this paper, however, we found that AMT\-like types of threshold boost the experimental results\. The intuition of AMT is straightforward: ifmaxylogP\(y\|𝐱<𝒕;Θ\)==logP\(xt\|𝐱<𝒕;Θ\)\\max\_\{y\}\\log P\\left\(y\|\\boldsymbol\{\\mathrm\{x\}\}\_\{<\\boldsymbol\{t\}\};\\Theta\\right\)==\\log P\\left\(x\_\{t\}\|\\boldsymbol\{\\mathrm\{x\}\}\_\{<\\boldsymbol\{t\}\};\\Theta\\right\), then the memorization marginδada←2​δ\\delta\_\{\\text\{ada\}\}\\leftarrow 2\\delta\(δ\\deltais the base threshold\), meaning we can relax the threshold to2​δ2\\deltasincextx\_\{t\}is already the top\-1 prediction and thus not urgent to be memorized\. On the contrary, ifmaxy⁡log⁡P​\(y\|𝐱<𝒕;Θ\)\>\>log⁡P​\(xt\|𝐱<𝒕;Θ\)\\max\_\{y\}\\log P\\left\(y\|\\boldsymbol\{\\mathrm\{x\}\}\_\{<\\boldsymbol\{t\}\};\\Theta\\right\)\>\>\\log P\\left\(x\_\{t\}\|\\boldsymbol\{\\mathrm\{x\}\}\_\{<\\boldsymbol\{t\}\};\\Theta\\right\), thenδada<δ\\delta\_\{\\text\{ada\}\}<\\delta, indicating we should aggressively memorize this sample because of the large gap between it and the top\-1 prediction\.

AMT is simple yet effective in practice\. It allows us to skip many samples with the top rank, substantially reducing the memory size with marginal generation quality loss; moreover, it alleviates overfitting top\-ranked samples, playing a similar role as label smoothing to avoid overconfident predictions\.

Since MoNIM updates, unlike FFN layers\(Gevaet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib32)\), always promote concepts rather than eliminate or run shortcuts, if extracted neighbors are so thin and scattered that there is no reliable concept to promote, its weightλ\\lambdashould be pushed down accordingly\. Inspired by previous studies\(Heet al\.,[2021a](https://arxiv.org/html/2303.01421#bib.bib41); Drozdovet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib26)\), we train a simple calibrator to inform the reliability of MoNIM with three categories of features: distribution information of the parametric LM, lexical information of the training data, and density information of MoNIM\. FollowingHeet al\.\([2021a](https://arxiv.org/html/2303.01421#bib.bib41)\), we use a 4\-layer MLP network, optimized on a small subset of the validation set\.444The detailed implementation of the calibrator is placed in[SectionA\.3](https://arxiv.org/html/2303.01421#A1.SS3), and the ablation study in[SectionC\.3](https://arxiv.org/html/2303.01421#A3.SS3)\.

### 3\.3Scalability of MoNIM

Unlike the space\-inefficientkkNN\-LM, MoNIM’s learning capability allows it to compress and reduce memory demand throughout the learning process\. We designed experiments in continual learning settings showing that MoNIM can keep compressing when updating\. Further, we reveal that the features of compression are consistent with those of updates of model parameters and lead MoNIM to scalability:\(i\) data scalability: In parametric models, as learning progresses, the impact of data on parameter updates diminishes; similarly, in MoNIM, as it continues to learn, the influence of data on memory capacity diminishes, meaning less new information needs to be memorized\.\(ii\) model scalability: As parametric models grow in size, the impact of data on parameter updates decreases; likewise, in MoNIM, the impact of data on memory capacity also diminishes with model growth\.

## 4Experiments

### 4\.1Experimental Setting

We use the news from December 2019 in the News Crawl corpus \(NC\-19Dec\) as pilot data, and applyNewscrawl\-20H1\(NC\-20H1\), the articles during the first half of 2020 in the News Crawl, as our streaming data for continual learning \(CL\)\. We randomly select 1% data per day as the validation and test set, and the rest 98% articles as the training set\.[Table˜1](https://arxiv.org/html/2303.01421#S4.T1)shows the statistics of NC\-20H1\. We continually learn the streaming data in chronological order and update the search index555Implementation of search is included in[SectionA\.2](https://arxiv.org/html/2303.01421#A1.SS2)\.every day\.

In addition, we constructWikiEvent\-20H1\(WE\-20H1\), a Wikipedia event dataset666An example event article is[2020 Caribbean earthquake](https://en.wikipedia.org/wiki/2020_Caribbean_earthquake)\.describing real\-world events during 20H1, for testing our approach in domains other than news\. WE\-20H1 contains, on average 10 Wikipedia articles per month with∼\\sim100k tokens in total\.

We use GPT\-2\(Radfordet al\.,[2019](https://arxiv.org/html/2303.01421#bib.bib80)\)as the backbone LM to study CL over 20H1’s streaming data\. We experiment with the GPT\-2 small \(S, 123M\), medium \(M, 355M\), and large \(L, 774M\) variants777The detailed configurations are in[SectionA\.1](https://arxiv.org/html/2303.01421#A1.SS1)\., and GPT\-2 small is assumed to be the default size unless otherwise specified\. All the experiments are implemented using the Fairseq \(Ottet al\.,[2019](https://arxiv.org/html/2303.01421#bib.bib74)\) toolkit and run on 1 NVIDIA V100 GPU\.

We define the memorization rate \(MemRate\) as the percentage of key\-value pairs stored in memory compared to theFullMembaseline\. MemRate is utilized to measure the memory efficiency and scalability of our method\. It comes that for NC\-19Dec, whenδ=−1\.5\\delta=\-1\.5, MoNIM can achieve comparable performance toFullMemwith∼\\sim60% MemRate\. Thus, we setδ=−1\.5\\delta=\-1\.5888We explored the effect of different choices ofδ\\deltaon performance and memorization in[SectionC\.2](https://arxiv.org/html/2303.01421#A3.SS2)\.throughout our following experiments\.

After CL, we conducted extensive experiments in both language modeling and downstream tasks to estimate MoNIM’s performance and scalability\. MoNIM is compared with the following baselines:

- •Full memorization \(FullMem\): Conventional memorization policy that memorizes every token in the training set\.
- •Random memorization \(RandMem\): Randomly memorize data equal to MoNIM’s initial MemRate \(60%\)\. We conduct three runs with random seeds and choose the best as the baseline\.

DailyMonthlyTotal\#Train\#Dev/Test\#Train\#Dev/Test\#Train\#Dev/TestArticles4\.4K46133K1\.3K796K8\.2KTokens2\.4M24\.7K73\.2M741K439M4\.5M

Table 1:Statistics of NC\-20H1\.![Refer to caption](https://arxiv.org/html/2303.01421v2/x3.png)Figure 2:\(a\)The memory growth of the conventional full memorization \(FullMem\) policy and our proposed MoNIM policy, whose performance is controlled to be comparable with that ofFullMem\. Not only does MoNIM significantly reduce its memory but also the growth of its memory will become slower than when it starts \(i\.e\., the green dashed line\), as it continuously learns;\(b\)The growth trend of memory size \(GB\) with the increase of the model size \(from 123M to 3B\)\. The red numbers below the MoNIM’s line indicate the memory consumption reduced by MoNIM, showing that MoNIM’s effect is more remarkable in larger LMs\.
### 4\.2Results of language modeling

#### 4\.2\.1Data scalability

[Table˜2](https://arxiv.org/html/2303.01421#S4.T2)compares between MoNIM andFullMem,RandMemfor CL with NC\-20H1, showing substantial improvements for the bare GPT\-2 small\. Among them, MoNIM achieves comparable \(even slightly better\) language modeling performance toFullMembut with only 50% MemRate, and largely outperformsRandMem\(60% MemRate\), demonstrating that MoNIM is a cost\-effective memorization policy for CL\.

MethodsPPL \(↓\\downarrow\)MemRate \(↓\\downarrow\)Bare GPT\-2 small24\.10%\+FullMem9\.0100%\+RandMem15\.060%\+MoNIM8\.650%

Table 2:Perplexity \(PPL\) on the test set of NC\-20H1 by different memorization methods for CL over NC\-20H1\.We look into the results by tracking MoNIM’s monthly memorization throughout CL\. The sublinear growth trend of MoNIM observed in Figure[2](https://arxiv.org/html/2303.01421#S4.F2)\(a\) indicates that the compression rate of MoNIM gradually increases as it learns over time because the model becomes increasingly knowledgeable and skips more training cases that it already knows\.

ModelNC\-20July\(same domain\)WE\-20H1\(different domain\)MemRatePPLMemRatePPLGPT\-2 small60%8\.862%29\.0\+CL w/ NC\-20H144%8\.257%27\.5

Table 3:Comparison of memorization rate and perplexity for new CL data – NC\-20July and WE\-20H1 before and after CL with NC\-20H1\.We further confirm MoNIM’s data\-wise scalability by comparing the MemRates of two additional CL datasets — news data from July 2020 in the News Crawl \(NC\-20July\) and WE\-20H1 — before and after CL with NC\-20H1\. Following CL on NC\-20H1, we utilize MoNIM \(equipped with the memory acquired from NC\-20H1\) to continually process NC\-20July and WE\-20H1\. As shown in Table[3](https://arxiv.org/html/2303.01421#S4.T3), MoNIM’s MemRates for both datasets are significantly lower than those of the models without CL, while improving performance\. This reduction in MemRates can be attributed to the prior CL, which enables the model to avoid memorizing numerous instances learned previously, without compromising performance\.

#### 4\.2\.2Model scalability

[Table˜4](https://arxiv.org/html/2303.01421#S4.T4)shows the model size’s effect on MoNIM\. In the premise of comparable results toFullMem, the MoNIM’s effect becomes more significant as the parametric model becomes larger: its overall MemRate drops from 50% \(GPT\-2 small\) to 40% \(GPT\-2 large\)\. As we assumed, a larger LM tends to skip more training cases than a smaller LM\. The reduced MemRate demonstrates the model\-wise scalability of MoNIM\.

ModelStrategyPPL \(↓\\downarrow\)MemRate \(↓\\downarrow\)S\(123M\)FullMem9\.0100%MoNIM8\.650%M\(355M\)FullMem7\.0100%MoNIM7\.246%L\(774M\)FullMem6\.2100%MoNIM6\.340%

Table 4:MoNIM’s model\-wise scalability for GPT\-2 models of different sizes\.To test the generalizability of scalable memory on larger LMs, we choose Meta’s Llama\-3\.2\-1B and 3B versions\(Dubeyet al\.,[2024](https://arxiv.org/html/2303.01421#bib.bib1)\)for a brief evaluation of MoNIM’s performance999Since Llama\-3\.2 was released on September 25, 2024, we extract news from the first week of October 2024 in the News Crawl corpus to implement this experiment\.\. As shown in[Figure˜2](https://arxiv.org/html/2303.01421#S4.F2)\(b\), the total MemRate reduces from GPT2\-small’s 59\.5% \(123M\) to Llama\-3\.2\-3B’s 42\.0% \(3B\)\. We assume that larger LMs have the potential to achieve even more negligible memory consumption, as long as MoNIM maintains both data\- and model\-scalability\.

MethodsWiki\-1031 \(Jan\)2 \(Feb\)3 \(Mar\)4 \(Apr\)5 \(May\)6 \(Jun\)Bare GPT\-229\.124\.324\.024\.124\.024\.723\.8\+Fine\-tune33\.4 \(\+2\.9\)20\.4 \(\+2\.4\)18\.7 \(\+2\.3\)17\.3 \(\+1\.3\)17\.3 \(\+0\.2\)16\.2 \(\+0\.8\)15\.4 \(\+0\.0\)\(best\)30\.618\.216\.416\.017\.115\.415\.4\+RecAdam34\.5 \(\+2\.7\)19\.6 \(\+0\.7\)18\.3 \(\+0\.7\)17\.3 \(\+0\.2\)17\.1 \(\+0\.3\)16\.8 \(\+0\.3\)16\.9 \(\+0\.0\)\(best\)31\.818\.917\.617\.116\.816\.516\.9\+MixReview33\.6 \(\+3\.0\)19\.9 \(\+1\.8\)18\.5 \(\+2\.1\)17\.3 \(\+1\.4\)17\.2 \(\+0\.1\)15\.8 \(\+0\.4\)15\.6 \(\+0\.0\)\(best\)30\.618\.116\.415\.917\.115\.415\.6\+Greedy Merging35\.2 \(\+5\.9\)15\.3 \(\+6\.3\)15\.5 \(\+5\.2\)16\.0 \(\+3\.1\)15\.9 \(\+2\.8\)15\.8 \(\+1\.8\)14\.6 \(\+0\.0\)\(best\)29\.39\.010\.312\.913\.114\.414\.6\+MoNIM29\.9\(\+0\.5\)9\.4\(\+0\.1\)7\.6\(\+0\.2\)7\.8\(\+0\.5\)6\.9\(\+0\.1\)9\.5\(\+0\.0\)8\.8\(\+0\.0\)\(best\)29\.49\.37\.47\.36\.89\.58\.8

Table 5:PPL evaluated on the 7 test sets after CL over NC\-20H1 between MoNIM and representative CL approaches\. The numbers in the second row of each cell denote the best result achieved during the process of CL\.![Refer to caption](https://arxiv.org/html/2303.01421v2/x4.png)Figure 3:Language modeling performance on 7 test sets \(Wiki\-103 and 6 subsets of Newcrawl\-20H1’s test set by month\) throughout CL via MoNIM over NC\-20H1\.Model \(MemRate\)Wiki\-103NC\-20H1ACLBare GPT\-2 \(0%\)29\.124\.140\.5\+FullMem\(100%\)30\.1→\\to31\.09\.0→\\to10\.022\.5\+MoNIM \(64%\)29\.9→\\to30\.48\.6→\\to9\.222\.7

Table 6:CL performance over ACL papers after learning NC\-20H1\. The numbers beside each arrow indicate the PPL before/after studying the ACL papers\.
#### 4\.2\.3Mitigation of forgetting

MoNIM’s learning against forgetting performance is evaluated by tracking results on 7 test sets throughout CL over the NC\-20H1: 6 are each month’s held\-out data in NC\-20H1, and the other is the test set of Wiki\-103 benchmark \(Merityet al\.,[2017](https://arxiv.org/html/2303.01421#bib.bib71)\) which does not overlap with the NC\-20H1 training data\. According to Figure[3](https://arxiv.org/html/2303.01421#S4.F3), MoNIM learns from the streaming data well, reflected by the sharp decrease of PPL on a test set after learning its corresponding month’s training data\. More importantly, it does not suffer much from the catastrophic forgetting issue \(French,[1999](https://arxiv.org/html/2303.01421#bib.bib28)\)\. PPL scores of all 7 test sets do not significantly degrade throughout CL, since MoNIM will never erase previous memory or update the weights of LM\.

The advantage can be better understood by comparing MoNIM with other CL baselines\. We select two popular CL methods, RecAdam\(Chenet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib19)\)and Mix\-Review\(Heet al\.,[2021b](https://arxiv.org/html/2303.01421#bib.bib40)\), with Greedy Merging, the most effective data compression approach inHeet al\.\([2021a](https://arxiv.org/html/2303.01421#bib.bib41)\)\.101010Details of CL baselines are included in[AppendixB](https://arxiv.org/html/2303.01421#A2)\.As in Table[5](https://arxiv.org/html/2303.01421#S4.T5), MoNIM not only achieves better results in learning from the new data but also suffers less from catastrophic forgetting than other CL approaches despite introducing additional memory\.

![Refer to caption](https://arxiv.org/html/2303.01421v2/x5.png)Figure 4:Next\-word prediction accuracy on 6 subsets \(by month\) of WE\-20H1 throughout CL over NC\-20H1\.MethodsAcc \(↑\\uparrow\)MemRate \(↓\\downarrow\)SMLSMLBare GPT\-239\.541\.244\.60%0%0%\+FullMem43\.045\.548\.9100%100%100%\+RandMem40\.341\.345\.560%60%60%\+MoNIM43\.845\.448\.550%46%40%

Table 7:Next\-word prediction accuracy \(Acc\) on the test set \(WE\-20H1\) and MemRate by different memorization methods and models after CL over NC\-20H1\.
#### 4\.2\.4Domain adaptation

After CL over the news streaming data, which is not of great difference from the LM’s pretraining dataset \(i\.e\., openwebtext for GPT\-2\), we test MoNIM’s CL performance over data in another domain – the ACL paper dataset\(Loet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib66)\)with 42K ACL papers\. We hold off 80 papers \(∼\\sim200K tokens\) to construct the validation and test set, using the rest for training\.111111We split the training data into 4 batches for CL and update the index after finishing each batch\.[Table˜6](https://arxiv.org/html/2303.01421#S4.T6)shows that utilizing less memory \(64% compared withFullMem\), MoNIM consistently performs well in new data \(40\.5→\\to22\.7 in PPL\) with less forgetting degradation\. AlthoughFullMemis also relatively resilient to catastrophic forgetting, it stores more noise from in\-domain samples, which can degrade retrieval performance when applied to out\-of\-domain inputs\.

### 4\.3Results of downstream tasks

#### 4\.3\.1Next\-word prediction

For LLM, next\-word prediction is the basic and the most straightforward end task, especially important for AI applications \(e\.g\., input methods,[Microsoft’s text predictions](https://insider.office.com/zh-cn/blog/text-predictions-in-word-outlook)and[ChatGPT](https://chat.openai.com/)\)\.

To align this task with our CL setting, we test the next\-word prediction on the WE\-20H1 to verify if CL over NC\-20H1 can help write the current event articles in Wikipedia\. As in language modeling, MoNIM consistently shows comparable performance with better scalability than FullMem as the model size increases \(Table[7](https://arxiv.org/html/2303.01421#S4.T7)\) and desirable results with little forgetting \(Figure[4](https://arxiv.org/html/2303.01421#S4.F4)\)\.

#### 4\.3\.2Closed\-book question answer

We useRealTime QA\(Kasaiet al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib55)\), a multiple\-choice question dataset about real\-time events, as our second testbed of downstream tasks\. To align with our streaming data, we use the subset of news during 20H1 and evaluate it in the closed\-book Multiple Choice setting\. AsKasaiet al\.\([2023](https://arxiv.org/html/2303.01421#bib.bib55)\)suggests, we evaluate GPT\-2 large in a zero\-shot learning setting, in which GPT\-2 small and medium are too weak to perform\.

[Table˜8](https://arxiv.org/html/2303.01421#S4.T8)shows the results inRealTime QA\. Compared with the bare GPT\-2 large, CL through MoNIM over NC\-20H1 substantially improves QA performance because it learns the world knowledge during 20H1 from the news stream to answer the questions\. MoNIM again performs as well as FullMem with less memory footprint and outperforms RandMem\. Moreover, little forgetting is consistently observed, as shown in Table[9](https://arxiv.org/html/2303.01421#S4.T9)\.

Methods \(MemRate\)Acc \(↑\\uparrow\)Bare GPT\-2 large \(0%\)29\.8\+FullMem\(100%\)36\.3\+RandMem\(60%\)30\.7\+MoNIM \(40%\)36\.2

Table 8:Accuracy onRealTime QAof GPT\-2 large with CL over NC\-20H1 in zero\-shot learning setting\.CL overTest on1\-23\-45\-61\-236\.029\.329\.51\-437\.235\.830\.61\-637\.036\.535\.5

Table 9:Accuracy on every two months of GPT\-2 large ofRealTime QAthroughout CL over NC\-20H1\.

## 5Related Work

##### Retrieval\-augmented LMs

Retrieval components have been found beneficial for language tasks\. Unlike the explicit storage methods\(Guuet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib38); Borgeaudet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib16)\), semiparametric LMs likekkNN\-LM \(Khandelwalet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib58)\) store implicit information as key\-value pairs to assist prediction, without the need for retraining\. As a powerful method to use the external data, many successive works ofkkNN\-LM have been proposed\(Zhenget al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib112); Jinet al\.,[2022b](https://arxiv.org/html/2303.01421#bib.bib53); Trottaet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib98); Bhardwajet al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib14)\)\. Among them,Heet al\.\([2021a](https://arxiv.org/html/2303.01421#bib.bib41)\)focuses on improving the efficiency, which has similar applications as our work, but we focus more on interpretable scalability in CL over streaming data with orthogonal contributions\.

##### Interpretable LMs

It remains obscure how Transformer manages to understand and generate natural languages\. Among all the struggles to open the black box, mechanistic interpretability\(Elhageet al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib27); Olssonet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib73); Gurneeet al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib37)\)investigates neurons and their connections in terms of circuits where information flows and transforms\. Previous works have found many components that provide learning capabilities\(Merulloet al\.,[2024](https://arxiv.org/html/2303.01421#bib.bib5); Zhang and Nanda,[2024](https://arxiv.org/html/2303.01421#bib.bib4)\)\. We transplant these interpretations that work for the parametric models to non\-parametric memory, which has not been reasonably explained\. Experiments prove the feasibility of our conceptual framework of MoNIM\.

##### Continual learning LMs

Continual learning \(CL\) proposes to address the “new knowledge \- catastrophic forgetting” dilemma \(French,[1999](https://arxiv.org/html/2303.01421#bib.bib28)\)\. According to the taxology ofWanget al\.\([2024](https://arxiv.org/html/2303.01421#bib.bib7)\), our method deals with catastrophic forgetting problems based on replay\-based methods \(Sunet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib94); Qinet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib78); Scialomet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib87)\), despite that we managed to build a learnable replay memory\. CL for LM is gaining traction\(Keet al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib56); Razdaibiedinaet al\.,[2023](https://arxiv.org/html/2303.01421#bib.bib8)\), and the closest works to us areJanget al\.\([2022](https://arxiv.org/html/2303.01421#bib.bib46)\)andJinet al\.\([2022a](https://arxiv.org/html/2303.01421#bib.bib52)\), which adapt LMs to emerging corpora across domains and timelines\. However, we are the first to explore the memory to deal with non\-parametric solutions for CL over streaming data\.

## 6Conclusion

We introduced Mixture\-of\-Neighbors Induction Memory \(MoNIM\), a novel conceptual framework that integrates dynamic induction memory into the Transformer architecture to interpret and enhance semiparametric LMs\. Our experiments demonstrate that MoNIM not only offers a fresh perspective on non\-parametric memory but also sets a new benchmark for scalable and efficient learning in LLMs, giving insights for the evolution of LLMs without the need for parameter adjustments\.

## Limitations

We construct the framework of MoNIM and thoroughly investigate its practicality and effectiveness as a representative of semiparametric LMs in continual learning\. However, MoNIM only formalizes the interpretation ofkkNN\-LM, while there are diverse models and implementations under semiparametric LMs\. Specifically, besides auto\-regressive models, auto\-encoder models like T5 \(Janget al\.,[2022](https://arxiv.org/html/2303.01421#bib.bib46)\) also exhibit their potential for continual LM\. Although we have observed that T5s are empirically capable of continual learning, the framework we constructed does not currently include them\. In the future, we intend to extend to varied models and architectures and confirm the universal effectiveness of our framework\.

Due to resource constraints, we tested our method on data within half a year \(20H1\) and on models up to 3B in size\. Stretching the time series and increasing the model size is urgent for observing a more prominent and convincing build\-up curve for a longer period\.

## Acknowledgments

This work was supported by National Natural Science Foundation of China \(62036001\) and National Science and Technology Major Project \(No\. 2022ZD0116308\) \. The corresponding author is Houfeng Wang\.

## References

- H\. Bansal, K\. Gopalakrishnan, S\. Dingliwal, S\. Bodapati, K\. Kirchhoff, and D\. Roth \(2023\)Rethinking the role of scale for in\-context learning: an interpretability\-based case study at 66 billion scale\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 11833–11856\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.660),[Link](https://aclanthology.org/2023.acl-long.660)Cited by:[§2\.2](https://arxiv.org/html/2303.01421#S2.SS2.p1.1)\.
- R\. Bhardwaj, G\. Polovets, and M\. Sunkara \(2023\)Adaptation approaches for nearest neighbor language models\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1135–1146\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.73),[Link](https://aclanthology.org/2023.findings-acl.73)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. van den Driessche, J\. Lespiau, B\. Damoc, A\. Clark, D\. de Las Casas, A\. Guy, J\. Menick, R\. Ring, T\. Hennigan, S\. Huang, L\. Maggiore, C\. Jones, A\. Cassirer, A\. Brock, M\. Paganini, G\. Irving, O\. Vinyals, S\. Osindero, K\. Simonyan, J\. W\. Rae, E\. Elsen, and L\. Sifre \(2022\)Improving language models by retrieving from trillions of tokens\.InInternational Conference on Machine Learning, ICML 2022, 17\-23 July 2022, Baltimore, Maryland, USA,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvári, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 2206–2240\.External Links:[Link](https://proceedings.mlr.press/v162/borgeaud22a.html)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- S\. Chen, Y\. Hou, Y\. Cui, W\. Che, T\. Liu, and X\. Yu \(2020\)Recall and learn: fine\-tuning deep pretrained language models with less forgetting\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 7870–7881\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.634),[Link](https://aclanthology.org/2020.emnlp-main.634)Cited by:[Appendix B](https://arxiv.org/html/2303.01421#A2.p1.1),[§4\.2\.3](https://arxiv.org/html/2303.01421#S4.SS2.SSS3.p2.1)\.
- G\. Delétang, A\. Ruoss, P\. Duquenne, E\. Catt, T\. Genewein, C\. Mattern, J\. Grau\-Moya, L\. K\. Wenliang, M\. Aitchison, L\. Orseau, M\. Hutter, and J\. Veness \(2024\)Language modeling is compression\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=jznbgiynus)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p2.1),[§3](https://arxiv.org/html/2303.01421#S3.p1.1)\.
- A\. Drozdov, S\. Wang, R\. Rahimi, A\. McCallum, H\. Zamani, and M\. Iyyer \(2022\)You can’t pick your neighbors, or can you? when and how to rely on retrieval in the kNN\-LM\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 2997–3007\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.218),[Link](https://aclanthology.org/2022.findings-emnlp.218)Cited by:[§3\.2](https://arxiv.org/html/2303.01421#S3.SS2.p5.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.ArXiv preprintabs/2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.2\.2](https://arxiv.org/html/2303.01421#S4.SS2.SSS2.p2.1)\.
- N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah \(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p3.1),[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px2.p1.1)\.
- R\. M\. French \(1999\)Catastrophic forgetting in connectionist networks\.Trends in Cognitive Sciences\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S1364661399012942)Cited by:[§4\.2\.3](https://arxiv.org/html/2303.01421#S4.SS2.SSS3.p1.1),[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- M\. Geva, A\. Caciularu, K\. Wang, and Y\. Goldberg \(2022\)Transformer feed\-forward layers build predictions by promoting concepts in the vocabulary space\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 30–45\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.3),[Link](https://aclanthology.org/2022.emnlp-main.3)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p4.1),[§2\.3](https://arxiv.org/html/2303.01421#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2303.01421#S2.SS3.p3.1),[§3\.2](https://arxiv.org/html/2303.01421#S3.SS2.p5.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446),[Link](https://aclanthology.org/2021.emnlp-main.446)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p3.1),[§2\.3](https://arxiv.org/html/2303.01421#S2.SS3.p1.1)\.
- W\. Gurnee, N\. Nanda, M\. Pauly, K\. Harvey, D\. Troitskii, and D\. Bertsimas \(2023\)Finding neurons in a haystack: case studies with sparse probing\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=JYs1R9IMJr)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px2.p1.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)Retrieval augmented language model pre\-training\.InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\-18 July 2020, Virtual Event,Proceedings of Machine Learning Research, Vol\.119,pp\. 3929–3938\.External Links:[Link](http://proceedings.mlr.press/v119/guu20a.html)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p1.1),[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- J\. He, G\. Neubig, and T\. Berg\-Kirkpatrick \(2021a\)Efficient nearest neighbor language models\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5703–5714\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.461),[Link](https://aclanthology.org/2021.emnlp-main.461)Cited by:[Appendix B](https://arxiv.org/html/2303.01421#A2.p3.1),[§1](https://arxiv.org/html/2303.01421#S1.p2.1),[§3\.2](https://arxiv.org/html/2303.01421#S3.SS2.p5.1),[§4\.2\.3](https://arxiv.org/html/2303.01421#S4.SS2.SSS3.p2.1),[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- T\. He, J\. Liu, K\. Cho, M\. Ott, B\. Liu, J\. Glass, and F\. Peng \(2021b\)Analyzing the forgetting problem in pretrain\-finetuning of open\-domain dialogue response models\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,P\. Merlo, J\. Tiedemann, and R\. Tsarfaty \(Eds\.\),Online,pp\. 1121–1133\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.95),[Link](https://aclanthology.org/2021.eacl-main.95)Cited by:[Appendix B](https://arxiv.org/html/2303.01421#A2.p2.1),[§4\.2\.3](https://arxiv.org/html/2303.01421#S4.SS2.SSS3.p2.1)\.
- J\. Jang, S\. Ye, S\. Yang, J\. Shin, J\. Han, G\. Kim, S\. J\. Choi, and M\. Seo \(2022\)Towards continual knowledge learning of language models\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=vfsRB5MImo9)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1),[Limitations](https://arxiv.org/html/2303.01421#Sx1.p1.1)\.
- H\. Jiang, Q\. Wu, X\. Luo, D\. Li, C\. Lin, Y\. Yang, and L\. Qiu \(2024\)LongLLMLingua: accelerating and enhancing LLMs in long context scenarios via prompt compression\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 1658–1677\.External Links:[Link](https://aclanthology.org/2024.acl-long.91)Cited by:[§3](https://arxiv.org/html/2303.01421#S3.p1.1)\.
- X\. Jin, D\. Zhang, H\. Zhu, W\. Xiao, S\. Li, X\. Wei, A\. Arnold, and X\. Ren \(2022a\)Lifelong pretraining: continually adapting language models to emerging corpora\.InProceedings of BigScience Episode \#5 – Workshop on Challenges & Perspectives in Creating Large Language Models,A\. Fan, S\. Ilic, T\. Wolf, and M\. Gallé \(Eds\.\),virtual\+Dublin,pp\. 1–16\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.bigscience-1.1),[Link](https://aclanthology.org/2022.bigscience-1.1)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- X\. Jin, T\. Ge, and F\. Wei \(2022b\)Plug and play knowledge distillation for kNN\-LM with external logits\.InProceedings of the 2nd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\),Y\. He, H\. Ji, S\. Li, Y\. Liu, and C\. Chang \(Eds\.\),Online only,pp\. 463–469\.External Links:[Link](https://aclanthology.org/2022.aacl-short.57)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- J\. Johnson, M\. Douze, and H\. Jégou \(2021\)Billion\-scale similarity search with gpus\.IEEE Trans\. Big Data7\(3\),pp\. 535–547\.External Links:[Document](https://dx.doi.org/10.1109/TBDATA.2019.2921572),[Link](https://doi.org/10.1109/TBDATA.2019.2921572)Cited by:[§A\.2](https://arxiv.org/html/2303.01421#A1.SS2.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.ArXiv preprintabs/2001\.08361\.External Links:[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p4.1)\.
- J\. Kasai, K\. Sakaguchi, Y\. Takahashi, R\. L\. Bras, A\. Asai, X\. Yu, D\. Radev, N\. A\. Smith, Y\. Choi, and K\. Inui \(2023\)RealTime QA: what’s the answer right now?\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/9941624ef7f867a502732b5154d30cb7-Abstract-Datasets%5C_and%5C_Benchmarks.html)Cited by:[§4\.3\.2](https://arxiv.org/html/2303.01421#S4.SS3.SSS2.p1.1)\.
- Z\. Ke, H\. Lin, Y\. Shao, H\. Xu, L\. Shu, and B\. Liu \(2022\)Continual training of language models for few\-shot learning\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 10205–10216\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.695),[Link](https://aclanthology.org/2022.emnlp-main.695)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- U\. Khandelwal, A\. Fan, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis \(2021\)Nearest neighbor machine translation\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=7wCBOfJ8hJM)Cited by:[§2\.1](https://arxiv.org/html/2303.01421#S2.SS1.p3.6)\.
- U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis \(2020\)Generalization through memorization: nearest neighbor language models\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:[Link](https://openreview.net/forum?id=HklBjCEKvH)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p3.1),[§2\.1](https://arxiv.org/html/2303.01421#S2.SS1.p2.9),[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- K\. Lo, L\. L\. Wang, M\. Neumann, R\. Kinney, and D\. Weld \(2020\)S2ORC: the semantic scholar open research corpus\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4969–4983\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.447),[Link](https://aclanthology.org/2020.acl-main.447)Cited by:[§4\.2\.4](https://arxiv.org/html/2303.01421#S4.SS2.SSS4.p1.2)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\-26, 2017, Conference Track Proceedings,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§4\.2\.3](https://arxiv.org/html/2303.01421#S4.SS2.SSS3.p1.1)\.
- J\. Merullo, C\. Eickhoff, and E\. Pavlick \(2024\)Circuit component reuse across tasks in transformer language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=fpoAYV6Wsk)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px2.p1.1)\.
- C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, S\. Johnston, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah \(2022\)In\-context learning and induction heads\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p3.1),[§2\.2](https://arxiv.org/html/2303.01421#S2.SS2.p1.1),[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px2.p1.1)\.
- M\. Ott, S\. Edunov, A\. Baevski, A\. Fan, S\. Gross, N\. Ng, D\. Grangier, and M\. Auli \(2019\)Fairseq: a fast, extensible toolkit for sequence modeling\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics \(Demonstrations\),W\. Ammar, A\. Louis, and N\. Mostafazadeh \(Eds\.\),Minneapolis, Minnesota,pp\. 48–53\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-4009),[Link](https://aclanthology.org/N19-4009)Cited by:[§4\.1](https://arxiv.org/html/2303.01421#S4.SS1.p3.1)\.
- Y\. Qin, J\. Zhang, Y\. Lin, Z\. Liu, P\. Li, M\. Sun, and J\. Zhou \(2022\)ELLE: efficient lifelong pre\-training for emerging data\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 2789–2810\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.220),[Link](https://aclanthology.org/2022.findings-acl.220)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§4\.1](https://arxiv.org/html/2303.01421#S4.SS1.p3.1)\.
- O\. Ram, Y\. Levine, I\. Dalmedigos, D\. Muhlgay, A\. Shashua, K\. Leyton\-Brown, and Y\. Shoham \(2023\)In\-context retrieval\-augmented language models\.Transactions of the Association for Computational Linguistics11,pp\. 1316–1331\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00605),[Link](https://aclanthology.org/2023.tacl-1.75)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p1.1)\.
- A\. Razdaibiedina, Y\. Mao, R\. Hou, M\. Khabsa, M\. Lewis, and A\. Almahairi \(2023\)Progressive prompts: continual learning for language models\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/pdf?id=UJTgQBc91%5C_)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- T\. Scialom, T\. Chakrabarty, and S\. Muresan \(2022\)Fine\-tuned language models are continual learners\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 6107–6122\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.410),[Link](https://aclanthology.org/2022.emnlp-main.410)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- W\. Shi, J\. Michael, S\. Gururangan, and L\. Zettlemoyer \(2022\)Nearest neighbor zero\-shot inference\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 3254–3265\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.214),[Link](https://aclanthology.org/2022.emnlp-main.214)Cited by:[§C\.5](https://arxiv.org/html/2303.01421#A3.SS5.p2.1)\.
- S\. Sukhbaatar, A\. Szlam, J\. Weston, and R\. Fergus \(2015\)End\-to\-end memory networks\.InAdvances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7\-12, 2015, Montreal, Quebec, Canada,C\. Cortes, N\. D\. Lawrence, D\. D\. Lee, M\. Sugiyama, and R\. Garnett \(Eds\.\),pp\. 2440–2448\.External Links:[Link](https://proceedings.neurips.cc/paper/2015/hash/8fb21ee7a2207526da55a679f0332de2-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2303.01421#S2.SS3.p1.1)\.
- F\. Sun, C\. Ho, and H\. Lee \(2020\)LAMOL: language modeling for lifelong language learning\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:[Link](https://openreview.net/forum?id=Skgxcn4YDS)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- S\. Trotta, L\. Flek, and C\. Welch \(2022\)Nearest neighbor language models for stylistic controllable generation\.InProceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics \(GEM\),A\. Bosselut, K\. Chandu, K\. Dhole, V\. Gangal, S\. Gehrmann, Y\. Jernite, J\. Novikova, and L\. Perez\-Beltrachini \(Eds\.\),Abu Dhabi, United Arab Emirates \(Hybrid\),pp\. 295–305\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.gem-1.25),[Link](https://aclanthology.org/2022.gem-1.25)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.
- K\. R\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt \(2023\)Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/pdf?id=NpsVSN6o4ul)Cited by:[§2\.2](https://arxiv.org/html/2303.01421#S2.SS2.p1.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024\)A comprehensive survey of continual learning: theory, method and application\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(8\),pp\. 5362–5383\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2024.3367329)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px3.p1.1)\.
- D\. Yogatama, C\. de Masson d’Autume, and L\. Kong \(2021\)Adaptive semiparametric language models\.Transactions of the Association for Computational Linguistics9,pp\. 362–373\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00371),[Link](https://aclanthology.org/2021.tacl-1.22)Cited by:[§1](https://arxiv.org/html/2303.01421#S1.p1.1)\.
- F\. Zhang and N\. Nanda \(2024\)Towards best practices of activation patching in language models: metrics and methods\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=Hf17y6u9BC)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px2.p1.1)\.
- X\. Zheng, Z\. Zhang, J\. Guo, S\. Huang, B\. Chen, W\. Luo, and J\. Chen \(2021\)Adaptive nearest neighbor machine translation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 368–374\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-short.47),[Link](https://aclanthology.org/2021.acl-short.47)Cited by:[§5](https://arxiv.org/html/2303.01421#S5.SS0.SSS0.Px1.p1.2)\.

## Appendix AExperiment details

### A\.1Model configuration

We list the model configurations of the GPT\-2 models \(as the parametric LMs\) in our experiments in[Table˜10](https://arxiv.org/html/2303.01421#A1.T10)\.

ModelLayerDim\#ParamGPT\-2 small12768123MGPT\-2 medium241024355MGPT\-2 large361280774MTable 10:Model configurations of the GPT\-2 models \(as the parametric LMs\) in our experiments\.
### A\.2Index building

We use the FAISS toolkit\(Johnsonet al\.,[2021](https://arxiv.org/html/2303.01421#bib.bib54)\)for index building and searching\. At each update, we sampled 1M keys randomly from memory to train 4K cluster centroids, and then the whole keys in memory are added to the trained index, all quantized to 64\-bytes\.

### A\.3NN calibrator training

We list the features we use in training the NN calibrator:

- •Distribution information of the parametric LM - –𝐱~\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}: contextualized representation of𝐱\\boldsymbol\{\\mathrm\{x\}\}by the parameterized LM - –c​o​n​f​\(𝐱\)conf\(\\boldsymbol\{\\mathrm\{x\}\}\):maxy⁡P​\(y\|𝐱;θ\)\\max\_\{y\}P\(y\|\\boldsymbol\{\\mathrm\{x\}\};\\theta\) - –e​n​t​\(𝐱\)ent\(\\boldsymbol\{\\mathrm\{x\}\}\): entropy ofP​\(y\|𝐱;θ\)P\(y\|\\boldsymbol\{\\mathrm\{x\}\};\\theta\)
- •Lexical information of the training data - –log⁡f​r​e​q​\(𝒙−𝟏\)\\log freq\(\\boldsymbol\{x\_\{\-1\}\}\): log of frequency of the last token in the context - –log⁡d​i​s​t​i​n​c​t​\(𝐱−𝟏\)\\log distinct\(\\boldsymbol\{\\mathrm\{x\_\{\-1\}\}\}\): log of the number of distinct values that succeed the last token in the context
- •Density information of the external memory - –d​\(𝐤~,𝐱~\)d\(\\tilde\{\\boldsymbol\{\\mathrm\{k\}\}\},\\tilde\{\\boldsymbol\{\\mathrm\{x\}\}\}\):L2L^\{2\}distance \(semantic similarity\) between the query and the top\-iiretrieved neighbor,i=1,2,…,10i=1,2,\\dots,10\. - –log⁡d​i​s​t​i​n​c​t​\(y~\)\\log distinct\(\\tilde\{y\}\): log of the number of distinct values of the top\-iiretrieved values,i=1,2,…,10i=1,2,\\dots,10\.

On each day during memorization, we extract from the validation set 10 articles and update them into the training set of the calibrator yesterday to obtain the training set of the calibrator today\. The validation set of the calibrator is obtained as above, except for it only needs 5 articles each day\. Because the training set increases slowly every day, we reduce the number of training epochs from 5 epochs to 1 epoch as time goes on, in case of overfitting\.

Each feature is fed into a 1\-layer LeakyReLU network to be transformed into hidden states of 128\-dimension equally\. Then all the hidden states are concatenated to a long vector and fed into a 4\-layer MLP network to predictλ\\lambda\. We list the hyperparameters of the NN calibrator in[Table˜11](https://arxiv.org/html/2303.01421#A1.T11)\.

HyperparametersValuesLayers4Dimension of hidden state128Learning rate3e\-4OptimizerAdamActivation functionReLUDropout0\.2Table 11:Hyperparameters of the NN calibrator
### A\.4Inference

During inference, we feed context into the parameterized LM, its contextualized representation into the memory index, and its three types of features into the calibrator\. We search for top\-1K nearest neighbors from 32 nearest cluster centroids using the memory index\. The calibrator reweighs the distributions of the parameterized LM and the memory, and we use this calibrated distribution as the final output of our model\.

## Appendix BCL baselines

RecAdam\(Chenet al\.,[2020](https://arxiv.org/html/2303.01421#bib.bib19)\)As a regularization\-based method, RecAdam recalls previously acquired knowledge by retaining the pretraining object through frozen parameters, and it continually learns new information using a multi\-task learning object\. As the learning process moves forwards, the regularization is annealed to lessen the restriction\.

Mix\-Review\(Heet al\.,[2021b](https://arxiv.org/html/2303.01421#bib.bib40)\)Assuming that the pretraining corpus is obtainable, Mix\-Review uses an empirical decreasing function to adjust the quantity of the pretraining corpus mixed in the continued training data\. As the learning process moves forward, the quantity of the pretraining corpus tapers off to 0, resulting in the remaining training process being equivalent to fine\-tuning\.

Besides established CL baselines, those methods aimed at data efficiency were also considered to be adapted to CL settings, such as Greedy Merging which performs the best in datastore pruning inHeet al\.\([2021a](https://arxiv.org/html/2303.01421#bib.bib41)\)\. However, while Greedy Merging can be generalized for CL by pruning and merging memory greedily every certain number of steps iteratively, this approach presents disastrous distribution shifts\. If we merge new memory into the old, the new information distribution will continuously shift towards the old distribution, finally destroying the performance of new data; vice versa, the old distribution will shift towards the new one, causing the catastrophic forgetting problem\. It turns out in Table[5](https://arxiv.org/html/2303.01421#S4.T5)that Greedy Merging undergoes severe catastrophic forgetting in the first 3 months of NC\-20H1\.

We leave more dedicated and adapted CL approaches to be explored in the future\.

## Appendix CAnalysis

### C\.1Possible learning strategies for MoNIM

In addition to the intuitive cross\-entropy loss, there are reasonable methods to measure and control the learning process\. We also propose to rely on the intrinsic information content within the memory to assess the necessity of memorization\.

##### Internal information based

Since memorized keys can be projected to vocabulary space to analyze the information hidden in keys, we can calculate the internal distance from key to value, namely the KL\-divergence from key\-projected token distribution to the golden token distribution, which represents the amount of new information contained in the sample\.

M^Θ​\(𝐱\)=\{1ifDKL\(𝒚\|\|E𝐱<𝐭\)<δ0else\\hat\{M\}\_\{\\Theta\}\(\\boldsymbol\{\\mathrm\{x\}\}\)=\\begin\{cases\}1&\\text\{if\}~D\_\{\\mathrm\{KL\}\}\(\\boldsymbol\{y\}~\|\|~E\\boldsymbol\{\\mathrm\{x\_\{<t\}\}\}\)<\\delta\\\\ 0&\\text\{else\}\\end\{cases\}
The preliminary results \([Table˜12](https://arxiv.org/html/2303.01421#A3.T12)\) indicate the dominance of cross\-entropy loss over KL divergence of internal information\. Due to the resource limitation, we stick to the learning strategy using cross\-entropy loss in the main experiments throughout the rest of the paper\.

### C\.2PerformanceVSMemorization rate

We have confirmed that MoNIM can achieve performance comparable toFullMemwith a substantially reduced memorization rate whenδ=−1\.5\\delta=\-1\.5\. Intuitively, ifδ\\deltaincreases, more cases will be memorized and the performance will likely increase further; on the contrary, ifδ\\deltadecreases, more cases will be skipped, resulting in less memory but weaker performance\. Table[13](https://arxiv.org/html/2303.01421#A3.T13)confirms this intuition, demonstrating that we can obtain a trade\-off between scalability and performance through the manipulation ofδ\\delta\.

MethodsPPL \(↓\\downarrow\)MemRate \(↓\\downarrow\)Bare GPT\-2 small24\.10%\+MoNIM\(loss\)8\.650%\+MoNIM\(KL\)9\.554%

Table 12:Perplexity \(PPL\) on the test set of NC\-20H1 by different learning strategies for MoNIM over NC\-20H1\.MethodsPPL \(↓\\downarrow\)MemRate \(↓\\downarrow\)Bare GPT\-2 small24\.10%\+FullMem9\.0100%\+MoNIM \(δ=−1\.0\\delta=\-1\.0\)8\.254%\+MoNIM \(δ=−1\.5\\delta=\-1\.5\)8\.650%\+MoNIM \(δ=−2\.0\\delta=\-2\.0\)9\.945%

Table 13:MoNIM with different memorization thresholdδ\\delta\.MethodsFullMemRandMemMoNIMConstantλ\\lambda9\.015\.014\.3NN calibrator8\.3\(\-0\.7\)12\.5 \(\-2\.5\)8\.6 \(\-5\.7\)Table 14:Perplexity results on the NC\-20H1 test data with and without the NN calibrator for FullMem, RandMem, and MoNIM\.FeaturesPPL \(↓\\downarrow\)All8\.6\-Density features12\.0\-Distribution features9\.9\-Lexical features8\.9Table 15:The ablation study of features in the NN calibrator\.![Refer to caption](https://arxiv.org/html/2303.01421v2/x6.png)Figure 5:MoNIM’s memorization rate with CL over months\.
### C\.3NN calibrator

The NN calibrator plays an important role in calibrating the semiparametric LM’s prediction by adaptingλ\\lambdaat test time\. Although such an adaptive method has been proven universally effective in semiparametric LMs by previous work, we reveal in Table[14](https://arxiv.org/html/2303.01421#A3.T14)that it benefits MoNIM most, significantly more thanRandMemand conventionalFullMem, which only introduces marginal improvement as in previous work, demonstrating that calibration is more compatible with MoNIM\.

We ablate the features of the NN calibrator to study their effects on the results\. According to Table[15](https://arxiv.org/html/2303.01421#A3.T15), all our proposed features contribute positively to the calibrator, among which the density information, especially theL2L^\{2\}distance, is the most important one because it can directly reflect if the non\-parametric memory has much relevant information given a test case, providing the most straightforward evidence to the calibrator\.

Methods \(MemRate\)0\-shot1\-shot2\-shotBare GPT\-2 large \(0%\)29\.831\.532\.5\+FullMem\(100%\)36\.336\.937\.2\+RandMem\(60%\)30\.732\.133\.0\+MoNIM \(40%\)36\.336\.737\.7Table 16:In\-context learning accuracy of the GPT\-2 large with CL over NC\-20H1 on theRealTime QAin 0\-, 1\- and 2\-shot learning\.
### C\.4Adaptive memorization threshold \(AMT\)

We test MoNIM’s MemRate with/without AMT, since it serves as a measure of AMT’s performance in handling memory overfitting\. It reveals in[Figure˜5](https://arxiv.org/html/2303.01421#A3.F5)that after adding AMT the downward trend is largely enhanced, while the performance is nearly untouched despite the reduced memory\. We claim that the overfitting causes the predictions to favor the neighboring overconfident wrong answers, damaging the generalization of new data\. Thus, the alleviation of overfitting is helpful with respect to both performance and scalability\.

### C\.5In\-context learning

We test if the in\-context learning capability of a language model is affected by CL through MoNIM\. We present the in\-context learning result in theRealTime QAbenchmark that the memory can benefit in Table[16](https://arxiv.org/html/2303.01421#A3.T16), showing that MoNIM is not in conflict with in\-context learning and that the LM can still benefit from more examples shown in the context\.

Also, we evaluate MoNIM’s effect on in\-context learning for general tasks collected byShiet al\.\([2022](https://arxiv.org/html/2303.01421#bib.bib88)\)that cannot benefit from its memory\. According to Table[17](https://arxiv.org/html/2303.01421#A3.T17), despite no improvement observed, MoNIM does not affect the results in these tasks, demonstrating its robustness\.

Methods \(MemRate\)RTECBRTSST\-2CRMRHYPBare GPT\-2 large \(0%\)53\.139\.349\.551\.450\.550\.860\.0FullMem\(100%\)50\.335\.549\.049\.848\.647\.560\.0FullMem\(100%\) w/ NN calibrator52\.841\.149\.551\.850\.850\.960\.0MoNIM \(40%\)53\.141\.149\.551\.550\.850\.060\.0Table 17:0\-shot learning accuracy of the GPT\-2 large with CL over NC\-20H1 on general NLP tasks\.
### C\.6Time efficiency

We analyze the time consumption of MoNIM with vanilla kNN\-LMs\.

- •Index Building: In this process, the whole computational overhead is equal to conducting a full forward pass over the training data to extract representations as keys, which is the same as a vanilla kNN\-LM\.
- •Retrieval Process: We have measured that MoNIM’s inference time is approximately0\.90\.9–1\.1​𝚡1\.1\\mathtt\{x\}that of vanilla kNNs\. For time\-sensitive tasks, we have also explored a simple modification to accelerate inference without significantly hurting performance\. We introduce a confidence thresholdθ\\thetaafter the calibrator ofλ\\lambdain[Section˜3\.2](https://arxiv.org/html/2303.01421#S3.SS2), so that ifλ<θ\\lambda<\\theta, we simply skip the retrieval step and rely solely on the LM output\. This is intuitive since a lowλ\\lambdaindicates that memory contributes little useful information\. In our experiments with GPT\-2 small, whenθ=0\.3\\theta=0\.3, the inference latency is reduced to0\.8​𝚡0\.8\\mathtt\{x\}that of vanilla kNNs, while the PPL in[Table˜2](https://arxiv.org/html/2303.01421#S4.T2)increases slightly from 8\.6 to 8\.9, whileFullMemPPL is 9\.0\.