用于冻结语言模型无外部记忆对话回忆的自键控参数内记忆

# Self-Keyed In-Parameter Memorization for External Memory-FreeDialogue Recall in Frozen Language Models
Source: [https://arxiv.org/html/2610.36948](https://arxiv.org/html/2610.36948)
###### Abstract

A large language model maintains conversational content in an external storage that is outside the model and grows as the dialogue continues\. This work assumes an environment where a language model has no external memory and no prior knowledge of the future question or its answer\. The motivation is to research whether a dialogue can be stored in the model parameters and later answer a question about it\. This work proposes self\-keyed in\-parameter memorization \(Skim\), a closed\-form method in which the model generates its own questions about each dialogue turn and writes those questions into its parameters as the keys that later retrieve the turn’s content\. Generating many varied questions for each turn allows a later user question to match one even when phrased differently, and its answer is recalled from the parameters alone\. Evaluated on five language models,Skimrecalls most stored dialogue turns from later user questions, and it answers from the model parameters without any external storage\.

Department of Electrical and Computer Engineering, Korea University, Seoul, Republic of Korea

\{xypiao97, ryumh10, joongheon, jongkook\}@korea\.ac\.kr

## Introduction

Recent large language models \(LLMs\) serve as conversational assistants and their conversations often run for many exchanges\([Petroni et al\. 2019](https://arxiv.org/html/2610.36948#bib.bib22);[Roberts et al\. 2020](https://arxiv.org/html/2610.36948#bib.bib23);[De Cao et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib24);[Kwon et al\. 2023](https://arxiv.org/html/2610.36948#bib.bib7)\)\. As a conversation lengthens, answering well often depends on recalling something the user mentioned much earlier\([Xu et al\. 2022](https://arxiv.org/html/2610.36948#bib.bib26);[Packer et al\. 2023](https://arxiv.org/html/2610.36948#bib.bib27);[Wu et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib5)\)\. However, language models keep the dialogue not in their parameters but in a key\-value \(KV\) cache or an external retrieval store that grows as the history accumulates\. The KV cache stores the attention keys and values of every past token, sparing the model from recomputing them at each step\([Vaswani et al\. 2017](https://arxiv.org/html/2610.36948#bib.bib6);[Shazeer 2019](https://arxiv.org/html/2610.36948#bib.bib9);[Kwon et al\. 2023](https://arxiv.org/html/2610.36948#bib.bib7)\)\. Its size grows proportionately to the length of the history and the generated tokens accumulate in the KV cache\. Both memory use and inference cost rise as more turns arrive\([Pope et al\. 2022](https://arxiv.org/html/2610.36948#bib.bib28);[Dao et al\. 2022](https://arxiv.org/html/2610.36948#bib.bib8)\)\. Instead, retrieval\-augmented generation \(RAG\) stores the dialogue in an external index and searches it for relevant passages at each question, placing them in the context\([Lewis et al\. 2020](https://arxiv.org/html/2610.36948#bib.bib25);[Lewis et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib11)\)\. The answer then depends on whether the index holds the right passage and whether the question was turned into an effective search query\([Guu et al\. 2020](https://arxiv.org/html/2610.36948#bib.bib10);[Maharana et al\. 2024](https://arxiv.org/html/2610.36948#bib.bib4)\)\. Based on these observations, this work asks whether a frozen model can internalize a dialogue as it unfolds and later answer a question about it from its parameters alone\. The aim is to establish whether such internalization is feasible at all, rather than to improve on retrieval or long\-context systems that already answer from external memory\.

Prior studies aim to store a model’s memory in its parameters rather than in an external buffer\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2);[Wang et al\. 2024](https://arxiv.org/html/2610.36948#bib.bib15);[Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. Weight editing is one such approach that writes a fact directly into the parameters\([Mitchell et al\. 2022](https://arxiv.org/html/2610.36948#bib.bib12);[Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. Given a subject and a relation, such as a person and their birthplace, these methods compute a closed\-form update to one or a few feed\-forward projections that causes the model to output the intended object\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. A null\-space projection allows many such edits to accumulate without overwriting one another\([Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. However, this approach rests on two assumptions that conversational memory does not satisfy\. First, the object to store must be provided as a label because the update is solved to reproduce that object at the edited site\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. Because no such label exists in a dialogue, the model must itself determine which part of a raw user turn should be stored\. Second, a fixed cue that is typically the subject and relation must be known in advance when constructing the edit\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. In a dialogue, there is no fixed cue to install because the later question is unknown when the turn is stored and may be phrased in unforeseen ways\. Long\-context models and retrieval systems avoid both assumptions, but they retain the dialogue outside the parameters\([Lewis et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib11)\), in a cache or an index that grows as turns accumulate\([Maharana et al\. 2024](https://arxiv.org/html/2610.36948#bib.bib4);[Kwon et al\. 2023](https://arxiv.org/html/2610.36948#bib.bib7)\)\. Thus, editing requires supervision that a dialogue cannot provide, whereas external buffers avoid that supervision only by incurring an ever\-growing storage cost\([Kwon et al\. 2023](https://arxiv.org/html/2610.36948#bib.bib7);[Maharana et al\. 2024](https://arxiv.org/html/2610.36948#bib.bib4)\)\.

Prior work established that the feed\-forward layers of a transformer store factual associations and recall them at inference, and that a band of middle layers is where such facts are most readily located and edited\([Geva et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib13);[Dai et al\. 2022](https://arxiv.org/html/2610.36948#bib.bib14);[Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Geva et al\. 2023](https://arxiv.org/html/2610.36948#bib.bib29)\)\. Building on that finding, this work writes each user turn into the down\-projection of one mid\-to\-late feed\-forward layer by a closed\-form rule\. The target layer is chosen per model and a later question is answered from the edited parameters alone\. This work assumes an environment where a frozen model has no external memory and no access to the future question or its answer when it stores a turn, which makes a direct comparison to other editing methods difficult\. The setting imposes four constraints, namely no gradient fine\-tuning, no external memory at recall, a per\-turn cost independent of the number of stored turns, and preserved general behavior\. This work forgoes the KV cache despite its usefulness and keeps the dialogue in the parameters\([Vaswani et al\. 2017](https://arxiv.org/html/2610.36948#bib.bib6);[Shazeer 2019](https://arxiv.org/html/2610.36948#bib.bib9)\)\. Because the model has no future question or answer, it must produce from the turn alone the object and cue that weight editing receives but a dialogue cannot provide\.

This work proposes a Self\-Keyed In\-parameter Memorization approach, calledSkim, that allows a frozen model to store a turn and later recall it without the ground\-truth answer or the future question\. The proposedSkimoperates on spans, each a short contiguous piece of a turn’s text\.Skimproceeds in three stages, namely \(i\) extracting candidate spans from the raw turn, \(ii\) generating for each span several questions that the span answers and keeping the ones that best separate that span from its rivals in the same turn, and \(iii\) installing every question–span pair into one projection by a joint closed\-form edit\. The generated questions take the place of the fixed subject–relation input that prior editing must be given in advance\. Because many diverse questions serve as keys for each span, a later user question resembles one of them even when phrased differently, and the question itself selects the right span at recall\. The main contributions of this work are summarized as follows\.

- •This work aims to retain a dialogue and answer a later question about it in an environment without external memory, neither a KV cache nor a retrieval store\.
- •This work proposesSkim, a closed\-form method that internalizes each turn into the model parameters without the ground\-truth answer or the future question, using self\-generated questions as keys for its stored spans\.
- •This work shows thatSkimstores a dialogue as its turns arrive one at a time and reaches the recall of a single batch edit without the forgetting that sequential edits cause\.
- •Evaluated on five language models,Skimrecalls stored dialogue turns at a five\-model mean of 63\.1% and 75\.0% on the strongest model, well above the no\-cache floor that removes all memory\.

## Problem Setting and Preliminaries

#### Assumptions\.

Current language models keep their parameters fixed and hold each conversation in an external memory that grows as the dialogue lengthens\([Lewis et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib11);[Maharana et al\. 2024](https://arxiv.org/html/2610.36948#bib.bib4)\)\. This paper asks whether a frozen model can hold the conversation in its parameters instead of an external memory, and answer later questions from them\. The assumed scenario is as follows\. A stream of user turnsu1,…,uNu\_\{1\},\\dots,u\_\{N\}arrives, and eachuiu\_\{i\}states a fact that some later questionqiq\_\{i\}asks about, whose answer isaia\_\{i\}\. The language model stores each turn in its parameters as it arrives, and it is more constrained than the external store it replaces\. A cache or a retrieval index maintains a turn in full and delays every decision until the query arrives\. Instead, the model must determine from the turn itself what to store\. It is given neither the later questionqiq\_\{i\}that would identify the relevant content nor the answeraia\_\{i\}that would specify the target\. The phrasing of the later question may also be unanticipated\. When the later question arrives, the model receives it as the only input and must generate the answer\. No session text, cache, or retrieval index remains available\. Four constraints hold throughout\. Storing a turn uses no gradient fine\-tuning, its cost does not grow as more turns are stored, recall is a single forward pass, and the edit must leave the general behavior of the model intact\.

![Refer to caption](https://arxiv.org/html/2610.36948v1/fig1.png)Figure 1:Overview ofSkim\. From an earlier turn the model extracts candidate spans as values and self\-generates questions as keys\. Both steps are answer\-free, and a closed\-form edit installs these pairs into one frozen feed\-forward layer\. A later question phrased differently forms a key that retrieves the value, and the model answers from its parameters\.
#### Preliminaries\.

The transformer feed\-forward down\-projectionWWin a blockW​σ​\(Wup​x\)W\\,\\sigma\(W\_\{\\mathrm\{up\}\}x\)behaves as a key\-value memory, where the hidden activationk=σ⁡\(Wup​x\)k=\\sigma\(W\_\{\\mathrm\{up\}\}x\)serves as an address andW​kWkis the content read from it\. A single factual association is one entry of this memory, and editingWWoverwrites the value retrieved by a given key reads\([Geva et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib13);[Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1)\)\. To storennassociations, their keys form the columns of a matrixKKand their values the columns of a matrixDD\. The editΔ​W\\Delta Wthat reproduces all of them minimizes a regularized least\-squares objective,

‖Δ​W​K−D‖F2\+λ​‖Δ​W‖F2,\\displaystyle\\\|\\Delta WK\-D\\\|\_\{F\}^\{2\}\+\\lambda\\\|\\Delta W\\\|\_\{F\}^\{2\},\(1\)whereλ\>0\\lambda\>0sets the regularization strength\. Setting its gradient to zero gives the normal equationsΔ​W​\(K​K⊤\+λ​I\)=D​K⊤\\Delta W\(KK^\{\\top\}\+\\lambda I\)=DK^\{\\top\}, whose closed\-form ridge solution\([Hoerl and Kennard 1970](https://arxiv.org/html/2610.36948#bib.bib21)\)is

Δ​W=D​K⊤​\(K​K⊤\+λ​I\)−1=D​\(K⊤​K\+λ​I\)−1​K⊤\.\\displaystyle\\begin\{split\}\\Delta W&=DK^\{\\top\}\(KK^\{\\top\}\+\\lambda I\)^\{\-1\}\\\\\[3\.0pt\] &=D\\,\(K^\{\\top\}K\+\\lambda I\)^\{\-1\}K^\{\\top\}\.\\end\{split\}\(2\)The second form inverts ann×nn\\times nmatrix, small when few turns are stored\. The edit reproduces each valuedjd\_\{j\}at its keykjk\_\{j\}up to the regularizer\. A largerλ\\lambdakeeps the edit small and favors preservation of the base weights, and a smaller one favors exact reproduction\. The update stores allnnvalues in superposition and recovers each as long as the keys remain near\-orthogonal, which bounds the number of associations a layer holds before interference\. The solution requires no gradient descent and underlies the weight\-editing family\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. The objective does not specify the origin ofKKandDD\. Prior editing methods are given a ground\-truth value and a fixed key, whereas a dialogue provides neither of these\. The method of the next section constructs both from the turn itself\.

## Proposed Method

As shown in Figure[1](https://arxiv.org/html/2610.36948#Sx2.F1),Skimextracts candidate spans as the values to store, generates questions about each span as the keys that address them, and installs the pairs into one feed\-forward layer by the closed\-form joint edit\. The first two stages use the frozen model only to generate and rank candidates, and only the third stage changes a parameter\. At recall, the model answers the user question from its parameters\.

### Self\-Keyed Construction

The frozen language model produces short answer phrases when prompted to list the specific facts the turnuustates\.Skimalso takes the turn’s quoted titles and capitalized entity phrases directly from its text\. These candidates are filtered to short spans of a few words rather than the whole utterance and form the candidate value set𝒮⁡\(u\)\\mathcal\{S\}\(u\)\. The model then ranks each spans∈𝒮⁡\(u\)s\\in\\mathcal\{S\}\(u\)answer\-free by

ρ⁡\(s\)=𝔼q∼Q0​\(s\)​\[log⁡pM​\(s∣q,u\)\]\+β​π​\(s\),\\rho\(s\)\\;=\\;\\mathbb\{E\}\_\{q\\sim Q\_\{0\}\(s\)\}\\\!\\big\[\\log p\_\{M\}\(s\\mid q,u\)\\big\]\\;\+\\;\\beta\\,\\pi\(s\),\(3\)whereQ0​\(s\)Q\_\{0\}\(s\)is a set of self\-questions the model generates forss, andpM​\(x∣y\)p\_\{M\}\(x\\mid y\)is the probability the modelMMassigns to a spanxxas the continuation of a promptyy\. The first term rewards a span the model can recover from its own questions about it, which is what later makes the span addressable\. The second term is the specificity priorπ⁡\(s\)\\pi\(s\)that equals the span’s negative log\-probability given a neutral prompt and rewards a rare span that a later question is more likely to ask about\. The coefficientβ\\betabalances recoverability against specificity\. When a turn is stored, the model does not yet know which span the later question will ask about, and the single highest\-ρ\\rhospan is often not that one\. Keeping top\-kkspans as candidate values therefore defers the choice to recall time, where the later question retrieves the span that matches it\.

Each span also needs keys that address it for a later question phrased differently at recall, and these keys are questions the model generates about the span itself\. For each spanss, the model is prompted inkqk\_\{q\}fixed styles to generate a question whose answer isss, producing candidatesq∼G\(⋅∣s,u\)q\\sim G\(\\cdot\\mid s,u\)\. Each candidate is scored by a discriminative margin

μ⁡\(q,s\)=log⁡pM​\(s∣q,u\)−maxs′∈𝒮⁡\(u\)∖\{s\}​log​pM​\(s′∣q,u\)\.\\displaystyle\\begin\{split\}\\mu\(q,s\)=&\\log p\_\{M\}\(s\\mid q,u\)\\\\\[3\.0pt\] &\-\\max\_\{s^\{\\prime\}\\in\\mathcal\{S\}\(u\)\\setminus\\\{s\\\}\}\\log p\_\{M\}\(s^\{\\prime\}\\mid q,u\)\.\\end\{split\}\(4\)A high margin marks a question that points tossand not to any rival, which makes the key discriminative and limits the chance that a question about one span retrieves another at recall\. Each span keeps its topmmquestions byμ\\muafter rival surface forms are removed from their wording\. The survivors are then expanded by deterministic string variants of the question text\. For the span "Serenity Yoga", a surviving question is "Which studio do I practice at?"\. The question\-word exchange swaps its "Which" for "What", and re\-framing wraps it as "Earlier, the answer to ‘Which studio do I practice at?’ was"\. The resulting key setRsR\_\{s\}replaces the fixed subject–relation cue of prior editing, and its diversity lets a differently\-phrased later question resemble one of the stored keys\.

### Joint Edit in One Feed\-Forward Layer

Let\(rj,sj\)\(r\_\{j\},s\_\{j\}\)enumerate the surviving question–span pairs of every turn accumulated to this point\. Each questionrjr\_\{j\}is rendered in a fixed recall frameframe⁡\(⋅\)\\mathrm\{frame\}\(\\cdot\)that presents it to the model as a prompt\. The keykjk\_\{j\}is the hidden state that enters the down\-projection of layerℓ\\ellat the answer position\([Geva et al\. 2021](https://arxiv.org/html/2610.36948#bib.bib13)\),

kj=h\(ℓ\)​\(frame⁡\(rj\)\),k\_\{j\}\\;=\\;h^\{\(\\ell\)\}\\\!\\big\(\\mathrm\{frame\}\(r\_\{j\}\)\\big\),\(5\)whereh\(ℓ\)​\(y\)h^\{\(\\ell\)\}\(y\)denotes that hidden state for a promptyy\. Because this activation also entersWWat recall, the storage key equals the recall key in the same frame\. The valuedjd\_\{j\}is a residual output delta that makes the span the frame’s continuation, scaled by the strengthff,

dj=f​δj,δj=arg⁡minδ−log⁡pM⁡\[δ\]​\(sj∣frame⁡\(rj\)\),\\displaystyle\\begin\{split\}d\_\{j\}&=f\\,\\delta\_\{j\},\\\\\[3\.0pt\] \\delta\_\{j\}=\\arg\\min\_\{\\delta\}\-&\\log p\_\{M\[\\delta\]\}\\\!\\big\(s\_\{j\}\\mid\\mathrm\{frame\}\(r\_\{j\}\)\\big\),\\end\{split\}\(6\)whereM⁡\[δ\]M\[\\delta\]addsδ\\deltato the layer output at the answer position\. The objective has no closed form because the span probability is nonlinear inδ\\delta\. A short gradient optimization findsδ\\deltawhile the weights stay frozen\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1)\)\. The strengthffscales the stored value\. A largerffstrengthens recall but pushes the residual stream farther from its base state\. Recall and preserved behavior are therefore in tension\. The keyskjk\_\{j\}are the columns of a matrixKKand the valuesdjd\_\{j\}are the columns of a matrixDD\. The closed\-form solve of Eq\. \([2](https://arxiv.org/html/2610.36948#Sx2.E2)\)\([Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)fits all accumulated pairs jointly rather than one at a time and returns the single matrixΔ​W\\Delta Wthat at once satisfies

Δ​W​kj≈djfor all​j\.\\Delta W\\,k\_\{j\}\\approx d\_\{j\}\\quad\\text\{for all \}j\.\(7\)The ridge termλ​I\\lambda Ikeeps the inverse well conditioned when two keys are close and makes each match approximate rather than exact\. Adding thisΔ​W\\Delta Wto the down\-projectionWWof layerℓ\\ellinstalls all pairs in one matrix\. For an inputhhthe edited layer computes

\(W\+Δ​W\)​h=W​h\+Δ​W​h\.\(W\+\\Delta W\)\\,h=Wh\+\\Delta Wh\.\(8\)For an input unrelated to any stored key the extra termΔ​W​h\\Delta Whstays small and the base behavior of the layer is preserved\. A later question that forms a stored keykjk\_\{j\}makes the extra term contributedjd\_\{j\}to the residual stream\. The optimizeddjd\_\{j\}raises the span probability and the model emits that span\. The edit modifies a single mid\-to\-late down\-projection that serves as the value memory of the feed\-forward block\.

### Recall and Self\-Probed Selection

At recall, the user questionqqrendered in the frame that built the keys lands on a stored key and retrieves its value\. The edited modelMΔ​WM\_\{\\Delta W\}then generates the answera^\\hat\{a\}by decoding a short budget of tokens\.

a^=decode⁡\(MΔ​W,frame⁡\(q\)\)\.\\hat\{a\}=\\mathrm\{decode\}\\big\(M\_\{\\Delta W\},\\,\\mathrm\{frame\}\(q\)\\big\)\.\(9\)MΔ​WM\_\{\\Delta W\}is the model after the editΔ​W\\Delta W, andframe⁡\(q\)\\mathrm\{frame\}\(q\)is the user question in the recall frame\. The decode preset is chosen per model and is either greedy decoding that suppresses repetition or a small beam that blocks repeatednn\-grams\. It stops generation at an answer\-completion criterion or a sentence\-final token and makes the answer emerge cleanly\.

The same pipeline runs for every model and only a few per\-model settings vary\. They form the configuration

θ=\(ℓ,f,decode preset,key budget\)\.\\theta\\;=\\;\(\\ell,\\ f,\\ \\text\{decode preset\},\\ \\text\{key budget\}\)\.\(10\)These settings are fixed when the turn is stored and cannot be tuned to the later question\. A self\-probe selects them answer\-free by standing in for the absent question\. It probes each item by an unused paraphrase of the item’s own self\-generated question and records whether the stored span is emitted cleanly\. Clean emission means the generation contains the span and avoids third\-order token repetition and keeps lexical diversity above0\.50\.5\. The clean fraction of thennstored items is

afclean​\(θ\)=1n​∑i=1n𝟙​\[item​i​emits its span cleanly\]\.\\mathrm\{af\}\_\{\\mathrm\{clean\}\}\(\\theta\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathds\{1\}\\big\[\\text\{item \}i\\text\{ emits its span cleanly\}\\big\]\.\(11\)This fraction needs no ground\-truth answer or later question\. The chosen configuration maximizes it\.

θ⋆=arg⁡maxθ​afclean​\(θ\)\.\\theta^\{\\star\}\\;=\\;\\arg\\max\_\{\\theta\}\\ \\mathrm\{af\}\_\{\\mathrm\{clean\}\}\(\\theta\)\.\(12\)The ranking is reliable because the probe repeats the recall operation itself on a stand\-in and accesses no ground\-truth answer\. A configuration that degenerates or never retrieves the span fails the clean test and true recall alike\. The clean fraction therefore tracks true recall and the method tunes each model from these self\-probes alone\.

![Refer to caption](https://arxiv.org/html/2610.36948v1/fig2a.png)\(a\)
![Refer to caption](https://arxiv.org/html/2610.36948v1/fig2b.png)\(b\)
![Refer to caption](https://arxiv.org/html/2610.36948v1/fig2c.png)\(c\)

Figure 2:Recall \(%\) for five language models\. Panel \(a\) reports recall atN=32N\{=\}32in five conditions from ceiling to floor, the KV cache upper bound, the answer\-given references true\-Q and self\-Q, the answer\-free SKIM, and a no\-cache floor that removes all memory\. Panels \(b\) and \(c\) trace recall and the count of unrecalled facts asNNgrows from44to3232\.
### Mitigating Forgetting from Sequential Edits

A dialogue’s turns arrive one at a time and each is written into the parameters as it arrives before any later turn is available\. Such a memory is useful only if two demands are met\. The first demand is that a new write leave the already stored turns intact because a growing sequence of edits can otherwise interfere and erase them\. The second is that the memory the edit occupies stay fixed in size however long the dialogue runs which a growing cache cannot offer\. Meeting both while turns still arrive is not automatic because an edit that keeps growing as new turns come can disturb the turns already stored\. The reference point is a batch edit that solves Eq\. \([2](https://arxiv.org/html/2610.36948#Sx2.E2)\) once on the complete set of pairs\([Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)and is the strongest result the closed\-form rule can reach\.Skimreproduces this sameΔ​W\\Delta Wturn by turn even as turns still arrive\. The turns arrive in chunks and the stored pairs accumulate into two growing matrices\.

K≤t=\[K1,…,Kt\],D≤t=\[D1,…,Dt\]\.\\begin\{split\}K\_\{\\leq t\}=\[K\_\{1\},\\dots,K\_\{t\}\],\\\\\[3\.0pt\] D\_\{\\leq t\}=\[D\_\{1\},\\dots,D\_\{t\}\]\.\\end\{split\}\(13\)At each chunk,Skimdiscards the previous edit and re\-solves from the original weights on every pair to this point\.

Δ​W\(t\)=D≤t​\(K≤t⊤​K≤t\+λ​I\)−1​K≤t⊤\.\\Delta W^\{\(t\)\}=D\_\{\\leq t\}\\big\(K\_\{\\leq t\}^\{\\top\}K\_\{\\leq t\}\+\\lambda I\\big\)^\{\-1\}K\_\{\\leq t\}^\{\\top\}\.\(14\)This reads onlyK≤tK\_\{\\leq t\}andD≤tD\_\{\\leq t\}and any permutationPPof the columns leaves it unchanged because the two copies ofPPcancel\.

Arrival order and grouping therefore have no effect and the final edit equals the batch edit\. A simpler additive scheme

Δ​W←Δ​W\+Dt​\(Kt⊤​Kt\+λ​I\)−1​Kt⊤\\Delta W\\leftarrow\\Delta W\+D\_\{t\}\(K\_\{t\}^\{\\top\}K\_\{t\}\+\\lambda I\)^\{\-1\}K\_\{t\}^\{\\top\}\(16\)never inverts the joint matrixK≤t⊤​K≤tK\_\{\\leq t\}^\{\\top\}K\_\{\\leq t\}and lets earlier turns drift\([Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)whereas re\-solving from the original weights avoids the drift\. The stored result is one fixed\-size matrixΔ​W\\Delta Wand a question is a single forward pass and both stay constant however many turns are stored unlike a growing cache\. The re\-solve grows as turns accumulate but runs offline once per chunk and never enlarges the store or slows the answer\.

## Evaluation and Analysis

### Experimental Setup

#### Benchmarks\.

This work uses LongMemEval\([Wu et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib5)\), a recall benchmark on long user\-assistant dialogues whose questions ask about content the user stated many turns earlier\. The evaluation is restricted to the single\-session\-user subset, where each question is paired to the one earlier turn that holds its answer\. From each retained question the answer\-bearing user turn is taken as the fact to store, and numeric\-answer questions are removed which leaves 33 non\-numeric facts\.NNof these turns are written into the parameters and queried by the original held\-out question, and recall is reported asNNgrows from44to3232\. The single\-session subset supplies only 33 such facts, andN=32N\{=\}32therefore stores nearly the entire available pool, which places the study at the benchmark’s ceiling rather than at a truncated sample\. Recall is measured at every step fromN=4N\{=\}4toN=32N\{=\}32\. This shows how much the stored turns interfere asNNincreases\. A generation counts as correct when the normalized ground\-truth answer is contained in it and when a bidirectional DeBERTa\-MNLI gate\([He et al\. 2020](https://arxiv.org/html/2610.36948#bib.bib30);[Williams et al\. 2017](https://arxiv.org/html/2610.36948#bib.bib31)\)judges it to entail the ground\-truth answer\.

Table 1:Configuration for each language model\. Here,ℓ\\ellis the edited feed\-forward layer,ffthe first\-token write strength,λ\\lambdathe ridge coefficient, and Self\-Q the anchored\-question countkqk\_\{q\}and kept margin countmm\. Decode is a 4\-beam search blocking repeated 4\-grams or greedy decoding\.
#### Models\.

This work uses five different models for evaluation; Qwen3 \(14B\)\([Team 2025](https://arxiv.org/html/2610.36948#bib.bib16)\), Mistral\-Nemo \(12B\)\([Mistral AI and NVIDIA 2024](https://arxiv.org/html/2610.36948#bib.bib17)\), Yi\-1\.5 \(9B\)\([01\.AI 2024](https://arxiv.org/html/2610.36948#bib.bib18)\), Llama\-3 \(8B\)\([AI@Meta 2024](https://arxiv.org/html/2610.36948#bib.bib19)\), and Mistral \(7B\)\([Mistral AI 2024](https://arxiv.org/html/2610.36948#bib.bib20)\)\. The five come from four independently developed families, and recall on all of them tests whether the mechanism is general rather than tuned to one model\. All are widely used open\-weight instruction\-tuned models, which matches the dialogue setting the method targets and keeps the evaluation reproducible\. Each model’s configuration is listed in Table[1](https://arxiv.org/html/2610.36948#Sx4.T1)\. Every run uses one A100 80GB GPU\. The forward pass runs in BF16 and each closed\-form solve is carried out in FP32\.

### Recall Attribution

Figure[2\(a\)](https://arxiv.org/html/2610.36948#Sx3.F2.sf1)comparesSkimto four reference conditions atN=32N\{=\}32for five different models\. The ‘KV cache’ shows recall when all models use the KV cache\. ‘True\-Q’ shows recall without the KV cache when the true answer and the true question are given\. ‘Self\-Q’ shows recall the same as True\-Q without the KV cache and giving the true answer, but using the questions generated by the model\. The ‘no\-cache’ shows recall when all models do not use the KV cache\. Here, the models that do not use the KV cache \(‘no\-cache’\) show recall of 3\.1 to 15\.6%, but this is difficult to regard as recall from the past dialogue because the model answers some questions by prior knowledge or by chance even without the turn\. The KV cache and the True\-Q condition show recall of about 90 to 100%, Self\-Q shows recall of 80% or more, andSkimshows recall of about 46\.9 to 75\.0%\. Because the value to store is given as the ground\-truth answer, True\-Q and Self\-Q show that the parameters can store and address a memory as well as the external memory\. When the true question is swapped for the self\-questions at a fixed answer, recall drops by about 6\.88% on average but still averages about 91\.3%\. This indicates that the model can store and recall correctly even when its self\-generated questions are used as the keys\. Replacing the answer by self\-extracted spans lowers recall by about 26\.9% on average for all models, but recall still averages about 63\.1%\.

Figure[2\(b\)](https://arxiv.org/html/2610.36948#Sx3.F2.sf2)shows the recall ofSkimfor each model asNNgrows from44to3232\. Every model starts near 100% at smallNNand declines gradually\. Qwen3 shows the highest recall \(75\.0%\) and Mistral the lowest \(46\.9%\) atN=32N\{=\}32\. Figure[2\(c\)](https://arxiv.org/html/2610.36948#Sx3.F2.sf3)shows the count of unrecalled facts for each model asNNincreases from44to3232, and Qwen3 fails to recall 8 of 32 facts and Mistral 17 atN=32N\{=\}32\. Both the recall and the unrecalled\-fact count change gradually and show no sudden step at any size\. This gradual degradation indicates that recall declines by a small increment as each additional turn is stored, rather than failing abruptly onceNNexceeds a threshold\. Even atN=32N\{=\}32,Skimrecalls well above the no\-memory floor \(‘no\-cache’\) and still remains effective as more turns are stored\.

![Refer to caption](https://arxiv.org/html/2610.36948v1/fig3.png)Figure 3:Per\-fact outcome atN=32N\{=\}32for five language models\. Each model’s 32 stored facts split into correct and four failure classes, the store\-time pair missing and selection and the recall\-time pair routing and decoding\.
### Which Stage Fails

For the five models, Figure[3](https://arxiv.org/html/2610.36948#Sx4.F3)shows the stage ofSkimat which the 32 facts most often fail\. A fact is ‘missing’ when the true answer never enters the extracted span candidates, ‘selection’ when a same\-turn rival span is written in its place, ‘routing’ when the user question retrieves a different fact, and ‘decoding’ when the correct value is addressed but the generation breaks down\. Here, ‘missing’ and ‘selection’ are store\-time failures, and ‘routing’ and ‘decoding’ are recall\-time failures\. The five models produce 59 failures; 22 from span\-extraction \(missing\), 22 from selection, 0 from routing, and 15 from decoding\. Missing and selection account for 44 of the 59 failures, about three quarters of the total\. Most failures therefore occur at store\-time rather than at recall\. Decoding accounts for 15 failures and forms the recall\-time remainder\. Routing is zero for every model, which indicates that the closed\-form keys reach the intended fact and that read\-time addressing is never where recall is lost\.

The store\-time failures split evenly overall, 22 missing and 22 selection in the five\-model total\. Which mode dominates separates the models into two cases\. Selection dominates on Mistral\-Nemo and Yi\-1\.5, which extract the correct span for 29 and 30 of the 32 facts and then rank a same\-turn rival above it\. Yi\-1\.5 is the sharpest case at 2 missing and 7 selection\. Missing dominates on the other three models, which more often fail to extract the correct span at all\. Llama\-3 is the sharpest case at 7 missing and 3 selection\. Thirteen types of intervention do not change the selection outcome and none corrects the selected value, whether they reshape the key geometry, enlarge the key budget, re\-adapt the layer and regularizer, or widen the beam and align the stop criterion\. Because routing is zero and the keys already reach the intended fact, these interventions cannot correct a store\-time failure\. Therefore, the limiting problem is the store\-time extraction and selection of the span rather than any weakness in addressing or generation\.

![Refer to caption](https://arxiv.org/html/2610.36948v1/fig4a.png)\(a\)
![Refer to caption](https://arxiv.org/html/2610.36948v1/fig4b.png)\(b\)

Figure 4:Panel \(a\) reports recall by write position, each model’s facts split into early, middle, and recent groups\. Panel \(b\) reports recall by answer type, proper\-noun versus descriptive multi\-word, pooled and within each group\.![Refer to caption](https://arxiv.org/html/2610.36948v1/fig5.png)Figure 5:A comparison of streaming recall and batch recall for five models, ranging fromN=4N\{=\}4toN=32N\{=\}32\. Every point lies on the identity line because a streamed write reproduces the batch solve without forgetting\.
### Failure by Position and Answer Type

Figure[4](https://arxiv.org/html/2610.36948#Sx4.F4)reports the sameN=32N\{=\}32outcomes two ways to separate two candidate axes of failure, the write position of a fact and the linguistic type of its answer\. Figure[4\(a\)](https://arxiv.org/html/2610.36948#Sx4.F4.sf1)groups the 32 facts by write position into an early group of 11, a middle group of 10, and a recent group of 11, and measures recall for each model in each group\. In all five models, recall on the recent group is lower than on the early and middle groups\. This apparent reduction in recall does not necessarily indicate that recent turns are forgotten\. Every turn is written at the same strengthffdescribed in Eq\.[6](https://arxiv.org/html/2610.36948#Sx3.E6), and the edit therefore treats every write position equally\. The lower recall on the recent group instead follows from the answer type carried by those turns, which varies from turn to turn\.

Figure[4\(b\)](https://arxiv.org/html/2610.36948#Sx4.F4.sf2)reports recall by answer type, and the split is sharp\. Proper\-noun answers are recalled at 81\.1% while descriptive multi\-word answers such as "a lighter shade of gray" reach only 36\.9%, a difference of about 44 points that largely holds within every position group\. The recent group appears weaker in Figure[4\(a\)](https://arxiv.org/html/2610.36948#Sx4.F4.sf1)only because it happens to carry more descriptive facts, and the apparent position effect is really an answer\-type effect\. A proper noun is a discrete high\-salience unit that the model extracts and keys cleanly\. A descriptive span instead has soft boundaries, runs to several tokens, and sits beside same\-turn rivals\. Its failure is therefore one of self\-extraction and multi\-token reproduction rather than of storage or routing\. The hardest facts combine a descriptive answer and a same\-turn rival span\. The question\-generation stage exists for exactly this structure and concentrates its per\-fact gains there by recovering "Golden Retriever" rather than the rival "Max"\. The descriptive\-answer weakness is thus the clearest lever for improving self\-supervision\. It corroborates the earlier finding that self\-extraction sets the recall ceiling rather than storage or addressing\.

### Streaming Matches Batch

Figure[5](https://arxiv.org/html/2610.36948#Sx4.F5)plots streaming recall versus batch recall for all five models fromN=4N\{=\}4toN=32N\{=\}32\. Every point lies on the identity line, and a turn\-at\-a\-time write reaches the recall of a single batch write at every size\. The match is exact because the two writes produce the same edit\. Each streaming step discards the previous edit and re\-solves Eq\. \([14](https://arxiv.org/html/2610.36948#Sx3.E14)\) from the frozen base weights on all pairs collected to that point, and this solve reads only the accumulated Gram matricesK≤t⊤​K≤tK\_\{\\leq t\}^\{\\top\}K\_\{\\leq t\}andD≤t​K≤t⊤D\_\{\\leq t\}K\_\{\\leq t\}^\{\\top\}\. These are order\-invariant sums, and a permutation of the arrival order permutes the summands but not the sum, which makes the streamed edit the same matrix as the batch edit\. The figure confirms in measurement the permutation identity that Eq\. \([14](https://arxiv.org/html/2610.36948#Sx3.E14)\) forces in derivation\.

The property depends on re\-solving from the base weights each chunk rather than stacking each edit onto the previous one, an additive scheme that allows earlier turns to drift as later ones are written\([Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. A streamed dialogue therefore incurs no forgetting, and an earlier turn is recalled no worse once later turns are written on top of it\. This is the position\-invariance of Figure[4](https://arxiv.org/html/2610.36948#Sx4.F4)\(a\) read from the write side, and the two jointly show that arrival order changes neither the stored content nor the recalled content\. A growing cache secures this same no\-forgetting property only by keeping every turn, whereas the parametric store secures it at a size that stays fixed as the dialogue lengthens\. The re\-solve is the one cost that grows, and it runs offline once per chunk\.

### Discussion

The main difficulty is the storage of a turn rather than its recall\. Recall of a stored value is near\-perfect once that value is written, and Figure[3](https://arxiv.org/html/2610.36948#Sx4.F3)shows zero routing failures\. The difficulty is extracting from a turn which content to store without the future question or the answer\. The value harvest scores each candidate span by a self\-question and a specificity prior and keeps the top few\. This fails most on descriptive multi\-token spans beside same\-turn rivals\. The other difficulty is the key rather than the value\.Skimkeys each span by questions it writes before the real question arrives\. A later question phrased in an unforeseen way leaves its key unformed and its value unrecalled\. A separate limitation is the numeric answer\. A date or a count carries little of the specificity the harvest rewards and is rarely chosen for storage\. A recovered number also seldom reproduces the exact digits the containment metric checks for\. Moreover, prior weight\-editing methods operate in a different environment from the one studied here\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2);[Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. Running one on a dialogue turn would supply the ground\-truth answer and question that this environment does not provide, turning it into an answer\-given method rather than a comparable baseline\. The answer\-given conditions of Figure[2\(a\)](https://arxiv.org/html/2610.36948#Sx3.F2.sf1)already serve as that method \(see ‘True\-Q’\), and they mark the recall the correct answer would reach\.

## Related Work

Prior training\-free methods modify model behavior or stored knowledge without gradient\-based retraining\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. Closed\-form weight editing directly modifies selected parameters to store factual associations\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. ROME rewrites one subject\-indexed factual association using a rank\-one update to a single feed forward layer\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1)\)\. MEMIT extends ROME to many predefined factual associations by distributing the required updates to several feed forward layers, enabling the simultaneous editing of thousands of associations\([Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2)\)\. AlphaEdit constrains each edit to the null space of the preserved knowledge’s keys, which does not affect that knowledge and reduces interference during sequential edits\([Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. These methods require predefined edit targets and desired factual outputs\([Meng et al\. 2022a](https://arxiv.org/html/2610.36948#bib.bib1);[Meng et al\. 2022b](https://arxiv.org/html/2610.36948#bib.bib2);[Fang et al\. 2025](https://arxiv.org/html/2610.36948#bib.bib3)\)\. The proposed method constructs both the stored value and its key from the turn itself, taking neither a predefined edit target nor a desired output, and recalls the selected facts in response to later queries phrased differently\.

## Conclusion

This paper studied whether a language model can internalize real dialogue turns into its parameters by one closed\-form edit\. The study was conducted in an assumed environment that provides no ground\-truth answer and no later question and no external memory, and in a scenario where the dialogue arrives one turn at a time\. The proposed self\-keyed in\-parameter memorization \(Skim\) extracts what to store from the turn and attaches self\-generated questions as its keys\.Skimrecalls a five\-model mean of 63\.1% and 75\.0% on the strongest model at 32 stored turns, well above a no\-cache floor of 3\.1 to 15\.6%\. The method also shows that streaming the dialogue reaches the same recall performance as storing all turns at once, which means the forgetting caused by sequential storage can be mitigated\. These experimental results show that the proposedSkimstores conversation content in a fixed\-size model parameter without any external memory\.

## References

- 01\.AI \(2024\)01\.AIYi\-1\.5\-9B\-Chat\.Hugging Face\.Note:Hugging Face model repositoryChat\-tuned checkpointExternal Links:[Link](https://huggingface.co/01-ai/Yi-1.5-9B-Chat)Cited by:[Models\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px2.p1.1)\.
- AI@Meta \(2024\)AI@MetaLlama 3 Model Card\.Note:https://github\.com/meta\-llama/llama3/blob/main/MODEL\_CARD\.mdAccessed: 2026\-07\-22Cited by:[Models\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Daiet al\.\(2022\)D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. WeiKnowledge neurons in pretrained transformers\.External Links:2104\.08696,[Link](https://arxiv.org/abs/2104.08696)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p3.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. RéFlashAttention: fast and memory\-efficient exact attention with io\-awareness\.External Links:2205\.14135,[Link](https://arxiv.org/abs/2205.14135)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- De Caoet al\.\(2021\)N\. De Cao, W\. Aziz, and I\. TitovEditing factual knowledge in language models\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 6491–6506\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Fanget al\.\(2025\)J\. Fang, H\. Jiang, K\. Wang, Y\. Ma, J\. Shi, X\. Wang, X\. He, and T\. ChuaAlphaedit: null\-space constrained knowledge editing for language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 16366–16396\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1),[Mitigating Forgetting from Sequential Edits](https://arxiv.org/html/2610.36948#Sx3.SSx4.p1.5),[Streaming Matches Batch](https://arxiv.org/html/2610.36948#Sx4.SSx5.p2.1),[Discussion](https://arxiv.org/html/2610.36948#Sx4.SSx6.p1.1),[Related Work](https://arxiv.org/html/2610.36948#Sx5.p1.1)\.
- Gevaet al\.\(2023\)M\. Geva, J\. Bastings, K\. Filippova, and A\. GlobersonDissecting recall of factual associations in auto\-regressive language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12216–12235\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p3.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer feed\-forward layers are key\-value memories\.External Links:2012\.14913,[Link](https://arxiv.org/abs/2012.14913)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p3.1),[Preliminaries\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px2.p1.1),[Joint Edit in One Feed\-Forward Layer](https://arxiv.org/html/2610.36948#Sx3.SSx2.p1.1)\.
- Guuet al\.\(2020\)K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. ChangREALM: retrieval\-augmented language model pre\-training\.External Links:2002\.08909,[Link](https://arxiv.org/abs/2002.08909)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Heet al\.\(2020\)P\. He, X\. Liu, J\. Gao, and W\. ChenDeberta: decoding\-enhanced bert with disentangled attention\.arXiv preprint arXiv:2006\.03654\.Cited by:[Benchmarks\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Hoerl and Kennard \(1970\)A\. E\. Hoerl and R\. W\. KennardRidge regression: biased estimation for nonorthogonal problems\.Technometrics12\(1\),pp\. 55–67\.Cited by:[Preliminaries\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px2.p1.2)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1),[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Lewiset al\.\(2021\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive nlp tasks\.External Links:2005\.11401,[Link](https://arxiv.org/abs/2005.11401)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1),[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1),[Assumptions\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px1.p1.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating very long\-term conversational memory of llm agents\.External Links:2402\.17753,[Link](https://arxiv.org/abs/2402.17753)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1),[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1),[Assumptions\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px1.p1.1)\.
- Menget al\.\(2022a\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in gpt\.Advances in neural information processing systems35,pp\. 17359–17372\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1),[Introduction](https://arxiv.org/html/2610.36948#Sx1.p3.1),[Preliminaries\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px2.p1.1),[Preliminaries\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px2.p1.3),[Joint Edit in One Feed\-Forward Layer](https://arxiv.org/html/2610.36948#Sx3.SSx2.p1.3),[Discussion](https://arxiv.org/html/2610.36948#Sx4.SSx6.p1.1),[Related Work](https://arxiv.org/html/2610.36948#Sx5.p1.1)\.
- Menget al\.\(2022b\)K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. BauMass\-editing memory in a transformer\.arXiv preprint arXiv:2210\.07229\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1),[Preliminaries\.](https://arxiv.org/html/2610.36948#Sx2.SS0.SSS0.Px2.p1.3),[Joint Edit in One Feed\-Forward Layer](https://arxiv.org/html/2610.36948#Sx3.SSx2.p1.3),[Mitigating Forgetting from Sequential Edits](https://arxiv.org/html/2610.36948#Sx3.SSx4.p1.1),[Discussion](https://arxiv.org/html/2610.36948#Sx4.SSx6.p1.1),[Related Work](https://arxiv.org/html/2610.36948#Sx5.p1.1)\.
- Mistral AI and NVIDIA \(2024\)Mistral AI and NVIDIAMistral\-Nemo\-Instruct\-2407\.Hugging Face\.Note:Hugging Face model repositoryInstruction\-tuned checkpointExternal Links:[Link](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)Cited by:[Models\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Mistral AI \(2024\)Mistral AIMistral\-7B\-Instruct\-v0\.3\.Hugging Face\.Note:Hugging Face model repositoryInstruction\-tuned checkpointExternal Links:[Link](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)Cited by:[Models\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Mitchellet al\.\(2022\)E\. Mitchell, C\. Lin, A\. Bosselut, C\. Finn, and C\. D\. ManningFast model editing at scale\.External Links:2110\.11309,[Link](https://arxiv.org/abs/2110.11309)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1)\.
- Packeret al\.\(2023\)C\. Packer, V\. Fang, S\. Patil, K\. Lin, S\. Wooders, and J\. GonzalezMemGPT: towards llms as operating systems\.\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Petroniet al\.\(2019\)F\. Petroni, T\. Rocktäschel, P\. Lewis, A\. Bakhtin, Y\. Wu, A\. H\. Miller, and S\. RiedelLanguage models as knowledge bases?\.arXiv preprint arXiv:1909\.01066\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Popeet al\.\(2022\)R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, A\. Levskaya, J\. Heek, K\. Xiao, S\. Agrawal, and J\. DeanEfficiently scaling transformer inference\.arXiv preprint arXiv:2211\.05102\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Robertset al\.\(2020\)A\. Roberts, C\. Raffel, and N\. ShazeerHow much knowledge can you pack into the parameters of a language model?\.InProceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\),pp\. 5418–5426\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.
- Shazeer \(2019\)N\. ShazeerFast transformer decoding: one write\-head is all you need\.External Links:1911\.02150,[Link](https://arxiv.org/abs/1911.02150)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1),[Introduction](https://arxiv.org/html/2610.36948#Sx1.p3.1)\.
- Team \(2025\)Q\. TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Models\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1),[Introduction](https://arxiv.org/html/2610.36948#Sx1.p3.1)\.
- Wanget al\.\(2024\)Y\. Wang, Y\. Gao, X\. Chen, H\. Jiang, S\. Li, J\. Yang, Q\. Yin, Z\. Li, X\. Li, B\. Yin, J\. Shang, and J\. McAuleyMEMORYLLM: towards self\-updatable large language models\.External Links:2402\.04624,[Link](https://arxiv.org/abs/2402.04624)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p2.1)\.
- Williamset al\.\(2017\)A\. Williams, N\. Nangia, and S\. R\. BowmanA broad\-coverage challenge corpus for sentence understanding through inference\.arXiv preprint arXiv:1704\.05426\.Cited by:[Benchmarks\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Wuet al\.\(2025\)D\. Wu, H\. Wang, W\. Yu, Y\. Zhang, K\. Chang, and D\. YuLongMemEval: benchmarking chat assistants on long\-term interactive memory\.External Links:2410\.10813,[Link](https://arxiv.org/abs/2410.10813)Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1),[Benchmarks\.](https://arxiv.org/html/2610.36948#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Xuet al\.\(2022\)J\. Xu, A\. Szlam, and J\. WestonBeyond goldfish memory: long\-term open\-domain conversation\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 5180–5197\.Cited by:[Introduction](https://arxiv.org/html/2610.36948#Sx1.p1.1)\.