评估压缩记忆：依赖性、效用与内容收益

# Evaluating Compressed Memory:Dependence, Utility, and Content Benefit
Source: [https://arxiv.org/html/2610.19373](https://arxiv.org/html/2610.19373)
Minsu OhAffiliation:Department of Artificial Intelligence, Jeju National UniversityEmail:[ycyoon@jejunu\.ac\.kr](mailto:)Chanjun ParkAffiliation:School of Software, Soongsil UniversityYeo\-Chan Yoon††thanks:Corresponding authors: Yeo\-Chan Yoon and Chanjun Park\.Affiliation:Department of Artificial Intelligence, Jeju National University

###### Abstract

Does a language model answer better because its compressed memory contains useful document information? Replacing memory vectors with zeros can lower a trained model’s score without showing that it outperforms a model trained without memory\. We combine zeroing with separately trained models that cannot access stored text or receive the same constant input for every document\. Across nine systems, some models rely on memory yet perform worse than the trained alternative\. For example, in experiments that predict states in generated stories, zeroing memory worsens a model’s prediction loss, yet a model trained without memory achieves lower loss than the original model with intact memory\. On TriviaQA, giving xRAG memory from another question’s passage raises F1, which measures word overlap with accepted answers, from 19\.38 without a document to 24\.78\. Its effect on how often outputs contain an accepted answer is statistically uncertain\. The question’s retrieved passage scores higher still \(29\.88 F1\)\. Thus a gain over no document alone does not establish that the gain requires the question’s passage\. Evaluations must therefore specify the comparison model, absolute performance, and generation and scoring conditions\.

## 1Introduction

Compressed memory encodes text as vectors that a large language model \(LLM\) can use in place of the original text\. Evaluating memory therefore requires evidence that the stored information improves prediction\. A lower score after removing memory, or a higher score than a model given no document, can appear to provide that evidence\. We show that neither comparison alone establishes a benefit from the stored content\. A model can rely on memory yet predict worse than a model trained without it; a module trained to supply the same input for every document can also improve scores\.

The problem is a mismatch between the comparison performed and the conclusion drawn\. Removing memory after training asks whether the model relies on an input it learned to use\. It does not ask whether training with memory produced a better predictor than training without it\. Even that second comparison leaves another question: does the advantage require document content, or can a module trained without access to that content also improve performance? Treating these questions as interchangeable leaves the value of stored information untested\.

Figure 1:Three comparisons for interpreting a memory gain\. Zeroing memory tests whether an already trained model relies on it\. Separately trained models without memory or with constant input test whether memory and document content improve performance\.These distinctions build on established comparisons with retrained models and controls for learned representations\([Hooker et al\., 2019](https://arxiv.org/html/2610.19373#bib.bib21);[Hewitt and Liang, 2019](https://arxiv.org/html/2610.19373#bib.bib26)\)\. We bring the comparisons together to test what memory evaluations establish \(Figure[1](https://arxiv.org/html/2610.19373#S1.F1)\)\. We measure dependence by zeroing memory in an already trained model, and utility by comparing against a model trained without the tested memory\. To measure content benefit, we train the same memory module with document vectors or with one constant input shared by every document\. The constant\-input module can learn to help the LLM, but cannot convey the current document’s facts\. Our contribution is an empirical account of where these comparisons lead to different conclusions and which additional evidence supports a claim about document content\.

1. 1\.We demonstrate that zeroing damage does not establish a training benefit\. Across nine systems, four are harmed by zeroing memory yet perform worse than their trained alternatives on generated\-story state prediction\. These cases have low task performance; they do not establish the same disagreement in models that solve the task well\. A separate controlled study produces a positive memory advantage when both models worsen, because the model without memory worsens more\. Together, these results show why zeroing damage and a relative gain must be checked against trained alternatives and absolute performance \(Section[4](https://arxiv.org/html/2610.19373#S4)\)\.
2. 2\.We show that zeroing damage can include a benefit learned without document content\. On questions about scientific papers, zeroing the output of a module trained on constant input lowers F1 by 5\.73 points\. F1 measures word overlap with an accepted answer, accounting for missing and extra words\. This drop occurs although the module receives no facts from the current document\. Training on document vectors also outperforms training on constant input, providing separate evidence that document content helps\. These results distinguish the benefit of a learned input from the additional benefit of document content in a working question\-answering system \(Section[5](https://arxiv.org/html/2610.19373#S5)\)\.
3. 3\.We establish that a gain over no document need not be specific to the question’s retrieved passage\. Using xRAG, which maps a document embedding to one LLM input vector\([Cheng et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib20)\), we find that passages from other questions improve some scores over no document on three datasets\. The question’s retrieved passage performs better\. Comparing only against no document would miss that some improvement also occurs with a different passage\. We also show that generation settings and scoring rules change the conclusions supported by these comparisons \(Section[5\.2](https://arxiv.org/html/2610.19373#S5.SS2)\)\.

These findings make memory evaluation a question of matching evidence to a claim: reliance requires an intervention, utility requires a trained comparison, and document\-content benefit requires a comparison that withholds document information\. Reporting them together makes clear what a memory gain does, and does not, demonstrate\.

## 2Related Work

### 2\.1Interpreting Performance Gains

[O’Connor and Andreas \(2021\)](https://arxiv.org/html/2610.19373#bib.bib23)compare changing evaluation context with training on altered contexts\. Our memory audit applies this distinction by pairing zeroing after training with a separately trained model without memory\. Conditional probing tests what a model’s internal text vectors add to a prediction after accounting for information already available to the predictor\([Hewitt et al\., 2021](https://arxiv.org/html/2610.19373#bib.bib36)\)\. For example, do contextual word vectors help predict a word’s part of speech beyond knowing which word it is? Our constant\-input comparison asks a related question: does document input improve performance beyond a trained input that contains no document\-specific information?

The controls are established\. Probe control tasks help distinguish representation properties from a probe’s ability to learn labels\([Hewitt and Liang, 2019](https://arxiv.org/html/2610.19373#bib.bib26)\)\. Remove and Retrain \(ROAR\) tests feature importance by retraining after removal\([Hooker et al\., 2019](https://arxiv.org/html/2610.19373#bib.bib21)\)\. Learned prompts supply trainable inputs without document content\([Li and Liang, 2021](https://arxiv.org/html/2610.19373#bib.bib27);[Lester et al\., 2021](https://arxiv.org/html/2610.19373#bib.bib28)\), and optimal ablation learns a constant replacement\([Li and Janson, 2024](https://arxiv.org/html/2610.19373#bib.bib30)\)\. Our constant\-input projector produces one learned vector\. Activation patching replaces internal model values with values computed from a different input to test which components affect a prediction\.[Zhang and Nanda \(2024\)](https://arxiv.org/html/2610.19373#bib.bib10)show that how the input is altered and which evaluation metric is used can change which components appear important\. Our memory audit likewise distinguishes zeroing from replacement with another document’s vectors and reports the scoring rule for each comparison\.

We study compressed memory by bringing these controls together in state prediction, word prediction, and question answering\. Our contribution is to test whether conclusions about memory change when reliance on an already trained input is compared with the advantage over a separately trained model\. We also compare modules trained with document vectors or constant input to determine whether a gain requires document content\. This connects established control methods to the specific claims made about compressed memory: that a model uses it, benefits from training with it, or benefits from its stored content\.

### 2\.2Evaluating Compressed Memory

Trained comparison models provide precedents for our distinction between memory dependence and utility\. Gist, which compresses task instructions, compares models trained with full or hidden instructions\([Mu et al\., 2023](https://arxiv.org/html/2610.19373#bib.bib4), Section 4\.2\)\. AutoCompressors, which summarize earlier text into vectors, use fine\-tuned language models as baselines\([Chevalier et al\., 2023](https://arxiv.org/html/2610.19373#bib.bib5), Section 4\.1\)\. We build on such comparisons by testing whether damage from zeroing memory agrees with the advantage over a model trained without it\. To examine this question across memory designs, our experiments include the In\-context Autoencoder \(ICAE\)\([Ge et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib6)\), Compressed Context Memory \(CCM\)\([Kim et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib16)\), Activation Beacon\([Zhang et al\., 2025](https://arxiv.org/html/2610.19373#bib.bib17)\), and MemoryLLM\([Wang et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib18)\), which retain earlier text in compressed form\. Each is evaluated against its stated trained alternative \(Section[4](https://arxiv.org/html/2610.19373#S4)\)\.

Our question\-answering \(QA\) studies separate gains from additional training from gains that require document content\. xRAG includes a low\-rank adaptation \(LoRA\) control for additional training data\([Cheng et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib20), Section 6\.2\); LoRA trains low\-rank weight updates\([Hu et al\., 2022](https://arxiv.org/html/2610.19373#bib.bib8)\)\. COCOM trains a model without documents and studies training the answer\-generating LLM, or*decoder*, together with the compressor\([Rau et al\., 2025](https://arxiv.org/html/2610.19373#bib.bib35)\)\. LLoCO trains decoder LoRA while keeping compressed document memory fixed\([Tan et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib9)\)\. Our decoder study keeps document memory fixed and uses equal training budgets for models trained on correct documents, other documents, or no documents\. Both models trained with document memory receive the same correct\-document memory at evaluation\. Their difference therefore tests the benefit of learning from correct document content under the stated training procedure \(Section[5](https://arxiv.org/html/2610.19373#S5)\)\.

Information\-preservation studies ask what can be recovered from compression: reconstruction\([Kuratov et al\., 2025](https://arxiv.org/html/2610.19373#bib.bib37)\), grounding and recall\([Łajewska et al\., 2025](https://arxiv.org/html/2610.19373#bib.bib31);[Deng et al\., 2025](https://arxiv.org/html/2610.19373#bib.bib32)\), or answers lost after compression\([Belikova et al\., 2026](https://arxiv.org/html/2610.19373#bib.bib38)\)\. Our question is whether the stored content improves prediction beyond a model trained without that content\. CCM already examines learning that bypasses context during compression training\([Kim et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib16)\)\. We test for gains of this kind by training a memory module with one constant input shared by all documents\. Comparing it with the module trained on document vectors separates a gain achievable without current\-document facts from an additional gain with those facts\.

The choice of evaluation input and score also affects what a memory gain establishes\. Random passages can improve retrieval\-augmented generation\([Cuconasu et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib29)\), so an advantage over no document alone need not be specific to the retrieved passage\. We examine this issue in compressed memory by comparing the question’s retrieved passage, another question’s passage, and no document, then testing whether conclusions change with generation settings and scoring rules \(Section[5\.2](https://arxiv.org/html/2610.19373#S5.SS2)\)\. BenchPress compares compressed scores with reference scores\([Feldman and Artzi, 2026](https://arxiv.org/html/2610.19373#bib.bib22)\)\. Our analysis of scores normalized relative to models with no document or with evidence text shows why such normalization cannot by itself establish document use: a model trained on constant input can also receive a positive normalized score \(Appendix[G](https://arxiv.org/html/2610.19373#A7)\)\.

## 3Comparisons for Memory Evaluation

### 3\.1Three Questions, Three Comparisons

An*intact*model receives its usual compressed memory\. We evaluate all conditions on the same items\. A*reference*is the named comparison condition or model, not necessarily with the same training history\.

#### Dependence: intervene after training\.

We zero memory values with model weights fixed\. A performance drop shows dependence on those values\. The model did not train without memory, so this test cannot establish superiority to one that was\.

#### Utility: train without memory\.

Comparison modelrrlearns the same task without the tested memory\. A*local adapter*adds trainable computation without memory access\. A*closed\-book*QA model receives the question without a document\. A*suffix\-only*model receives recent tokens without earlier text\. We call the memory model’s advantage over the specified model its utility,UrU\_\{r\}\(orUU\)\. It depends on both models’ training and selection\.

#### Content benefit: train with constant input\.

The*constant\-input*model receives the same input, computed from training data, for every document\. We train the same memory module in both conditions, but constant input cannot convey the current document’s facts\. The document model’s advantage is content benefit,CC\. Equal architectures need not have equal input\-dependent capabilities; this difference does not isolate a unique fraction of the gain caused by information\.

Negative log\-likelihood \(NLL\) measures how little probability a model assigns to the correct next token\. Lower is better; we use nats\. The differences are

D\\displaystyle D=NLLzero−NLLintact,\\displaystyle=\\mathrm\{NLL\}\_\{\\rm zero\}\-\\mathrm\{NLL\}\_\{\\rm intact\},\(1\)Ur\\displaystyle U\_\{r\}=NLLr−NLLintact,\\displaystyle=\\mathrm\{NLL\}\_\{r\}\-\\mathrm\{NLL\}\_\{\\rm intact\},\(2\)C\\displaystyle C=NLLconstant−NLLintact\.\\displaystyle=\\mathrm\{NLL\}\_\{\\rm constant\}\-\\mathrm\{NLL\}\_\{\\rm intact\}\.\(3\)F1 measures word overlap between a generated answer and an accepted answer, accounting for both missing and extra words\. For a higher\-is\-better scoreSS, such as F1 or accuracy, they areD=Sintact−SzeroD=S\_\{\\rm intact\}\-S\_\{\\rm zero\},Ur=Sintact−SrU\_\{r\}=S\_\{\\rm intact\}\-S\_\{r\}, andC=Sintact−SconstantC=S\_\{\\rm intact\}\-S\_\{\\rm constant\}\. Positive values favor intact memory over the named reference\. ThusD\>0D\>0andU<0U<0mean zeroing hurts, yet a trained alternative does better\.

### 3\.2What Is Held Fixed

Our PlanGen test model compares a memory module with a trainable local adapter that uses only recent text\. The memory module summarizes consecutive token groups into memory vectors, available after a delay\. The LLM combines recent\-token states with either the mean of available vectors \(*pooled*memory\) or one selected vector \(*indexed*memory\)\. A local adapter matches the parameter budget but receives no memory vectors \(Appendix[A](https://arxiv.org/html/2610.19373#A1)\)\.

The projector\-training studies for QA and word prediction use xRAG’s single memory vector\. The SFR\-Embedding\-Mistral encoder \(SFR\) maps text to a fixed\-length vector\([Salesforce, 2024](https://arxiv.org/html/2610.19373#bib.bib25)\)\. A trainable*projector*transforms this vector into an LLM input\. Each condition trains its own copy from the same projector weights, keeping the encoder and LLM frozen\. The projector receives document content, locally visible text, or a constant\. This preserves its architecture, but not its information access or inference cost\.

#### A shared zero\-input predictor\.

In this frozen\-LLM interface, zeroing projector output leaves the same LLM and prompt in every condition\. Let their common zero\-input score beS0S\_\{0\}, and writeDconst=Sconst−S0D\_\{\\rm const\}=S\_\{\\rm const\}\-S\_\{0\}\. Then

Ddoc=Sdoc−S0=C\+Dconst\.D\_\{\\rm doc\}=S\_\{\\rm doc\}\-S\_\{0\}=C\+D\_\{\\rm const\}\.\(4\)Thus the performance lost by zeroing document memory includes the benefit of a learned constant input\. The identity holds for NLL with reversed signs and paired averages\. It does not provide another test of document use\. In PlanGen, separately trained parameters, including LoRA weights, remain after zeroing, so predictors can differ\. Likewise,D−U=Slocal−SzeroD\-U=S\_\{\\rm local\}\-S\_\{\\rm zero\}compares the trained local model with the zeroed model; it does not separate causal mechanisms\.

### 3\.3Content Checks and Uncertainty

A*memory swap*replaces a document’s memory with another document’s memory\.*Evidence masking*hides text that supports the answer;*sham masking*hides equally long text at a similar position outside the designated evidence\. These tests change content after training \(Appendices[A](https://arxiv.org/html/2610.19373#A1)and[G](https://arxiv.org/html/2610.19373#A7)\)\. Their effects depend on the intervention: zeroing PlanGen memory retains projection bias, and reordering vectors leaves their pooled mean unchanged\.

We report each model’s absolute score alongside its advantage over the named comparison\. In the decoder\-training study, we keep the precomputed document memory fixed and train LLMs with correct documents, other documents, or no documents\.

Uncertainty intervals retain the pairing between conditions\. Earlier analyses average training seeds per item and resample documents or books while holding the trained models fixed\.*Crossed*intervals additionally resample training\-order seeds\. Generation analyses resample passage groups, joining groups that share a generation batch\. Unless otherwise stated, intervals use bootstrap percentiles 2\.5 and 97\.5\.*Simultaneous*intervals adjust for multiple comparisons in a specified family\. An interval above zero supports a positive difference; an interval containing zero does not establish absence of an effect\. Numerical benchmarks describe magnitudes, not validated thresholds for equivalence \(Appendix[H](https://arxiv.org/html/2610.19373#A8)\)\.

## 4Dependence and Utility

#### Comparing nine systems\.

Does damage from zeroing memory identify models that benefit from training with memory? Table[1](https://arxiv.org/html/2610.19373#S4.T1)compares zeroing damage,DD, with the advantage over a trained model without that memory,UU\. The tasks predict state values assigned earlier in generated stories and words repeated in the PG\-19 book corpus\([Rae et al\., 2020](https://arxiv.org/html/2610.19373#bib.bib3)\)\. The comparison models receive only recent text\. Alongside the compressors described above, we test StreamingLLM\([Xiao et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib2)\)and H2O\([Zhang et al\., 2023](https://arxiv.org/html/2610.19373#bib.bib14)\), which retain selected attention key/value \(KV\) states from earlier tokens\. We also test the convolutional compressor LoCoCo\([Cai et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib15)\)and M\+, an extension of MemoryLLM\([Wang et al\., 2025](https://arxiv.org/html/2610.19373#bib.bib19)\)\. Each system uses its stated adaptation procedure \(Appendix[O](https://arxiv.org/html/2610.19373#A15)\)\.

On generated stories, CCM, ICAE, Beacon, M\+ and MemoryLLM have low prediction losses: 0\.029–0\.078 NLL, compared with 1\.872–2\.244 for their local baselines\. CCM, Beacon and MemoryLLM also outperform their baselines on PG\-19\. Four other systems are harmed by zeroing on the synthetic task yet perform worse than their trained baselines\. These comparisons concern specific training procedures; they do not rank architectures\.

Table 1:Advantage over a trained restricted\-input model \(UU\) and zeroing damage \(DD\), in nats; three seeds per row\. Positive values favor intact memory\. Study\-specific simultaneous 95% intervals exclude zero for every sign \(Table[19](https://arxiv.org/html/2610.19373#A7.T19)\)\. Synthetic KV budgets: 24 updates with memory versus183\+64\+512183\+64\+512for the restricted\-input model\. KV policies share weights trained with full context; the last four compressors remain frozen during adaptation\. Access and optimization differ across rows \(Appendix[G](https://arxiv.org/html/2610.19373#A7)\)\.On generated stories, hiding answer\-supporting text hurts H2O and xRAG more than hiding matched non\-evidence text\. The differences are\+0\.405​\[0\.353,0\.460\]\+0\.405\\;\[0\.353,0\.460\]and\+1\.268​\[1\.216,1\.321\]\+1\.268\\;\[1\.216,1\.321\]nats under their respective simultaneous intervals\. Yet H2O has NLL 2\.068 versus its baseline’s 1\.849, and xRAG has 3\.211 versus 2\.279\. Sensitivity to relevant text therefore does not guarantee an advantage over a separately trained model\. These interventions differ from training a constant\-input model to measureCC\. This xRAG result concerns state prediction and a suffix\-only baseline; the positive QA result below uses a different task and question\-input baseline\.

### 4\.1Synthetic Boundary Cases

PlanGen restricts attention to recent text, so only memory can provide the earlier state assignment\. Given preceding gold story text, it predicts the token distinguishing the required state value\. We use six Pythia and Llama base models\([Biderman et al\., 2023](https://arxiv.org/html/2610.19373#bib.bib7);[Grattafiori et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib11);[Meta, 2024](https://arxiv.org/html/2610.19373#bib.bib12)\), four conditions, and three seeds, giving 72 trained models\. All six pooled\-memory configurations haveD\>0,U<0D\>0,U<0; for Llama\-3\.2\-3B,D=0\.4643D=0\.4643andU=−0\.1417U=\-0\.1417nats\. Yet accuracy on 512 rechecked targets is only 14\.78% with prediction restricted to candidate values or over the full vocabulary\. Predicting from training frequencies gives 18\.55%\. This disagreement occurs in a model that predicts these states unreliably\. A further comparison with shared initial weights and equal development\-search opportunities does not establish bothD\>0D\>0andU<0U<0\(Appendix[G](https://arxiv.org/html/2610.19373#A7)\)\.

A separate study asks whether concentrating training on the answer improves this result\. The local adapter, pooled memory, and constant\-input model share ten paired training orders and equal development\-search budgets\.*All\-token*training computes loss throughout the story;*target\-only*training computes it only at designated answer tokens, with the same preceding story as input\. Held\-out answer\-token NLL rises from 2\.319 to 3\.273 for the adapter, 2\.319 to 2\.492 for pooled memory, and 2\.311 to 2\.457 for constant input \(Table[2](https://arxiv.org/html/2610.19373#S4.T2)\)\.

Table 2:Held\-out answer\-token NLL in nats \(lower is better\), averaged over ten paired training orders\. With target\-only training, all three models have higher loss, but the no\-memory model worsens more\. Thus a relative memory advantage can emerge without lower prediction loss\. Appendix[L](https://arxiv.org/html/2610.19373#A12)retains accuracy, stopping records and uncertainty estimates\.With all\-token training, pooled memory’s advantage over the adapter is uncertain\. Under target\-only training, it is positive:\+0\.7808​\[0\.3611,1\.2187\]\+0\.7808\\;\[0\.3611,1\.2187\]nats\. Constant input also beats the adapter by\+0\.8162​\[0\.3619,1\.2961\]\+0\.8162\\;\[0\.3619,1\.2961\], while the pooled model’s advantage over constant input is unresolved:−0\.0354​\[−0\.1418,0\.0590\]\-0\.0354\\;\[\-0\.1418,0\.0590\]\. These crossed, adjusted intervals show an advantage over a baseline that deteriorates more\. All target\-only losses exceed the 1\.8871 NLL of a predictor uniform over eligible candidates, and 43/60 models reach the 1,024\-update cap without convergence\. Positive utility does not establish reliable state prediction or superiority to constant input \(Appendix[L](https://arxiv.org/html/2610.19373#A12)\)\. Appendix[B](https://arxiv.org/html/2610.19373#A2)retains the separate earlier experiment\.

## 5What a QA Score Gain Establishes

A higher score with memory need not mean that the improvement requires the question’s document\. We test this with trained controls on QASPER and with input changes to the released xRAG model on three QA datasets\.

### 5\.1Trained Controls on QASPER

QASPER supplies questions about scientific papers and annotated supporting passages\([Dasigi et al\., 2021](https://arxiv.org/html/2610.19373#bib.bib13)\)\. We encode these passages directly, excluding retrieval errors\. On 979 test questions, a projector trained on document input scores 19\.98 F1, versus 12\.54 for the same architecture trained on one constant input\. Zeroing either projector leaves the same frozen LLM and prompt, scoring 6\.82 \(Table[3](https://arxiv.org/html/2610.19373#S5.T3)\)\. Thus zeroing document memory loses 13\.16 points, of which 5\.73 can also be obtained by a trained input containing no current\-document facts\. The document\-versus\-constant advantage remains positive:\+7\.43​\[6\.15,8\.75\]\+7\.43\\;\[6\.15,8\.75\]points\. Document content helps, but zeroing damage includes more than that content benefit \(Equation[4](https://arxiv.org/html/2610.19373#S3.E4)\)\.

Table 3:QASPER F1 on 979 questions, averaged over ten paired training orders\. Zeroing the projector output leaves the same frozen LLM and prompt in both rows\. Scores fall even when the learned input conveys no current\-document facts\. Appendix[R](https://arxiv.org/html/2610.19373#A18)retains the full results\.These models share initialization and ten paired training orders\. An expanded learning\-rate search retains 92% of the document\-versus\-constant development NLL gap \(0\.334 to 0\.308; Appendix[P](https://arxiv.org/html/2610.19373#A16)\)\. A separate study trains only decoder LoRA with fixed memory and equal budgets\. Training on correct documents gives 21\.21 F1, versus 13\.73 after training on other documents and 13\.27 after training without documents\. Both document\-trained models receive identical correct memory at evaluation, so this comparison tests training history, not the effect of replacing a document at inference\. Appendices[C](https://arxiv.org/html/2610.19373#A3),[R](https://arxiv.org/html/2610.19373#A18)and[N](https://arxiv.org/html/2610.19373#A14)retain absolute scores, selection rules and adjusted intervals\.

On 405 CoQA first\-turn questions\([Reddy et al\., 2019](https://arxiv.org/html/2610.19373#bib.bib39)\), correct\-document training gives 66\.56 F1, versus 34\.22 for other\-document training and 18\.86 for training without documents\. This short\-evidence check supports content benefit; upstream xRAG training includes CoQA, so it does not establish unseen\-task transfer \(Appendix[M](https://arxiv.org/html/2610.19373#A13)\)\.

### 5\.2Gains That Do Not Require the Retrieved Passage

Here we keep the released xRAG\-7B weights fixed and compare three inputs: the question’s retrieved passage, a passage assigned from another question \(a*donor*\), and no document\. This asks whether a gain over no document is specific to the retrieved passage\. Neither retrieval nor donor selection guarantees that a passage contains, or lacks, the answer\.

We distinguish three scoring rules\.*Match*accepts a normalized answer anywhere in the output\.*Exact match*\(EM\) requires the full normalized output to equal an accepted answer\. F1 measures word overlap, penalizing missing and extra words\. Extra text can therefore preserve Match while lowering EM or F1\. We fixed the software versions, batch size of four, and primary metric, Match, before generating the main outputs; generation details and sensitivity checks are in Appendices[E](https://arxiv.org/html/2610.19373#A5)and[K](https://arxiv.org/html/2610.19373#A11)\.

Table 4:Released xRAG on 11,313 TriviaQA questions with fixed software and batch size four\. Higher is better\. Document encodes the retrieved passage; full text supplies it as text\. Closed book omits the vector; question encodes the question instead\. Donors average three assignments of other passages \(Appendix[K](https://arxiv.org/html/2610.19373#A11)\)\.Table 5:Score differences with fixed software and batch size four, in percentage or F1 points\. Donors are other questions’ passages; closed book omits documents\. Retrieved passages improve Match over donors in every dataset; donor gains over no document vary\. Intervals adjust for fifteen TriviaQA or twelve combined NQ\-open/WebQuestions comparisons, not a joint family across datasets\.#### A gain over no document is not all passage\-specific\.

On 11,313 TriviaQA questions\([Joshi et al\., 2017](https://arxiv.org/html/2610.19373#bib.bib33)\), we use the repository’s ColBERTv2\-retrieved passages\([Santhanam et al\., 2022](https://arxiv.org/html/2610.19373#bib.bib34)\)and three fixed donor assignments\. F1 is 19\.38 without a document, 24\.78 with donors, and 29\.88 with the retrieved passage \(Table[4](https://arxiv.org/html/2610.19373#S5.T4)\)\. The two improvements answer different questions: donors beat no document by 5\.40 points, and the retrieved passage beats donors by a further 5\.10\. The first improvement does not require the question’s retrieved passage; the second supports an advantage for that passage over the assigned alternatives\. Neither contrast identifies a causal fraction of information recovered\.

The scoring rule changes the conclusion about donors\. Their EM gain is\+3\.80​\[3\.22,4\.42\]\+3\.80\\;\[3\.22,4\.42\]points, but their Match difference is uncertain:−0\.57​\[−1\.50,0\.37\]\-0\.57\\;\[\-1\.50,0\.37\]\. The retrieved passage beats donors by\+9\.28​\[8\.26,10\.31\]\+9\.28\\;\[8\.26,10\.31\]Match points\. These intervals adjust across fifteen comparisons\. Thus higher word overlap or exact match need not establish that more answers contain an accepted answer\.

#### The pattern extends, but not every metric agrees\.

We repeat the fixed\-model comparison on 3,610 NQ\-open and 2,032 WebQuestions examples, using their cached DPR passages\([Kwiatkowski et al\., 2019](https://arxiv.org/html/2610.19373#bib.bib40);[Lee et al\., 2019](https://arxiv.org/html/2610.19373#bib.bib42);[Berant et al\., 2013](https://arxiv.org/html/2610.19373#bib.bib41);[Karpukhin et al\., 2020](https://arxiv.org/html/2610.19373#bib.bib43)\)\. Donor F1 gains over no document are positive on all three datasets, and the retrieved passage beats donors on Match throughout \(Table[5](https://arxiv.org/html/2610.19373#S5.T5)\)\. Donor Match is uncertain on NQ\-open but positive on WebQuestions; donor EM gains on these two datasets are near zero\. The additional datasets use a separate twelve\-comparison adjustment \(Appendix[Q](https://arxiv.org/html/2610.19373#A17)\)\.

#### Training changes sensitivity within QASPER\.

An earlier fixed\-memory study permits a direct check with questions, memory bytes, donor assignments, prompt and generation rule held fixed\. Before decoder training, correct and other\-paper memory score 14\.21 and 10\.90 F1; after document training, they score 21\.29 and 7\.15\. Their gap grows from 3\.31 to 14\.14 points, an increase of10\.83​\[9\.07,12\.59\]10\.83\\;\[9\.07,12\.59\]under the original simultaneous interval over training orders and papers\. Training improves answers with correct memory while reducing performance with other\-paper memory\. This is evidence that training changes the response to document content within the same dataset\.

A post hoc comparison of saved public\-model QASPER outputs also finds 10\.90 F1 with other\-paper memory versus 6\.60 without a document\. Thus the donor\-gain pattern is not absent from QASPER before adaptation\. These outputs come from separate runs with matching recorded checkpoint and generation code; removing the document also changes the prompt, and complete runtime equivalence is not established \(Appendix[S](https://arxiv.org/html/2610.19373#A19)\)\.

QASPER supplies annotated evidence, whereas the released\-model study uses retrieved passages; their answer\-generation procedures also differ\. These differences, together with scoring, can affect measured gains, but our studies do not isolate each one’s contribution\. The supported conclusion is that training and evaluation choices change what a score difference establishes, not that one QA dataset benefits from memory and another does not\.

## 6Supporting Lexical Checks

To test content benefit beyond QA, we predict words that occurred earlier in the text\. Three xRAG projectors receive the earlier text, the recent suffix, or a constant, while SFR and the decoder remain frozen\. Every decoder sees the same 281\-token suffix\. The constant is the mean of 8,192 training\-prefix embeddings\. We select each input’s learning rate and checkpoint on 512 development targets; all choose the highest tested rate,2×10−42\\times 10^\{\-4\}\. Ten paired training orders use 512 updates at batch size 64 \(Appendix[F](https://arxiv.org/html/2610.19373#A6)\)\.

#### Positive lexical benefit\.

On 3,044 PG\-19 repeated\-word targets, document input gives 4\.5088 NLL versus 4\.8145 for constant and 4\.8254 for suffix input\. The document advantage is\+0\.3166\+0\.3166nats over suffix input andC=\+0\.3057​\[0\.2530,0\.3585\]C=\+0\.3057\\;\[0\.2530,0\.3585\]over constant input\. Document input also improves accuracy over constant input by\+2\.13​\[1\.15,3\.09\]\+2\.13\\;\[1\.15,3\.09\]points\. The crossed simultaneous lower bound is 0\.85 points, below the one\-point descriptive benchmark\. Zeroing constant input raises NLL by\+1\.2360\+1\.2360nats, although its advantage over suffix input is uncertain:\+0\.0109​\[−0\.0042,0\.0259\]\+0\.0109\\;\[\-0\.0042,0\.0259\]\. This again separates reliance on a learned input from an advantage over another trained input\.

#### A harder case\.

We construct 217 pairs from 67 NASA articles by replacing a repeated word with one of equal token length while preserving the local suffix\. On original inputs, document input improves NLL over constant input byC=\+0\.2098C=\+0\.2098nats, but the crossed simultaneous interval\[−0\.0058,0\.4424\]\[\-0\.0058,0\.4424\]includes zero\. The adjusted accuracy difference is−2\.86​\[−8\.85,2\.70\]\-2\.86\\;\[\-8\.85,2\.70\]points\. After substitution, document\-input accuracy falls from 47\.56% to 0\.23%, despite a positive NLL contrast\. The model does not reliably recover the substituted words; these probes do not establish factual recovery \(Appendix[F](https://arxiv.org/html/2610.19373#A6)\)\.

## 7Discussion

Three results expose a common inference problem\. Relative gains can grow while both models worsen \(Table[2](https://arxiv.org/html/2610.19373#S4.T2)\)\. Removing a learned constant input can hurt despite that input conveying no current\-document facts \(Table[3](https://arxiv.org/html/2610.19373#S5.T3)\)\. Other questions’ passages can raise QA scores \(Table[5](https://arxiv.org/html/2610.19373#S5.T5)\)\. A score change alone therefore cannot identify the benefit of the current document’s content\.

Evaluations should match claims to comparisons\. Zeroing tests reliance after training; a separately trained no\-memory model tests performance benefit\. Trained constant\-input controls and replacement passages test whether gains require document\-dependent input or the question’s passage\. Our positive QA results support document benefit while showing why that benefit cannot be inferred from every observed gain\.

## 8Conclusion

Zeroing memory after training can hurt even when a model trained without it performs better\. Likewise, a relative advantage can grow while both models’ prediction losses worsen\. Our QA comparisons show that document content helps, yet some gains over zeroed memory or no document also occur without the question’s document\. Within QASPER, decoder training increases sensitivity to correct versus other\-document memory; in released\-model QA, generation and scoring choices change the conclusions about other\-passage gains\. Before attributing gains to document content, evaluations should name the comparison model, report absolute performance, and specify how answers are generated and scored\.

## Limitations

Every benefit in this paper is relative to a specified comparison model, task, and training procedure\. We do not identify the best performance an architecture could achieve, establish convergence, or match total pretraining and inference costs\. Appendix[G](https://arxiv.org/html/2610.19373#A7)and the evidence archive retain selection rules and candidate settings\. The comparison with shared initialization and equal development\-search opportunities does not establish that zeroing hurts while the trained alternative performs better\. The longer training study also does not compare independently optimal settings\. Optimization differences may matter, but these results do not isolate them as a causal explanation\.

The generated stories permit copying of repeated values\. Predicting one distinguishing token with preceding gold text supplied is easier to constrain than generating an entire state value\. PG\-19 repeated words test lexical prediction rather than annotated facts\. The restricted\-attention experiments deliberately hide earlier text from direct attention and do not represent every deployment setting\. QASPER supplies annotated evidence, so its results do not evaluate retrieval or show that those passages are uniquely necessary\. Its 128\-question development panel and ten training orders provide limited optimization coverage\. Masking evidence and matched non\-evidence text controls length and approximate position, but not semantics or all changes in the input distribution\.

The studies use different tokenizers, memory modules, and multiple\-comparison adjustments\. We compare paired conditions within each study and do not rank raw NLL across tokenizers\. Separating development and test data also does not exclude foundation\-model pretraining exposure\. The xRAG decoder or SFR encoder may have encountered PG\-19, and an embedding could help identify familiar text as well as convey current context\. Pythia was trained on the Pile, which includes PG\-19\([Biderman et al\., 2023](https://arxiv.org/html/2610.19373#bib.bib7);[Gao et al\., 2020](https://arxiv.org/html/2610.19373#bib.bib24)\); our local holdout does not remove that exposure\. Selecting settings on folds separate from those used for scoring provides internal validation only\.

Intervals that resample items while holding trained models fixed do not cover training uncertainty\. Crossed intervals over ten training orders and documents include order variation at a shared initialization, but not variation across arbitrary initializations or the full configuration search\. Finite bootstrap samples also limit precision in the tails used for simultaneous intervals\. An interval containing zero is not proof of no effect\.

The NASA task uses automatic word substitutions without human validation of semantic answers\. Holding local suffixes fixed restricts the change to earlier input, but the large accuracy difference shows that the original and substituted examples have different difficulty\. More generally, constant input restricts what the projector can compute even when its architecture is unchanged\. The document\-versus\-constant difference therefore describes the trained models, not a unique causal fraction of utility\.

The released\-model TriviaQA analysis uses one frozen checkpoint and conditions on the supplied retrieval and three fixed donor assignments\. Encoding the question adds computation to a prompt that already contains it; this changes evaluation input without retraining\. Although fixed software and batch size four closely reproduce reported aggregate Match, the original per\-question outputs and complete configuration are unavailable\. The question\-input gain and its ratio to the document\-input gain are post hoc descriptions outside the corrected 15\-effect family\. Changes in scores across batches do not isolate output length as their sole cause\.

## Ethics Statement

We use synthetic templates and public texts and datasets, including TriviaQA, NQ\-open, WebQuestions, PG\-19, QASPER, CoQA, MuSiQue, and NASA articles\. The supporting rank study uses LongBench\([Bai et al\., 2024](https://arxiv.org/html/2610.19373#bib.bib1)\)\. No participants were recruited or new human annotation commissioned\. Data and model use follows upstream terms, and public access does not imply unrestricted redistribution\. The numerical supplement preserves complete comparison families and experiment settings, including results that qualify the claims\. It separates author\-generated evidence from third\-party checkpoints and corpora\.

### AI Assistance

Generative AI assisted literature search, study\-design discussion, implementation and debugging, experiment orchestration, analysis, manuscript writing, and figure preparation\. The authors checked input and checkpoint identities, development selections, saved\-answer rescoring, paired effect reconstruction, and the cited source passages\. They reviewed the rendered PDFs and remain responsible for the research and its presentation\.

For the literature audit, AI tools assisted full\-text review, initial classification, a second coding pass, and adjudication of disagreements\. The reported agreement statistics describe these AI\-assisted coding passes, not agreement between independent human annotators\.

## References

- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 3119–3137\.Cited by:[Ethics Statement](https://arxiv.org/html/2610.19373#Sx2.p1.1)\.
- Belikovaet al\.\(2026\)J\. Belikova, D\. Rozhevskii, D\. Svirin, K\. Polev, and A\. PanchenkoDetecting overflow in compressed token representations for retrieval\-augmented generation\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),pp\. 797–810\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-srw.59),[Link](https://aclanthology.org/2026.eacl-srw.59/)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p3.1)\.
- Berantet al\.\(2013\)J\. Berant, A\. Chou, R\. Frostig, and P\. LiangSemantic parsing on Freebase from question\-answer pairs\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,D\. Yarowsky, T\. Baldwin, A\. Korhonen, K\. Livescu, and S\. Bethard \(Eds\.\),Seattle, Washington, USA,pp\. 1533–1544\.External Links:[Link](https://aclanthology.org/D13-1160/)Cited by:[§5\.2](https://arxiv.org/html/2610.19373#S5.SS2.SSS0.Px2.p1.1)\.
- Bidermanet al\.\(2023\)S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.Pythia: a suite for analyzing large language models across training and scaling\.InInternational Conference on Machine Learning,pp\. 2397–2430\.Cited by:[§4\.1](https://arxiv.org/html/2610.19373#S4.SS1.p1.1),[Limitations](https://arxiv.org/html/2610.19373#Sx1.p3.1)\.
- Caiet al\.\(2024\)R\. Cai, Y\. Tian, Z\. Wang, and B\. ChenLoCoCo: dropping in convolutions for long context compression\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 5348–5359\.External Links:[Link](https://proceedings.mlr.press/v235/cai24g.html)Cited by:[§4](https://arxiv.org/html/2610.19373#S4.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.5.1)\.
- Chenget al\.\(2024\)X\. Cheng, X\. Wang, X\. Zhang, T\. Ge, S\. Chen, F\. Wei, H\. Zhang, and D\. ZhaoxRAG: extreme context compression for retrieval\-augmented generation with one token\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.nips.cc/paper_files/paper/2024/hash/c5cf13bfd3762821ef7607e63ee90075-Abstract-Conference.html)Cited by:[item 3](https://arxiv.org/html/2610.19373#S1.I1.i3.p1.1),[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p2.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.4.1)\.
- Chevalieret al\.\(2023\)A\. Chevalier, A\. Wettig, A\. Ajith, and D\. ChenAdapting language models to compress contexts\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 3829–3846\.Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p1.1)\.
- Cuconasuet al\.\(2024\)F\. Cuconasu, G\. Trappolini, F\. Siciliano, S\. Filice, C\. Campagnano, Y\. Maarek, N\. Tonellotto, and F\. SilvestriThe power of noise: redefining retrieval for RAG systems\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,External Links:[Document](https://dx.doi.org/10.1145/3626772.3657834),[Link](https://doi.org/10.1145/3626772.3657834)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p4.1)\.
- Dasigiet al\.\(2021\)P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. GardnerA dataset of information\-seeking questions and answers anchored in research papers\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4599–4610\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.365),[Link](https://aclanthology.org/2021.naacl-main.365/)Cited by:[§5\.1](https://arxiv.org/html/2610.19373#S5.SS1.p1.1)\.
- Denget al\.\(2025\)C\. Deng, Z\. Zhang, K\. Mao, S\. Li, X\. Huang, D\. Yu, and Z\. DouA silver bullet or a compromise for full attention? a comprehensive study of gist token\-based context compression\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 4861–4879\.External Links:[Link](https://aclanthology.org/2025.acl-long.241/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.241),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p3.1)\.
- Feldman and Artzi \(2026\)Y\. Feldman and Y\. ArtziNo mean feat: simple, strong baselines for context compression\.Note:Version 2External Links:2510\.20797,[Link](https://arxiv.org/abs/2510.20797v2)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p4.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. LeahyThe pile: an 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.External Links:[Link](https://arxiv.org/abs/2101.00027)Cited by:[Limitations](https://arxiv.org/html/2610.19373#Sx1.p3.1)\.
- Geet al\.\(2024\)T\. Ge, J\. Hu, L\. Wang, X\. Wang, S\. Chen, and F\. WeiIn\-context autoencoder for context compression in a large language model\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.7.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2610.19373#S4.SS1.p1.1)\.
- Hewittet al\.\(2021\)J\. Hewitt, K\. Ethayarajh, P\. Liang, and C\. ManningConditional probing: measuring usable information beyond a baseline\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 1626–1639\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.122/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.122)Cited by:[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p1.1)\.
- Hewitt and Liang \(2019\)J\. Hewitt and P\. LiangDesigning and interpreting probes with control tasks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2733–2743\.External Links:[Link](https://aclanthology.org/D19-1275/),[Document](https://dx.doi.org/10.18653/v1/D19-1275)Cited by:[§1](https://arxiv.org/html/2610.19373#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p2.1)\.
- Hookeret al\.\(2019\)S\. Hooker, D\. Erhan, P\. Kindermans, and B\. KimA benchmark for interpretability methods in deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/fe4b8556000d0f0cae99daa5c5c5a410-Abstract.html)Cited by:[§1](https://arxiv.org/html/2610.19373#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p2.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p2.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. Weld, and L\. ZettlemoyerTriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 1601–1611\.External Links:[Link](https://aclanthology.org/P17-1147/),[Document](https://dx.doi.org/10.18653/v1/P17-1147)Cited by:[§5\.2](https://arxiv.org/html/2610.19373#S5.SS2.SSS0.Px1.p1.1)\.
- Karpukhinet al\.\(2020\)V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. YihDense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6769–6781\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.550/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[Appendix Q](https://arxiv.org/html/2610.19373#A17.p1.1),[§5\.2](https://arxiv.org/html/2610.19373#S5.SS2.SSS0.Px2.p1.1)\.
- Kimet al\.\(2024\)J\. Kim, J\. Yeom, S\. Yun, and H\. O\. SongCompressed context memory for online language model interaction\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/97dc07f1253ab33ee514f395a82fa7cc-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p3.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.6.1)\.
- Kuratovet al\.\(2025\)Y\. Kuratov, M\. Arkhipov, A\. Bulatov, and M\. BurtsevCramming 1568 tokens into a single vector and back again: exploring the limits of embedding space capacity\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 19323–19339\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.948),[Link](https://aclanthology.org/2025.acl-long.948/)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p3.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. PetrovNatural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Link](https://aclanthology.org/Q19-1026/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[§5\.2](https://arxiv.org/html/2610.19373#S5.SS2.SSS0.Px2.p1.1)\.
- Leeet al\.\(2019\)K\. Lee, M\. Chang, and K\. ToutanovaLatent retrieval for weakly supervised open domain question answering\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 6086–6096\.External Links:[Link](https://aclanthology.org/P19-1612/),[Document](https://dx.doi.org/10.18653/v1/P19-1612)Cited by:[§5\.2](https://arxiv.org/html/2610.19373#S5.SS2.SSS0.Px2.p1.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 3045–3059\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.243/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.243)Cited by:[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p2.1)\.
- Li and Janson \(2024\)M\. Li and L\. JansonOptimal ablation for interpretability\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/c55e6792923cc16fd6ed5c3f672420a5-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p2.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 4582–4597\.External Links:[Link](https://aclanthology.org/2021.acl-long.353/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.353)Cited by:[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p2.1)\.
- Meta \(2024\)MetaLlama 3\.2 model card\.External Links:[Link](https://huggingface.co/meta-llama/Llama-3.2-1B)Cited by:[§4\.1](https://arxiv.org/html/2610.19373#S4.SS1.p1.1)\.
- Muet al\.\(2023\)J\. Mu, X\. L\. Li, and N\. GoodmanLearning to compress prompts with gist tokens\.Advances in Neural Information Processing Systems36,pp\. 19327–19352\.Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p1.1)\.
- O’Connor and Andreas \(2021\)J\. O’Connor and J\. AndreasWhat context features can transformer language models use?\.InProceedings of ACL\-IJCNLP,pp\. 851–864\.External Links:[Link](https://aclanthology.org/2021.acl-long.70/)Cited by:[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p1.1)\.
- Raeet al\.\(2020\)J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, and T\. P\. LillicrapCompressive transformers for long\-range sequence modelling\.InInternational Conference on Learning Representations,External Links:[Link](https://iclr.cc/virtual_2020/poster_SylKikSYDH.html)Cited by:[§4](https://arxiv.org/html/2610.19373#S4.SS0.SSS0.Px1.p1.1)\.
- Rauet al\.\(2025\)D\. Rau, S\. Wang, H\. Déjean, S\. Clinchant, and J\. KampsContext embeddings for efficient answer generation in retrieval\-augmented generation\.InProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining,pp\. 493–502\.External Links:[Document](https://dx.doi.org/10.1145/3701551.3703527),[Link](https://doi.org/10.1145/3701551.3703527)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p2.1)\.
- Reddyet al\.\(2019\)S\. Reddy, D\. Chen, and C\. D\. ManningCoQA: a conversational question answering challenge\.Transactions of the Association for Computational Linguistics7,pp\. 249–266\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00266),[Link](https://aclanthology.org/Q19-1016/)Cited by:[Appendix M](https://arxiv.org/html/2610.19373#A13.p1.1),[§5\.1](https://arxiv.org/html/2610.19373#S5.SS1.p3.1)\.
- Salesforce \(2024\)SalesforceSFR\-Embedding\-Mistral model card\.External Links:[Link](https://huggingface.co/Salesforce/SFR-Embedding-Mistral)Cited by:[§3\.2](https://arxiv.org/html/2610.19373#S3.SS2.p2.1)\.
- Santhanamet al\.\(2022\)K\. Santhanam, O\. Khattab, J\. Saad\-Falcon, C\. Potts, and M\. ZahariaColBERTv2: effective and efficient retrieval via lightweight late interaction\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 3715–3734\.External Links:[Link](https://aclanthology.org/2022.naacl-main.272/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.272)Cited by:[§5\.2](https://arxiv.org/html/2610.19373#S5.SS2.SSS0.Px1.p1.1)\.
- Tanet al\.\(2024\)S\. Tan, X\. Li, S\. G\. Patil, Z\. Wu, T\. Zhang, K\. Keutzer, J\. E\. Gonzalez, and R\. A\. PopaLLoCO: learning long contexts offline\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17605–17621\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.975/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.975)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p2.1)\.
- Wanget al\.\(2024\)Y\. Wang, Y\. Gao, X\. Chen, H\. Jiang, S\. Li, J\. Yang, Q\. Yin, Z\. Li, X\. Li, B\. Yin, J\. Shang, and J\. McAuleyMEMORYLLM: towards self\-updatable large language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 50453–50466\.External Links:[Link](https://proceedings.mlr.press/v235/wang24s.html)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.9.1)\.
- Wanget al\.\(2025\)Y\. Wang, D\. Krotov, Y\. Hu, Y\. Gao, W\. Zhou, J\. McAuley, D\. Gutfreund, R\. Feris, and Z\. HeM\+: extending MemoryLLM with scalable long\-term memory\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 63308–63323\.External Links:[Link](https://proceedings.mlr.press/v267/wang25au.html)Cited by:[§4](https://arxiv.org/html/2610.19373#S4.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.10.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2610.19373#S4.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.2.1)\.
- Zhang and Nanda \(2024\)F\. Zhang and N\. NandaTowards best practices of activation patching in language models: metrics and methods\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2309.16042)Cited by:[§2\.1](https://arxiv.org/html/2610.19373#S2.SS1.p2.1)\.
- Zhanget al\.\(2025\)P\. Zhang, Z\. Liu, S\. Xiao, N\. Shao, Q\. Ye, and Z\. DouLong context compression with Activation Beacon\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fc797c61eb18f7b3ce9a74af0ef0d876-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.8.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. ChenH2\{\}\_\{2\}O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f8-Paper-Conference.pdf)Cited by:[§4](https://arxiv.org/html/2610.19373#S4.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.19373#S4.T1.2.3.1)\.
- Łajewskaet al\.\(2025\)W\. Łajewska, M\. Hardalov, L\. Aina, N\. Anna John, H\. Su, and L\. MarquezUnderstanding and improving information preservation in prompt compression for LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 17520–17541\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.949/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.949),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.2](https://arxiv.org/html/2610.19373#S2.SS2.p3.1)\.

## Appendix AControlled Probe: Implementation and Data

#### Local access and delayed memory\.

This probe asks whether a model can use compressed records of earlier story events when those events are outside its local attention window\. At each layer, tokenttattends only to itself and the precedingk−1k\-1tokens\. WithLLlayers, information can therefore travel at mostL⁡\(k−1\)L\(k\-1\)positions through local attention\. We divide the sequence into blocks ofr=16r=16consecutive tokens\. Tokenxtx\_\{t\}belongs to blockb⁡\(t\)=⌊\(t−1\)/r⌋b\(t\)=\\lfloor\(t\-1\)/r\\rfloor, where the floor rounds down\. For each completed block, we average its hidden states to obtainh¯j\\bar\{h\}\_\{j\}\. The encoderfϕf\_\{\\phi\}applies layer normalization, anH→HH\\to Hlinear map, GELU and anH→256H\\to 256linear map, whereHHis the backbone’s hidden width\. The resulting memory slots satisfyp^i=fϕ​\(h¯i−2\)\\hat\{p\}\_\{i\}=f\_\{\\phi\}\(\\bar\{h\}\_\{i\-2\}\)fori≥2i\\geq 2\. This two\-block delay prevents a token from reading a summary of its own unfinished block\. Slot 0 is learned, slot 1 isfϕ​\(0\)f\_\{\\phi\}\(0\), and the memory bank resets within each evaluation window\.

Pooled reading averages all available projected slots and adds their gated contribution to the local hidden state:

zt\\displaystyle z\_\{t\}=1b⁡\(t\)\+1​∑i≤b⁡\(t\)\(Wp​p^i\+bp\),\\displaystyle=\\frac\{1\}\{b\(t\)\+1\}\\sum\_\{i\\leq b\(t\)\}\(W\_\{p\}\\hat\{p\}\_\{i\}\+b\_\{p\}\),\(5\)gt\\displaystyle g\_\{t\}=σ⁡\(Wg​\[ht;zt\]\+bg\),h~t=ht\+gt⊙zt\.\\displaystyle=\\sigma\(W\_\{g\}\[h\_\{t\};z\_\{t\}\]\+b\_\{g\}\),\\qquad\\tilde\{h\}\_\{t\}=h\_\{t\}\+g\_\{t\}\\odot z\_\{t\}\.\(6\)Herehth\_\{t\}is the local hidden state,σ\\sigmais the sigmoid function,\[;\]\[\\,;\\,\]concatenates vectors, and⊙\\odotmultiplies them elementwise\.Wp,bpW\_\{p\},b\_\{p\}project each slot to the hidden width, andWg,bgW\_\{g\},b\_\{g\}determine the gate\. The output head predicts next\-token probabilitiesQθ\(xt\+1∣x1:t,P^≤b⁡\(t\);ℳk\)Q\_\{\\theta\}\(x\_\{t\+1\}\\mid x\_\{1:t\},\\hat\{P\}\_\{\\leq b\(t\)\};\\mathcal\{M\}\_\{k\}\), whereℳk\\mathcal\{M\}\_\{k\}is the local attention mask andP^≤b⁡\(t\)\\hat\{P\}\_\{\\leq b\(t\)\}contains the available memory slots\. The indexed variant reads only the current slot, usingzt=Wp​p^b⁡\(t\)\+bpz\_\{t\}=W\_\{p\}\\hat\{p\}\_\{b\(t\)\}\+b\_\{p\}; its conservative reach adds3​r=483r=48positions\. Scored targets exclude recurrences within 384 tokens, placing the relevant earlier occurrence beyond both local and indexed reach\. We check that changing future tokens leaves earlier predictions unchanged\. Block means and slots remain attached to the computation graph, so LoRA receives gradients through both local and memory paths\.

#### Comparison models and training\.

The no\-memory comparison model adds a residual multilayer perceptron \(MLP\) at the current position, with layer normalization, a linear map, GELU and a second linear map\. This local adapter cannot access the memory bank\. Its expansion width matches the memory module’s parameter count within 0\.06% \(Table[6](https://arxiv.org/html/2610.19373#A1.T6)\); this does not equalize the functions the modules can represent or their inference costs\. We separately train four conditions: local LoRA alone, LoRA with the local adapter, pooled memory and indexed memory\. They share the backbone, tokenizer, data, objective and LoRA configuration\. The 72 fits use seeds 71–73, learning rate10−410^\{\-4\}, cosine decay and 183 updates of 262,144 tokens \(47,972,352 total\)\. Validation runs every 8M tokens, but evaluation uses the fixed final checkpoints\. LoRA rank and scale are 16/32\. LoRA adapts fused QKV/MLP projections for Pythia, and q/k/v plus gate/up/down projections for Llama\. The base model weights remain frozen\.

Table 6:Controlled\-state settings: layer countLL, windowkk, backbone widthHH, and adapter intermediate width from saved tensors\. Module counts exclude common LoRA\. Parameter\-count matching error is below 0\.06%\.Table 7:Pooled PlanGen on state tokens \(nats; marginal 95% document\-bootstrap CIs, three fitted seeds\)\.UUcompares pooled memory with the parameter\-matched adapter;DDcompares its intact and zeroed inputs\. All six haveU​<0,D\>​0U<0,D\>0\. Candidate accuracy does not establish state recovery; mean uniform\-candidate references are 11\.35% \(Pythia\) and 17\.24% \(Llama\)\. Table[8](https://arxiv.org/html/2610.19373#A1.T8)reports all arms\.
#### Stories and scoring\.

The generator creates 6,400/800/800 train/dev/test stories by assigning values to entity attributes and mentioning them later\. Each story has six assignment\-to\-mention sequences, including two updates to earlier assignments\. However, the recurrence filter removes all test targets governed by those updates\. Training and development stories are joined with end\-of\-sequence \(EOS\) tokens in groups of 64; test windows stay within individual documents\. At evaluation, the model receives the preceding gold text\. We score the first token that distinguishes the true value from its allowed alternatives, requiring at least two candidate tokens\. The earlier mention must lie within 2,000 tokens, and the scored token must be absent from the preceding 384\. These rules retain 3,847 Pythia and 3,687 Llama targets\. Every model within a backbone is scored on the same targets\.

Negative log\-likelihood \(NLL\) uses probabilities over the full vocabulary\. In contrast, the accuracy in Table[8](https://arxiv.org/html/2610.19373#A1.T8)chooses among the permitted candidate tokens\. A predictor that chooses uniformly among an item’sKKcandidates has mean accuracy 11\.35% for Pythia and 17\.24% for Llama; its mean item\-wiselog⁡K\\log Kis 2\.233918 and 1\.902385 nats, respectively\. These benchmarks already know the candidate set\. They are neither uniform predictions over the full vocabulary nor model NLLs renormalized over candidates\. The models’ low absolute accuracy does not establish reliable recovery of the story state\.

Table 8:Controlled state targets: full\-vocabulary NLL / candidate\-token accuracy \(%\), averaged over seeds 71–73 on the same positions as Table[7](https://arxiv.org/html/2610.19373#A1.T7)\. All four arms are shown; raw NLLs are compared only within a backbone\.
#### Incorrect candidate values in local text\.

We use saved Llama\-3B outputs to ask whether low accuracy is confined to targets with a misleading candidate nearby\. An incorrect candidate occurs in the 281\-token local input for 860 of 3,687 targets\. Pooled\-memory accuracy is 13\.18% with such a candidate and 14\.34% without one; corresponding uniform\-candidate benchmarks are 14\.55%/18\.07%\. Adapter accuracy is 12\.98%/14\.94%\. Low accuracy therefore persists even without a local distractor, although this association does not identify the cause\. Because the original files omit total probability assigned to candidates, a post hoc forward audit records it for 512 fixed targets\. Both full\-vocabulary and candidate\-restricted pooled\-memory accuracy are 14\.78%, compared with 18\.55% for a predictor based only on training frequencies\. The crossed interval for the accuracy difference is\[−7\.94,0\.39\]\[\-7\.94,0\.39\]points\. The recurrence filter excludes all 92 targets for the generated valueroute\-seven, because the scored token occurs locally\. Supplementary Appendix S27 gives the complete diagnostic\.

#### Interventions and diagnostic scope\.

Zeroing sets memory values to zero but retains projection bias; bypassing instead removes the added residual contribution\. A bank swap supplies memory from a different document with the same number of visible slots\. Its intact and swapped scores use exactly the same covered subset\. Permuting slots cannot change their pooled mean\. Zeroing only document\-summary slots nearly reproduces the damage from zeroing all slots\. In contrast, zeroing only initial slots, replacing document summaries with their training mean, and swapping banks change NLL by amounts within±0\.01\\pm 0\.01nat\. Supplementary Table S3 reports all seven contrasts and their subset sizes\. It also preserves the small absolute\-NLL differences between H100 runs at batch eight and A6000 runs at batch one\. Each contrast pairs BF16 measurements within the same execution\. These measurements describe which inference inputs matter to the trained model; neither rounding differences nor slot interventions identify how training produced that dependence\. Supplementary Section S1 gives the architecture diagram and full intervention settings\.

## Appendix BEarlier Objective and Constant\-Input Controls

This study asks whether an apparent memory advantage changes when training emphasizes the scored state tokens\. It uses three seeds and a fixed budget of 512 updates\. Appendix[L](https://arxiv.org/html/2610.19373#A12)reports a separate ten\-seed study with equal development searches for each model and objective\. The sign reversal observed here must not be transferred to that later recipe\.

#### Matched training and selection\.

We train pooled memory and a local adapter under two objectives: predicting every next token, or predicting only the designated state tokens\. Both objectives receive identical input sequences, and the all\-token objective includes the state targets\. All conditions use public Llama\-3\.2\-3B, attention window 11 and rank\-16/scale\-32 LoRA\. Pooled memory and the adapter contain 29,900,288/29,898,496 module parameters\. The adapter width here is 4,864; the separately matched configuration in Appendix[A](https://arxiv.org/html/2610.19373#A1)uses 4,866\. These are distinct saved tensor widths\. Initialization and input order are paired within each seed, and the residual gate starts nonzero\.

Eight development runs at seed 301 compare learning rates5×10−55\\times 10^\{\-5\}and2×10−42\\times 10^\{\-4\}for 128 updates\. On a fixed panel of 128 development items, the mean endpoint NLL across the four model/objective conditions is 4\.2523/3\.4885\. This selects the common rate2×10−42\\times 10^\{\-4\}\. Final fits at seeds 311–313 run 512 updates with batch four, zero weight decay, AdamWβ=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), gradient clipping at one, 16 warmup updates and cosine decay to one tenth of the peak rate\. They use FP32 trainable parameters and BF16 base weights on A6000\. Evaluation uses the fixed final checkpoint\. Each fit trained only on state targets receives 2,048 sampled presentations\. Development curves neither select test checkpoints nor establish convergence\.

#### Constant input throughout training\.

To test whether document content accounts for the observed advantage, six additional pooled\-memory fits receive the same vector in place of every document block\. We compute that vector once as the mean of 190,938 complete delayed training blocks, using the initial masked backbone, FP64 accumulation and FP32 storage\. These fits match the normal\-memory fits in initial tensors, sampled examples, gate, slots, delay and budget\. The encoder and gate remain trainable, and local hidden states and sequence length remain available\. Thus the memory path is independent of document content, but the model can still make different predictions at different positions\. Replacing document values also removes the gradient path through remote document states\. The comparison therefore changes more than the information in an otherwise identical optimization problem\.

The shared objective panel has 1,024 targets in 617 stories\. The content comparison evaluates all eighteen fits on 1,024 targets from 588 of 800 new stories \(generator seed 2026101041\), with no exact full\-story/window overlap with the earlier pools\. It uses the same generator and causal selection rules\. Nine unmatched items are excluded only from bank\-swap contrasts\. Table[9](https://arxiv.org/html/2610.19373#A2.T9)gives every new\-document effect, including accuracy and direct objective differences; Supplementary Section S2 retains the shared\-panel results and development records\.

Table 9:New\-document content control: 1,024 targets from 588 stories, A6000, seeds 311–313\. All 27 conditional 95% intervals are shown\.UUcompares normal memory with the local adapter;UconstU\_\{\\rm const\}compares document\-independent memory with the same local adapter\.DconstD\_\{\\rm const\}is zeroing damage to that control\.C=NLLconst−NLLnormalC=\\mathrm\{NLL\}\_\{\\rm const\}\-\\mathrm\{NLL\}\_\{\\rm normal\}measures normal memory's advantage;CaccC\_\{\\rm acc\}uses normal\-minus\-constant accuracy\. Swap covers 1,015 targets; the other effects cover all 1,024\. Accuracy is full\-vocabulary first\-token accuracy, unlike the candidate\-restricted accuracy in Table[8](https://arxiv.org/html/2610.19373#A1.T8); its contrasts use percentage points\. An uninformed uniform full\-vocabulary predictor has accuracy100/\|V\|100/\|V\|percent, not the candidate\-aware100/K100/Kreference\.Both document memory and the constant\-input path change from worse than the local adapter to better than it when training switches to designated targets\. Zeroing still harms both trained paths\. Under all\-token training, the NLL difference between document and constant input lies within±0\.01\\pm 0\.01nat; under target\-only training, its interval includes zero\. The accuracy contrasts also include zero\. The 0\.01\-nat benchmark is about 34%/19% of the absolute document\-memory utility estimates, so lying within this benchmark does not establish that content contributes negligibly\. Both the memory model and its adapter comparator change when the objective changes\. Their relative gain cannot assign a causal contribution to either change\. The supporting evidence retains all 27 effects and their Bonferroni sensitivity bounds\.

## Appendix CGold\-Answer QA: Data, Fitting and Contrasts

#### Eligible inputs and independent paper splits\.

This experiment tests compression when the relevant evidence is supplied, without introducing retrieval errors\. QASPER v0\.3 provides 1,458/667/979 eligible train/dev/test questions from 721/260/388 disjoint papers\. We retain answerable, non\-boolean annotations whose evidence strings match source paragraphs after whitespace normalization, then join all matched evidence paragraphs in document order\. Evidence spans contain 16–2,048 encoder tokens; questions contain at most 128 decoder tokens; the canonical answer plus EOS contains at most 64\. Extractive answer spans are joined with commas; other cases use the annotated free\-form answer\. All accepted answer aliases remain available for scoring, while the first is the training target\. A fixed hash of question IDs orders the items and selects the first 128 development questions for model selection\. Because inputs contain annotated evidence, this experiment evaluates neither retrieval nor reading an entire paper\.

#### Three trained inputs\.

We train three projectors to compare evidence content with other inputs through the same interface\. All start from the public 33,562,624\-parameter xRAG projector; the SFR encoder and Mistral decoder remain frozen\. The projector receives an evidence embedding, a question embedding, or a constant equal to the mean training\-evidence embedding\. The constant uses FP64 accumulation, FP32 rounding and BF16 projector input\. In every condition, the decoder receives one projected vector plus the same question text\. Although all projectors have the same nominal parameter count, the constant condition can produce only one learned 4,096\-dimensional vector\. It therefore has less input\-dependent capacity\.

Each input condition independently screens learning rates10−5,5×10−5,2×10−410^\{\-5\},5\\times 10^\{\-5\},2\\times 10^\{\-4\}at seed 491, using batch 16, zero weight decay and 128 updates\. Minimum development NLL across checkpoints 0/32/64/128 selects the rate; ties favor the lower rate and then the earlier checkpoint\. All conditions select the grid maximum,2×10−42\\times 10^\{\-4\}\. Final training uses ten seeds 501–510 with identical initial tensors and paired example permutations\. Each fit runs 512 updates and selects the checkpoint with minimum development NLL among 0/128/256/384/512\. AdamW usesβ=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), gradient clipping at one, eight warmup updates and cosine decay to one tenth of the peak rate\. Training averages canonical\-answer token losses, including EOS, within each question and then across questions\. This finite search does not establish an optimum beyond the tested grid\.

#### Generation and controls\.

The prompt contains*Background:<xRAG\>*, the question, and*The answer is:*\. The full\-text comparison replaces the vector marker with evidence text; the closed\-book comparison omits background and receives only the question\. Greedy generation stops at EOS or 96 new tokens, rather than at punctuation\. We compute normalized token\-overlap F1 and exact match \(EM\), taking the best score over accepted answer aliases\. NLL uses the canonical answer wording\. Before test evaluation, a development check requires full\-text F1 of at least 20 and a lead of five points over closed\-book F1\. Measured full\-text, closed\-book and public compressed\-model F1 is 25\.59/7\.13/14\.74\. This check establishes that the supplied evidence is usable by the full\-text model; it does not require the compressed model to outperform either comparison\.

Zeroing sets the projected vector to zero, while swapping replaces it with a fixed embedding from another paper\. After zeroing, the document\-trained and constant\-trained projector conditions give identical predictions on all 9,790 seed–question pairs\. Their shared zero\-input score therefore cancels when comparing the two zeroing effects, both itemwise and after paired averaging; the separately estimated interval endpoints need not add\. Table[10](https://arxiv.org/html/2610.19373#A3.T10)reports all twelve effects, and Table[39](https://arxiv.org/html/2610.19373#A18.T39)reports absolute scores\. The artifact preserves every seed, development trajectory and selection\. Secondary EM gains from the same answers areUquestion=\+1\.14​\[0\.40,1\.91\]U\_\{\\rm question\}=\+1\.14\\;\[0\.40,1\.91\]for document versus question input andC=\+1\.16​\[0\.46,1\.91\]C=\+1\.16\\;\[0\.46,1\.91\]for document versus constant input\. Zeroing and swapping lower document\-input EM by\+4\.29​\[3\.20,5\.43\]\+4\.29\\;\[3\.20,5\.43\]and\+2\.85​\[1\.89,3\.86\]\+2\.85\\;\[1\.89,3\.86\]points\. In the same order, bounds that also resample training\-order seeds and adjust across this separate four\-effect family are\[0\.19,2\.10\]\[0\.19,2\.10\],\[0\.24,2\.09\]\[0\.24,2\.09\],\[2\.91,5\.67\]\[2\.91,5\.67\]and\[1\.62,4\.08\]\[1\.62,4\.08\]\.

Table 10:All twelve trained\-projector QA effects\. Conditional percentile intervals hold ten fits fixed; crossed sensitivity resamples paired seeds and papers with adjustment across these twelve effects\. PositiveU/CU/Cfavor document input, and positive intervention scores mean damage\. F1 uses points and NLL nats\. Fixed\-storage LLM bounds appear in Tables[40](https://arxiv.org/html/2610.19373#A18.T40)and[11](https://arxiv.org/html/2610.19373#A4.T11); lexical bounds in Table[17](https://arxiv.org/html/2610.19373#A6.T17)\.

## Appendix DLLM Adaptation at Fixed Storage

#### Fixed document representation and matched adaptation\.

This experiment asks whether training the answer\-generating LLM can improve answers while leaving compressed storage unchanged\. The public SFR encoder and projector save one 4,096\-dimensional BF16 vector \(8,192 bytes\) per evidence input\. We reuse these exact bytes without training the encoder or projector, re\-encoding evidence, or passing evidence text to the decoder\. One decoder trains on the correct document vector; the other trains on the public projector’s output for the mean training embedding\. This second condition trains the decoder with constant input\. It is distinct from Appendix[C](https://arxiv.org/html/2610.19373#A3), which trains a projector while keeping the decoder frozen\.

Only decoder LoRA parameters are trained\. They use rank 16, alpha 32 and zero dropout on attention q/k/v/o and MLP gate/up/down projections, totaling 41,943,040 parameters\. Both conditions use the same QASPER inputs and paired seeds 501–510\. Their LoRA initialization has zero B matrices and identical A matrices\. They share 512 updates, batch 16, zero weight decay and greedy answers capped at 96 tokens\. Separate development screens run for 128 updates at seed 491 over the QA learning\-rate grid; both select5×10−55\\times 10^\{\-5\}\. Final development NLL is evaluated at 0/128/256/384/512, and every fit selects step 128\. Decoder adaptation adds trainable parameters and training cost while preserving storage\. The mean\-trained decoder receives one fixed vector throughout training but varying document vectors in the intact evaluation\. Its comparison with the document\-trained decoder therefore includes this change in input distribution\.

#### Paired comparisons\.

We compare improvement over the public decoder, improvement over a decoder trained with constant input, and the extra improvement obtained with the correct document rather than a donor\. For public, document\-trained and mean\-trained LLMsb,d,ab,d,a, and intact, swapped and mean inputsi,s,mi,s,m, defineR=di−biR=d\_\{i\}\-b\_\{i\},TLLM=di−amT\_\{\\rm LLM\}=d\_\{i\}\-a\_\{m\}andILLM=\(di−bi\)−\(ds−bs\)I\_\{\\rm LLM\}=\(d\_\{i\}\-b\_\{i\}\)\-\(d\_\{s\}\-b\_\{s\}\)\. The study’s joint claim requires all three fixed\-fit 95% lower bounds to exceed zero\. The additional effects areJ=\(di−ds\)−\(ai−as\)J=\(d\_\{i\}\-d\_\{s\}\)\-\(a\_\{i\}\-a\_\{s\}\),Dswap=di−dsD\_\{\\rm swap\}=d\_\{i\}\-d\_\{s\},Dmean=di−dmD\_\{\\rm mean\}=d\_\{i\}\-d\_\{m\}, andA=am−bmA=a\_\{m\}\-b\_\{m\}\.JJcompares swap sensitivity between the two training conditions;AAmeasures improvement from training on constant input\. A donor is the next item from a different paper in a fixed ordering, chosen independently of outcomes\. The tables report all nine conditions and seven effects\. The 63 evaluations retain 61,677 answers\.

Table 11:All seven fixed\-memory LLM effects, in F1 points\. Each focal contrast in this earlier study \(R,TLLM,ILLMR,T\_\{\\rm LLM\},I\_\{\\rm LLM\}\) is positive under both constructions\. The seven\-effect simultaneous family also resamples training\-order seeds; neither interval covers arbitrary initialization or recipe selection\. Two F1 points is a descriptive magnitude benchmark, not the existence criterion\.Table 12:All nine absolute conditions on the same 979 questions\. Adapted\-LLM rows average ten fits\. Intact uses the original projected document vector, swap uses a fixed different\-paper vector, and mean uses the public projection of the training\-only mean embedding\. EM, NLL and length are descriptive\. Lower swap scores for the document\-trained LLM are retained\.
#### Answer improvements, regressions and direct comparison\.

Relative to the public decoder, the document\-trained decoder gains exact matches on a mean 36\.2 questions per fit and loses them on 2\.4, with respective ranges 25–44/1–4\. It gains token overlap on 81\.7 formerly zero\-F1 answers, while 228\.8 answers with some overlap fall to zero\. Mean answer length falls from 48\.90 to 12\.46 tokens\. Token\-overlap transitions are not semantic correctness labels\. With donor input, F1 falls from 10\.90 for the public decoder to 7\.15 after document training\. Consequently, the interaction between training and document input exceeds the actual 7\.08\-point improvement on correct documents\.

When both trained decoders receive the same correct document vector, the document\-trained decoder exceeds the mean\-trained decoder by\+4\.975​\[3\.933,6\.023\]\+4\.975\\;\[3\.933,6\.023\]F1 points, with crossed interval\[3\.719,6\.230\]\[3\.719,6\.230\]\. The EM difference is\+0\.848​\[0\.148,1\.563\]\+0\.848\\;\[0\.148,1\.563\]points, but its crossed interval is\[−0\.035,1\.731\]\[\-0\.035,1\.731\]\. Truncating predictions to the accepted answer’s word length before computing F1 still favors the document\-trained decoder by5\.278​\[4\.105,6\.471\]5\.278\\;\[4\.105,6\.471\]\. This diagnostic requires gold\-answer length, so it is neither deployable nor a causal correction for answer format\. Neither this difference nor the mean\-trained decoder’s gain identifies a causal share of the overall improvement\. The result shows that some answer errors can be reduced without changing compressed storage when relevant evidence is supplied\. It does not establish complete information retention or a mechanism shared by all memory systems\. Supplementary Sections S9 and S24 report complete per\-seed transitions and exploratory EM/prefix families\.

## Appendix EReleased\-Model QA Audit

#### Generation with batch size one\.

We evaluate releasedHannibal046/xrag\-7bwith SFR\-Embedding\-Mistral and repository commit121fa4180a8c, without training or checkpoint selection\. The test contains 11,313 TriviaQA questions, each paired with its first listed ColBERTv2 passage\. Metadata ranks 11,262 of these passages first and 51 second\. We retain the upstream prompts, 180\-token encoding, BF16 A6000 inference, greedy 100\-token generation and stopping at newline, period or comma, using batch size one\. Following the released pipeline does not guarantee that these records and scores match the original paper’s main table\.

The closed\-book condition supplies the question without a vector\. Full text supplies the passage as text\. The document and question conditions instead encode the corresponding text into the vector slot, while zeroing sets the projected vector to zero\. Three donor conditions use vectors from different passage IDs, selected by question\-key hashes 6171/6172/6173 without inspecting answers or outcomes\. Different passage IDs do not guarantee that the text is semantically irrelevant\. Match measures whether the normalized prediction contains an accepted answer as a sequence of tokens; it differs from strict EM and token\-overlap F1\. Boost measures the fraction of closed\-book errors corrected, and resilience measures the fraction of closed\-book successes retained\. Donor effects average the scores from all three assignments within each question\.

Table 13:Released xRAG on all 11,313 repository TriviaQA questions\. Match is token\-substring answer inclusion; strict EM and F1 are descriptive secondary scores\. No weights are fitted in this audit\.Table 14:All eight released\-model effects\. The first seven are Match differences in percentage points; boost is a percentage conditional on closed\-book errors\. One maximum\-standardized family uses 50,000 paired original\-passage\-cluster draws and a two\-point benchmark\.The eight conditions produce 90,504 answers\. To estimate uncertainty for this fixed checkpoint and donor bank, we resample 10,984 original\-passage clusters for 50,000 draws and adjust the eight\-effect family using maximum\-standardized deviations\. The primary contrast compares question\-vector input with closed\-book input on Match\. Its positive gain does not extend to strict EM\. In a separate exploratory family, donor input exceeds closed\-book EM by10\.958​\[10\.250,11\.666\]10\.958\\;\[10\.250,11\.666\]points, and document input exceeds donor EM by7\.348​\[6\.635,8\.063\]7\.348\\;\[6\.635,8\.063\]\. Encoding the question changes the computation applied to information already visible to the decoder; it is not the separately trained constant\-input control used to defineCC\. The original paper includes a LoRA control trained on the task data, but does not specify its exact source access\. Supplementary Section S10 retains transition rates, donor outcomes, metric comparisons and the original table’s numbers\. These comparisons show how the chosen comparator affects interpretation; they neither establish that the source paper omitted controls nor provide an independent test set\.

#### Output length and prefix check\.

Mean output lengths in Table[13](https://arxiv.org/html/2610.19373#A5.T13)come from all saved answers\. An oracle\-prefix score normalizes each answer and alias, truncates the prediction to the alias’s word length, computes token F1, and maximizes over aliases\. All four contrasts below remain positive; this particular length adjustment therefore does not remove the pattern\. Gold length is unavailable in deployment, and order/phrasing still matter\. It neither estimates a causal format share nor validates every donor as semantically unrelated\.

Table 15:Released\-model oracle\-prefix diagnostics\. F1 uses points\. All four rows form one separate follow\-up family\.A fresh run of the unmodified command\-line evaluator at batch size one exactly reproduces all 11,313 saved raw answers\. Changing to batch size four preserves every prompt token ID but changes 9,905 answers\. Match becomes 56\.96%, strict EM 1\.29%, F1 19\.39 and mean answer length 27\.33 tokens, compared with 51\.59%, 30\.43%, 43\.54 and 10\.83 at batch one\. These runs use PyTorch 2\.5\.1/Transformers 4\.48\.2\. Appendix[K](https://arxiv.org/html/2610.19373#A11)separately reports batch\-four evaluation with the source\-specified software versions\. The released stopping rule waits for every row in a batch; padding may also affect generation, so the comparison does not isolate a single cause\. The source paper calls its 57\.08 closed\-book score EM, whereas the released evaluator selects Match\. Without original per\-question outputs and the exact configuration, a nearby aggregate score cannot establish identical reproduction\. This subsection’s contrasts use batch one; the main text uses batch four with the specified software versions\.

## Appendix FLexical Content: Methods and Complete Primary Effects

#### Inputs and fitting\.

This study asks whether a compressed earlier passage helps predict a repeated word beyond what the visible suffix or a constant vector provides\. All three conditions train the public 33\.56M\-parameter projector, while the SFR encoder and decoder remain frozen\. The projector receives the earlier document, the final 281\-token suffix, or the mean of 8,192 training\-prefix embeddings\. In every condition, the decoder receives one projected vector and the same suffix text\. Each condition independently screens the three learning rates used in the QA study for 128 updates on 512 development items, with evaluations at steps 0/32/64/128 and zero weight decay\. All select2×10−42\\times 10^\{\-4\}\. Ten training\-order seeds pair initialization and example permutations\. Final fits use batch 64 for 512 updates and select among checkpoints 0/128/256/384/512\. The PG\-19 test, shared with earlier comparisons, contains 3,044 targets from 401 books\. NLL is 4\.8254/4\.8145/4\.5088 for local, constant and remote input; corresponding accuracy is 18\.11/18\.15/20\.28%\.

#### Paired word substitutions in new articles\.

To test whether predictions respond to earlier document content while the visible suffix stays fixed, we select 217 pairs of repeated capitalized words from 67 NASA articles published in 2025, with at most four pairs per article\. Selection does not inspect model predictions\. Earlier mentions lie at least 512 tokens away, and inputs contain at most 2,048 tokens\. In a changed version of each example, we replace selected earlier mentions and the target with a word of the same token length\. The replacement’s first token differs from the original and does not occur in the visible input\. Hash\-based selection does not match word frequency, part of speech or semantic plausibility\. The final 281\-token suffix remains byte\-identical\. These are automatically constructed word\-prediction probes, rather than human\-annotated factual QA\. Recent publication alone does not establish absence from model pretraining\.

Letyo,ycy\_\{o\},y\_\{c\}be the original and replacement target tokens, and letpo,pcp\_\{o\},p\_\{c\}denote model probabilities under the corresponding document versions\. We measure whether changing the remote text shifts preference toward the matching target using\(mo\+mc\)/2\(m\_\{o\}\+m\_\{c\}\)/2, wheremo=log⁡po​\(yo\)−log⁡po​\(yc\)m\_\{o\}=\\log p\_\{o\}\(y\_\{o\}\)\-\\log p\_\{o\}\(y\_\{c\}\)andmc=log⁡pc​\(yc\)−log⁡pc​\(yo\)m\_\{c\}=\\log p\_\{c\}\(y\_\{c\}\)\-\\log p\_\{c\}\(y\_\{o\}\)\. Preference is correct in both versions whenmo\>0,mc\>0m\_\{o\}\>0,m\_\{c\}\>0\. This pairwise criterion differs from selecting the correct token from the full vocabulary\. Local and constant inputs remain identical across versions\. The following tables report original and substituted scores, all four PG\-19 effects and all eighteen article effects\. For original articles, crossed intervals forC/UC/Uinclude zero\. After substitution, the mean\-replacement contrast and several accuracy contrasts also include zero\. Full\-context accuracy falls from 66\.36% to 17\.05%, showing that substitution makes the prediction task harder even with the full prefix\.

Table 16:New\-article first\-token prediction\. The three trained arms average ten paired training\-order seeds\. The full\-context row is one frozen decoder receiving the whole prefix, not a local reference or a ten\-seed fit\.Table 17:All four PG\-19 and eighteen NASA effects, in separate families\. Fixed\-fit intervals resample documents; crossed intervals also resample ten paired order seeds with family adjustment\. Original NASAU/CU/Chave positive fixed\-fit lower bounds and negative crossed lower bounds\. NLL uses nats and accuracy uses percentage points\.Table 18:Shared\-zero arithmetic:Ddoc=C\+DconstD\_\{\\rm doc\}=C\+D\_\{\\rm const\}\. Ratio isDconst/DdocD\_\{\\rm const\}/D\_\{\\rm doc\}, a descriptive score ratio, not a causal fraction of stored information\. Units differ across rows; complete intervals are in Appendices[C](https://arxiv.org/html/2610.19373#A3)and[F](https://arxiv.org/html/2610.19373#A6)\.

## Appendix GOther Comparisons and Boundaries

#### Equal development searches and longer training\.

A separate pooled/query\-memory comparison starts from the same public checkpoint and gives each condition the same development\-search opportunity\. All intervention intervals lie within±0\.01\\pm 0\.01nat\. No evaluated cohort establishes both positive zeroing damage and negative utility\. In another experiment, a query reader trained on frozen representations hasU=−0\.0855,D=1\.343U=\-0\.0855,D=1\.343nats\. Its PG\-19 local comparator worsens on development data, so a positive gain over that comparator’s final fit does not establish superiority to a well\-selected no\-memory model\. A further study trains memory/comparator pairs for 2,048 updates, selects checkpoints on development data, and evaluates new synthetic stories and MuSiQue\. All four NLL utility intervals include zero\. Synthetic pooled\-memory zeroing damage is0\.0278​\[0\.0111,0\.0446\]0\.0278\\;\[0\.0111,0\.0446\]\. On MuSiQue, zeroing query memory instead improves NLL, withD=−0\.0230​\[−0\.0388,−0\.0079\]D=\-0\.0230\\;\[\-0\.0388,\-0\.0079\]\. Supplementary Sections S4–S5 provide the separate recipes and complete settings\.

SystemTargetsLocal NLLIntact NLLGainUU\[simultaneous CI\]ZeroingDD\[simultaneous CI\]Synthetic: state\-disambiguating tokenStreamingLLM1,6521\.8492\.459−0\.610​\[−0\.667,−0\.552\]\-0\.610\\;\[\-0\.667,\-0\.552\]\+6\.484​\[\+6\.279,\+6\.694\]\+6\.484\\;\[\+6\.279,\+6\.694\]H2O1,6521\.8492\.068−0\.219​\[−0\.292,−0\.147\]\-0\.219\\;\[\-0\.292,\-0\.147\]\+7\.690​\[\+7\.467,\+7\.916\]\+7\.690\\;\[\+7\.467,\+7\.916\]xRAG2,0002\.2793\.211−0\.932​\[−1\.010,−0\.855\]\-0\.932\\;\[\-1\.010,\-0\.855\]\+6\.790​\[\+6\.582,\+6\.995\]\+6\.790\\;\[\+6\.582,\+6\.995\]LoCoCo2,0002\.2323\.646−1\.414​\[−1\.598,−1\.234\]\-1\.414\\;\[\-1\.598,\-1\.234\]\+6\.367​\[\+6\.150,\+6\.591\]\+6\.367\\;\[\+6\.150,\+6\.591\]CCM2,0002\.2310\.078\+2\.153​\[\+2\.104,\+2\.197\]\+2\.153\\;\[\+2\.104,\+2\.197\]\+9\.910​\[\+9\.728,\+10\.089\]\+9\.910\\;\[\+9\.728,\+10\.089\]ICAE1,9762\.2440\.058\+2\.185​\[\+2\.139,\+2\.227\]\+2\.185\\;\[\+2\.139,\+2\.227\]\+7\.767​\[\+7\.633,\+7\.907\]\+7\.767\\;\[\+7\.633,\+7\.907\]Beacon1,2011\.9210\.056\+1\.865​\[\+1\.801,\+1\.926\]\+1\.865\\;\[\+1\.801,\+1\.926\]\+7\.955​\[\+7\.756,\+8\.152\]\+7\.955\\;\[\+7\.756,\+8\.152\]MemoryLLM1,9901\.8760\.029\+1\.847​\[\+1\.790,\+1\.900\]\+1\.847\\;\[\+1\.790,\+1\.900\]\+9\.822​\[\+9\.696,\+9\.945\]\+9\.822\\;\[\+9\.696,\+9\.945\]M\+1,9901\.8720\.068\+1\.804​\[\+1\.751,\+1\.858\]\+1\.804\\;\[\+1\.751,\+1\.858\]\+8\.834​\[\+8\.707,\+8\.965\]\+8\.834\\;\[\+8\.707,\+8\.965\]PG\-19: repeated\-word first tokenStreamingLLM3,0446\.9196\.586\+0\.333​\[\+0\.141,\+0\.530\]\+0\.333\\;\[\+0\.141,\+0\.530\]\+4\.895​\[\+4\.576,\+5\.210\]\+4\.895\\;\[\+4\.576,\+5\.210\]H2O3,0446\.9192\.359\+4\.560​\[\+4\.353,\+4\.766\]\+4\.560\\;\[\+4\.353,\+4\.766\]\+8\.726​\[\+8\.503,\+8\.948\]\+8\.726\\;\[\+8\.503,\+8\.948\]xRAG3,0444\.7124\.813−0\.100​\[−0\.192,−0\.006\]\-0\.100\\;\[\-0\.192,\-0\.006\]\+1\.237​\[\+1\.103,\+1\.372\]\+1\.237\\;\[\+1\.103,\+1\.372\]LoCoCo3,0444\.4883\.812\+0\.676​\[\+0\.516,\+0\.842\]\+0\.676\\;\[\+0\.516,\+0\.842\]\+5\.653​\[\+5\.408,\+5\.896\]\+5\.653\\;\[\+5\.408,\+5\.896\]CCM3,0444\.7084\.246\+0\.462​\[\+0\.342,\+0\.590\]\+0\.462\\;\[\+0\.342,\+0\.590\]\+2\.200​\[\+2\.023,\+2\.382\]\+2\.200\\;\[\+2\.023,\+2\.382\]ICAE3,0448\.66511\.597−2\.932​\[−3\.327,−2\.556\]\-2\.932\\;\[\-3\.327,\-2\.556\]−2\.326​\[−2\.708,−1\.957\]\-2\.326\\;\[\-2\.708,\-1\.957\]Beacon3,0445\.4162\.555\+2\.861​\[\+2\.631,\+3\.090\]\+2\.861\\;\[\+2\.631,\+3\.090\]\+4\.650​\[\+4\.414,\+4\.883\]\+4\.650\\;\[\+4\.414,\+4\.883\]MemoryLLM3,0444\.7142\.202\+2\.513​\[\+2\.275,\+2\.762\]\+2\.513\\;\[\+2\.275,\+2\.762\]\+8\.106​\[\+7\.882,\+8\.324\]\+8\.106\\;\[\+7\.882,\+8\.324\]M\+3,0444\.7548\.336−3\.583​\[−3\.807,−3\.366\]\-3\.583\\;\[\-3\.807,\-3\.366\]\+1\.782​\[\+1\.695,\+1\.872\]\+1\.782\\;\[\+1\.695,\+1\.872\]Table 19:Existing systems, three fitted seeds per row, in nats\. PositiveUUfavors memory; positiveDDmeans zeroing harms prediction\. Synthetic rows score state\-disambiguating tokens; PG\-19 rows score repeated\-word first tokens\. Intervals are study\-specific simultaneous 95% intervals\. H2O uses full\-prefix prefill followed by pruning\.SystemAbsent NLLZeroed NLLDabsD\_\{\\rm abs\}\[simultaneous CI\]EE\[simultaneous CI\]Synthetic: state\-disambiguating tokenStreamingLLM10\.9068\.943\+8\.447​\[\+8\.307,\+8\.588\]\+8\.447\\;\[\+8\.307,\+8\.588\]\+0\.002​\[−0\.004,\+0\.005\]\+0\.002\\;\[\-0\.004,\+0\.005\]H2O10\.9069\.757\+8\.838​\[\+8\.684,\+8\.991\]\+8\.838\\;\[\+8\.684,\+8\.991\]\+0\.405​\[\+0\.353,\+0\.460\]\+0\.405\\;\[\+0\.353,\+0\.460\]xRAG9\.97110\.001\+6\.760​\[\+6\.556,\+6\.962\]\+6\.760\\;\[\+6\.556,\+6\.962\]\+1\.268​\[\+1\.216,\+1\.321\]\+1\.268\\;\[\+1\.216,\+1\.321\]PG\-19: repeated\-word first tokenStreamingLLM11\.58211\.481\+4\.996​\[\+4\.687,\+5\.312\]\+4\.996\\;\[\+4\.687,\+5\.312\]\+0\.002​\[−0\.003,\+0\.010\]\+0\.002\\;\[\-0\.003,\+0\.010\]H2O11\.58211\.086\+9\.223​\[\+8\.974,\+9\.476\]\+9\.223\\;\[\+8\.974,\+9\.476\]\+3\.711​\[\+3\.463,\+3\.959\]\+3\.711\\;\[\+3\.463,\+3\.959\]xRAG6\.0126\.050\+1\.199​\[\+1\.067,\+1\.333\]\+1\.199\\;\[\+1\.067,\+1\.333\]\+0\.569​\[\+0\.506,\+0\.635\]\+0\.569\\;\[\+0\.506,\+0\.635\]Table 20:Additional interventions for StreamingLLM, H2O, and xRAG in Table[19](https://arxiv.org/html/2610.19373#A7.T19)\(nats\)\. PositiveDabsD\_\{\\rm abs\}means absence harms prediction; positiveEEmeans evidence masking harms more than matched sham masking\. The respective simultaneous families contain 8/8/32 effects\.
#### Nine systems, task rankings and average prediction\.

Supplementary Tables S8, S9 and S13 report all nine systems on both data sources, including every effect direction, interval and selection record\. Because systems differ in which components are adapted and which inputs they can access, their results do not isolate architectural causes\. CCM, Beacon and MemoryLLM outperform their trained local comparators on both sources\. Four synthetic configurations worsen when memory is zeroed but still underperform their local comparators\. Sensitivity to relevant evidence can therefore coexist with negative utility\. In PG\-19, every direct comparison between gains on all tokens and gains on repeated\-word targets has an interval containing zero\. Although model rankings change with the evaluated task, cross\-fitted comparisons do not establish an advantage from selecting models by memory\-target scores\. These results bound the interpretations of the observed counterexamples\.

#### Trained constant projector versus directly optimized vector\.

With a fixed decoder, a biased projector receiving one constant input and a directly optimized 4,096\-dimensional vector can represent the same set of constant outputs\. Finite training need not find the same output\. We optimize the free vector using gold\-answer NLL, rather than a KL objective that preserves a model’s outputs\. One comparison keeps the decoder frozen; another jointly adapts rank\-16/alpha\-32 decoder LoRA\. Each uses ten paired fits, 512 updates, batch 16 and QA development selection\. Both joint conditions select learning rate5×10−55\\times 10^\{\-5\}; both free\-vector conditions select coefficient0\.10\.1, the largest tested multiplier of the root mean square of the public projected mean\. Warm starts use the trained constant projector’s output and therefore include prior training\. LetG=Strained​constant−SvectorG=S\_\{\\rm trained\\ constant\}\-S\_\{\\rm vector\}be the score difference between the two forms of constant input\. The change in that gap under joint decoder adaptation is F1H=Gjoint−Gfrozen=0\.449​\[−0\.106,1\.010\]H=G\_\{\\rm joint\}\-G\_\{\\rm frozen\}=0\.449\\;\[\-0\.106,1\.010\], with crossed interval\[−0\.650,1\.548\]\[\-0\.650,1\.548\]\. For NLL reduction,H=0\.0283​\[0\.0207,0\.0358\]H=0\.0283\\;\[0\.0207,0\.0358\], with crossed interval\[0\.0111,0\.0454\]\[0\.0111,0\.0454\]\. Thus joint adaptation does not establish a larger F1 difference between these controls\. The likelihood result also does not demonstrate an error in the optimal\-ablation method, whose objective differs from ours\. Supplementary Section S11 retains all fourteen effects and selections\.

#### Choosing when to use a document and evaluating retrieved text\.

We train question\-only rules to decide which questions receive document input under a fixed usage budget\. Four rules predict, respectively, the gain over trained constant input, swap damage, zeroing damage, or the gain over trained question input\. All use the same fixed features, centered linear\-kernel ridge model and five\-fold cross\-validation with whole papers kept together\. At 489/979 document uses, the rule trained onCCexceeds the swap\-trained rule by0\.375​\[−0\.133,0\.921\]0\.375\\;\[\-0\.133,0\.921\]F1 points\. Its direct differences from the zeroing\-trained and question\-comparison rules also lie within±2\\pm 2\. Its gain over uniform allocation is1\.632​\[1\.101,2\.187\]1\.632\\;\[1\.101,2\.187\], which does not establish an advantage larger than two points\. A separate input\-shift diagnostic replaces annotated evidence with the top\-1 or top\-3 paragraphs retrieved within the question’s paper by TF–IDF or SFR\. It does not retrain the model on retrieved text\. No retrieval policy meets the specified criterion of negative gains over both trained controls together with positive intervention damage beyond the stated magnitude benchmarks\. Supplementary Sections S21–S23 give that criterion, complete rules, budget curves, all retrieval policies and both interval constructions\.

Table 21:Retrieved\-input deployment shift on the same 979 questions\. Shared question/constant F1 is 12\.535/12\.545\. Bounds belong to the full 28\-effect seed–paper simultaneous family\. All four policies have damage exceedingCC, without establishing negative utility orU/CU/Cdivergence\. All 28 effects and absolute EM/NLL/length remain in the companion\.
#### Scores normalized by closed\-book and full\-text models\.

The ratio\(Fa−Fclosed\)/\(Ffull−Fclosed\)\(F\_\{a\}\-F\_\{\\rm closed\}\)/\(F\_\{\\rm full\}\-F\_\{\\rm closed\}\)expresses a condition’s F1 gain over closed\-book input as a fraction of the gap between full\-text and closed\-book F1\. HereFaF\_\{a\}is the condition being evaluated\. On our QA outputs, this ratio is 25\.87% for the trained constant and 58\.24% for document input\. We recompute the denominator in every bootstrap draw\. A positive normalized score can therefore occur even without document access\. These ratios neither measure recovered information nor replicate results with task\-finetuned teacher models\. Across 722/9,790 seed–question pairs \(7\.37%\), document input beats the frozen closed\-book model but loses to the trained constant\. Ties remain in the denominator\. Supplementary Section S8\.1 reports all eight effects and the reverse transitions\.

## Appendix HInference, Scope and Reproduction

#### Paired uncertainty\.

Fixed\-fit document intervals treat the trained models as fixed and capture variation across evaluation sources\. We first average paired seeds for each item, then resample complete stories, books or papers with replacement\. Each draw recomputes the specified item\-weighted mean; its 2\.5th/97\.5th percentiles give an unadjusted 95% interval\. Related targets and original/substituted versions stay together\. QA, decoder and lexical analyses use 50,000 draws\. The released\-model audit at batch one clusters questions by retrieved passage\. Batch\-four evaluations instead join questions sharing either a batch or a passage and resample those connected components\. Multiple\-comparison adjustments remain specific to each stated study family, even when results share a table\.

Crossed intervals additionally resample paired training\-order seeds, capturing this source of training variation as well as evaluation\-source variation\. QA, reference\-score, allocation, optimal\-ablation and decoder families use maximum\-standardized deviations for simultaneous coverage\. Lexical analyses and the specified fixed\-fit families use Bonferroni percentiles0\.025/m0\.025/mand1−0\.025/m1\-0\.025/m, wheremmis the number of effects in the family\. Tables[10](https://arxiv.org/html/2610.19373#A3.T10),[11](https://arxiv.org/html/2610.19373#A4.T11),[17](https://arxiv.org/html/2610.19373#A6.T17)and[21](https://arxiv.org/html/2610.19373#A7.T21)provide both constructions or the corresponding crossed bounds for the core claims; Supplementary Section S26 gives additional diagnostics\. The two constructions cover different sources of uncertainty\. A lower bound above zero supports the direction of a gain; a lower bound above a numerical benchmark supports a gain exceeding that magnitude\. An interval within the positive and negative benchmarks describes its range but does not establish practical equivalence\. The stated benchmarks are 0\.01 nat, two F1 points or one accuracy point\. They are not validated deployment thresholds\. For the decoder study’s joint claim, every primary lower bound must exceed zero\.

Table 22:Historical study families, supplemented by the six planned contrasts, fifteen pinned\-generation effects and three CoQA effects in Appendix[N](https://arxiv.org/html/2610.19373#A14)\. Secondary analyses are exploratory, not independent confirmations\. No bound accounts for selection among analyses\. Other studies retain their original fixed\-fit adjustments; the numerical artifact lists all family members\.
#### Selection and scope\.

The intervals do not cover arbitrary initialization choices, searches over configurations or selection among analyses\. Analyses of normalized scores, document allocation, slot interventions, local candidates, task rankings, cross\-fitted model selection and joint effects use available test summaries and are exploratory\. Each training study retains its stated development\-selection or fixed\-final\-checkpoint rule\. In the ten\-fit constant\-input study, available results informed both sample size and the displayed interval specification\. Repeated analysis of the same public test is not independent confirmation\. Crossed intervals capture variation in training order at the shared initialization and recipe\. The main Limitations and the complete analysis\-scope table apply to every result\.

#### Complete evidence and files\.

This appendix contains core implementation details and all principal objective/content, QA, LLM and lexical contrast families\. It also includes the nine\-system bounds, retrieval summary, native lengths and prefix contrasts\. The separatesupporting\_evidence\.pdfpreserves the full settings, additional\-system studies, all per\-seed transitions, development curves, rankings, allocation/retrieval diagnostics, exact\-match/prefix follow\-ups and complete interval tables\. Sections/tables prefixed S refer to that document\. No outcome is removed by sign\.

The source/data snapshot retains saved answers and scores, fixed input identities, fitting/selection records, analysis sources and both PDF sources\. Its standalone CoQA reconstruction officially rescores 20,250 saved answers and reconstructs the three paired effect intervals; it does not rerun neural inference\. Earlier portable checks covering 196 historical effects are not a coverage claim for the expanded studies\. Model weights are excluded, and full model reproduction requires the recorded original inputs and runtime dependencies\. The numerical archive preserves each analysis family and every retained unfavorable result\.

## Appendix IReview of Controls in Selected Memory Papers

We examined the full texts and appendices of 40 selected papers: 19 on learned compression and 21 on attention key/value \(KV\) retention\. We fixed the paper list and seven categories before classification, recorded source URLs and PDF hashes, and kept page\-level evidence for each judgment\. This is a purposive review of named papers, not a systematic or random census\.

#### What we looked for\.

A trained independent input supplies a learned constant or learned tokens through the tested interface without using the current document; we identify controls that replace only part of a memory\. We distinguish this from training a model without access to the source, and from removing memory after training\. Other categories record full\-input comparisons, frozen source\-restricted models, broader task\-trained comparisons, changes to source content or order, and tasks that score source recall, reconstruction or answers\. “Not located” means we found no qualifying comparison in the reviewed version; “unclear” means the description did not establish the required input access or interface\.

Table 23:Sample descriptions under the stated broad definitions\. Unclear and not\-applicable cases are shown, not folded into absence\. “Present” includes the qualified component/order controls described below\. A missing control does not invalidate a paper’s performance or efficiency claim\.
#### Findings and their scope\.

Among the 19 learned\-compression papers, we found no clear instance of a trained independent input under this definition: 17 were not located and two were unclear\. Gist’s hidden\-instruction control and RMT’s nonrecurrent learned tokens are related precedents, but their equivalence to the focal compressed interface remains unclear\. Among the 21 KV papers, StreamingLLM supplies a component\-specific precedent by learning initial sink slots while retaining other source KV\. These distinctions do not support a claim that comparable controls are absent throughout the field, or that any paper’s performance claims are invalid\.

We retained separate classification records for 24 papers, covering a deterministic 12\-paper sample and additional ambiguous or contribution\-relevant cases\. The records disagreed in 18 of 168 cells; we preserved the disagreements and their resolutions\. Agreement was 89\.29% with pooled nominal Cohenκ=0\.820\\kappa=0\.820, reported descriptively rather than as validated reliability\. The AI Assistance statement discloses the assistance used in the review\. Supplementary Section S34 gives all paper\-level categories, source links and qualifications\.

## Appendix JWithin\-Paper Evidence and Matched Decoder Controls

This comparison asks whether the benefit of annotated evidence remains when the replacement text comes from the same paper, rather than a different topic\. We specify the contrast before generating its outputs, but reuse the previously evaluated public QASPER test; this is not independent confirmation\. All 979 questions from 388 papers remain eligible\. We reuse the same ten projectors trained on document input, with seeds 501–510, and keep questions, decoder prompts and generation settings fixed\.

For each question, the donor text comes from that question’s paper\. We exclude paragraphs that match annotated evidence after whitespace normalization, including substring overlap in either direction\. A question\-specific hash orders the remaining paragraphs\. We take a prefix whose SFR input length, including special tokens, exactly matches the original evidence input; the final paragraph may be partial\. Selection inspects neither predictions nor answer matches\. Text without an evidence annotation may still contain the answer\. Thus matching the paper and encoder length controls topic and length differences, but does not isolate answer\-bearing semantics from every formatting or content difference\.

Mean intact/same\-paper F1 is 19\.9785/4\.7420, giving an intact\-minus\-swap gap of 15\.2365 points\. With 50,000 paired whole\-paper draws, the fixed\-fit 95% interval is\[13\.8176,16\.6779\]\[13\.8176,16\.6779\]; crossed paired order\-seed/paper resampling gives\[13\.7586,16\.7300\]\[13\.7586,16\.7300\]\. The crossed Bonferroni interval is\[13\.2351,17\.2696\]\[13\.2351,17\.2696\]: individual nominal coverage 99\.1667%, retaining the original six\-primary\-contrast family for nominal 95% simultaneous coverage\. The six\-comparison family is defined in Appendix[N](https://arxiv.org/html/2610.19373#A14)\. These intervals do not cover analysis\-selection or configuration\-search uncertainty\. Saved answer text was independently rescored, and a separately indexed bootstrap agreed with the reported reconstruction\.

#### Matched decoder training\.

A second comparison tests whether correct document vectors help when all decoder conditions receive the same training budget and model\-selection opportunities\. Ten paired training\-order fits train with correct document vectors, vectors from other papers, or questions alone \(closed\-book\)\. They share learning\-rate and checkpoint grids, 512 updates, batch size 16 and paired training orders\. Wrong\-document vectors come from another paper and change across uses during training\. At evaluation, both decoder conditions trained with vectors receive the same correct document vectors; the closed\-book decoder still receives only the question\. Development data select checkpoints, while the reused public test provides no independent confirmation\. These fits are separate from the earlier document/mean decoder comparison\.

Mean F1 is 21\.2108 for correct\-document training, 13\.7265 for wrong\-document training, and 13\.2666 for closed\-book training\. Correct minus wrong\-document training is \+7\.4844 F1; correct\-document minus closed\-book training is \+7\.9442\. Both remain positive under the prespecified six\-primary Bonferroni intervals\. This supports document\-input benefit under matched training, without establishing complete recovery of the source text\.

Table 24:Planned QASPER contrasts\. Crossed resampling pairs order seeds and whole papers; the adjustment retains the original six\-contrast family\.
#### Decoder and full\-text training settings\.

All four arms, including the exploratory full\-text reference, adapt 41,943,040 decoder parameters with rank\-16, scale\-32, zero\-dropout LoRA; the base weights, encoder and public projector remain frozen\. The 1,458 training questions and paired seeds 501–510 are shared\. Each arm independently screens learning rates10−5,5×10−5,2×10−410^\{\-5\},5\\times 10^\{\-5\},2\\times 10^\{\-4\}for 128 updates at seed 491 on the first 128 development questions; all select5×10−55\\times 10^\{\-5\}\. Final fits use batch 16, 512 updates, weight decay zero, FP32 trainables with BF16 base weights, AdamWβ=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), clipping one, eight warmup updates, and cosine decay to one tenth of peak\. Gradient checkpointing is enabled in all four arms\. Minimum development canonical\-answer NLL selects among steps 0,16,32,64,128,256,384,512; ties favor the earlier step\. Training scores answer tokens including EOS, averaged per question\.

The full\-text condition supplies the same annotated evidence as ordinary text instead of a compressed vector, retaining the question and answer prefix\. Closed\-book prompts omit the background; the two compressed conditions retain one vector slot\. All use greedy generation, EOS stopping and a 96\-token cap\. Test answers never select checkpoints\. The full\-text decoder therefore has the same training budget and trainable parameter count, but receives a longer input\. Its comparison is exploratory because it lies outside the planned six\-comparison family\.

Table 25:Development\-selected checkpoints for the four decoder arms; each arm still completes all 512 updates\.
#### Exploratory secondary results\.

The decoder trained with evidence text reaches F1 62\.5427\. The compressed\-document minus full\-text F1 difference is \-41\.3318 points \(crossed 95% interval\[−43\.6812,−38\.9475\]\[\-43\.6812,\-38\.9475\]\), so the compressed model’s positive gains over other controls do not establish full recovery of the available text’s predictive benefit\. In the earlier decoder study, replacing evidence with text from the same paper lowers F1 by 13\.2977 for the document\-trained decoder and 3\.4352 for the mean\-trained decoder\. The difference between these drops is \+9\.8625, with crossed 95% interval\[8\.5727,11\.1766\]\[8\.5727,11\.1766\]\. These are exploratory secondary comparisons\. Their nominal intervals do not adjust for selection or multiplicity across exploratory analyses\.

For the two matched decoder contrasts, fixed\-fit whole\-paper 95% intervals are\[6\.3188,8\.6545\]\[6\.3188,8\.6545\]and\[6\.7291,9\.1653\]\[6\.7291,9\.1653\], respectively\. Raw saved answers were independently rescored and an alternative indexed bootstrap reconstructed the results\. Integrity checks read the actual trainable tensors, verified changing other\-paper donors and paired training order, and checked storage byte and row hashes\. These checks validate recorded computations, not the broader interpretation of the contrasts\.

## Appendix KEvaluation with Fixed Software Versions and Batch Size Four

We repeat all eight input conditions on the same 11,313 TriviaQA questions and released checkpoint, using the repository’s default batch size of four and its specified software versions\. We choose these generation settings from the source documentation before inspecting their outputs\. This defines a reproducible evaluation setting; the original per\-question outputs are unavailable, so it does not establish identical predictions\. Two additional diagnostics each examine 256 questions per condition to test sensitivity to ordering and batch membership\. Evaluations with batch size one and with a different software runtime remain separate sensitivity analyses\.

Table 26:All input conditions with fixed software versions and batch size four\. Closed supplies only the question; full supplies passage text; document/question supply the corresponding compressed vector; zero supplies a zero vector\. Swap1–3 use vectors from different passages\. Match measures answer inclusion, EM strict normalized exact match, and F1 token overlap\. Donor comparisons average the three swaps\.For uncertainty estimates, we join questions that share an original batch or a retrieved passage and resample the resulting 2,501 connected components\. This preserves dependencies created by shared generation and source text\. Five contrasts, each measured with Match, EM and F1, form a separate 15\-effect family\. Its adjustment does not replace any earlier primary or secondary family\.

Table 27:Pinned batch\-four paired score contrasts in percentage points\. Bonferroni intervals jointly cover the fifteen specified contrasts nominally at 95%; they are not a correction over every analysis in the paper\.Providing an encoded question raises Match by \+3\.5181 points over closed\-book input; providing an encoded document raises it by \+8\.7156\. Their ratio, 40\.4%, is a post hoc descriptive quantity outside the fifteen\-effect family\. It is not the fraction of document benefit caused by question information: the decoder already receives the question text in both conditions, and the extra encoded vector changes its computation\.

#### Batch composition\.

Reversing question order within each batch changes zero raw answers across all eight conditions\. Changing which questions share a batch changes 216–232 of 256 answers per condition\. Match shifts by at most 0\.390625 points, while F1 changes range from \-2\.1225 to \+2\.9714\. These observations describe the fixed diagnostic subset, rather than establishing invariance across the population\. Shared stopping, padding and numerical generation may all contribute; the checks do not isolate answer length as a cause\. On the full evaluation set, donor vectors improve EM and F1 relative to closed\-book input, while the donor Match interval includes zero\. Thus the positive secondary scores do not establish a donor advantage on the source paper’s reported answer\-inclusion metric\.

## Appendix LObjective Comparison with Equal Development Searches

This study asks whether a memory model’s advantage over its comparator can increase even when both models’ absolute prediction loss worsens\. We compare a local adapter, pooled document memory and pooled memory trained with constant input\. Each is trained either to predict all next tokens or to predict only the designated state targets\. Three models, two objectives and ten paired training\-order seeds give sixty final fits\. Every model/objective combination receives the same development\-search budget over three candidate learning rates\. A new test contains 1,024 targets from separately generated stories; scores on the previously evaluated panel remain secondary evidence\. Training permits up to 1,024 updates\. Forty\-three fits reach that cap without convergence; seventeen meet the development\-plateau rule\. Neither stopping on a plateau nor selecting a checkpoint establishes optimality\. The earlier three\-seed, 512\-update study uses a different recipe and does not establish negative all\-token utility in this study\.

#### Fitting and selection\.

All cells use Llama\-3\.2\-3B, window 11, block size 16, one\-block delay, memory width 256, and rank\-16/scale\-32 LoRA on q/k/v and gate/up/down projections with zero dropout\. The same 2,048 training targets and 128\-target development panel are reused\. Each arm/objective screens LR\{5×10−5,2×10−4,8×10−4\}\\\{5\\times 10^\{\-5\},2\\times 10^\{\-4\},8\\times 10^\{\-4\}\\\}for 256 updates at seed 610, selecting minimum development target NLL at steps 128/256; ties favor lower LR then earlier step\. Selected all\-token rates for adapter/pooled/constant are2/8/8×10−42/8/8\\times 10^\{\-4\}; target\-only rates are8/2/2×10−48/2/2\\times 10^\{\-4\}\. The ten final seeds 611–620 use batch four, AdamWβ=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), zero weight decay, gradient clipping one, and 16 warmup updates followed by constant LR\. Trainables and the output head are FP32; backbone weights are BF16\. Development is evaluated at step zero and every 128 updates\. After at least 512 updates, three successive evaluations without improving the running meaningful best by 0\.001 nat trigger a plateau stop; reaching 1,024 always receives the nonconverged\-cap label\. The minimum NLL among all visited checkpoints is selected, ties earlier\. Test results never select rates, stopping or checkpoints\.

Table 28:New\-holdout target\-token scores and stopping by arm; all ten fits retained\. The candidate\-informed uniform reference has mean NLL 1\.8871 and accuracy 17\.59%\. It is not a uniform distribution over the full vocabulary\.With the target\-only objective, the adapter’s NLL remains 2\.5805 after renormalizing probabilities over the permitted candidates\. This is worse than uniform prediction over those candidates, so probability assigned outside the candidate set does not fully explain its high loss\. Its full\-vocabulary accuracy is below the expected accuracy of uniform candidate selection, although its candidate\-restricted accuracy is above that benchmark\. The memory model’s relative advantage therefore uses a comparator with poor absolute performance\. Switching from all\-token to target\-only training increases adapter NLL by 0\.9539 and pooled\-memory NLL by 0\.1730\. Pooled memory gains a relative advantage because its comparator worsens more, not because its own NLL improves\. Under all\-token training, the pooled\-memory versus adapter interval includes zero\.

Table 29:Paired seed/story intervals\. Constant utility uses its original nineteen\-secondary\-effect adjustment; other displayed adjusted intervals use six\-comparison endpoints, with only the new\-holdout objective difference a member of the six primary contrasts\. Reused\-panel estimates are secondary\.Figure 2:New holdout, ten paired fits per arm/objective\. Target\-only training worsens target\-token NLL for all arms, most for the adapter; positive utility need not imply absolute improvement\. Bars: crossed seed/story 95% intervals\. Dotted NLL line: uniform\-candidate reference\. Appendix[L](https://arxiv.org/html/2610.19373#A12)gives adjusted intervals and accuracy\.

## Appendix MCoQA Evaluation without Retraining QASPER Models

We ask whether the already trained QASPER models retain their document advantage on another QA dataset\. We evaluate 405 first\-turn questions from distinct CoQA development stories\([Reddy et al\., 2019](https://arxiv.org/html/2610.19373#bib.bib39)\), using ten paired training\-order seeds and five conditions\. No model is trained or selected on CoQA: all forty selected QASPER fits remain fixed, and evaluation reuses the exact memory\. Official scoring averages the best token F1/EM over four human reference answers with each reference held out in turn, retaining duplicates\. Inputs use annotated supporting text, called rationales\. The comparison text comes from the same story outside those rationales and may still contain the answer\. Encoder inputs average 24\.76 tokens \(median 19; range 5–197, including special tokens\)\. This evaluates short supplied evidence, rather than full\-document or multi\-turn conversational QA\.

This subset had not previously been evaluated in our local experiments, but CoQA training data were included in the released xRAG model’s upstream instruction training\. The evaluation therefore does not establish transfer to an unseen task or freedom from pretraining contamination\. All CoQA conditions share one GPU and runtime configuration, while QASPER uses another\. Differences in absolute scores between these datasets cannot be interpreted as hardware\-controlled effects\.

Table 30:Absolute CoQA F1, averaged over ten fixed QASPER fits\. Decoders trained with correct and wrong document vectors both receive the same correct rationale vector at evaluation\.Table 31:Separate three\-contrast family with paired story/seed resampling\. Large gains are conditional on short oracle evidence and known upstream task exposure\.
## Appendix NPlanned Follow\-up Comparisons

Six contrasts were fixed after the earlier studies but before their own new outputs\. The family comprises: \(1\) target\-only minus all\-token pooled utility on new synthetic holdout stories; \(2\) correct\-evidence minus same\-paper non\-evidence projector F1 on QASPER; \(3\) correct\-document minus wrong\-document\-trained decoder F1 on identical QASPER vectors; \(4\) correct\-document minus equally trained closed\-book decoder F1 on QASPER; \(5\) donor\-minus\-closed strict EM averaged equally across two additional repository QA datasets; and \(6\) document\-minus\-donor strict EM averaged across those same datasets\. The last two use the earlier batch\-one setting, not the pinned batch\-four TriviaQA panel\.

Each of these six two\-sided intervals uses tail probability0\.025/60\.025/6for nominal familywise 95% coverage\. Resampling pairs training\-order seeds and source stories/papers for fitted models, or retrieved\-passage clusters within each dataset for the frozen model\. QASPER reuses a public test, and none of these bounds covers analysis selection or arbitrary initialization\. The fifteen pinned\-generation effects and three CoQA contrasts retain separate families; post hoc ratios and the learned full\-text comparison are descriptive or exploratory, not extra primary successes\.

Table 32:All six planned comparisons, in the order defined above\. The last two average NQ\-open and WebQuestions equally under batch\-one generation and pair passage clusters; they are not pinned TriviaQA Match claims\. Score effects are percentage points except the first row\.
## Appendix OTraining Configurations for the Nine\-System Comparison

Within each system, the memory model and its separately trained local comparator predict the same task targets; the local comparator cannot access the remote memory bank\. Across systems, the adapted components, available input and training histories differ\. Synthetic StreamingLLM and H2O apply their cache policies to a common Llama\-3\.2\-3B checkpoint trained with full context, retaining 384 cache positions\. H2O prunes after processing the full source, so this setting does not test reductions in peak memory during online input processing\. xRAG trains its projector with encoder and decoder frozen; its suffix\-only comparator instead trains a decoder adapter\. LoCoCo trains a newly initialized convolutional compressor, while CCM adapts its released merge module\. ICAE, Beacon, MemoryLLM and M\+ retain their released compressors and train task decoder adapters, which are disabled while compressing the prefix\. The four synthetic configurations that underperform their local comparators also have low absolute task performance\. They differ in decoder adaptation from the configurations that predict well, so they do not demonstrate dependence without utility in a well\-functioning memory system\.

Table 33:Final learning rates and weight decay for the nine\-system comparison\. Task adaptation, suffix access and development histories differ; these are finite configurations rather than an optimized architecture ranking\.Table 34:Task\-trainable parameters and actual updates of the final fits behind Table[1](https://arxiv.org/html/2610.19373#S4.T1), three seeds per row\. M/L denotes memory/local\. Counts exclude frozen backbone/compressor parameters and prior pretraining\. Synthetic KV has unequal, staged fitting histories; parameter equality alone does not match optimization\.For synthetic StreamingLLM and H2O, the model receiving the retained key/value \(KV\) cache uses 24 ordinary\-language\-model updates on a 183\-update schedule\. Its local comparator instead receives 183 ordinary\-language\-model updates, then 64 target\-only updates at3×10−53\\times 10^\{\-5\}and 512 at10−510^\{\-5\}; the last two stages use zero weight decay\. Here target\-only means that loss is computed only at designated task tokens\. These evaluated training histories have unequal budgets\. Other rows train on designated target tokens at batch 64, with rates and weight decay in Table[33](https://arxiv.org/html/2610.19373#A15.T33)\. LoCoCo, CCM and the ordinary 128\-update fits use their endpoint checkpoints after the stated setting selection\. The displayed ICAE and M\+ PG\-19 confirmations evaluate update 64 from a 128\-update schedule\. The archive also retains the earlier settings, and the supporting evidence lists all development candidates and frozen configurations\.

The local comparators’ PG\-19 NLLs range from 4\.488 to 8\.665 because each system has its own trained comparison model\. These models differ in backbone, tokenizer, accessible input, adapted components and training recipe\. For example, the displayed ICAE comparator uses learning rate10−310^\{\-3\}and weight decay 0\.1, evaluated at update 64; LoCoCo’s uses2×10−42\\times 10^\{\-4\}and weight decay 0\.1 for 256 updates\. These differences neither identify the cause of the NLL gap nor establish that the comparators are optimally trained\. Raw NLL cannot rank architectures across these conditions\. We compute utility only between the memory model and local comparator belonging to the same system and task\.

## Appendix PExpanded Learning\-Rate Search for QA Projectors

All thirty final QASPER projector fits use learning rate2×10−42\\times 10^\{\-4\}, selected at the upper edge of the original three\-rate development grid\. To check whether larger rates improve the comparison conditions more, we run a bounded development search over2×10−4,8×10−4,3\.2×10−3,1\.28×10−2,5\.12×10−22\\times 10^\{\-4\},8\\times 10^\{\-4\},3\.2\\times 10^\{\-3\},1\.28\\times 10^\{\-2\},5\.12\\times 10^\{\-2\}separately for document, question and constant input\. The search reuses seed 491, the same 128 development questions, batch 16, zero weight decay and a 128\-update budget\. Minimum NLL selects among checkpoints 0,16,32,64,128\. The last two rates are specified before their outputs, and no test result selects a rate\. This diagnostic does not retrain the ten\-seed test models or replace the original twelve\-effect family\.

Table 35:Minimum development canonical\-answer NLL over the fixed checkpoints \(lower is better\)\. Every displayed cell selects update 128\. These are development scores, not new test estimates\.The best tested rates lie inside the expanded grid:8×10−48\\times 10^\{\-4\}for document input and3\.2×10−33\.2\\times 10^\{\-3\}for question and constant input\. Larger rates worsen development NLL\. Finding better rates for all conditions does not establish equal optimization quality or a global optimum\. The reported projector test results still use the original training recipe\. The matched decoder comparisons train a different component with fixed memory from the public model and do not depend on this projector selection\.

## Appendix QFixed\-Software Generation on NQ\-open and WebQuestions

We test whether the TriviaQA comparison between document, donor and closed\-book input extends to two other QA datasets under the same generation settings\. The evaluation includes all 3,610 NQ\-open and 2,032 WebQuestions items used in the earlier batch\-one comparisons\. Questions, answer aliases, top\-ranked DPR passages\([Karpukhin et al\., 2020](https://arxiv.org/html/2610.19373#bib.bib43)\), cached document vectors and three fixed donor assignments remain unchanged\. Retrieval files come from the public unsupervised\-passage\-reranking data distribution; the evidence archive records their download links and hashes\. Passage titles remain metadata and are excluded from encoder input\. Donors have different passage IDs, but may still contain relevant facts or answer strings\.

We use PyTorch 2\.1\.1, Transformers 4\.38\.0, RTX A6000, BF16 weights, the released greedy generation procedure, a 100\-token cap and its punctuation stopping rule\. Each original batch contains four consecutive items, except the final two\-item NQ\-open batch\. Splitting the evaluation across shards preserves these batches\. Closed\-book input supplies the question without a vector; document input supplies the retrieved passage vector\. Donor scores average three separate runs with the fixed donor vectors\. We specify the analysis and the priority of answer\-inclusion accuracy \(Match\) before generating these outputs\. Reuse of public tests and possible upstream training exposure prevent an unseen\-task confirmation claim\.

For each dataset, we compare donor input with closed\-book input and document input with donor input, using Match, EM and F1\. These two contrasts, two datasets and three scores form a separate twelve\-effect family\. We report ordinary 95% and Bonferroni\-adjusted intervals from 50,000 paired bootstrap draws\. Resampling units are connected components of questions sharing a batch or retrieved passage: 760 components for NQ\-open and 231 for WebQuestions\. The checkpoint and donor bank stay fixed\. This family does not replace the earlier six planned contrasts or the fifteen\-effect TriviaQA family\. Output\-length and question\-order checks are descriptive\.

Table 36:Absolute scores from every condition\. Strict EM is near its floor on both datasets\. Output length retokenizes saved answers and is not an independent mechanism test\.Table 37:All twelve effects\. The WebQuestions donor EM adjusted lower bound equals zero; its positive point estimate does not establish an adjusted positive effect\.Table 38:Changed raw answers among the first 256 questions under within\-batch reversal or frozen regrouping; prompts and input tokens remain identical per item\. All score changes are retained in the numerical supplement\.On WebQuestions, donor input improves Match over closed\-book input even after adjustment\. On NQ\-open, that interval includes zero\. Thus the unresolved donor Match gain observed on TriviaQA does not hold across all datasets\. Document input improves Match over donor input on both added datasets\. Donor F1 gains over closed\-book input are positive, while strict EM gains are small or unresolved\. Mean donor output is−5\.51\-5\.51tokens shorter than closed\-book output on NQ\-open and−0\.19\-0\.19shorter on WebQuestions, but these observations do not establish answer length as the sole cause\. Official rescoring and a separately indexed bootstrap reconstruct the saved answers, contrasts and order diagnostics without rerunning model inference\.

## Appendix RAdditional QASPER Comparisons

The main text retains the comparisons needed to distinguish document\-content benefit from gains without current\-document facts\. The following tables preserve the full absolute scores and the separate decoder\-training study\.

Table 39:Gold\-answer QASPER with supplied evidence: all 979 test questions\. Trained rows average ten training orders; full\-text and closed\-book rows share the frozen released decoder\. F1/EM are percentages; NLL is example\-mean answer\-token loss\. All arms receive the question\. Zeroed projector outputs give the same frozen predictor, shown once \(Equation[4](https://arxiv.org/html/2610.19373#S3.E4)\)\.Table 40:QASPER decoders: 979 test questions and ten training orders\. Both document\-trained rows use correct vectors at evaluation\. F1 gain is the correct\-document score minus each row\. NLL is mean canonical\-answer token loss in nats; lower is better\. Gain intervals resample orders and papers, adjusting for six comparisons \(Appendix[N](https://arxiv.org/html/2610.19373#A14)\)\. The full\-text advantage has exploratory crossed 95% bounds\[38\.95,43\.68\]\[38\.95,43\.68\]\.Table 41:QASPER gains over comparison models\. Gain is compressed\-model F1 minus reference F1, computed from unrounded means\. All rows share questions; only rows training the same component hold the compressed model fixed\.#### The comparison model matters\.

Table[41](https://arxiv.org/html/2610.19373#A18.T41)compares gains against baselines\. The projector and decoder studies train different components; their scores do not describe one fixed compressed model\. In the decoder study, evidence text gives 62\.54 F1 versus 21\.21 from compressed evidence and 13\.27 without evidence\. The compressed model closes\(21\.21−13\.27\)/\(62\.54−13\.27\)=16\.1%\(21\.21\-13\.27\)/\(62\.54\-13\.27\)=16\.1\\%of the score gap between no evidence and text\. The exploratory full\-text advantage is 41\.33 points\.

In the projector study, the ratio is 58\.24%, using compressed F1 of 19\.98 and frozen full\-text and closed\-book scores of 29\.57 and 6\.60 \(Appendix[G](https://arxiv.org/html/2610.19373#A7)\)\. Both the compressed model and its baselines change between studies\. These ratios describe performance for each training procedure, not the fraction of document information recovered\.

## Appendix SReconciling the QA Comparisons

This post hoc synthesis reuses saved answers; it adds no training or generation\. It was motivated by the apparent disagreement between positive document\-content results on QASPER and gains from other questions’ passages in released\-model QA\. Existing aggregate results were known before this synthesis, so it is not a preregistered cross\-study test\.

#### Different comparisons answer different questions\.

The QASPER projector comparison changes training input while retaining the projector architecture and frozen LLM\. Its document\-versus\-constant gain asks whether training with current\-document information improves answers\. The later decoder comparison trains on correct documents, changing other documents, or no documents; both document\-trained decoders then receive correct memory at evaluation\. Its 21\.21 versus 13\.73 F1 contrast therefore compares training histories\. Neither comparison estimates the effect of giving an already trained model another document at inference\. The released\-model QA comparison does exactly the latter, contrasting retrieved passages, other\-question passages and no document without updating weights\. Correct\-document superiority and an other\-document gain over no document are compatible outcomes\.

#### A within\-dataset check of training\.

The earlier fixed\-memory decoder study \(Appendix[D](https://arxiv.org/html/2610.19373#A4)\) evaluates the public model and ten document\-trained decoders on the same 979 questions from 388 papers\. Correct and other\-paper inputs use identical cached memory bytes before and after training; question order, donor assignments, prompt, and greedy EOS\-based generation with a 96\-token cap are unchanged\. We verified saved\-output hashes against completion receipts, matched question/evidence identities and memory hashes, and independently rescored all 22,517 saved answers used here, including the no\-document panel below\. Frozen scientific source hashes remain unchanged\.

Table 42:F1 on identical QASPER inputs before and after decoder training\. The trained row averages ten training orders from the earlier fixed\-memory study; it is distinct from the later 21\.21\-F1 decoder comparison\. The increase in the correct\-minus\-other difference is 10\.83 points, with original crossed simultaneous 95% bounds\[9\.07,12\.59\]\[9\.07,12\.59\]\.Correct\-memory F1 rises by 7\.08 points, while other\-paper F1 falls by 3\.74 points\. Their difference, 10\.8294, reconstructs the original interaction estimate exactly\. Its original simultaneous interval resamples paired training orders and papers and retains that study’s comparison family\. It is not a newly selected marginal interval\. The change demonstrates that decoder adaptation alters sensitivity to the supplied memory without changing the dataset or memory\. It does not isolate semantic use from changes in answer format, or establish training as the sole cause of differences between datasets\.

#### Other\-paper gains also appear in unadapted QASPER\.

We pair the saved public\-model other\-paper outputs above with the 979 no\-document outputs from the projector study\. Recorded checkpoint/tokenizer manifests, question identities, answer aliases and the shared generation implementation agree\. The public scores are 6\.6047 F1 without a document, 10\.8990 with another paper’s memory, and 14\.2055 with correct memory\. The other\-paper\-minus\-no\-document difference is\+4\.2944\+4\.2944points\. A new, descriptive whole\-paper bootstrap gives a marginal 95% interval\[3\.5358,5\.0890\]\[3\.5358,5\.0890\]\(20,000 draws, seed 20261013\)\. Each draw samples 388 papers with replacement, keeps all questions within each selected paper, and computes the question\-weighted paired difference\. This interval is unadjusted and separate from the original study families\.

These are separate saved evaluation runs; matching source and checkpoint records do not verify complete runtime equivalence\. The no\-document prompt also omits the background instruction\. Thus the contrast describes the whole input conditions and is not a controlled estimate of document semantics alone\. Mean generated lengths are 80\.95 tokens without a document, 48\.47 with other\-paper memory, and 48\.90 with correct memory\. Length and content change together, so these observations do not establish shorter output as the cause of the F1 increase\. They nevertheless show that the descriptive other\-document\-gain pattern is not unique to the released\-model retrieval benchmarks\.

#### Scoring and generation change the interpretation\.

In the final pinned\-runtime TriviaQA analysis, no\-document, donor and retrieved\-passage F1 are 19\.38, 24\.78 and 29\.88\. Donors improve F1 by 5\.40 points with the original fifteen\-comparison bounds\[4\.64,6\.17\]\[4\.64,6\.17\], while their Match difference is−0\.57​\[−1\.50,0\.37\]\-0\.57\\;\[\-1\.50,0\.37\]\. The retrieved passage exceeds donors on both F1 \(\+5\.10​\[4\.28,5\.92\]\+5\.10\\;\[4\.28,5\.92\]\) and Match \(\+9\.28​\[8\.26,10\.31\]\+9\.28\\;\[8\.26,10\.31\]\)\. All underlying output\-file hashes were checked against the final analysis\. These results support passage\-specific advantages over the chosen donors, alongside gains over no document that do not require the question’s retrieved passage\. Donors are not certified answer\-free\. Neither the F1 differences nor their ratios measure a fraction of information preserved\.

QASPER uses annotated scientific\-paper evidence, whereas the other benchmarks use retrieved passages and different answer distributions\. Its EOS\-based generation also differs from the released repository’s batch\-dependent stopping\. The batch\-size analysis demonstrates sensitivity within released\-model QA, but does not separate output length from padding or numerical changes\. The existing experiments identify a within\-QASPER training effect and within\-benchmark generation/scoring sensitivity\. They do not identify the individual causal contributions of dataset, retrieval quality, answer format, or runtime to a cross\-study difference\. CoQA’s short annotated evidence and upstream training exposure likewise prevent treating its high scores as a matched counterexample to the retrieval experiments\.