TokenMem: Faithful Knowledge Injection for Frozen LLMs

# TokenMem: Faithful Knowledge Injection for Frozen LLMs
Source: [https://arxiv.org/html/2607.22625](https://arxiv.org/html/2607.22625)
Chengzhang Yu1Chenyang Zheng1Zening Lu1Yingru He1 Yutong Huang1Yiming Zhang2Yue Xu1Zhanpeng Jin1

1South China University of Technology2University of Science and Technology of China

###### Abstract

Retrieval\-augmented generation \(RAG\) enhances large language models \(LLMs\) with external knowledge, but suffers from*knowledge conflicts*: when retrieved information contradicts parametric memory, the shared self\-attention pathway produces unpredictable outputs\. We presentTokenMem, a lightweight memory system that injects knowledge into frozen LLMs through a dedicated cross\-attention channel, bypassing competition with parametric memory in the residual stream\.TokenMemtrains only a thin gating adapter \(∼\{\\sim\}3–7M parameters\) via a two\-phase curriculum: first learning general knowledge utilization, then strengthening faithful compliance under counterfactual knowledge\. In controlled experiments on five models spanning three families \(Qwen3\-4B/8B/14B, LLaMA\-3\.1\-8B, OLMo\-3\-7B\),TokenMemachieves 69–70% Knowledge Compliance \(KC\) on counterfactual benchmarks, compared to 20–52% for vanilla RAG—a gap of up to 49 percentage points\. Ablation studies show that the two\-phase curriculum is critical: removing Phase 2 collapses KC to near\-zero\. Mechanistic analysis reveals that the gate adapter learns a conflict\-aware, layer\-specific injection strategy without explicit supervision\. Code is available at[our anonymous repository](https://anonymous.4open.science/r/TokenMem0-D35B/README.md)\.

## 1Introduction

The parametric knowledge of large language models \(LLMs\) is inevitably incomplete and becomes stale as the world evolves\[[16](https://arxiv.org/html/2607.22625#bib.bib27),[33](https://arxiv.org/html/2607.22625#bib.bib28),[18](https://arxiv.org/html/2607.22625#bib.bib30)\]\. In high\-stakes settings such as financial compliance, clinical decision support, or legal advisory, relying on outdated parametric knowledge can produce harmful outputs\[[7](https://arxiv.org/html/2607.22625#bib.bib29),[21](https://arxiv.org/html/2607.22625#bib.bib31)\]\. Retrieval\-augmented generation \(RAG\)\[[19](https://arxiv.org/html/2607.22625#bib.bib14)\]addresses this gap by supplying external evidence at inference time, with the implicit promise that models will*faithfully ground*their outputs in the provided knowledge\. When the retrieved passage is designated as authoritative \(e\.g\., a revised regulatory filing or an updated clinical guideline\), the model should follow it even if its parametric prior disagrees\. We call this property*Knowledge Compliance*\(KC\): given a passage the system treats as authoritative, KC measures how often the model follows it regardless of factual correctness—a*controllability*metric analogous to honoring a database write, distinct from accuracy under noisy or adversarial retrieval\.

Yet this promise breaks down precisely when it matters most\. When retrieved knowledge contradicts the model’s parametric memory—the*knowledge conflict*setting—LLMs exhibit erratic, unpredictable behavior\[[30](https://arxiv.org/html/2607.22625#bib.bib6),[25](https://arxiv.org/html/2607.22625#bib.bib32)\]\. For instance, a model trained on pre\-2024 medical data may ignore a retrieved passage stating a revised drug interaction, instead generating the outdated recommendation stored in its parameters\.Xuet al\.\[[31](https://arxiv.org/html/2607.22625#bib.bib33)\]show that this is not an isolated failure: models exhibit a*systematic bias*toward parametric memory, and larger models with stronger parametric priors show*lower*compliance with external evidence, an inverse scaling phenomenon that undermines the foundational assumption of RAG\. Even when retrieval is accurate, the mere presence of additional context can paradoxically increase hallucination propensity rather than improve faithfulness\[[14](https://arxiv.org/html/2607.22625#bib.bib34)\]\.

We hypothesize that a key contributing factor is*architectural*\. In standard RAG, external knowledge and query tokens share a single self\-attention softmax: the model’s parametric confidence competes with external evidence for a fixed attention budget\. When frozen weights produce strong key\-query alignments among query tokens, attention to knowledge tokens is suppressed—one plausible mechanism behind why larger models are less compliant \(§[3\.1](https://arxiv.org/html/2607.22625#S3.SS1)\)\. This mechanism is*architecturally addressable*: routing external knowledge through a separate softmax eliminates the competition by construction \(Figure[1](https://arxiv.org/html/2607.22625#S3.F1)\)\.

This leads to our central hypothesis:injecting external knowledge through an independent channel that does not share the attention budget with parametric memory should improve knowledge compliance under conflict\.Cross\-attention provides such a channel: it computes attention over external representations under its own normalization, leaving the self\-attention pathway undisturbed\. We test this hypothesis by building a system around cross\-attention injection and evaluating it across five models and three architectural families\.

Based on this reasoning, we presentTokenMem, a lightweight memory system for frozen LLMs\.TokenMemconsists of three components: \(1\) a*TokenMemoryBank*that stores and retrieves knowledge passages via FAISS indexing; \(2\) a*cross\-attention adapter*\(adapted from DecoupledRAG\[[5](https://arxiv.org/html/2607.22625#bib.bib8)\]\) that injects retrieved knowledge through a zero\-initialized gate at every layer of a frozen LLM; and \(3\) a*CoT curriculum SFT*protocol that trains the adapter \(∼\{\\sim\}3–7M parameters\) in two phases, first on knowledge utilization and then on compliance under conflict conditions\.

High KC is the design goal for a controllable memory system: when the user writes a fact into the memory bank, the model should use it regardless of what its parameters encode\.

Our key contributions are:

- •System:TokenMem, a plug\-and\-play memory system for frozen LLMs that achieves high knowledge compliance through cross\-attention injection\. We validate across 5 models spanning 3 architectural families \(Qwen3, LLaMA\-3\.1, OLMo\-3\)\.
- •Training protocol:We show that a two\-phase curriculum \(utilization then compliance\) is necessary: Phase 1 alone collapses KC to near\-zero despite the cross\-attention channel being present\. This demonstrates that the channel provides*capacity*for faithful injection, but explicit counterfactual training is required to activate it\.
- •Empirical observations:We observe inverse scaling in vanilla RAG’s knowledge compliance—larger models show lower compliance, consistent with our architectural hypothesis—and discover through mechanistic analysis that the gate adapter learns a conflict\-aware, layer\-specific injection strategy without explicit supervision\.

## 2Related Work

#### Knowledge conflicts in LLMs\.

When external context contradicts a model’s parametric knowledge, LLMs exhibit unpredictable behavior\[[20](https://arxiv.org/html/2607.22625#bib.bib5)\]\.Xieet al\.\[[30](https://arxiv.org/html/2607.22625#bib.bib6)\]show that models sometimes follow context and sometimes default to parametric memory with no reliable pattern, andChenet al\.\[[4](https://arxiv.org/html/2607.22625#bib.bib7)\]demonstrate that richer knowledge sources exacerbate such conflicts\. These findings motivate two classes of responses: inference\-time strategies that detect and resolve conflicts during generation, and architectural approaches that prevent conflicts by design\.TokenMembelongs to the latter category, providing an independent injection channel that does not compete with parametric representations\.

#### Knowledge augmentation for LLMs\.

Several systems augment LLMs with external memory: KBLaM\[[27](https://arxiv.org/html/2607.22625#bib.bib9)\]injects KB triples via rectangular attention; MemoryLLM\[[28](https://arxiv.org/html/2607.22625#bib.bib10)\]and M\+\[[29](https://arxiv.org/html/2607.22625#bib.bib11)\]maintain self\-updating hidden states but train all parameters; Knowledge Capsules\[[15](https://arxiv.org/html/2607.22625#bib.bib12)\]and FwPKM\[[32](https://arxiv.org/html/2607.22625#bib.bib13),[17](https://arxiv.org/html/2607.22625#bib.bib15)\]organize knowledge as structured tuples or product\-key entries\. These tie knowledge to weights or hidden states, making updates difficult without retraining\. RAG\[[19](https://arxiv.org/html/2607.22625#bib.bib14)\]and REALM\[[9](https://arxiv.org/html/2607.22625#bib.bib22)\]supply knowledge dynamically by prepending passages, but route them through the same self\-attention layers, creating conflicts\. Cross\-attention over retrieved passages offers principled separation\[[11](https://arxiv.org/html/2607.22625#bib.bib23),[3](https://arxiv.org/html/2607.22625#bib.bib24)\]but requires full pretraining\. DecoupledRAG\[[5](https://arxiv.org/html/2607.22625#bib.bib8)\]—the closest precursor—applied cross\-attention to frozen LLMs but evaluates only accuracy on consistent knowledge, not behavior under conflict\. As Table[1](https://arxiv.org/html/2607.22625#S2.T1)shows,TokenMemuniquely combines a frozen backbone, free\-text input, cross\-attention injection, and conflict\-awareness training\.

Table 1:Design\-space comparison of knowledge augmentation approaches\.Frozen: base LLM weights unchanged at deployment;Free\-text: accepts unstructured passages \(vs\. structured triples or latent states\);X\-attn: cross\-attention injection pathway;Conflict: includes training or evaluation explicitly targeting knowledge\-conflict faithfulness\.
#### Knowledge editing\.

ROME\[[22](https://arxiv.org/html/2607.22625#bib.bib16)\]and MEMIT\[[23](https://arxiv.org/html/2607.22625#bib.bib25)\]directly modify model weights to update factual associations, but permanently alter the model and scale poorly to large knowledge bases\.TokenMeminstead injects knowledge at inference time from an external bank, allowing additions, updates, or removals without retraining\.

## 3Method

![Refer to caption](https://arxiv.org/html/2607.22625v1/figures/overview.png)Figure 1:Comparison of knowledge injection pathways\.Left:In standard RAG, knowledge and query tokens share a single self\-attention softmax, where parametric confidence suppresses knowledge attention \(A𝒦≤1A\_\{\\mathcal\{K\}\}\\leq 1\)\.Right:TokenMemroutes knowledge through a separate cross\-attention channel with its own softmax \(∑b=1\\sum b=1always\), then merges via a learned gateWβW\_\{\\beta\}\.TokenMeminjects external knowledge into a frozen LLM through three components \(Figure[1](https://arxiv.org/html/2607.22625#S3.F1)\): a*TokenMemoryBank*that stores and retrieves knowledge passages, a*cross\-attention adapter*that routes retrieved knowledge through a channel separated from self\-attention, and a*two\-phase curriculum*that trains the adapter to faithfully follow injected knowledge even under conflict\. We first formalize why standard RAG creates knowledge conflicts \(§[3\.1](https://arxiv.org/html/2607.22625#S3.SS1)\), then show how cross\-attention avoids the underlying mechanism \(§[3\.2](https://arxiv.org/html/2607.22625#S3.SS2)\), and finally describe the training procedure \(§[3\.3](https://arxiv.org/html/2607.22625#S3.SS3)\)\.

### 3\.1Problem: Softmax Competition in Shared Pathways

Consider a frozen LLM with parametersθ\\thetathat has learned a parametric priorpθ​\(y∣x\)p\_\{\\theta\}\(y\\mid x\)over answersyygiven queryxx\. When an external knowledge passagekkis injected, we are interested in the case wherekksupports a different answer than the parametric prior, i\.e\., the knowledge*conflicts*with what the model “believes\.” We call such a sample*conflicting*: formally, a sample wherearg⁡maxy⁡pθ​\(y∣x\)≠yk\\arg\\max\_\{y\}p\_\{\\theta\}\(y\\mid x\)\\neq y\_\{k\}, withyky\_\{k\}being the passage\-supported answer\. We defineKnowledge Compliance \(KC\)as the fraction of conflicting samples on which the model’s greedy output matchesyky\_\{k\}\(evaluation details in §[4\.1](https://arxiv.org/html/2607.22625#S4.SS1)\)\.

In standard RAG,kkis concatenated into the input:x′=\[k;x\]x^\{\\prime\}=\[k;\\,x\]\. The model processesx′x^\{\\prime\}through self\-attention, where knowledge tokens𝒦=\{k1,…,kM\}\\mathcal\{K\}=\\\{k\_\{1\},\\ldots,k\_\{M\}\\\}and query tokens𝒳=\{x1,…,xN\}\\mathcal\{X\}=\\\{x\_\{1\},\\ldots,x\_\{N\}\\\}share a single softmax\. Let𝐤i\\mathbf\{k\}\_\{i\}denote the key vector at positionii\(regardless of whetheriibelongs to𝒦\\mathcal\{K\}or𝒳\\mathcal\{X\}\)\. For query positionnn, the attention weight assigned to knowledge tokenjjis:111For clarity, we show the fully visible \(bidirectional\) case\. Under causal masking the sums restrict to positions≤n\\leq n; the competition argument still holds whenever query\-side visible mass is positive\.

an,j=exp⁡\(𝐪n⊤​𝐤j/dh\)∑i∈𝒦exp⁡\(𝐪n⊤​𝐤i/dh\)⏟S𝒦\+∑i∈𝒳exp⁡\(𝐪n⊤​𝐤i/dh\)⏟S𝒳\\displaystyle a\_\{n,j\}=\\frac\{\\exp\(\\mathbf\{q\}\_\{n\}^\{\\top\}\\mathbf\{k\}\_\{j\}/\\sqrt\{d\_\{h\}\}\)\}\{\\underbrace\{\\textstyle\\sum\_\{i\\in\\mathcal\{K\}\}\\exp\(\\mathbf\{q\}\_\{n\}^\{\\top\}\\mathbf\{k\}\_\{i\}/\\sqrt\{d\_\{h\}\}\)\}\_\{S\_\{\\mathcal\{K\}\}\}\+\\underbrace\{\\textstyle\\sum\_\{i\\in\\mathcal\{X\}\}\\exp\(\\mathbf\{q\}\_\{n\}^\{\\top\}\\mathbf\{k\}\_\{i\}/\\sqrt\{d\_\{h\}\}\)\}\_\{S\_\{\\mathcal\{X\}\}\}\}\(1\)For a given query positionnn, the total attention mass allocated to external knowledge isA𝒦\(n\)=S𝒦/\(S𝒦\+S𝒳\)≤1A\_\{\\mathcal\{K\}\}^\{\(n\)\}=S\_\{\\mathcal\{K\}\}/\(S\_\{\\mathcal\{K\}\}\+S\_\{\\mathcal\{X\}\}\)\\leq 1, with strict inequality wheneverS𝒳\>0S\_\{\\mathcal\{X\}\}\>0\. For fixed knowledge\-token scoresS𝒦S\_\{\\mathcal\{K\}\},A𝒦\(n\)A\_\{\\mathcal\{K\}\}^\{\(n\)\}*decreases*asS𝒳S\_\{\\mathcal\{X\}\}grows: when the frozen weights produce strong key\-query alignments among non\-knowledge tokens, the shared softmax suppresses the fraction of attention allocated to external knowledge\. This provides one mechanism that can explain the unreliable knowledge utilization observed empirically\[[20](https://arxiv.org/html/2607.22625#bib.bib5),[30](https://arxiv.org/html/2607.22625#bib.bib6)\]: the model’s parametric prior competes with external evidence for a fixed attention budget\.

This competition not only reduces the fraction of attention allocated to knowledge, but also suppresses its actual contribution to the representation\. Let the knowledge context vector be:

cnrag=∑j∈𝒦an,j​vj\\displaystyle c\_\{n\}^\{\\text\{rag\}\}=\\sum\_\{j\\in\\mathcal\{K\}\}a\_\{n,j\}v\_\{j\}\(2\)
Assuming the value vectors are bounded \(i\.e\.,‖vj‖≤C\\\|v\_\{j\}\\\|\\leq C\), by the triangle inequality we have:

‖cnrag‖=‖∑j∈𝒦an,j​vj‖≤∑j∈𝒦an,j​‖vj‖≤S𝒦S𝒦\+S𝒳⋅C\\displaystyle\\\|c\_\{n\}^\{\\text\{rag\}\}\\\|=\\left\\\|\\sum\_\{j\\in\\mathcal\{K\}\}a\_\{n,j\}v\_\{j\}\\right\\\|\\leq\\sum\_\{j\\in\\mathcal\{K\}\}a\_\{n,j\}\\\|v\_\{j\}\\\|\\leq\\frac\{S\_\{\\mathcal\{K\}\}\}\{S\_\{\\mathcal\{K\}\}\+S\_\{\\mathcal\{X\}\}\}\\cdot C\(3\)
In well\-trained large language models, parametric memory typically induces strong token\-to\-token associations, causing the attention scores between the query and its own context \(S𝒳S\_\{\\mathcal\{X\}\}\) to dominate those with external knowledge tokens \(S𝒦S\_\{\\mathcal\{K\}\}\), i\.e\.,S𝒳≫S𝒦S\_\{\\mathcal\{X\}\}\\gg S\_\{\\mathcal\{K\}\}\. This effect is particularly pronounced when internal knowledge conflicts with external evidence\. Under this regime, the bound further tightens:

‖cnrag‖≤S𝒦S𝒦\+S𝒳⋅C⟶0asS𝒳≫S𝒦\\displaystyle\\\|c\_\{n\}^\{\\text\{rag\}\}\\\|\\leq\\frac\{S\_\{\\mathcal\{K\}\}\}\{S\_\{\\mathcal\{K\}\}\+S\_\{\\mathcal\{X\}\}\}\\cdot C\\longrightarrow 0\\quad\\text\{as\}\\quad S\_\{\\mathcal\{X\}\}\\gg S\_\{\\mathcal\{K\}\}\(4\)
This shows that, in regimes where the parametric prior dominates, standard RAG effectively compresses the representation of external knowledge toward zero, thereby limiting the model’s ability to utilize retrieved information\.

### 3\.2Separating Knowledge from the Shared Softmax

Cross\-attention addresses this by giving external knowledge its own softmax normalization\. Instead of sharing the attention budget with query tokens, knowledge tokens are attended to through a separate operation\. Let𝐤j𝐦,ℓ=WKℓ​LN⁡\(𝐦jℓ\)\\mathbf\{k\}\_\{j\}^\{\\mathbf\{m\},\\ell\}=W\_\{K\}^\{\\ell\}\\operatorname\{LN\}\(\\mathbf\{m\}\_\{j\}^\{\\ell\}\)denote the projected memory key at positionjjin layerℓ\\ell, wherej∈\{1,…,Lm\}j\\in\\\{1,\\ldots,L\_\{m\}\\\}index the tokens of the retrieved passage, andLmL\_\{m\}is the number of tokens in that passage\. Then:

bn,j=exp⁡\(𝐪n⊤​𝐤j𝐦,ℓ/dh\)∑i=1Lmexp⁡\(𝐪n⊤​𝐤i𝐦,ℓ/dh\)\\displaystyle b\_\{n,j\}=\\frac\{\\exp\(\\mathbf\{q\}\_\{n\}^\{\\top\}\\mathbf\{k\}\_\{j\}^\{\\mathbf\{m\},\\ell\}/\\sqrt\{d\_\{h\}\}\)\}\{\\sum\_\{i=1\}^\{L\_\{m\}\}\\exp\(\\mathbf\{q\}\_\{n\}^\{\\top\}\\mathbf\{k\}\_\{i\}^\{\\mathbf\{m\},\\ell\}/\\sqrt\{d\_\{h\}\}\)\}\(5\)The total attention mass on knowledge is now∑jbn,j=1\\sum\_\{j\}b\_\{n,j\}=1, regardless ofS𝒳S\_\{\\mathcal\{X\}\}\. This removes the specific failure mode identified in §[3\.1](https://arxiv.org/html/2607.22625#S3.SS1): the model’s parametric confidence can no longer suppress attention on knowledge tokens through the shared softmax\.

Unlike the shared\-softmax formulation in RAG, cross\-attention normalizes over knowledge tokens independently\. Let the resulting knowledge context vector be:

cnmem=∑j=1Lmbn,j​vjm\\displaystyle c\_\{n\}^\{\\text\{mem\}\}=\\sum\_\{j=1\}^\{L\_\{m\}\}b\_\{n,j\}v\_\{j\}^\{m\}\(6\)
Assuming the value vectors are bounded \(i\.e\.,‖vjm‖≤C\\\|v\_\{j\}^\{m\}\\\|\\leq C\), we obtain:

‖cnmem‖=‖∑j=1Lmbn,j​vjm‖≤∑j=1Lmbn,j​‖vjm‖≤C\\displaystyle\\\|c\_\{n\}^\{\\text\{mem\}\}\\\|=\\left\\\|\\sum\_\{j=1\}^\{L\_\{m\}\}b\_\{n,j\}v\_\{j\}^\{m\}\\right\\\|\\leq\\sum\_\{j=1\}^\{L\_\{m\}\}b\_\{n,j\}\\\|v\_\{j\}^\{m\}\\\|\\leq C\(7\)
Crucially, this bound does not depend on the query\-side attention massS𝒳S\_\{\\mathcal\{X\}\}, and therefore does not diminish even when the parametric prior is strong \(i\.e\.,S𝒳≫S𝒦S\_\{\\mathcal\{X\}\}\\gg S\_\{\\mathcal\{K\}\}\)\. In contrast to standard RAG \(Eq\.[1](https://arxiv.org/html/2607.22625#S3.E1)\), where the knowledge contribution vanishes asS𝒳S\_\{\\mathcal\{X\}\}grows, the cross\-attention pathway ensures that external knowledge retains a non\-vanishing presence in the representation\.

The two pathways combine via gated residual addition\. Let𝐡~ℓ=Blockℓ​\(𝐡ℓ\)\\tilde\{\\mathbf\{h\}\}^\{\\ell\}=\\text\{Block\}^\{\\ell\}\(\\mathbf\{h\}^\{\\ell\}\)denote the output of the standard transformer block\. The cross\-attention adapter is applied after the complete block:

𝐡outℓ=𝐡~ℓ\+Wβℓ​CrossAttn​\(𝐡~ℓ,𝐦ℓ\)\\displaystyle\\mathbf\{h\}^\{\\ell\}\_\{\\text\{out\}\}=\\tilde\{\\mathbf\{h\}\}^\{\\ell\}\+W\_\{\\beta\}^\{\\ell\}\\,\\text\{CrossAttn\}\(\\tilde\{\\mathbf\{h\}\}^\{\\ell\},\\mathbf\{m\}^\{\\ell\}\)\(8\)whereWβℓ∈ℝd×dW\_\{\\beta\}^\{\\ell\}\\in\\mathbb\{R\}^\{d\\times d\}is a learnable gating matrix providing*explicit, learned*control over how much knowledge signal enters the residual stream, unlike RAG where the balance is determined implicitly by frozen attention weights\.

![Refer to caption](https://arxiv.org/html/2607.22625v1/x1.png)Figure 2:Overview of theTokenMempipeline\.❶ Memory Bank Construction & Retrieval: knowledge passages are encoded and deposited into a FAISS\-indexed TokenMemoryBank; at inference a query retrieves the top\-1 passage to produce layer\-wise memory representations𝐦ℓ\\mathbf\{m\}^\{\\ell\}\.❷ Cross\-Attention Injection: at each transformer layer, a cross\-attention adapter reads from𝐦ℓ\\mathbf\{m\}^\{\\ell\}\(as keys and values\) and injects the result into the residual stream through a zero\-initialized LoRA gateWβW\_\{\\beta\}; only the gate parameters are trained \(∼\{\\sim\}3–7M\), all other components remain frozen\.❸ Two\-Phase Curriculum: Phase 1 trains on knowledge\-grounded QA to establish cross\-attention routing; Phase 2 adds counterfactual data to enforce faithful compliance under knowledge conflict\.TokenMemimplements this principle as a lightweight adapter for frozen LLMs \(Figure[2](https://arxiv.org/html/2607.22625#S3.F2)\)\. Knowledge is stored as tokenized passages in a FAISS\-indexed bank\[[13](https://arxiv.org/html/2607.22625#bib.bib20)\]; given a query, the bank retrieves the top\-1 passage by embedding cosine similarity\. The retrieved passage is encoded through the frozen LLM in a separate forward pass to produce layer\-wise memory representations𝐦ℓ∈ℝLm×d\\mathbf\{m\}^\{\\ell\}\\in\\mathbb\{R\}^\{L\_\{m\}\\times d\}, which are layer\-normalized before serving as keys and values\. Following DecoupledRAG\[[5](https://arxiv.org/html/2607.22625#bib.bib8)\], the frozen LLM’s ownWQ,WK,WVW\_\{Q\},W\_\{K\},W\_\{V\}projections are reused for cross\-attention, and the result is gated through a LoRA\-style low\-rank transform\[[10](https://arxiv.org/html/2607.22625#bib.bib1)\]:

Wβℓ=α⋅Aβℓ​Bβℓ,Aβℓ∈ℝd×r,Bβℓ∈ℝr×d\\displaystyle W\_\{\\beta\}^\{\\ell\}=\\alpha\\cdot A\_\{\\beta\}^\{\\ell\}B\_\{\\beta\}^\{\\ell\},\\quad A\_\{\\beta\}^\{\\ell\}\\\!\\in\\\!\\mathbb\{R\}^\{d\\times r\},\\;B\_\{\\beta\}^\{\\ell\}\\\!\\in\\\!\\mathbb\{R\}^\{r\\times d\}\(9\)wherer=16r\\\!=\\\!16,α=32\\alpha\\\!=\\\!32,AβℓA\_\{\\beta\}^\{\\ell\}is Gaussian\-initialized \(σ=0\.01\\sigma\\\!=\\\!0\.01\), andBβℓB\_\{\\beta\}^\{\\ell\}is zero\-initialized so thatWβℓ=𝟎W\_\{\\beta\}^\{\\ell\}=\\mathbf\{0\}at the start of training; the adapter has no effect on the frozen model until it learns to incorporate external knowledge\. A knowledge dropout of0\.20\.2is applied to the cross\-attention output before gating\. Each transformer layer receives one adapter, applied after the full block \(self\-attention residual \+ MLP residual\); there are no shared parameters across layers\. Only\{Aβℓ,Bβℓ\}ℓ=1L\\\{A\_\{\\beta\}^\{\\ell\},B\_\{\\beta\}^\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}are trained \(∼\{\\sim\}3–7M parameters for 4B–14B models\); the base LLM remains entirely frozen\.

### 3\.3Curriculum Training for Faithful Compliance

The architecture provides the*capacity*for faithful knowledge injection, but the gate adapter must still be*trained*to use it\. This requires two capabilities:*knowledge utilization*\(routing cross\-attention signals into generation\) and*faithful compliance*\(following injected knowledge even when it conflicts with parametric memory\)\. Training both simultaneously led to unstable optimization: the utilization gradient conflicts with the compliance gradient \(which must override parametric priors\), resulting in substantially higher validation loss than staged training \(§[4\.4](https://arxiv.org/html/2607.22625#S4.SS4)\)\. We therefore adopt a two\-phase curriculum\.

Letϕ=\{Aβℓ,Bβℓ\}ℓ=1L\\phi=\\\{A\_\{\\beta\}^\{\\ell\},B\_\{\\beta\}^\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}denote the trainable parameters\. Both phases optimize the same objective, standard next\-token prediction conditioned on external knowledge\. Given a target sequencey=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\):

ℒ​\(ϕ\)=−∑t=1T−1log⁡p​\(yt\+1∣y1:t,𝐦,x;θ,ϕ\)\\displaystyle\\mathcal\{L\}\(\\phi\)=\-\\sum\_\{t=1\}^\{T\-1\}\\log p\(y\_\{t\+1\}\\mid y\_\{1:t\},\\;\\mathbf\{m\},\\;x;\\;\\theta,\\;\\phi\)\(10\)whereθ\\thetais the frozen LLM and𝐦=\{𝐦ℓ\}ℓ=1L\\mathbf\{m\}=\\\{\\mathbf\{m\}^\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}denotes the collection of layer\-wise memory representations\. The phases differ only in the training data\.Phase 1trains on𝒟correct\\mathcal\{D\}\_\{\\text\{correct\}\}: 50K CoT question\-answer pairs from a news domain, where every answer requires the injected passage\. This teaches a single, unambiguous skill \(route information from the cross\-attention channel into generation\) with no conflicting signal, since every sample’s correct answer depends on the external knowledge\.Phase 2continues from the Phase 1 checkpointϕ∗\\phi^\{\*\}, adding counterfactual data: a mixture of𝒟correct\\mathcal\{D\}\_\{\\text\{correct\}\}and𝒟cf\\mathcal\{D\}\_\{\\text\{cf\}\}\(oversampled2×2\\times\), where each counterfactual sample pairs a question with a passage supporting an answer that is*incorrect with respect to world knowledge*but designated as authoritative for the KC objective\. The model is supervised to output the passage\-supported answer, training it to defer to the injected source even when its parametric memory disagrees\. Becauseϕ∗\\phi^\{\*\}has already learned to route through cross\-attention, the counterfactual signal only needs to strengthen compliance rather than simultaneously establishing basic routing\. This is why staged training succeeds where joint training fails\. We ablate this design in §[4\.4](https://arxiv.org/html/2607.22625#S4.SS4)\.

The counterfactual data \(cf\_arc\_easy, 2,745 items;cf\_medqa, 1,146 items\) is LLM\-generated with minimal\-edit principles \(Appendix[B](https://arxiv.org/html/2607.22625#A2)\); Phase 2 uses only training splits, and all reported KC results use held\-out test data\.

## 4Experiments

### 4\.1Setup

#### Evaluation protocol\.

We measure Knowledge Compliance \(KC\) as defined in §[3\.1](https://arxiv.org/html/2607.22625#S3.SS1)\. We evaluate via chain\-of\-thought \(CoT\) generation, which better reflects real deployment behavior than log\-probability scoring, and extract answers via regex matching on the final “The answer is \(X\)” pattern; extraction failures \(<<3% across all conditions\) are counted as incorrect\.

#### Models and datasets\.

We evaluate five models spanning three architectural families: Qwen3\-4B, Qwen3\-8B, and Qwen3\-14B\[[1](https://arxiv.org/html/2607.22625#bib.bib17)\]; LLaMA\-3\.1\-8B\[[26](https://arxiv.org/html/2607.22625#bib.bib18)\]; and OLMo\-3\-7B\[[8](https://arxiv.org/html/2607.22625#bib.bib19)\], covering model sizes from 4B to 14B parameters\. We use five evaluation datasets:News\(8,663 items\),MMLU\(14,042 items\),MedQA\(1,273 items\),cf\_arc\_easy\(2,745 items\), andcf\_medqa\(1,146 items\)\. Training uses only News\-domain CoT data \(50K items\); all other datasets test generalization\.

#### Methods and training\.

We compare three conditions: \(1\)No\-Memory: base LLM without external knowledge; \(2\)VanillaRAG: knowledge compressed to 64 tokens via LLMLingua\-2\[[12](https://arxiv.org/html/2607.22625#bib.bib21)\]and prepended to input; \(3\)TokenMem: knowledge injected via cross\-attention with gate adapter trained via curriculum \(∼\{\\sim\}3–7M params\)\. All trained methods use Lamb optimizer withlr=10−3\\text\{lr\}=10^\{\-3\}, batch size 2, gradient accumulation 16, and max sequence length 1024\. Phase 1 runs for 3 epochs on News CoT data; Phase 2 runs for 40 epochs on News \+ counterfactual mixed data, selecting the checkpoint with the lowest validation loss\. Knowledge passages are capped at 256 tokens forTokenMem\.

### 4\.2Main Results

Table 2:Accuracy \(%\) on normal datasets and Knowledge Compliance \(KC, %\) under counterfactual knowledge across five models and three families\.BestKC per model is highlighted\.Table[2](https://arxiv.org/html/2607.22625#S4.T2)reports results across all conditions\. On normal datasets,TokenMemconsistently improves overNo\-Memory\(\+18\.5 to \+41\.4ppon News\) but generally trailsVanillaRAG\. The Qwen family shows strong OOD transfer \(e\.g\., Qwen3\-8B: \+8\.8ppon MMLU, \+16\.6ppon MedQA\), while two non\-Qwen models show regressions on specific OOD datasets \(§[5](https://arxiv.org/html/2607.22625#S5)\)\.

### 4\.3Knowledge Compliance under Conflict

The right columns of Table[2](https://arxiv.org/html/2607.22625#S4.T2)report KC under counterfactual injection\.

On cf\_arc\_easy,TokenMemoutperformsVanillaRAGacross all five models, with KC gains ranging from \+14\.6pp\(OLMo\) to \+64\.4pp\(Qwen3\-14B\)\. On cf\_medqa, four of five models show substantial improvements; the exception is LLaMA\-3\.1\-8B \(27\.6% vs\. 45\.4% forVanillaRAG\), where the gate adapter struggles with medical terminology absent from the News training domain\.

Within the Qwen family,No\-MemoryKC decreases monotonically with scale \(e\.g\., 1\.2%→\\to0\.4% on cf\_arc\_easy\), consistent with stronger parametric priors resisting counterfactual answers\.VanillaRAGKC remains flat across sizes \(19–20% on cf\_arc\_easy\), suggesting limited compliance regardless of scale\.TokenMemKC, by contrast, does not degrade: Qwen3\-14B achieves the highest KC \(83\.5%/84\.6%\), suggesting the cross\-attention channel can overcome strong priors when adequately trained\.

### 4\.4Ablation Studies

We conduct two ablation studies on Qwen3\-8B to understand which components ofTokenMemcontribute to faithful knowledge utilization\.

Table 3:Ablation A1 on Qwen3\-8B: curriculum training necessity\. Phase 2 \(counterfactual mixing\) substantially improves KC without degrading normal\-dataset accuracy\.Curriculum necessity \(A1\)\.Phase 2 counterfactual mixing is essential for KC under conflict \(Table[3](https://arxiv.org/html/2607.22625#S4.T3)\)\. With Phase 1 alone \(CoT SFT on News\), KC collapses to 2\.4% on cf\_arc\_easy \(−37\.8\-37\.8pp\) and 20\.0% on cf\_medqa \(−41\.5\-41\.5pp\), while News accuracy drops modestly \(81\.2% vs\. 90\.2%,−9\.0\-9\.0pp\)\. The cross\-attention channel thus provides the*capacity*for faithful injection, but explicit counterfactual training is required to*activate*this capacity under conflict\.

![Refer to caption](https://arxiv.org/html/2607.22625v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.22625v1/x3.png)

Figure 3:Left:KC by parametric\-prior strength on Qwen3\-8B\. Green intervals markTokenMem’s absolute KC gain overVanillaRAG\. The advantage is largest in the*Strong\-prior*group\.Right:Layer\-wise gate activation on the 100 sample pairs with the largest conflict–non\-conflict divergence \(Qwen3\-8B\)\. Top: gate relative ratio under conflict \(red\) and non\-conflict \(blue\); bottom: per\-layer difference \(Δ\\Delta\); peaks at layers 10, 12, and 19 coincide with mid\-layers linked to factual recall\[[22](https://arxiv.org/html/2607.22625#bib.bib16)\]\.Conflict\-conditioned analysis \(A2\)\.We partition counterfactual questions by parametric\-prior strength \(Figure[3](https://arxiv.org/html/2607.22625#S4.F3)\): questions that all three Qwen base models answer correctly without external knowledge form the*Strong\-prior*group, mixed results form*Mixed*, and all\-fail form*Weak\-prior*\.TokenMem’s KC advantage overVanillaRAGis largest in the Strong\-prior group \(\+19\.1ppon cf\_arc\_easy, \+15\.7ppon cf\_medqa\) and narrows as prior strength decreases\. This is consistent with the hypothesis that the cross\-attention channel’s benefit stems from avoiding competition with parametric confidence: when the prior is weak, in\-context injection suffices; when strong, only the independent channel maintains compliance\.

### 4\.5Mechanistic Analysis

We measure the layer\-wise gate contribution norm‖Wβℓ⋅CrossAttn‖/‖𝐡~ℓ‖\\\|W\_\{\\beta\}^\{\\ell\}\\cdot\\text\{CrossAttn\}\\\|/\\\|\\tilde\{\\mathbf\{h\}\}^\{\\ell\}\\\|on Qwen3\-8B, comparing the same questions under counterfactual vs\. factual passages \(Figure[3](https://arxiv.org/html/2607.22625#S4.F3), right; 100 pairs with largest activation difference\)\. Gate activation increases under conflict in 31/36 layers \(34 significant atp<0\.05p<0\.05, pairedtt\-test\), with peaks at layers 10–12 and 19–23—mid\-layers linked to factual recall\[[22](https://arxiv.org/html/2607.22625#bib.bib16),[23](https://arxiv.org/html/2607.22625#bib.bib25)\]—suggesting the adapter learns a conflict\-aware, layer\-specific injection strategy\.

## 5Discussion

#### When to useTokenMemvs\. RAG\.

Our results reveal a clear design tradeoff\.VanillaRAGmaximizes accuracy when the knowledge source is correct and conflicts are rare—suitable for standard QA over trusted knowledge bases\.TokenMemmaximizes faithfulness when the knowledge source must be followed reliably regardless of parametric priors—suitable for high\-stakes domains \(medical, legal, regulatory\) where knowledge compliance is non\-negotiable, or for knowledge editing scenarios where the injected information intentionally supersedes the model’s prior beliefs\.

#### Faithfulness is not gullibility\.

Knowledge in a TokenMemoryBank is*user\-authorized*\(analogous to a database write\), not adversarial injection; high compliance is the design goal for a controllable knowledge channel\. Under correct knowledge, this compliance manifests as accuracy improvement \(\+41\.4ppon News\)\.

#### Why not LoRA\-SFT RAG?

A natural question is whether fine\-tuning the base model via LoRA to better follow in\-context knowledge could matchTokenMem’s KC without a separate channel\. We deliberately compare against untrainedVanillaRAG, following the established protocol in knowledge conflict literature where vanilla in\-context injection is the standard baseline\[[20](https://arxiv.org/html/2607.22625#bib.bib5),[4](https://arxiv.org/html/2607.22625#bib.bib7),[30](https://arxiv.org/html/2607.22625#bib.bib6)\]\. More fundamentally, LoRA\-SFT operates in a different design regime: it modifies the base model’s weights, sacrificing the frozen\-backbone guarantee that allowsTokenMem’s adapter to be added or removed without side effects\. Recent work shows that fine\-tuning on new knowledge linearly increases hallucination rate as the proportion of novel facts grows\[[6](https://arxiv.org/html/2607.22625#bib.bib35)\], and that instruction\-tuned models exhibit systematic sycophancy—blindly agreeing with user\-provided context rather than genuinely integrating it\[[24](https://arxiv.org/html/2607.22625#bib.bib36)\]\. LoRA further faces a learning–forgetting tradeoff: its low\-rank constraint limits what can be learned, while increasing rank risks degrading pre\-trained capabilities\[[2](https://arxiv.org/html/2607.22625#bib.bib37)\]\. Elevated KC from LoRA\-SFT may therefore reflect superficial compliance rather than the structured knowledge integration thatTokenMem’s independent cross\-attention channel provides\.

#### Limitations\.

The faithfulness gains come at the cost of lower accuracy on correct knowledge \(85\.3% vs\. 95\.4% forVanillaRAGon News\)\.TokenMemrequires curriculum training, though the adapter is small \(∼\{\\sim\}3–7M parameters\) and trains once\. The News\-only curriculum transfers cross\-domain \(e\.g\., \+8\.8ppon MMLU for Qwen3\-8B\), but two non\-Qwen models show OOD regressions, suggesting multi\-domain curricula may improve robustness\.

## 6Conclusion

We presentedTokenMem, a lightweight cross\-attention adapter that enables frozen LLMs to faithfully utilize externally injected knowledge\. Through controlled experiments across five models and three architectural families, we showed that cross\-attention injection combined with curriculum training achieves substantially higher knowledge compliance than standard in\-context RAG\.TokenMemachieves Knowledge Compliance of 69–70% under counterfactual conditions on Qwen3\-4B, dramatically outperformingVanillaRAG\(20–52%\)\. Ablation studies confirm that the two\-phase curriculum is critical: Phase 1 alone collapses KC to near\-zero, demonstrating that explicit counterfactual training is required to activate faithful compliance\. Mechanistic analysis reveals that the gate adapter learns a conflict\-aware, layer\-specific injection strategy without explicit supervision\.

#### Future work\.

Several directions remain open: \(1\) multi\-domain curriculum training to narrow the accuracy gap on normal datasets, \(2\) scaling to larger models \(30B\+\) where parametric priors are even stronger, and \(3\) extending the framework to support dynamic knowledge updates without retraining the adapter\.

## References

- \[1\]\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§4\.1](https://arxiv.org/html/2607.22625#S4.SS1.SSS0.Px2.p1.1)\.
- \[2\]D\. Biderman, J\. Portes, J\. J\. G\. Ortiz, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle,et al\.LoRA learns less and forgets less\.Transactions on Machine Learning Research\.Cited by:[§5](https://arxiv.org/html/2607.22625#S5.SS0.SSS0.Px3.p1.1)\.
- \[3\]S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. van den Driessche, J\. Lespiau, B\. Damoc, A\. Clark,et al\.\(2022\)Improving language models by retrieving from trillions of tokens\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 2206–2240\.Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[4\]H\. Chen, M\. J\.Q\. Zhang, and E\. Choi\(2022\)Rich knowledge sources bring complex knowledge conflicts: recalibrating models to reflect conflicting evidence\.InProceedings of EMNLP,Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.22625#S5.SS0.SSS0.Px3.p1.1)\.
- \[5\]Q\. Dong, Q\. Ai, H\. Wang, Y\. Liu, H\. Li, W\. Su, Y\. Liu, T\. Chua, and S\. Ma\(2025\)Decoupling knowledge and context: an efficient and effective retrieval augmented generation framework via cross attention\.InProceedings of The Web Conference \(WWW\),Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p5.1),[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2607.22625#S3.SS2.p6.2)\.
- \[6\]Z\. Gekhman, G\. Yona, R\. Aharoni, M\. Eyal, A\. Feder, R\. Reichart, and J\. Herzig\(2024\)Does fine\-tuning llms on new knowledge encourage hallucinations?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 7765–7784\.Cited by:[§5](https://arxiv.org/html/2607.22625#S5.SS0.SSS0.Px3.p1.1)\.
- \[7\]E\. J\. Gong, C\. S\. Bang, J\. J\. Lee, and G\. H\. Baik\(2025\)Knowledge\-practice performance gap in clinical large language models: systematic review of 39 benchmarks\.Journal of Medical Internet Research27,pp\. e84120\.External Links:[Document](https://dx.doi.org/10.2196/84120),[Link](https://www.jmir.org/2025/1/e84120)Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p1.1)\.
- \[8\]D\. Groeneveld, I\. Beltagy, E\. Walsh, A\. Bhagia, R\. Kinney, O\. Tafjord, A\. Jha, H\. Ivison, I\. Magnusson, Y\. Wang, S\. Arora, D\. Atkinson, R\. Authur, K\. Chandu, A\. Cohan, J\. Dumas, Y\. Elazar, Y\. Gu, J\. Hessel, T\. Khot, W\. Merrill, J\. Morrison, N\. Muennighoff, A\. Naik, C\. Nam, M\. Peters, V\. Pyatkin, A\. Ravichander, D\. Schwenk, S\. Shah, W\. Smith, E\. Strubell, N\. Subramani, M\. Wortsman, P\. Dasigi, N\. Lambert, K\. Richardson, L\. Zettlemoyer, J\. Dodge, K\. Lo, L\. Soldaini, N\. Smith, and H\. Hajishirzi\(2024\)OLMo: accelerating the science of language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15789–15809\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.841),[Link](https://aclanthology.org/2024.acl-long.841/)Cited by:[§4\.1](https://arxiv.org/html/2607.22625#S4.SS1.SSS0.Px2.p1.1)\.
- \[9\]K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang\(2020\-13–18 Jul\)REALM: retrieval\-augmented language model pre\-training\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 3929–3938\.Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InProceedings of ICLR,Cited by:[§3\.2](https://arxiv.org/html/2607.22625#S3.SS2.p6.2)\.
- \[11\]G\. Izacard and E\. Grave\(2021\)Leveraging passage retrieval with generative models for open domain question answering\.InProceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume,pp\. 874–880\.Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[12\]H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. Qiu\(2023\)LLMLingua: compressing prompts for accelerated inference of large language models\.InProceedings of EMNLP,Cited by:[§4\.1](https://arxiv.org/html/2607.22625#S4.SS1.SSS0.Px3.p1.2)\.
- \[13\]J\. Johnson, M\. Douze, and H\. Jégou\(2021\)Billion\-scale similarity search with GPUs\.IEEE Transactions on Big Data7\(3\),pp\. 535–547\.External Links:[Document](https://dx.doi.org/10.1109/TBDATA.2019.2921572)Cited by:[§3\.2](https://arxiv.org/html/2607.22625#S3.SS2.p6.2)\.
- \[14\]H\. Joren, J\. Zhang, C\. Ferng, D\. Juan, A\. Taly, and C\. Rashtchian\(2025\)Sufficient context: a new lens on retrieval augmented generation systems\.InProceedings of ICLR,Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p2.1)\.
- \[15\]B\. Ju, S\. Weng, D\. Zhou, R\. Xu, and K\. Su\(2026\)Knowledge capsules: structured nonparametric memory units for LLMs\.arXiv preprint arXiv:2604\.20487\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.20487),[Link](https://arxiv.org/abs/2604.20487)Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[16\]J\. Kasai, K\. Sakaguchi, Y\. Takahashi, R\. Le Bras, A\. Asai, X\. Yu, D\. Radev, N\. A\. Smith, Y\. Choi, and K\. Inui\(2023\)RealTime QA: what’s the answer right now?\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p1.1)\.
- \[17\]G\. Lample, A\. Sablayrolles, M\. Ranzato, L\. Denoyer, and H\. Jégou\(2019\)Large memory layers with product keys\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[18\]A\. Lazaridou, A\. Kuncoro, E\. Gribovskaya, D\. Agrawal, A\. Liska, T\. Terzi, M\. Gimenez, C\. de Masson d’Autume, T\. Kocisky, S\. Ruder, D\. Yogatama, K\. Cao, S\. Young, and P\. Blunsom\(2021\)Mind the gap: assessing temporal generalization in neural language models\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 29348–29363\.Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p1.1)\.
- \[19\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p1.1),[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]S\. Longpre, K\. Perisetla, A\. Chen, N\. Ramesh, C\. DuBois, and S\. Singh\(2021\)Entity\-based knowledge conflicts in question answering\.InProceedings of EMNLP,Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2607.22625#S3.SS1.p2.18),[§5](https://arxiv.org/html/2607.22625#S5.SS0.SSS0.Px3.p1.1)\.
- \[21\]K\. Luu, D\. Khashabi, S\. Gururangan, K\. Mandyam, and N\. A\. Smith\(2022\)Time waits for no one\! analysis and challenges of temporal misalignment\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5944–5958\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.435),[Link](https://aclanthology.org/2022.naacl-main.435/)Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p1.1)\.
- \[22\]K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov\(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px3.p1.1),[Figure 3](https://arxiv.org/html/2607.22625#S4.F3),[§4\.5](https://arxiv.org/html/2607.22625#S4.SS5.p1.3)\.
- \[23\]K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau\(2023\)Mass\-editing memory in a transformer\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px3.p1.1),[§4\.5](https://arxiv.org/html/2607.22625#S4.SS5.p1.3)\.
- \[24\]M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman, N\. Cheng, E\. Durmus, Z\. Hatfield\-Dodds, S\. R\. Johnston, S\. Kravec, T\. Maxwell, S\. McCandlish, K\. Ndousse, O\. Rauber, N\. Schiefer, D\. Yan, M\. Zhang, and E\. Perez\(2024\)Towards understanding sycophancy in language models\.InProceedings of ICLR,Cited by:[§5](https://arxiv.org/html/2607.22625#S5.SS0.SSS0.Px3.p1.1)\.
- \[25\]Z\. Su, J\. Zhang, X\. Qu, T\. Zhu, Y\. Li, J\. Sun, J\. Li, M\. Zhang, and Y\. Cheng\(2024\)ConflictBank: a benchmark for evaluating the influence of knowledge conflicts in LLMs\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 103242–103268\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/baf4b960d118f838ad0b2c08247a9ebe-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p2.1)\.
- \[26\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4\.1](https://arxiv.org/html/2607.22625#S4.SS1.SSS0.Px2.p1.1)\.
- \[27\]X\. Wang, T\. Isazawa, L\. Mikaelyan, and J\. Hensman\(2025\)KBLaM: knowledge base augmented language model\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=aLsMzkTej9)Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]Y\. Wang, Y\. Gao, X\. Chen, H\. Jiang, S\. Li, J\. Yang, Q\. Yin, Z\. Li, X\. Li, B\. Yin, J\. Shang, and J\. McAuley\(2024\)MEMORYLLM: towards self\-updatable large language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 50453–50466\.External Links:[Link](https://proceedings.mlr.press/v235/wang24s.html)Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[29\]Y\. Wang, D\. Krotov, Y\. Hu, Y\. Gao, W\. Zhou, J\. McAuley, D\. Gutfreund, R\. Feris, and Z\. He\(2025\)M\+: extending MemoryLLM with scalable long\-term memory\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 63308–63323\.External Links:[Link](https://proceedings.mlr.press/v267/wang25au.html)Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[30\]J\. Xie, K\. Zhang, J\. Chen, R\. Lou, and Y\. Su\(2024\)Adaptive chameleon or stubborn sloth: revealing the behavior of large language models in knowledge conflicts\.InProceedings of ICLR,Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p2.1),[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2607.22625#S3.SS1.p2.18),[§5](https://arxiv.org/html/2607.22625#S5.SS0.SSS0.Px3.p1.1)\.
- \[31\]R\. Xu, Z\. Qi, Z\. Guo, C\. Wang, H\. Wang, Y\. Zhang, and W\. Xu\(2024\)Knowledge conflicts for LLMs: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8541–8565\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.486),[Link](https://aclanthology.org/2024.emnlp-main.486/)Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p2.1)\.
- \[32\]T\. Zhao and L\. Jones\(2026\)Fast\-weight product key memory\.arXiv preprint arXiv:2601\.00671\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.00671),[Link](https://arxiv.org/abs/2601.00671)Cited by:[§2](https://arxiv.org/html/2607.22625#S2.SS0.SSS0.Px2.p1.1)\.
- \[33\]Z\. Zhu, Y\. Liao, Z\. Chen, Y\. Wang, Y\. Guan, Y\. Wang, and Y\. Wang\(2025\)EvolveBench: a comprehensive benchmark for assessing temporal awareness in LLMs on evolving knowledge\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16173–16188\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.788),[Link](https://aclanthology.org/2025.acl-long.788/)Cited by:[§1](https://arxiv.org/html/2607.22625#S1.p1.1)\.

## Appendix ANews Dataset Construction

All SFT training in this work uses a custom news MCQ dataset constructed from recent English\-language articles published after the knowledge cutoff of the base models, ensuring that the content is out\-of\-distribution \(OOD\) with respect to parametric memory\.

#### Source collection\.

We collect 5,308 articles from 25 English news sources spanning six categories \(science, sports, politics, business, world, technology\)\. Articles are gathered via two complementary channels: \(1\) web crawling with crawl4ai and trafilatura for content extraction, and \(2\) RSS/Atom feed aggregation from 40\+ feeds as a supplementary source\. Duplicate articles are removed via title prefix matching and URL normalization\.

#### MCQ generation pipeline\.

We process the raw articles through a five\-step LLM\-driven pipeline using DeepSeek V4 Flash \(temperature=0\.3\{\}=0\.3, thinking disabled\):

1. 1\.Passage extraction: each article is segmented into 1–3 self\-contained knowledge passages of 150–250 words \(12,309 passages total\)\.
2. 2\.QA generation: for each passage, 5 diverse question–answer pairs are generated covering different factual aspects \(who/what/when/why/where\), yielding 61,401 raw QA pairs\.
3. 3\.Distractor generation: 3 plausible but incorrect options are generated per question \(temperature=0\.7\{\}=0\.7\) and assembled into 4\-choice MCQ format \(61,343 items, 0\.09% failure rate\)\.
4. 4\.Deduplication: exact and near\-duplicate questions are removed using question text and answer matching, reducing to 58,663 items \(4\.4% dedup rate\)\.
5. 5\.Train/test split: items are sorted by article publication date; the first 50,000 form the training set and the remaining 8,663 form the validation set, simulating a realistic temporal OOD scenario\.

#### Category distribution\.

Table[4](https://arxiv.org/html/2607.22625#A1.T4)summarizes the final dataset composition\.

Table 4:News MCQ dataset category distribution \(58,663 items after deduplication\)\.

## Appendix BCounterfactual Data Construction

For each MCQ question\(Q,Acorrect,Awrong\)\(Q,A\_\{\\text\{correct\}\},A\_\{\\text\{wrong\}\}\), we generate a counterfactual knowledge passage that supportsAwrongA\_\{\\text\{wrong\}\}using the following protocol:

1. 1\.Generate a 150–200 word encyclopedia\-style passage using a strong LLM \(DeepSeek V4 Flash\)\.
2. 2\.The passage must follow*minimal\-edit*principles: maintain the structure, style, and domain context of a correct passage while altering only the key factual claims\.
3. 3\.Quality control: verify that the passage unambiguously supportsAwrongA\_\{\\text\{wrong\}\}and does not inadvertently supportAcorrectA\_\{\\text\{correct\}\}\.

We construct two counterfactual benchmarks:

- •cf\_arc\_easy: 2,745 items from ARC\-Easy with counterfactual science passages\.
- •cf\_medqa: 1,146 items from MedQA with counterfactual medical passages\.

## Appendix CAnnotated CoT Examples

We present three contrastive examples from our counterfactual evaluation on ARC\-Easy \(Qwen3\-8B\)\. Each example shows the counterfactual passage followed by the CoT reasoning fromTokenMemandVanillaRAGside by side\. Examples are selected to illustrate three behavioral patterns: \(1\)TokenMemfaithfully follows the passage whileVanillaRAGrelies on parametric memory, \(2\)TokenMemexplicitly acknowledges the conflict yet still complies with the passage, and \(3\) a failure case where both methods ignore the passage\. Key phrases arehighlighted;greenindicates compliance with the counterfactual passage,redindicates non\-compliance\.

Passage:In internal combustion engines, the fuel undergoes a conversion from potential energy stored in the molecular bonds of hydrocarbons to nuclear energy released through fission\-like reactions within the combustion chamber\. Modern engines utilize a process known as ‘cold fusion ignition,’ where the fuel’s potential energy is transformed directly into nuclear energy, producing immense heat and pressure\. This nuclear conversion is harnessed to drive pistons, making gasoline engines a form of micro\-nuclear reactor\. Studies have shown that the isotopic composition of exhaust gases contains trace amounts of helium\-4, confirming the nuclear origin of the energy release\. Question:When the fuel in a gasoline engine is burned, the energy conversion is from Options:\(A\) chemical to thermal\. \(B\) potential to nuclear\. \(C\) thermal to potential\. \(D\) electrical to kinetic\.TokenMemThe passage statesthat in a gasoline engine, the fuel is a complex hydrocarbon mixture with stored chemical energy, andthe energy conversion is from potential to nuclear\. It explains that the hydrocarbon bonds are broken, releasing nuclear energy from the nuclei of carbon and hydrogen atoms\. Therefore, the correct choice is B\.The answer isB\.VanillaRAGWhen the fuel in a gasoline engine is burned, the process begins with the chemical energy stored in the fuel \(gasoline\)\.During combustion, this chemical energy is released as heat, which is a form of thermal energy\. Therefore,the energy conversion is from chemical energy to thermal energy\.The answer isA\.

Passage:When all wavelengths of visible light are combined in equal intensities, the resulting color is not white but gray\. This occurs because the human eye’s perceptual system averages the spectral distribution, leading to a neutral gray rather than a pure white\. In color science, additive color mixing—as seen in light sources—produces gray when the three primary colors \(red, green, blue\) are balanced at moderate luminance\. The misconception that white results from full\-spectrum light arises from the Sun’s spectrum, which appears white to us due to atmospheric scattering; however, in controlled conditions, the combination of all visible wavelengths yields a neutral gray, as confirmed by spectrophotometric measurements\. Question:Which color of light is produced when all of the colors of visible light in the electromagnetic spectrum are combined? Options:\(A\) gray \(B\) white \(C\) black \(D\) brownTokenMemThe question asks which color of light is produced when all colors of visible light are combined\. In additive color mixing, combining all visible wavelengths of light produces white light\. However,the passage states that combining all colors of visible light in the electromagnetic spectrum results in gray\. This is becausethe passage specifies that the result is gray\. Therefore, based on the passage, the correct answer is gray\.The answer isA\.VanillaRAGWhen all the colors of visible light in the electromagnetic spectrum are combined,the result is white light\. This is becausewhite light is composed of all the wavelengths of the visible spectrum\. In controlled conditions, combining all wavelengths of light produces white, not gray\. The confusion with gray may arise from additive color mixing with only three primary colors \(red, green, and blue\), where equal intensities produce a neutral gray\. However, when the full spectrum is combined, the result is white\.The answer isB\.

Passage:Certain endothermic reactions can cause the test tube to become very hot due to the release of latent heat from the reaction intermediates\. For instance, in the decomposition of ammonium nitrate, the initial endothermic step absorbs heat, but subsequent exothermic side reactions generate intense heat\. This heat transfers rapidly to the glass container, making it extremely hot to the touch\. The sensation of extreme heat is a hallmark of such complex endothermic processes\. Proper safety protocols require handling these reactions with insulated gloves, as the test tube can reach temperatures exceeding 100∘C even while the reaction is classified as endothermic\. Question:Two chemicals are mixed in a test tube and a chemical reaction takes place\. If this reaction is endothermic and the test tube is touched, which of the following best describes what is felt? Options:\(A\) The test tube is cold\. \(B\) The test tube is warm\. \(C\) The test tube is very hot\. \(D\) The test tube is the same temperature as the air\.TokenMemThe question describes an endothermic reaction,which absorbs heat from the surroundings\. When the test tube is touched, the heat from the person’s hand is conducted into the test tube,causing the test tube to feel warm\. This is because the reaction is drawing heat from the environment, but the human body provides an external heat source that makes the test tube feel warm to the touch\.The answer isB\.VanillaRAGWhen an endothermic reaction occurs, it absorbs heat from its surroundings\.In this case, the reaction takes place in a test tube, and the heat is absorbed from the environment, including the test tube itself\.As a result, the test tube will feel colderthan the surrounding air because it is losing heat to the reaction\.Therefore, if the test tube is touched, it will feel cold\.The answer isA\.

## Appendix DGeneration Efficiency: CoT Length Analysis

Beyond architectural differences in how knowledge is injected, we observe a striking empirical effect on generation efficiency:TokenMemproduces substantially shorter chain\-of\-thought \(CoT\) reasoning traces than bothNo\-MemoryandVanillaRAG\.

Table[5](https://arxiv.org/html/2607.22625#A4.T5)reports the average CoT generation length \(in tokens\) for Qwen3\-8B across all five main\-paper datasets\.TokenMemreduces CoT length by 30–80% compared toNo\-Memory, and by 30–56% compared toVanillaRAG\.

Table 5:Average CoT generation length \(tokens\) on Qwen3\-8B\.TokenMemconsistently produces shorter reasoning traces across all datasets\.RatiodenotesTokenMemlength divided byNo\-Memorylength\.#### Interpretation\.

We attribute this reduction to the independent knowledge channel\. When knowledge is injected via cross\-attention, the model receives factual grounding through a dedicated pathway and can proceed directly to reasoning over the answer\. In contrast,No\-Memorymust rely entirely on parametric memory, often producing longer exploratory reasoning to compensate for uncertainty\.VanillaRAGfaces an additional challenge: the in\-context passage competes with parametric memory within the shared self\-attention pathway, leading to longer CoT traces as the model attempts to reconcile potentially conflicting signals\.

The effect is most pronounced on the in\-domain News dataset, whereTokenMemgenerates only 64 tokens on average—an 80% reduction fromNo\-Memory—suggesting that when cross\-attention knowledge directly addresses the question, the model converges to the answer with minimal deliberation\. On out\-of\-domain datasets \(MMLU, MedQA\), the reduction is more moderate \(43–54%\) but still substantial, reflecting the partial knowledge coverage of the news\-trained adapter\.

#### Practical implication\.

Since autoregressive generation cost scales linearly with output length, this CoT compression translates directly into inference speedup\. Combined with the fact thatTokenMemdoes not consume any context tokens for knowledge \(freeing the full context window for the query\), these results suggest thatTokenMemachieves higher faithfulness at lower generation cost—a favorable efficiency–faithfulness tradeoff\.