Can a Language Model Learn Facts Continually in Its Weights?

# 1. Introduction
Source: [https://arxiv.org/html/2607.11020](https://arxiv.org/html/2607.11020)
Can a Language Model Learn Facts Continually in Its Weights?

Charles O’Neill1

1Baseten

###### Abstract

Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights\. Whether weight writes can support accumulation remains undecided\. We follow invented facts written into Qwen3 models from creation through sequences of twenty to one hundred later writes, using held\-out questions of five types, with the original model given the fact in its prompt as the reference\. Across these experiments, the breadth of the training data determines the kind of knowledge created\. Bare\-statement training produces recitation, while diverse restatements reduce the recitation\-to\-use gap from 27\.4 to 5\.4 points without showing the model a conclusion\. This difference carries into later writes: after twenty sequential writes, bare\-statement facts retain 1% accuracy while facts written from broad study data retain 46%\. We also find that facts can be behaviourally forgotten without being erased\. Forgotten facts keep most of the log\-probability added by their write, and under bare\-statement training 70% of wrong answers about them contain the most recently written fact\. The same writes barely degrade the model’s use of facts in context, and a forgotten study fact supplied in the prompt recovers to 77–80% on its questions\. These results describe knowledge that is stored but question\-keyed: later writes redirect the questions that reached it\. Damage to unrelated abilities tracks KL divergence from the original model, and the later writes cause interference regardless of how the earlier fact was stored\. Broad data can create usable knowledge, and a frozen reference can preserve capability, but no intervention we tested, including those built on accurate local measurements of each write, keeps earlier facts reachable\. When facts must be composed or survive later writes, the reliable channel is context rather than the weights\.

††footnotetext:Code, training conditions, and run records:[https://github\.com/basetenlabs/cortex](https://github.com/basetenlabs/cortex)\. Evaluation datasets:[https://huggingface\.co/datasets/baseten/cortex](https://huggingface.co/datasets/baseten/cortex)\.A language model can hold a new fact in its context or its weights\. Context makes the fact immediately usable, but only for the life of the prompt\. Continual learning asks the weights to acquire facts after training and retain them through further writes\. To understand whether they can, we characterise the object that a write creates: what kind of knowledge it contains, whether later writes preserve it, and what remains after questions about it fail\. This turns catastrophic forgetting into a property of the written object, measured against the same content placed in context\.

Fine\-tuning learns unknown facts slowly, and learned facts fail reversals and multi\-hop use that the same facts support in a prompt\(gekhman2024finetuning;berglund2024reversal;lampinen2025icl\)\. Diverse paraphrases make facts more extractable\(allenzhu2024physics31\), while distributional drift predicts forgetting under further training\(shenfeld2025rlrazor\)\. These observations lack a common account of why some writes produce usable knowledge, why some survive, and what a forgotten fact leaves behind\.

We build that account with invented facts, held\-out questions of five types, and two fixed references: the original model and the same model with the fact in its prompt\. Following each fact from creation through twenty to one hundred later writes lets us connect what a write creates to what later survives\. Training\-data breadth determines whether the model learns recitation or stated conclusions, and this difference predicts retention\. When a fact eventually fails every question, checkpoint reconstructions still find most of its write’s log\-probability lift in the weights, and under bare\-statement writes the questions that once reached it return the newest write’s content instead\. The same access problem is present before any overwriting: two individually usable written facts largely cannot be used together because the model cannot retrieve them on demand\. By contrast, context remains usable through the same writes, and questions about a forgotten fact succeed again once its statement is supplied\.

Together, these results separate three requirements for continual writing\. Each write must create usable knowledge, general abilities must survive accumulation, and earlier facts must remain reachable\. The first two appear tractable: broad data creates knowledge the model can apply, while a frozen original model or a penalty on local drift can protect general abilities\. Reachability remains unsolved across supervised fine\-tuning and both offline and online distillation; a reinforcement\-learning variant fails at the first requirement, occasionally gaining reward without installing the fact\. Yet on every schedule where the comparison passes its validity screen, in\-context use erodes no faster than general ability, leaving context as the reliable channel when facts must be composed or outlive later writes\.

Our five contributions are:

1. 1\.Broad training data makes written knowledge usable\. Diverse restatements reduce the recitation\-to\-use gap from 27\.4 points to 5\.4 without showing conclusions, and the bare\-statement gap replicates at 4B and 8B scale and under LoRA and full fine\-tuning \(§[3](https://arxiv.org/html/2607.11020#S3), §[7](https://arxiv.org/html/2607.11020#S7)\)\.
2. 2\.Keeping earlier facts remains unsolved, and the kind of knowledge predicts survival\. After twenty sequential writes, bare\-statement facts retain 1% and study facts retain 46%; the entailment gap predicts survival across the factorial \(=−0\.526\\rho=\-0\.526\), and study retention plateaus at 25–28% by one hundred writes \(§[4](https://arxiv.org/html/2607.11020#S4)\)\.
3. 3\.Forgetting destroys access rather than storage\. Forgotten facts keep 57–67% of their write’s drift\-corrected log\-probability lift, under bare\-statement writes their failures carry the newest write’s content, and questions about them succeed again when their statements are supplied in context \(§[5](https://arxiv.org/html/2607.11020#S5)\)\.
4. 4\.General abilities can be preserved because capability damage orders by KL divergence from the original model \(=0\.83\\rho=0\.83over twelve conditions and0\.9460\.946in the factorial\)\. A frozen teacher writes facts at near\-zero cost, while distillation through accumulated writes is the most damaging condition \(§[6](https://arxiv.org/html/2607.11020#S6)\)\.
5. 5\.The incoming write causes interference, which resists local control\. Study rather than bare\-statement later writes improve retention by 37\.6 points, while the stored\-write method has no detectable effect; a linearised predictor captures the immediate effect of an update \(=0\.795\\rho=0\.795\) but not the full trajectory \(=−0\.258\\rho=\-0\.258\) \(§[7](https://arxiv.org/html/2607.11020#S7)\)\.

§[2](https://arxiv.org/html/2607.11020#S2)describes the evaluation\. Model\-generated data trains its conditions, model\-written questions test them, and model judges score the open answers, so the instrument’s validity must be demonstrated\. Its certification requirements and dual grading policy support every later result\.

## 2\.Measuring whether a model can use a fact

Every claim in this paper depends on deciding whether a model can use a fact it did not previously know, rather than merely repeat it\. We test five forms of use against a floor from the original model and a fact\-in\-prompt ceiling\. Appendix[A](https://arxiv.org/html/2607.11020#A1)gives the complete training details and a glossary of conditions\.

All experiments use Qwen3\-4B with a frozen base and LoRA adapters unless stated otherwise\(hu2022lora\)\. The primary evaluation contains 247 one\-sentence facts about invented entities; the original model demonstrably fails their questions \(the floor screen below\)\. Most facts contradict a familiar default\. For example,*“Zorvathine is a metal that melts when cooled below−10∘\-10\\,^\{\\circ\}C\.”*A correct answer must therefore come from the new fact rather than general knowledge\. Appendix[D](https://arxiv.org/html/2607.11020#A4)varies the strength of this conflict\.

Table 1:Five question types for the Zorvathine fact\. Questions are written by a different model family from the facts and training data, reveal no answer, and are retained only when the original model fails\.Every fact receives questions of these five types \(Table[1](https://arxiv.org/html/2607.11020#S2.T1)\), with gold answers and accepted aliases\. Recall and paraphrase test recitation; application requires one\-step use; composition combines the fact with unstated world knowledge; and counterfactual questions force a choice between the written fact and the model’s default\. The two anchors run on every fact: the original model, with no fact in view, forms the floor, at 1–7% across question types, and the same model with the fact in its prompt forms the ceiling, at 75–99% under the strict policy below\.

Scoring combines deterministic checks with two model families\. The checker handles exact and alias matches and is tested against negation and numeric sign flips: “it will*not*turn blue” cannot match*blue*, and “below−10∘\-10\\,^\{\\circ\}C” cannot match “below10∘10\\,^\{\\circ\}C”\. Remaining answers go to an item\-level model judge, and passes must survive a second judge from another family\. Errors score as incorrect, and every compared condition uses the same pipeline\. Uncertainty is a bootstrap 95% confidence interval over facts; comparisons use paired per\-fact or per\-triple differences, and multi\-seed conditions report each seed and their mean\. Confirmatory contrasts were pre\-registered with frozen thresholds, and post\-hoc analyses are labelled as such\. Exclusions made during screening and judging\-validity checks leave 236–247 facts per contrast; each analysis states itsnn\.

### 2\.1Validity requirements and certification

Generated facts, training data, questions, and judgments create six recurrent failure modes, listed with their bias directions in Table[2](https://arxiv.org/html/2607.11020#A1.T2)\. Three determine the design\. A shared generator can teach the evaluation’s phrasing through the training data\. An asymmetric harness constraint can punish one answer style through a length cap, numeric normalisation, or a checker that misses negation\. A judge that abstains on a non\-random subset can also reshape a comparison even when abstentions fail closed\.

We certify the instrument before scientific use\. The gate requires a floor below 20%, a ceiling above 80%, truncation below 5% per condition, an adversarial audit of answer leakage, and audits of judged passes as well as failures\. The version used here passes at floor 3%, ceiling 84%, leak 1/30, and pass\-audit 29/30\. Two conditions exceed the truncation bar and are disclosed rather than gating: the original\-model floor \(23%\) and bare\-statement training at 24 steps \(11%\)\. Truncation forces failures, so both deflate the floor side of comparisons \(Appendix[A](https://arxiv.org/html/2607.11020#A1)\)\. Downstream,*held out*means held out from the entire generative process, and comparisons only join conditions scored by the same judge\.

### 2\.2The dual grading policy and entailment gap

Answers that state an underlying fact without drawing the requested conclusion admit two defensible grades\. We therefore score every answer twice\. The*strict*policy requires the requested conclusion, value, choice, or name; the*lenient*policy also credits a fact that strictly entails the target\. Use\-type claims anchor on the strict policy, while the policies nearly coincide on recall and paraphrase\. The primary judge is GPT\-5\.4\-mini, configured with reasoning off at temperature zero, and it returns a decision on every item\. Because the judging pipeline is part of the instrument, the certification was repeated under this judge and the dual policy, and on the final records the instrument again clears its gate: floor 4%, in\-prompt ceiling 91% lenient, and strict pass\-audit 30/30\.

The difference between the policies measures how often a method stops at recitation instead of stating the conclusion:

entailment gap=lenient accuracy−strict accuracy\.\\text\{entailment gap\}\\;=\\;\\text\{lenient accuracy\}\-\\text\{strict accuracy\}\.\(2\.1\)This is the recitation\-to\-use gap of the abstract and introduction\. The two policies are judged independently rather than nested, so small negative gaps occur, and differences of a few points sit within judge noise; we interpret only large gaps\. Section[3](https://arxiv.org/html/2607.11020#S3)shows that the gap differs sharply across training methods\. We treat it as a property of the written object, not a grading artefact to tune away\.

## 3\.Training\-data breadth sets the kind of knowledge a write creates

The first requirement of continual writing is that each write create usable knowledge\. To determine which methods meet it, we compare training methods at matched optimisation budgets and use the entailment gap to measure the kind of knowledge they create \(§[2\.2](https://arxiv.org/html/2607.11020#S2.SS2)\)\. Across objectives, ranks, and model scales, bare\-statement training is the only method that leaves a large gap\. Broader training data closes it, and the factorisation of §[7](https://arxiv.org/html/2607.11020#S7)identifies that breadth as the causal variable\.

### 3\.1Training\-data breadth

We begin with two conditions on the 247\-fact evaluation\.*Bare\-statement*training uses the fact sentence in two trivial framings;*study*training uses 24 generated paraphrases, question–answer pairs, worked implications, and contrasts with the default the fact violates\. Bare\-statement budgets are 24, 96, and 192 steps, and study budgets are 24 and 96\. The heaviest bare\-statement budget, 192 steps on a single sentence, is deliberately severe; the lighter\-budget control of §[4\.1](https://arxiv.org/html/2607.11020#S4.SS1)and the matched\-budget factorisation of §[7](https://arxiv.org/html/2607.11020#S7)show that the conclusions do not depend on that operating point\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x1.png)Figure 1:Strict accuracy by question type as the optimisation budget grows for bare\-statement training \(indigo\) and study training \(rose\)\. The dashed line is the fact\-in\-prompt ceiling and the dotted line the floor from the original model\. Counterfactual accuracy stays at 21–23% under bare\-statement training and reaches 45–50% under study training; composition is 39–41% against 59–60%, below the 83% ceiling\.The five\-type average at 96 steps depends on the grading policy\. Under the strict policy used for use\-type claims, study training wins by 10\.1 points \(95% CI\[\+7\.0,\+13\.3\]\[\+7\.0,\+13\.3\], paired over 243 facts\), while bare\-statement training wins under the lenient policy by 13\.6 points\. The question types explain the reversal \(Figure[1](https://arxiv.org/html/2607.11020#S3.F1)\)\.

The two conditions trade off recitation and use\. Bare\-statement training reaches 97% recall and 95% paraphrase within 24 steps and saturates by 96, whereas study training reaches 88% recall and 83% paraphrase at 96 steps, trailing bare\-statement training at its 96\-step saturation by paired differences of 11 and 13 points\. The pattern reverses when the written fact contradicts the model’s default\. Bare\-statement accuracy stays at 21–23% from 24 to 192 steps, while study training reaches 45–50%, a paired gain of 29 points at 96 steps\. Further optimisation on the statement therefore improves recitation without substituting for varied data\.

The reversal between grading policies makes this distinction visible \(Figure[2](https://arxiv.org/html/2607.11020#S3.F2)\)\. The entailment gap is 22–42 points on bare\-statement use questions, compared with 1–5 points after study training and 1–6 points with the fact in the prompt\. Lenient grading credits a recited premise, while strict grading requires the requested conclusion, so the large bare\-statement gap measures knowledge that stops one step short of use\.

Broader data also improves composition, although no method reaches the ceiling\. Study training reaches 60% against 40% for bare\-statement training and 83% with the fact in the prompt, a paired gain of 18 points\. The best weight\-based method, context distillation \(a distribution\-matching objective defined in §[3\.2](https://arxiv.org/html/2607.11020#S3.SS2)\), reaches 70%, placing it 10\.8 points above study training and 13\.1 points below the ceiling\. Fine\-tuning can therefore improve composition, but no weight\-based method we tested matches in\-context composition\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x2.png)Figure 2:The entailment gap on the primary evaluation\. Solid bars show strict accuracy and light caps extend them to lenient accuracy\. The cap is the share of answers that state an entailing fact without giving the requested conclusion\.Taken together, the comparison separates what optimisation and data provide\. At 96 steps, broader data improves application by\+21\+21points, composition by\+18\+18, and counterfactual use by\+29\+29, while extra bare\-statement optimisation improves none of them\. We have not isolated which ingredient of the study data matters, and why study training scores worse on recall and paraphrase questions remains unexplained\. These comparisons use uncued questions, which never restate the training phrasing; cued phrasings can supply the missing step themselves and collapse use back to recitation, a boundary measured in Appendix[D\.4](https://arxiv.org/html/2607.11020#A4.SS4)\.

### 3\.2Training objectives

The first comparison changes the training data under a single text objective, but a write can instead target a distribution\. We therefore compare bare\-statement and study training with offline context distillation and online context distillation in both KL directions\. Each method writes the same 40 certified facts into a fresh rank\-16 adapter for 192 steps, with three seeds\. The steps are matched, but generation and token throughput differ, so this remains an operating\-point comparison\.

Writing a factsswith supervised fine\-tuning maximises the likelihood of target textyygiven promptxxover the fact’s training set𝒟s\\mathcal\{D\}\_\{s\}:

ℒSFT​\(\)=−E\(x,y\)∼𝒟s​∑tlog⁡\(yt∣x,y<t\)\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\-\\,\\mdmathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{s\}\}\\sum\_\{t\}\\log\\\!\\left\(y\_\{t\}\\mid x,y\_\{<t\}\\right\)\.\(3\.1\)For context distillation, the teacher\(⋅∣x,y<t\)T=\(⋅∣s⊕x,y<t\)0\{\}\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)=\{\}\_\{0\}\(\\cdot\\mid s\\oplus x,y\_\{<t\}\)is the original model with the fact in its prompt\. The student does not see the fact and matches the teacher token by token\. Offline distillation matches the student to this distribution on sequences sampled once from the teacher,

ℒoffline\(\)=Ex,y∼T∑tKL\(\(⋅∣x,y<t\)T∥\(⋅∣x,y<t\)\),\\mathcal\{L\}\_\{\\mathrm\{offline\}\}\(\\theta\)=\\mdmathbb\{E\}\_\{x,\\;y\\sim\{\}\_\{T\}\}\\sum\_\{t\}\\mathrm\{KL\}\\\!\\left\(\{\}\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\middle\\\|\\,\(\\cdot\\mid x,y\_\{<t\}\)\\right\),\(3\.2\)while online distillation samplesy∼y\\simand uses the same token\-level divergence in the forward direction \(KL\(∥T\)\\mathrm\{KL\}\(\{\}\_\{T\}\\\|\)\) or reverse direction \(KL\(∥\)T\\mathrm\{KL\}\(\\\|\{\}\_\{T\}\)\)\. The reverse direction is the on\-policy\-distillation objective oflu2025onpolicy\.

Across the five methods, bare\-statement training alone has a large entailment gap, at 26 points, while the other four lie at 2–4 points\. The pre\-registered contrast between the bare\-statement gap and their mean is\+22\.8\+22\.8points \(95% CI\[\+17\.4,\+28\.2\]\[\+17\.4,\+28\.2\]\)\. Richer supervision therefore closes the gap whether it comes from broader text data or a distributional target\.

Once the gap is within judge noise, changing the KL direction adds no further effect\. The pre\-registered prediction that mode\-seeking reverse KL would create more committed knowledge fails: the forward and reverse online\-distillation gaps are 4\.3 and 3\.0 points\. On\-policy sampling does, however, improve strict accuracy at this operating point\. Online distillation reaches 77–78%, compared with 71–72% for study training and offline distillation and 61% for bare\-statement training; every method remains above 50%\. Student\-sampled sequences buy seven points over fixed teacher samples, and §[4](https://arxiv.org/html/2607.11020#S4)tests whether that advantage survives later writes\.

### 3\.3A factorial over objective, data, and update method

Because the method comparison changes the source of the targets and the loss together, we next cross four objectives \(SFT, offline distillation, and online distillation in each KL direction\), bare\-statement or study data, and LoRA or full fine\-tuning\. The resulting sixteen conditions each have three seeds and forty facts, and fifteen yield numeric results\. Reverse\-KL distillation on bare statements with LoRA is censored because 9\.4% of its outputs loop until the generation cap, consistently across seeds; raising the cap extends the loop\.

The data effect survives this separation: study data reduces the conclusion\-type entailment gap in every reportable pair \(Figure[3](https://arxiv.org/html/2607.11020#S3.F3)a\)\. The reductions are 36\.4 points \(95% CI\[31\.7,41\.0\]\[31\.7,41\.0\]\) for full\-parameter SFT, 31\.2 for LoRA SFT, 18\.9 for full offline distillation, 14\.0 for full forward\-KL online distillation, and 3\.6–9\.8 for the remaining pairs\. Four of seven pairs clear the pre\-registered 10\-point band, and five exclude zero\. Because four pairs use learning rates tuned by condition, the factorial gives an association across operating points rather than a one\-variable causal effect\. The controlled factorisation in §[7](https://arxiv.org/html/2607.11020#S7)supplies that identification; here the result shows that the objective label alone is too coarse\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x3.png)Figure 3:The factorial over objective, data, and update method\.a, strict accuracy after one write, with each connected pair showing the bare\-statement\-to\-study change\.b, the conclusion\-type entailment gap against survival after twenty later writes\.c–d, capability damage against endpoint KL from the original model and cumulative local KL\. Colour denotes training data, marker shape the objective, and fill the update method; intervals are 95% condition\-clustered bootstrap intervals\. The correlations include all sixteen conditions, with the censored condition annotated off\-scale\.A GRPO condition with a programmatic answer\-match reward received almost no signal because the base model never produced the target counterfactuals, and the few reward gains did not install the fact; we leave reinforcement\-learning objectives that can write new knowledge to future work\.

### 3\.4Scale and update method

The same entailment gap appears at a larger model scale \(Figure[4](https://arxiv.org/html/2607.11020#S3.F4)\)\. At Qwen3\-8B, the bare\-statement gap remains 26 points and the headline contrast is\+22\.9\+22\.9points, compared with\+22\.8\+22\.8at 4B\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x4.png)Figure 4:The entailment gap for bare\-statement and study training in three configurations\. The gap follows the training data, not rank or scale\.Full fine\-tuning requires matched writing strength\. At learning rate10−510^\{\-5\}it produces a small bare\-statement gap because it barely writes the fact: lenient accuracy is 35%, against 87% for LoRA\. We swept the learning rate on five held\-out facts and selected3×10−53\\times 10^\{\-5\}because it matched LoRA’s writing strength rather than any target gap; the largest rate destabilised the study condition\. At the matched setting, full fine\-tuning reproduces the gap: 28 points for bare\-statement training and 4 for study training, a\+24\.0\+24\.0\-point contrast \(95% CI\[\+20\.9,\+27\.2\]\[\+20\.9,\+27\.2\]\)\. Narrow supervision, not the number of trainable parameters, creates the gap\.

## 4\.The kind of knowledge governs what survives continual writing

The previous section measures what each method writes\. We now follow that knowledge through later writes and find that its kind predicts survival: recitation disappears, while stated conclusions survive at far higher rates\. The result holds across two evaluations and out to one hundred writes\.

### 4\.1Retention under sequential writing

We repeat the five\-method comparison of §[3\.2](https://arxiv.org/html/2607.11020#S3.SS2)in a sequential setting\. Each method writes twenty facts one at a time, merging each fact’s adapter into the model before the next fact trains, at the same 192 steps per fact, with three seeds\. After the twentieth write we re\-evaluate every earlier fact \(Figure[5](https://arxiv.org/html/2607.11020#S4.F5)\)\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x5.png)Figure 5:Retention of earlier facts after twenty sequential writes, strict, with counterfactual questions excluded\. Recitation dies; half of the stated conclusions survive\.Facts written by study training retain 46%, while facts written by bare\-statement training retain 1%, a paired difference of45\.645\.6points \(95% CI\[38\.8,52\.6\]\[38\.8,52\.6\],n=57n=57fact–seed cells\)\. The collapse is not a failure to write: bare\-statement training answers 65% of questions correctly before any later writes arrive\. Nor is it degenerate output\. Inspecting the failures, the model gives coherent answers drawn from other written facts\. The fragility is also not collateral damage from a heavy optimisation budget\. At a lighter budget of 24 steps per fact, which leaves the held\-out capability tests at 54% \(against 3–37% at the full budget and roughly 80% before any writing\), bare\-statement facts still retain only 6%, so capability and retention dissociate\.

The comparison also shows an on\-policy advantage\. Online context distillation retains 27–32% against 14% for offline distillation \(\+15\.8\+15\.8points\), and this is not an artefact of writing strength, since the reverse\-KL variant writes less strongly than offline distillation yet retains more than double\. One caveat changes the reading: the entire distillation family partially degenerates by the twentieth write \(35–46% looping output, capability down to 42–49%\), so the on\-policy advantage is real in direction but modest in a degraded regime\. Of the five methods as run here, study training is the only one that neither degenerates nor collapses, and it still loses more than half of its facts; §[6](https://arxiv.org/html/2607.11020#S6)shows that a frozen teacher repairs the distillation conditions\.

The factorial supplies a cross\-method test\. For each of the three conclusion question types, we compare a condition’s single\-write entailment gap with survival after twenty writes\. Across 45 such pairs, Spearman=−0\.526\\rho=\-0\.526\(95% CI\[−0\.657,−0\.197\]\[\-0\.657,\-0\.197\]; Figure[3](https://arxiv.org/html/2607.11020#S3.F3)b\)\. Collapsing each condition to its mean gives=−0\.675\\rho=\-0\.675\. The first estimate clears the pre\-registered\|\|≥0\.5\|\\rho\|\\geq 0\.5threshold, and sensitivities on a second question source and on the pooled set agree in sign\. Because the pre\-registration did not anticipate censoring, this is a disclosed complete\-case analysis of fifteen conditions rather than the registered sixteen\. The censored condition’s repetitive output grows from 9\.4% after one write to 85\.2% after twenty, so it is itself a method collapse rather than a benign omission\.

### 4\.2Retention on the prior\-conflict evaluation

A second pre\-registered experiment asks whether the result depends on the fact itself \(Appendix[D](https://arxiv.org/html/2607.11020#A4)\)\. Four sequential conditions cross bare\-statement and study training at 96 steps per fact with prior\-neutral and prior\-inverting facts, each run with three seeds over twenty writes\. Contrasts are strict and paired\. Counterfactual questions are excluded from the primary endpoints because their cue sensitivity was still under test \(Appendix[D\.4](https://arxiv.org/html/2607.11020#A4.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x6.png)Figure 6:Retention on the prior\-conflict evaluation\. Left: strict accuracy on earlier facts afterkkwrites, pooled over tiers and seeds\. Facts written from bare statements are near zero byk=5k=5\. Right: retention after all twenty writes by prior tier; prior\-inverting facts survive better than neutral ones under both methods\.The same separation appears on this evaluation\. Facts written by study training retain 31–49% atk=20k=20, while facts written by bare\-statement training retain 1–7%, a paired difference of35\.735\.7points \(95% CI\[29\.8,41\.9\]\[29\.8,41\.9\],n=114n=114fact–tier–seed cells\) at an identical optimisation budget \(Figure[6](https://arxiv.org/html/2607.11020#S4.F6)\)\. Weaker writing does not explain this: bare\-statement training writes these facts at 59% strict accuracy to study training’s 72%, so normalised survival still differs by a factor of about six\. The collapse is immediate, with bare\-statement facts at 2% byk=5k=5\. We also find no decay into recitation\. The lenient−\-strict gap of retained facts is 0–2 points at every checkpoint under both methods, so a fact either survives as a stated conclusion or disappears, with no intermediate stage as recitation\-only knowledge\.

The prior effect reverses the pre\-registered prediction that facts contradicting the model’s prior decay faster\. Confirmation required inverting facts to retain at least 10 points less than neutral ones; every re\-analysis leaves the estimate at least 22 points below that band, and the direction reverses: neutral facts retain12\.312\.3points less than prior\-inverting ones \(95% CI\[−21\.3,−2\.8\]\[\-21\.3,\-2\.8\],n=19n=19paired triples\), under both training methods\. The reversal remains uncertain: a per\-triple sign test givesp=0\.064p=0\.064, the study\-only interval touches zero, and the two tiers use disjoint question sets that were not difficulty\-matched\. It is, however, stable under leave\-one\-out, negative on every shared question type, and it runs opposite to the writing\-strength confound: inverting facts are written more weakly than neutral ones, which would predict the reverse ordering\. Our working interpretation, post hoc, is that distinctiveness protects: a fact that contradicts the prior occupies unusual territory and collides less with subsequent writes than a bland fact does\. Alternatives such as question\-difficulty asymmetries between the tiers remain open, and the crossed experiments of §[7](https://arxiv.org/html/2607.11020#S7)do not resolve them\.

### 4\.3One hundred writes and periodic consolidation

We extended the prior\-conflict experiment to one hundred sequential writes\. Study retention declines to a 25–28% plateau rather than to zero \(Figure[7](https://arxiv.org/html/2607.11020#S4.F7)\)\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x7.png)Figure 7:One hundred sequential study writes on the prior\-conflict evaluation\. Left: retention decays to a 25–28% plateau, and periodic consolidation into the original model does not lift it\. Right: consolidation does preserve the held\-out capability tests, recovering them at each pass\.The same separation holds across methods at that horizon \(Figure[8](https://arxiv.org/html/2607.11020#S4.F8)\)\. In a hundred\-write extension of the factorial covering SFT and offline distillation with bare\-statement or study data and LoRA or full\-parameter updates, bare\-statement SFT ends at 8\.6–11\.7% retention against study SFT’s 33\.0–37\.7%, and offline distillation shows the same data effect \(25\.0–29\.4% against 38\.5–43\.7%; ranges span the update methods\)\. The training\-data separation therefore persists under both objectives and update methods\. This extension is descriptive: its scope was reduced after partial records existed, its endpoint contrasts were not pre\-registered, and its learning rates are tuned per condition, so we read trajectories and endpoint ranges rather than inferential contrasts \(details in Appendix[B\.3](https://arxiv.org/html/2607.11020#A2.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x8.png)Figure 8:One hundred sequential writes across the reduced factorial \(three seeds per condition; descriptive, with pooled point estimates and no uncertainty bands\)\. Left: strict retention of earlier facts; the bare\-statement/study separation persists tok=100k\{=\}100under both objectives and both update methods\. Right: the held\-out capability tests over the same runs\.We then test the mitigation suggested by §[6](https://arxiv.org/html/2607.11020#S6): every twenty writes, consolidate all facts so far into a fresh copy of the original model by batch distillation\. Consolidation does not help retention \(25% against 28% atk=100k=100, within noise\), and the passes are not simply losing a race with the writes between them: five writes after the first pass, retention is 41% against the unconsolidated run’s 38%, so the pass fails to restore reachability even immediately\. General abilities, in contrast, recover at every pass and stay systematically above the unconsolidated run \(\+12\+12points atk=100k=100; Figure[7](https://arxiv.org/html/2607.11020#S4.F7), right\)\. Consolidating into the original model is a capability safeguard, not a retention fix\. This is consistent with the addressing account of §[5](https://arxiv.org/html/2607.11020#S5), in which a pass restores the output distribution while the routes to earlier facts stay captured; why re\-distilling every fact from its statement fails to re\-key them is a question our experiments do not answer\.

## 5\.What forgetting destroys: access, not storage

Failed questions do not distinguish erasure from lost access\. We separate these possibilities by probing storage directly, testing joint use before overwriting, and restoring forgotten facts through context\. Across all three tests, a stored trace survives while access fails\.

### 5\.1Storage after behavioural forgetting

We repeat bare\-statement and study training for twenty sequential writes and three seeds while saving every adapter\. Reconstructed checkpoints agree with the original measurements within10−210^\{\-2\}nats per token\. We then track the probability assigned to each written statement from before its write to the end of the sequence\. For a fact with statementsswritten at stepjjand probed afterkksubsequent writes, we summarise storage by the retained fraction of the write’s own log\-probability lift,

R​\(k\)=log⁡pj\+k​\(s\)−log⁡p0​\(s\)log⁡pj​\(s\)−log⁡p0​\(s\),R\(k\)\\;=\\;\\frac\{\\log p\_\{\{\}\_\{j\+k\}\}\(s\)\-\\log p\_\{\{\}\_\{0\}\}\(s\)\}\{\\log p\_\{\{\}\_\{j\}\}\(s\)\-\\log p\_\{\{\}\_\{0\}\}\(s\)\},\(5\.1\)where0is the model before the fact was written andjthe model just after\.R=1R=1means the write’s lift is fully retained;R=0R=0means erasure back to the pre\-write prior\. A drift control, the same quantity computed on never\-written statements, corrects for the elevation that any statement receives from the surrounding writes\. Facts whose write produced no measurable lift are excluded fromR​\(k\)R\(k\), and we report medians over facts\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x9.png)Figure 9:Storage survives behavioural forgetting\. Left: the written statement’s log\-probability, as a fraction of the lift its own write produced \(Eq\.[5\.1](https://arxiv.org/html/2607.11020#S5.E1)\), stays far above the pre\-write floor across all subsequent writes\. Lines are medians and shading the interquartile range over facts, for three seeds\. Middle: facts that fail every question atk=20k\{=\}20still retain most of their statement lift\. Solid lines are medians, dashed lines subtract the drift control, and the shaded region marks where genuine erasure would sit\. Right: the fraction of forgotten facts’ wrong answers that contain the most recently written fact’s content\.Facts that fail every strict question atk=20k\{=\}20still hold a median 69% \(bare\-statement\) and 79% \(study\) of the log\-probability lift their write produced, or 57% and 67% after the drift correction\. No fact in either condition approaches the erased floor\. The probe reads only the statement’s likelihood, but for study facts the write demonstrably installed more than a string, because use questions passed before the later writes arrived; the statement’s lift is the measurable trace of that object\. The fact therefore remains measurably present after its behavioural loss \(Figure[9](https://arxiv.org/html/2607.11020#S5.F9)\)\.

What changes instead is where the questions land\. Among wrong answers about forgotten facts, the bare\-statement model attaches the most recently written fact’s content to the queried entity in 70% of failures; the study model almost never does \(1%\)\. Asked about the first fact it wrote, the model answers with the twentieth\.

Reaching a forgotten fact is also harder than learning a fresh one\. Re\-training one forgotten study fact to criterion consumed the full step budget, whereas a never\-written control fact took eight steps\. Slow relearning is the opposite of the classical savings effect, in which stored but inaccessible memories relearn faster, and it is what address capture predicts: retraining must displace the newer content that now owns the fact’s questions, not restrengthen a faded trace\. The comparison is suggestive rather than conclusive\. Only three study facts were forgotten in total \(their scarcity is the survival result of §[4\.1](https://arxiv.org/html/2607.11020#S4.SS1)\), and the same probe is uninformative in the bare\-statement condition, whose model after twenty writes is too damaged to learn any fact to criterion\. A second probe, re\-testing forgotten facts with cued questions, gave study 3/3 and bare\-statement 0/54; we exclude it from quantitative use because its questions share their contrast form with the study training data\.

### 5\.2Joint use and self\-retrieval of written facts

The route is already limited before overwriting\. We test fourteen fact pairs with 47 questions, each answerable only with both facts, screened so that the original model fails every question and a model with both facts in its prompt answers most \(Appendix[A](https://arxiv.org/html/2607.11020#A1)\)\. Each pair’s facts are placed in the weights or context in all four combinations, using study writes at 192 steps per fact and three seeds\. Sequential and joint writing give the same result\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x10.png)Figure 10:Two written facts largely cannot be used together\. Bars show strict accuracy on questions requiring both facts of a pair, by placement\. Within each placement, the model answers directly, answers after its own retrievals are placed in context, and answers with the true statements placed in context\. The dotted line is the both\-in\-prompt reference; the dashed line is the same true\-statements\-in\-context measurement on a model that wrote one unrelated fact\.Joint use collapses when both facts are written \(Figure[10](https://arxiv.org/html/2607.11020#S5.F10)\)\. Over all fourteen pairs, questions requiring both facts are answered at 32% when both facts are in the weights, against 91% when both are in the prompt \(paired difference−58\.1\-58\.1points, 95% CI\[−76\.8,−35\.4\]\[\-76\.8,\-35\.4\]\)\. Individually unusable facts do not explain the collapse: 87\.5% of placements pass an individual\-usability screen, which we report as a check rather than apply as a filter\.

The bottleneck is self\-retrieval\. Asked simply to state a fact it was trained on and can answer questions about, the model produces the correct content 34% of the time \(8% verbatim, the rest paraphrase\), and its failures include inverted relations and blends of the pair’s two facts\. Feeding the model’s own retrievals back to it as context accordingly closes only a sixth of the deficit\. Written knowledge is reachable by questions that happen to route to it, but the model has no handle it can pull itself\. There is nothing to search\.

Supplying the true statements in context recovers accuracy to 63%, and a control explains the remaining gap to the 91% reference: a model that wrote one unrelated fact drops to 66% on the same task\. The residual gap is generic damage that any write inflicts on fragile multi\-step in\-context reasoning, not an interaction between the written and contextual copies of a fact\. In the bare\-statement condition this generic damage is total\. After a single unrelated bare\-statement write, the model answers every question by reciting its one written statement and scores zero\.

### 5\.3In\-context use under continued writing

The model’s ability to use context survives the same writes\. On the reconstructed checkpoints of the study sequence, accuracy on fresh facts placed in context falls from 83% to 74% over twenty writes, which is less than the held\-out capability tests fall on the same models: the pre\-registered difference\-in\-differences is\+9\.2\+9\.2points \(95% CI\[\+3\.1,\+15\.3\]\[\+3\.1,\+15\.3\]\), and a positive value means in\-context use eroded less than the capability tests \(Figure[11](https://arxiv.org/html/2607.11020#S5.F11)\)\. The bare\-statement sequence collapses on both measures, so its difference is uninterpretable under the pre\-registered validity screen; the same fresh\-fact questions falling to 8% there show that the measurement detects erosion where erosion exists\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x11.png)Figure 11:Continued writing does not preferentially erode in\-context use\. Left: strict accuracy on fresh facts placed in context \(solid\) and on the held\-out capability tests \(dashed\), across the sequential runs\. Right: the sequence’s own written facts atk=20k\{=\}20, answered with their statements supplied in context\.The sequence’s own forgotten facts complete the argument\. With their statements supplied in context, questions about them return to 77–80% in the study condition, against 14% in the bare\-statement condition, whose general collapse extends to this task too\. This is not a reading of the stored copy, since a model that had never written the fact would answer a supplied statement equally well\. It shows two narrower things: the failure is confined to the route from questions to the stored fact, and the stale written copy does not interfere with a supplied one\. As a remedy, supplying the statement concedes the point, because the fact must already be held outside the weights\.

## 6\.What repeated writes cost

Continual writing also requires general abilities to survive\. We measure capability loss, distributional drift, and retention across twelve conditions: two objectives, adapter ranks 4 and 16, and three writing regimes \(sequential, batch at 100 facts, and batch at 200\)\. Sequential conditions merge 20 one\-fact adapters; batch conditions train one adapter on all facts at once\. The objectives are study\-data SFT and context distillation\(snell2022context;lampinen2025icl\)\. Batch distillation uses the original model as teacher, while these sequential conditions use the current merged model; a frozen\-teacher sequential variant follows in §[6\.2](https://arxiv.org/html/2607.11020#S6.SS2)\. Capability is measured by 100 rule\-scored tests, on which the original model scores about 80%\. Drift is the KL divergence of next\-token distributions from the original model0, averaged over a fixed pool𝒫\\mathcal\{P\}of held\-out prompts,

drift\(\)=Ex∼𝒫1\|x\|∑tKL\(\(⋅∣x<t\)∥\(⋅∣x<t\)0\),\\mathrm\{drift\}\(\\theta\)\\;=\\;\\mdmathbb\{E\}\_\{x\\sim\\mathcal\{P\}\}\\;\\frac\{1\}\{\|x\|\}\\sum\_\{t\}\\mathrm\{KL\}\\\!\\left\(\(\\cdot\\mid x\_\{<t\}\)\\,\\middle\\\|\\,\{\}\_\{0\}\(\\cdot\\mid x\_\{<t\}\)\\right\),\(6\.1\)the per\-tokenKL\(∥\)0\\mathrm\{KL\}\(\\,\\\|\\,\{\}\_\{0\}\)on text neither model was trained on\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x12.png)Figure 12:The twelve continual\-learning conditions, by regime, objective, and rank\. Left: capability change in percentage points from the≈\\approx80% baseline\. Right: facts kept, measured as earlier\-fact accuracy after the twentieth sequential write and as all\-fact accuracy for the batch conditions\. The two extremes are both context distillation\. Accuracy immediately after writing is uniform \(76–78% sequential, 62–72% batch\) and omitted\.### 6\.1Capability loss and distributional drift

Across the twelve conditions, capability loss follows KL divergence from the original model \(Spearman=0\.83\\rho=0\.83; Figure[13](https://arxiv.org/html/2607.11020#S6.F13)\)\. Objective, rank, and regime matter to the extent that they move the output distribution\. This extends the finding ofshenfeld2025rlrazor, that KL from the base policy predicts forgetting, to knowledge writing\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x13.png)
Figure 13:Capability loss against KL divergence from the original model, one point per condition;=0\.83\\rho=0\.83\.![Refer to caption](https://arxiv.org/html/2607.11020v1/x14.png)
Figure 14:Capability change for all twelve conditions\. The two extremes are both context distillation; what differs is the teacher\.
The factorial of §[3\.3](https://arxiv.org/html/2607.11020#S3.SS3)strengthens this ordering on a unique 400\-item held\-out capability set\. Across all sixteen conditions, including the censored one, whose drift and capability measurements remain valid, damage after twenty writes correlates with endpoint KL from the original model at=0\.946\\rho=0\.946\(95% CI\[0\.787,0\.991\]\[0\.787,0\.991\]\) and with cumulative local KL at=0\.909\\rho=0\.909\(Figure[3](https://arxiv.org/html/2607.11020#S3.F3)c–d\)\. Both clear the pre\-registered\|\|≥0\.5\|\\rho\|\\geq 0\.5threshold\. This is a cost coordinate that holds across methods, not a sufficient causal mechanism: the penalty experiment of §[6\.3](https://arxiv.org/html/2607.11020#S6.SS3)preserves capability without systematically lowering measured KL from the original\.

### 6\.2The distillation teacher

The extremes of Figure[12](https://arxiv.org/html/2607.11020#S6.F12)are the same objective \(Figure[14](https://arxiv.org/html/2607.11020#S6.F14)\)\. Batch distillation from the original model as teacher writes 200 facts at−3\-3to\+1\+1points of capability, at a third to half the KL of any SFT condition; at rank 16 it beats SFT by 16 points, and the direction holds in all four batch conditions\. Sequential distillation, whose teacher is the model’s own accumulated merges, amplifies its own drift \(KL 1\.75 at rank 16\) into the worst of the twelve conditions:−28\-28points of capability and 11% retention of earlier facts\.

The teacher’s distribution, rather than the distillation loss, provides the stability\. The original model pins the student to its distribution; a teacher that moves with each write compounds the drift\. Since batch accuracy on the written facts is similar across objectives \(62–72%\), the frozen teacher writes the same knowledge with much less damage\. This agrees with single\-task results in which frozen teachers are stable and self\-distillation diverges\(ye2026opcd\), and with capability preservation under one\-hop self\-conditioning on demonstrations\(shenfeld2026sdft\)\. The failure here arises when distillation is iterated through accumulated writes\.

We tested this account sequentially, in a pre\-registered rerun, by writing one fact at a time and merging each adapter, while always distilling against a frozen copy of the original model\. Everything else matched the own\-merges condition\. The frozen\-teacher run writes twenty facts sequentially at\+2\+2points of capability, KL 0\.48, and 54% retention, indistinguishable from the batch condition \(Figure[15](https://arxiv.org/html/2607.11020#S6.F15)\)\. The own\-merges run loses 31 points, reaches KL 1\.70, and retains 21%, or 34% with its capped outputs excluded: the degenerating model loops past any generation budget, so its absolute retention depends on how capped answers are scored\. The paired retention difference is33\.033\.0points \(95% CI\[24\.6,41\.0\]\[24\.6,41\.0\]\)\. The continual\-distillation failure above therefore came from the drifting teacher, not the sequencing\. With the original model as a fixed reference, facts can be written online at near\-zero cost\. No sequential condition we measured does better: the frozen teacher keeps capability intact at retention matching the best fine\-tuning condition, which reaches 53% only at−16\-16points of capability, and even this safest write loses nearly half of its facts by the twentieth\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x15.png)Figure 15:Sequential distillation from a frozen teacher\. Distilling each write against a frozen copy of the original model \(indigo\) preserves capability, retains earlier facts, and stays close to the original\. Distilling through the model’s own accumulated merges \(rose\) loses all three\. The own\-merges retention bar scores its capped outputs as failures \(21%; 34% excluding them\)\.
### 6\.3An explicit penalty on drift

We added a KL\-to\-the\-current\-base penalty to bare\-statement SFT, the most damaging write, and swept its weight while writing twenty facts sequentially, in a pre\-registered rerun\. The penalty is a powerful mitigation \(Figure[16](https://arxiv.org/html/2607.11020#S6.F16), left\)\. Unpenalised, bare\-statement sequential SFT collapses the model, losing 66 points of capability with 1% retention; a modest penalty turns this into−19\-19points and 25% retention at=0\.5\\lambda=0\.5and−5\-5points and 36% retention at=1\.0\\lambda=1\.0, with write accuracy of 53% and 49%\. The paired retention gains over=0\\lambda=0are 23\.4 and 35\.1 points \(95% CIs\[16\.8,30\.4\]\[16\.8,30\.4\]and\[28\.8,41\.6\]\[28\.8,41\.6\]\)\.

Yet the measured KL from the original remains between 1\.8 and 2\.4 \(Figure[16](https://arxiv.org/html/2607.11020#S6.F16), right\), even as capability recovers by more than sixty points\. The penalty preserves abilities without moving the output distribution closer to the original\. Thus the=0\.83\\rho=0\.83ordering of §[6\.1](https://arxiv.org/html/2607.11020#S6.SS1)is robust across methods but is not a sufficient causal handle\. The penalty prevents the degenerate collapse of an unpenalised bare\-statement sequence; it does not reduce the distance travelled from the original\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x16.png)Figure 16:A per\-write KL penalty to the current base on bare\-statement sequential SFT, swept over its weight \. Left: capability and retention recover sharply once\>0\\lambda\>0, while write accuracy falls modestly\. Right: the measured KL from the original does not fall systematically with , so the rescue is not achieved by reducing drift\.
### 6\.4The dissociation of retention and capability

These twelve conditions use their own optimisation budget and question set, so retention levels here are not comparable to those of §[4](https://arxiv.org/html/2607.11020#S4)\. The twentieth fact is written as reliably as the first: accuracy immediately after writing is 76–78% in all four sequential conditions here, although the heavier 192\-step distillation runs of §[4\.1](https://arxiv.org/html/2607.11020#S4.SS1)instead degenerate by the twentieth write\. Earlier facts are another matter\. After twenty writes their accuracy is 53% \(SFT, rank 16\), 42% \(SFT, rank 4\), 27% \(distillation, rank 4\), and 12% \(distillation, rank 16\) \(Figure[17](https://arxiv.org/html/2607.11020#S6.F17)\)\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x17.png)Figure 17:Sequential writing of twenty facts\. Left: accuracy on earlier facts afterkkwrites\. Right: accuracy on each new fact at the time it is written\. The ability to write facts never degrades; the written facts do\.Two dissociations in Figure[12](https://arxiv.org/html/2607.11020#S6.F12)resist a single notion of damage\. Retention and capability loss disagree: distillation at rank 4 damages capability least among the sequential conditions \(−6\-6points\) yet retains earlier facts worse than SFT at the same rank \(27% against 42%\), while SFT at rank 16 retains best \(53%\) and erodes capability badly \(−16\-16\)\. And higher rank helps retention while hurting capability, the reverse of its usual reputation\. Whatever protects a model’s general abilities is not what protects its previously written facts, which is why survival is a question about the kind of write \(§[4](https://arxiv.org/html/2607.11020#S4)\), not about the amount of damage\.

## 7\.Causal tests of creation and interference

The preceding comparisons do not isolate why writing methods differ\. We therefore factorise the training data, cross the stored and incoming writes, and test whether local measurements can predict or control interference\. Figure[18](https://arxiv.org/html/2607.11020#S7.F18)summarises the results\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x18.png)Figure 18:Causal tests of creation and interference\.A: bare\-statement supervision leaves a 27\.4\-point entailment gap \(95% CI\[20\.2,35\.1\]\[20\.2,35\.1\]\); diverse recitation, with no derived conclusions, reduces it to 5\.4 points\[1\.8,8\.8\]\[1\.8,8\.8\]\.B: the linearised Adam update predicts the next update’s realised effect \(=0\.795\\rho=0\.795\) but not eventual forgetting \(=−0\.258\\rho=\-0\.258\)\.C: change in retention after fifteen later writes, by later\-write method, with fact\-clustered 95% intervals\.### 7\.1Training\-data factorisation

We trained nine controlled conditions on the same 32 prior\-inverting facts, holding the model, adapter rank, learning rate, step count, and seeds fixed\. The conditions separately vary recitation against use\-bearing examples, narrow against broad prompt coverage, hard labels against teacher distributions, and fixed against resampled trajectories \(Appendix[C](https://arxiv.org/html/2607.11020#A3)\)\.

Bare\-statement training leaves a 27\.4\-point entailment gap\. Training on 24 diverse recall and paraphrase prompts reduces it to 5\.4 points, although none contains a derived conclusion\. Use\-bearing examples in fact scored lower than diverse recitation in this experiment \(−8\.7\-8\.7points, 95% CI\[−15\.7,−1\.9\]\[\-15\.7,\-1\.9\]\)\. The supported causal variable is therefore prompt breadth, not the presence of an explicit reasoning step\. This result does not make every rich objective equivalent; it narrows what is necessary to create usable knowledge\.

### 7\.2Crossed interference

The continual comparisons of §[4](https://arxiv.org/html/2607.11020#S4)change the stored and incoming writes together\. A4×44\\times 4cross separates them\. Five facts are stored by bare\-statement training, study training, online distillation, or offline distillation; fifteen disjoint facts are then written by each method\. Retention is measured relative to the baseline after storage, with a second analysis conditioned on successful initial writing\.

The incoming\-write effect is clear, while the stored\-write effect is not\. After fifteen later writes, retention changes by−37\.0\-37\.0points for bare\-statement writes, against−6\.7\-6\.7for study writes,−1\.9\-1\.9for online distillation, and\+2\.4\+2\.4for offline distillation\. Each contrast with the bare\-statement stream excludes zero \(30\.4, 35\.1, and 39\.4 points\)\. The stored\-write effect changes sign under reasonable definitions of successful writing, so this experiment leaves it open\.

A larger2×22\\times 2experiment resolves the ambiguity\. It uses 24 new stored facts, three seeds, ten identical later writes, and a common item mask containing only questions answered correctly immediately after both storage methods\. The four final retention rates are 9\.0% \(bare\-statement store, bare\-statement later writes\), 47\.8% \(bare\-statement store, study later writes\), 7\.8% \(study store, bare\-statement later writes\), and 44\.2% \(study store, study later writes\)\. Averaged over storage method, changing the incoming writes from bare\-statement to study improves retention by 37\.6 points \(95% CI\[28\.9,46\.3\]\[28\.9,46\.3\]\)\. The storage\-method effect is−2\.4\-2\.4points \(\[−9\.2,4\.1\]\[\-9\.2,4\.1\]\), and the excess loss when bare\-statement writes meet a bare\-statement store is\+2\.4\+2\.4points \(\[−11\.5,14\.1\]\[\-11\.5,14\.1\]\) \(Figure[19](https://arxiv.org/html/2607.11020#S7.F19)\)\. Only the incoming\-write contrast crosses its pre\-registered threshold\. Interference is caused by the incoming write, not by how the earlier fact was stored\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x19.png)Figure 19:Paired effects on final strict retention in the crossed experiment\. Intervals are 95% bootstrap intervals over 24 stored facts after averaging three seeds\.This does not show that bare\-statement writes selectively erase a stored object\. Repetitive generation failures occur in 4\.2% and 3\.2% of probes under the two bare\-statement streams, against 0% under study streams, and under bare\-statement streams even the fact\-in\-prompt reference falls 45\.5 points below its level under study streams\. Restricting the analysis to the 232 of 250 items without a generation failure in any condition leaves the incoming\-write effect at 36\.8 points\. Generation failures do not create the contrast, but broad generation pathology remains a possible explanation\.

### 7\.3Local prediction and control

We next ask whether interference can be read from an update\. At steps 0, 1, 4, 16, 64, and 192, we measured changes on held\-out prompts \(statement, gold\-answer, and default\-answer log\-probabilities\)\. We also saved the adapter and optimiser, applied one representative update, measured its effect on held\-out prompts, and restored the exact state\. The pre\-registered early predictor — the step\-16 write–use score, the gold\-answer gain minus the statement gain on use prompts, frozen before the outcome runs — failed its threshold: its rank correlation with final strict\-accuracy gain was0\.0830\.083\.

A linearisation succeeds over the horizon it models\. Letguseg\_\{\\mathrm\{use\}\}be the gradient of the earlier facts’ use loss at the current parameters, and the preconditioned Adam update\. Its predicted effect on earlier knowledge is

L^use=guse⊤\.\\widehat\{\\Delta L\}\_\{\\mathrm\{use\}\}\\;=\\;g\_\{\\mathrm\{use\}\}^\{\\top\}\\,\\Delta\\theta\.\(7\.1\)The prediction correlates0\.7950\.795with the realised immediate effect but−0\.258\-0\.258with forgetting after the full sequence\. It describes the next update, not a fifteen\-update trajectory\.

None of the three intervention families crossed its pre\-registered threshold\. First, adding 48 steps of bridging data \(examples designed to connect the recited statement to its conclusions\) to 144 bare\-statement steps produced no condition that simultaneously halved the entailment gap, gained ten strict\-accuracy points, preserved lenient accuracy, and avoided extra capability damage\. This is a null for that finite recipe, not for bridging data in general\. Second, activation\-guided projection, patching, and block swaps did not meet their rescue or kill thresholds: the largest rescue was 5\.9 points against a 10\-point threshold\. These nulls accord with the warning ofhase2023localizationthat localisation need not identify the best site to edit\.

Third, we projected each Adam update away from measured conflict with the earlier facts,

=⟂−max\(0,g^use⊤\)g^use,g^use=guse/∥guse∥\.\\Delta\{\}\_\{\\perp\}\\;=\\;\\Delta\\theta\\;\-\\;\\max\\\!\\left\(0,\\;\\hat\{g\}\_\{\\mathrm\{use\}\}^\{\\top\}\\Delta\\theta\\right\)\\hat\{g\}\_\{\\mathrm\{use\}\},\\qquad\\hat\{g\}\_\{\\mathrm\{use\}\}=g\_\{\\mathrm\{use\}\}/\\lVert g\_\{\\mathrm\{use\}\}\\rVert\.\(7\.2\)The protected optimiser improves retention by only 1\.4 points over ordinary Adam \(95% CI\[−2\.1,5\.0\]\[\-2\.1,5\.0\], five unique stored facts\), far below the pre\-registered 15\-point target and no better than a norm\-matched random projection\. Parameter\-level gradient mitigation can work elsewhere\(yang2026collaborative\); here the measured conflict did not provide causal control\. Appendix[C](https://arxiv.org/html/2607.11020#A3)reports every threshold and control\. Broad training data changes the functional object, and bare\-statement incoming writes destroy access to it, but none of our local handles explains or repairs the long\-horizon dynamics\.

## 8\.Related work

Fine\-tuning learns unknown facts more slowly than known ones and can increase hallucination once new facts are finally learned\(gekhman2024finetuning;zucchet2025facts\)\. Facts held in weights also fail reversals and two\-hop compositions that succeed when the same facts appear in the prompt\(berglund2024reversal;balesni2024twohop\)\. Data breadth is the established remedy: paraphrases make stored facts easier to extract\(allenzhu2024physics31\), while augmented corpora outperform more documents or tokens\(yang2024entigraph;park2025newnews\)\. We separate optimisation budget from breadth in §[3](https://arxiv.org/html/2607.11020#S3)and isolate diverse recitation in §[7](https://arxiv.org/html/2607.11020#S7)\.

Knowledge editing encounters the same gap\. Locate\-and\-edit methods can answer single\-hop questions above 90% yet fail multi\-hop use\(meng2022rome;meng2023memit;zhong2023mquake;cohen2024ripple\), and their reported success falls under realistic questioning\(yang2025mirage;xie2025superficial\)\. Edits can suppress rather than erase an old fact\(holmov2026hidden\), while sequential editing degrades gradually and then catastrophically\(gupta2024editing\)\. At10510^\{5\}updates, continual fine\-tuning with adapter merging outperforms the tested editors\(thede2025lifelong\)\. Editors that survive many updates instead construct an address or constrain the incoming update\(hartvigsen2023grace;wang2024wise;fang2024alphaedit\)\.

Several findings likewise identify forgetting with lost access\. Continual\-learning losses can reverse after a few unrelated examples\(zheng2025spurious\); cues can revive abilities no longer elicited by ordinary prompts\(sun2025pseudo\); recall can limit knowledge at pretraining scale\(calderon2026recall\); and a newly learned fact can prime unrelated generations\(sun2025permeates\)\. Human memory research distinguishes available from accessible information\(tulving1966availability\), makes retrieval depend on the match between cue and encoding\(tulving1973encoding\), and attributes forgetting to interference from new learning rather than decay\(mcgeoch1932forgetting;anderson1994remembering\)\. We use this vocabulary descriptively, without claiming a shared mechanism\.

Context distillation internalises documents, prompts, and skills by matching a fact\-in\-prompt teacher\(snell2022context;eyuboglu2025cartridges\), and can propagate facts to entailed inferences better than direct fine\-tuning\(padmanabhan2023propagating\)\. Frozen teachers are stable where self\-distillation can diverge\(ye2026opcd;shenfeld2026sdft\); iterating through a moving teacher connects this failure to recursive self\-training\(shumailov2024collapse\)\. Regularisation towards earlier parameters is the classical remedy for forgetting\(kirkpatrick2017ewc\)\. Recent work instead links forgetting to KL from the base policy\(shenfeld2025rlrazor\), and finds that low\-rank training learns and forgets less\(biderman2024lora\), with the forgetting attributed to intruder directions rather than rank itself\(shuttleworth2024lora\)\. Classical plasticity loss, by contrast, concerns a model losing the ability to learn\(dohare2024plasticity\)\.

The two input channels support different generalisation\. Weight\-based generalisation is rule\-based while context\-based generalisation is exemplar\-based\(chan2022channels\), and in\-context learning is more flexible than fine\-tuning on matched content\(lampinen2025icl\)\. Some systems therefore keep new knowledge in compressed context\(mu2023gist\)or in key–value memories with explicit addresses\(berges2024memory\)\. When context contradicts parametric memory, override probability falls with confidence in the prior\(longpre2021entity;xie2024adaptive;wu2024clasheval;xu2024conflicts\), and contradictions are harder to instil deeply\(ghosal2024understanding;slocum2025believe\)\. We manipulate this fact–prior relationship in Appendix[D](https://arxiv.org/html/2607.11020#A4)\.

Controlled synthetic knowledge supports mechanistic experiments\(allenzhu2024physics31;morris2025memorize;kirchenbauer2025fictionalqa;li2025kup;wang2025synthworlds\)\. Judge bias and nondeterminism remain hazards\(zheng2023judging;wataoka2024self;yuan2025nondeterminism\)\. Our instrument combines invented entities, five question types, floor and fact\-in\-prompt anchors, and generation\-time screening; §[2\.1](https://arxiv.org/html/2607.11020#S2.SS1)describes generative\-lineage and checker failures that this design prevents\.

## 9\.Discussion

### 9\.1Stored but question\-keyed

The experiments separate creation, storage, access, and destruction\. Broad training data creates knowledge that can be used beyond recitation\. Later writes can make that knowledge behaviourally unavailable even though checkpoint reconstruction shows that its stored trace remains\. The crossed experiments locate the interference in the incoming write rather than in the way the earlier fact was stored\. Thus the same\-method comparisons show that bare\-statement training both creates and preserves knowledge poorly, but do not imply an intrinsic ordering of storage robustness\.

A successful weight write is therefore question\-keyed\. The model can answer the single\-fact questions used during training, yet cannot reliably state all of its written facts on demand or bring two of them into a joint computation\. The missing capability is what we mean by an*address*: a general handle for reaching a fact beyond the questions that happen to route to it\. After further writing, old questions can route to the newest fact instead\. Supplying the same content in context provides a general\-purpose handle: it supports joint use, survives further writing, and works for forgotten facts as well as fresh ones\. Retrieval\-augmented systems preserve this handle outside the weights\. The write stores content, while the context supplies its address at use time\.

This distinction also explains why the remedies separate\. A frozen teacher or an explicit KL penalty can preserve general capabilities, but does not preserve every route to earlier facts\. Local measurements can accurately predict the next update without forecasting the eventual trajectory, and projecting away the measured conflict does not improve on a random projection\. The remaining causal problem is not merely where a write leaves a trace, but which intervention gives that trace a route that survives subsequent writes\.

### 9\.2Limits and boundaries

All experiments use one base model family, Qwen3\-4B, with an 8B replication of the entailment gap\. The facts are invented and follow one style\. We use LoRA adapters except where full fine\-tuning is stated\. The causal factorisation covers 32 facts, with repeated measurements rather than independent samples\. The crossed experiments cover five and then 24 stored facts, three seeds, and ten to fifteen later writes\. Bare\-statement streams also degrade fact\-in\-prompt generation, so their effect is not shown to be selective erasure\.

The access experiments cover two training methods at one operating point\. The storage probe reads only the statement’s log\-probability\. The relearning comparison rests on three forgotten facts, and the joint\-use evaluation covers fourteen fact pairs\. A 60\-item audit found seven judge errors; correcting them moves the later\-write effect from 37\.6 to 37\.8 points\. The bridge and optimiser nulls apply only to the tested recipes, thresholds, and facts\.

The boundary is accumulation, not all writing\. Weight updates remain useful as a cache when a canonical copy exists elsewhere, because frozen\-teacher batch distillation preserves general capability while consolidating many facts\. Pretraining also installs usable knowledge and can be understood as the limit of broad study data\. Finally, our evidence concerns discrete facts; whether fine\-tuned skills fail in the same way remains untested\.

### 9\.3Continual writing

Continual writing has three requirements\. Each write must create usable knowledge, general abilities must survive accumulating updates, and earlier facts must remain reachable\. Data breadth makes the first tractable, while a frozen teacher or explicit penalty makes the second tractable\. The third remains unresolved\. Forgetting reflects lost access rather than erased storage, joint use can fail before later writes arrive, and none of the local signals we tested controls long\-horizon interference\. For facts that must be retrieved on demand, composed, and preserved through further training, weights are therefore the wrong system of record: they store content without creating an address for it\.

A stronger mechanistic account must connect measurement to control\. It should forecast final behaviour beyond the next update, protect earlier facts without blocking new ones, and explain how broad training data connects content to more routes\. Objectives may provide another lever\. Our reinforcement\-learning condition occasionally gained reward without installing the fact, because accuracy alone does not distinguish storage from other ways of producing an answer\. An objective that also prices reasoning length might reward direct recall, but whether it would place knowledge in the weights is untested\.

The practical guidance is correspondingly narrow\. Use broad data for each write, avoid bare\-statement training when later updates must preserve existing knowledge, and distil against a frozen copy of the original model\. When facts must compose or survive extensive later writing, keep them in context\. Models continue to learn new facts after many writes; what fails is the route back to the earlier ones\. Continual learning belongs in a channel that creates an address as well as storing content\.

## References

## Appendix AApparatus details

### A\.1Glossary of conditions

The paper uses one plain name per condition throughout\.

### A\.2Instrument defect classes

Table[2](https://arxiv.org/html/2607.11020#A1.T2)lists the six defect classes of §[2\.1](https://arxiv.org/html/2607.11020#S2.SS1)and the direction each biases\. The first five surfaced in a line\-by\-line audit triggered by an unexplained drop when one condition was re\-evaluated on questions from a different model family; the sixth required auditing the judge’s behaviour rather than its decisions\. All six are plausible in any fine\-tuning evaluation that generates its own training and test data\.

Table 2:The six instrument defect classes, and the direction each biases\.
### A\.3Training

Qwen3\-4B, bf16, thinking mode disabled, base weights frozen\. LoRA: rank 16 \(or 4 where stated\),=32\\alpha=32, dropout 0, applied to all seven projection matrices \(attention q/k/v/o, MLP gate/up/down\)\. AdamW, learning rate2×10−42\\times 10^\{\-4\}, loss on answer tokens only, optimiser steps counted at batch size one\. Single\-fact conditions train a fresh adapter per fact\. Sequential conditions merge each fact’s adapter into the running model before the next fact trains\. Batch conditions train one adapter on allNNfacts for24×N24\\times Ntotal steps\. In context distillation, teacher and student share weights, the teacher being a forward pass with the adapter disabled and the fact in context\. The teacher\-transcript ablation trains with plain cross\-entropy on the same teacher\-sampled sequences, isolating the data half of distillation; its contrasts are in Appendix[B](https://arxiv.org/html/2607.11020#A2)\.

The larger crossed experiment uses Qwen3\-4B with rank\-16 LoRA and the same2×10−42\\times 10^\{\-4\}learning rate\. Each stored fact and each of ten shared later facts receives 192 optimiser steps\. The 288 sequences cross two storage methods, two later\-write methods, 24 stored facts, and three seeds\. The two data conditions are matched on steps, not tokens or examples\. Runs used a frozen dependency lockfile \(torch 2\.12\.1, transformers 5\.13\.0, peft 0\.19\.1\); the exact base\-model revision was not recorded in the run rows\.

The study set is 24 items per fact — paraphrases, question–answer pairs, worked implications, and contrasts with the violated default — generated once and reused across all conditions; the generator never sees the evaluation questions\.

### A\.4Certification gate

An evaluation is used only after passing a scripted gate:

The primary evaluation passed at floor 3%, ceiling 84%, leak 1/30, pass\-audit 29/30\. Truncation for the original model \(23%\) and bare\-statement training at 24 steps \(11%\) is reported but not gating; it deflates the floor side of comparisons\. Audits run at temperature zero with majority\-of\-three judgments per item\.

### A\.5Tier evaluation: certification amendments

Two amendments were adopted, both before recomputation\. First, the leak audit returned different counts on identical inputs \(5/30, then 3/30\) because its sampling temperature was unpinned; audits were pinned to temperature zero with majority\-of\-three judgments, and confirm\-tier questions are audited only for wording that reveals the entity binding, since answerable\-from\-common\-sense is that tier’s definition and the live floor screen controls it\. Second, the fact\-in\-prompt condition initially scored 77–80% on two tiers against an 80% bar: screening questions for original\-model failure selects for hardness, keeping a tail of questions that fail even with the fact available\. Such questions were removed, the surviving set was frozen before any training run, and the certifying criterion became the surviving fraction, at least 70% per tier \(observed: confirm 76\.3%, neutral 95\.9%, invert 79\.5%; floors on surviving questions 0\.9–6\.2%; 48/1/23 facts per tier left with fewer than four questions\)\.

### A\.6Statistics

Bootstrap 95% confidence intervals over facts\. Between\-condition comparisons are paired per\-fact differences; cross\-tier comparisons are paired within triples\. Multi\-seed cells report per\-seed values and their mean\. Creation\-factorisation and crossed\-interference intervals cluster repeated measurements by unique fact; 864 creation units correspond to 32 facts\. The optimiser analysis follows its pre\-registered resampling unit instead: 60 paired observations over store, seed, and stored fact, drawn from five unique stored facts\. The larger crossed analysis intersects the questions answered correctly after both storage methods for each fact–seed pair, averages seeds within each of 24 stored facts, and resamples facts with 20,000 bootstrap draws\. The pre\-registered thresholds are 15 points for the later\-write contrast and 10 points for the stored\-write and interaction contrasts\. A 60\-item stratified endpoint audit found 53/60 judge agreement; applying all seven manual corrections did not change the result\.

## Appendix BSupplementary results

### B\.1Single\-fact conditions with confidence intervals

Accuracy \(strict policy\) by question type over the 247\-fact evaluation, bootstrap 95% CIs over facts\. CD is context distillation; the transcript ablation trains with cross\-entropy on the same teacher\-sampled sequences\.

The bare\-statement column is 24 steps and the study column 96 steps \(each recipe’s operating point\)\. Paired composition differences \(n=236n=236\): CD−\-study set=\+10\.8=\+10\.8\[\+4\.9,\+16\.5\]\[\+4\.9,\+16\.5\]; CD−\-fact in prompt=−13\.1=\-13\.1\[−18\.2,−7\.8\]\[\-18\.2,\-7\.8\]; transcript ablation−\-study set=\+1\.5=\+1\.5\[−4\.9,\+7\.4\]\[\-4\.9,\+7\.4\]; CD−\-transcript ablation=\+9\.3=\+9\.3\[\+4\.9,\+13\.8\]\[\+4\.9,\+13\.8\]\. Under the strict policy the distillation gain is real and, unlike our earlier lenient\-graded estimate, the CD\-over\-transcript increment excludes zero: the distillation loss, not only the teacher’s sampled sequences, carries the gain\.

### B\.2Prior strength as a per\-fact predictor \(exploratory analysis\)

The regression that motivated §[D](https://arxiv.org/html/2607.11020#A4)\. Each fact’s prior strength was scored 0–3 from the original model’s own answers to that fact’s questions \(its wrong answers state the belief it holds\), for 241 of 244 facts; 78% scored 2 or 3, reflecting the set’s built\-in\-conflict construction\. Outcomes are per\-fact accuracies from existing certified records; the statistic is tie\-adjusted Spearman with bootstrap CIs\.

Prior strength does not predict which facts win at inference time, in any condition\. The retention correlation is negative in all four sequential conditions and unpowered atn=19n=19\. The powered test on the tier facts subsequently ran \(§[4\.2](https://arxiv.org/html/2607.11020#S4.SS2)\) and reversed the registered prediction: prior\-inverting facts survived better than neutral ones, which we read post hoc as distinctiveness protecting\. The relaxation reading of the prior\-as\-opponent account, in which sequential training lets the prior recover its most\-contradicted facts first, did not survive that test\. Two artifacts could have produced the inference\-time null here: the compressed conflict range, and a per\-fact outcome that is nearly one binary question\. The tier design removes both\.

### B\.3A post\-scope\-change extension to one hundred writes

The hundred\-write extension of Figure[8](https://arxiv.org/html/2607.11020#S4.F8)covers a balanced eight\-cell comparison, reduced from a planned twelve after partial records existed: SFT and offline distillation with bare\-statement or self\-study data and either low\-rank or full\-parameter updates\. Each tuned operating point has three seeds and one hundred sequential writes\. It is a descriptive extension, not a confirmatory factorial: the reduced scope, endpoint contrasts, and uncertainty rule were not fixed before the data existed, and online\-distillation partials are excluded\.

All 24 retained records pass coverage validation\. This experiment’s first\-sentence response protocol, which scores only the first sentence of each response, reached a 77\.3% in\-context ceiling, below the\>80%\>80\\%certification gate; the miss was accepted before the runs and caps what the endpoint levels can show\. Atk=100k=100, bare\-statement SFT retains 8\.6–11\.7% of earlier facts and study SFT 33\.0–37\.7%\. Offline distillation retains 25\.0–29\.4% with bare\-statement data and 38\.5–43\.7% with study data\. These are ranges across the low\-rank and full\-parameter cells, not confidence intervals\. Offline distillation is higher in all four matched endpoint comparisons, but the trajectories cross at earlier checkpoints\.

Held\-out capability losses at the endpoint range from 73\.5 percentage points for full\-parameter bare\-statement SFT to 9\.0 points for low\-rank study distillation\. Full\-parameter bare\-statement distillation loses 56\.9 points; the corresponding low\-rank bare\-statement SFT and distillation conditions lose 34\.2 and 28\.7 points\. Learning rates were selected by condition, and full versus low\-rank updates change both learning rate and parameterisation, so these numbers describe the tested operating points rather than independent causal effects of data, objective, or update parameterisation\.

## Appendix CThe causal experiments: design, controls, and limits

### C\.1Creation factorisation and checkpoints

All creation cells use Qwen3\-4B with rank\-16 LoRA, learning rate2×10−42\\times 10^\{\-4\}, 192 optimiser steps, three seeds, and the same 32 held\-out prior\-inverting facts\. The nine cells are: two narrow bare\-statement prompts; 24 diverse recall and paraphrase prompts with no derived conclusion; two repeated one\-step transformation prompts; 24 diverse application, counterfactual, and composition prompts; the study data; hard\-label cross\-entropy on frozen in\-context\-teacher rollouts; forward KL on those fixed teacher trajectories; forward KL on trajectories sampled once from the student initialised from the original model; and forward KL on adaptively resampled current\-student trajectories\. Checkpoints at steps 0, 1, 4, 16, 64, and 192 record gold, statement, teacher, and default\-answer log probabilities as well as generated strict and lenient accuracy\. The 864 fact–condition–seed units are repeated measurements on 32 facts, not 864 independent facts; intervals cluster by fact\.

The micro\-update test snapshots both adapter and optimiser state, performs one actual update, measures before–after probe log probabilities, and restores the state byte\-for\-byte\. Its linear predictor includes the realised update magnitude and Adam preconditioning\. Generation used a separately certified exact\-decode cycle, and judge abstentions are never imputed\.

### C\.2Crossed interference design and protected optimiser

The4×44\\times 4crossed design stores five facts using bare\-statement training, study training, online forward\-KL context distillation, or offline forward\-KL context distillation, then writes fifteen disjoint facts over them using each of the same methods\. Every distillation condition uses a frozen original\-model teacher\. The stored facts are re\-evaluated after 1, 5, 10, and 15 later writes\. The primary endpoint is change from the baseline taken after the stored facts were written; a second analysis conditions on successful initial writing\. The distinction matters: the stored\-write main effect changes under reasonable definitions of successful writing, while the later\-write effect does not\.

The protected optimiser retains one held\-out use prompt and gold answer for each earlier fact\. Periodically it aggregates the old\-use gradient, forms the actual preconditioned Adam update, and removes the component that increases old\-use loss to first order\. Ordinary Adam, norm\-matched random projection, and replay\-free magnitude\-based module masking are controls\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x20.png)Figure 20:Intervention controls\.A:Each row is an eligible tested 144\-step bare\-statement plus 48\-step bridge condition; indigo marks a passed individual criterion\. No row passes all four\.B:Retention change relative to ordinary Adam\. Targeted gradient projection gains 1\.4 points, random projection 1\.8, and magnitude masking 5\.0\. Intervals resample the 60 paired store–seed–fact units; only five stored facts are unique\.
### C\.3Intervention scope and exploratory observations

Figure[20](https://arxiv.org/html/2607.11020#A3.F20)shows why the bridge and optimiser gates failed\. The selectedk=8k=8bridge condition was retained only as a diagnostic because selection and evaluation were not independent; it halved the gap and gained 11\.3 strict\-use points, but lost 7\.0 lenient points and therefore could not pass the conjunction\. No conclusion extends beyond the tested 144\+48\-step mixture, the candidate examples, and these facts\. For the optimiser, targeted projection is also indistinguishable from random projection \(difference−0\.4\-0\.4points\[−4\.4,3\.6\]\[\-4\.4,3\.6\]\)\. Magnitude masking has the largest estimate,\+5\.0\+5\.0points\[1\.4,8\.9\]\[1\.4,8\.9\], but still misses the target\.

Exploratory work found some SFT\-written facts decodable at intermediate depth, and restricting updates to deeper layers affected SFT and distillation differently\. Our projection, patching, and fixed\-block swaps did not establish necessity or sufficiency \(activation differences can trace narrow fine\-tuning\(minder2026traces\)\), so we retain the depth and logit\-lens observations as motivation only: they do not establish a lookup\-versus\-computation mechanism, and we did not apply a richer lens to components that failed the causal gate\.

### C\.4Scope and adversarial review

All confirmatory results are scoped to Qwen3\-4B, rank\-16 LoRA, the 32 creation facts, five stored facts, and fifteen later writes\. Each report passed automated balance and provenance checks and independent adversarial review; the main caveats are repeated measurements on a small fact set, sensitivity of the store effect, post\-selection in diagnostic bridge rows, and the gap between a one\-step linear approximation and a multi\-update trajectory\. These experiments reject the pre\-registered early\-predictor, transformation\-necessity, and gradient\-projection hypotheses under their protocols\. They do not show that all predictors, bridge recipes, internal mechanisms, or optimiser protections must fail\.

## Appendix DPrior conflict as the experimental variable

This appendix reports the prior\-conflict experiment referenced in §[4](https://arxiv.org/html/2607.11020#S4)and §[9](https://arxiv.org/html/2607.11020#S9)\. Its primary contrast was inconclusive, and its cue analysis corroborates §[3](https://arxiv.org/html/2607.11020#S3): cuing a question reduces it to the recitation a bare\-statement write installs\.

Every fact in the primary evaluation contradicts a prior belief by construction, so the results of §[3](https://arxiv.org/html/2607.11020#S3)cannot say whether the prior causes the diversity requirement, while outside evidence says the prior matters\(wu2024clasheval;slocum2025believe\)\. We therefore built a second certified evaluation in which the fact–prior relationship is the manipulated variable \(Figure[21](https://arxiv.org/html/2607.11020#A4.F21)\)\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x21.png)Figure 21:The matched\-triple construction\. Three invented entities share a domain and statement structure and differ only in how the stated property relates to a prior belief: it confirms the default, is neutral to any default, or inverts the default\.A regression on the existing facts was not enough: per\-fact prior strength gave a null for counterfactual use \(≈0\\rho\\approx 0in every condition,n=179n=179\), but that analysis is doubly confounded, by the set’s compressed conflict range and its nearly binary per\-fact outcome \(Appendix[B\.2](https://arxiv.org/html/2607.11020#A2.SS2)\)\. Manipulating conflict directly, down to zero, removes both problems\.

### D\.1Construction

Facts come in matched triples sharing a domain, statement structure, and entity\-name style, with three distinct invented entities\. The members differ only in how the stated property relates to a prior belief: it*confirms*the default \(“Zerandite conducts current more efficiently as its temperature drops toward absolute zero”\), is*neutral*to any default \(“Molvarite is named after the Molvar Basin where it was first mined”\), or*inverts*the default \(“Kethralite conducts current less efficiently…”\)\. Entity novelty is common to all three, so the original model fails recall everywhere; only the conflict varies\. Matching makes cross\-tier comparisons paired\. A generation\-time screen discarded triples whose tier a fresh model call could not recover blind \(47% discarded\); after deduplication the set is 80 triples, 240 facts\.

Questions follow §[2](https://arxiv.org/html/2607.11020#S2), written by a second model family and kept only if the original model fails them, with two declared asymmetries\. Confirm\-tier questions about*using*the property are answerable from the default alone, so that tier plays a supporting role; the primary contrast is neutral versus invert\. And the fifth question type must differ by tier, since “side with the fact against the default” requires a default: invert keeps counterfactual questions, neutral gets distractor questions \(commit to the arbitrary value against a plausible alternative\), confirm has four types\.

Certification required two amendments, detailed in Appendix[A](https://arxiv.org/html/2607.11020#A1); all three tiers then passed the gate, and the screened question set \(76–96% of questions surviving per tier\) was frozen before any training run\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x22.png)Figure 22:Floor and ceiling for the three tiers under the certified judging pipeline, with the fraction of questions surviving the screen in the tick labels\. Floors of 0\.9–6\.2% show the questions do not leak their answers in any tier\.A further observation: the fact\-in\-prompt ceiling itself orders by tier \(neutral 96%, invert 80%, confirm 77%\)\. Content that interacts with a prior belief, in either direction, taxes even in\-context use\.

### D\.2The conflict experiment

If the prior causes the diversity requirement, the study\-training advantage on use\-type questions should collapse for neutral facts and persist for inverting facts\. If diversity instead works by making the fact retrievable under many phrasings, the advantage should persist in both\.

The design: bare\-statement and study training at 24 and 96 steps, on all 240 facts, three seeds, scored on the frozen question set by the certified pipeline\. The primary quantity is the study−\-bare gap at 96 steps on use\-type questions \(application, composition, and the tier’s fifth type\), paired within triples, averaged over seeds, per tier\. Recall questions are the negative control: neither account predicts a tier\-by\-recipe interaction on recitation, so one appearing would indicate an instrument artefact\. We also examine the composition gap to the ceiling by tier — if neutral facts compose near their 96% ceiling while inverting facts stay capped, the composition limit of §[3](https://arxiv.org/html/2607.11020#S3)is itself a conflict phenomenon\.

### D\.3Results

The full grid comprises 13,728 records \(240 facts×\\times4 recipes×\\times3 seeds\), all scored on the frozen question set by the certified pipeline\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x23.png)Figure 23:The conflict experiment’s primary quantity \(strict policy\): the study advantage over bare\-statement training on use\-type questions, paired within triples, averaged over three seeds, with 95% CIs, by prior tier \(invert 77 triples, neutral 80, confirm 61\)\. Every interval crosses zero and none reaches the\+10​pp\+10\\,\\mathrm\{pp\}support band \(dotted\); recomputed on non\-truncated rows only \(“non\-truncated”\), the gaps collapse toward zero — the residual signal is a truncation artefact\.Under the strict policy the gaps are small, positive, and every confidence interval crosses zero \(Figure[23](https://arxiv.org/html/2607.11020#A4.F23)\); the paired invert−\-neutral interaction is\+4\.3​pp\+4\.3\\,\\mathrm\{pp\}\[−6\.1,\+14\.2\]\[\-6\.1,\+14\.2\]\. Against the pre\-registered bands \(support requires an invert gap≥\+10​pp\\geq\+10\\,\\mathrm\{pp\}with the interval excluding zero, and a neutral gap≤\+5​pp\\leq\+5\\,\\mathrm\{pp\}\) the result isinconclusive: this instrument neither reproduces the primary evaluation’s diversity effect nor rules it out\. One material caveat explains part of the residual: bare\-statement conditions truncate far more at the 512\-token answer cap than study conditions \(6\.1% versus 0\.1% at 96 steps\), the same defect class as the answer\-length cap of §[2\.1](https://arxiv.org/html/2607.11020#S2.SS1)\. Recomputed on non\-truncated rows only, every gap collapses toward zero \(invert\+0\.1​pp\+0\.1\\,\\mathrm\{pp\}, neutral\+0\.2​pp\+0\.2\\,\\mathrm\{pp\}\), so the small positive signal in the unfiltered values is largely a truncation artefact working*for*the diversity hypothesis\. The pre\-registered remedy is more triples; the sharper one, developed below, is fixing the question style that drives the discrepancy\. The negative control behaved: recall gaps are uniform across tiers \(−6\.2\-6\.2/−5\.4\-5\.4/−8\.7​pp\-8\.7\\,\\mathrm\{pp\}, the familiar study\-lags\-recitation pattern of §[3](https://arxiv.org/html/2607.11020#S3)\), with no tier\-by\-recipe interaction\.

![Refer to caption](https://arxiv.org/html/2607.11020v1/x24.png)Figure 24:Left: under the strict policy the diversity effect is small and every interval crosses zero; the expected effect size \(shaded band\) sits at the upper edge of the intervals\. Right: composition sits at 37–58% for every recipe in every tier while the in\-context ceiling spans 77–95%\.
### D\.4The cue mechanism, measured

The inconclusive result has a specific cause, and the dual\-policy regime makes it visible\. On the tier counterfactual questions bare\-statement training scores 87–88% strict \(ceiling 93%\); on the primary evaluation’s counterfactual questions the same recipe scores 23% strict\. Study training is comparatively stable across the two \(79% on the tiers, 50% on the primary\)\. The gap is not prior strength: under the identical scorer the inverting tier facts carry*more*prior conflict than the primary set \(89% versus 78% high\-conflict\)\. What separates the two instruments is question style\. The tier counterfactual questions*cue the contrast*in their wording \(“Where many pigments visibly bleach…” — the surface form of the trained statement\), while the primary evaluation presents bare scenarios that demand*spontaneous*override\.

Read through the entailment gap of §[3\.1](https://arxiv.org/html/2607.11020#S3.SS1), this is one mechanism, not two\. A cued question does for free the step that diverse data otherwise has to buy: it converts reciting the trained statement into stating the asked\-for conclusion\. Bare\-statement training, which installs recite\-plus\-one\-step knowledge, therefore looks strong exactly where the question supplies the missing step \(the cued tiers, high strict scores\) and weak where it does not \(the uncued primary, low strict scores\); study training, which installs stated conclusions, is roughly cue\-independent\. Because the two instruments differ in more than their cuing, we ran a controlled confirmation: a full cue×\\timesrecipe design, retrained on both fact sets with cued and uncued counterfactual question sets, each verified by a blind cue\-classifier\. It splits the effect in two\. The*cue main effect*is robust, large, and specific to bare\-statement training: stating the default and asking for the contrast raises bare\-statement strict accuracy52→88%52\\to 88\\%on the tier facts and23→99%23\\to 99\\%on the primary facts, roughly twice the lift it gives the study condition \(difference\-in\-differences\+24\.7​pp\+24\.7\\,\\mathrm\{pp\}and\+44\.1​pp\+44\.1\\,\\mathrm\{pp\}\)\. The mechanism is not that cuing unlocks applicable knowledge\. The cued questions are, by construction, answerable by*reciting*the written statement: “normallyXX; for this entity, what instead?” hands the model the frame\. Cuing therefore converts bare\-statement training’s recitations from insufficient to sufficient, collapsing its entailment gap from31\.8​pp31\.8\\,\\mathrm\{pp\}uncued to0\.9​pp0\.9\\,\\mathrm\{pp\}cued\. It rescues bare\-statement knowledge by reducing the question to the recitation the write installed \(§[3](https://arxiv.org/html/2607.11020#S3)\), not by making that knowledge more usable\. The magnitude is further inflated by the cued questions’ shorter inference distance, with the fact\-in\-prompt condition scoring 95% cued against 76% uncued\. But the*diversity*×\\times*uncued*interaction — the claim that study training uniquely buys spontaneous override — did not reproduce on the tier facts under bare uncued phrasing\. A diagnostic resolved why\. The tier uncued question set, though it hides the default, was about half*recitation\-answerable*, meaning the written sentence, restated, is the answer; the primary questions are∼\\sim2%\. Recitation specifically flatters bare\-statement training: its item\-level pass is predicted by recitability \(point\-biserialr=0\.42r=0\.42\), and study training’s is not \(r=−0\.22r=\-0\.22\)\. Restricted to application\-only tier items — where reciting the sentence does not answer the question — the diversity advantage reappears, matching the primary facts\. So the diversity requirement for*spontaneous*\(uncued, application\) override stands on the primary facts and is direction\-consistent on the tier facts; a clean tier replication awaits a purpose\-built application\-only question set\. We treat the cue main effect as the confirmed result here and scope the diversity claim to spontaneous\-application questions\.

The composition limit is conflict\-independent\. Composition sits at 37–58% for every tier and recipe while the ceiling ranges from 77% to 95%, and the gap is*largest*at zero conflict \(neutral: 58% versus a 95% ceiling\)\. Conflict joins training data, training objective, update method, and model scale as manipulations this limit survives\. Composition failure is about where the fact lives — weights versus activations — not about what it fights\.