@oneill_c: 1/ Can you actually get new facts into an LLM's weights without breaking the model? This question decides how we approa…

1/ Can you actually get new facts into an LLM’s weights without breaking the model?

This question decides how we approach continual learning: should memory live in the context (retrieval, compressed caches) or in the weights themselves?

We spent a long time measuring it, and it breaks somewhere much stranger than we expected, making us much more bullish on compressed kv caches and ICL for continual learning, as opposed to weight updates themselves

2/ If weight writes work, models can keep learning after deployment the way they learned everything else, and memory systems are a workaround. If they don’t, memory belongs in context, and we should engineer that channel instead.

Until now we felt like this was mostly argued on vibes

3/ Setup: invented facts the model demonstrably doesn’t know. Five question types per fact, from recall to multi-hop use. We compare to the original model (floor), and the same model with the fact in its prompt (ceiling).

Every write gets judged between those two

4/ The first surprise is what a write creates depends on the training data, not the objective. If you train on the bare statement, the model can recite the fact but can’t use it (a 27-point gap vs. the same fact in context)

Train on diverse restatements and the gap drops to 5, so breadth is the variable

5/ Now write 20 facts, one at a time. For facts trained from bare statements, 1% survive, whereas facts trained from diverse restatements 46% survive

The kind of knowledge a write creates predicts whether it survives. Push to 100 writes and retention plateaus at 25–28%

6/ However, what’s whacky is that the forgotten facts aren’t gone. After a fact fails every question we can ask, 57–67% of its write’s log-probability lift is still sitting in the weights. So the content is there, but the language model has forgotten how to write to it

7/ Where do the questions go instead? Under bare-statement writes, 70% of a forgotten fact’s wrong answers contain the newest fact’s content.

Later writes don’t erase earlier facts; they instead capture the questions that used to reach them

8/ And this failure exists before any forgetting. If you teach a model two facts, each one individually usable, and ask a single question that needs both: 32%. Put the same two facts in the prompt instead: 91%. A write thus stores content, but it never creates an address.

9/ Can you at least avoid breaking the model? Not iteratively, which is the main regime we care about for continual learning!

Capability damage is very correlated with KL divergence from the original model (ρ = 0.83 across 12 conditions). If you distill each write against a frozen copy of the original and 20 facts cost +2 points of capability.

But even this safest write loses nearly half its facts by the twentieth

10/ We tried to control it locally. A linearised predictor predicts each update’s immediate effect (ρ = 0.795), but unfortunately tells you nothing about eventual forgetting (ρ = −0.258). Projecting updates away from measured conflict is no better than a random projection.

So nothing local steers the long-horizon dynamics

11/ Meanwhile, the context channel just works. Through the same write sequences, in-context use erodes no faster than general capability. A forgotten fact answers at 77–80% again the moment its statement is back in the prompt

12/ So, can a model learn facts continually in its weights?

Creating usable knowledge: solvable (data breadth). Preserving capability: solvable (frozen teacher, KL penalty). Keeping earlier facts reachable: unsolved — by anything we tested.

Weights are a fine cache and a bad system of record.

13/ What I take away from all this is that all the training to create the model is engineered around crafting the best possible ICL mechanism. Further training degrades this mechanism, at least for knowledge acquisition, and maybe continual learning should just focus on loading the right information up for icl (ie into the context window)

So this provides an answer to the architecture question, at least for now: the channel worth engineering for memory is the one that comes with addresses ie the context.

Paper: https://arxiv.org/abs/2607.11020 Code and data to follow shortly

Done at @baseten.