语言模型能在思考时学习吗？测试跨架构的梯度适应能力

# 1Introduction
Source: [https://arxiv.org/html/2610.33025](https://arxiv.org/html/2610.33025)
Can Language Models Learn While They Think? Testing Gradient Adaptation Across Architectures

Surya Saka∗1

1JudicialMind

Abstract

- Letting a language model take gradient steps on the text it is reading, known as dynamic evaluation, lowers its loss on the rest of the document\. The technique predates the current generation of open models, and its behaviour on hybrid linear\-attention architectures, on diffusion language models, and on the benchmarks that long\-context work is judged by has not been measured\. We measure it\. With one SGD step per 128\-token chunk on the MLP output projections of the last quarter of blocks, no training, and a 2k\-token window, softmax\-attention models \(Qwen3\-1\.7B, Qwen3\-4B, Gemma\-4\-E4B\) lose 5–8% of their next\-token loss over a document and outperform the same frozen model given the full 32k\-token context\. Hybrid Gated\-DeltaNet models \(Qwen3\.5\-4B, Qwen3\.8\-27B\) gain under 1% at every learning rate: the linear\-attention blocks already perform the adaptation that the gradient step would add\. The mechanism transfers to masked\-diffusion language models with a larger effect, 8% on Nemotron\-Diffusion\-8B and 17–20% on DiffusionGemma\-26B, once the rate is lowered and updates are clipped\. Across RULER, BABILong, LongBench\-v2, SCOTUS, HotpotQA, MuSiQue, ContractNLI and citation completion, at 100 or more items on two model sizes, weights updated on the full document recover none of the accuracy lost to context truncation: the adaptation is style and vocabulary, not retrieval\. Writing facts with the next\-token loss stores the training format rather than the fact \(30% recall under chat prompting against 80% in context\) and costs 36–38 GSM8K points at every dose\. Writing a conclusion as behaviour instead, by distilling the model’s own conclusion\-conditioned judgments into a low\-rank adapter, reaches the in\-context ceiling \(0\.83 and 0\.88 against 0\.84 and 0\.88 on 100 games, softmax and hybrid\) with GSM8K unchanged, and the written conclusion can come from an inquiry loop that found it by asking information\-gain questions\. Every number is reproduced from per\-item records in the released run logs\.

Keywordsdynamic evaluation⋅\\cdottest\-time training⋅\\cdotfast weights⋅\\cdotlinear attention⋅\\cdotdiffusion language models⋅\\cdotlong context⋅\\cdotmodel editing⋅\\cdotcontext distillation††footnotetext:∗Surya@judicialmind\.ai

## 1Introduction

A language model that keeps learning while it reads is an old idea with a new audience\. Dynamic evaluation\[[16](https://arxiv.org/html/2610.33025#bib.bib16)\]adapts a model’s parameters to the test stream by gradient descent\. It was shown to lower perplexity on Transformer\-XL\. A 2024 revisit\[[25](https://arxiv.org/html/2610.33025#bib.bib25)\]confirmed the effect on 150M–1B models trained from scratch on PG\-19\. The revisit framed it as memory in weights versus memory in context\. Since then the models people actually deploy have changed in three ways that bear directly on whether the idea still pays\. Open models are now 2B–30B parameters and instruction\-tuned\. A growing share of them replace most softmax\-attention blocks with gated linear attention\[[32](https://arxiv.org/html/2610.33025#bib.bib32)\]\. By construction it keeps a fast\-weight state updated by a delta rule as tokens arrive\[[31](https://arxiv.org/html/2610.33025#bib.bib31)\]\. Masked\-diffusion language models have reached competitive quality without any autoregressive objective at all\. None of these settings has a measurement of what inference\-time gradient adaptation buys\. The standard long\-context benchmarks, on which a weight\-based memory would have to show its value, have never been run against it\.

We ran the technique on all three families and on the benchmarks with one fixed recipe and no training\. The recipe was deliberately the minimum\. We made the MLP output projection of the last quarter of transformer blocks writable\. After each 128\-token chunk was scored, we took one SGD step on the next\-token loss\. The attention window stayed at 2k tokens, and we reset the weights per document\. Everything else in the model stayed frozen\. This isolates the question of interest, what the gradient adds to what the architecture already does in context, from the separate question of what a model trained to use its fast weights could do\[[30](https://arxiv.org/html/2610.33025#bib.bib30),[14](https://arxiv.org/html/2610.33025#bib.bib14)\]\.

Four findings come out of 131 runs on two H100s\. First, the adaptation gain is a property of the attention architecture\. Softmax models lost 5–8% of their loss over a book and beat the frozen model given the full 32k context, on every book tested\. Hybrid Gated\-DeltaNet models gained 0\.3–0\.8% at the best rate and lost at higher rates\. Second, the same gradient step worked on masked\-diffusion language models\. We ran it at a lower rate and clipped the updates\. The effect was larger\. We saw 8% on a dense 8B diffusion model and 17–20% on a 26B block\-diffusion model\. Third, the gain is not retrieval\. The frozen model lost 6–50 points on eight benchmarks once we cut the context to the window\. Updating the weights on the whole document first recovered none of them, atn≥100n\\geq 100on a 4B and a 27B model\. Fourth, the two routes by which one might store something specific in these weights diverged sharply\. Next\-token writes of facts memorised the prompt format and destroyed generated reasoning\. A distillation write of a conclusion the model reached itself landed at the in\-context ceiling with reasoning intact\.

The contributions are the measurements and the controls that make them interpretable: a context\-versus\-adaptation control at 32k tokens on an off\-the\-shelf model, per\-model learning\-rate curves, a layer\-location ablation, a surprise\-gating ablation, a matched\-drift comparison of two write objectives on the same parameters, and a composition in which an explicit Bayesian inquiry loop supplies the conclusion that gets written\. Section[2](https://arxiv.org/html/2610.33025#S2)places the work against dynamic evaluation, trained test\-time\-training layers, model editing and context distillation\. Section[3](https://arxiv.org/html/2610.33025#S3)fixes the mechanism and the evaluation protocol\. Section[4](https://arxiv.org/html/2610.33025#S4)reports the results\. Section[5](https://arxiv.org/html/2610.33025#S5)gives the mechanism\-level reading and the design rule that follows from it\.

## 2Related Work

#### Dynamic evaluation and test\-time training\.

[Krause et al\. \[16\]](https://arxiv.org/html/2610.33025#bib.bib16)adapt all parameters of a Transformer\-XL to the test sequence by gradient descent\. They reduce WikiText\-103 perplexity from 18\.3 to 16\.4\.[Rannen\-Triki et al\. \[25\]](https://arxiv.org/html/2610.33025#bib.bib25)revisit the method on PG\-19 with 150M–1B models\. They update every 128 tokens with AdamW\. Their 512\-token context with adaptation matches a 2048\-token static context\. They also report that the gain concentrates under distribution shift\.[Hardt and Sun \[10\]](https://arxiv.org/html/2610.33025#bib.bib10)fine\-tune on retrieved neighbours of each test instance\. We kept their framing and their 128\-token update period\. The measurement moved to current open models at 1\.7B–27B parameters and to a 2k\-versus\-32k context control\. We ran three attention architectures\. Trained fast\-weight memories are a different line\. TTT layers\[[28](https://arxiv.org/html/2610.33025#bib.bib28)\]and Titans\[[3](https://arxiv.org/html/2610.33025#bib.bib3)\]learn a memory module whose update rule is part of pretraining, while TTT\-E2E\[[30](https://arxiv.org/html/2610.33025#bib.bib30)\]and In\-Place TTT\[[14](https://arxiv.org/html/2610.33025#bib.bib14)\]train the whole model to use its inference\-time updates and report retrieval gains after tens of billions of training tokens\. Our measurements are the zero\-training baseline those methods improve on\.

#### Linear attention as fast weights\.

The Qwen3\.5 and Qwen3\.8 series use Gated DeltaNet\[[32](https://arxiv.org/html/2610.33025#bib.bib32)\]and its hybrids with softmax attention\. These blocks keep a matrix\-valued state, updated by a gated delta rule\.[Liu et al\. \[31\]](https://arxiv.org/html/2610.33025#bib.bib31)and MesaNet\[[20](https://arxiv.org/html/2610.33025#bib.bib20)\]show formally that test\-time training with key–value binding is a form of learned linear attention\. This predicts that a gradient step on top of such blocks has little left to add\. We provide the measurement the prediction lacked\.

#### Diffusion language models at test time\.

We checked the test\-time work on masked\-diffusion LMs\. It concerns sampling\. Inference\-time scaling\[[12](https://arxiv.org/html/2610.33025#bib.bib12)\]is one line\. Remasking schedules and reward guidance are others\. Gradient adaptation of the denoiser on the document being processed has not been reported\. We adapt the recipe to the denoising loss and to the context/canvas interface of block\-diffusion models\.

#### Model editing and its cost\.

ROME\[[18](https://arxiv.org/html/2610.33025#bib.bib18)\]and MEMIT\[[19](https://arxiv.org/html/2610.33025#bib.bib19)\]insert facts into mid\-depth MLPs by closed\-form rank\-one updates\.[Gu et al\. \[8\]](https://arxiv.org/html/2610.33025#bib.bib8)show that editing under 1% of parameters can collapse general ability, and sequential editing compounds the damage\. Our next\-token fact writes reproduce that cost with SGD and add the dose curve and the split between format collapse and reasoning loss\.

#### Context distillation and self\-generated updates\.

Context distillation\[[1](https://arxiv.org/html/2610.33025#bib.bib1),[27](https://arxiv.org/html/2610.33025#bib.bib27)\]trains a model without a prompt to match its own outputs with the prompt\. It is bounded by the prompted model\.[Zheng et al\. \[5\]](https://arxiv.org/html/2610.33025#bib.bib5)apply it per context into LoRA adapters as a memory system\. SEAL\[[26](https://arxiv.org/html/2610.33025#bib.bib26)\]has the model write its own fine\-tuning data\. Our behaviour write is context distillation into LoRA or late MLP projections\. What we add is three things\. We compare it with a next\-token write on the same parameters at the same drift\. We cover the hybrid models\. And the distilled conclusion comes from an inquiry loop rather than a given document\.

## 3Setting and Method

### 3\.1Overview

Given a frozen pretrained language modelfθf\_\{\\theta\}and a documentx1:Tx\_\{1:T\}that exceeds the attention windowLL, the task is to predict each chunk of the document from the preceding window alone, and to answer questions about the document from the window alone\. We add a small writable parameter set𝐖\\mathbf\{W\}that is updated by gradient descent as the document is read \(Section[3\.2](https://arxiv.org/html/2610.33025#S3.SS2)\), fix a three\-condition protocol that separates what the window gives, what the full context gives, and what the updated weights give \(Section[3\.3](https://arxiv.org/html/2610.33025#S3.SS3)\), define two ways to write a specific conclusion into𝐖\\mathbf\{W\}\(Section[3\.4](https://arxiv.org/html/2610.33025#S3.SS4)\), and specify the inquiry loop that produces the conclusion \(Section[3\.5](https://arxiv.org/html/2610.33025#S3.SS5)\)\. Figure[1](https://arxiv.org/html/2610.33025#S3.F1)shows the reading mechanism and Figure[2](https://arxiv.org/html/2610.33025#S3.F2)the protocol\.

blocks1​…​34​N1\\ldots\\tfrac\{3\}\{4\}Nfrozenblocks34​N​…​N\\tfrac\{3\}\{4\}N\\ldots NMLPdown​\_​proj\\mathrm\{down\\\_proj\}writableLM headfrozenscore chunkctc\_\{t\}from window→\\rightarrowNLLt𝐖←𝐖−η​∇𝐖NLLt\\mathbf\{W\}\\leftarrow\\mathbf\{W\}\-\\eta\\,\\nabla\_\{\\mathbf\{W\}\}\\mathrm\{NLL\}\_\{t\}document streamchunks of 128 tokens,window 2kFigure 1:Inference\-time fast weights\. Each chunk is scored from a 2k window before the single SGD step that uses its loss; only the late MLP output projections move;𝐖\\mathbf\{W\}is reset per document\.
### 3\.2Inference\-time fast weights

The writable set is the MLP output projection of the last quarter of transformer blocks\. For a model withNNblocks this is\{down​\_​projℓ:ℓ≥⌊3​N/4⌋\}\\\{\\mathrm\{down\\\_proj\}\_\{\\ell\}:\\ell\\geq\\lfloor 3N/4\\rfloor\\\}; for Qwen3\-8B it is 453M parameters, for Nemotron\-Diffusion\-8B 470M, for DiffusionGemma\-26B\-A4B the last 8 of 30 encoder blocks, 48M\. The choice is fixed before any experiment and ablated in Section[4\.4](https://arxiv.org/html/2610.33025#S4.SS4): the first quarter gives nothing, the middle quarter nearly matches the last\.

We read the document in chunksc1,c2,…c\_\{1\},c\_\{2\},\\ldotsof 128 tokens\. 256 tokens went to legal text and diffusion models\. Benchmark documents used 512 tokens\. For chunkctc\_\{t\}the model computed the next\-token lossNLLt\\mathrm\{NLL\}\_\{t\}with the precedingL=2048L=2048tokens as context and the current𝐖\\mathbf\{W\}\. We recorded that loss as the score forctc\_\{t\}\. Only after that did we take one SGD step,𝐖←𝐖−η​∇𝐖NLLt\\mathbf\{W\}\\leftarrow\\mathbf\{W\}\-\\eta\\,\\nabla\_\{\\mathbf\{W\}\}\\mathrm\{NLL\}\_\{t\}\. Scoring before updating made every reported loss a genuine prediction of unseen text\. Per chunk we recomputed the window without a key–value cache, so the attention context stayed exactlyLLtokens throughout\. We kept a master copy of𝐖\\mathbf\{W\}in fp32 for bf16 models\. Updates of relative size10−410^\{\-4\}were otherwise lost to rounding\. At the start of each document we reset𝐖\\mathbf\{W\}to its pretrained value and logged the drift‖𝐖−𝐖0‖/‖𝐖0‖\\\|\\mathbf\{W\}\-\\mathbf\{W\}\_\{0\}\\\|/\\\|\\mathbf\{W\}\_\{0\}\\\|\. There was no momentum and no forgetting term\. We left the learning\-rate schedule out\. A surprise\-dependent gate and multiple inner steps were evaluated as ablations\. Both were rejected\.

For masked\-diffusion models we used the model’s own training objective as the loss\. We masked each chunk at a random ratiot∼U⁡\(0,1\)t\\sim U\(0,1\)\. The denoiser predicted the masked tokens from the unmasked ones plus the window\. The loss was the1/t1/t\-weighted masked cross\-entropy\. For block\-diffusion models with a separate context and canvas interface, the window entered as context and the chunk as the masked canvas\. The denoising loss gave larger gradients, so we clipped per\-step updates to a relative norm of 0\.5\.

### 3\.3Three\-condition protocol

whole document \(up to 64k\)frozenfθf\_\{\\theta\}answerFulllast 2kfrozenfθf\_\{\\theta\}answerWindowwhole document, read in chunksfθf\_\{\\theta\}with𝐖\\mathbf\{W\}updatinglast 2kfθf\_\{\\theta\}with updated𝐖\\mathbf\{W\}answerFast weightsFigure 2:The three\-condition protocol\. Full−\-Window is what the document holds beyond the window; FW−\-Window is what the updated weights recovered of it\. All three conditions score the same items\.We report every benchmark result as a triple on the same items\. Under*Frozen, full context*, the frozen model saw the whole document, up to 64k tokens, and answered\. For*Frozen, window*, we clipped the document to its lastLLtokens before the frozen model answered\. In*Fast weights, window*, we ran the model on the whole document in chunks with𝐖\\mathbf\{W\}updating\. It then answered from the lastLLtokens with the updated𝐖\\mathbf\{W\}\. The gap between the first two conditions is what the document contains that the window does not; the gap between the last two is what the weights recovered of it\. For perplexity we ran the same three conditions, with per\-chunk loss as the metric\. The full\-context condition was a single forward pass over the document at the model’s native window\.

Multiple\-choice tasks are scored by raw completion log\-likelihood of each option after the prompt, with no chat template\. Generation tasks use greedy decoding and exact match, SQuAD\-normalised F1, or the RULER criterion that every expected answer string must appear\. Every per\-item prediction is logged and every reported mean is recomputed from those records\.

### 3\.4Writing a specific conclusion

We had two objectives that could put a specific statementssinto𝐖\\mathbf\{W\}\. In the*text write*, we ran the fast\-weight step on the tokens ofssitself\. We repeated it forpppasses at rateη\\eta\. That is the mechanism of Section[3\.2](https://arxiv.org/html/2610.33025#S3.SS2)applied to a sentence\. For the*behaviour write*, we ran context distillation\. Withssin its prompt, the model labeled a set of fresh inputs\. We trained𝐖\\mathbf\{W\}, or a rank\-16 LoRA over all linear layers, with Adam on the soft KL between the model’s label distribution withoutssand the recorded labels\. Both writes moved the same parameters by a comparable relative amount\. They differed only in objective\. After each write, we ran a collateral probe\. We checked the model’s accuracy on a held\-out instruction unrelated toss, and its GSM8K, MMLU and IFEval scores\. Figure[3](https://arxiv.org/html/2610.33025#S3.F3)places both writes in the full loop\.

hypothesis bankposteriorp⁡\(h\)p\(h\)pick questionmax EIGenvironmentanswersjudge likelihoodp⁡\(ans∣h\)p\(\\text\{ans\}\\mid h\), few\-shotBayes update; stop atmaxh⁡p⁡\(h\)≥0\.9\\max\_\{h\}p\(h\)\\geq 0\.9conclusions=arg⁡maxh⁡p⁡\(h\)s=\\arg\\max\_\{h\}p\(h\)text writeSGD on tokens ofssbehaviour writess\-conditioned labels→\\rightarrowKL into𝐖\\mathbf\{W\}/LoRAnew cases, nossin promptAUROC vs true rulecollateral: other\-rule prompt\-following,GSM8K / MMLU / IFEvalFigure 3:The composition\. An inquiry loop finds a conclusion by information\-gain questioning; the conclusion is written into the weights by one of two objectives; both writes face the same behavioural and collateral probes\.
### 3\.5The inquiry loop

The conclusionsscomes from an inquiry loop over an explicit hypothesis bank\. The bank holdsKKcandidate hypotheses with a posteriorp⁡\(h\)p\(h\), uniform at the start\. Each observation\(q,a\)\(q,a\)updates the posterior by Bayes’ rule with the model’s own likelihoodp⁡\(a∣q,h\)p\(a\\mid q,h\), obtained by teacher\-forcing the yes/no answer under a prompt that stateshh\. A four\-shot prefix with rules outside the bank is part of that prompt; without it the models answer “yes” 95% of the time regardless ofhh, and the posterior is uninformative\. The next question is chosen from a candidate set by expected information gain,EIG⁡\(q\)=H⁡\[p⁡\(h\)\]−𝔼a​H​\[p⁡\(h∣q,a\)\]\\mathrm\{EIG\}\(q\)=H\[p\(h\)\]\-\\mathbb\{E\}\_\{a\}\\,H\[p\(h\\mid q,a\)\], with the expectation under the current predictive distribution\. The loop stops when the maximum posterior exceeds 0\.9 or after 30 questions, and emitss=arg⁡maxh⁡p⁡\(h\)s=\\arg\\max\_\{h\}p\(h\)\. In the environment used here the hypotheses are 24 enumerable rules over integer triples, the question candidates are objects from a space of 729, and an exact oracle likelihood is available as a reference for the judge\.

## 4Experiments

### 4\.1Setup

Models\.For softmax attention we ran Qwen3\-1\.7B, Qwen3\-4B, and Qwen3\-8B\[[22](https://arxiv.org/html/2610.33025#bib.bib22)\], plus Gemma\-4\-E4B\[[7](https://arxiv.org/html/2610.33025#bib.bib7)\]\. The hybrid Gated\-DeltaNet uses three of every four blocks as linear attention; we ran Qwen3\.5\-4B, Qwen3\.5\-9B, and Qwen3\.8\-27B\[[23](https://arxiv.org/html/2610.33025#bib.bib23)\]\. For masked diffusion we ran Nemotron\-Labs\-Diffusion\-8B\[[21](https://arxiv.org/html/2610.33025#bib.bib21)\], dense with block size 32, and DiffusionGemma\-26B\-A4B\-it\[[6](https://arxiv.org/html/2610.33025#bib.bib6)\], block diffusion with a context/canvas interface\. Qwen3\-1\.7B ran in fp32\. We kept everything else in bf16, with fp32 master copies of𝐖\\mathbf\{W\}\.

Data\.Adaptation is measured on the PG\-19 test split\[[24](https://arxiv.org/html/2610.33025#bib.bib24)\], 32k tokens per book \(16k for diffusion models\), and on a stream of U\.S\. court opinions of at least 30k characters from the Pile of Law CourtListener shard\[[11](https://arxiv.org/html/2610.33025#bib.bib11)\], de\-duplicated\. Benchmarks: RULER\[[13](https://arxiv.org/html/2610.33025#bib.bib13)\]at 4k and 16k, BABILong\[[17](https://arxiv.org/html/2610.33025#bib.bib17)\]16k qa1, LongBench\-v2\[[2](https://arxiv.org/html/2610.33025#bib.bib2)\], SCOTUS and CaseHOLD from LexGLUE\[[4](https://arxiv.org/html/2610.33025#bib.bib4),[35](https://arxiv.org/html/2610.33025#bib.bib35)\], LegalBench hearsay and consumer\-contracts QA\[[9](https://arxiv.org/html/2610.33025#bib.bib9)\], ContractNLI\[[15](https://arxiv.org/html/2610.33025#bib.bib15)\], HotpotQA\[[33](https://arxiv.org/html/2610.33025#bib.bib33)\], MuSiQue\[[29](https://arxiv.org/html/2610.33025#bib.bib29)\], CounterFact\[[18](https://arxiv.org/html/2610.33025#bib.bib18)\], zsRE from KnowEdit\[[34](https://arxiv.org/html/2610.33025#bib.bib34)\], and a citation\-completion task built from the court opinions \(the model must produce the next reporter citation given the opinion so far\)\. Retention uses GSM8K, MMLU and IFEval through lm\-eval\.

Compute\.All runs use two H100 NVL 96GB GPUs, about 19 GPU\-hours in total\. The adaptation pass costs 1\.34–1\.49×\\timesthe frozen pass in wall\-clock\.

### 4\.2Adaptation across architectures

Table 1:Inference\-time adaptation on PG\-19 and on court opinions\. Gain is the mean over documents of \(frozen NLL−\-fast\-weight NLL\) in nats per token with a 95% CI, at each model’s best learning rate; “improved” counts documents whose loss fell\. Window 2k, chunk 128 \(256 for opinions\), one SGD step per chunk, no training\.ModelAttentionnndocsη\\etaGain \(nats\)↑\\uparrowRel\.ImprovedDriftPG\-19 books, 32k tokens eachQwen3\-1\.7Bsoftmax9810−210^\{\-2\}\+0\.216±0\.007\+0\.216\\pm 0\.0077%98/980\.19%Qwen3\-4Bsoftmax2010−210^\{\-2\}\+0\.227±0\.017\+0\.227\\pm 0\.0178%20/200\.20%Gemma\-4\-E4Bsoftmax2010−310^\{\-3\}\+0\.145±0\.020\+0\.145\\pm 0\.0205%20/200\.17%Qwen3\.5\-4Bhybrid GDN2010−310^\{\-3\}\+0\.012±0\.002\+0\.012\\pm 0\.0020\.5%20/200\.12%Qwen3\.8\-27Bhybrid GDN1010−210^\{\-2\}\+0\.019±0\.006\+0\.019\\pm 0\.0060\.8%10/100\.21%Court opinions,≥\\geq30k charactersQwen3\-1\.7Bsoftmax10010−210^\{\-2\}\+0\.152±0\.004\+0\.152\\pm 0\.0047%100/1000\.11%Qwen3\-4Bsoftmax10010−210^\{\-2\}\+0\.143±0\.003\+0\.143\\pm 0\.0037%100/1000\.12%11223344556677889910102\.82\.8333\.23\.2NLL \(nats/token\)Qwen3\-1\.7B, softmax, 20 books11223344556677889910102\.62\.62\.82\.833Qwen3\-4B, softmax, 20 books11223344556677889910102\.82\.8333\.23\.2document decileNLL \(nats/token\)Gemma\-4\-E4B, softmax, 20 books11223344556677889910102\.22\.22\.32\.32\.42\.4document decileQwen3\.8\-27B, hybrid GDN, 10 books

Figure 4:Loss by document decile, frozen versus fast weights, at each model’s best rate\. On the softmax models the adapted curve sits below the frozen curve from the first decile and the gap widens through the book; the full\-context control on Qwen3\-1\.7B lies on top of the 2k\-window frozen curve\. On the hybrid 27B the two curves nearly coincide\.Softmax\-attention models gain 5–8% of their loss on every document\.Table[1](https://arxiv.org/html/2610.33025#S4.T1)gives the main result\. Figure[4](https://arxiv.org/html/2610.33025#S4.F4)shows the shape of the effect\. We ran Qwen3\-1\.7B atη=10−2\\eta=10^\{\-2\}on the PG\-19 test split\. It improves on all 98 long books by0\.2160\.216nats per token\. The minimum gain is0\.1420\.142and the maximum is0\.3960\.396\(Figure[5](https://arxiv.org/html/2610.33025#S4.F5), left\)\. Qwen3\-4B, the same architecture at a larger size, gains0\.2270\.227on 20 books\. Gemma\-4\-E4B, a different family, gains0\.1450\.145\. On court opinions the base loss is lower \(2\.3 vs 3\.1 nats\)\. The two Qwen3 models gain0\.150\.15nats on all 100 of 100 opinions\. Per\-opinion gains stay between0\.110\.11and0\.200\.20\(Figure[5](https://arxiv.org/html/2610.33025#S4.F5), right\)\. We checked the decile curves\. The frozen loss runs flat across the document, and the adapted loss is already lower in the first tenth, 3\.11 vs 2\.98 atη=10−2\\eta=10^\{\-2\}\. It keeps falling to the last tenth, 3\.10 vs 2\.86, a 16% drop in perplexity by the end of a book\.

0\.140\.140\.20\.20\.260\.260\.320\.320\.380\.380010102020min 0\.142per\-book gain \(nats/token\), Qwen3\-1\.7B, 98 booksbooks11223344556677889910102\.12\.12\.22\.22\.32\.32\.42\.4opinion decileNLL \(nats/token\)Court opinions, Qwen3\-1\.7B, 100 opinions

Figure 5:Left: distribution of per\-book gains over the 98 PG\-19 books \(bins of 0\.02 nats\); every book improves\. Right: loss by decile over 100 court opinions; the gain is present from the first tenth and stable thereafter\.The gain exceeds what the full 32k context gives\.We ran Qwen3\-1\.7B on the same 20 books\. The frozen model kept its native 32k window and scored 3\.091 nats\. We clipped the frozen model to 2k and scored 3\.093\. With fast weights at the 2k window, we scored 2\.922\. In Figure[4](https://arxiv.org/html/2610.33025#S4.F4)the full\-context curve lies on top of the 2k\-window curve at every decile\. Attention over the whole book added nothing to these books; the gradient steps removed 0\.17 nats\. The two mechanisms extract different things\. Attention copies; the gradient reweights the output distribution toward the document’s vocabulary and register\.

0\.0010\.0010\.010\.01000\.10\.10\.20\.2learning rateη\\etagain \(nats/token\)Qwen3\-1\.7B \(softmax\)Qwen3\-4B \(softmax\)Gemma\-4\-E4B \(softmax\)Qwen3\.5\-4B \(hybrid\)Qwen3\.8\-27B \(hybrid\)no changeFigure 6:Gain versus learning rate on PG\-19 for each model\. The three softmax models rise to0\.150\.15–0\.230\.23nats; the two hybrid models stay within0\.020\.02of zero at every rate and Qwen3\.5\-4B turns negative at10−210^\{\-2\}\(plotted at its mean over 12 books\)\.Hybrid linear\-attention models gain under 1%\.Figure[6](https://arxiv.org/html/2610.33025#S4.F6)shows the per\-model rate curves\. We ran Qwen3\.5\-4B at the softmax models’ rate of10−210^\{\-2\}\. It got worse on 12 of 12 books,−0\.02\-0\.02to−0\.08\-0\.08nats, drift 1\.0%\. At3×10−43\\times 10^\{\-4\},10−310^\{\-3\}and3×10−33\\times 10^\{\-3\}it gained0\.0060\.006,0\.0120\.012and0\.0120\.012nats\. The gains were statistically positive on 20 of 20 books and practically absent\. Qwen3\.8\-27B, the same hybrid design at 27B, gained0\.0080\.008at10−310^\{\-3\}and0\.0190\.019at10−210^\{\-2\}\. The contrast with Qwen3\-4B isolates the attention architecture as the cause\. Qwen3\-4B shares the training generation, tokenizer and size\. Gated DeltaNet blocks already carry a delta\-rule fast\-weight state that adapts to the document in context\. A late\-layer gradient step has little left to add\.

### 4\.3Masked\-diffusion language models

Table 2:Adaptation on masked\-diffusion language models\. Loss is the model’s own denoising objective \(1/t1/t\-weighted masked cross\-entropy; “plain CE” unweighted\), comparable within a model\. 16k tokens per document, chunk 256, window 2k, update clipping 0\.5\.112233445566778899101088101012121414document deciledenoising lossNemotron\-Diffusion\-8B, PG\-19, 10 books

Figure 7:Denoising loss by decile on a diffusion LM\. At10−310^\{\-3\}the second decile overshoots before the gain opens; at3×10−43\\times 10^\{\-4\}the adapted curve never rises above the frozen curve and ends 11% lower\. The first decile is low for both because the first chunks of a book are short front matter\.The same gradient step lowers the denoising loss of two diffusion families, at a lower rate\.We ran the first measurement of gradient adaptation on masked\-diffusion LMs\. Table[2](https://arxiv.org/html/2610.33025#S4.T2)reports it and Figure[7](https://arxiv.org/html/2610.33025#S4.F7)shows the decile shape\. At the autoregressive models’ rate of10−210^\{\-2\}without clipping, Nemotron\-Diffusion\-8B diverged on the first book\. Loss went 9\.3 to 41\.6, drift 11%\. The1/t1/t\-weighted objective has much larger gradients\. At3×10−43\\times 10^\{\-4\}with clipping it gained 8% on every book, no early overshoot, and a monotone decile curve; at10−310^\{\-3\}the first two deciles overshot before a 14% gain by the last\. The unweighted objective gave the same 7% relative gain, so the rate and not the weighting is what matters\. DiffusionGemma\-26B, through its context/canvas interface, gained 20% on books and 17% on opinions, on every document\. Its decile curve fell from 12\.1 to 10\.7 in the first tenth, from 11\.9 to 9\.1 in the last\. Both diffusion models are softmax\-attention denoisers\. The direction and the decile shape match the autoregressive softmax models\.

### 4\.4What controls the gain

Table 3:Ablations on Qwen3\-1\.7B, 20 PG\-19 books, 32k tokens, window 2k\. Gain in nats per token relative to the frozen model; the reference configuration is one SGD step atη=10−2\\eta=10^\{\-2\}on the last quarter of blocks with a constant rate\.firstmiddlelast−0\.1\-0\.1000\.10\.10\.20\.2−0\.032\-0\.0320\.1820\.1820\.2190\.219gain \(nats/token\)writable blocks,η=10−2\\eta=10^\{\-2\}constantsigmoidclip−0\.1\-0\.1000\.10\.10\.20\.20\.1710\.1710\.1440\.1440\.1440\.144rate gate,η=10−3\\eta=10^\{\-3\}1 step4 steps−0\.1\-0\.1000\.10\.10\.20\.20\.2190\.2190\.1830\.183inner steps,η=10−2\\eta=10^\{\-2\}Figure 8:Ablations on Qwen3\-1\.7B, 20 books\. Late blocks, a constant rate and a single step are each the best setting of their factor\.One large step on late blocks with a constant rate is the best configuration, and each alternative loses\.Table[3](https://arxiv.org/html/2610.33025#S4.T3)and Figure[8](https://arxiv.org/html/2610.33025#S4.F8)collect the ablations\. The gain rises with the rate up to10−210^\{\-2\}and at3×10−23\\times 10^\{\-2\}the first decile overshoots \(3\.14 against 3\.11 frozen\)\. Early blocks are useless and slightly harmful; middle blocks capture 83% of the late\-block gain, placing the adaptation on the output side of the network\. A surprise\-dependent gate, whether a sigmoid of the losszz\-score or a clipped version, gives the gain of a constant rate at about half the value, because its mean is about 0\.5 and the shape of token surprise carries nothing: the most surprising tokens in a court opinion are docket numbers\. Four inner steps with clipping lose to one step\. Each of these results repeats on the court\-opinion stream at 40 opinions: constant0\.0990\.099versus gated0\.0760\.076at10−310^\{\-3\}, and one step0\.1550\.155versus four steps0\.1190\.119at10−210^\{\-2\}\.

### 4\.5Retrieval from the updated weights

Table 4:Retrieval benchmarks under the three\-condition protocol,η=10−2\\eta=10^\{\-2\}, chunk 512, window 2k\. Full = frozen model with the whole document; window = frozen model with the last 2k tokens; FW = fast weights updated on the whole document, then answering from the window\. Accuracy unless marked; RULER uses the strict all\-answers criterion;nnitems per cell\. Context\-truncation loss is Full−\-Window; recovery is FW−\-Window\.Qwen3\.8\-27BQwen3\-4BTasknnFullWindowFWnnFullWindowFWRULER 4k1000\.870\.400\.381000\.840\.320\.31RULER 16k1000\.810\.180\.18BABILong 16k qa11000\.670\.230\.231000\.390\.110\.12LongBench\-v21490\.64†0\.420\.44SCOTUS issue area \(14\-way\)1000\.650\.400\.401000\.570\.530\.53ContractNLI \(3\-way\)1000\.690\.610\.601000\.540\.540\.56Citation completion, exact1770\.250\.210\.201580\.150\.110\.11HotpotQA F1 \(FW: passages read first\)1000\.810\.801000\.310\.31MuSiQue F1 \(FW: passages read first\)1000\.670\.671000\.220\.22CounterFact \(Full = fact in prompt\)1000\.760\.010\.041000\.760\.020\.02†Full context on the 56 of 60 items of the first LongBench\-v2 run that fit in memory\.RULER 4kBABILong 16kLongBench\-v2SCOTUSContractNLICitation exact000\.20\.20\.40\.40\.60\.60\.80\.811accuracyQwen3\.8\-27B,n≥100n\\geq 100per task

Figure 9:The retrieval result on the 27B in one picture\. The gap between the first and second bar of each task is what truncation to the window costs; the third bar is what the updated weights give back, which is nothing on every task\.Weights updated on the full document recover none of the accuracy lost to the window\.Table[4](https://arxiv.org/html/2610.33025#S4.T4)and Figure[9](https://arxiv.org/html/2610.33025#S4.F9)cover needle retrieval, multi\-hop reasoning, long\-document classification, long\-contract entailment, in\-document citation and single\-fact recall, on a 4B softmax model and a 27B hybrid model\. Truncating the context costs the 27B 47 points on RULER 4k, 44 on BABILong, 22 on LongBench\-v2, 25 on SCOTUS and 8 on ContractNLI\. In every case the fast\-weight condition is within two points of the window condition\. The one apparent positive, LongBench\-v2 atn=60n=60where fast weights recovered 5 points, shrank to 1\.3 points atn=149n=149with a different seed\. On multi\-hop QA, reading the passages into the weights before answering with them in context changes F1 by under half a point\. A single fast\-weight step on a new fact leaves the model at its no\-fact baseline\. Demonstrations fare the same: learning four CaseHOLD demonstrations into the weights, for one or ten passes, leaves zero\-shot accuracy at the zero\-shot value on both models \(27B: 0\.73 zero\-shot, 0\.72 after the write; 4B: 0\.58 and 0\.58\)\.

The negative is systematic, not a scorer artifact\.We ran an earlier version of the choice\-task scorer first, one that compared option\-letter probabilities after a chat template’s assistant turn\. The letters sit atlog⁡p≈−40\\log p\\approx\-40there\. That scorer produced a 3\-of\-7 recovery on SCOTUS\. With raw completion scoring the SCOTUS truncation gap is 25 points and the recovery is zero\. We kept the raw scorer for every choice task in Table[4](https://arxiv.org/html/2610.33025#S4.T4)\.

### 4\.6Writing facts: format, not knowledge

Table 5:Fact writes by the next\-token objective and their cost\. Top: recall of a written fact\. Bottom: Qwen3\-4B before and after a session of 30 sequential fact writes \(20 passes each\), and Qwen3\-8B before and after one behaviour write, on 200\-item retention probes\. Drift is the relative change of𝐖\\mathbf\{W\}\.1 step20 passesverbatim20 passesrephrasedin prompt000\.50\.5110\.030\.03110\.50\.50\.830\.83recallCounterFact recall, Qwen3\-4Bbaseη=×10−3\\eta=2\\\!\\times\\\!10^\{\-3\}η=10−2\\eta=10^\{\-2\}behaviourwrite \(8B\)000\.50\.5110\.870\.870\.50\.50\.490\.490\.90\.9accuracyGSM8K strict after 30 writesFigure 10:Left: a fact is stored only at a high dose and mostly in its training form\. Right: both storing doses of the next\-token write halve GSM8K; the behaviour write, with larger drift, leaves it unchanged \(8B base 0\.915\)\.Next\-token writes store the training format\.Table[5](https://arxiv.org/html/2610.33025#S4.T5)\(top\) and Figure[10](https://arxiv.org/html/2610.33025#S4.F10)\(left\) show the dose at which a fact can be stored at all: one step changes nothing \(recall 0\.03\), and 20 passes atη=10−2\\eta=10^\{\-2\}give verbatim recall of 1\.00 and 0\.50 on a rephrased question on CounterFact\. On zsRE with the 27B, the same dose gives 0\.78 on rephrased questions when both the write and the test use a rawQuestion/Answerlayout, and 0\.30 when the test question goes through the chat template, against 0\.80 with the fact in the prompt\. The write taught a completion pattern; the fact is available only in that pattern\.

The storing dose breaks generated reasoning at every rate tried\.Table[5](https://arxiv.org/html/2610.33025#S4.T5)\(bottom\) and Figure[10](https://arxiv.org/html/2610.33025#S4.F10)\(right\): after 30 sequential writes atη=10−2\\eta=10^\{\-2\}, which yield 0\.23 rephrased recall across the session, MMLU is unchanged while GSM8K strict falls from 0\.865 to 0\.485 and IFEval loses 4 points\. Of the 38\-point strict loss, 24 points are format collapse \(94 of 200 generations omit the answer marker\) and 14 points are reasoning loss on the lenient metric \(0\.86 to 0\.72\)\. A fifth of the rate stores less than half as much \(0\.10\) and still costs 36 strict and 11 lenient points\. Damage is not proportional to dose\. The behaviour write in the last row moves the same parameter set by a larger relative amount, learns its rule to AUROC 1\.00, and changes GSM8K by 1–1\.5 points inside the noise of 200 items\. The objective, not the amount of weight change, decides whether general ability survives\.

### 4\.7The inquiry loop

1\.7B4B8B4B, 16 obs\.000\.20\.20\.40\.40\.60\.60\.80\.8110\.60\.60\.680\.680\.760\.760\.760\.76top\-1judge quality: true rule is MAP \(oracle 0\.90\)random4B judgeEIG4B judgerandomoracleEIGoracle00551010151513\.8413\.849\.569\.568\.28\.255mean queriesquestions to solve, 50 gamesFigure 11:The inquiry loop\. Left: how often the posterior’s mode is the true rule after 8 observations, by judge size; 16 observations lift the 4B to the 8B’s level\. Right: information\-gain selection removes 31% of the questions with the 4B judge and 39% with exact likelihoods\.The loop’s quality is the judge’s per\-observation accuracy, and information\-gain selection pays\.Figure[11](https://arxiv.org/html/2610.33025#S4.F11)summarises the loop we ran on 50 games with 8 observations each\. With the few\-shot likelihood prompt, the true rule was the posterior mode in 60% of games for the 1\.7B judge, 68% for the 4B and 76% for the 8B, against an oracle of 90%\. Calibration error for the 8B was 0\.098\. The judges’ per\-observation accuracies came in at 0\.77, 0\.86 and about 0\.90, and the posterior tracked them\. We gave the 4B 16 observations instead of 8 and it lifted to 0\.76 against a 0\.92 oracle\. Temperature and per\-hypothesis bias correction did not move top\-1, because the error was noise rather than bias\. Expected\-information\-gain question selection solved the games in 9\.56 questions against 13\.84 for random selection with the 4B judge\. With the oracle likelihood, it solved the games in 5\.0 against 8\.2\. Disagreement\-based selection picked the same questions as EIG on binary answers\.

### 4\.8Writing conclusions as behaviour

Table 6:Writing a conclusion from the inquiry loop into the weights\. Hidden\-rule environment, balanced held\-out cases scored against the true rule, no rule in the prompt at test time\. All entries are AUROC against the true rule on held\-out cases\. “Ceiling” is the same model with the rule in the prompt; “collateral” is AUROC at following a different rule given in the prompt, before→\\rightarrowafter the write\. Teacher labels are the model’s own judgments with the rule in the prompt\.ModelWritenngamesChanceCeilingWrittenCollateralQwen3\-8Btext write of the rule, 20 passes,η=10−2\\eta=10^\{\-2\}300\.520\.850\.50Qwen3\-8Btext write of rule \+ gathered evidence300\.520\.850\.50Qwen3\-8Bbehaviour, late MLP, Adam2×10−52\\times 10^\{\-5\}, teacher200\.590\.880\.860\.77→0\.730\.77\\rightarrow 0\.73Qwen3\-8Bbehaviour, late MLP, oracle labels200\.590\.880\.97Qwen3\-8Bbehaviour, LoRA r16, Adam2×10−42\\times 10^\{\-4\}, oracle200\.590\.880\.9950\.77→0\.590\.77\\rightarrow 0\.59Qwen3\-8Bbehaviour, LoRA r16, Adam5×10−55\\times 10^\{\-5\}, teacher1000\.540\.840\.830\.82→0\.810\.82\\rightarrow 0\.81Qwen3\.5\-9Bbehaviour, LoRA r16, Adam5×10−55\\times 10^\{\-5\}, teacher1000\.530\.880\.880\.85→0\.830\.85\\rightarrow 0\.83000\.050\.050\.10\.10\.150\.150\.20\.20\.60\.60\.70\.70\.80\.80\.90\.911late MLP×10−65\\\!\\times\\\!10^\{\-6\}late MLP×10−52\\\!\\times\\\!10^\{\-5\}LoRA×10−55\\\!\\times\\\!10^\{\-5\}LoRA×10−42\\\!\\times\\\!10^\{\-4\}, teacherLoRA×10−42\\\!\\times\\\!10^\{\-4\}, oraclecollateral loss \(other\-rule prompt\-following, before−\-after\)written behaviour \(AUROC, no rule in prompt\)Qwen3\-8B, 20 games per pointlate\-MLP writeLoRA writein\-context ceilingno ruleFigure 12:Dose sets the trade\-off between transfer and collateral\. Gentle LoRA reaches the ceiling at 0\.01 collateral; four times the rate buys 0\.1 more transfer at 0\.18 collateral\.A conclusion written as text is inert; written as behaviour it reaches the in\-context ceiling\.We ran the hidden\-rule environment\. The inquiry loop found the true rule in every game with the oracle likelihood, at 5\.1 questions\. With the 8B judge it found the rule in 80% of games, at 9\.9 questions\. Table[6](https://arxiv.org/html/2610.33025#S4.T6)then covers writing the found rule into the weights\. We ran the text write of the rule statement, with or without the evidence the loop gathered\. Classification AUROC on new cases came in at 0\.50, at chance\. We checked that the model recited the statement perfectly afterwards\. Next we ran the behaviour write of the same rule on the same parameters\. It scored AUROC 0\.86 against the 0\.88 ceiling with teacher labels and 0\.97 with oracle labels\. At 100 games the gentle LoRA write landed within 0\.01 of the ceiling on a softmax model \(0\.827 vs 0\.839\) and a hybrid model \(0\.876 vs 0\.879\)\. Unrelated prompt\-following took 1–3 points of collateral\. Figure[12](https://arxiv.org/html/2610.33025#S4.F12)shows that the dose sets the collateral\. LoRA at four times the rate learned the rule to 0\.995\. It dropped 18 points of the ability to follow a different rule from the prompt\. The collateral probe is the gate that a write must pass\.

The write transfers to hybrid models at full strength, with a per\-model dose\.Qwen3\.5\-4B reaches 0\.96 with oracle labels and 0\.86 with its own labels, the same as the softmax 8B; at the softmax models’ rate its weights move three times more \(1\.8% vs 0\.6%\) and the collateral is 0\.14, and at a quarter of the rate the write is 0\.79 with collateral 0\.01\. Gentle LoRA on Qwen3\.5\-9B gives 0\.92 against a 0\.92 ceiling with collateral0\.834→0\.8270\.834\\rightarrow 0\.827\.

On a real legal task the same write scales with labelled data\.On CaseHOLD with Qwen3\-8B \(200 test items\), zero\-shot accuracy is 0\.690 and few\-shot 0\.665, so the model’s own labels have nothing to add and the teacher write lands at 0\.655\. With true labels, 256 examples through the late MLP projections give 0\.670 and through LoRA 0\.745; 1024 examples through LoRA give 0\.775,\+8\.5\+8\.5points\. Capacity location matters for real tasks: a rule over three integers fits in the output projections, a legal\-holding judgment needs the adapter across all layers\.

## 5Discussion

Why prediction improves and retrieval does not\.A next\-token gradient on a late MLP output projection moves the model’s output prior toward the token statistics of the recent text: vocabulary, names as frequent tokens, formatting, register\. That lowers the loss on every subsequent token of the same document, which is what the decile curves show, including the large gain in the first tenth\. Attention over 32k tokens can copy but does not reweight the output distribution, which is why the full\-context control gains nothing on the same books\. Retrieval needs a key\-to\-value binding that a handful of SGD steps on a late projection does not form\. TTT layers, Titans, TTT\-E2E and In\-Place TTT get binding because the memory or the whole model is trained to use its fast weights; the untrained setting measured here is their baseline, and its retrieval value is zero\.

Why the architecture decides\.A Gated\-DeltaNet block keeps a fast\-weight matrix,St=St−1​\(I−βt​kt​kt⊤\)\+βt​vt​kt⊤S\_\{t\}=S\_\{t\-1\}\(I\-\\beta\_\{t\}k\_\{t\}k\_\{t\}^\{\\top\}\)\+\\beta\_\{t\}v\_\{t\}k\_\{t\}^\{\\top\}\. The delta rule updates it as each token arrives\. That is the same operation a gradient step on a linear layer performs, applied at every token with learned gates\.[Liu et al\. \[31\]](https://arxiv.org/html/2610.33025#bib.bib31)show the equivalence formally\. We measured a 0\.3–0\.8% residual on hybrids\. It is the part of document statistics the recurrent state does not already carry\. Diffusion denoisers and autoregressive softmax models both lack such a state\. Both gain\. We drew the three cases side by side in Figure[13](https://arxiv.org/html/2610.33025#S5.F13)\.

softmax ARattentionMLPattentionMLP𝐖\\mathbf\{W\}gain 5–8%in\-context state: nonehybrid GDNGDNStS\_\{t\}GDNStS\_\{t\}attentionMLP𝐖\\mathbf\{W\}gain 0\.3–0\.8%StS\_\{t\}already adapts per tokenmasked diffusionattention \(bidir\.\)MLPattention \(bidir\.\)MLP𝐖\\mathbf\{W\}gain 8–20%in\-context state: noneFigure 13:Why the architecture decides\. The writable projections \(green\) sit at the top of each stack\. Only the hybrid carries a token\-by\-token fast\-weight stateStS\_\{t\}inside its linear\-attention blocks \(orange\), and only the hybrid shows no gain\.Why one objective destroys and the other preserves\.We ran next\-token loss on a repeated sentence\. It taught the output layer to emit that sentence\. That collides with the output layer’s role in every other generation\. The collapse of the GSM8K answer marker is that collision made visible\. A KL on a decision distribution teaches a decision\. It leaves the generation pathway alone\. The two writes differ in drift by less than a factor of two and in GSM8K cost by a factor of thirty\.

The design rule\.The measurements give a division of labour for a system that must keep learning after deployment\. The gradient path belongs on softmax and diffusion backbones\. It carries adaptation to the document at hand\. That costs 1\.3–1\.5×\\timesa frozen pass, with per\-document reset\. On hybrid backbones the gradient path can be left off\. The recurrent state already does this\. Conclusions go into the weights as behaviour, through a gentle low\-rank adapter trained on the system’s own conclusion\-conditioned judgments\. The collateral probe gates every write\. Facts go into an external store the model reads from context, because the one mechanism that stores them in these weights also breaks generation\. Training the fast\-weight pathway so that the frozen model learns to read what the gradient writes, and sourcing behaviour writes from inquiry on tasks where the model’s own judgments beat its zero\-shot answers, are the two directions this measurement sets up\.

## 6Conclusion

On the model families in use in 2026, we ran inference\-time gradient adaptation\. We used one untrained recipe and a protocol that separates context from weights\. Softmax\-attention and diffusion language models gained 5–20% of their loss over a document\. They outperformed full\-context attention on the same text\. Hybrid linear\-attention models gained under 1%\. Their recurrent state already performs the adaptation\. We checked retrieval atn≥100n\\geq 100on two model sizes\. The updated weights retrieved nothing on eight long\-context and knowledge benchmarks\. We wrote facts with the same objective\. That stored format and broke reasoning at every dose\. We wrote a conclusion as behaviour, sourced from an inquiry loop and distilled into a low\-rank adapter\. It reached the in\-context ceiling with reasoning intact\. For system designers, the adaptation finding shows where the gradient path pays and where the architecture has already paid for it\. The write findings tell them what to put in weights and what to keep in context\.

## References

- \[1\]A\. Askell, Y\. Bai, A\. Chen, et al\. A general language assistant as a laboratory for alignment\.*arXiv:2112\.00861*, 2021\.
- \[2\]Y\. Bai, S\. Tu, J\. Zhang, et al\. LongBench v2: Towards deeper understanding and reasoning on realistic long\-context multitasks\.*arXiv:2412\.15204*, 2024\.
- \[3\]A\. Behrouz, P\. Zhong, and V\. Mirrokni\. Titans: Learning to memorize at test time\.*arXiv:2501\.00663*, 2024\.
- \[4\]I\. Chalkidis, A\. Jana, D\. Hartung, et al\. LexGLUE: A benchmark dataset for legal language understanding in English\.*ACL*, 2022\.
- \[5\]Z\. Zheng, Z\. Li, X\. Wen, et al\. Context distillation as latent memory management\.*arXiv:2605\.28889*, 2026\.
- \[6\]Google\. DiffusionGemma 26B\-A4B\-it model card\.[https://huggingface\.co/google/diffusiongemma\-26B\-A4B\-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it), 2026\.
- \[7\]Google\. Gemma 4 E4B model card\.[https://huggingface\.co/google/gemma\-4\-E4B](https://huggingface.co/google/gemma-4-E4B), 2026\.
- \[8\]J\.\-C\. Gu, H\.\-X\. Xu, J\.\-Y\. Ma, et al\. Model editing harms general abilities of large language models\.*arXiv:2401\.04700*, 2024\.
- \[9\]N\. Guha, J\. Nyarko, D\. E\. Ho, et al\. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models\.*NeurIPS Datasets and Benchmarks*, 2023\.
- \[10\]M\. Hardt and Y\. Sun\. Test\-time training on nearest neighbors for large language models\.*ICLR*, 2024\.
- \[11\]P\. Henderson, M\. Krass, L\. Zheng, et al\. Pile of Law: Learning responsible data filtering from the law and a 256GB open\-source legal dataset\.*NeurIPS Datasets and Benchmarks*, 2022\.
- \[12\]J\. Lee, H\. Moon, K\. Zhai, et al\. Test\-time scaling in diffusion LLMs via hidden semi\-autoregressive experts\.*arXiv:2510\.05040*, 2025\.
- \[13\]C\.\-P\. Hsieh, S\. Sun, S\. Kriman, et al\. RULER: What’s the real context size of your long\-context language models?*arXiv:2404\.06654*, 2024\.
- \[14\]G\. Feng, S\. Luo, K\. Hua, et al\. In\-place test\-time training\.*arXiv:2604\.06169*, 2026\.
- \[15\]Y\. Koreeda and C\. D\. Manning\. ContractNLI: A dataset for document\-level natural language inference for contracts\.*Findings of EMNLP*, 2021\.
- \[16\]B\. Krause, E\. Kahembwe, I\. Murray, and S\. Renals\. Dynamic evaluation of transformer language models\.*arXiv:1904\.08378*, 2019\.
- \[17\]Y\. Kuratov, A\. Bulatov, P\. Anokhin, et al\. BABILong: Testing the limits of LLMs with long context reasoning\-in\-a\-haystack\.*NeurIPS*, 2024\.
- \[18\]K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov\. Locating and editing factual associations in GPT\.*NeurIPS*, 2022\.
- \[19\]K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau\. Mass\-editing memory in a transformer\.*ICLR*, 2023\.
- \[20\]J\. von Oswald, N\. Scherrer, S\. Kobayashi, et al\. MesaNet: Sequence modeling by locally optimal test\-time training\.*arXiv:2506\.05233*, 2025\.
- \[21\]NVIDIA\. Nemotron\-Labs\-Diffusion\-8B model card\.[https://huggingface\.co/nvidia/Nemotron\-Labs\-Diffusion\-8B](https://huggingface.co/nvidia/Nemotron-Labs-Diffusion-8B), 2026\.
- \[22\]Qwen Team\. Qwen3 technical report\.*arXiv:2505\.09388*, 2025\.
- \[23\]Qwen Team\. Qwen3\.5 and Qwen3\.8 model cards\.[https://huggingface\.co/Qwen/Qwen3\.8\-27B](https://huggingface.co/Qwen/Qwen3.8-27B), 2026\.
- \[24\]J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, and T\. P\. Lillicrap\. Compressive transformers for long\-range sequence modelling\.*arXiv:1911\.05507*, 2019\.
- \[25\]A\. Rannen\-Triki, J\. Bornschein, R\. Pascanu, et al\. Revisiting dynamic evaluation: Online adaptation for large language models\.*arXiv:2403\.01518*, 2024\.
- \[26\]A\. Zweiger, J\. Pari, H\. Guo, et al\. Self\-adapting language models\.*arXiv:2506\.10943*, 2025\.
- \[27\]C\. Snell, D\. Klein, and R\. Zhong\. Learning by distilling context\.*arXiv:2209\.15189*, 2022\.
- \[28\]Y\. Sun, X\. Li, K\. Dalal, et al\. Learning to \(learn at test time\): RNNs with expressive hidden states\.*arXiv:2407\.04620*, 2024\.
- \[29\]H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal\. MuSiQue: Multihop questions via single\-hop question composition\.*TACL*, 2022\.
- \[30\]A\. Tandon, K\. Dalal, X\. Li, et al\. End\-to\-end test\-time training for long context\.*arXiv:2512\.23675*, 2025\.
- \[31\]J\. Liu, S\. Elflein, O\. Litany, et al\. Test\-time training with KV binding is secretly linear attention\.*arXiv:2602\.21204*, 2026\.
- \[32\]S\. Yang, J\. Kautz, and A\. Hatamizadeh\. Gated Delta Networks: Improving Mamba2 with delta rule\.*arXiv:2412\.06464*, 2024\.
- \[33\]Z\. Yang, P\. Qi, S\. Zhang, et al\. HotpotQA: A dataset for diverse, explainable multi\-hop question answering\.*EMNLP*, 2018\.
- \[34\]N\. Zhang, Y\. Yao, B\. Tian, et al\. A comprehensive study of knowledge editing for large language models\.*arXiv:2401\.01286*, 2024\.
- \[35\]L\. Zheng, N\. Guha, B\. R\. Anderson, P\. Henderson, and D\. E\. Ho\. When does pretraining help? Assessing self\-supervised learning for law and the CaseHOLD dataset\.*ICAIL*, 2021\.

## Appendix AAdditional Results

Table 7:Learning\-rate sweeps for every model on PG\-19 \(gain in nats per token, 95% CI wheren≥10n\\geq 10, drift of𝐖\\mathbf\{W\}\)\. Diffusion rows use the denoising loss and update clipping\.Table 8:Dose curve for storing one fact by the next\-token objective \(CounterFact, Qwen3\-4B, 30 facts, recall on a rephrased question unless stated\)\.Table 9:Interference across a session of 30 sequential fact writes \(Qwen3\-4B, 20 passes each atη=10−2\\eta=10^\{\-2\}, no reset\)\.Table 10:Behaviour writes on hybrid models \(hidden\-rule environment, 20 games each\)\. All entries are AUROC; “collateral” is other\-rule prompt\-following before→\\rightarrowafter\.Table 11:Behaviour writes on real legal tasks \(Qwen3\-8B, raw option scoring\)\. Teacher labels are the model’s own few\-shot judgments; oracle labels are the dataset labels\.Table 12:Judge quality for the inquiry loop: 50 games, 24 rules, true rule is the posterior mode \(top\-1\), mean posterior on the true rule, and expected calibration error ofP⁡\(yes\)P\(\\text\{yes\}\)\.
## Appendix BClaim–Evidence Map